Flex lexical analyser

Infobox_Software
name = flex

developer = Vern Paxson
latest_release_version = 2.5.35
latest_release_date = February 26, 2008
operating_system = Unix-like
genre = Lexical analyzer generator
license = BSD license
website = [http://flex.sourceforge.net/ flex.sf.net]

flex (fast lexical analyzer generator) is a free software alternative to lex. It is frequently used with the free Bison parser generator. Flex was originally written in C by Vern Paxson around 1987.

A similar lexical scanner for C++ is flex++, which is included as part of the flex package.

Flex is a non-GNU project, but the GNU project developed the manual for Flex.

Example lexical analyzer

This is an example of a scanner (written in C) for the instructional programming language PL/0.

The symbols recognized are: '+', '-', '*', '/', '=', '(', ')', ',', ';', '.', ':=', '<', '<=', '<>', '>', '>=';numbers: 0-9 {0-9}; identifiers: a-zA-Z {a-zA-Z0-9} and keywords: begin, call, const, do, end, if, odd, procedure, then, var, while.

External variables used:FILE *source /* The source file */int cur_line, cur_col, err_line, err_col /* For error reporting */int num /* Last number read stored here, for the parser */char id [] /* Last identifier read stored here, for the parser */Hashtab *keywords /* List of keywords */

External routines called:error(const char msg [] ) /* Report an error */Hashtab *create_htab(int estimate) /* Create a lookup table */int enter_htab(Hashtab *ht, char name [] , void *data) /* Add an entry to a lookup table */Entry *find_htab(Hashtab *ht, char *s) /* Find an entry in a lookup table */void *get_htab_data(Entry *entry) /* Returns data from a lookup table */FILE *fopen(char fn [] , char mode [] ) /* Opens a file for reading */fgetc(FILE *stream) /* Read the next character from a stream */ungetc(int ch, FILE *stream) /* Put-back a character onto a stream */isdigit(int ch), isalpha(int ch), isalnum(int ch) /* Character classification */

External types:Symbol /* An enumerated type of all the symbols in the PL/0 language */Hashtab /* Represents a lookup table */Entry /* Represents an entry in the lookup table */

Scanning is started by calling init_scan, passing the name of the source file. If the source file is successfully opened, the parser calls getsym repeatedly to return successive symbols from the source file.

The heart of the scanner, getsym, should be straightforward. First, whitespace is skipped. Then the retrieved character is classified. If the character represents a multiple-character symbol, additional processing must be done. Numbers are converted to internal form, and identifiers are checked to see if they represent a keyword.

int read_ch(void) { int ch = fgetc(source); cur_col++; if (ch = ' ') { cur_line++; cur_col = 0; } return ch;}

void put_back(int ch) { ungetc(ch, source); cur_col--; if (ch = ' ') cur_line--;}

Symbol getsym(void) { int ch;

while ((ch = read_ch()) != EOF && ch <= ' ') ; err_line = cur_line; err_col = cur_col; switch (ch) { case EOF: return eof; case '+': return plus; case '-': return minus; case '*': return times; case '/': return slash; case '=': return eql; case '(': return lparen; case ')': return rparen; case ',': return comma; case ';': return semicolon; case '.': return period; case ':': ch = read_ch(); return (ch = '=') ? becomes : nul; case '<': ch = read_ch(); if (ch = '>') return neq; if (ch = '=') return leq; put_back(ch); return lss; case '>': ch = read_ch(); if (ch = '=') return geq; put_back(ch); return gtr; default: if (isdigit(ch)) { num = 0; do { /* no checking for overflow! */ num = 10 * num + ch - '0'; ch = read_ch(); } while ( ch != EOF && isdigit(ch)); put_back(ch); return number; } if (isalpha(ch)) { Entry *entry; id_len = 0; do { if (id_len < MAX_ID) { id [id_len] = (char)ch; id_len++; } ch = read_ch(); } while ( ch != EOF && isalnum(ch)); id [id_len] = '