HANDBOOK / REFLEX REFERENCE MANUAL /

# How To Describe a Token

For each token `Token` introduced in the Gentle specification, there must be a token description file `Token.t` . This file must specify a Lex rule that handles the token. Such a rule is given by a regular expression that matches the token, and an action that computes the attributes of the token.

       pattern { action }

If the rule does not fit on one line, the action may be written on several lines, but the opening brace must appear on the first line.

By way of an example, a token introduced in a Gentle specification as

       'token' Number (-> INT)

may be described in file `Number.t`:

       [0-9]+ {
          yylval.attr[1] = atoi(yytext);
          yysetpos();
          return Number;
       }

The pattern

       [0-9]+

matches a non-empty sequence of digits.

The line

       yylval.attr[1] = atoi(yytext);

converts the matched input token (`yytext`) into an integer (using the C function `atoi`) and assigns it as the first attribute. The Lex variable `yylval` has a field `attr` which is an array of token attributes. attr[ i] stands for the i-th attribute.

The macro

       yysetpos();

defines the source position of the token (the source position is stored in yylval.attr[0]). In a Gentle specification, this value may be accessed with the @-predicate.

The line

       return Number;

returns the code of the token. The token code is a constant that has the same name as the token.