<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
        <!-- -*- xhtml -*- -->
        <title>JavaCC Lexer Generator Integration Tutorial for NetBeans Platform 7.1</title>
        <link rel="stylesheet" type="text/css" href="https://netbeans.org/netbeans.css"/>
        <meta name="AUDIENCE" content="NBUSER"/>
        <meta name="TYPE" content="ARTICLE"/>
        <meta name="EXPIRES" content="N"/>
        <meta name="developer" content="gwielenga@netbeans.org"/>
        <meta name="indexed" content="y"/>
        <meta name="description"
              content="A short guide to integrating JavaCC into the NetBeans Platform."/>
        <!--      Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved. -->
        <!--     Use is subject to license terms.-->
    </head>
    <body>
        <h1>JavaCC Lexer Generator Integration Tutorial for NetBeans Platform 7.1</h1>

        <p>This tutorial shows you how to generate a lexer with <a href="http://javacc.java.net/">JavaCC</a>
            and use it to create editor features for
            applications created on top of the NetBeans Platform.</p>

        <b>Contents</b>

        <p><img src="../../images/articles/71/netbeans-stamp.png" class="stamp" width="114" height="114" alt="Content on this page applies to NetBeans IDE 7.1" title="Content on this page applies to NetBeans IDE 7.1"/></p>
        <ul class="toc">
            <li><a href="#intro">Introduction</a></li>
            <li><a href="#creating">Creating the Module</a></li>
            <li><a href="#recognizing">Recognizing SJ Files</a></li>
            <li><a href="#generating">Generating a Lexer from JavaCC</a></li>
            <li><a href="#integrating">Integrating the JavaCC Lexer with NetBeans APIs</a></li>
            <li><a href="#registering">Registering the NetBeans Lexer</a></li>
            <li><a href="#defining">Defining Tokens and Assigning Colors</a></li>
        </ul>

        <p><b>To follow this tutorial, you need the software and resources listed in the following
                table.</b></p>

        <table>
            <tbody>
                <tr>
                    <th class="tblheader" scope="col">Software or Resource</th>
                    <th class="tblheader" scope="col">Version Required</th>
                </tr>
                <tr>
                    <td class="tbltd1"><a href="https://netbeans.org/downloads/index.html">NetBeans IDE</a></td>
                    <td class="tbltd1">version 7.1 or above</td>
                </tr>
                <tr>
                    <td class="tbltd1"><a href="http://java.sun.com/javase/downloads/index.jsp">Java Developer Kit (JDK)</a></td>
                    <td class="tbltd1">version 6 or above</td>
                </tr>
            </tbody>
        </table>
        <p></p>
        <p class="notes">Though some explanations of JavaCC are given in this tutorial,
            the focus here is not on JavaCC as such. For background information and
            a thorough understanding of JavaCC, see the related website. The intended
        audience of this tutorial is the JavaCC user who would like to integrate a
        generated JavaCC lexer into NetBeans Platform applications. JavaCC is discussed
        only in so far as it is needed to set up a scenario for using the related
        NetBeans APIs.</p>
        <p class="tips">This tutorial is the official version of
            the first part of <a href="http://wiki.netbeans.org/How_to_create_support_for_a_new_language">http://wiki.netbeans.org/How_to_create_support_for_a_new_language</a>,
            which, aside from being a rough draft, is partly
            obsolete and out of date for NetBeans Platform 7.1.</p>
        
        <p>A basic overview of the steps to be taken during the instructions
            that follow can be seen here:</p>
        <p><object><iframe width="560" height="315" src="http://www.youtube.com/embed/aOwdeoOlZ7E" frameborder="0" allowfullscreen></iframe></object></p>

        <h2 class="tutorial"><a name="intro"></a>Introduction</h2>

        <p>JavaCC is a generator for Java lexers and Java parsers.
            A Java lexer and parser generator is a tool that reads a grammar
            specification and converts it to a Java program that can recognize
            matches to the grammar. </p>

        <p>This tutorial focuses specifically on lexers.
            Lexical analysis, or "lexing", is the process of
            converting a sequence of characters into a sequence of tokens.
            Functionality that performs lexical analysis is called a
            lexical analyzer, a lexer, or a scanner. In this tutorial,
            the term "lexer" will be used. A lexer typically
            exists as a single function called by a parser.
            For example, while a typical lexer recognizes parentheses as tokens,
            it does nothing to ensure that an opening parentheses is matched
            by a closing parentheses, which is a task for a parser.
        </p>
        <p class="tips"> Part 2 of this tutorial, the <a href="nbm-javacc-parser.html">JavaCC Parser Generator Integration Tutorial</a>,
            shows you how to create a parser that parses the same text
            that the lexer in this tutorial lexes.</p>

        <p>The NetBeans Platform
            provides hooks for integrating a lexer into your own Java desktop
            application on the NetBeans Platform. For example, in this tutorial,
            you use a sample grammar specification provided by JavaCC to generate a lexer and use
            it to create syntax coloring
            in an editor for a simplified
            Java language defined by the sample grammar specification:</p>

        <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/result-1.png" alt="editor."/></p>
        <p>Users of your editor will be able to customize syntax colors via the Options window
            in the application:</p>
        <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/result-2.png" alt="options."/></p>
        <p>The source code you will have by the end of this tutorial
            will be as follows:</p>
        <p><img src="../../images/tutorials/javacc/71/lexer-5.png" alt="source."/></p>

        <p>The principles you learn in this tutorial should help you integrate
            a JavaCC-based lexer of your own.</p>

        <!-- ===================================================================================== -->

        <h2 class="tutorial"><a name="creating"></a>Creating the Module</h2>

        <p>In this section, we use a wizard to create the source structure that every NetBeans module requires. The
            source structure consists of certain folders in specific places and a set of files that are
            always needed. For example, every NetBeans module requires a <tt>nbproject</tt> folder, which holds
            the project's metadata.</p>

<div class="indent">

        <ol>

            <li>Choose File &gt; New Project (Ctrl-Shift-N). Under Categories, select NetBeans Modules. Under Projects,
                select Module. Click Next.</li>
            <li><p>In the Name and Location panel, type <tt>SimpleJava</tt> in Project Name.
                    Change the
                    Project Location to any directory on your computer. Click Next.</p></li>

            <li><p>In the Basic Module Configuration panel, type <tt>org.simplejava</tt>
                    as the Code Name Base. Click Finish.</p></li></ol>

        <p> The IDE creates the <tt>SimpleJava</tt>
            project:</p>
        <p><img src="../../images/tutorials/javacc/71/code-1.png" alt="source."/></p>
    </div>
        <p>The project contains all of your sources and
            project metadata, such as the project's Ant build script. The project
            opens in the IDE. You can view its logical structure in the Projects
            window (Ctrl-1) and its
            file structure in the Files window (Ctrl-2). </p>

        <!-- ===================================================================================== -->
        <h2><a name="recognizing"></a>Recognizing SJ Files</h2>

        <p>In this section, we use a wizard to create the classes necessary for
            recognizing "SimpleJava" files, for which we will create the
            new "text/x-sj" MIME type, which will recognize files with
            ".sj" as their file extension.</p>

<div class="indent">
        <ol>

            <li>Right-click the project node and
                choose New &gt; Other &gt; Module Development &gt; File Type.</li>
            <li>In the File Recognition panel, do the following:
                <ul>
                    <li>Type <tt>text/x-sj</tt> in the MIME Type edit box.</li>
                    <li>Type <tt>.sj .SJ</tt> in the by Filename Extension edit box.</li>
                </ul>
                <p>The File Recognition panel should now look as follows:</p>
                <p><img src="../../images/tutorials/javacc/71/code-2.png" alt="source."/></p>

                <p>Note the following about the fields in the File Recognition panel:</p>
                <ul>
                    <li><b>MIME Type.</b> Specifies the data object's unique MIME type.</li>
                    <li><b>Filename Extension.</b> Specifies one or more file extensions that the IDE will recognize
                        as belonging to the specified MIME type. The file extension can optionally be preceded
                        by a dot. Separators are commas, spaces, or both. Therefore, all of the following are valid:
                        <ul><li><tt>.abc,.def</tt></li>
                            <li><tt>.abc .def</tt></li>
                            <li><tt>abc def</tt></li>
                            <li><tt>abc,.def ghi, .wow</tt></li></ul>
                        <p>Let's imagine that SJ files are not case-sensitive. For
                            this reason, you specify <i>two</i> file extensions
                            in this tutorial&#8212;<tt>.sj</tt> and <tt>.SJ</tt>.</p></li>
                </ul>

                <p>Click Next.</p></li>

            <li><p>In the Name and Location panel, type <tt>Abc</tt> as the Class Name Prefix
                    and browse to any 16x16 pixel image file as the new file type's icon,
                    as shown below.</p>

                <p><img src="../../images/tutorials/javacc/71/code-3.png" alt="source."/></p>

                <p class="notes"><b>Note:</b> You can use any icon of a 16x16 pixel dimension. If you like, you can
                    right-click on this one and save it locally, and then
                    specify it in the wizard step above:
                    <img src="../../images/tutorials/javacc/71/JavaIcon.gif" alt="gif."/></p>
            </li>
            <li><p>Click Finish.</p>

                <p>The Projects window should now look as follows:</p>

                <p><img src="../../images/tutorials/javacc/71/code-4.png" alt="source."/></p>

                <p>Each of the newly generated files is briefly introduced:</p>
                <ul>
                    <li><b>SJDataObject.java.</b> Represents one or more file types, letting
                        you parse and process the underlying file content. For more information, see <a href="http://wiki.netbeans.org/wiki/view/DevFaqDataObject">What is a DataObject?</a>.</li>
                    <li><b>SJResolver.xml.</b> Maps the <tt>.sj</tt> and <tt>.SJ</tt> extensions to the MIME type.
                        The MIME type resolver,
                        uses <a href="http://bits.netbeans.org/dev/javadoc/org-openide-filesystems/org/openide/filesystems/doc-files/resolverDocumentation.html">these</a>
                        rules to determine whether a file is recognized.</li>
                    <li><b>SJTemplate.sj.</b> Provides the basis of a file template that is registered
                        in the system via the <tt>package-info.java</tt>
                        class, which has been generated into the module as can be seen in the screenshot above,
                        such that it will be installed in the New File dialog
                        as a new template.</li>
                    <li><b>SJVisualElement.java.</b> Is a <tt>JPanel</tt> annotated such that it
                        will integrate into your editor for the <tt>text/x-sj</tt> file type, providing
                        a first visual panel. Copy it and tweak its annotations to begin creating
                        additional visual panels.</li>
                </ul>

                <p>In the <tt>layer.xml</tt> file, you should see the following:</p>

                <pre class="examplecode">&lt;folder name="Loaders"&gt;
    &lt;folder name="text"&gt;
        &lt;folder name="x-sj"&gt;
            &lt;folder name="Actions"&gt;
                &lt;file name="org-openide-actions-CopyAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/Edit/org-openide-actions-CopyAction.instance"/&gt;
                    &lt;attr name="position" intvalue="400"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-CutAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/Edit/org-openide-actions-CutAction.instance"/&gt;
                    &lt;attr name="position" intvalue="300"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-DeleteAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/Edit/org-openide-actions-DeleteAction.instance"/&gt;
                    &lt;attr name="position" intvalue="600"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-FileSystemAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/System/org-openide-actions-FileSystemAction.instance"/&gt;
                    &lt;attr name="position" intvalue="1100"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-OpenAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/System/org-openide-actions-OpenAction.instance"/&gt;
                    &lt;attr name="position" intvalue="100"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-PropertiesAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/System/org-openide-actions-PropertiesAction.instance"/&gt;
                    &lt;attr name="position" intvalue="1400"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-RenameAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/System/org-openide-actions-RenameAction.instance"/&gt;
                    &lt;attr name="position" intvalue="700"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-SaveAsTemplateAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/System/org-openide-actions-SaveAsTemplateAction.instance"/&gt;
                    &lt;attr name="position" intvalue="900"/&gt;
                &lt;/file&gt;
                &lt;file name="org-openide-actions-ToolsAction.shadow"&gt;
                    &lt;attr name="originalFile" stringvalue="Actions/System/org-openide-actions-ToolsAction.instance"/&gt;
                    &lt;attr name="position" intvalue="1300"/&gt;
                &lt;/file&gt;
                &lt;file name="sep-1.instance"&gt;
                    &lt;attr name="instanceClass" stringvalue="javax.swing.JSeparator"/&gt;
                    &lt;attr name="position" intvalue="200"/&gt;
                &lt;/file&gt;
                &lt;file name="sep-2.instance"&gt;
                    &lt;attr name="instanceClass" stringvalue="javax.swing.JSeparator"/&gt;
                    &lt;attr name="position" intvalue="500"/&gt;
                &lt;/file&gt;
                &lt;file name="sep-3.instance"&gt;
                    &lt;attr name="instanceClass" stringvalue="javax.swing.JSeparator"/&gt;
                    &lt;attr name="position" intvalue="800"/&gt;
                &lt;/file&gt;
                &lt;file name="sep-4.instance"&gt;
                    &lt;attr name="instanceClass" stringvalue="javax.swing.JSeparator"/&gt;
                    &lt;attr name="position" intvalue="1000"/&gt;
                &lt;/file&gt;
                &lt;file name="sep-5.instance"&gt;
                    &lt;attr name="instanceClass" stringvalue="javax.swing.JSeparator"/&gt;
                    &lt;attr name="position" intvalue="1200"/&gt;
                &lt;/file&gt;
            &lt;/folder&gt;
            &lt;folder name="Factories"&gt;
                &lt;file name="SJDataLoader.instance"&gt;
                    &lt;attr name="dataObjectClass" stringvalue="org.simplejava.SJDataObject"/&gt;
                    &lt;attr name="iconBase" stringvalue="org/simplejava/JavaIcon.gif"/&gt;
                    &lt;attr name="instanceCreate" methodvalue="org.openide.loaders.DataLoaderPool.factory"/&gt;
                    &lt;attr name="mimeType" stringvalue="text/x-sj"/&gt;
                &lt;/file&gt;
            &lt;/folder&gt;
        &lt;/folder&gt;
    &lt;/folder&gt;
&lt;/folder&gt;
&lt;folder name="Services"&gt;
    &lt;folder name="MIMEResolver"&gt;
        &lt;file name="SJResolver.xml" url="SJResolver.xml"&gt;
            &lt;attr name="displayName" bundlevalue="org.simplejava.Bundle#Services/MIMEResolver/SJResolver.xml"/&gt;
        &lt;/file&gt;
    &lt;/folder&gt;
&lt;/folder&gt;</pre>

<p class="notes"><b>Note:</b> Each <tt>&lt;file&gt;</tt> under <tt>&lt;folder name="Actions"&gt;</tt> associates the files of the <tt>text/x-sj</tt> mime type with one of the predefined actions provided by the NetBeans Platform.
A "<tt>.shadow</tt>" file means that the action specification is actually somewhere else in the
NetBeans virtual filesystem, as specified by the <tt>originalFile</tt> attribute. To add more actions for SJ files,
see the <a href="https://platform.netbeans.org/tutorials/nbm-filetype.html">File Type Integration Tutorial</a>.</p>

	    </li>
            <li><p>Right-click the module and choose Run. The application starts up, installing
                    your module. Go to the Favorites window (Ctrl-3) and open a few
                    files on computer that have ".sj" as their file extension.</p>
                <p>You should see your icon is shown for the file type, the Actions
                    registered in the <tt>layer.xml</tt> are shown when you right-click
                    the file in the Favorites window, and three tabs are shown in
                    the editor (the source tab, the visual tab created above, and the history tab):</p>
                <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/code-5.png" alt="source."/></p>
                <p>Create a new Java project and then go to the New File dialog, where you should
                    see that your template has been registered:</p>
                <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/code-6.png" alt="source."/></p>
            </li>
        </ol>
</div>
        <p>You now have a basic file type recognition infrastructure set up. In the next section, we generate
            a lexer via JavaCC and then use it to add syntax coloring for the SJ file type.</p>
        <!-- ======================================================================================= -->

        <h2><a name="generating"></a>Generating a Lexer from JavaCC</h2>
        <p>Let's now use JavaCC to generate a lexer!</p>
        <div class="indent">

            <ol>
                <li><p>Download "javacc-4.0.zip" from
                        "https://javacc.dev.java.net"
                        and unpack it to a folder somewhere in your computer, named,
                        for example, "/myjavacc40":</p>
                    <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/javacc-1.png" alt="source."/></p>
                    <p>In this section, for purposes of this example, we will use the grammar
                        specified in the <tt>Java1.5.jj</tt> file,
                        together with the <tt>Token.java</tt> file from the example</p>
                    <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/javacc-2.png" alt="source."/></p>
                    <p class="notes"><b>Note:</b> You can try to use a different version of JavaCC, but there could
                        be a differences in the generated files making the result
                        incompatible with the sections that follow. </p></li>
                <li><p>Create a new package named <tt>org.simplejava.jcclexer</tt> in your project.
                        Copy the two files mentioned above, <tt>Java1.5.jj</tt>
                        and <tt>Token.java</tt>, into the
                        new package:</p>
                    <p><img src="../../images/tutorials/javacc/71/javacc-3.png" alt="source."/></p>
                    <p class="notes"><b>Note:</b> The <tt>Token</tt> class does not yet compile
                        because the class it references, <tt>JavaParserConstants</tt>, does not
                        yet exist. It will be generated at the end of this section of the tutorial
                        by JavaCC and at that stage the <tt>Token</tt> class will compile.</p>
                    <p>We're now going to tweak the <tt>Java1.5.jj</tt> file so that it fits our
                        specific needs.</p>
                </li>
                <li><p>Firstly, we need to make sure that the classes
                        that JavaCC will generate for us will be generated into the correct
                        package, that is, the package where we copied the two files above.
                        Add "package org.simplejava.jcclexer;"
                        to <tt>Java1.5.jj</tt> file after the "PARSER_BEGIN(JavaParser)" line, so
                        that the files will be generated in the correct package:</p>

                    <pre class="examplecode">PARSER_BEGIN(JavaParser)

<b>package org.simplejava.jcclexer;</b>

import java.io.*;</pre></li>
                <li><p>The <tt>Java1.5.jj</tt> file contains the
                        descriptions of tokens for the Java parser.
                        That's nearly what we need for our own Java lexer, though
                        there are some differences. The lexer defined for the parser
                        hides some types of tokens, such as comments and whitespaces.
                        However, we need to see such tokens in the NetBeans lexer
                        because we want to define special colors for comments.
                        Therefore, we need to change that in our JavaCC file. </p>
                    <ul><li><p>Change:</p>
                            <pre class="examplecode">SKIP :
{
  " "
| "\t"
| "\n"
| "\r"
| "\f"
}</pre>

                            to:

                            <pre class="examplecode">TOKEN :
{
  &lt; WHITESPACE:
  " "
| "\t"
| "\n"
| "\r"
| "\f">
}</pre>
                        </li>
                        <li><p>For the same reason, change all SPECIAL_TOKEN definitions:</p>

                            <pre class="examplecode">SPECIAL_TOKEN :
{
  &lt;SINGLE_LINE_COMMENT: "//" (~["\n","\r"])* ("\n" | "\r" | "\r\n")?&gt;
}

&lt;IN_FORMAL_COMMENT&gt;
SPECIAL_TOKEN :
{
  &lt;FORMAL_COMMENT: "*/" &gt; : DEFAULT
}

&lt;IN_MULTI_LINE_COMMENT&gt;
SPECIAL_TOKEN :
{
  &lt;MULTI_LINE_COMMENT: "*/" &gt; : DEFAULT
}</pre>

                            <p>to TOKEN definitions:</p>

                            <pre class="examplecode">TOKEN :
{
  &lt;SINGLE_LINE_COMMENT: "//" (~["\n","\r"])* ("\n" | "\r" | "\r\n")?&gt;
}

&lt;IN_FORMAL_COMMENT&gt;
TOKEN :
{
  &lt;FORMAL_COMMENT: "*/" &gt; : DEFAULT
}

&lt;IN_MULTI_LINE_COMMENT&gt;
TOKEN :
{
  &lt;MULTI_LINE_COMMENT: "*/" &gt; : DEFAULT
}</pre>

                        </li></ul></li>
                <li><p>Because we will use our <tt>Java1.5.jj</tt> grammar
                        file to create a lexer only, we can simplify it.
                        Add this line, which sets the BUILD_PARSER property to false:</p>

                    <pre class="examplecode">options {
  JAVA_UNICODE_ESCAPE = true;
  ERROR_REPORTING = false;
  STATIC = false;
  JDK_VERSION = "1.5";
  <b>BUILD_PARSER = false;</b>
}</pre>

                </li>

                <li><p>Part of the <tt>Java1.5.jj</tt> file is obsolete for our purposes, so let's delete some sections.
                        Firstly, keep the PARSER_BEGIN and PARSER_END
                        sections, but delete the JavaParser class body, so that you're
                        left with exactly this:</p>

                    <pre class="examplecode">PARSER_BEGIN(JavaParser)

package org.simplejava.jcclexer;

public class JavaParser {}

PARSER_END(JavaParser)</pre>
                    <p class="notes"><b>Note:</b> Though the parser start and end lines remain, the body of the
                        class should now be empty and have exactly the content shown above.</p>
                    <p>Also delete everything from these lines down to the end of the file:</p>

                    <pre class="examplecode">/*****************************************
 * THE JAVA LANGUAGE GRAMMAR STARTS HERE *
 *****************************************/</pre>


                </li>

                <li><p>The <tt>Java1.5.jj</tt> file is ready now and we can "compile" it from the command line:</p>

                    <pre class="examplecode">cd /myprojects/simplejava/src/org/simplejava/jcclexer
/myjavacc40/bin/javacc Java1.5.jj</pre>
                    <p>The result should be as follows:</p>
                    <p><img src="../../images/tutorials/javacc/71/code-7.png" alt="source."/></p>
                    <p>As you can see, JavaCC has generated
                        the following files:</p>
                    <ul>
                        <li>JavaCharStream.java</li>
                        <li>JavaParserConstants.java</li>
                        <li>JavaParserTokenManager.java</li>
                        <li>ParseException.java</li>
                        <li>TokenMgrError.java</li>
                    </ul>
                    <p>All the files should be compilable, that is, there should be no error
                        marks anywhere in the module, as can be seen in the screenshot above.</p>
                </li>

            </ol>

            <p>You've now completed the JavaCC part of the tutorial.
                The time has come to use the generated files to create a new NetBeans Lexer plugin.</p>

        </div>


        <!-- ======================================================================================= -->

        <h2><a name="integrating"></a>Integrating the JavaCC Lexer with NetBeans APIs</h2>
        <p>In this section, we take the files generated in the previous section
            and integrate them with the <a href="http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-lexer/overview-summary.html">NetBeans Lexer API</a>.</p>
        <div class="indent">
            <ol>
                <li><p>In the Projects window, right-click the Libraries node, and choose
                        Add Module Dependency, as shown below:</p>
                    <p><img src="../../images/tutorials/javacc/71/lexer-1.png" alt="source."/></p>
                    <p>Look for the "Lexer" module in the list:</p>
                    <p><img src="../../images/tutorials/javacc/71/lexer-2.png" alt="source."/></p>
                    <p>When you click OK, you should see the "Lexer" module is now
                        a dependency in your module:</p>
                    <p><img src="../../images/tutorials/javacc/71/lexer-3.png" alt="source."/></p>
                </li>
                <li><p>In your module, create a new package named <tt>org.simplejava.lexer</tt>.</p>
                </li>
                <li><p>The first class you need to implement is <tt><a href="http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-lexer/org/netbeans/api/lexer/TokenId.html">org.netbeans.api.lexer.TokenId</a></tt>.
                        <tt>TokenId</tt> represents one type of token.
                        It has three properties:</p>
                    <ul>
                        <li><tt>name.</tt> Unique name of the token type, such as <tt>"KEYWORD_IF".</tt></li>
                        <li><tt>id.</tt> Unique number.</li>
                        <li><tt>primaryCategory.</tt> Used
                            for sharing a token coloring
                            among multiple token types.</li>
                    </ul>
                    <p>Create a class named <tt>SJTokenId</tt> and define it as follows:</p>
                    <pre class="examplecode">package org.simplejava.lexer;

import org.netbeans.api.lexer.TokenId;

public class SJTokenId implements TokenId {

    private final String name;
    private final String primaryCategory;
    private final int id;

    SJTokenId(
            String name,
            String primaryCategory,
            int id) {
        this.name = name;
        this.primaryCategory = primaryCategory;
        this.id = id;
    }

    @Override
    public String primaryCategory() {
        return primaryCategory;
    }

    @Override
    public int ordinal() {
        return id;
    }

    @Override
    public String name() {
        return name;
    }

}</pre></li>
                <li><p>The next class you need to implement is <tt><a href="http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-lexer/org/netbeans/spi/lexer/LanguageHierarchy.html">org.netbeans.spi.lexer.LanguageHierarchy</a></tt>.
                        <tt>LanguageHierarchy</tt> provides a list of token types for our language and creates a new
                        instance of our lexer.</p>
                    <p>Create a class named <tt>SJLanguageHierarchy</tt> and define it as follows:</p>
                    <pre class="examplecode">package org.simplejava.lexer;

import java.util.*;
import org.netbeans.spi.lexer.LanguageHierarchy;
import org.netbeans.spi.lexer.Lexer;
import org.netbeans.spi.lexer.LexerRestartInfo;

public class SJLanguageHierarchy extends LanguageHierarchy&lt;SJTokenId&gt; {

    private static List&lt;SJTokenId&gt; tokens;
    private static Map&lt;Integer, SJTokenId&gt; idToToken;

    private static void init() {
        tokens = Arrays.&lt;SJTokenId&gt;asList(new SJTokenId[]{
            //[PENDING]
        });
        idToToken = new HashMap&lt;Integer, SJTokenId&gt;();
        for (SJTokenId token : tokens) {
            idToToken.put(token.ordinal(), token);
        }
    }

    static synchronized SJTokenId getToken(int id) {
        if (idToToken == null) {
            init();
        }
        return idToToken.get(id);
    }

    @Override
    protected synchronized Collection&lt;SJTokenId&gt; createTokenIds() {
        if (tokens == null) {
            init();
        }
        return tokens;
    }

    @Override
    protected synchronized Lexer&lt;SJTokenId&gt; createLexer(LexerRestartInfo&lt;SJTokenId&gt; info) {
        return new SJLexer(info);
    }

    @Override
    protected String mimeType() {
        return "text/x-sj";
    }

}</pre>
                    <p class="notes"><b>Note:</b> Because the <tt>SJLexer</tt> class does not
                        yet exist, a red error mark is shown in the NetBeans editor in the new
                        declaration for the non-existent <tt>SJLexer</tt> class. You will define
                        this class in the next step.</p>
                </li>

                <li><p>The last class you need to implement is <tt><a href="http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-lexer/org/netbeans/spi/lexer/Lexer.html">org.netbeans.spi.lexer.Lexer</a></tt>.
                        <tt>Lexer</tt> reads input text and returns tokens for it.
                        In our case, the Lexer implementation needs to delegate
                        to the lexer generated by JavaCC.</p>
                    <p>Create a class named <tt>SJLexer</tt> and define it as follows:</p>
                    <pre class="examplecode">package org.simplejava.lexer;

import org.netbeans.spi.lexer.Lexer;
import org.netbeans.spi.lexer.LexerRestartInfo;
import org.simplejava.jcclexer.JavaCharStream;
import org.simplejava.jcclexer.JavaParserTokenManager;
import org.simplejava.jcclexer.Token;

class SJLexer implements Lexer&lt;SJTokenId&gt; {

    private LexerRestartInfo&lt;SJTokenId&gt; info;
    private JavaParserTokenManager javaParserTokenManager;

    SJLexer(LexerRestartInfo&lt;SJTokenId&gt; info) {
        this.info = info;
        JavaCharStream stream = new JavaCharStream(info.input());
        javaParserTokenManager = new JavaParserTokenManager(stream);
    }

    @Override
    public org.netbeans.api.lexer.Token&lt;SJTokenId&gt; nextToken() {
        Token token = javaParserTokenManager.getNextToken();
        if (info.input().readLength() &lt; 1) {
            return null;
        }
        return info.tokenFactory().createToken(SJLanguageHierarchy.getToken(token.kind));
    }

    @Override
    public Object state() {
        return null;
    }

    @Override
    public void release() {
    }

}</pre>
                    <p class="notes"><b>Note:</b> The class above does not compile at the moment
                        because  <tt>JavaCharStream</tt> has not been defined to receive
                        a <tt>LexerInput</tt>. In the next step, we rewrite the
                        <tt>JavaCharStream</tt> class generated by JavaCC.
                        Our new version of <tt>JavaCharStream</tt>, listed in the next step,
                        reads input characters from <tt><a href="http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-lexer/org/netbeans/spi/lexer/LexerInput.html">org.netbeans.spi.lexer.LexerInput</a></tt>, instead of the
                        standard <tt>InputStream</tt>.</p>
                </li>
                <li><p>Because we are now using
                        <tt>LexerInput</tt> instead of <tt>InputStream</tt>, another change
                        in <tt>JavaCharStream</tt> is needed because
                        the <tt>JavaParserTokenManager</tt> created by JavaCC is designed to
                        work with a <tt>java.io.Reader</tt> and recognizes a &lt;EOF&gt;
                        when the <tt>io.Reader</tt> throws an <tt>IOException</tt>.
                        However, though the <tt>LexerInput</tt> class logically corresponds to
                        <tt>java.io.Reader</tt>, its <tt>read()</tt>
                        method does not throw a checked exception.
                        Hence the <tt>BeginToken</tt> and the <tt>readChar</tt> methods below validate the
                        returned character and throw the exception, if necesary.</p>

                    <pre class="examplecode">package org.simplejava.jcclexer;

import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
import java.io.UnsupportedEncodingException;
import org.netbeans.spi.lexer.LexerInput;

public class JavaCharStream {

    private LexerInput input;

    static boolean staticFlag;

    public JavaCharStream(LexerInput input) {
        this.input = input;
    }

    JavaCharStream(Reader stream, int i, int i0) {
        throw new UnsupportedOperationException("Not yet implemented");
    }

    JavaCharStream(InputStream stream, String encoding, int i, int i0) throws UnsupportedEncodingException {
        throw new UnsupportedOperationException("Not yet implemented");
    }

    char BeginToken() throws IOException {
        return readChar();
    }

    String GetImage() {
        return input.readText().toString();
    }

    public char[] GetSuffix(int len) {
        if (len > input.readLength()) {
            throw new IllegalArgumentException();
        }
        return input.readText(input.readLength() - len, input.readLength()).toString().toCharArray();
    }

    void ReInit(Reader stream, int i, int i0) {
        throw new UnsupportedOperationException("Not yet implemented");
    }

    void ReInit(InputStream stream, String encoding, int i, int i0) throws UnsupportedEncodingException {
        throw new UnsupportedOperationException("Not yet implemented");
    }

    void backup(int i) {
        input.backup(i);
    }

    int getBeginColumn() {
        return 0;
    }

    int getBeginLine() {
        return 0;
    }

    int getEndColumn() {
        return 0;
    }

    int getEndLine() {
        return 0;
    }

    char readChar() throws IOException {
        int result = input.read();
        if (result == LexerInput.EOF) {
            throw new IOException("LexerInput EOF");
        }
        return (char) result;
    }

}</pre></li>
                <li><p> After replacing the generated <tt>JavaCharStream</tt> with the
                        code in the previous step, everything should compile and your module structure
                        should be as follows:</p>
                    <p><img src="../../images/tutorials/javacc/71/lexer-4.png" alt="source."/></p>

                </li>
            </ol>

            <p>You now have an implementation of the NetBeans Lexer API based
                on a JavaCC lexer generated from a JavaCC grammar definition. In
                the next section, you register your NetBeans lexer so that the
                NetBeans Platform infrastructure can find it and load it
                into the application.</p>

        </div>

        <!-- ======================================================================================= -->

        <h2><a name="registering"></a>Registering the NetBeans Lexer</h2>
        <p>You now have a NetBeans lexer. We need to register it so that it can be used.</p>
        <div class="indent">
            <ol>
                <li><p>In <tt>SJTokenId</tt>, define the following method,
                        which returns an instance of <tt>org.netbeans.api.lexer.Language</tt>:</p>
                    <pre class="examplecode">public static Language&lt;SJTokenId&gt; getLanguage() {
    return new SJLanguageHierarchy().language();
}</pre>
                </li>
                <li><p>The instance created statically above needs to be called from somewhere.
			The call
                        is done from the <tt>layer.xml</tt> file, within the
			<tt>CslPlugins</tt>
                        folder, where you register the class below as
			a language instance, via the class annotation <tt>@LanguageRegistration</tt>:</p>
                    <pre class="examplecode">package org.simplejava;

import org.netbeans.api.lexer.Language;
import org.netbeans.modules.csl.spi.DefaultLanguageConfig;
import org.netbeans.modules.csl.spi.LanguageRegistration;
import org.simplejava.lexer.SJTokenId;

@LanguageRegistration(mimeType = "text/x-sj")
public class SJLanguage extends DefaultLanguageConfig {

    @Override
    public Language getLexerLanguage() {
        return SJTokenId.getLanguage();
    }

    @Override
    public String getDisplayName() {
        return "SJ";
    }

}</pre>
		    <p class="tips"> When the module containing the above class is built,
			 the <tt>generated-layer.xml</tt> file in the module's 'build' folder,
			 which is visible in the Files window (Ctrl - 2) contains many new
			 entries, providing many default features for your language.</p>
                </li>
            </ol>
        </div>
        <!-- ======================================================================================= -->

        <h2><a name="defining"></a>Defining Tokens and Assigning Colors</h2>
        <p>Let's now work with the actual tokens that we're going to need. First,
            we'll update the <tt>SJLanguageHierarchy</tt> with our tokens. After that,
            we'll map the tokens to fonts and colors. Finally, we'll register our
            new files in the virtual filesystem of the application we're working on.</p>
        <div class="indent">
            <ol>
                <li><p>Look in the generated <tt>JavaParserConstants</tt> file
                        and notice the tokens that have been generated by JavaCC.</p>

                    <pre class="examplecode">
public interface JavaParserConstants {

  int EOF = 0;
  int WHITESPACE = 1;
  int SINGLE_LINE_COMMENT = 4;
  int FORMAL_COMMENT = 5;
  int MULTI_LINE_COMMENT = 6;
  int ABSTRACT = 8;
  int ASSERT = 9;
  int BOOLEAN = 10;
  int BREAK = 11;
  int BYTE = 12;
  ...
  ...
  ...</pre>
                    <p>Now tweak and then
                        copy the tokens above into your <tt>SJLanguageHierarchy</tt> file:</p>
                    <pre class="examplecode">tokens = Arrays.asList(new SJTokenId[]{
    new SJTokenId("EOF", "whitespace", 0),
    new SJTokenId("WHITESPACE", "whitespace", 1),
    new SJTokenId("SINGLE_LINE_COMMENT", "comment", 4),
    new SJTokenId("FORMAL_COMMENT", "comment", 5),
    new SJTokenId("MULTI_LINE_COMMENT", "comment", 6),
    new SJTokenId("ABSTRACT", "keyword", 8),
    new SJTokenId("ASSERT", "keyword", 9),
    new SJTokenId("BOOLEAN", "keyword", 10),
    new SJTokenId("BREAK", "keyword", 11),
    new SJTokenId("BYTE", "keyword", 12),
    ...
    ...
    ...</pre>

                    <p class="tips">See the <a href="#appendix">Appendix</a> for the complete
                        list of tokens. Copy them from the appendix into your module.</p>
                </li>
                <li><p>Next, we need to map the categories to fonts and colors. This is
                        done declaratively, in an XML file, where we list the categories
                        and then declare the fonts and colors that should be applied.
                        In the main package
                        of the module, that is, <tt>org.simplejava</tt>, create a new XML file
                        named <tt>FontAndColors.xml</tt>, with the following content:</p>
                    <pre class="examplecode">&lt;!DOCTYPE fontscolors PUBLIC
    "-//NetBeans//DTD Editor Fonts and Colors settings 1.1//EN"
    "https://netbeans.org/dtds/EditorFontsColors-1_1.dtd"&gt;
&lt;fontscolors&gt;
    &lt;fontcolor name="character" default="char"/&gt;
    &lt;fontcolor name="errors" default="error"/&gt;
    &lt;fontcolor name="identifier" default="identifier"/&gt;
    &lt;fontcolor name="keyword" default="keyword" foreColor="red"/&gt;
    &lt;fontcolor name="literal" default="keyword" /&gt;
    &lt;fontcolor name="comment" default="comment"/&gt;
    &lt;fontcolor name="number" default="number"/&gt;
    &lt;fontcolor name="operator" default="operator"/&gt;
    &lt;fontcolor name="string" default="string"/&gt;
    &lt;fontcolor name="separator" default="separator"/&gt;
    &lt;fontcolor name="whitespace" default="whitespace"/&gt;
    &lt;fontcolor name="method-declaration" default="method"&gt;
        &lt;font style="bold" /&gt;
    &lt;/fontcolor&gt;
&lt;/fontscolors&gt;</pre>

                    <p>This file defines how to visualize the tokens produced by the lexer.</p>

                    <p>The <tt>fontcolor</tt> tag properties are as follows:</p>

                    <ul>
                        <li>name: Name or primaryCategory of your token (or tokens).</li>
                        <li>default: Name of default coloring. All properties that are not defined
                            explicitly are inherited from this default coloring. Default coloring
                            is customizable in the Options window.</li>
                        <li>foreColor: Foreground color.</li>
                        <li>bgColor: Background color.</li>
                        <li>underline: Underlined color. Token will be underlined if specified.</li>
                        <li>strikeThrough: Strike through color.</li>
                        <li>waveUnderlined: Wave underlined color.</li>
                    </ul>
                    <p>The <tt>fontcolor</tt> tag can contain
                        a nested font tag. The <tt>font</tt> tag has
                        the following properties:</p>
                    <ul>
                        <li>name: Name of font.</li>
                        <li>size: Font size.</li>
                        <li>style: Bold or italic style.</li>
                    </ul>

                </li>
                <li><p>Copy the following code into the template file. Not only will
                        you use the template file, that is, <tt>SJTemplate.sj</tt>, as
                        a template in the New File dialog, but also as example text to
                        be shown in the Options window, where the user will be able to
                        see the effect of their customized fonts and colors.</p>
                    <pre class="examplecode">/**
 * SimpleJavadoc comment for <code>SimpleJavaExample</code> class.
 * @author Simple Joe Smith
 */
public class SimpleJavaExample {

    @Deprecated public String method (int param) {
        return "SimpleString " + '-' + 1.2;
    }// line comment

}</pre>
                </li>

                <li><p>Add the following key/value pairs into the <tt>Bundle.properties</tt>
                        file of the main package, that is, the <tt>Bundle.properties</tt> file
                        found in <tt>org.simplejava</tt>:</p>
                    <pre class="examplecode">text/x-sj=Simple Java
character=Character
errors=Error
identifier=Identifier
keyword=Keyword
literal=Literal
comment=Comment
number=Number
operator=Operator
string=String
separator=Separator
whitespace=Whitespace
method-declaration=Method Declaration</pre>
                    <p>The above values will be shown in the Options window, in the panel
                        where the user will be able to change the predefined fonts
                        and colors per category.</p>
                </li>
                <li><p>Register the <tt>FontAndColors.xml</tt> file, as well as the
                        example file into the <tt>layer.xml</tt> file:</p>
                    <pre class="examplecode">&lt;folder name="Editors"&gt;
    &lt;folder name="text"&gt;
        &lt;folder name="x-sj"&gt;
            <b>&lt;attr name="SystemFileSystem.localizingBundle" stringvalue="org.simplejava.Bundle"/&gt;</b>
            <b>&lt;folder name="FontsColors"&gt;
                &lt;folder name="NetBeans"&gt;
                    &lt;folder name="Defaults"&gt;
                        &lt;file name="FontAndColors.xml" url="FontAndColors.xml"&gt;
                            &lt;attr name="SystemFileSystem.localizingBundle" stringvalue="org.simplejava.Bundle"/&gt;
                        &lt;/file&gt;
                    &lt;/folder&gt;
                &lt;/folder&gt;
            &lt;/folder&gt;</b>
        &lt;/folder&gt;
    &lt;/folder&gt;
&lt;/folder&gt;
<b>&lt;folder name="OptionsDialog"&gt;
    &lt;folder name="PreviewExamples"&gt;
        &lt;folder name="text"&gt;
            &lt;file name="x-sj" url="SJTemplate.sj"/&gt;
        &lt;/folder&gt;
    &lt;/folder&gt;
&lt;/folder&gt;</b></pre>
                </li>
                <li><p>Check that your module now has this content:</p>
                    <p><img src="../../images/tutorials/javacc/71/lexer-5.png" alt="source."/></p>

                </li>
            </ol>
            <p>Run the module, open an SJ file, and you should see the correct coloring,
                as defined in the files above:</p>
            <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/result-1.png" alt="editor."/></p>
            <p>Look in the Options window, under the Tools menu,
                and you should be able to change the fonts and colors for the file type.</p>
            <p><img style="border:1px solid black" src="../../images/tutorials/javacc/71/result-2.png" alt="editor."/></p>

        </div>

        <!-- ======================================================================================= -->

        <p></p>
        <div class="feedback-box"><a href="https://netbeans.org/about/contact_form.html?to=3&amp;subject=Feedback:%20JavaCC%20Lexer%207.1%20Tutorial">Send Us Your Feedback</a></div>

        <!-- ======================================================================================== -->

        <h2><a name="nextsteps"></a>Next Steps</h2>
        <p class="tips">This tutorial is the official version of
            the first part of <a href="http://wiki.netbeans.org/How_to_create_support_for_a_new_language">http://wiki.netbeans.org/How_to_create_support_for_a_new_language</a>,
            which, aside from being a rough draft, is partly
            obsolete and out of date for NetBeans Platform 7.1.</p>
         <p class="tips"> Part 2 of this tutorial, the <a href="nbm-javacc-parser.html">JavaCC Parser Generator Integration Tutorial</a>,
            shows you how to create a parser that parses the same text
            that the lexer in this tutorial lexes.</p>
        <p>For more information about creating and developing NetBeans modules, see the following resources: </p>
        <ul>
            <li><a href="https://platform.netbeans.org/index.html">NetBeans Platform Homepage</a></li>
            <li><a href="https://netbeans.org/download/dev/javadoc/">NetBeans API List (Current Development Version)</a></li>
            <li><a href="https://netbeans.org/kb/trails/platform.html">Other Related Tutorials</a></li>
        </ul>

        <!-- ======================================================================================== -->

        <h2><a name="appendix"></a>Appendix</h2>

        <p>The complete list of tokens, referred to in step 1 of the
            section <a href="#defining">Defining Tokens and Assigning Colors</a> above, is as follows: </p>

        <pre class="examplecode">tokens = Arrays.asList(new SJTokenId[]{
    new SJTokenId("EOF", "whitespace", 0),
    new SJTokenId("WHITESPACE", "whitespace", 1),
    new SJTokenId("SINGLE_LINE_COMMENT", "comment", 4),
    new SJTokenId("FORMAL_COMMENT", "comment", 5),
    new SJTokenId("MULTI_LINE_COMMENT", "comment", 6),
    new SJTokenId("ABSTRACT", "keyword", 8),
    new SJTokenId("ASSERT", "keyword", 9),
    new SJTokenId("BOOLEAN", "keyword", 10),
    new SJTokenId("BREAK", "keyword", 11),
    new SJTokenId("BYTE", "keyword", 12),
    new SJTokenId("CASE", "keyword", 13),
    new SJTokenId("CATCH", "keyword", 14),
    new SJTokenId("CHAR", "keyword", 15),
    new SJTokenId("CLASS", "keyword", 16),
    new SJTokenId("CONST", "keyword", 17),
    new SJTokenId("CONTINUE", "keyword", 18),
    new SJTokenId("_DEFAULT", "keyword", 19),
    new SJTokenId("DO", "keyword", 20),
    new SJTokenId("DOUBLE", "keyword", 21),
    new SJTokenId("ELSE", "keyword", 22),
    new SJTokenId("ENUM", "keyword", 23),
    new SJTokenId("EXTENDS", "keyword", 24),
    new SJTokenId("FALSE", "keyword", 25),
    new SJTokenId("FINAL", "keyword", 26),
    new SJTokenId("FINALLY", "keyword", 27),
    new SJTokenId("FLOAT", "keyword", 28),
    new SJTokenId("FOR", "keyword", 29),
    new SJTokenId("GOTO", "keyword", 30),
    new SJTokenId("IF", "keyword", 31),
    new SJTokenId("IMPLEMENTS", "keyword", 32),
    new SJTokenId("IMPORT", "keyword", 33),
    new SJTokenId("INSTANCEOF", "keyword", 34),
    new SJTokenId("INT", "keyword", 35),
    new SJTokenId("INTERFACE", "keyword", 36),
    new SJTokenId("LONG", "keyword", 37),
    new SJTokenId("NATIVE", "keyword", 38),
    new SJTokenId("NEW", "keyword", 39),
    new SJTokenId("NULL", "keyword", 40),
    new SJTokenId("PACKAGE", "keyword", 41),
    new SJTokenId("PRIVATE", "keyword", 42),
    new SJTokenId("PROTECTED", "keyword", 43),
    new SJTokenId("PUBLIC", "keyword", 44),
    new SJTokenId("RETURN", "keyword", 45),
    new SJTokenId("SHORT", "keyword", 46),
    new SJTokenId("STATIC", "keyword", 47),
    new SJTokenId("STRICTFP", "keyword", 48),
    new SJTokenId("SUPER", "keyword", 49),
    new SJTokenId("SWITCH", "keyword", 50),
    new SJTokenId("SYNCHRONIZED", "keyword", 51),
    new SJTokenId("THIS", "keyword", 52),
    new SJTokenId("THROW", "keyword", 53),
    new SJTokenId("THROWS", "keyword", 54),
    new SJTokenId("TRANSIENT", "keyword", 55),
    new SJTokenId("TRUE", "keyword", 56),
    new SJTokenId("TRY", "keyword", 57),
    new SJTokenId("VOID", "keyword", 58),
    new SJTokenId("VOLATILE", "keyword", 59),
    new SJTokenId("WHILE", "keyword", 60),
    new SJTokenId("INTEGER_LITERAL", "literal", 61),
    new SJTokenId("DECIMAL_LITERAL", "literal", 62),
    new SJTokenId("HEX_LITERAL", "literal", 63),
    new SJTokenId("OCTAL_LITERAL", "literal", 64),
    new SJTokenId("FLOATING_POINT_LITERAL", "literal", 65),
    new SJTokenId("DECIMAL_FLOATING_POINT_LITERAL", "literal", 66),
    new SJTokenId("DECIMAL_EXPONENT", "number", 67),
    new SJTokenId("HEXADECIMAL_FLOATING_POINT_LITERAL", "literal", 68),
    new SJTokenId("HEXADECIMAL_EXPONENT", "number", 69),
    new SJTokenId("CHARACTER_LITERAL", "literal", 70),
    new SJTokenId("STRING_LITERAL", "literal", 71),
    new SJTokenId("IDENTIFIER", "identifier", 72),
    new SJTokenId("LETTER", "literal", 73),
    new SJTokenId("PART_LETTER", "literal", 74),
    new SJTokenId("LPAREN", "operator", 75),
    new SJTokenId("RPAREN", "operator", 76),
    new SJTokenId("LBRACE", "operator", 77),
    new SJTokenId("RBRACE", "operator", 78),
    new SJTokenId("LBRACKET", "operator", 79),
    new SJTokenId("RBRACKET", "operator", 80),
    new SJTokenId("SEMICOLON", "operator", 81),
    new SJTokenId("COMMA", "operator", 82),
    new SJTokenId("DOT", "operator", 83),
    new SJTokenId("AT", "operator", 84),
    new SJTokenId("ASSIGN", "operator", 85),
    new SJTokenId("LT", "operator", 86),
    new SJTokenId("BANG", "operator", 87),
    new SJTokenId("TILDE", "operator", 88),
    new SJTokenId("HOOK", "operator", 89),
    new SJTokenId("COLON", "operator", 90),
    new SJTokenId("EQ", "operator", 91),
    new SJTokenId("LE", "operator", 92),
    new SJTokenId("GE", "operator", 93),
    new SJTokenId("NE", "operator", 94),
    new SJTokenId("SC_OR", "operator", 95),
    new SJTokenId("SC_AND", "operator", 96),
    new SJTokenId("INCR", "operator", 97),
    new SJTokenId("DECR", "operator", 98),
    new SJTokenId("PLUS", "operator", 99),
    new SJTokenId("MINUS", "operator", 100),
    new SJTokenId("STAR", "operator", 101),
    new SJTokenId("SLASH", "operator", 102),
    new SJTokenId("BIT_AND", "operator", 103),
    new SJTokenId("BIT_OR", "operator", 104),
    new SJTokenId("XOR", "operator", 105),
    new SJTokenId("REM", "operator", 106),
    new SJTokenId("LSHIFT", "operator", 107),
    new SJTokenId("PLUSASSIGN", "operator", 108),
    new SJTokenId("MINUSASSIGN", "operator", 109),
    new SJTokenId("STARASSIGN", "operator", 110),
    new SJTokenId("SLASHASSIGN", "operator", 111),
    new SJTokenId("ANDASSIGN", "operator", 112),
    new SJTokenId("ORASSIGN", "operator", 113),
    new SJTokenId("XORASSIGN", "operator", 114),
    new SJTokenId("REMASSIGN", "operator", 115),
    new SJTokenId("LSHIFTASSIGN", "operator", 116),
    new SJTokenId("RSIGNEDSHIFTASSIGN", "operator", 117),
    new SJTokenId("RUNSIGNEDSHIFTASSIGN", "operator", 118),
    new SJTokenId("ELLIPSIS", "operator", 119),
    new SJTokenId("RUNSIGNEDSHIFT", "operator", 120),
    new SJTokenId("RSIGNEDSHIFT", "operator", 121),
    new SJTokenId("GT", "operator", 122)
});</pre>

        <!-- ======================================================================================== -->

        <!--<h2><a name="version"></a>Versioning </h2>

        <table width="76%" >
            <tbody>
                <tr>
                    <td>
                        <b>Version</b>
                    </td>
                    <td>
                        <b>Date</b>
                    </td>
                    <td>
                        <b>Changes</b>
                    </td>
                </tr>
                <tr>
                    <td>
                        1
                    </td>
                    <td>
                        3 January 2012
                    </td>
                    <td>
                        <ul><li>Initial version.</li>
                            <li>To do:
                                <ul>
                                    <li></li>
                                    <li></li>
                                    <li></li>
                                    <li></li>
                                    <li></li>
                                </ul>
                            </li>
                        </ul>
                    </td>
                </tr>
            </tbody>
        </table>-->

    </body>
</html>