netbeans.apache.org/src/old_content/content/tutorials/68/nbm-mfsyntax_migrate_lexer.adoc

// 
//     Licensed to the Apache Software Foundation (ASF) under one
//     or more contributor license agreements.  See the NOTICE file
//     distributed with this work for additional information
//     regarding copyright ownership.  The ASF licenses this file
//     to you under the Apache License, Version 2.0 (the
//     "License"); you may not use this file except in compliance
//     with the License.  You may obtain a copy of the License at
// 
//       http://www.apache.org/licenses/LICENSE-2.0
// 
//     Unless required by applicable law or agreed to in writing,
//     software distributed under the License is distributed on an
//     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
//     KIND, either express or implied.  See the License for the
//     specific language governing permissions and limitations
//     under the License.
//

= Lexer Migration Guide
:jbake-type: platform_tutorial
:jbake-tags: tutorials 
:jbake-status: published
:syntax: true
:source-highlighter: pygments
:toc: left
:toc-title:
:icons: font
:experimental:
:description: Lexer Migration Guide - Apache NetBeans
:keywords: Apache NetBeans Platform, Platform Tutorials, Lexer Migration Guide

Please note that this tutorial is *obsolete*. Go here instead:

link:http://wiki.netbeans.org/How_to_create_support_for_a_new_language[http://wiki.netbeans.org/How_to_create_support_for_a_new_language]

This document shows you how to migrate from an implementation of  ``org.netbeans.editor.Syntax``  to an implementation of  ``org.netbeans.api.lexer`` . At the end of this document, you will have a NetBeans module that provides syntax highlighting for Manifest files. You will have tokens for the 'name', 'colon', and 'value' in manifest entries and each will have a distinct color, as illustrated below:


image::images/mf_syntax_result1.png[]

The starting point of this migration guide is the  link:https://netbeans.apache.org/tutorials/nbm-mfsyntax.html[NetBeans Manifest File Syntax Highlighting Module Tutorial], which implements the  `` link:https://bits.netbeans.org/dev/javadoc/org-netbeans-modules-editor-lib/org/netbeans/editor/Syntax.html[org.netbeans.editor.Syntax]``  class. To migrate from an implementation of that class to an implementation of  ``org.netbeans.api.lexer`` , you will need to replace the files listed in the left column in the table below with those listed in the right column:

|===
|*Pre-Lexer* |*Lexer* 

| ``ManifestEditorKit.java``  extends  `` link:https://bits.netbeans.org/dev/javadoc/org-netbeans-modules-editor/org/netbeans/modules/editor/NbEditorKit.html[NbEditorKit]``  | ``ManifestEditorKit.java``  extends  ``LexerEditorKit``  

| ``ManifestSyntax.java``  extends  ``org.netbeans.editor.Syntax``  | ``ManifestLexer.java``  extends  ``org.netbeans.spi.lexer.Lexer``  

| ``ManifestTokenContext.java``  extends  ``org.netbeans.editor.TokenContext``  | ``ManifestTokenId.java``  extends  ``org.netbeans.api.lexer.TokenId``  

| ``ManifestEditorKit.java``  |Not needed. 

| ``ManifestSettingsInitializer.java``  |Not needed. 

| ``org.netbeans.modules.manifestsupport.options``  |Not needed. 
|===

The following topics are covered in this document:

* <<setting-up,Setting Up>>

* <<installing-software,Installing the Software>>
* <<installing-sample,Installing the Sample>>
* <<migrating,Migrating the Module>>

* <<tokenizing,Migrating the Tokens>>
* <<syntaxing,Migrating the Lexical Analyzer>>
* <<bundle,Migrating the Bundle.properties File>>
* <<layer,Migrating the XML Layer>>
* <<building,Building and Installing the Module>>
* <<explore,Exploring Further>>

Once the software is installed, the migration can be complete in 15 minutes.

For more information on creating NetBeans plug-in modules, see the  link:https://netbeans.apache.org/platform/index.html[ NetBeans Development Project home] on the NetBeans website. If you have questions, visit the  link:http://wiki.netbeans.org/wiki/view/NetBeansDeveloperFAQ[NetBeans Developer FAQ] or use the feedback link at the top of this page.


== Setting Up

Before you start writing the module, you have to make sure you have all of the necessary software. In addition, you might want to play with the sample before building it yourself.


=== Installing the Software

Before you begin, you need to install the following software on your computer:

* NetBeans IDE 5.5, together with the Lexer modules provided by the Update Center (to come) or Milestone 5 (or later) of NetBeans IDE 6.0
* Java Standard Development Kit (JDK™) version 1.4.2 ( link:https://www.oracle.com/technetwork/java/javase/downloads/index.html[download]) or 5.0 ( link:https://www.oracle.com/technetwork/java/javase/downloads/index.html[download])


=== Installing the Sample

Take the following steps to install the sample:


[start=1]
1. Download and unzip  link:https://netbeans.org/files/documents/4/583/ManifestSupport.zip[ManifestSupport.zip].

[start=2]
1. In the IDE, choose File > Open Project and browse to the folder that contains the unzipped file. Open the module project. The files that we will deal with are in the main package,  ``org.netbeans.modules.manifestsupport`` :


image::images/mf_syntax_migrate1.png[]


[start=3]
1. Right-click the project node and choose Project Properties. In the Sources panel, change the source level to 1.5. In the Libraries panel, add dependencies on Lexer and Lexer to Editor Bridge and Plain Editor Library.


== Migrating the Module

After taking the preparatory steps above, you need to change four files and delete several others. The files to be changed are listed below, with explanations after the code listing and, in some cases, within the code listings themselves. After that a list of files to be deleted is provided.


=== Migrating the Tokens

Do the following:


[start=1]
1. *Migrate the file.* Refactor  ``ManifestTokenContext``  to  ``ManifestTokenId`` . Replace the code with the following:

[source,java]
----

package org.netbeans.modules.manifestsupport;

import java.util.Collection;
import java.util.EnumSet;
import java.util.Map;
import org.netbeans.api.lexer.Language;
import org.netbeans.api.lexer.TokenId;
import org.netbeans.spi.lexer.LanguageHierarchy;
import org.netbeans.spi.lexer.Lexer;
import org.netbeans.spi.lexer.LexerRestartInfo;

/**
 *
 * @author Administrator
 */
public enum ManifestTokenId implements TokenId {

    // The token ids may be assigned to categories and the coloring information
    // may then be assigned directly to a tokenId.name() e.g. "NAME"
    // or to a token category e.g. "separator"
    NAME(null),
    COLON("separator"),
    VALUE(null),
    END_OF_LINE("whitespace");

    private String primaryCategory;
   
    private ManifestTokenId(String primaryCategory) {
        this.primaryCategory = primaryCategory;
    }
    
    public String primaryCategory() {
        return primaryCategory;
    }

    private static final Language<ManifestTokenId> language = new LanguageHierarchy<ManifestTokenId>() {

        protected String mimeType() {
            return "text/x-java-jar-manifest";
        }

        protected Collection<ManifestTokenId> createTokenIds() {
            return EnumSet.allOf(ManifestTokenId.class);
        }
        
        protected Map<String,Collection<ManifestTokenId>> createTokenCategories() {
            return null;
        }

        protected Lexer<ManifestTokenId> createLexer(LexerRestartInfo<ManifestTokenId> info) {
            return new ManifestLexer(info);
        }

    }.language();

    public static Language<ManifestTokenId> language() {
        return language;
    }

}
----


[start=2]
1. *Understand the file.* This Java class specifies a token for each item in the Manifest file with which we want to work. Each distinct item in a Manifest file is a token: 'NAME', 'COLON', and 'VALUE'. In addition, there is also a token for the end of the line, because we need to work with the end of the line—the end of the line determines where a value ends and the next name begins. We can also assign tokens to categories and then bind colors to categories. For example, above COLON is assigned to the category 'separator', while END_OF_LINE is assigned to the category 'whitespace'. If the example was not so simple, we would probably assign multiple tokens to the same category, which is superfluous here.


=== Migrating the Lexical Analyzer

Do the following:


[start=1]
1. *Migrate the file.* Refactor  ``ManifestSyntax``  to  ``ManifestLexer`` . Replace the default code with the following:

[source,java]
----

package org.netbeans.modules.manifestsupport;

import org.netbeans.api.lexer.Token;
import org.netbeans.spi.lexer.Lexer;
import org.netbeans.spi.lexer.LexerInput;
import org.netbeans.spi.lexer.LexerRestartInfo;
import org.netbeans.spi.lexer.TokenFactory;

public class ManifestLexer implements Lexer<ManifestTokenId> {
    
    private static final int EOF = LexerInput.EOF;

    // Lexer internal states - preferably small integers for more compact token storage
    private static final int INIT = 0;
    private static final int AFTER_COLON = 1;
    private static final int AFTER_NAME = 2;

    private LexerInput input;
    
    private TokenFactory<ManifestTokenId> tokenFactory;
    
    private int state;
    
    public Object state() {
		// autoconversion uses Integer.valueOf() which caches <-127,127>
        return state;
    }

    public ManifestLexer(LexerRestartInfo<ManifestTokenId> info) {
        this.input = info.input();
        this.tokenFactory = info.tokenFactory();
		this.state = (info.state() != null) ? (Integer)info.state() : INIT;
    }
    
    public Token<ManifestTokenId> nextToken() {
        int c = input.read();
        switch (state) {
            case INIT:
                return nextTokenInit(c);
            case AFTER_COLON:
                return nextTokenAfterColon(c);
            case AFTER_NAME:
                return nextTokenAfterName(c);
            default:
                throw new IllegalStateException();
        }
    }
        
    private Token<ManifestTokenId> nextTokenInit(int c) {
        switch (c) {
            case ':': // ":"
                state = AFTER_COLON;
                return token(ManifestTokenId.COLON);
            case '\r':
                input.consumeNewline(); // continue to '\n' handling
            case '\n':
                //state = INIT;
                return token(ManifestTokenId.END_OF_LINE);
            case EOF: // no chars -> finish lexing by returning null
                return null;
            default: // Name follows
                return finishName(c);
        }
    }

    private Token<ManifestTokenId> nextTokenAfterColon(int c) {
        switch (c) {
            case ':': // ":"
                state = AFTER_COLON;
                return token(ManifestTokenId.COLON);
            case '\r':
                input.consumeNewline(); // continue to '\n' handling
            case '\n':
                state = INIT;
                return token(ManifestTokenId.END_OF_LINE);
            case EOF: // no chars -> finish lexing by returning null
                return null;
            default:
                return finishValue(c);
        }
    }

    private Token<ManifestTokenId> nextTokenAfterName(int c) {
        switch (c) {
            case ':': // ":"
                state = AFTER_COLON;
                return token(ManifestTokenId.COLON);
            case '\r':
                input.consumeNewline(); // continue to '\n' handling
            case '\n':
                state = INIT;
                return token(ManifestTokenId.END_OF_LINE);
            case EOF: // no chars -> finish lexing by returning null
                return null;
            default:
                throw new IllegalStateException();
        }
    }

    private Token<ManifestTokenId> finishName(int c) {
        while (true) {
            switch (c) {
                case ':':
                case '\r':
                case '\n':
                case EOF:
                    input.backup(1);
                    state = AFTER_NAME;
                    return token(ManifestTokenId.NAME);
            }
            c = input.read();
        }
    }

    private Token<ManifestTokenId> finishValue(int c) {
        while (true) {
            switch (c) {
                case '\r':
                case '\n':
                case EOF:
                    input.backup(1);
                    state = INIT;
                    return token(ManifestTokenId.VALUE);
            }
            c = input.read();
        }
    }

    private Token<ManifestTokenId> token(ManifestTokenId id) {
        Token<ManifestTokenId> t = tokenFactory.createToken(id);
        return t;
    }
    
    public void release() {
    }

}
----


[start=2]
1. *Understand the file.* This Java class tells the IDE which part of the text is which token. It does this by starting in an initial state and sequentially looking at each character in the text and deciding if it stays in that state, moves to another state, or announces that a token was found.


=== Migrating the  ``Bundle.properties``  File

Your  ``Bundle.properties``  file, which is in the same package as the previous two classes, should have this content:


[source,java]
----

#Layer.xml entries for fonts &amp; colors in Options window:
NetBeans=NetBeans
NAME=Name
VALUE=Value
COLON=Colon
separator=Separator
whitespace=Whitespace
----


=== Migrating the XML Layer

This is how your new Lexer implementation should be registered:


[source,xml]
----

<folder name="Editors">
    <folder name="text">
        <folder name="x-java-jar-manifest">
            <attr name="SystemFileSystem.localizingBundle" stringvalue="org.myorg.manifestfiletype.Bundle"/>
            <folder name="NetBeans">
                <folder name="Defaults">
                    <file name="coloring.xml" url="resources/NetBeans-Manifest-fontsColors.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.manifestsupport.Bundle"/>
                    </file>
                </folder>
            </folder> 
            <folder name="CityLights">
                <folder name="Defaults">
                    <file name="coloring.xml" url="resources/CityLights-Properties-fontsColors.xml">
                        <attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.manifestsupport.Bundle"/>
                    </file>
                </folder>
            </folder>
            <file name="language.instance">
                <attr name="instanceCreate" methodvalue="org.netbeans.modules.manifestsupport.ManifestTokenId.language"/>
                <attr name="instanceOf" stringvalue="org.netbeans.api.lexer.Language"/>
            </file>
            <file name="EditorKit.instance">
                <attr name="instanceCreate" methodvalue="org.netbeans.modules.lexer.editorbridge.LexerEditorKit.create"/>
                <attr name="instanceClass" stringvalue="org.netbeans.modules.lexer.editorbridge.LexerEditorKit"/>
            </file>
        </folder>
    </folder>
</folder>

<folder name="OptionsDialog">
    <folder name="PreviewExamples">
        <folder name="text">
            <file name="x-java-jar-manifest" url="resources/ManifestExample"/>
        </folder>
    </folder>
</folder>
----


=== Finishing Up

Do the following:


[start=1]
1. *Delete  ``ManifestEditorKit`` .* You do not need this anymore, because of the LexerEditorKit registered in the XML layer. Right-click the  ``ManifestSettingsInitializer.java``  file and choose Delete.

[start=2]
1. *Delete  ``ManifestSettingsInitializer`` .* You do not need this anymore, because initizialiatio is done through the XML layer. Right-click the  ``ManifestSettingsInitializer.java``  file and choose Delete.

[start=3]
1. *Change the installer.* Previously, you used a module installer to call the  ``ManifestSettingsInitializer`` . You no longer have it, so there is no need to call it. In  ``RestoreColoring.java`` , delete the  ``addInitializer()``  method and remove it from the  ``restored()``  method.

[start=4]
1. *Delete the Options package.* Delete  ``org.netbeans.modules.manifestsupport.options`` .


== Building and Installing the Module

The IDE uses an Ant build script to build and install your module. The build script is created for you when you create the module project.


=== Installing and Testing the NetBeans Module


[start=1]
1. In the Projects window, right-click the  ``ManifestSupport``  project and choose Install/Reload in Target Platform.

The module is built and installed in the target IDE or Platform. The target IDE or Platform opens so that you can try out your new module. The default target IDE or Platform is the installation used by the current instance of the development IDE. Note that when you run your module, you will be using a temporary test user directory, not the development IDE's user directory.


[start=2]
1. Choose File > New Project (Ctrl-Shift-N). Create a new Java application project or a new Plug-in Module project. Open the project's Manifest file in the Source Editor and notice the syntax highlighting.


link:http://netbeans.apache.org/community/mailing-lists.html[Send Us Your Feedback]


== Next Steps

For more information about creating and developing NetBeans modules, see the following resources:

*  link:https://netbeans.apache.org/platform/index.html[Module Developer's Resources]
*  link:https://bits.netbeans.org/dev/javadoc/[NetBeans API List (Current Development Version)]
*  link:http://apisupport.netbeans.org/new-apisupport.html[New API Support-Proposal]


== Versioning

|===
|*Version* |*Date* |*Changes* 

|1 |5 November 2006 |

* Initial version.
 

|2 |12 December 2006 |

* Removed editorkit. Not necessary anymore. LexerEditorkit is declared in layer XML.
* Added migration of Bundle.properties and XML layer.
* Changed token ID class and Lexer class, to reflect latest state of the implementation.
 

|3 |13 December 2006 |

* Removed the Options package. (Forgot to do that yesterday.)
* Small tweaks.
 
|===