<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> | |
<html xmlns="http://www.w3.org/1999/xhtml"> | |
<head> | |
<!-- -*- xhtml -*- --> | |
<title>Lexer Migration Guide</title> | |
<link rel="stylesheet" type="text/css" href="https://netbeans.org/netbeans.css"> | |
<meta name="AUDIENCE" content="NBUSER"> | |
<meta name="TYPE" content="ARTICLE"> | |
<meta name="EXPIRES" content="N"> | |
<meta name="developer" content="gwielenga@netbeans.org"> | |
<meta name="indexed" content="y"> | |
<meta name="description" | |
content="A short guide to using the Lexer API."> | |
<!-- Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. --> | |
<!-- Use is subject to license terms.--> | |
</head> | |
<body> | |
<h1>Lexer Migration Guide</h1> | |
<p> | |
<div style="border: 1px solid rgb(14, 27, 85); padding: 10px 30px; background-color: rgb(231, 231, 231); margin-left: 60px; margin-right: 40px;"> | |
Please note that this tutorial is <b>obsolete</b>. Go here instead:<p> | |
<a href="http://wiki.netbeans.org/How_to_create_support_for_a_new_language">http://wiki.netbeans.org/How_to_create_support_for_a_new_language</a> | |
</div> | |
<p>This document shows you how to migrate from an implementation of | |
<tt>org.netbeans.editor.Syntax</tt> to an implementation of <tt>org.netbeans.api.lexer</tt>. | |
At the end of this document, you will have a NetBeans module that | |
provides syntax highlighting for Manifest files. You will have tokens | |
for the 'name', 'colon', and 'value' | |
in manifest entries and each will have a distinct color, as illustrated below: | |
<p><img src="../../images/tutorials/mf_syntax/result1.png" alt="Highlighted manifest file"> | |
<p>The starting point of this migration guide is the | |
<a href="https://platform.netbeans.org/tutorials/nbm-mfsyntax.html">NetBeans Manifest File Syntax Highlighting Module Tutorial</a>, | |
which implements the <tt><a href="https://netbeans.org/download/dev/javadoc/org-netbeans-modules-editor-lib/org/netbeans/editor/Syntax.html">org.netbeans.editor.Syntax</a></tt> class. To migrate from | |
an implementation of that class to an implementation of <tt>org.netbeans.api.lexer</tt>, | |
you will need to replace the files listed in the left column in the table below | |
with those listed in the right column: | |
<p><table width="76%" border="1"> | |
<tbody><tr> | |
<td> | |
<div align="left"><b>Pre-Lexer</b></div> | |
</td> | |
<td> | |
<div align="left"><b>Lexer</b></div> | |
</td> | |
</tr> | |
<tr> | |
<td align="left" valign="top"><tt>ManifestEditorKit.java</tt> extends | |
<tt><a href="https://netbeans.org/download/dev/javadoc/org-netbeans-modules-editor/org/netbeans/modules/editor/NbEditorKit.html">NbEditorKit</a></tt></a></td> | |
<td><tt>ManifestEditorKit.java</tt> extends <tt>LexerEditorKit</tt></td> | |
</tr> | |
<tr> | |
<td align="left" valign="top"><tt>ManifestSyntax.java</tt> extends | |
<tt>org.netbeans.editor.Syntax</tt></a></td> | |
<td><tt>ManifestLexer.java</tt> extends <tt>org.netbeans.spi.lexer.Lexer</tt></td> | |
</tr> | |
<tr> | |
<td align="left" valign="top"><tt>ManifestTokenContext.java</tt> | |
extends <tt>org.netbeans.editor.TokenContext</tt></a></td> | |
<td><tt>ManifestTokenId.java</tt> extends <tt>org.netbeans.api.lexer.TokenId</tt></td> | |
</tr> | |
<tr> | |
<td align="left" valign="top"><tt>ManifestEditorKit.java</tt></td> | |
<td>Not needed.</td> | |
</tr> | |
<tr> | |
<td align="left" valign="top"><tt>ManifestSettingsInitializer.java</tt></td> | |
<td>Not needed.</td> | |
</tr> | |
<tr> | |
<td align="left" valign="top"><tt>org.netbeans.modules.manifestsupport.options</tt></td> | |
<td>Not needed.</td> | |
</tr> | |
</tbody> | |
</table><p> | |
<p>The following topics are covered in this document:</p> | |
<ul> | |
<li><a href="#setting-up">Setting Up</a></li> | |
<ul> | |
<li><a href="#installing-software">Installing the Software</a></li> | |
<li><a href="#installing-sample">Installing the Sample</a></li> | |
</ul> | |
<li><a href="#migrating">Migrating the Module</a></li> | |
<ul> | |
<li><a href="#tokenizing">Migrating the Tokens</a> | |
<li><a href="#syntaxing">Migrating the Lexical Analyzer</a> | |
<li><a href="#bundle">Migrating the Bundle.properties File</a> | |
<li><a href="#layer">Migrating the XML Layer</a> | |
</ul> | |
<li><a href="#building">Building and Installing the Module</a></li> | |
<li><a href="#explore">Exploring Further</a></li> | |
</ul> | |
<p><a name="top"></a>Once the software is installed, the migration can be complete in 15 minutes. | |
<p>For more information on creating NetBeans plug-in modules, see the <a href="https://platform.netbeans.org/index.html"> | |
NetBeans Development Project home</a> on the NetBeans website. If you have questions, visit the | |
<a href="http://wiki.netbeans.org/wiki/view/NetBeansDeveloperFAQ">NetBeans Developer FAQ</a> or use the feedback link | |
at the top of this page.</p> | |
<!-- ===================================================================================== --> | |
<h2><a name="gettingtoknowthesample"></a>Setting Up</h2> | |
<p>Before you start writing the module, you have to make sure you have all of the necessary software. | |
In addition, you might want to play with the sample before building it yourself.</p> | |
<div class="indent"> | |
<h3 class="tutorial"><a name="installing-software"></a>Installing the Software</h3> | |
<p>Before you begin, you need to install the following software on your | |
computer:</p> | |
<ul> | |
<li>NetBeans IDE 5.5, together with the Lexer modules provided by the Update Center (to come) | |
or Milestone 5 (or later) of NetBeans IDE 6.0 </li> | |
<li>Java Standard Development Kit (JDK™) version | |
1.4.2 (<a href="http://java.sun.com/j2se/1.4.2/download.html">download</a>) | |
or 5.0 (<a href="http://java.sun.com/j2se/1.5.0/download.jsp">download</a>)</li> | |
</ul> | |
<h3 class="tutorial"><a name="installing-sample"></a>Installing the Sample</h3> | |
<p>Take the following steps to install the sample: | |
<ol><li>Download and unzip <a href="https://netbeans.org/files/documents/4/583/ManifestSupport.zip">ManifestSupport.zip</a>. | |
<li>In the IDE, choose File > Open Project and browse to the folder that contains the unzipped file. | |
Open the module project. The files that we will deal with are in the main package, | |
<tt>org.netbeans.modules.manifestsupport</tt>: | |
<p><p><img src="../../images/tutorials/mf_syntax/migrate1.png" alt="Final Projects window"></p></p> | |
<li>Right-click the project node and choose Project Properties. In the Sources panel, | |
change the source level to 1.5. In the Libraries panel, add dependencies on Lexer | |
and Lexer to Editor Bridge and Plain Editor Library.</ol> | |
<!-- ===================================================================================== --> | |
<h2><a name="migrating"></a>Migrating the Module</h2> | |
<p>After taking the preparatory steps above, you need to change four files | |
and delete several others. The files to be changed are listed below, with | |
explanations after the code listing and, in some cases, within the code | |
listings themselves. After that a list of files to be deleted is provided.</p> | |
<div class="indent"> | |
<p><h3 class="tutorial"><a name="tokenizing"></a>Migrating the Tokens</h3> | |
<p>Do the following: | |
<ol> | |
<li><b>Migrate the file.</b> Refactor <tt>ManifestTokenContext</tt> to <tt>ManifestTokenId</tt>. | |
Replace the code with the following: | |
<pre class=examplecode>package org.netbeans.modules.manifestsupport; | |
import java.util.Collection; | |
import java.util.EnumSet; | |
import java.util.Map; | |
import org.netbeans.api.lexer.Language; | |
import org.netbeans.api.lexer.TokenId; | |
import org.netbeans.spi.lexer.LanguageHierarchy; | |
import org.netbeans.spi.lexer.Lexer; | |
import org.netbeans.spi.lexer.LexerRestartInfo; | |
/** | |
* | |
* @author Administrator | |
*/ | |
public enum ManifestTokenId implements TokenId { | |
// The token ids may be assigned to categories and the coloring information | |
// may then be assigned directly to a tokenId.name() e.g. "NAME" | |
// or to a token category e.g. "separator" | |
NAME(null), | |
COLON("separator"), | |
VALUE(null), | |
END_OF_LINE("whitespace"); | |
private String primaryCategory; | |
private ManifestTokenId(String primaryCategory) { | |
this.primaryCategory = primaryCategory; | |
} | |
public String primaryCategory() { | |
return primaryCategory; | |
} | |
private static final Language<ManifestTokenId> language = new LanguageHierarchy<ManifestTokenId>() { | |
protected String mimeType() { | |
return "text/x-java-jar-manifest"; | |
} | |
protected Collection<ManifestTokenId> createTokenIds() { | |
return EnumSet.allOf(ManifestTokenId.class); | |
} | |
protected Map<String,Collection<ManifestTokenId>> createTokenCategories() { | |
return null; | |
} | |
protected Lexer<ManifestTokenId> createLexer(LexerRestartInfo<ManifestTokenId> info) { | |
return new ManifestLexer(info); | |
} | |
}.language(); | |
public static Language<ManifestTokenId> language() { | |
return language; | |
} | |
}</pre></li> | |
<li><b>Understand the file.</b> This Java class specifies a token for each item in the Manifest file with which we want to | |
work. Each distinct item in a Manifest file is a token: | |
'NAME', 'COLON', and 'VALUE'. In addition, there is also a token for the end of the line, because we need to | |
work with the end of the line—the end of the line | |
determines where a value ends and the next name begins. We can also assign tokens to categories | |
and then bind colors to categories. For example, above COLON is assigned to the category | |
'separator', while END_OF_LINE is assigned to the category 'whitespace'. If the example was not | |
so simple, we would probably assign multiple tokens to the same category, which is superfluous | |
here. | |
</ol> | |
<h3 class="tutorial"><a name="migrating"></a>Migrating the Lexical Analyzer</h3> | |
<p>Do the following: | |
<ol> | |
<li><b>Migrate the file.</b> Refactor <tt>ManifestSyntax</tt> | |
to <tt>ManifestLexer</tt>. Replace the default code with the following: | |
<pre class=examplecode>package org.netbeans.modules.manifestsupport; | |
import org.netbeans.api.lexer.Token; | |
import org.netbeans.spi.lexer.Lexer; | |
import org.netbeans.spi.lexer.LexerInput; | |
import org.netbeans.spi.lexer.LexerRestartInfo; | |
import org.netbeans.spi.lexer.TokenFactory; | |
public class ManifestLexer implements Lexer<ManifestTokenId> { | |
private static final int EOF = LexerInput.EOF; | |
// Lexer internal states - preferably small integers for more compact token storage | |
private static final int INIT = 0; | |
private static final int AFTER_COLON = 1; | |
private static final int AFTER_NAME = 2; | |
private LexerInput input; | |
private TokenFactory<ManifestTokenId> tokenFactory; | |
private int state; | |
public Object state() { | |
// autoconversion uses Integer.valueOf() which caches <-127,127> | |
return state; | |
} | |
public ManifestLexer(LexerRestartInfo<ManifestTokenId> info) { | |
this.input = info.input(); | |
this.tokenFactory = info.tokenFactory(); | |
this.state = (info.state() != null) ? (Integer)info.state() : INIT; | |
} | |
public Token<ManifestTokenId> nextToken() { | |
int c = input.read(); | |
switch (state) { | |
case INIT: | |
return nextTokenInit(c); | |
case AFTER_COLON: | |
return nextTokenAfterColon(c); | |
case AFTER_NAME: | |
return nextTokenAfterName(c); | |
default: | |
throw new IllegalStateException(); | |
} | |
} | |
private Token<ManifestTokenId> nextTokenInit(int c) { | |
switch (c) { | |
case ':': // ":" | |
state = AFTER_COLON; | |
return token(ManifestTokenId.COLON); | |
case '\r': | |
input.consumeNewline(); // continue to '\n' handling | |
case '\n': | |
//state = INIT; | |
return token(ManifestTokenId.END_OF_LINE); | |
case EOF: // no chars -> finish lexing by returning null | |
return null; | |
default: // Name follows | |
return finishName(c); | |
} | |
} | |
private Token<ManifestTokenId> nextTokenAfterColon(int c) { | |
switch (c) { | |
case ':': // ":" | |
state = AFTER_COLON; | |
return token(ManifestTokenId.COLON); | |
case '\r': | |
input.consumeNewline(); // continue to '\n' handling | |
case '\n': | |
state = INIT; | |
return token(ManifestTokenId.END_OF_LINE); | |
case EOF: // no chars -> finish lexing by returning null | |
return null; | |
default: | |
return finishValue(c); | |
} | |
} | |
private Token<ManifestTokenId> nextTokenAfterName(int c) { | |
switch (c) { | |
case ':': // ":" | |
state = AFTER_COLON; | |
return token(ManifestTokenId.COLON); | |
case '\r': | |
input.consumeNewline(); // continue to '\n' handling | |
case '\n': | |
state = INIT; | |
return token(ManifestTokenId.END_OF_LINE); | |
case EOF: // no chars -> finish lexing by returning null | |
return null; | |
default: | |
throw new IllegalStateException(); | |
} | |
} | |
private Token<ManifestTokenId> finishName(int c) { | |
while (true) { | |
switch (c) { | |
case ':': | |
case '\r': | |
case '\n': | |
case EOF: | |
input.backup(1); | |
state = AFTER_NAME; | |
return token(ManifestTokenId.NAME); | |
} | |
c = input.read(); | |
} | |
} | |
private Token<ManifestTokenId> finishValue(int c) { | |
while (true) { | |
switch (c) { | |
case '\r': | |
case '\n': | |
case EOF: | |
input.backup(1); | |
state = INIT; | |
return token(ManifestTokenId.VALUE); | |
} | |
c = input.read(); | |
} | |
} | |
private Token<ManifestTokenId> token(ManifestTokenId id) { | |
Token<ManifestTokenId> t = tokenFactory.createToken(id); | |
return t; | |
} | |
public void release() { | |
} | |
}</pre></li> | |
<li><b>Understand the file.</b> This Java class tells the IDE which part of the text is which token. | |
It does this by starting in an initial state and sequentially looking at each character in the text | |
and deciding if it stays in that state, moves to another state, or announces that a token was found. | |
</ol> | |
<h3 class="tutorial"><a name="bundle"></a>Migrating the <tt>Bundle.properties</tt> File</h3> | |
<p>Your <tt>Bundle.properties</tt> file, which is in the same package as the | |
previous two classes, should have this content: | |
<pre class=examplecode>#Layer.xml entries for fonts & colors in Options window: | |
NetBeans=NetBeans | |
NAME=Name | |
VALUE=Value | |
COLON=Colon | |
separator=Separator | |
whitespace=Whitespace</pre> | |
<h3 class="tutorial"><a name="layer"></a>Migrating the XML Layer</h3> | |
<p>This is how your new Lexer implementation should be registered: | |
<pre class=examplecode><folder name="Editors"> | |
<folder name="text"> | |
<folder name="x-java-jar-manifest"> | |
<attr name="SystemFileSystem.localizingBundle" stringvalue="org.myorg.manifestfiletype.Bundle"/> | |
<folder name="NetBeans"> | |
<folder name="Defaults"> | |
<file name="coloring.xml" url="resources/NetBeans-Manifest-fontsColors.xml"> | |
<attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.manifestsupport.Bundle"/> | |
</file> | |
</folder> | |
</folder> | |
<folder name="CityLights"> | |
<folder name="Defaults"> | |
<file name="coloring.xml" url="resources/CityLights-Properties-fontsColors.xml"> | |
<attr name="SystemFileSystem.localizingBundle" stringvalue="org.netbeans.modules.manifestsupport.Bundle"/> | |
</file> | |
</folder> | |
</folder> | |
<file name="language.instance"> | |
<attr name="instanceCreate" methodvalue="org.netbeans.modules.manifestsupport.ManifestTokenId.language"/> | |
<attr name="instanceOf" stringvalue="org.netbeans.api.lexer.Language"/> | |
</file> | |
<file name="EditorKit.instance"> | |
<attr name="instanceCreate" methodvalue="org.netbeans.modules.lexer.editorbridge.LexerEditorKit.create"/> | |
<attr name="instanceClass" stringvalue="org.netbeans.modules.lexer.editorbridge.LexerEditorKit"/> | |
</file> | |
</folder> | |
</folder> | |
</folder> | |
<folder name="OptionsDialog"> | |
<folder name="PreviewExamples"> | |
<folder name="text"> | |
<file name="x-java-jar-manifest" url="resources/ManifestExample"/> | |
</folder> | |
</folder> | |
</folder></pre> | |
<h3 class="tutorial"><a name="syntaxcoloring"></a>Finishing Up</h3> | |
<p>Do the following: | |
<ol> | |
<li><b>Delete <tt>ManifestEditorKit</tt>.</b> You do not need this anymore, | |
because of the LexerEditorKit registered in the XML layer. Right-click | |
the <tt>ManifestSettingsInitializer.java</tt> | |
file and choose Delete. | |
<li><b>Delete <tt>ManifestSettingsInitializer</tt>.</b> You do not need this anymore, | |
because initizialiatio is done through the XML layer. Right-click | |
the <tt>ManifestSettingsInitializer.java</tt> | |
file and choose Delete. | |
<li><b>Change the installer.</b> Previously, you used a module installer to call | |
the <tt>ManifestSettingsInitializer</tt>. You no longer have it, so there | |
is no need to call it. In <tt>RestoreColoring.java</tt>, delete the | |
<tt>addInitializer()</tt> method and remove it from the <tt>restored()</tt> method. | |
<li><b>Delete the Options package.</b> Delete <tt>org.netbeans.modules.manifestsupport.options</tt>. | |
</ol> | |
<h2><a name="building"></a>Building and Installing the Module</h2> | |
<p>The IDE uses an Ant build script to build and install your module. The build script is created for you | |
when you create the module project.</p> | |
<div class="indent"> | |
<h3 class="tutorial">Installing and Testing the NetBeans Module</h3> | |
<p> | |
<ol> | |
<li>In the Projects window, right-click the <tt>ManifestSupport</tt> project and choose Install/Reload | |
in Target Platform. | |
<p>The module is built and installed in the target IDE or Platform. The target IDE or Platform opens so that you | |
can try out your new module. The default target IDE or Platform is the | |
installation used by the current instance of the development IDE. Note that when you run your module, you will be using | |
a temporary test user directory, not the development IDE's user directory. | |
<li>Choose File > New Project (Ctrl-Shift-N). Create a new Java application project | |
or a new Plug-in Module project. Open the project's Manifest file in the Source Editor | |
and notice the syntax highlighting. | |
</ol> <p> | |
<!-- ===================================================================================== --> | |
<br> | |
<div class="feedback-box"><a href="https://netbeans.org/about/contact_form.html?to=3&subject=Feedback:%20Lexer%20Migration%20Guide">Send Us Your Feedback</a></div> | |
<br style="clear:both;" /> | |
<!-- ======================================================================================== --> | |
<h2><a name="nextsteps"></a>Next Steps</h2> | |
<p>For more information about creating and developing NetBeans modules, see the following resources: | |
<ul> | |
<li><a href="https://platform.netbeans.org/index.html">Module Developer's Resources</a></li> | |
<li><a href="https://netbeans.org/download/dev/javadoc/">NetBeans API List (Current Development Version)</a></li> | |
<li><a href="http://apisupport.netbeans.org/new-apisupport.html">New API Support-Proposal</a></li></ul> | |
</p> | |
<hr> | |
<!-- ======================================================================================== --> | |
<h2><a name="version"></a>Versioning </h2> | |
<p> | |
<table width="76%" border="1"> | |
<tbody> | |
<tr> | |
<td> | |
<div align="left"><b>Version</b></div> | |
</td> | |
<td> | |
<div align="left"><b>Date</b></div> | |
</td> | |
<td> | |
<div align="left"><b>Changes</b></div> | |
</td> | |
<tr> | |
<td> | |
1 | |
</td> | |
<td> | |
5 November 2006 | |
</td> | |
<td><ul> | |
<li>Initial version. | |
</td> | |
</tr> | |
<tr> | |
<td> | |
2 | |
</td> | |
<td> | |
12 December 2006 | |
</td> | |
<td><ul> | |
<li>Removed editorkit. Not necessary anymore. LexerEditorkit is declared in layer XML. | |
<li>Added migration of Bundle.properties and XML layer. | |
<li>Changed token ID class and Lexer class, to reflect latest state of the implementation. | |
</td> | |
</tr> | |
<tr> | |
<td> | |
3 | |
</td> | |
<td> | |
13 December 2006 | |
</td> | |
<td><ul> | |
<li>Removed the Options package. (Forgot to do that yesterday.) | |
<li>Small tweaks. | |
</td> | |
</tr> | |
</tbody> | |
</table> | |
</body> | |
</html> |