ide/csl.api/doc/overview.html - netbeans - Git at Google

 <!--

     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.

 -->
 <html>
     <body>
         <span style="float:right">Tor Norbye &lt;tor.norbye@sun.com&gt;</span>

         <h2><a href="apichanges.html">API changes</a></h2>

         <h1>GSF</h1>
         <p>
             GSF is a language infrastructure for NetBeans.
             The purpose of GSF is to help create first-class editing support
             for a group languages (like Ruby, Groovy, JavaScript, Python, Scala,
             PHP, etc). By "first-class" editing support I mean all the features
             that we currently have for Java in NetBeans. Deep editing support
             including intelligent code completion, quick fixes and hints,
             and so on.
         </p>
         <p>
             The abbreviation
             "GSF" is an historical artifact and a new name should probably be
             assigned.
         </p>
         <h2>Philosophy</h2>
         <p>
             The GSF API is intended for "language plugins". These are modules
             which add support for new languages (such as Ruby, JavaScript,
             HTML, CSS and so on) to NetBeans.

             The <b>key philosophy</b> behind GSF is:
             <blockquote style="background-color: #ffddff; color: black; padding: 20px; border: solid 1px black">
                 The language plugin deals <b>only</b> with language specific details.
             </blockquote>
         </p>
         <p>
             The implication of this is that nearly all UI, and all "infrastructure"
             code like listening on editor changes, scheduling parsing, implementing editor
             actions etc. are handled by the GSF infrastructure. A language plugin should
             just focus on specific features as they pertain to this language.
         </p>
         <p>
             If you are trying to get started writing a new language using GSF,
             see the <a href="getting-started.html">Getting Started</a> document,
             which will try to order the tasks involved and point to the relevant
             sections.
         </p>
         <h3>Example: Instant Rename</h3>
         <p>
             For an example of this, take a look at the following diagram. This shows
             how "Instant Rename" works in GSF. The Instant Rename action itself
             (editor action implementation, keybinding registration, etc.) is provided
             by GSF.   The language plugin implements two services:
             <ul>
                 <li> A parser - which produces a parse tree. This is used for most
                 features in GSF.
                 <li> An "InstantRenamer" - a simple interface where the language plugin
                 is handed one of its own parse trees, and it needs to answer two
                 questions:
                 <ol>
                     <li>Is the given caret offset renameable? (And if not, provide
                         some error message to explain why, or an indication that refactoring
                     needs to be involved.</li>)
                     <li>Produce a set of offset ranges (begin to end) for all the
                         instances of this symbol that should be renamed together.
                     </li>
                 </ol>
             </ul>
             Here's the flow - the left side is what happens on the GSF side, and
             the right side is what happens in the language plugin:
             <br/>
             <img src="instant-rename.png" />
             <br/>
             <br/>
             One important thing to notice here is that on the GSF side, there
             is <b>NO</b> interpretation of the parser result in any way. GSF asks
             you to parse your code, and hands the parser result back to you
             as part of the implementation of many features.
         </p>
         <p>
             As you can see, GSF provides implementations for a lot of the UI
             and interactions that you need for deep editing support for your language.
             You implement your lexer, parser, and feature implementations like
             code completion, go to declaration, quickfixes etc. based on analyzing
             your own parse trees.  You talk through feature specific APIs to GSF,
             and GSF does the rest.
         </p>
         <h2>Architecture</h2>
         <p>
             Here's a rough architecture diagram for GSF:
             <br/>
             <img src="gsf-architecture.png" />
             <br/>
             <br/>
             As you can see, language plugins talk mostly through the GSF APIs
             to implement their features. GSF is in turn implemented in many cases
             on top of existing NetBeans APIs. In some cases, the abstraction on
             top of existing APIs (such as the Hyperlink API) is very thin. In other
             cases, such as for quickfixes, mark occurrences or semantic highlighting,
             GSF provides a lot of UI implementation under the hood on top of the
             existing APIs.  You might wonder why GSF should bother with features
             such as code completion and hyperlinking when the existing APIs are
             pretty complete. There are several reasons for this:
             <ul>
                 <li>Simpler to learn from one place. The goal for GSF is to bring
                     language support of many languages in NetBeans up to the same depth
                     of support as we have for Java.  (See
                     <a href="http://wiki.netbeans.org/EditorFeaturesChecklist">http://wiki.netbeans.org/EditorFeaturesChecklist</a>
                     for a document which tries to list all these features.)
                     If you have to go and hunt for various different NetBeans APIs
                     (navigation, hyperlinking, highlighting, etc.) it's a lot harder
                     to make sure you cover everything, and you in some cases have
                     to learn new concepts.
                 </li>
                 <li>
                     GSF can take advantage of synergy. For example, GSF knows how
                     to (a) parse your code, and (b) index your code, so it can hand
                     you an up to date parse tree when it's asking you to for example
                     compute the declaration location for an editor click. That makes
                     your job easier. As another example, because GSF implements both
                     your Hyperlink provider (Go To Declaration), and your code completion
                     documentation provider, it can register a tooltip which shows the
                     documentation for the symbol under the caret when you are
                     ctrl-hovering over that symbol. And so on.
                 </li>
                 <li>
                     GSF makes unit testing easier. This is another aspect of the synergy
                     point I brought up above. When you for example add a new
                     "semantic highlighting" implementation, writing a unit test for it
                     is as simple as writing a simple function call - pass in a source
                     file, and a caret location. The GSF unit testing infrastructure
                     will do the rest: It will parse your file, pass it to your semantic
                     highlighting feature implementation, produce a nice "golden file"
                     pretty print of your highlighting (see the unit testing section
                     of this document for details), and diff that result with the recorded
                     golden file (or create it if it doesn't exist). Thus, to unit test,
                     just create new testcases in the form of a source file, run the test
                     once to create a golden file, look at the golden file to make sure
                     it looks correct, and from now on you have a regression test which
                     makes sure that your semantic highlighter always produces the same
                     highlights for this source file.
                 </li>
                 <li>
                     GSF supports embedding. Thus, if you for example implement a GSF
                     hints provider for your language, when your language is embedded in
                     some other files, your quickfixes continue to work; they get run,
                     the error offsets from your AST get translated back to embedding
                     appropriate offsets, etc.
                 </li>
             </ul>
             The bigger point though is that for nearly all editing actions, there is
             shared UI that should not be duplicated across modules. Things like
             color and font definitions for syntax highlighting, error messages,
             action names, keybindings, icons etc. should all be defined in one place
             such that periodic maintenance keeps everything in sync, and such that
             for example editor themes work across all intentional and unintentional
             file types.
         </p>
         <p>
             Furthermore, for many actions, there is a LOT of code behind the scenes
             that is not language specific.  Take mark occurrences for example.
             The only language specific thing there is determining, for a given offset,
             what the other corresponding occurrences are in terms of editor offsets.
             Everything else: ensuring that the parse information is up to date,
             listening on caret changes, painting highlights for the occurrences,
             defining and implementing "goto previous occurrence" and "goto next occurrence",
             and so on, should all be implemented in just one shared place.
         </p>
         <p>
             One thing which is not obvious from the diagram above is that clients
             are free to use NetBeans APIs directly. For example, in NetBeans 6.1,
             the HTML module uses GSF for parsing, navigator, embedding etc. - but
             it registers its own CompletionProvider using the NetBeans APIs
             instead of GSF's code completion handler interface. It is possible
             to combine GSF with for example Schliemann or a custom written
             editor kit or custom written data loaders. More details about this
             are found in the <a href="registration.html#UseCustomEditorKit">custom editor kit</a> section.
         </p>

         <h2>Lack of Extensibility</h2>
         <p>
             One of the key principles of GSF is to keep all UI in GSF. In the old days,
             each language support would define all its own logical editor types, color
             definitions, and so on. That made it really tricky to provide editor themes,
             as well as keep things consistent. A literal string may have one color in one
             file type, and another color in another filetype.
         </p>
         <p>
             This has some implications for GSF:

             <ul>
                 <li>
                     In the Java module, one of the key APIs you get is the ability
                     to schedule a user task: run the parse tree on a Java file,
                     and then run a task over the parse tree. (This is
                     <code>JavaSource.runUserActionTask()</code>).
                     This lets modules create arbitrary features for Java - and this
                     is how the various Java modules (navigation, hints, etc)
                     operate.
                     <br/><br/>
                     GSF does not push this approach (it's actually there, through
                     the <code>SourceModel</code> class, but you're discouraged
                     from using it).
                     <br/><br/>
                     This facility is typically used to implement specific features.
                     And the GSF Way is to implement the feature INSIDE GSF, and expose
                     a lean feature API to other language clients. If it's a feature
                     needed for language A, chances are it's needed for language B
                     as well, and putting user <b>feature</b> code in language plugins
                     is the opposite approach of what GSF is trying to do.
                     <br/><br/>
                 </li>
                 <li>
                     GSF tries to make it really easy to describe UI. It provides this
                     through the
                     <a href="org/netbeans/modules/gsf/api/ElementKind.html">ElementKind</a>
                     enumeration class. When creating code completion items, or navigation
                     items, just reference one of the constants there, such as
                     ElementKind.METHOD, to create an item which will look like a method,
                     consistent across languages. This will provide an icon, a color,
                     a code completion sorting priority etc. None of these are language
                     specific, which is why GSF wants to provide the definitions, and
                     language clients just reference them through the enumeration name.
                     <br/><br/>
                     This of course may seem like a straightjacket - the types of code
                     elements that GSF can address are hardcoded!!
                     <br/><br/>
                     Once again, the philosophy here is that GSF is trying to address
                     a wide swath of languages, but perhaps not every single one.
                     Rather than saying that say a Prolog language has a particular
                     code element that isn't provided by GSF today so Prolog should have
                     the option of adding it, I'd like to add new types to the ElementKind
                     enum. They can be added in every release without any backwards
                     compatibility problems. Chances are, if something is available in
                     one language, it's going to be needed by some other language as well.
                     Second, there's a pretty simple workaround - pick something SIMILAR.
                     In Ruby for example, I could have used the FIELD kind if there
                     wasn't an ATTRIBUTE kind.
                     <br/><br/>
                     Finally, there are backdoors. For example, the CompletionProposal
                     interface (implemented by code completion items) actually has
                     a <code>getIcon()</code> method, where you normally return null,
                     but you can override the icon there. This is how I return a
                     Ruby specific icon for Ruby keywords in code completion (and
                     ditto for JavaScript). But overriding UI is the exception, not
                     the norm.
                     <br/><br/>
                 </li>
                 <li>
                     GSF lets client format the data shown in code completion items;
                     there's the left hand side, the right hand side, as well as
                     different colors and attributes for parameter types, return types,
                     overridden methods, deprecated methods, etc.  Rather than
                     letting clients format HTML directly (which would for example
                     cause color definitions to get hardcoded into the clients),
                     GSF provides an HTML formatter to clients, and when they need
                     to produce HTML, they use logical formatter methods to emit
                     HTML. The actual color definitions etc. are therefore provided
                     by GSF and clients are a bit more high level.
                     <br/><br/>
                 </li>
             </ul>
         </p>


         <h2>Service Implementations</h2>
         <p>
             A language plugin basically registers a bunch of callbacks. These are invoked
             at appropriate times by the infrastructure.  There are two types of callbacks:
             <ol>
                 <li>Service Implementations, such as lexing and parsing</li>
                 <li>Feature Implementations, such as code completion and quickfixes</li>
             </ol>
             Service Implementations basically implement basic services that are used
             to drive other features. There service implementations are:
             <ul>
                 <li>Lexing: Tokenize a document into syntactic elements</li>
                 <li>Parsing: Parse a document into semantic structure</li>
                 <li>Indexing: Given a parse result, store some information in a persistent
                 (and quickly searchable) index</li>
             </ul>
             The infrastructure will call your services at the right times. For example,
             the lexer is asked to lex your document when it is opened in the editor.
             It is also asked to update the token hierarchy immediately as the document
             is edited. Similarly, the parser is called in a background thread shortly
             after the document has been edited, or immediately when the result is needed
             right away such as during code completion. And finally, the indexer is called
             when you leave a file (to keep the index up to date), or at startup for
             files that are new or have been updated outside the IDE.
         </p>

         <h3>Lexing</h3>
         <p>
             GSF uses the Lexer API in NetBeans directly. All GSF languages must provide
             a Lexer language. It must also be registered in the Editors mime folder.
             See the <a href="registration.html">Registration</a> section for more details.
         </p>

         <h3>Parsing</h3>
         <p>
             GSF provides a <a href="org/netbeans/modules/gsf/api/Parser.html">parser</a> interface.
             You can register a parser, and GSF will call your parser when needed.
         </p>
         <blockquote style="background-color: #ffdddd; color: black; padding: 20px; border: solid 1px black">
             NOTE: There is a new Parsing API underway. This will be something
             equivalent to the Lexer API, but applied to parsing. The plan is for GSF
             to remove its own Parser registration and interfaces and replace it with
             the standard Parser API. This will allow embedding and coordination
             not just among GSF-based languages, but for all languages that implement
             the parsing API (such as Java, and C/C++, which are not GSF based).
         </blockquote>
         <p>
             The Parser API basically just asks you to parse a given CharSequence,
             and return the result as your own subclass of the
             <a href="org/netbeans/modules/gsf/api/ParserResult.html">ParserResult</a> interface.
             It is in your own parser result that you store your own AST reference.
             You will probably need it to implement most of the features.
             There is a separate document describing the <a href="parsing.html">parsing aspects</a>
             of GSF. (Eventually you'll also want to implement
             <a href="incremental-parsing.html">incremental parsing</a>.)
         </p>
         <h3>Indexing</h3>
         <p>
             Indexing lets you register a service which will extract information from your
             parse results, and store them in an index which can be queried quickly later.
             Your indexer will be called "at the right times" by the infrastructure:
             <ol>
                 <li>At startup for all source files that are new or have changed since the
                     last IDE session
                 </li>
                 <li>After a file has been edited, and closed/left</li>
                 <li>Immediately if a file has been edited and somebody is trying to query the index
             </ol>

             This topic is described in a lot more detail in the separate
             <a href="indexer.html">Indexing and Querying</a> document.
         </p>
         <h2>Feature Implementations</h2>

         The service implementations above aren't really useful in their own right,
         but they allow a number of features to be implemented. When implementing
         additional feature interfaces, you get handed your token hierarchy from
         your lexer, your parse tree and a handle to search the index populated
         by your indexer, which you can use to respond to the feature requests.

         <h3>Semantic Highlighting</h3>
         <p>

         </p>
         <p>
             To implement semantic highlighting, you need to
             <a href="registration.html">register</a> an implementation of the
             <a href="org/netbeans/modules/gsf/api/SemanticAnalyzer.html">SemanticAnalyzer</a> interface.
             You will be given your own <code>ParserResult</code> object, and
             you need to return a <code>Map&lt;OffsetRange, Set&lt;ColoringAttributes&gt;&gt;</code>.
             So, you'll iterate your own AST, look for things to highlight, such as
             method definitions, parameters and unused variables, and for each node,
             look up its source offsets (start, end integer offsets in the document),
             and then place them in the document:
             <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
     // Aha, this is an unused variable reference!
     int nodeStart = node.getStartOffset();
     int nodeEnd = node.getEndOffset();
     result.put(new OffsetRange(nodeStart, nodeEnd), ColoringAttribute.UNUSED_SET);
             </pre>
             Mark Occurrences is very similar to this. You implement the
             <a href="org/netbeans/modules/gsf/api/OccurrencesFinder.html">OccurrencesFinder</a> interface,
             where you return a Set&lt;OffsetRange&gt; for the symbols that should be
             highlighted as other occurrences of the symbol at the given offset.
         </p>

         <h3>Other Features</h3>
         <p>
             TODO: I plan to write some documents explaining how to do keystroke handling,
             formatting, declaration finding, code completion, etc.
         </p>

         <a name="registration"/>
         <h2>Registration</h2>
         <p>
             GSF services, such as the lexer, parser, and feature implementations,
             must all be registered with the GSF infrastructure. For details on
             how to do this, see the <a href="registration.html">Registration document</a>.
         </p>

         <h2>Embedding</h2>
         <p>
             GSF supports language embedding - supporting nested languages, like
             JavaScript and CSS support inside HTML files, Ruby support (and JavaScript
             and CSS support) inside ERb/RHTML files, and so on.
             The mechanism of how this works is described fully in the
             <a href="embedding.html">embedding document</a>.
         </p>

         <h2>GSF Diagnostic Tools</h2>
         <p>
             There is a separate module in the contrib repository, <code>gsf.tools</code>,
             which provides a number of tools in the <b>Tools | GSF Development</b> menu.
             Details are described in the <a href="gsf-tools.html">GSF Development Tools</a> document.
         </p>

         <a name="Classpath"/>
         <h2>Classpath</h2>
         GSF needs to integrated with the project system, in order to index source files at startup,
         in order to support project-wide operations like Open Type, and so on. Details about
         the project and classpath integration are discussed in more detail in the
         <a href="classpath.html">Classpath document</a>.

         <h2>Unit Testing</h2>
         <p>
             GSF makes it really easy to unit test your feature implementations.
             Details on how to do this is described in the <a href="unit-testing.html">unit
             testing document</a>.
         </p>


         <h2>Implementation Issues/Limitations</h2>
         <p>
             I will describe the various GSF limitations here. For now, they are
             listed (in very brief form) in the following Wiki page:<br/>
             <a href="http://wiki.netbeans.org/GsfIssues">http://wiki.netbeans.org/GsfIssues</a>
             <br/>
         </p>

         <h2>TODO</h2>
         <p>
             <ul>
                 <li>
                     Describe the philosophy of the builtin ElementKind UI, icons, fonts etc. and
                     the general philosophy of pushing features INTO gsf rather than supporting
                     generic user task execution.
                 </li>
                 <li>
                     Describe the modules, imported and exported APIs, etc.
                 </li>
                 <li>
                     Document the GSF limitations
                 </li>
                 <li>Write documents for specific features, like code completion and keystroke handling</li>
             </ul>
         </p>

         <br/>
         <span style="color: #cccccc">Tor Norbye &lt;tor@netbeans.org&gt;</span>
     </body>
 </html>
	<!--

	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.

	-->
	<html>
	<body>
	<span style="float:right">Tor Norbye <tor.norbye@sun.com></span>

	<h2><a href="apichanges.html">API changes</a></h2>

	<h1>GSF</h1>
	<p>
	GSF is a language infrastructure for NetBeans.
	The purpose of GSF is to help create first-class editing support
	for a group languages (like Ruby, Groovy, JavaScript, Python, Scala,
	PHP, etc). By "first-class" editing support I mean all the features
	that we currently have for Java in NetBeans. Deep editing support
	including intelligent code completion, quick fixes and hints,
	and so on.
	</p>
	<p>
	The abbreviation
	"GSF" is an historical artifact and a new name should probably be
	assigned.
	</p>
	<h2>Philosophy</h2>
	<p>
	The GSF API is intended for "language plugins". These are modules
	which add support for new languages (such as Ruby, JavaScript,
	HTML, CSS and so on) to NetBeans.

	The <b>key philosophy</b> behind GSF is:
	<blockquote style="background-color: #ffddff; color: black; padding: 20px; border: solid 1px black">
	The language plugin deals <b>only</b> with language specific details.
	</blockquote>
	</p>
	<p>
	The implication of this is that nearly all UI, and all "infrastructure"
	code like listening on editor changes, scheduling parsing, implementing editor
	actions etc. are handled by the GSF infrastructure. A language plugin should
	just focus on specific features as they pertain to this language.
	</p>
	<p>
	If you are trying to get started writing a new language using GSF,
	see the <a href="getting-started.html">Getting Started</a> document,
	which will try to order the tasks involved and point to the relevant
	sections.
	</p>
	<h3>Example: Instant Rename</h3>
	<p>
	For an example of this, take a look at the following diagram. This shows
	how "Instant Rename" works in GSF. The Instant Rename action itself
	(editor action implementation, keybinding registration, etc.) is provided
	by GSF. The language plugin implements two services:
	<ul>
	<li> A parser - which produces a parse tree. This is used for most
	features in GSF.
	<li> An "InstantRenamer" - a simple interface where the language plugin
	is handed one of its own parse trees, and it needs to answer two
	questions:
	<ol>
	<li>Is the given caret offset renameable? (And if not, provide
	some error message to explain why, or an indication that refactoring
	needs to be involved.</li>)
	<li>Produce a set of offset ranges (begin to end) for all the
	instances of this symbol that should be renamed together.
	</li>
	</ol>
	</ul>
	Here's the flow - the left side is what happens on the GSF side, and
	the right side is what happens in the language plugin:
	<br/>
	<img src="instant-rename.png" />
	<br/>
	<br/>
	One important thing to notice here is that on the GSF side, there
	is <b>NO</b> interpretation of the parser result in any way. GSF asks
	you to parse your code, and hands the parser result back to you
	as part of the implementation of many features.
	</p>
	<p>
	As you can see, GSF provides implementations for a lot of the UI
	and interactions that you need for deep editing support for your language.
	You implement your lexer, parser, and feature implementations like
	code completion, go to declaration, quickfixes etc. based on analyzing
	your own parse trees. You talk through feature specific APIs to GSF,
	and GSF does the rest.
	</p>
	<h2>Architecture</h2>
	<p>
	Here's a rough architecture diagram for GSF:
	<br/>
	<img src="gsf-architecture.png" />
	<br/>
	<br/>
	As you can see, language plugins talk mostly through the GSF APIs
	to implement their features. GSF is in turn implemented in many cases
	on top of existing NetBeans APIs. In some cases, the abstraction on
	top of existing APIs (such as the Hyperlink API) is very thin. In other
	cases, such as for quickfixes, mark occurrences or semantic highlighting,
	GSF provides a lot of UI implementation under the hood on top of the
	existing APIs. You might wonder why GSF should bother with features
	such as code completion and hyperlinking when the existing APIs are
	pretty complete. There are several reasons for this:
	<ul>
	<li>Simpler to learn from one place. The goal for GSF is to bring
	language support of many languages in NetBeans up to the same depth
	of support as we have for Java. (See
	<a href="http://wiki.netbeans.org/EditorFeaturesChecklist">http://wiki.netbeans.org/EditorFeaturesChecklist</a>
	for a document which tries to list all these features.)
	If you have to go and hunt for various different NetBeans APIs
	(navigation, hyperlinking, highlighting, etc.) it's a lot harder
	to make sure you cover everything, and you in some cases have
	to learn new concepts.
	</li>
	<li>
	GSF can take advantage of synergy. For example, GSF knows how
	to (a) parse your code, and (b) index your code, so it can hand
	you an up to date parse tree when it's asking you to for example
	compute the declaration location for an editor click. That makes
	your job easier. As another example, because GSF implements both
	your Hyperlink provider (Go To Declaration), and your code completion
	documentation provider, it can register a tooltip which shows the
	documentation for the symbol under the caret when you are
	ctrl-hovering over that symbol. And so on.
	</li>
	<li>
	GSF makes unit testing easier. This is another aspect of the synergy
	point I brought up above. When you for example add a new
	"semantic highlighting" implementation, writing a unit test for it
	is as simple as writing a simple function call - pass in a source
	file, and a caret location. The GSF unit testing infrastructure
	will do the rest: It will parse your file, pass it to your semantic
	highlighting feature implementation, produce a nice "golden file"
	pretty print of your highlighting (see the unit testing section
	of this document for details), and diff that result with the recorded
	golden file (or create it if it doesn't exist). Thus, to unit test,
	just create new testcases in the form of a source file, run the test
	once to create a golden file, look at the golden file to make sure
	it looks correct, and from now on you have a regression test which
	makes sure that your semantic highlighter always produces the same
	highlights for this source file.
	</li>
	<li>
	GSF supports embedding. Thus, if you for example implement a GSF
	hints provider for your language, when your language is embedded in
	some other files, your quickfixes continue to work; they get run,
	the error offsets from your AST get translated back to embedding
	appropriate offsets, etc.
	</li>
	</ul>
	The bigger point though is that for nearly all editing actions, there is
	shared UI that should not be duplicated across modules. Things like
	color and font definitions for syntax highlighting, error messages,
	action names, keybindings, icons etc. should all be defined in one place
	such that periodic maintenance keeps everything in sync, and such that
	for example editor themes work across all intentional and unintentional
	file types.
	</p>
	<p>
	Furthermore, for many actions, there is a LOT of code behind the scenes
	that is not language specific. Take mark occurrences for example.
	The only language specific thing there is determining, for a given offset,
	what the other corresponding occurrences are in terms of editor offsets.
	Everything else: ensuring that the parse information is up to date,
	listening on caret changes, painting highlights for the occurrences,
	defining and implementing "goto previous occurrence" and "goto next occurrence",
	and so on, should all be implemented in just one shared place.
	</p>
	<p>
	One thing which is not obvious from the diagram above is that clients
	are free to use NetBeans APIs directly. For example, in NetBeans 6.1,
	the HTML module uses GSF for parsing, navigator, embedding etc. - but
	it registers its own CompletionProvider using the NetBeans APIs
	instead of GSF's code completion handler interface. It is possible
	to combine GSF with for example Schliemann or a custom written
	editor kit or custom written data loaders. More details about this
	are found in the <a href="registration.html#UseCustomEditorKit">custom editor kit</a> section.
	</p>

	<h2>Lack of Extensibility</h2>
	<p>
	One of the key principles of GSF is to keep all UI in GSF. In the old days,
	each language support would define all its own logical editor types, color
	definitions, and so on. That made it really tricky to provide editor themes,
	as well as keep things consistent. A literal string may have one color in one
	file type, and another color in another filetype.
	</p>
	<p>
	This has some implications for GSF:

	<ul>
	<li>
	In the Java module, one of the key APIs you get is the ability
	to schedule a user task: run the parse tree on a Java file,
	and then run a task over the parse tree. (This is
	<code>JavaSource.runUserActionTask()</code>).
	This lets modules create arbitrary features for Java - and this
	is how the various Java modules (navigation, hints, etc)
	operate.
	<br/><br/>
	GSF does not push this approach (it's actually there, through
	the <code>SourceModel</code> class, but you're discouraged
	from using it).
	<br/><br/>
	This facility is typically used to implement specific features.
	And the GSF Way is to implement the feature INSIDE GSF, and expose
	a lean feature API to other language clients. If it's a feature
	needed for language A, chances are it's needed for language B
	as well, and putting user <b>feature</b> code in language plugins
	is the opposite approach of what GSF is trying to do.
	<br/><br/>
	</li>
	<li>
	GSF tries to make it really easy to describe UI. It provides this
	through the
	<a href="org/netbeans/modules/gsf/api/ElementKind.html">ElementKind</a>
	enumeration class. When creating code completion items, or navigation
	items, just reference one of the constants there, such as
	ElementKind.METHOD, to create an item which will look like a method,
	consistent across languages. This will provide an icon, a color,
	a code completion sorting priority etc. None of these are language
	specific, which is why GSF wants to provide the definitions, and
	language clients just reference them through the enumeration name.
	<br/><br/>
	This of course may seem like a straightjacket - the types of code
	elements that GSF can address are hardcoded!!
	<br/><br/>
	Once again, the philosophy here is that GSF is trying to address
	a wide swath of languages, but perhaps not every single one.
	Rather than saying that say a Prolog language has a particular
	code element that isn't provided by GSF today so Prolog should have
	the option of adding it, I'd like to add new types to the ElementKind
	enum. They can be added in every release without any backwards
	compatibility problems. Chances are, if something is available in
	one language, it's going to be needed by some other language as well.
	Second, there's a pretty simple workaround - pick something SIMILAR.
	In Ruby for example, I could have used the FIELD kind if there
	wasn't an ATTRIBUTE kind.
	<br/><br/>
	Finally, there are backdoors. For example, the CompletionProposal
	interface (implemented by code completion items) actually has
	a <code>getIcon()</code> method, where you normally return null,
	but you can override the icon there. This is how I return a
	Ruby specific icon for Ruby keywords in code completion (and
	ditto for JavaScript). But overriding UI is the exception, not
	the norm.
	<br/><br/>
	</li>
	<li>
	GSF lets client format the data shown in code completion items;
	there's the left hand side, the right hand side, as well as
	different colors and attributes for parameter types, return types,
	overridden methods, deprecated methods, etc. Rather than
	letting clients format HTML directly (which would for example
	cause color definitions to get hardcoded into the clients),
	GSF provides an HTML formatter to clients, and when they need
	to produce HTML, they use logical formatter methods to emit
	HTML. The actual color definitions etc. are therefore provided
	by GSF and clients are a bit more high level.
	<br/><br/>
	</li>
	</ul>
	</p>


	<h2>Service Implementations</h2>
	<p>
	A language plugin basically registers a bunch of callbacks. These are invoked
	at appropriate times by the infrastructure. There are two types of callbacks:
	<ol>
	<li>Service Implementations, such as lexing and parsing</li>
	<li>Feature Implementations, such as code completion and quickfixes</li>
	</ol>
	Service Implementations basically implement basic services that are used
	to drive other features. There service implementations are:
	<ul>
	<li>Lexing: Tokenize a document into syntactic elements</li>
	<li>Parsing: Parse a document into semantic structure</li>
	<li>Indexing: Given a parse result, store some information in a persistent
	(and quickly searchable) index</li>
	</ul>
	The infrastructure will call your services at the right times. For example,
	the lexer is asked to lex your document when it is opened in the editor.
	It is also asked to update the token hierarchy immediately as the document
	is edited. Similarly, the parser is called in a background thread shortly
	after the document has been edited, or immediately when the result is needed
	right away such as during code completion. And finally, the indexer is called
	when you leave a file (to keep the index up to date), or at startup for
	files that are new or have been updated outside the IDE.
	</p>

	<h3>Lexing</h3>
	<p>
	GSF uses the Lexer API in NetBeans directly. All GSF languages must provide
	a Lexer language. It must also be registered in the Editors mime folder.
	See the <a href="registration.html">Registration</a> section for more details.
	</p>

	<h3>Parsing</h3>
	<p>
	GSF provides a <a href="org/netbeans/modules/gsf/api/Parser.html">parser</a> interface.
	You can register a parser, and GSF will call your parser when needed.
	</p>
	<blockquote style="background-color: #ffdddd; color: black; padding: 20px; border: solid 1px black">
	NOTE: There is a new Parsing API underway. This will be something
	equivalent to the Lexer API, but applied to parsing. The plan is for GSF
	to remove its own Parser registration and interfaces and replace it with
	the standard Parser API. This will allow embedding and coordination
	not just among GSF-based languages, but for all languages that implement
	the parsing API (such as Java, and C/C++, which are not GSF based).
	</blockquote>
	<p>
	The Parser API basically just asks you to parse a given CharSequence,
	and return the result as your own subclass of the
	<a href="org/netbeans/modules/gsf/api/ParserResult.html">ParserResult</a> interface.
	It is in your own parser result that you store your own AST reference.
	You will probably need it to implement most of the features.
	There is a separate document describing the <a href="parsing.html">parsing aspects</a>
	of GSF. (Eventually you'll also want to implement
	<a href="incremental-parsing.html">incremental parsing</a>.)
	</p>
	<h3>Indexing</h3>
	<p>
	Indexing lets you register a service which will extract information from your
	parse results, and store them in an index which can be queried quickly later.
	Your indexer will be called "at the right times" by the infrastructure:
	<ol>
	<li>At startup for all source files that are new or have changed since the
	last IDE session
	</li>
	<li>After a file has been edited, and closed/left</li>
	<li>Immediately if a file has been edited and somebody is trying to query the index
	</ol>

	This topic is described in a lot more detail in the separate
	<a href="indexer.html">Indexing and Querying</a> document.
	</p>
	<h2>Feature Implementations</h2>

	The service implementations above aren't really useful in their own right,
	but they allow a number of features to be implemented. When implementing
	additional feature interfaces, you get handed your token hierarchy from
	your lexer, your parse tree and a handle to search the index populated
	by your indexer, which you can use to respond to the feature requests.

	<h3>Semantic Highlighting</h3>
	<p>

	</p>
	<p>
	To implement semantic highlighting, you need to
	<a href="registration.html">register</a> an implementation of the
	<a href="org/netbeans/modules/gsf/api/SemanticAnalyzer.html">SemanticAnalyzer</a> interface.
	You will be given your own <code>ParserResult</code> object, and
	you need to return a <code>Map<OffsetRange, Set<ColoringAttributes>></code>.
	So, you'll iterate your own AST, look for things to highlight, such as
	method definitions, parameters and unused variables, and for each node,
	look up its source offsets (start, end integer offsets in the document),
	and then place them in the document:
	<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
	// Aha, this is an unused variable reference!
	int nodeStart = node.getStartOffset();
	int nodeEnd = node.getEndOffset();
	result.put(new OffsetRange(nodeStart, nodeEnd), ColoringAttribute.UNUSED_SET);
	</pre>
	Mark Occurrences is very similar to this. You implement the
	<a href="org/netbeans/modules/gsf/api/OccurrencesFinder.html">OccurrencesFinder</a> interface,
	where you return a Set<OffsetRange> for the symbols that should be
	highlighted as other occurrences of the symbol at the given offset.
	</p>

	<h3>Other Features</h3>
	<p>
	TODO: I plan to write some documents explaining how to do keystroke handling,
	formatting, declaration finding, code completion, etc.
	</p>

	<a name="registration"/>
	<h2>Registration</h2>
	<p>
	GSF services, such as the lexer, parser, and feature implementations,
	must all be registered with the GSF infrastructure. For details on
	how to do this, see the <a href="registration.html">Registration document</a>.
	</p>

	<h2>Embedding</h2>
	<p>
	GSF supports language embedding - supporting nested languages, like
	JavaScript and CSS support inside HTML files, Ruby support (and JavaScript
	and CSS support) inside ERb/RHTML files, and so on.
	The mechanism of how this works is described fully in the
	<a href="embedding.html">embedding document</a>.
	</p>

	<h2>GSF Diagnostic Tools</h2>
	<p>
	There is a separate module in the contrib repository, <code>gsf.tools</code>,
	which provides a number of tools in the <b>Tools \| GSF Development</b> menu.
	Details are described in the <a href="gsf-tools.html">GSF Development Tools</a> document.
	</p>

	<a name="Classpath"/>
	<h2>Classpath</h2>
	GSF needs to integrated with the project system, in order to index source files at startup,
	in order to support project-wide operations like Open Type, and so on. Details about
	the project and classpath integration are discussed in more detail in the
	<a href="classpath.html">Classpath document</a>.

	<h2>Unit Testing</h2>
	<p>
	GSF makes it really easy to unit test your feature implementations.
	Details on how to do this is described in the <a href="unit-testing.html">unit
	testing document</a>.
	</p>


	<h2>Implementation Issues/Limitations</h2>
	<p>
	I will describe the various GSF limitations here. For now, they are
	listed (in very brief form) in the following Wiki page:<br/>
	<a href="http://wiki.netbeans.org/GsfIssues">http://wiki.netbeans.org/GsfIssues</a>
	<br/>
	</p>

	<h2>TODO</h2>
	<p>
	<ul>
	<li>
	Describe the philosophy of the builtin ElementKind UI, icons, fonts etc. and
	the general philosophy of pushing features INTO gsf rather than supporting
	generic user task execution.
	</li>
	<li>
	Describe the modules, imported and exported APIs, etc.
	</li>
	<li>
	Document the GSF limitations
	</li>
	<li>Write documents for specific features, like code completion and keystroke handling</li>
	</ul>
	</p>

	<br/>
	<span style="color: #cccccc">Tor Norbye <tor@netbeans.org></span>
	</body>
	</html>