<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head></head><script charset="utf-8" id="injection_graph_func" src="nbm_interview_toni-1_bestanden/injection_graph_func.js"></script><link href="nbm_interview_toni-1_bestanden/injection_graph.css" type="text/css" rel="stylesheet" charset="utf-8" id="injection_graph_css"><body>
        <h1>Meet a NetBeans Module Writer: Jim Clarke</h1>
          
    <p>30 March 2007
    
               <p><small><a href="mailto:nbdocs_feedback@usersguide.NetBeans.org?subject=Feedback:%20Meet%20A%20NetBeans%20Module%20Writer:%20Jim">Feedback</a></small></p>
        <table class="ltblue" align="right" border="1" cellpadding="5" cellspacing="0" width="190">
            <tbody>
                <tr>
                    <td align="center"><img src="nbm_interviews/jim/jim.png" alt="Jim's picture" align="middle">
                        
                    <br><b>Jim Clarke</b></td>
                </tr>
                  <td>Jim's Hot Links:
		 <ul>
                     <li><a href="http://www28.cplan.com/cc158/sessions_catalog.jsp?ilc=158-1&ilg=english&isort=&isort_type=&is=yes&icriteria1=+&icriteria2=+&icriteria7=+&icriteria9=TS-1205&icriteria8=&icriteria3=">The Sun Java Real-Time System meets Wall Street</a>
                     <li><a href="http://groovy.codehaus.org/">Groovy</a>
                     <li><a href="https://groovychart.dev.java.net/">GroovyChart</a>
		 </ul>
	
                
            </tbody>
        </table>
  
        <p>Jim Clarke has been doing some interesting work, investigating
        ways to automate the generation of NBS files (used by the new <a href="http://languages.netbeans.org/">Schliemann</a>
        project to define tokens and grammar) from ANTLR files. He is interested
        in this in connection with the GroovyChart project, about which he also
        talks in the interview below.
        
<H3>Hi Jim, please tell us a little bit about yourself. Who are you
and what do you do? 
</H3>
<P>I am presently a Principal Engineer in Sun Microsystem's Original Equipment Manufacturer (OEM) Software Sales
group. OEM's goal is to get partners that
create hardware or software products to include Sun's software in their solutions. 
A central example is our team's responsibility for licensing
Java to Cell Phone handset manufacturers. My main duties are to help explain Sun software to our OEM
partners and to explore new technologies that might benefit our OEM
partners. Lately, I have been focused on <a href="http://java.sun.com/javase/technologies/realtime.jsp">Java Realtime</a>,
<a href="http://java.sun.com/products/javatv/">Java TV</a>, and
<a href="http://en.wikipedia.org/wiki/Service_Delivery_Platform">Telco Service Delivery Platforms</a>. 
I have been with Sun for 9 years,
and all of that time working with Java. Prior to Sun, I was doing
independent consulting primarily based on C++ and Corba in Telco. I 
reside in Orlando, Florida., USA.</P>
<H3>What's your connection with NetBeans? How long have you used it
and what do you think of it? 
</H3>
<P>I have been using NetBeans since Sun acquired it in 1999 and it
was called Forte for Java. 
Prior to this time, I was an ardent UNIX/vi type of developer, having
used that since my early days with Unix at AT&amp;T. I have to admit
NetBeans was clumsy at first, but I have seen a marked improvement
    over the years to where <i>I now use it exclusively</i> for development. 
</P>
<P>On a previous assignment, we used NetBeans IDE 3.5 for our RFID
product, with a team of about 20 developers, and found it to be more
than adequate to support that rapid development environment. Part of
this product even included a NetBeans module for partners to
add in their own extensions to the product base.</P>
<P>Since that time, I have used NetBeans 4.0 through 5.5 and I am now
starting to play with the 6.0 development release.</P>

<h3>What specific things do you like about NetBeans IDE? And what could be improved, in your opinion?</h3>

<p>What I like:
<ul><li>Project generation from templates.
    <li>Integration with Ant.
    <li>The Java Editor (and it gets better each release!).
    <li>Hyperlinks, Javadoc, code usage, go to source, etc.
    <li>Code completion.
    <li>Fix imports.
    <li>Java Debugger.
    <li>CVS/Subversion integration.</ul>

<p>Things that I think are needed:

<ul><li>Find/replace across many files. I see 6.0 has a 
    field for this on the Project Find command, but it does not seem to work yet.
    <li>I would like to do a global rename of files. For example, if I originally 
    created files like <tt>GYParser</tt>, <tt>GRFileLoader</tt>, <tt>GRFoo.java</tt>, I would like ability to 
    rename all files, replacing <tt>GR</tt> with <tt>Groovy</tt> to produce <tt>GroovyParser</tt>, and so on.
    <li>Library mappings to JAR file needs to be maintained in the project directory, 
    so that when you JAR/TAR up the source, and someone else downloads it, they are good to go, 
    they would not have to resolve references.
    <li>GUI builder should understand subclasses of Swing object, like "MyPanel extends JPanel". 
    Right now, if I link in MyPanel to the GUI designer it does not understand size, etc, and this
    is hard to visualize.
    <li>Should be an easy way to flag certain non-Java files to be included in <tt>dist/xxx.jar</tt>.
    <li>Would be nice to have a global search capability to find a jar that needs to be added
    to your project, similar to the way module dependencies work.
</ul>

<p>For Groovy, I would like integration with Java Project types, a GUI Editor, debugger support,
and the ability to add Libraries to a Groovy scripting project.

<H3>You have some background and interest in Groovy, can you tell us
a bit about that? 
</H3>
<P>I started looking at Groovy about a year ago in preparation for a
    talk at JavaOne on <a href="http://jddac.dev.java.net/">Sensor Networks</a>. We decided to use Groovy to see
how fast we could develop the demo and to see how well Groovy works.
I have been hooked ever since.</P>
<P>My philosophy on tools and languages is that I want them to be
mainly simple and quick, but at the same time allow me the
flexibility to do complex things for those occasional problems that
always come up in projects. Groovy does this by its seamless
integration with the Java language. I can create Groovy
scripts/classes most of the time, but when I need to, I can just use
a Java class right inside Groovy. That is cool...</P>
<P>More recently, we had to create a Java Realtime Demo that included
displays of performance and JVM metrics. We ended up using an open
source package, <A HREF="http://www.jfree.org/jfreechart/"></A><A HREF="http://www.jfree.org/jfreechart/">JFreeChart</A><A HREF="http://www.jfree.org/jfreechart/"></A>.
At the time, I thought: "Wouldn't it be cool to apply the Groovy
SwingBuilder concept to JFreeChart?" So, that's when I started work on
the java.net project <A HREF="http://groovychart.dev.java.net/">GroovyChart</A>.
So, right now I can use Groovy to create the following kinds of charts:

<ul><li>PieCharts
    <li>BarCharts
    <li>LineCharts
    <li>AreaCharts
    <li>TimeSeries
    <li>FinancialCharts
    <li>XYCharts
    <li>MeterCharts
    <li>MultipleAxisCharts
    <li>CombinedAxisCharts
    <li>GanttCharts
    <li>BoxAndWhiskerCharts
    <li>BubbleCharts
<li>FreeForm charts (in progress)</ul>

<p>So, just about any kind of chart. What I have to finish is some of the low level mapping of Datasets, etc.
In the end, you'll get the same benefits that you get from SwingBuilder, with
a top down view of the component tree, simplified building commands, etc.
The results are illustrated below. First, you see a graph of the 
Java Memory Pools in the JRTS 2.0 VM (Sun Java Real-time System),
taken from the VM JMX MBeans:

<p><img src="nbm_interviews/jim/chart.PNG" border="1" alt="Groovy chart" align="middle">

<p>Next, you see a graph of the execution time for a trade, where the peaks 
indicate a delay caused by either GC or system interupts:

<p><img src="nbm_interviews/jim/executive-groovychart.png" border="1" alt="Executive Groovy chart" align="middle">

<p>And here's a graph of the Garbage Collectors
in the JRTS 2.0 VM, taken from the VM JMX MBeans: 

<p><img src="nbm_interviews/jim/gc-groovychart.png" border="1" alt="GC Groovy chart" align="middle">

<p>Finally, this graph show the money lost due to delays caused by 
GC or System interrupts in executing trades:

<p><img src="nbm_interviews/jim/trade-groovychart.png" border="1" alt="Trade Groovy chart" align="middle">

<p>This last graph shows the opportunity (gain/loss)
for a trade because it executed later than it should have. 
For a limit buy order, the stock price would have risen after 
the instant the stock should have traded, hence the stock price
is higher than the limit buy order requested. For a stop-loss, 
or limit sell order, the stock price would have fallen after 
the instant the stock should have traded, hence the actual trade
price is lower than the sell order wanted. There is a potential 
for a gain if the stock goes the opposite direction, but it is felt
that because stocks trend in a direction and the limit orders are placed
based on the trend, that this usually will be a loss. Also, some brokerage
houses guarantee limit orders, so it is the brokerage house that is taking
the hit in this case...

<p>This year at JavaOne, my colleague Jim Connors and I have a session, "<a href="http://www28.cplan.com/cc158/sessions_catalog.jsp?ilc=158-1&ilg=english&isort=&isort_type=&is=yes&icriteria1=+&icriteria2=+&icriteria7=+&icriteria9=TS-1205&icriteria8=&icriteria3=">The Sun Java Real-Time System meets Wall Street</a>", where we will show the
GroovyChart-enabled GUI. The source for this will be posted on the
GroovyChart java.net site.</P>
<P>Obviously, being a NetBeans fanatic, I found the <A HREF="https://coyote.dev.java.net/">Coyote</A>
NetBeans plugin that supports Groovy. While this did some basic
things, I found it lacking. It needed an indentation engine,
code-completion, support for adding in Java files, the ability to add
libraries just as it is done in a Java project, hyperlinks to JavaDoc
and a GUI editor that supports SwingBuilder. So several months back I
started developing an Indentation Engine. I actually completed one,
which was a nasty experience, when I heard about the upcoming
<a href="http://languages.netbeans.org/">Schliemann Project</a> in NetBeans 6.0.  It was obvious to me that I
could do more with  Schliemann in a shorter time, so I abandoned that
original effort.</P>
<H3>Also, you seem to have quite a bit of experience with compilers
such as ANTLR. True? Tell us about it! 
</H3>
<P>Actually, my experience with lexical analyzers and parsers goes
back to my 'C' and Unix days with Lex and Yacc. We would have to
write conversion routines from legacy system data structures to Unix,
and I figured out that by using Lex/Yacc we could save a lot of hand
coding time.</P>
<P>In my first Java project, I had to convert interface data from IBM
Mainframes into our Java program using Cobol CopyBooks.  I ran across
a Java lexer/parser developed in 1995 at Georgia Tech called CUP
and found that it did a reasonable job.  (CUP was created by Scott Hudson who 
is part of the Graphics Visualization and Usability Center and the College 
of Computing at Georgia Tech. Cup is now maintained at <a href="http://www2.cs.tum.edu/projects/cup/">University of Munich</a>.) 
After that I worked some
with JavaCC, and more recently came across ANTLR in the Coyote and
Groovy projects.</P>
<H3>Now, recently you've begun investigating the sticky problem of
converting ANTLR to NBS (the Schliemann file type). Can you share
some code while you tell us more about that? 
</H3>
<P>My interest in Groovy and Coyote crossed paths with  Schliemann
and I was seeing a lot of traffic on the Groovy mailing list on how
to create an NBS (NetBeans Scripting) grammar file. I knew there was an
ANTLR grammar for Groovy, so it was just obvious that the best thing
to do was look at generating the NBS grammar from the ANTLR grammar. 
The main ANTLR program takes in "antlr" grammar files and
produces Java files  to support the particular grammar provided.  I
then started to investigate how to plugin a different backend
processor to the ANTLR program, that would produce NBS output rather
than Java output.</P>

<p>Before going further, here's a visual summary of the flow from ANTLR format to NBS format:

<p><img src="nbm_interviews/jim/groovy-antlr.png" border="1" alt="Conversion flow" align="middle">


<P>Typically, lexer/parsers have an analyze phase where they scan
over the input and when certain syntax is recognized, an action
occurs. In the main ANTLR program, there are options for plugging in
different backends, or <tt>CodeGenerators</tt>, that do different things when
syntactical elements are recognized. The default is to produce Java
code to support the particular grammar, other options include HTML
and DocBook. HTML produces HTML files that represent the Lexer Tokens
and Grammar from the input grammar. Because this was the most similar
generator to what NBS required, I started off by using this one as
the basis for creating an <tt>NBSCodeGenerator</tt> class.</P>
<P>There were some design issues with ANTLR itself that I had to deal
with. First, all the generators are hard-coded into the main Java
class, <tt>antrl.Tool</tt>. So when the command line option "-html"
is encountered, it sets up a hardcoded class name that points to
<tt>antr.HTMLCodeGenerator.class</tt>.  To add in my <tt>NBSCodeGenerator</tt>, I had
to do the same kind of thing. A better approach would be to make this
more extensible, by allowing a mechanism to define this on the fly.</P>
<P>In <tt>antlr.Tool.java</tt>, I made three changes, as outlined below.</P>
<ol>
    <li>Add in a flag to tell the system to generate NBS output:</P>
    <pre><P STYLE="margin-bottom: 0in">boolean genHTML = false;</P>
<B>boolean genNBS = false;</B></P></pre>
    <P><li>Add in a statement to recognize the new "-nbs" argument:</P>
<pre><B>else if (args[i].equals(&quot;-nbs&quot;)) {</B>
    <B>genNBS = true;</B>

    genDiagnostics = false;
    setArgOK(i);
}</pre>
    <P><li>Return langugage type of "NBS", when <tt>genNBS</tt> is set.
        This allows <tt>antlr.Tool</tt> to create a class name,
    "<tt>antlr.NBSCodeGenerator.class</tt>".</P>
<pre>if (genHTML) {
     return &quot;HTML&quot;;
}
<B>if (genNBS) {
     return &quot;NBS&quot;;
}</B></pre>
</ol>
<P>The other design issue is that the <tt>CodeGeneration</tt> classes have a
    hard dependency on the <tt>antrl.Tool</tt> class. This constricts the
    <tt>CodeGeneration</tt> classes to be within the <tt>antlr</tt> package, because many methods
and members only have package visibility.  A better approach would be
to make this an interface dependency rather than a concrete class
dependency. 
</P>
<P>The first step was to modify the HTML <tt>genHeader</tt> and <tt>genTrail</tt>
methods. In <tt>genHeader</tt>, I took out the normal HTML header and just
replaced it with an 'NBS' style comment  block. In <tt>genTrail</tt>, I just
took out the HMTL close.</P>
<P>ANLTR treats each recognized entity as a <tt>RuleSymbol</tt>, no matter
whether it is a Token or Grammar Rule. However, there are differences
in the way NBS defines tokens and syntax that need to be handled
differently in the <tt>NBSGenerator</tt>. An example  of this is that a token
is represented as follows:</P>
<P><tt>TOKEN:name:</tt></P>
<P>While syntax is represented like this:</P>
<P><tt>name = </tt>
</P>
<P>Below is an example of how this is handled: 
</P>
<pre>if(doingLexRules)
{
    _print(&quot;TOKEN:&quot;);
    _print(s.getId());
    _print(&quot;:( &quot;);
} else {
    _print(s.getId());
    _print(&quot;= &quot;);
}
</pre>
<P>The <tt>CodeGeneration</tt> interface uses a
    call back mechanism, so that when certain types of <tt>RuleSymbols</tt> are
encountered, different generator methods are called back into the
    <tt>CodeGeneration</tt> implementation. Within these methods, I made changes
targeted to the NBS grammar syntax.</P>
<P>For example, in the
    <tt>gen(CharRangeElement)</tt> method, I changed the range character from "<tt>..</tt>"
to "<tt>-</tt>", to match the NBS format:</P>

<pre>     /** Generate code for the given grammar element.
     * @param blk The character-range reference to generate
     */
    public void gen(CharRangeElement r) {
        //print(r.beginText + <B>&quot;..&quot; </B>+ r.endText + &quot; &quot;);
        print(r.beginText + <B>&quot;-&quot;</B> + r.endText + &quot; &quot;);
    }
</pre>
<P><SPAN>The above would produce <tt>'a'-'z'</tt>, or
even <tt>'\u00C0'-'\u00D6'.</tt></SPAN></P>

<P>Some example output from the NBS Generator
using the "<tt>groovy.g</tt>" grammar file from the Groovy project
is as follows:</P>
<pre>############################################################
# ANTLR - generated NBS file from groovy.g
#
# ANTLR Version 2.7.7 (20070322); 1989-2005
############################################################
TOKEN:QUESTION: ( '?'  )
TOKEN:LPAREN: ( '('  )
TOKEN:RPAREN: ( ')'  )
TOKEN:LBRACK: ( '['  )
TOKEN:RBRACK: ( ']'  )
TOKEN:LCURLY: ( '{'  )
...
...
TOKEN:STRING_LITERAL: ( &quot;'''&quot;
        ( STRING_CH
        | ESC
        | '&quot;'
        | '$'
        | STRING_NL
        | '\''  )*
        &quot;'''&quot;
    | '\''
        ( STRING_CH
        | ESC
        | '&quot;'
        | '$'  )*
        '\''
    | &quot;\&quot;\&quot;\&quot;&quot; STRING_CTOR_END
    | '&quot;' STRING_CTOR_END  )
...
...
...
compilationUnit = ( SH_COMMENT  )
            nls
            ( packageDefinition
            | ( statement  )  )
            ( sep ( statement  ) )* EOF

nls = ( NLS  )

annotationsOpt = ( annotation nls )*
...
...
...

</pre>


<h3>Can you share the code of your NBS generator with us?</h3>

<p>Sure, but one thing I need to do is verify that
the NBS syntax being produced is correct. Also, the <tt>HTMLGenerator</tt>
creates two HTML files, one for the Lexer and one for the Parser. My
current <tt>NBSGenerator</tt> does the same thing, however, my intent is to
    generate only <i><b>one</b></i> NBS file. 

<p>Here is the current version of the generator: <tt><a href="nbm_interviews/jim/NBSCodeGenerator.java">NBSCodeGenerator</a></tt>

<h3>Can you foresee a wizard in NetBeans IDE that takes an ANTLR grammar 
    as an input and produces NBS as output? Does this seem feasible to you, 
based on your current investigations?</h3>

<p>Well, it would be nice to have support for automating the inclusion of new scripting
languages in NetBeans IDE. As a part of this, I could see a wizard 
to help in doing this, at least to bootstrap the NBS file. It may be harder to automatically
generate sections like syntax coloring, though.  Also, I would like to support JavaCC...
Perhaps, in addition, I would like to go  back to my roots and support "CUP".

<H3>Anything else you'd like to share? 
</H3>
<P>The key for supporting new languages within NetBeans IDE is to fully
integrate them with the existing features like those that support
Java today. I think that the Schliemann approach is the correct first
step toward this goal.</P>
<P>
            
</p></body></html>