You somehow found this GitHub project and wonder if it solves a problem you might have?!
The goal of freemarker-cli is to automate repeated transformation tasks
In December 2015 I needed a little bit of test data management for a customer project - to make a long story short (after writing a few more Groovy scripts) it boiled down to transforming one or more JSON files to something human readable.
What are the options?
So I went with ‘Apache Groovy’, ‘JsonPath’ and ‘Apache Velocity’
Using Velocity actually created some minor issues so I migrated to Apache FreeMarker during Christmas 2016
While I love Apache Velocity (Apache Turbine anyone?) I decided to give FreeMarker a chance and migrated my velocity-cli to FreeMarker.
Some years later the not-so-small-any-longer-and-not-having-tests Groovy script was still growing so I decided
System Properties, Enviroment VariablesDownload the latest release from GitHub, e.g. freemarker-cli-2.0.0-BETA-5-app.tar.gz and unpack it into of a directory of your choice, e.g /Application/Java/freemarker-cli
It is recommended
bin/freemarker-cli or bin/freemarker-cli.bat to your executable path~/.freemarker-cli directory to store your custom FTL templatesYou can test the installation by executing
> ./bin/freemarker-cli -t templates/info.ftl FreeMarker CLI Information --------------------------------------------------------------------------- FreeMarker version : 2.3.29 Template name : templates/info.ftl Language : en Locale : en_US Timestamp : Feb 22, 2020 4:42:01 PM Output encoding : UTF-8 Output format : plainText FreeMarker CLI Template Directories --------------------------------------------------------------------------- [1] /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler FreeMarker CLI Tools --------------------------------------------------------------------------- - CSVTool : Process CSV files using Apache Commons CSV (see https://commons.apache.org/proper/commons-csv/) - ExcelTool : Process Excels files (XLS, XLSX) using Apache POI (see https://poi.apache.org) - ExecTool : Execute command line tools using Apache Commons Exec (see https://commons.apache.org/proper/commons-exec/) - FreeMarkerTool : Expose useful Apache FreeMarker classes - GrokTool : Process text files using Grok expressions (see https://github.com/thekrakken/java-grok) - JsonPathTool : Process JSON files using Java JSON Path (see https://github.com/json-path/JsonPath) - JsoupTool : Process HTML files using Jsoup (see https://jsoup.org) - PropertiesTool : Process JDK properties files - SystemTool : Expose System-related utility methods - UUIDTool : Create UUIDs - XmlTool : Process XML files using Apache FreeMarker (see https://freemarker.apache.org/docs/xgui.html) - YamlTool : Process YAML files using SnakeYAML(see https://bitbucket.org/asomov/snakeyaml/wiki/Home) FreeMarker CLI Documents --------------------------------------------------------------------------- User Supplied Parameters --------------------------------------------------------------------------- User Supplied System Properties --------------------------------------------------------------------------- SystemTool --------------------------------------------------------------------------- Command line : -t, templates/info.ftl Host Name : W0GL5179.local Java Home : /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home User Name : sgoeschl Timestamp : 1,582,386,121,793 Writer : org.apache.freemarker.generator.base.util.NonClosableWriterWrapper
There a many examples (see below) available you can execute - run ./run-samples.sh and have a look at the generated output
./run-samples.sh templates/info.ftl templates/demo.ftl templates/csv/html/transform.ftl templates/csv/md/transform.ftl templates/csv/shell/curl.ftl templates/csv/md/filter.ftl templates/csv/fo/transform.ftl fop -fo target/out/locker-test-users.fo target/out/locker-test-users.pdf templates/csv/fo/transactions.ftl fop -fo target/out/transactions.fo target/out/transactions-fo.pdf templates/csv/html/transform.ftl wkhtmltopdf -O landscape target/out/transactions.html target/out/transactions-html.pdf templates/accesslog/combined-access.ftl templates/excel/html/transform.ftl templates/excel/md/transform.ftl templates/excel/csv/transform.ftl templates/excel/csv/custom.ftl templates/html/csv/dependencies.ftl templates/json/csv/swagger-endpoints.ftl templates/json/md/github-users.ftl templates/properties/csv/locker-test-users.ftl templates/yaml/txt/transform.ftl templates/xml/txt/recipients.ftl Created the following sample files in ./target/out total 1344 -rw-r--r-- 1 sgoeschl staff 646 Feb 22 16:43 combined-access.log.txt -rw-r--r-- 1 sgoeschl staff 22548 Feb 22 16:43 contract.html -rw-r--r-- 1 sgoeschl staff 7933 Feb 22 16:43 contract.md -rw-r--r-- 1 sgoeschl staff 784 Feb 22 16:43 curl.sh -rw-r--r-- 1 sgoeschl staff 232 Feb 22 16:43 customer.txt -rw-r--r-- 1 sgoeschl staff 15084 Feb 22 16:43 demo.txt -rw-r--r-- 1 sgoeschl staff 1310 Feb 22 16:43 dependencies.csv -rw-r--r-- 1 sgoeschl staff 2029 Feb 22 16:43 github-users-curl.md -rw-r--r-- 1 sgoeschl staff 2668 Feb 22 16:43 info.txt -rw-r--r-- 1 sgoeschl staff 66 Feb 22 16:43 interactive-html.txt -rw-r--r-- 1 sgoeschl staff 16 Feb 22 16:43 interactive-json.txt -rw-r--r-- 1 sgoeschl staff 10 Feb 22 16:43 interactive-xml.txt -rw-r--r-- 1 sgoeschl staff 285 Feb 22 16:43 locker-test-users.csv -rw-r--r-- 1 sgoeschl staff 6341 Feb 22 16:43 locker-test-users.fo -rw-r--r-- 1 sgoeschl staff 5526 Feb 22 16:43 locker-test-users.pdf -rw-r--r-- 1 sgoeschl staff 921 Feb 22 16:43 recipients.txt -rw-r--r-- 1 sgoeschl staff 910 Feb 22 16:43 sales-records.md -rw-r--r-- 1 sgoeschl staff 379 Feb 22 16:43 swagger-spec.csv -rw-r--r-- 1 sgoeschl staff 156 Feb 22 16:43 test-multiple-sheets.xlsx.csv -rw-r--r-- 1 sgoeschl staff 1917 Feb 22 16:43 test-multiple-sheets.xlsx.html -rw-r--r-- 1 sgoeschl staff 389 Feb 22 16:43 test-multiple-sheets.xlsx.md -rw-r--r-- 1 sgoeschl staff 150 Feb 22 16:43 test-transform-xls.csv -rw-r--r-- 1 sgoeschl staff 1556 Feb 22 16:43 test.xls.html -rw-r--r-- 1 sgoeschl staff 1558 Feb 22 16:43 test.xslx.html -rw-r--r-- 1 sgoeschl staff 25756 Feb 22 16:43 transactions-fo.pdf -rw-r--r-- 1 sgoeschl staff 66016 Feb 22 16:43 transactions-html.pdf -rw-r--r-- 1 sgoeschl staff 330128 Feb 22 16:43 transactions.fo -rw-r--r-- 1 sgoeschl staff 51008 Feb 22 16:43 transactions.html
Please note that generated PDF files are very likely not found since they require wkhtmltopdf and Apache FOP installation.
> ./bin/freemarker-cli -h Usage: freemarker-cli (-t=<template> | -i=<interactiveTemplate>) [-EhV] [--stdin] [-b=<baseDir>] [--config=<configFile>] [-e=<inputEncoding>] [--exclude=<exclude>] [--include=<include>] [-l=<locale>] [-o=<outputFile>] [--output-encoding=<outputEncoding>] [--times=<times>] [-D=<String=String>]... [-P=<String=String>]... [<sources>...] Apache FreeMarker CLI [<sources>...] List of input files and/or input directories -b, --basedir=<baseDir> Optional template base directory --config=<configFile> FreeMarker CLI configuration file -D, --system-property=<String=String> Set system property -e, --input-encoding=<inputEncoding> Encoding of input documents -E, --expose-env Expose environment variables and user-supplied properties globally --exclude=<exclude> File pattern for document input directory -h, --help Show this help message and exit. -i, --interactive=<interactiveTemplate> Interactive FreeMarker template --include=<include> File pattern for document input directory -l, --locale=<locale> Locale being used for the output, e.g. 'en_US' -o, --output=<outputFile> Output file --output-encoding=<outputEncoding> Encoding of output, e.g. UTF-8 -P, --param=<String=String> Set parameter --stdin Read input document from stdin -t, --template=<template> FreeMarker template to render --times=<times> Re-run X times for profiling -V, --version Print version information and exit.
The examples were tested with JDK 1.8 on Mac OS X
> java -version java version "1.8.0_192" Java(TM) SE Runtime Environment (build 1.8.0_192-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
It is assumed that you run the examples from the freemarker-cli installation directory.
A simple example with real JSON data to be transformed into Markdown
You can either use the existing JSON sample
./bin/freemarker-cli -t templates/json/md/github-users.ftl site/sample/json/github-users.json
or pipe a cURL response
curl -s https://api.github.com/users | ./bin/freemarker-cli -t templates/json/md/github-users.ftl --stdin
<#ftl output_format="plainText" > <#assign json = JsonPathTool.parse(Documents.get(0))> <#assign users = json.read("$[*]")> <#---------------------------------------------------------------------------> # GitHub Users Report generated at ${.now?iso_utc} <#compress> <#list users as user> <#assign userAvatarUrl = user.avatar_url> <#assign userHomeUrl = user.html_url> # ${user.login} | User | Homepage | |:--------------------------------------------------------|:----------------------------------------------| | <img src="${user.avatar_url}" width="48" height="48"/> | [${userHomeUrl}](${userHomeUrl}) | </#list> </#compress>
creates the following output
Sometimes you have a CSV file which needs to be translated in Markdown or HTML - there are on-line solutions available such as CSV To Markdown Table Generator but having a local solution gives you more flexibility.
> ./bin/freemarker-cli -t templates/csv/md/transform.ftl site/sample/csv/contract.csv > ./bin/freemarker-cli -t templates/csv/html/transform.ftl site/sample/csv/contract.csv
The FreeMarker template is shown below
<#ftl output_format="plainText">
<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader()>
<#assign csvParser = CSVTool.parse(Documents.get(0), cvsFormat)>
<#assign csvHeaders = csvParser.getHeaderMap()?keys>
<#assign csvRecords = csvParser.records>
<#--------------------------------------------------------------------------->
<#compress>
<@writeHeaders headers=csvHeaders/>
<@writeColums columns=csvRecords/>
</#compress>
<#--------------------------------------------------------------------------->
<#macro writeHeaders headers>
| ${csvHeaders?join(" | ", "")} |
<#list csvHeaders as csvHeader>| --------</#list>|
</#macro>
<#--------------------------------------------------------------------------->
<#macro writeColums columns>
<#list columns as column>
| ${column.iterator()?join(" | ", "")} |
</#list>
</#macro>
The resulting file actually looks pleasant when compared to raw CSV
Of course you can also transform a XML document
> ./bin/freemarker-cli -t templates/xml/txt/recipients.ftl site/sample/xml/recipients.xml
using the following template
<#ftl output_format="plainText" >
<#assign xml = XmlTool.parse(Documents.get(0))>
<#list xml.recipients.person as recipient>
To: ${recipient.name}
${recipient.address}
Dear ${recipient.name},
Thank you for your interest in our products. We will be sending you a catalog shortly.
To take advantage of our free gift offer, please fill in the survey attached to this
letter and return it to the address on the reverse. Only one participant is allowed for
each household.
Sincere salutations,
D. H.
---------------------------------------------------------------------------------------
</#list>
which generates the following output
To: John Smith 3033 Long Drive, Houston, TX Dear John Smith, Thank you for your interest in our products. We will be sending you a catalog shortly. To take advantage of our free gift offer, please fill in the survey attached to this letter and return it to the address on the reverse. Only one participant is allowed for each household. Sincere salutations, D. H.
One day I was asked a to prepare a CSV files containing REST endpoints described by Swagger - technically this is a JSON to CSV transformation. Of course I could create that CSV manually but writing a FTL template doing that was simply more fun and saves time in the future.
<#ftl output_format="plainText" strip_text="true">
<#assign json = JsonPathTool.parse(Documents.get(0))>
<#assign basePath = json.read("$.basePath")>
<#assign paths = json.read("$.paths")>
<#compress>
ENDPOINT;METHOD;CONSUMES;PRODUCES;SUMMARY;DESCRIPTION
<#list paths as endpoint,metadata>
<#assign relative_url = basePath + endpoint>
<#assign methods = metadata?keys>
<#list methods as method>
<#assign summary = sanitize(paths[endpoint][method]["summary"]!"")>
<#assign description = sanitize(paths[endpoint][method]["description"]!"")>
<#assign consumes = join(paths[endpoint][method]["consumes"]![])>
<#assign produces = join(paths[endpoint][method]["produces"]![])>
${relative_url};${method?upper_case};${consumes};${produces};${summary};${description}
</#list>
</#list>
</#compress>
${'\n'}
<#function sanitize str>
<#return (((str?replace(";", ","))?replace("(\\n)+", "",'r')))?truncate(250)>
</#function>
<#function join list>
<#if list?has_content>
<#return list?join(", ")>
<#else>
<#return "">
</#if>
</#function>
Invoking the FTL template
./bin/freemarker-cli -t templates/json/csv/swagger-endpoints.ftl site/sample/json/swagger-spec.json
gives you
ENDPOINT;METHOD;CONSUMES;PRODUCES;SUMMARY;DESCRIPTION /api/pets;GET;;;;Returns all pets from the system that the user has access to /api/pets;POST;;;;Creates a new pet in the store. Duplicates are allowed /api/pets/{id};GET;;;;Returns a user based on a single ID, if the user does not have access to the pet /api/pets/{id};DELETE;;;;Deletes a single pet based on the ID supplied
Another day my project management asked me to create a CSV configuration file based on an Excel documents - as usual manual copying was not an option due to required data cleanup and data transformation. So I thought about Apache POI which support XLS and XLSX documents - integration of Apache POI was a breeze but the resulting code was not particularly useful example. So a more generic transformation was provided to show the transformation of Excel documents ...
> ./bin/freemarker-cli -t templates/excel/html/transform.ftl site/sample/excel/test.xls > ./bin/freemarker-cli -t templates/excel/html/transform.ftl site/sample/excel/test.xlsx > ./bin/freemarker-cli -t templates/excel/html/transform.ftl site/sample/excel/test-multiple-sheets.xlsx > ./bin/freemarker-cli -t templates/excel/md/transform.ftl site/sample/excel/test-multiple-sheets.xlsx
The provided FTL transforms an Excel into a HTML document supporting multiple Excel sheets
<#ftl output_format="HTML" > <#assign document = Documents.get(0)> <#assign documentName = document.name> <#assign workbook = ExcelTool.parse(document)> <#assign date = .now?iso_utc> <#---------------------------------------------------------------------------> <!DOCTYPE html> <html> <head> <title>${documentName}</title> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css"> </head> <body> <div class="container-fluid"> <h1>Excel Test <small>${documentName}, ${date}</small> </h1> <@writeSheets workbook/> </div> </body> </html> <#---------------------------------------------------------------------------> <#-- writeSheets --> <#---------------------------------------------------------------------------> <#macro writeSheets workbook> <#assign sheets = ExcelTool.getSheets(workbook)> <#list sheets as sheet> <@writeSheet sheet/> </#list> </#macro> <#---------------------------------------------------------------------------> <#-- writeSheet --> <#---------------------------------------------------------------------------> <#macro writeSheet sheet> <#assign rows = ExcelTool.toTable(sheet)> <h2>${sheet.getSheetName()}</h2> <@writeRows rows/> </#macro> <#---------------------------------------------------------------------------> <#-- writeRows --> <#---------------------------------------------------------------------------> <#macro writeRows rows> <table class="table table-striped"> <#list rows as row> <#if row?is_first> <tr> <th>#</th> <#list row as column> <th>${column}</th> </#list> </tr> <#else> <tr> <td>${row?index}</td> <#list row as column> <td>${column}</td> </#list> </tr> </#if> </#list> </table> </#macro>
but the result looks reasonable
In this sample we transform all property files found in a directory (recursive search using include pattern) to a CSV file
> ./bin/freemarker-cli --include *.properties -t templates/properties/csv/locker-test-users.ftl site/sample/properties TENANT,SITE,USER_ID,DISPOSER_ID,PASSWORD,SMS_OTP,NAME,DESCRIPTION TENANT_A,fat,user_0004,user_0004,password_0004,,, TENANT_B,fat,user_0001,user_0001,password_0001,,, TENANT_B,uat,user_0003,user_0003,password_0003,,, TENANT_C,fat,user_0002,user_0002,password_0004,000000,,Many products
The FTL uses a couple of interesting features
strip_text and compress strips any white-spaces and line-breaks from the output so we can create a proper CSV filetenant and site, e.g. extractTenant${'\n'}<#ftl output_format="plainText" strip_text="true">
<#compress>
TENANT,SITE,USER_ID,DISPOSER_ID,PASSWORD,SMS_OTP,NAME,DESCRIPTION
<#list Documents.list as document>
<#assign properties = PropertiesTool.parse(document)>
<#assign environments = properties["ENVIRONMENTS"]!"">
<#assign tenant = extractTenant(environments)>
<#assign site = extractSite(environments)>
<#assign userId = properties["USER_ID"]!"">
<#assign disposerId = properties["USER_ID"]!"">
<#assign password = properties["PASSWORD"]!"">
<#assign smsOtp = properties["SMS_OTP"]!"">
<#assign name = properties["NAME"]!"">
<#assign description = properties["DESCRIPTION"]!"">
${tenant},${site},${userId},${disposerId},${password},${smsOtp},${name},${description}
</#list>
</#compress>
${'\n'}
<#function extractSite environments>
</#function>
<#function extractTenant environments>
</#function>
For a POC (proof of concept) I created a sample transformation from CSV to XML-FO in order to create a PDF document using Apache FOP using the following template file
<#ftl output_format="XML" > <#assign document = Documents.get(0)> <#assign name = document.name> <#assign cvsFormat = CSVTool.formats.DEFAULT.withDelimiter('\t').withHeader()> <#assign csvParser = CSVTool.parse(document, cvsFormat)> <#assign csvHeaders = csvParser.getHeaderMap()?keys> <#assign csvRecords = csvParser.records> <#---------------------------------------------------------------------------> <?xml version="1.0" encoding="UTF-8"?> <fo:root xmlns:fo="http://www.w3.org/1999/XSL/Format"> <fo:layout-master-set> <fo:simple-page-master master-name="first" page-height="21cm" page-width="29.7cm" margin-top="1cm" margin-bottom="1cm" margin-left="1cm" margin-right="1cm"> <fo:region-body margin-top="1cm"/> <fo:region-before extent="1cm"/> <fo:region-after extent="-1.2cm"/> </fo:simple-page-master> </fo:layout-master-set> <fo:page-sequence master-reference="first"> <fo:static-content flow-name="xsl-region-before"> <fo:block line-height="10pt" font-size="8pt" text-align="left">Transaction Export - ${.now}</fo:block> </fo:static-content> <fo:static-content flow-name="xsl-region-after"> <fo:block line-height="6pt" font-size="6pt" text-align="end">Page <fo:page-number/></fo:block> </fo:static-content> <fo:flow flow-name="xsl-region-body"> <fo:table table-layout="fixed" width="100%" border-collapse="separate"> <fo:table-column column-width="8%"/> <fo:table-column column-width="10%"/> <fo:table-column column-width="12%"/> <fo:table-column column-width="8%"/> <fo:table-column column-width="7%"/> <fo:table-column column-width="5%"/> <fo:table-column column-width="5%"/> <fo:table-column column-width="5%"/> <fo:table-column column-width="35%"/> <fo:table-column column-width="5%"/> <@writeTableHeader headers=csvHeaders/> <@writeTableBody columns=csvRecords/> </fo:table> </fo:flow> </fo:page-sequence> </fo:root> <#---------------------------------------------------------------------------> <#macro writeTableHeader headers> <fo:table-header> <fo:table-row> <#list headers as header> <fo:table-cell border-style="solid" border-width="0.1pt" padding-left="1.0px" padding-right="1.0px"> <fo:block font-size="6pt" font-weight="bold">${header}</fo:block> </fo:table-cell> </#list> </fo:table-row> </fo:table-header> </#macro> <#---------------------------------------------------------------------------> <#macro writeTableBody columns> <fo:table-body> <#list columns as column> <fo:table-row> <#list column.iterator() as field> <fo:table-cell border-style="solid" border-width="0.1pt" padding-left="1.0px" padding-right="1.0px"> <fo:block font-size="6pt">${field}</fo:block> </fo:table-cell> </#list> </fo:table-row> </#list> </fo:table-body> </#macro>
In order to create the PDF you need to execute the following commands (assuming that you have Apache FOP installed)
> ./bin/freemarker-cli -t templates/csv/fo/transform.ftl site/sample/csv/locker-test-users.csv > sample.fo > fop -fo sample.fo sample.pdf Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent WARNING: Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400". Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent WARNING: Font "ZapfDingbats,normal,700" not found. Substituting with "ZapfDingbats,normal,400". Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent INFO: Rendered page #1.
The result does not look very impressive but it is a PDF :-)
Further along the line of the POC we converted a transaction export from CSV to PDF using Apache FOP
> ./bin/freemarker-cli -t templates/csv/fo/transactions.ftl site/sample/csv/transactions.csv > transactions.fo > fop -fo transactions.fo transactions.pdf Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent WARNING: Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400". Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent WARNING: Font "ZapfDingbats,normal,700" not found. Substituting with "ZapfDingbats,normal,400". Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent WARNING: The contents of fo:block line 1 exceed the available area in the inline-progression direction by 11027 millipoints. (See position 1519:51) Jan 16, 2019 11:15:22 PM org.apache.fop.events.LoggingEventListener processEvent INFO: Rendered page #1. Jan 16, 2019 11:15:22 PM org.apache.fop.events.LoggingEventListener processEvent INFO: Rendered page #2.
Recently I got the rather unusual question how to determine the list of dependencies of an application - one easy way is the Maven “dependencies.html” but this is unstructured data. Having said that the Jsoup library is perfectly able to parse most real-life HTML and provides a DOM model
<#ftl output_format="plainText" strip_text="true">
<#assign document = Documents.get(0)>
<#assign documentName = document.name>
<#assign html = JsoupTool.parse(document)>
<#compress>
<@writeHeader/>
<@writeDependencies "Project_Dependencies_compile"/>
<@writeDependencies "Project_Transitive_Dependencies_compile"/>
<@writeDependencies "Project_Transitive_Dependencies_runtime"/>
<@writeDependencies "Project_Transitive_Dependencies_provided"/>
</#compress>
<#macro writeHeader>
GroupId,ArtifactId,Version,Type,Licenses
</#macro>
<#macro writeDependencies section>
<#assign selection = html.select("a[name=${section}]")>
<#if selection?has_content>
<#assign table = selection[0].nextElementSibling().child(2).child(0)>
<#assign rows = table.children()>
<#list rows as row>
<#if !row?is_first>
<#assign groupId = row.child(0).text()>
<#assign artificatId = row.child(1).text()>
<#assign version = row.child(2).text()>
<#assign type = row.child(3).text()>
<#assign licences = row.child(4).text()?replace(",", "")>
${groupId},${artificatId},${version},${type},${licences}
</#if>
</#list>
</#if>
</#macro>
Your dependencies as CSV can be generated as shown below
> ./bin/freemarker-cli -t templates/html/csv/dependencies.ftl site/sample/html/dependencies.html GroupId,ArtifactId,Version,Type,Licenses com.jayway.jsonpath,json-path,2.4.0,jar,The Apache Software License Version 2.0 commons-cli,commons-cli,1.4,jar,Apache License Version 2.0 org.apache.commons,commons-csv,1.5,jar,Apache License Version 2.0 org.apache.poi,poi,4.0.1,jar,The Apache Software License Version 2.0 org.apache.poi,poi-ooxml,3.17,jar,The Apache Software License Version 2.0 org.apache.poi,poi-ooxml-schemas,3.17,jar,The Apache Software License Version 2.0 org.freemarker,freemarker,2.3.28,jar,Apache License Version 2.0 org.jsoup,jsoup,1.11.3,jar,The MIT License org.slf4j,slf4j-api,1.7.21,jar,MIT License org.slf4j,slf4j-log4j12,1.7.21,jar,MIT License com.github.virtuald,curvesapi,1.04,jar,BSD License commons-codec,commons-codec,1.11,jar,Apache License Version 2.0 log4j,log4j,1.2.17,jar,The Apache Software License Version 2.0 net.minidev,accessors-smart,1.2,jar,The Apache Software License Version 2.0 net.minidev,json-smart,2.3,jar,The Apache Software License Version 2.0 org.apache.commons,commons-collections4,4.2,jar,Apache License Version 2.0 org.apache.commons,commons-math3,3.6.1,jar,Apache License Version 2.0 org.apache.xmlbeans,xmlbeans,2.6.0,jar,The Apache Software License Version 2.0 org.ow2.asm,asm,5.0.4,jar,BSD stax,stax-api,1.0.1,jar,The Apache Software License Version 2.0
For a customer project we wanted to record REST request / responses using WireMock - really quick and dirty. So we decided to avoid any sophisticated test tool but generate a ready-to-use shell script executing cURL commands. It turned out that handling of dollar signs is a bit tricky
noparse directive to disable parsing of dollar signs${r"${MY_BASE_URL}" to generate output with dollar signsand the final FTL is found below
<#ftl output_format="plainText">
<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader()>
<#assign csvParser = CSVTool.parse(Documents.get(0), cvsFormat)>
<#assign records = csvParser.records>
<#assign csvMap = CSVTool.toMap(records, "disposer")>
<#--------------------------------------------------------------------------->
#!/bin/sh
<#noparse>
MY_BASE_URL=${MY_BASE_URL:=https://postman-echo.com}
</#noparse>
echo "time,user,status,duration,size"
<#list records as record>
date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',${record.disposer},%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${r"${MY_BASE_URL}"}/get"
</#list>
Rendering the FreeMarker template
> ./bin/freemarker-cli -t ./templates/csv/shell/curl.ftl site/sample/csv/user.csv
generates the following shell script
#!/bin/sh
MY_BASE_URL=${MY_BASE_URL:=https://postman-echo.com}
echo "time,user,status,duration,size"
date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',AAAAAAA,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',BBBBBBB,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',CCCCCCC,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',DDDDDDD,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
Looks a bit complicated but lets dissect the things
date "+%FT%H:%M:%S" | tr -d '\n' creates a timestamp and removes the line feedcurl --write-out allows to print runtime data (see https://ec.haxx.se/usingcurl-writeout.html)Executing the result shell script creates the following output (which is a nice CSV for further processing)
time,user,status,duration,size 2019-09-27T21:02:52,AAAAAAA,200,0.522473,206 2019-09-27T21:02:53,BBBBBBB,200,0.498093,206 2019-09-27T21:02:54,CCCCCCC,200,0.529013,206 2019-09-27T21:02:54,DDDDDDD,200,0.528268,206
Think of Grok as modular regular expressions with a pre-defined functionality to parse access logs or any other data where you can't comprehend the regular expression any longer, one very simple example is QUOTEDSTRING
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
And with Grok the QUOTEDSTRING is just a building block for an even more complex regular expession such as COMBINEDAPACHELOG
bin/freemarker-cli -t templates/accesslog/combined-access.ftl site/sample/accesslog/combined-access.log
which gives you the following output
TIMESTAMP;VERB;REQUEST;HTTPVERSION 19/Jun/2005:06:44:17 +0200;GET;/wximages/wxwidgets02-small.png;1.1 19/Jun/2005:06:46:05 +0200;GET;/wximages/wxwidgets02-small.png;1.1 19/Jun/2005:06:47:37 +0200;GET;/wximages/wxwidgets02-small.png;1.1 19/Jun/2005:06:48:40 +0200;GET;/wiki.pl?WxWidgets_Bounties;1.1 19/Jun/2005:06:50:49 +0200;GET;/wiki.pl?WxWidgets_Compared_To_Other_Toolkits;1.1 19/Jun/2005:06:50:49 +0200;GET;/wxwiki.css;1.1 19/Jun/2005:06:50:49 +0200;GET;/wximages/wxwidgets02-small.png;1.1 19/Jun/2005:06:50:50 +0200;GET;/favicon.ico;1.1 19/Jun/2005:06:52:36 +0200;GET;/wximages/wxwidgets02-small.png;1.1 19/Jun/2005:06:53:14 +0200;GET;/;1.0
using the following FreeMarker template
<#ftl output_format="plainText" strip_whitespace=true>
<#assign grok = GrokTool.compile("%{COMBINEDAPACHELOG}")>
<#assign document = Documents.get(0)>
<#assign lines = document.getLineIterator()>
<#compress>
TIMESTAMP;VERB;REQUEST;HTTPVERSION
<#list lines as line>
<#assign parts = grok.match(line)>
<#assign timestamp = parts["timestamp"]>
<#assign verb = parts["verb"]>
<#assign request = parts["request"]>
<#assign httpversion = parts["httpversion"]>
${timestamp};${verb};${request};${httpversion}
</#list>
</#compress>
While this looks small and tidy there are some nifty features
GrokTool.compile("%{COMBINEDAPACHELOG}") builds the Grok instance to parse access logs in Combined Formatstdin so are able to parse GB of access log or other filesSometimes you have a CSV file which is not quite right - you need to change the format. Lets have a look how freemarker-cli can help
bin/freemarker-cli -Pcsv.in.delimiter=COMMA -Pcsv.out.delimiter=PIPE -t templates/csv/transform.ftl ./site/sample/csv/contract.csv
renders the following template
<#ftl output_format="plainText" strip_text="true">
<#assign csvParser = createCsvParser(Documents.get(0))>
<#assign csvPrinter = createCsvPrinter()>
<#-- Print each line without materializing the CSV in memory -->
<#compress>
<#list csvParser.iterator() as record>
${csvPrinter.printRecord(record)}
</#list>
</#compress>
<#function createCsvParser document>
<#assign initialCvsInFormat = CSVTool.formats[SystemTool.getParameter("csv.in.format", "DEFAULT")]>
<#assign csvInDelimiter = CSVTool.toDelimiter(SystemTool.getParameter("csv.in.delimiter", initialCvsInFormat.getDelimiter()))>
<#assign cvsInFormat = initialCvsInFormat.withDelimiter(csvInDelimiter)>
<#return CSVTool.parse(document, cvsInFormat)>
</#function>
<#function createCsvPrinter>
<#assign initialCvsOutFormat = CSVTool.formats[SystemTool.getParameter("csv.out.format", "DEFAULT")]>
<#assign csvOutDelimiter = CSVTool.toDelimiter(SystemTool.getParameter("csv.out.delimiter", initialCvsOutFormat.getDelimiter()))>
<#assign cvsOutFormat = initialCvsOutFormat.withDelimiter(csvOutDelimiter)>
<#return CSVTool.printer(cvsOutFormat, SystemTool.writer)>
</#function>
and generates the following output
contract_id|seller_company_name|customer_company_name|customer_duns_number|contract_affiliate|FERC_tariff_reference|contract_service_agreement_id|contract_execution_date|contract_commencement_date|contract_termination_date|actual_termination_date|extension_provision_description|class_name|term_name|increment_name|increment_peaking_name|product_type_name|product_name|quantity|units_for_contract|rate|rate_minimum|rate_maximum|rate_description|units_for_rate|point_of_receipt_control_area|point_of_receipt_specific_location|point_of_delivery_control_area|point_of_delivery_specific_location|begin_date|end_date|time_zone C71|The Electric Company|The Power Company|456543333|N|FERC Electric Tariff Original Volume No. 10|2|2/15/2001|2/15/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||||||ES C72|The Electric Company|Utility A|38495837|n|FERC Electric Tariff Original Volume No. 10|15|7/25/2001|8/1/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||||||ES C73|The Electric Company|Utility B|493758794|N|FERC Electric Tariff Original Volume No. 10|7|6/8/2001|7/6/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||" "|" "|||ep C74|The Electric Company|Utility C|594739573|n|FERC Electric Tariff Original Volume No. 10|25|6/8/2001|7/6/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||" "|" "|||ep
Some useful hints
Using Apache Commons Exec allows to execute arbitrary commands - nice but dangerous. It was recently quite useful to to invoke AWS CLI to generate a Confluence page about the overall setup of our AWS accounts.
A few snippets to illustrate the points
<#ftl output_format="plainText" strip_whitespace="true">
<#assign profile = SystemTool.getProperty("profile", "default")>
<#assign ec2Instances = ec2Instances()/>
h3. AWS EC2 Instance
<@printEc2Instances ec2Instances/>
<#function ec2Instances>
<#local json = awsCliToJson("aws ec2 describe-instances --profile ${profile}")>
<#local instances = json.read("$.Reservations[*].Instances[*]")>
<#return instances?sort_by(['InstanceType'])>
</#function>
<#function awsCliToJson line>
<#local output = ExecTool.execute(line)>
<#return JsonPathTool.parse(output)>
</#function>
<#function getAwsEc2InstanceTag tags name>
<#return tags?filter(x -> x["Key"] == name)?first["Value"]!"">
</#function>
<#macro printEc2Instances ec2Instances>
<#compress>
|| NAME || INSTANCE_TYPE || VCPUS || STATE || PRIVATE_IP_ADDRESS ||
<#list ec2Instances as ec2Instance>
<#assign instanceType = ec2Instance["InstanceType"]>
<#assign arn = ec2Instance["IamInstanceProfile"]["Arn"]>
<#assign privateIpAddress = ec2Instance["PrivateIpAddress"]>
<#assign state = ec2Instance["State"]["Name"]>
<#assign launchTime = ec2Instance["LaunchTime"]>
<#assign coreCount = ec2Instance["CpuOptions"]["CoreCount"]?number>
<#assign threadsPerCore = ec2Instance["CpuOptions"]["ThreadsPerCore"]?number>
<#assign nrOfVirtualCpus = coreCount * threadsPerCore>
<#assign tags = ec2Instance["Tags"]/>
<#assign awsCloudFormationStackId = getAwsEc2InstanceTag(tags, "aws:cloudformation:stack-id")>
<#assign awsCloudFormationStackName = getAwsEc2InstanceTag(tags, "aws:cloudformation:stack-name")>
<#assign name = getAwsEc2InstanceTag(tags, "Name")>
<#assign country = getAwsEc2InstanceTag(tags, "Country")>
<#assign environment = getAwsEc2InstanceTag(tags, "Environment")>
| ${name} | ${instanceType} | ${nrOfVirtualCpus} | ${state} | ${privateIpAddress} |
</#list>
</#compress>
</#macro>
Sometime you need to apply a CSS, JSON or XPath query in ad ad-hoc way without installing xmllint, jq or pup - in this case you can pass a FreeMarker template in an interactive fashion
> bin/freemarker-cli -i 'Hello ${SystemTool.envs["USER"]}'; echo Hello sgoeschl > bin/freemarker-cli -i '${JsonPathTool.parse(Documents.first).read("$.info.title")}' site/sample/json/swagger-spec.json; echo Swagger Petstore > bin/freemarker-cli -i '${XmlTool.parse(Documents.first)["recipients/person[1]/name"]}' site/sample/xml/recipients.xml; echo John Smith > bin/freemarker-cli -i '${JsoupTool.parse(Documents.first).select("a")[0]}' site/sample/html/dependencies.html; echo <a href="${project.url}" title="FreeMarker CLI">FreeMarker CLI</a> > ./bin/freemarker-cli -i '<#list SystemTool.envs as name,value>${name} ==> ${value}${"\n"}</#list>' TERM ==> xterm-256color LANG ==> en_US DISPLAY ==> :0.0 SHELL ==> /bin/bash EDITOR ==> vi
During an integration project we imported large transactions CSV files (500.000+ records) and in case of import failures the developers would be happy to get a nice outline of the transactions causing the problem (the CSV records have 60+ columns) - in essence it is filtering (based on some primary key) and and transforming into an output format (Markdown).
So lets start the filtering & transformation using the following command line
> bin/freemarker-cli -e UTF-8 -l de_AT -Dcolumn="Order ID" \ -Dvalues=226939189,957081544 \ -Dformat=DEFAULT \ -Ddelimiter=COMMA \ -t templates/csv/md/filter.ftl site/sample/csv/sales-records.csv
and Apache FreeMarker template
<#ftl output_format="plainText" strip_text="true">
<#assign document = Documents.get(0)>
<#assign parser = parser(document)>
<#assign headers = parser.getHeaderNames()>
<#assign column = SystemTool.getProperty("column")>
<#assign values = SystemTool.getProperty("values")?split(",")>
<#-- Process each line without materializing the whole file in memory -->
<#compress>
<@writePageHeader document/>
<#list parser.iterator() as record>
<#if filter(record)>
<@writeCsvRecord headers record/>
</#if>
</#list>
</#compress>
<#function parser document>
<#assign format = CSVTool.formats[SystemTool.getProperty("format", "DEFAULT")]>
<#assign delimiter = CSVTool.toDelimiter(SystemTool.getProperty("delimiter", format.getDelimiter()))>
<#return CSVTool.parse(document, format.withFirstRecordAsHeader().withDelimiter(delimiter))>
</#function>
<#function filter record>
<#return values?seq_contains(record.get(column))>
</#function>
<#macro writePageHeader document>
# ${document.name}
</#macro>
<#macro writeCsvRecord headers record>
## Line ${record.getRecordNumber()}
| Column | Value |
| --------- | --------------------------- |
<#list headers as header>
| ${header} | ${record.get(header)} |
</#list>
</#macro>
yields
# sales-records.csv ## Line 1 | Column | Value | | --------- | --------------------------- | | Region | Central America and the Caribbean | | Country | Antigua and Barbuda | | Item Type | Baby Food | | Sales Channel | Online | | Order Priority | M | | Order Date | 12/20/2013 | | Order ID | 957081544 | | Ship Date | 1/11/2014 | | Units Sold | 552 | | Unit Price | 255.28 | | Unit Cost | 159.42 | | Total Revenue | 140914.56 | | Total Cost | 87999.84 | | Total Profit | 52914.72 | ## Line 4.998 | Column | Value | | --------- | --------------------------- | | Region | Asia | | Country | Myanmar | | Item Type | Baby Food | | Sales Channel | Offline | | Order Priority | H | | Order Date | 11/23/2016 | | Order ID | 226939189 | | Ship Date | 12/10/2016 | | Units Sold | 5204 | | Unit Price | 255.28 | | Unit Cost | 159.42 | | Total Revenue | 1328477.12 | | Total Cost | 829621.68 | | Total Profit | 498855.44 |
There is a demo.ftl which shows some advanced FreeMarker functionality
Running
./bin/freemarker-cli -t templates/demo.ftl
gives you
1) FreeMarker Special Variables --------------------------------------------------------------------------- FreeMarker version : 2.3.29 Template name : templates/demo.ftl Language : en Locale : en_US Timestamp : Feb 22, 2020 4:54:19 PM Output encoding : UTF-8 Output format : plainText 2) Invoke a constructor of a Java class --------------------------------------------------------------------------- new java.utilDate(1000 * 3600 * 24): Jan 2, 1970 1:00:00 AM 3) Invoke a static method of an non-constructor class --------------------------------------------------------------------------- Random UUID : 1fdb5ead-80a3-418c-bcc7-242e41b4e950 System.currentTimeMillis : 1,582,386,859,782 4) Access an Enumeration --------------------------------------------------------------------------- java.math.RoundingMode#UP: UP 5) Loop Over The Values Of An Enumeration --------------------------------------------------------------------------- - java.math.RoundingMode.UP - java.math.RoundingMode.DOWN - java.math.RoundingMode.CEILING - java.math.RoundingMode.FLOOR - java.math.RoundingMode.HALF_UP - java.math.RoundingMode.HALF_DOWN - java.math.RoundingMode.HALF_EVEN - java.math.RoundingMode.UNNECESSARY 6) Display list of input files --------------------------------------------------------------------------- List all files: 7) SystemTool --------------------------------------------------------------------------- Host name : W0GL5179.local Command line : -t, templates/demo.ftl System property : sgoeschl Timestamp : 1582386860080 Environment var : sgoeschl 8) Access System Properties --------------------------------------------------------------------------- app.dir : app.home : /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler app.pid : 71792 basedir : /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler java.version : 1.8.0_192 user.name : sgoeschl user.dir : /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler user.home : /Users/sgoeschl 9) List Environment Variables --------------------------------------------------------------------------- - TERM ==> xterm-256color - LANG ==> en_US - DISPLAY ==> :0.0 - SHELL ==> /bin/bash - EDITOR ==> vi 10) List System Properties --------------------------------------------------------------------------- - java.runtime.name ==> Java(TM) SE Runtime Environment - java.vm.version ==> 25.192-b12 - java.vm.vendor ==> Oracle Corporation - java.vendor.url ==> http://java.oracle.com/ - java.vm.name ==> Java HotSpot(TM) 64-Bit Server VM 11) Access Documents --------------------------------------------------------------------------- Get the number of documents: - 0 List all files containing "README" in the name List all files having "md" extension Get all documents 12) FreeMarker CLI Tools --------------------------------------------------------------------------- - CSVTool : Process CSV files using Apache Commons CSV (see https://commons.apache.org/proper/commons-csv/) - ExcelTool : Process Excels files (XLS, XLSX) using Apache POI (see https://poi.apache.org) - ExecTool : Execute command line tools using Apache Commons Exec (see https://commons.apache.org/proper/commons-exec/) - FreeMarkerTool : Expose useful Apache FreeMarker classes - GrokTool : Process text files using Grok expressions (see https://github.com/thekrakken/java-grok) - JsonPathTool : Process JSON files using Java JSON Path (see https://github.com/json-path/JsonPath) - JsoupTool : Process HTML files using Jsoup (see https://jsoup.org) - PropertiesTool : Process JDK properties files - SystemTool : Expose System-related utility methods - UUIDTool : Create UUIDs - XmlTool : Process XML files using Apache FreeMarker (see https://freemarker.apache.org/docs/xgui.html) - YamlTool : Process YAML files using SnakeYAML(see https://bitbucket.org/asomov/snakeyaml/wiki/Home) 13) Document Data Model --------------------------------------------------------------------------- - CSVTool - Documents - ExcelTool - ExecTool - FreeMarkerTool - GrokTool - JsonPathTool - JsoupTool - PropertiesTool - SystemTool - UUIDTool - XmlTool - YamlTool 14) Create a UUID --------------------------------------------------------------------------- UUIDTool Random UUID : 936315a5-ee3d-44cd-a32c-2be10d2249a6 UUIDTool Named UUID : 298415f9-e888-3d98-90e7-6c0d63ad14dc 15) Printing Special Characters --------------------------------------------------------------------------- German Special Characters: äöüßÄÖÜ 16) Locale-specific output --------------------------------------------------------------------------- Small Number : 1.23 Large Number : 12,345,678.90 Date : Feb 22, 2020 Time : 4:54:20 PM 17) Execute a program --------------------------------------------------------------------------- > date Sat Feb 22 16:54:20 CET 2020
stdinstdoutWithin the script a FreeMarker data model is set up and passed to the template - it contains the documents to be processed and the following tools
| Entry | Description |
|---|---|
| CSVTool | Process CSV files using Apache Commons CSV |
| ExecTool | Execute command line tools using Apache Commons Exec |
| ExcelTool | Process Excels files (XLS, XLSX) using Apache POI |
| Documents | Helper class to find documents, e.g. by name, extension or index |
| FreeMarkerTool | Expose useful FreeMarker classes |
| GrokTool | Process text files using Grok instead of regular expressions |
| JsonPathTool | Process JSON file using Java JSON Path |
| JsoupTool | Processing HTML files using Jsoup |
| PropertiesTool | Process JDK properties files |
| SystemTool | System-related utility methods |
| XmlTool | Process XML files using Apache FreeMarker |
| YamlTool | Process YAML files using SnakeYAML |
| UUIDTool | Create UUIDs |
When doing some ad-hoc scripting it is useful to rely on a base directory to resolve the FTL templates
-b or --basedir command line parameter./bin/freemarker-cli -t templates/json/html/customer-user-products.ftl freemarker-cli/site/sample/json/customer-user-products.jso
When doing ad-hoc scripting it useful to pipe the output of one command directly into “freemarker-cli”
cat site/sample/json/customer-user-products.json | ./bin/freemarker-cli -t ./templates/json/html/customer-user-products.ftl --stdin