conversion-notes.adoc

= Notes on conversion from cms to Antora/asciidoc
:project: felix

== Previous history preservation

While all history remains in svn repositories, this is going to be increasingly inaccessible as people forget how to use svn.
Therefore, previous history for the source and rendered html versions of the website is transferred to git using svn2git.
The authors.txt file needs to be enhanced with an entry for buildbot.

[source,console,subs=+attributes]
----
curl https://gitbox.apache.org/authors.txt
----

Add `buildbot = buildbot <buildbot@apache.org>`.
This is probably not needed for CMS-only sites but was needed for TomEE.

[source,console,subs=+attributes]
----
mkdir {project}/site
cd {project}/site
svn2git https://svn.apache.org/repos/asf/aries/site/trunk/content/modules/ --authors ../authors.txt --rootistrunk --no-minimize-url -m
#git gc #sometimes there's a gc error at the end of previous command
git tag -m "last cms revision" last-cms
----

[source,console,subs=+attributes]
----
mkdir {project}/site-pub
cd {project}/site-pub
svn2git https://svn-master.apache.org/repos/infra/websites/production/{project}/content --authors ../authors.txt --rootistrunk --no-minimize-url -m
#git gc #sometimes there's a gc error at the end of previous command
git tag -m "content from last cms revision" last-cms
----

== Move to Antora structure

In {project}/site:

CMS assets are at the root.
Start by moving them into Antora structure under `module/ROOT/pages`.
After getting an "exact copy" version working, consider reorganizing with Antora components, versions, and modules.

.Commands used for Aries
[source,console,subs=+attributes]
----
mkdir -p modules/ROOT/pages
ls
git mv *.mdtext modules/ROOT/pages/
git mv community modules/ROOT/pages/
git mv development modules/ROOT/pages/
git mv documentation modules/ROOT/pages/
git mv downloads modules/ROOT/pages/
git mv images modules/ROOT/
git mv resources modules/ROOT/pages/
git mv schemas modules/ROOT/pages/
----


.Commands used for Felix
[source,console,subs=+attributes]
----
mkdir -p modules/ROOT/pages
ls
git mv *.mdtext modules/ROOT/pages/
git mv components/ modules/ROOT/pages/
git mv documentation modules/ROOT/pages/
git mv errors/ modules/ROOT/pages/
ls icons/
git mv ipojo/ modules/ROOT/pages/
git mv miscellaneous/ modules/ROOT/pages/
git mv obr/ modules/ROOT/pages/
ls res
ls site
ls apidocs/
mkdir -p modules/ROOT/attachments
git mv apidocs/  modules/ROOT/attachments/

git commit -m "move to basic antora structure"
----

Note moving existing javadoc to attachments.
The commands here will obviously depend on the structure of the existing site.
This also leaves icons and other website resources unmoved; whether they go into the UI or images directory can be determined later.

== Markdown to Asciidoc

Asciidoc-kramdown is the best converter.
It seems easiest to build a script to do the conversion, then run it.


In {project}/site:

[source,console,subs=+attributes]
----
echo '#!/bin/sh' >convert.sh
find . -name "*.mdtext" -type f -exec sh -c     'echo echo $0\\nkramdoc --format=GFM --wrap=ventilate $0' {} \; >> convert.sh
chmod u+x convert.sh
./convert.sh
----

There might be errors; the name of the file they occur in will be printed before the error.
Figure out what they are and fix them.
Note the stacktrace is backwards from Java.

Some errors with causes:

`.../writer.rb:101:in `+': no implicit conversion of nil into String (TypeError)`:: `<< \| >>`

`.../converter.rb:99:in `convert': undefined method `convert_raw' for #<Kramdown::AsciiDoc::Converter:0x00007fc0be123320> (NoMethodError)`
::Raw style html such as `<style type="text/css">img { width:auto;}</style>`

`.../writer.rb:101:in `append': no implicit conversion of String into Array (TypeError)`::
* A section `# Section` cannot be immediately followed by `:::xml`.
Insert a `* fake point`.
* A blank line in an `:::XML` block
* Sometimes, embedded html such as `<div...>`

When the convert scripts works without errors, commit the result:

[source,console,subs=+attributes]
----
git add modules
git commit -m "basic kramdoc conversion to asciidoc"
----

== Antora component descriptor

Add an antora.yml file at the root containing

[source,yml,subs=+attributes]
----
name: documentation
title: Documentation
version: master
start_page: index.adoc
nav:
  - modules/ROOT/nav.adoc
----

Add an empty nav.adoc file at modules/ROOT/.

== Playbook and UI

Install `@djencks/antora-schematics`.

`npm i -g @djencks/antora-schematics`

=== Construct UI project

In root directory,

[source,console,subs=+attributes]
----
antora-schematics uiExtension --path {project}-antora-ui --extensionName {project}-antora --namespace '@{project}' --gitPath {project}-antora-ui --no-commit
cd {project}-antora-ui
// remove build from .gitignore
npm i
gulp
git add build
git commit -m "initial {project} UI project"
----

=== Playbook

In root directory,

[source,console,subs=+attributes]
----
antora-schematics playbook --path {project}-antora --gitPath {project}-antora --siteURL https://{project}.apache.org --no-commit  --siteTitle Apache {project} --sources ./../{project}-site --packageManager none --uiBundle node_modules/@{project}/{project}-antora-ui/build/{project}-antora-ui-bundle.zip
cd {project}-antora
// add '  "@{project}/{project}-antora-ui": "./../{project}-antora-ui"` to package.json devDependencies
git commit -m "initial {project} playbook project" .
npm run clean-build
//or npm run clean-install
----

== Fix the errors

Most likely there will be lots of errors from the markdown conversion.
One obvious thing to check for is that the pages start with a title `= This is the Title`.
Many Apache markdown pages have migrated automatically from previous systems such as Confluence with the markdown in a basically broken state.
Since Asciidoc has some notion of grammar, you'll need to fix the problems.

`:doctype: book

Title: ` :: replace with `= `

Error `level 0 sections can only be used when doctype is book` :: More deeply nest sections so body sections are at least ``== ``.
This may be facilitated in jetbrains products with an on-page regex search/replace from ``(=+ )`` to `=$1`.

There are numerous classes of problems this does not cover.
The first step is probably to eliminate the command line errors/warnings.

== Auto index pages

Add to {project}-antora package.json devDependencies `"@djencks/asciidoctor-antora-indexer": "^0.0.4"`

Run `npm run clean-install`.

To the `antora-playbook.yml` playbook add

[source,yml]
----
asciidoc:
  extensions:
    - "@djencks/asciidoctor-antora-indexer"
----

Add a page `auto-index.adoc` to the {project-site} project containing

[source,adoc]
----
= Temporary auto-index page
//uncomment to generate temporary nav file contents on console.
:antora-indexer-log-lists:

indexList::[]
----

Run `npm run build`.
You should see an adoc list output on the console, listing all pages in order.

== Construct nav file

Copy this list from the console to the empty `nav.adoc`.

== Move images

[source,console]
----
echo '#!/bin/sh' >move-png.sh
find modules -name "*.png" -exec sh -c     'echo mkdir -p `dirname $0`\\ngit mv $0 $0' {} \; |sed 's/\([gp]\) modules\/ROOT\/pages/\1 modules\/ROOT\/images/g' >>move-png.sh
chmod u+x move-png.sh
./move-png.sh
----

You may need to move pngs with spaces in their filenames by hand.

== Fix xrefs