maintenance.txt 1.0.0                 UTF-8                       2015-01-08

                  MAINTAIN THE EXTERNAL DOWNLOADS PROCEDURES
                  ==========================================

    The external\ folder holds scripts that provide for specific downloads
    to be obtained over the internet and retained in a download\ sub-folder.
    The scripts also provide for the extraction of code and libraries from
    the downloaded archives and making them available for use in builds on
    Windows and in deployed packages on Windows.

    When the source or version of downloads change, and other downloads are
    added, these procedures must be maintained.  Almost all of the maintenance
    and related customization is in the two procedure fetch_downloads.bat
    and extract_downloads.bat.

    There are interdependencies among the two scripts and within the second
    script that must be maintained and carefully verified.

    This file provides guidance on how to preserve correct, tested operation
    as maintenance and customization are performed.

MAINTAINING FETCH_DOWNLOADS.BAT
-------------------------------

CREATING THE DOWNLOAD\ SUB-FOLDER

This file creates the download\ subdirectory in the same folder as the script
when it is not present already.  It does not remove anything from a folder
that is already there.

The single command

        MKDIR "%~dp0download" >nul 2>&1

accomplishes that.  The script parameter %~dp0 uses the drive and path of
the location of the script itself (ending in "\") and appends the download
name for the desired subdirectory.  (This scheme is used throughout the
scripts.)  Any console output is routed to nul, and any error message output
is routed to the same place as the console.  This eliminates nuisance warnings
about directories already existing, etc.

DOWNLOADING THE EXTERNAL CONTENT

Each download is accomplished with one of the lines that performs a
CALL :FETCH operation.  The one parameter on the same line is the full URL
of a file to download.

There are two important variables to observe.  The full URL must be the
complete location for accessing the file on the Internet using HTTP.  The
downloads must use HTTP in this implementation.

Furthermore, the name by which the download will be stored will be the
file name at the end of the URL.  The :FETCH procedure takes apart the
URL and specifies the correct file name to use when storing in the download\
folder.

The fetch_downloads.bat procedure is mostly silent.  If it is discovered
that a file to be downloaded is already present, it is left intact and not
downloaded again.  If there is a download operation, and the file is present
afterwards, the name of the file is announced on the console.  And if that
download fails, there is also silence.

TESTING AND TROUBLE-SHOOTING

If there is any doubt about a download, the easiest way to determine if it
is working is by putting the URL in the address bar of a browser and seeing
whether the file is delivered.

It is possible to run fetch_downloads.bat multiple times.  Once all of the
files are all present (if downloadable), there will be no further output.
Look in the download\ directory to see what files are present.

It may be necessary to trouble-shoot collisions of names that are the same
when case is ignored, and there can be other problems with special characters
that are not permitted in Windows file names (or that are reserved names, like
"nul").

Finally, the wget-win.js never fails, and it will always store *something*
at the destination location.  This can lead to file-not-found messages in the extract_downloads.bat script.


MAINTAINING EXTRACT_DOWNLOADS.BAT
---------------------------------

The extract_downloads.bat script first does a CALL fetch_downloads.bat.  This
does no harm and it automatically obtains the downloads if they are not there
already.

CHECKING THAT THE DOWNLOADS ARE PRESENT

The first part of extract_downloads.bat is a series of CALL :CHECK commands.
Each one of these names one of the downloads that should now be present.
If they are not all there, the procedure will report that and stop.

The important feature of the CALL :CHECK commands is they must list exactly
the same files that are specified in the CALL :FETCH commands in
fetch_downloads.bat.  These are the names the files should have been stored
with.

It is possible that a file is present but it is incorrect.  This will usually
show up with unusual results from the extract operations.

PERFORMING EXTRACTIONS

The actual extraction procedure is very specific to the individual downloads.
There is some abstraction into general, reusable procedures, but they are
still very specific to the nature of the download.

There is a consistent extraction API for these custom routines.  The first
parameter is the name of the downloaded file.  This should agree with the
name in the CALL :CHECK operation.

The second parameter indicates where inside of a Zip, the interesting data
starts.  Many of the Zip files unload a single directory as their top level
and then the actual payload is at the level below that directory.  When that
is the case, the path prior to the payload is specified with a non-empty
second parameter.

This can be seen by examining some of the calls and comparing with how the
Zip files are organized by inspecting them in the Windows file system.

What's important:
 1. When an existing download is upgraded to use a new version, the file name
    will change and so many any second parameter, since it often is a folder
    whose name is based on the version of the related library.
 2. When an existing download is upgraded, the structure of the main material
    may change, so a different set of XCOPY operations may be required to
    extract the material correctly.

This is even more involved when a new custom download is introduced.

TESTING AND TROUBLESHOOTING

The easiest way to test individual extractions is to remark-out all of the
CALL procedures but the one to be tested.  The extraction from the one
archive can be seen in the download\T\ folder, and one can verify the
download\include\, download\lib\, and download\bin results manually.


 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

      Licensed to the Apache Software Foundation (ASF) under one
      or more contributor license agreements.  See the NOTICE file
      distributed with this work for additional information
      regarding copyright ownership.  The ASF licenses this file
      to you under the Apache License, Version 2.0 (the
      "License"); you may not use this file except in compliance
      with the License.  You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing,
      software distributed under the License is distributed on an
      "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
      KIND, either express or implied.  See the License for the
      specific language governing permissions and limitations
      under the License.

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 1.0.0 2015-01-08-19:12 Create initial version to account for the way
       the scripts operate and will need to be modified on upgrades of
       any of the downloads.

                     *** end of maintenance.txt ***