src/site/markdown/mkcsv.md

<!---
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

# mkcsv

Creates a CSV file with a given path and a specified number of records; useful for scale testing CSV processing.
The records are of varying length and contain a calculated CRC32 column so corruption can be detected.

```
Usage: mkcsv
        -D <key=value>  Define a single configuration option
        -sysprop <file> Property file of system properties
        -tokenfile <file>       Hadoop token file to load
        -xmlfile <file> XML config file to load
        -verbose        verbose output
        -debug  enable JVM logs (ALL) and override log4j levels (DEBUG) on specified packages or classes
        -logoverrides <file>    A newline separated list of package and class names
        -header print a header row
        -quote  quote column text
        <records> <path>
```

```bash
hadoop jar cloudstore-1.6.jar mkcsv  -header -quote -verbose  10000 s3a://bucket/file.csv
```

The format is a variable width sequence, with entries cross referencing each other for validation.
```csv
"start","rowId","length","dataCrc","data","rowId2","rowCrc","end"
"start","1","87","691051183","bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","1","2707924207","end"
"start","2","40","2886466480","cccccccccccccccccccccccccccccccccccccccc","2","2141198053","end"
"start","3","98","3320970725","dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd","3","4203069111","end"
"start","4","8","1257926895","eeeeeeee","4","189792478","end"
"start","5","25","1630497970","fffffffffffffffffffffffff","5","1034603103","end"
"start","6","38","557554018","gggggggggggggggggggggggggggggggggggggg","6","1412646710","end"
"start","7","86","951894681","hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh","7","2062289315","end"
"start","8","45","3065088391","iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii","8","3774714774","end"
"start","9","70","2839984696","jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj","9","303056462","end"
```

## Invariants

For each row
```java
start == "start"
rowId == rowId2
length == a random int >= 0    
data = string where data.length() == length
       elements of data == char c where c in "[a-z][A-Z][0-9]"
dataCrc == new CRC32().update(data.getBytes(StandardCharsets.UTF_8))
rowCrc == crc32 of all previous fields, including quotes, *excluding separators*
end == "end"
// and ignoring headers    
forall n: row[n].rowId == n
```


## Schemas for Apache Spark
```scala

/**
 * Dataset class.
 * Latest build is "start","rowId","length","dataCrc","data","rowId2","rowCrc","end"
 */
case class CsvRecord(
    start: String,
    rowId: Long,
    length: Long,
    dataCrc: Long,
    data: String,
    rowId2: Long,
    rowCrc: Long,
    end: String)

/**
 * The StructType of the CSV data.
 * "start","rowId","length","dataCrc","data","rowId2","rowCrc","end"
 */
val csvSchema: StructType = {
  new StructType().
    add("start", StringType).
    add("rowId", LongType).
    add("length", LongType).
    add("dataCrc", LongType).
    add("data", StringType).
    add("rowId2", LongType).
    add("rowCrc", LongType).
    add("end", StringType)
}

```