commit | 4e93f7521a9d7d58b351b5f59a8be18b0a646534 | [log] [tgz] |
---|---|---|
author | wphyojpl <38299756+wphyojpl@users.noreply.github.com> | Wed Jun 22 13:03:54 2022 -0700 |
committer | GitHub <noreply@github.com> | Wed Jun 22 13:03:54 2022 -0700 |
tree | 466116c2d8e144607ac9e2c44b65f80b0a15cb7e | |
parent | f77a8338ac8094c9cf15e4619f023a30139bff42 [diff] |
feat: CLI script to ingest S3 files (#5) * feat: add CLI script * chore: update readme * chore: add changelog
Assumption: K8s is successfully deployed
Download this repo
(optional) create different python3.6 environment
install dependencies
python3 setup.py install
setup AWS tokens
export AWS_ACCESS_KEY_ID=xxx export AWS_SECRET_ACCESS_KEY=xxx export AWS_SESSION_TOKEN=really.long.token export AWS_REGION=us-west-2
default
profile under ~/.aws/credentials
can be setup as wellsetup current directory to PYTHONPATH
PYTHONPATH="${PYTHONPATH}:/absolute/path/to/current/dir/"
run the script:
python3 -m parquet_cli.ingest_s3 --help
sample script:
python3 -m parquet_cli.ingest_s3 \ --LOG_LEVEL 30 \ --CDMS_DOMAIN https://doms.jpl.nasa.gov/insitu \ --CDMS_BEARER_TOKEN Mock-CDMS-Flask-Token \ --PARQUET_META_TBL_NAME cdms_parquet_meta_dev_v1 \ --BUCKET_NAME cdms-dev-ncar-in-situ-stage \ --KEY_PREFIX cdms_icoads_2017-01-01.json
https://stackoverflow.com/questions/38487667/overwrite-specific-partitions-in-spark-dataframe-write-method?noredirect=1&lq=1 > Finally! This is now a feature in Spark 2.3.0: SPARK-20236 > To use it, you need to set the spark.sql.sources.partitionOverwriteMode setting to dynamic, the dataset needs to be partitioned, and the write mode overwrite. Example: > https://stackoverflow.com/questions/50006526/overwrite-only-some-partitions-in-a-partitioned-spark-dataset spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic") data.toDF().write.mode("overwrite").format("parquet").partitionBy("date", "name").save("s3://path/to/somewhere")