tag | a328b5ca39c8c59e92121addd3fcd55908d0ec13 | |
---|---|---|
tagger | nchung <nga.t.chung@jpl.nasa.gov> | Wed Jul 13 09:24:17 2022 -0700 |
object | 7664731acfc87be361e667db0d3b79bd43174b1c |
Release v0.3.0
commit | 7664731acfc87be361e667db0d3b79bd43174b1c | [log] [tgz] |
---|---|---|
author | nchung <nga.t.chung@jpl.nasa.gov> | Wed Jul 13 09:23:18 2022 -0700 |
committer | nchung <nga.t.chung@jpl.nasa.gov> | Wed Jul 13 09:23:18 2022 -0700 |
tree | 4a918247dc3f2171ef2b1f5b9bfb03b1600a7e52 | |
parent | 3609f2cc4bbbd0786bece381a53781a17b593a6d [diff] |
/version 0.3.0
Assumption: K8s is successfully deployed
Download this repo
(optional) create different python3.6 environment
install dependencies
python3 setup.py install
setup AWS tokens
export AWS_ACCESS_KEY_ID=xxx export AWS_SECRET_ACCESS_KEY=xxx export AWS_SESSION_TOKEN=really.long.token export AWS_REGION=us-west-2
default
profile under ~/.aws/credentials
can be setup as wellsetup current directory to PYTHONPATH
PYTHONPATH="${PYTHONPATH}:/absolute/path/to/current/dir/"
run the script:
python3 -m parquet_cli.ingest_s3 --help
sample script:
python3 -m parquet_cli.ingest_s3 \ --LOG_LEVEL 30 \ --CDMS_DOMAIN https://doms.jpl.nasa.gov/insitu \ --CDMS_BEARER_TOKEN Mock-CDMS-Flask-Token \ --PARQUET_META_TBL_NAME cdms_parquet_meta_dev_v1 \ --BUCKET_NAME cdms-dev-ncar-in-situ-stage \ --KEY_PREFIX cdms_icoads_2017-01-01.json
https://stackoverflow.com/questions/38487667/overwrite-specific-partitions-in-spark-dataframe-write-method?noredirect=1&lq=1 > Finally! This is now a feature in Spark 2.3.0: SPARK-20236 > To use it, you need to set the spark.sql.sources.partitionOverwriteMode setting to dynamic, the dataset needs to be partitioned, and the write mode overwrite. Example: > https://stackoverflow.com/questions/50006526/overwrite-only-some-partitions-in-a-partitioned-spark-dataset spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic") data.toDF().write.mode("overwrite").format("parquet").partitionBy("date", "name").save("s3://path/to/somewhere")