If you are running in Amazon AWS, Heron provides out of the box uploader for S3, the object storage. S3 uploader is useful when the topologies run in a distributed cluster of Amazon EC2 compute instances. Since S3 replicates the data, it provides a scalable mechanism for distributing the user topology jars.
You can make Heron use S3 uploader by modifying the uploader.yaml config file specific for the Heron cluster. You'll need to specify the following for each cluster:
heron.class.uploader --- Indicate the uploader class to be loaded. You should set this to org.apache.heron.uploader.s3.S3Uploader
heron.uploader.s3.bucket --- Specifies the S3 bucket where the topology jar should be uploaded.
heron.uploader.s3.access_key --- Specify the access key of the AWS account that has write access to the bucket
heron.uploader.s3.secret_key --- Specify the secret access of the AWS account that has write access to the bucket
Below is an example configuration (in uploader.yaml) for a S3 uploader:
# uploader class for transferring the topology jar/tar files to storage heron.class.uploader: org.apache.heron.uploader.s3.S3Uploader # S3 region bucket is created. Must specify. heron.uploader.s3.region: us-east-1 # S3 bucket to put the jar file into heron.uploader.s3.bucket: heron-topologies-company-com # AWS access key heron.uploader.s3.access_key: access_key # AWS secret access key heron.uploader.s3.secret_key: secret_access_key