Administrator setup

With Cloud Shell:

Assumed variables,

1. Create a user and a group

Create a user and a group, and join user to the created group.

create-user

[cloudshell-user@host ~]$ aws iam create-user --user-name systemds-bot
{
    "User": {
        "Path": "/",
        "UserName": "systemds-bot",
        "UserId": "AIDAQSHHX7DDAODFXYZ3",
        "Arn": "arn:aws:iam::12345:user/systemds-bot",
        "CreateDate": "2021-04-10T20:36:59+00:00"
    }
}

create-group

[cloudshell-user@host ~]$aws iam create-group --group-name systemds-group
{
    "Group": {
        "Path": "/",
        "GroupName": "systemds-group",
        "GroupId": "AGPAQSHHX7DDB3XYZABCW",
        "Arn": "arn:aws:iam::12345:group/systemds-group",
        "CreateDate": "2021-04-10T20:41:58+00:00"
    }
}

2. Attach roles to the group

attach-group-policy

aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole --group-name systemds-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role --group-name systemds-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonElasticMapReduceFullAccess --group-name systemds-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AWSKeyManagementServicePowerUser --group-name systemds-group
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMUserSSHKeys --group-name systemds-group

# Grant cloud shell access too.
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AWSCloudShellFullAccess --group-name systemds-group

# To create EC2 keys
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name systemds-group

3. Add user to the group

aws iam add-user-to-group --user-name systemds-bot --group-name systemds-group

4. Create the login-profile with credentials

$ aws iam create-login-profile --generate-cli-skeleton > login-profile.json

login-profile.json contains

{
    "LoginProfile": {
        "UserName": "",
        "Password": "",
        "PasswordResetRequired": false
    }
}

Create the credentials manually by editing login-profile.json.

Now, create the login profile.

aws iam create-login-profile --cli-input-json file://login-profile.json

With AWS CLI:

1. Create aws account / use your existing aws account

2. Install aws-cli specific to your Operating System.

3. Create a user

   • Create a new user (https://console.aws.amazon.com/iam/home?#/users)

   • Create new group and add the following policies to it:

     • AmazonElasticMapReduceRole

     • AmazonElasticMapReduceforEC2Role

     • AdministratorAccess

     • AmazonElasticMapReduceFullAccess

     • AWSKeyManagementServicePowerUser

     • IAMUserSSHKeys

4. Configure your aws-cli (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration)

User Setup

5. Spin up an EMR cluster with SystemDS

   • Put your SystemDS artifacts (dml-scripts, jars, config-file) in the directory systemds

   • Edit configuration in: systemds_cluster.config

   • Run: ./spinup_systemds_cluster.sh

6. Run a SystemDS script

   • Run: ./run_systemds_script.sh path/to/script.dml With args: ./run_systemds_script.sh path/to/script.dml “1.0, 2.6”

7. Terminate the EMR cluster: ./terminate_systemds_cluster.sh

Further work

• Finetune the memory

  https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/ https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html#spark-defaults

• Test if Scale to 100 nodes

• Make the cluster WebUIs (Ganglia, SparkUI,..) accessible from outside

• Integrate spot up instances