| { |
| "cells": [ |
| { |
| "cell_type": "code", |
| "source": [ |
| "# @title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n", |
| "\n", |
| "# Licensed to the Apache Software Foundation (ASF) under one\n", |
| "# or more contributor license agreements. See the NOTICE file\n", |
| "# distributed with this work for additional information\n", |
| "# regarding copyright ownership. The ASF licenses this file\n", |
| "# to you under the Apache License, Version 2.0 (the\n", |
| "# \"License\"); you may not use this file except in compliance\n", |
| "# with the License. You may obtain a copy of the License at\n", |
| "#\n", |
| "# http://www.apache.org/licenses/LICENSE-2.0\n", |
| "#\n", |
| "# Unless required by applicable law or agreed to in writing,\n", |
| "# software distributed under the License is distributed on an\n", |
| "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n", |
| "# KIND, either express or implied. See the License for the\n", |
| "# specific language governing permissions and limitations\n", |
| "# under the License" |
| ], |
| "metadata": { |
| "cellView": "form", |
| "id": "sARMhsXz8yR1" |
| }, |
| "execution_count": null, |
| "outputs": [] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "# Preprocessing with the Apache Beam DataFrames API\n", |
| "\n", |
| "[Pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) is one of the most common tools used for data exploration and preprocessing. Pandas is popular because of its ease of use. It has intuitive methods to perform common analytical tasks and data preprocessing. \n", |
| "\n", |
| "<table align=\"left\">\n", |
| " <td>\n", |
| " <a target=\"_blank\" href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb\"><img src=\"https://raw.githubusercontent.com/google/or-tools/main/tools/colab_32px.png\" />Run in Google Colab</a>\n", |
| " </td>\n", |
| " <td>\n", |
| " <a target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb\"><img src=\"https://raw.githubusercontent.com/google/or-tools/main/tools/github_32px.png\" />View source on GitHub</a>\n", |
| " </td>\n", |
| "</table>\n" |
| ], |
| "metadata": { |
| "id": "A8xNRyZMW1yK" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "For rapid execution, Pandas loads all of the data into memory on a single machine (one node). This configuration works well when dealing with small-scale datasets. However, many projects involve datasets that are too big to fit in memory. These use cases generally require parallel data processing frameworks, such as Apache Beam.\n", |
| "\n", |
| "Beam DataFrames provide a Pandas-like\n", |
| "API to declare and define Beam processing pipelines. It provides a familiar interface for machine learning practioners to build complex data-processing pipelines by only invoking standard pandas commands.\n", |
| "\n", |
| "To learn more about Apache Beam DataFrames, see the\n", |
| "[Beam DataFrames overview](https://beam.apache.org/documentation/dsls/dataframes/overview) page.\n", |
| "\n", |
| "## Overview\n", |
| "The goal of this example is to explore a dataset preprocessed with the Beam DataFrame API for machine learning model training.\n", |
| "\n", |
| "This example demonstrates the use of the Apache Beam DataFrames API to perform common data exploration as well as the preprocessing steps that are necessary to prepare your dataset for machine learning model training and inference. This example includes the following steps: \n", |
| "\n", |
| "* Removing unwanted columns.\n", |
| "* One-hot encoding categorical columns.\n", |
| "* Normalizing numerical columns.\n", |
| "\n", |
| "In this example, the first section demonstrates how to build and execute a pipeline locally using the interactive runner.\n", |
| "The second section uses a distributed runner to demonstrate how to run the pipeline on the full dataset.\n" |
| ], |
| "metadata": { |
| "id": "iFZC1inKuUCy" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "## Install Apache Beam\n", |
| "\n", |
| "To explore the elements within a `PCollection`, install Apache Beam with the `interactive` component to use the Interactive runner. The DataFrames API methods invoked in this example are available in Apache Beam SDK versions 2.43 and later.\n" |
| ], |
| "metadata": { |
| "id": "A0f2HJ22D4lt" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "pCjwrwNWnuqI" |
| }, |
| "source": [ |
| "Install the latest Apache Beam SDK version." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "-OJC0Xn5Um-C", |
| "beam:comment": "TODO(https://github.com/apache/issues/23961): Just install 2.43.0 once it's released, [`issue 23276`](https://github.com/apache/beam/issues/23276) is currently not implemented for Beam 2.42 (required fix for implementing `str.get_dummies()`" |
| }, |
| "outputs": [], |
| "source": [ |
| "!git clone https://github.com/apache/beam.git\n", |
| "\n", |
| "!cd beam/sdks/python && pip3 install -r build-requirements.txt \n", |
| "\n", |
| "%pip install -e beam/sdks/python/.[interactive,gcp]" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "## Local exploration with the Interactive Beam runner\n", |
| "Use the [Interactive Beam](https://beam.apache.org/releases/pydoc/2.20.0/apache_beam.runners.interactive.interactive_beam.html) runner to explore and develop your pipeline.\n", |
| "This runner allows you to test the code interactively, progressively building out the pipeline before deploying it on a distributed runner. \n", |
| "\n", |
| "\n", |
| "This section uses a subset of the original dataset, because the notebook instance has limited compute resources.\n" |
| ], |
| "metadata": { |
| "id": "3NO6RgB7GkkE" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "5I3G094hoB1P" |
| }, |
| "source": [ |
| "### Load the data\n", |
| "\n", |
| "To read CSV files into DataFrames, Pandas has the\n", |
| "[`pandas.read_csv`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)\n", |
| "function.\n", |
| "This notebook uses the Beam\n", |
| "[`beam.dataframe.io.read_csv`](https://beam.apache.org/releases/pydoc/current/apache_beam.dataframe.io.html#apache_beam.dataframe.io.read_csv)\n", |
| "function, which emulates `pandas.read_csv`. The main difference is that the Beam function returns a deferred Beam DataFrame, whereas the Pandas function returns a standard DataFrame.\n" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "X3_OB9cAULav" |
| }, |
| "outputs": [], |
| "source": [ |
| "import os\n", |
| "\n", |
| "import numpy as np\n", |
| "import pandas as pd \n", |
| "import apache_beam as beam\n", |
| "import apache_beam.runners.interactive.interactive_beam as ib\n", |
| "from apache_beam import dataframe\n", |
| "from apache_beam.runners.interactive.interactive_runner import InteractiveRunner\n", |
| "from apache_beam.runners.dataflow import DataflowRunner\n", |
| "\n", |
| "# Available options: [sample_1000, sample_10000, sample_100000, full], where\n", |
| "# sample contains the entire dataset (around 1000000 samples).\n", |
| "\n", |
| "source_csv_file = 'gs://apache-beam-samples/nasa_jpl_asteroid/sample_10000.csv'\n", |
| "\n", |
| "# Initialize the pipeline.\n", |
| "p = beam.Pipeline(InteractiveRunner())\n", |
| "\n", |
| "beam_df = p | beam.dataframe.io.read_csv(source_csv_file)\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "paf7yf3YpCh8" |
| }, |
| "source": [ |
| "### Preprocess the data\n", |
| "\n", |
| "This example uses the [NASA - Nearest Earth Objects dataset](https://cneos.jpl.nasa.gov/ca/).\n", |
| "This dataset includes information about objects in outer space. Some objects are close enough to Earth to cause harm.\n", |
| "This dataset compiles the list of NASA certified asteroids that are classified as the nearest earth objects to understand which objects pose a risk." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "\n", |
| "Inspect the dataset columns and their types." |
| ], |
| "metadata": { |
| "id": "cvAu5T0ENjuQ" |
| } |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "colab": { |
| "base_uri": "https://localhost:8080/" |
| }, |
| "id": "LwW77ixE-pjR", |
| "outputId": "3dfba30d-165e-46a6-b0b9-f12519db1c27" |
| }, |
| "outputs": [ |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| "spk_id int64\n", |
| "full_name object\n", |
| "near_earth_object object\n", |
| "absolute_magnitude float64\n", |
| "diameter float64\n", |
| "albedo float64\n", |
| "diameter_sigma float64\n", |
| "eccentricity float64\n", |
| "inclination float64\n", |
| "moid_ld float64\n", |
| "object_class object\n", |
| "semi_major_axis_au_unit float64\n", |
| "hazardous_flag object\n", |
| "dtype: object" |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 27 |
| } |
| ], |
| "source": [ |
| "beam_df.dtypes" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "When using Interactive Beam, to bring a Beam DataFrame into local memory as a Pandas DataFrame, use `ib.collect()`." |
| ], |
| "metadata": { |
| "id": "1Wa6fpbyQige" |
| } |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 746 |
| }, |
| "id": "DPxkAmkpq4Xv", |
| "outputId": "3f89126d-f6fb-43fc-d87b-5daf8563e057" |
| }, |
| "outputs": [ |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_79206f341d7de09f6cacdd05be309575\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_79206f341d7de09f6cacdd05be309575\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_79206f341d7de09f6cacdd05be309575\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| " spk_id full_name near_earth_object \\\n", |
| "0 2000001 1 Ceres N \n", |
| "1 2000002 2 Pallas N \n", |
| "2 2000003 3 Juno N \n", |
| "3 2000004 4 Vesta N \n", |
| "4 2000005 5 Astraea N \n", |
| "... ... ... ... \n", |
| "9994 2009995 9995 Alouette (4805 P-L) N \n", |
| "9995 2009996 9996 ANS (9070 P-L) N \n", |
| "9996 2009997 9997 COBE (1217 T-1) N \n", |
| "9997 2009998 9998 ISO (1293 T-1) N \n", |
| "9998 2009999 9999 Wiles (4196 T-2) N \n", |
| "\n", |
| " absolute_magnitude diameter albedo diameter_sigma eccentricity \\\n", |
| "0 3.40 939.400 0.0900 0.200 0.076009 \n", |
| "1 4.20 545.000 0.1010 18.000 0.229972 \n", |
| "2 5.33 246.596 0.2140 10.594 0.256936 \n", |
| "3 3.00 525.400 0.4228 0.200 0.088721 \n", |
| "4 6.90 106.699 0.2740 3.140 0.190913 \n", |
| "... ... ... ... ... ... \n", |
| "9994 15.10 2.564 0.2450 0.550 0.160610 \n", |
| "9995 13.60 8.978 0.1130 0.376 0.235174 \n", |
| "9996 14.30 NaN NaN NaN 0.113059 \n", |
| "9997 15.10 2.235 0.3880 0.373 0.093852 \n", |
| "9998 13.00 7.148 0.2620 0.065 0.071351 \n", |
| "\n", |
| " inclination moid_ld object_class semi_major_axis_au_unit \\\n", |
| "0 10.594067 620.640533 MBA 2.769165 \n", |
| "1 34.832932 480.348639 MBA 2.773841 \n", |
| "2 12.991043 402.514639 MBA 2.668285 \n", |
| "3 7.141771 443.451432 MBA 2.361418 \n", |
| "4 5.367427 426.433027 MBA 2.574037 \n", |
| "... ... ... ... ... \n", |
| "9994 2.311731 388.723233 MBA 2.390249 \n", |
| "9995 7.657713 444.194746 MBA 2.796605 \n", |
| "9996 2.459643 495.460110 MBA 2.545674 \n", |
| "9997 3.912263 373.848377 MBA 2.160961 \n", |
| "9998 3.198839 632.144398 MBA 2.839917 \n", |
| "\n", |
| " hazardous_flag \n", |
| "0 N \n", |
| "1 N \n", |
| "2 N \n", |
| "3 N \n", |
| "4 N \n", |
| "... ... \n", |
| "9994 N \n", |
| "9995 N \n", |
| "9996 N \n", |
| "9997 N \n", |
| "9998 N \n", |
| "\n", |
| "[9999 rows x 13 columns]" |
| ], |
| "text/html": [ |
| "\n", |
| " <div id=\"df-064477d9-d4b6-44a6-a8fd-31f0cad2dcb5\">\n", |
| " <div class=\"colab-df-container\">\n", |
| " <div>\n", |
| "<style scoped>\n", |
| " .dataframe tbody tr th:only-of-type {\n", |
| " vertical-align: middle;\n", |
| " }\n", |
| "\n", |
| " .dataframe tbody tr th {\n", |
| " vertical-align: top;\n", |
| " }\n", |
| "\n", |
| " .dataframe thead th {\n", |
| " text-align: right;\n", |
| " }\n", |
| "</style>\n", |
| "<table border=\"1\" class=\"dataframe\">\n", |
| " <thead>\n", |
| " <tr style=\"text-align: right;\">\n", |
| " <th></th>\n", |
| " <th>spk_id</th>\n", |
| " <th>full_name</th>\n", |
| " <th>near_earth_object</th>\n", |
| " <th>absolute_magnitude</th>\n", |
| " <th>diameter</th>\n", |
| " <th>albedo</th>\n", |
| " <th>diameter_sigma</th>\n", |
| " <th>eccentricity</th>\n", |
| " <th>inclination</th>\n", |
| " <th>moid_ld</th>\n", |
| " <th>object_class</th>\n", |
| " <th>semi_major_axis_au_unit</th>\n", |
| " <th>hazardous_flag</th>\n", |
| " </tr>\n", |
| " </thead>\n", |
| " <tbody>\n", |
| " <tr>\n", |
| " <th>0</th>\n", |
| " <td>2000001</td>\n", |
| " <td>1 Ceres</td>\n", |
| " <td>N</td>\n", |
| " <td>3.40</td>\n", |
| " <td>939.400</td>\n", |
| " <td>0.0900</td>\n", |
| " <td>0.200</td>\n", |
| " <td>0.076009</td>\n", |
| " <td>10.594067</td>\n", |
| " <td>620.640533</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.769165</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>1</th>\n", |
| " <td>2000002</td>\n", |
| " <td>2 Pallas</td>\n", |
| " <td>N</td>\n", |
| " <td>4.20</td>\n", |
| " <td>545.000</td>\n", |
| " <td>0.1010</td>\n", |
| " <td>18.000</td>\n", |
| " <td>0.229972</td>\n", |
| " <td>34.832932</td>\n", |
| " <td>480.348639</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.773841</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>2</th>\n", |
| " <td>2000003</td>\n", |
| " <td>3 Juno</td>\n", |
| " <td>N</td>\n", |
| " <td>5.33</td>\n", |
| " <td>246.596</td>\n", |
| " <td>0.2140</td>\n", |
| " <td>10.594</td>\n", |
| " <td>0.256936</td>\n", |
| " <td>12.991043</td>\n", |
| " <td>402.514639</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.668285</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>3</th>\n", |
| " <td>2000004</td>\n", |
| " <td>4 Vesta</td>\n", |
| " <td>N</td>\n", |
| " <td>3.00</td>\n", |
| " <td>525.400</td>\n", |
| " <td>0.4228</td>\n", |
| " <td>0.200</td>\n", |
| " <td>0.088721</td>\n", |
| " <td>7.141771</td>\n", |
| " <td>443.451432</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.361418</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>4</th>\n", |
| " <td>2000005</td>\n", |
| " <td>5 Astraea</td>\n", |
| " <td>N</td>\n", |
| " <td>6.90</td>\n", |
| " <td>106.699</td>\n", |
| " <td>0.2740</td>\n", |
| " <td>3.140</td>\n", |
| " <td>0.190913</td>\n", |
| " <td>5.367427</td>\n", |
| " <td>426.433027</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.574037</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>...</th>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9994</th>\n", |
| " <td>2009995</td>\n", |
| " <td>9995 Alouette (4805 P-L)</td>\n", |
| " <td>N</td>\n", |
| " <td>15.10</td>\n", |
| " <td>2.564</td>\n", |
| " <td>0.2450</td>\n", |
| " <td>0.550</td>\n", |
| " <td>0.160610</td>\n", |
| " <td>2.311731</td>\n", |
| " <td>388.723233</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.390249</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9995</th>\n", |
| " <td>2009996</td>\n", |
| " <td>9996 ANS (9070 P-L)</td>\n", |
| " <td>N</td>\n", |
| " <td>13.60</td>\n", |
| " <td>8.978</td>\n", |
| " <td>0.1130</td>\n", |
| " <td>0.376</td>\n", |
| " <td>0.235174</td>\n", |
| " <td>7.657713</td>\n", |
| " <td>444.194746</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.796605</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9996</th>\n", |
| " <td>2009997</td>\n", |
| " <td>9997 COBE (1217 T-1)</td>\n", |
| " <td>N</td>\n", |
| " <td>14.30</td>\n", |
| " <td>NaN</td>\n", |
| " <td>NaN</td>\n", |
| " <td>NaN</td>\n", |
| " <td>0.113059</td>\n", |
| " <td>2.459643</td>\n", |
| " <td>495.460110</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.545674</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9997</th>\n", |
| " <td>2009998</td>\n", |
| " <td>9998 ISO (1293 T-1)</td>\n", |
| " <td>N</td>\n", |
| " <td>15.10</td>\n", |
| " <td>2.235</td>\n", |
| " <td>0.3880</td>\n", |
| " <td>0.373</td>\n", |
| " <td>0.093852</td>\n", |
| " <td>3.912263</td>\n", |
| " <td>373.848377</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.160961</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9998</th>\n", |
| " <td>2009999</td>\n", |
| " <td>9999 Wiles (4196 T-2)</td>\n", |
| " <td>N</td>\n", |
| " <td>13.00</td>\n", |
| " <td>7.148</td>\n", |
| " <td>0.2620</td>\n", |
| " <td>0.065</td>\n", |
| " <td>0.071351</td>\n", |
| " <td>3.198839</td>\n", |
| " <td>632.144398</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.839917</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " </tbody>\n", |
| "</table>\n", |
| "<p>9999 rows × 13 columns</p>\n", |
| "</div>\n", |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-064477d9-d4b6-44a6-a8fd-31f0cad2dcb5')\"\n", |
| " title=\"Convert this dataframe to an interactive table.\"\n", |
| " style=\"display:none;\">\n", |
| " \n", |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", |
| " width=\"24px\">\n", |
| " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n", |
| " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n", |
| " </svg>\n", |
| " </button>\n", |
| " \n", |
| " <style>\n", |
| " .colab-df-container {\n", |
| " display:flex;\n", |
| " flex-wrap:wrap;\n", |
| " gap: 12px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert {\n", |
| " background-color: #E8F0FE;\n", |
| " border: none;\n", |
| " border-radius: 50%;\n", |
| " cursor: pointer;\n", |
| " display: none;\n", |
| " fill: #1967D2;\n", |
| " height: 32px;\n", |
| " padding: 0 0 0 0;\n", |
| " width: 32px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert:hover {\n", |
| " background-color: #E2EBFA;\n", |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", |
| " fill: #174EA6;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert {\n", |
| " background-color: #3B4455;\n", |
| " fill: #D2E3FC;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert:hover {\n", |
| " background-color: #434B5C;\n", |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", |
| " fill: #FFFFFF;\n", |
| " }\n", |
| " </style>\n", |
| "\n", |
| " <script>\n", |
| " const buttonEl =\n", |
| " document.querySelector('#df-064477d9-d4b6-44a6-a8fd-31f0cad2dcb5 button.colab-df-convert');\n", |
| " buttonEl.style.display =\n", |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", |
| "\n", |
| " async function convertToInteractive(key) {\n", |
| " const element = document.querySelector('#df-064477d9-d4b6-44a6-a8fd-31f0cad2dcb5');\n", |
| " const dataTable =\n", |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", |
| " [key], {});\n", |
| " if (!dataTable) return;\n", |
| "\n", |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", |
| " + ' to learn more about interactive tables.';\n", |
| " element.innerHTML = '';\n", |
| " dataTable['output_type'] = 'display_data';\n", |
| " await google.colab.output.renderOutput(dataTable, element);\n", |
| " const docLink = document.createElement('div');\n", |
| " docLink.innerHTML = docLinkHtml;\n", |
| " element.appendChild(docLink);\n", |
| " }\n", |
| " </script>\n", |
| " </div>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 28 |
| } |
| ], |
| "source": [ |
| "ib.collect(beam_df)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "The datasets contain the following two types of columns:\n", |
| "\n", |
| "* **Numerical columns:** Use [normalization](https://developers.google.com/machine-learning/data-prep/transform/normalization) to transform these columns so that they can be used to train a machine learning model.\n", |
| "\n", |
| "* **Categorical columns:** Transform those columns with [one-hot encoding](https://developers.google.com/machine-learning/data-prep/transform/transform-categorical) to use them during training. \n" |
| ], |
| "metadata": { |
| "id": "8jV9odKhNyF2" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "Use the standard pandas command `DataFrame.describe()` to generate descriptive statistics for the numerical columns, such as percentile, mean, std, and so on. " |
| ], |
| "metadata": { |
| "id": "MGAErO0lAYws" |
| } |
| }, |
| { |
| "cell_type": "code", |
| "source": [ |
| "with dataframe.allow_non_parallel_operations():\n", |
| " beam_df_description = ib.collect(beam_df.describe())\n", |
| "\n", |
| "beam_df_description" |
| ], |
| "metadata": { |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 378 |
| }, |
| "id": "Befv697VBGM7", |
| "outputId": "bb465020-94e4-4b3c-fda6-6e43da199be1" |
| }, |
| "execution_count": null, |
| "outputs": [ |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_98687cb0060a8077a8abab6e464e4a75\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_98687cb0060a8077a8abab6e464e4a75\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_98687cb0060a8077a8abab6e464e4a75\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| " spk_id absolute_magnitude diameter albedo \\\n", |
| "count 9.999000e+03 9999.000000 8688.000000 8672.000000 \n", |
| "mean 2.005000e+06 12.675380 19.245446 0.197723 \n", |
| "std 2.886607e+03 1.639609 30.190191 0.138819 \n", |
| "min 2.000001e+06 3.000000 0.300000 0.008000 \n", |
| "25% 2.002500e+06 11.900000 5.614000 0.074000 \n", |
| "50% 2.005000e+06 12.900000 9.814000 0.187000 \n", |
| "75% 2.007500e+06 13.700000 19.156750 0.283000 \n", |
| "max 2.009999e+06 20.700000 939.400000 1.000000 \n", |
| "\n", |
| " diameter_sigma eccentricity inclination moid_ld \\\n", |
| "count 8591.000000 9999.000000 9999.000000 9999.000000 \n", |
| "mean 0.454072 0.148716 7.890742 509.805237 \n", |
| "std 1.093676 0.083803 6.336244 205.046582 \n", |
| "min 0.006000 0.001003 0.042716 0.131028 \n", |
| "25% 0.120000 0.093780 3.220137 377.829197 \n", |
| "50% 0.201000 0.140335 6.018836 470.650523 \n", |
| "75% 0.375000 0.187092 10.918176 636.010802 \n", |
| "max 39.297000 0.889831 68.018875 4241.524913 \n", |
| "\n", |
| " semi_major_axis_au_unit \n", |
| "count 9999.000000 \n", |
| "mean 2.689836 \n", |
| "std 0.607190 \n", |
| "min 0.832048 \n", |
| "25% 2.340816 \n", |
| "50% 2.614468 \n", |
| "75% 3.005449 \n", |
| "max 24.667968 " |
| ], |
| "text/html": [ |
| "\n", |
| " <div id=\"df-238f6456-dbfb-4707-a725-c607c847f522\">\n", |
| " <div class=\"colab-df-container\">\n", |
| " <div>\n", |
| "<style scoped>\n", |
| " .dataframe tbody tr th:only-of-type {\n", |
| " vertical-align: middle;\n", |
| " }\n", |
| "\n", |
| " .dataframe tbody tr th {\n", |
| " vertical-align: top;\n", |
| " }\n", |
| "\n", |
| " .dataframe thead th {\n", |
| " text-align: right;\n", |
| " }\n", |
| "</style>\n", |
| "<table border=\"1\" class=\"dataframe\">\n", |
| " <thead>\n", |
| " <tr style=\"text-align: right;\">\n", |
| " <th></th>\n", |
| " <th>spk_id</th>\n", |
| " <th>absolute_magnitude</th>\n", |
| " <th>diameter</th>\n", |
| " <th>albedo</th>\n", |
| " <th>diameter_sigma</th>\n", |
| " <th>eccentricity</th>\n", |
| " <th>inclination</th>\n", |
| " <th>moid_ld</th>\n", |
| " <th>semi_major_axis_au_unit</th>\n", |
| " </tr>\n", |
| " </thead>\n", |
| " <tbody>\n", |
| " <tr>\n", |
| " <th>count</th>\n", |
| " <td>9.999000e+03</td>\n", |
| " <td>9999.000000</td>\n", |
| " <td>8688.000000</td>\n", |
| " <td>8672.000000</td>\n", |
| " <td>8591.000000</td>\n", |
| " <td>9999.000000</td>\n", |
| " <td>9999.000000</td>\n", |
| " <td>9999.000000</td>\n", |
| " <td>9999.000000</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>mean</th>\n", |
| " <td>2.005000e+06</td>\n", |
| " <td>12.675380</td>\n", |
| " <td>19.245446</td>\n", |
| " <td>0.197723</td>\n", |
| " <td>0.454072</td>\n", |
| " <td>0.148716</td>\n", |
| " <td>7.890742</td>\n", |
| " <td>509.805237</td>\n", |
| " <td>2.689836</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>std</th>\n", |
| " <td>2.886607e+03</td>\n", |
| " <td>1.639609</td>\n", |
| " <td>30.190191</td>\n", |
| " <td>0.138819</td>\n", |
| " <td>1.093676</td>\n", |
| " <td>0.083803</td>\n", |
| " <td>6.336244</td>\n", |
| " <td>205.046582</td>\n", |
| " <td>0.607190</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>min</th>\n", |
| " <td>2.000001e+06</td>\n", |
| " <td>3.000000</td>\n", |
| " <td>0.300000</td>\n", |
| " <td>0.008000</td>\n", |
| " <td>0.006000</td>\n", |
| " <td>0.001003</td>\n", |
| " <td>0.042716</td>\n", |
| " <td>0.131028</td>\n", |
| " <td>0.832048</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>25%</th>\n", |
| " <td>2.002500e+06</td>\n", |
| " <td>11.900000</td>\n", |
| " <td>5.614000</td>\n", |
| " <td>0.074000</td>\n", |
| " <td>0.120000</td>\n", |
| " <td>0.093780</td>\n", |
| " <td>3.220137</td>\n", |
| " <td>377.829197</td>\n", |
| " <td>2.340816</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>50%</th>\n", |
| " <td>2.005000e+06</td>\n", |
| " <td>12.900000</td>\n", |
| " <td>9.814000</td>\n", |
| " <td>0.187000</td>\n", |
| " <td>0.201000</td>\n", |
| " <td>0.140335</td>\n", |
| " <td>6.018836</td>\n", |
| " <td>470.650523</td>\n", |
| " <td>2.614468</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>75%</th>\n", |
| " <td>2.007500e+06</td>\n", |
| " <td>13.700000</td>\n", |
| " <td>19.156750</td>\n", |
| " <td>0.283000</td>\n", |
| " <td>0.375000</td>\n", |
| " <td>0.187092</td>\n", |
| " <td>10.918176</td>\n", |
| " <td>636.010802</td>\n", |
| " <td>3.005449</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>max</th>\n", |
| " <td>2.009999e+06</td>\n", |
| " <td>20.700000</td>\n", |
| " <td>939.400000</td>\n", |
| " <td>1.000000</td>\n", |
| " <td>39.297000</td>\n", |
| " <td>0.889831</td>\n", |
| " <td>68.018875</td>\n", |
| " <td>4241.524913</td>\n", |
| " <td>24.667968</td>\n", |
| " </tr>\n", |
| " </tbody>\n", |
| "</table>\n", |
| "</div>\n", |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-238f6456-dbfb-4707-a725-c607c847f522')\"\n", |
| " title=\"Convert this dataframe to an interactive table.\"\n", |
| " style=\"display:none;\">\n", |
| " \n", |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", |
| " width=\"24px\">\n", |
| " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n", |
| " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n", |
| " </svg>\n", |
| " </button>\n", |
| " \n", |
| " <style>\n", |
| " .colab-df-container {\n", |
| " display:flex;\n", |
| " flex-wrap:wrap;\n", |
| " gap: 12px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert {\n", |
| " background-color: #E8F0FE;\n", |
| " border: none;\n", |
| " border-radius: 50%;\n", |
| " cursor: pointer;\n", |
| " display: none;\n", |
| " fill: #1967D2;\n", |
| " height: 32px;\n", |
| " padding: 0 0 0 0;\n", |
| " width: 32px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert:hover {\n", |
| " background-color: #E2EBFA;\n", |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", |
| " fill: #174EA6;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert {\n", |
| " background-color: #3B4455;\n", |
| " fill: #D2E3FC;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert:hover {\n", |
| " background-color: #434B5C;\n", |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", |
| " fill: #FFFFFF;\n", |
| " }\n", |
| " </style>\n", |
| "\n", |
| " <script>\n", |
| " const buttonEl =\n", |
| " document.querySelector('#df-238f6456-dbfb-4707-a725-c607c847f522 button.colab-df-convert');\n", |
| " buttonEl.style.display =\n", |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", |
| "\n", |
| " async function convertToInteractive(key) {\n", |
| " const element = document.querySelector('#df-238f6456-dbfb-4707-a725-c607c847f522');\n", |
| " const dataTable =\n", |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", |
| " [key], {});\n", |
| " if (!dataTable) return;\n", |
| "\n", |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", |
| " + ' to learn more about interactive tables.';\n", |
| " element.innerHTML = '';\n", |
| " dataTable['output_type'] = 'display_data';\n", |
| " await google.colab.output.renderOutput(dataTable, element);\n", |
| " const docLink = document.createElement('div');\n", |
| " docLink.innerHTML = docLinkHtml;\n", |
| " element.appendChild(docLink);\n", |
| " }\n", |
| " </script>\n", |
| " </div>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 21 |
| } |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "D9uJtHLSSAMC" |
| }, |
| "source": [ |
| "Before running any transformations, verify that all of the columns need to be used for model training. Start by looking at the column description provided by the [JPL website](https://ssd.jpl.nasa.gov/sbdb_query.cgi):\n", |
| "\n", |
| "* **spk_id:** Object primary SPK-ID.\n", |
| "* **full_name:** Asteroid name.\n", |
| "* **near_earth_object:** Near-earth object flag.\n", |
| "* **absolute_magnitude:** The apparent magnitude an object would have if it were located at a distance of 10 parsecs.\n", |
| "* **diameter:** Object diameter (from equivalent sphere) km unit.\n", |
| "* **albedo:** A measure of the diffuse reflection of solar radiation out of the total solar radiation, measured on a scale from 0 to 1.\n", |
| "* **diameter_sigma:** 1-sigma uncertainty in object diameter km unit.\n", |
| "* **eccentricity:** A value between 0 and 1 that refers to how flat or round the asteroid is.\n", |
| "* **inclination:** The angle with respect to the x-y ecliptic plane.\n", |
| "* **moid_ld:** Earth Minimum Orbit Intersection Distance au unit.\n", |
| "* **object_class:** The classification of the asteroid. For a more detailed description, see [NASA object classifications](https://pdssbn.astro.umd.edu/data_other/objclass.shtml).\n", |
| "* **Semi-major axis au Unit:** The length of half of the long axis in AU unit.\n", |
| "* **hazardous_flag:** Identifies hazardous asteroids." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "DzYVKbwTp72d" |
| }, |
| "source": [ |
| "The **spk_id** and **full_name** columns are unique for each row. You can remove these columns, because they are not needed for model training." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "piRPwH2aqT06" |
| }, |
| "outputs": [], |
| "source": [ |
| "beam_df = beam_df.drop(['spk_id', 'full_name'], axis='columns', inplace=False)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "fRvNyahSuX_y" |
| }, |
| "source": [ |
| "Review the number of missing values." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 358 |
| }, |
| "id": "A2PLchW8vXvt", |
| "outputId": "14a4ac64-5b54-4ed4-959d-daea65bb6457" |
| }, |
| "outputs": [ |
| { |
| "output_type": "stream", |
| "name": "stderr", |
| "text": [ |
| "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in long_scalars\n", |
| " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n" |
| ] |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_868f8ad001ab00c7013b65472a513917\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_868f8ad001ab00c7013b65472a513917\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_868f8ad001ab00c7013b65472a513917\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| "near_earth_object 0.000000\n", |
| "absolute_magnitude 0.000000\n", |
| "diameter 13.111311\n", |
| "albedo 13.271327\n", |
| "diameter_sigma 14.081408\n", |
| "eccentricity 0.000000\n", |
| "inclination 0.000000\n", |
| "moid_ld 0.000000\n", |
| "object_class 0.000000\n", |
| "semi_major_axis_au_unit 0.000000\n", |
| "hazardous_flag 0.000000\n", |
| "dtype: float64" |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 30 |
| } |
| ], |
| "source": [ |
| "ib.collect(beam_df.isnull().mean() * 100)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "00MRdFGLwQiD" |
| }, |
| "source": [ |
| "Most of the columns do not have missing values. However, the columns **diameter**, **albedo**, and **diameter_sigma** have many missing values. Because these values cannot be measured or derived and aren't needed for training the ML model, remove the columns." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "tHYeCHREwvyB", |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 538 |
| }, |
| "outputId": "3be686d0-f56a-4054-a71a-d3019bf379e8" |
| }, |
| "outputs": [ |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_f88b77f183371d1a45fa87bed4a545f6\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_f88b77f183371d1a45fa87bed4a545f6\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_f88b77f183371d1a45fa87bed4a545f6\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| " near_earth_object absolute_magnitude eccentricity inclination \\\n", |
| "0 N 3.40 0.076009 10.594067 \n", |
| "1 N 4.20 0.229972 34.832932 \n", |
| "2 N 5.33 0.256936 12.991043 \n", |
| "3 N 3.00 0.088721 7.141771 \n", |
| "4 N 6.90 0.190913 5.367427 \n", |
| "... ... ... ... ... \n", |
| "9994 N 15.10 0.160610 2.311731 \n", |
| "9995 N 13.60 0.235174 7.657713 \n", |
| "9996 N 14.30 0.113059 2.459643 \n", |
| "9997 N 15.10 0.093852 3.912263 \n", |
| "9998 N 13.00 0.071351 3.198839 \n", |
| "\n", |
| " moid_ld object_class semi_major_axis_au_unit hazardous_flag \n", |
| "0 620.640533 MBA 2.769165 N \n", |
| "1 480.348639 MBA 2.773841 N \n", |
| "2 402.514639 MBA 2.668285 N \n", |
| "3 443.451432 MBA 2.361418 N \n", |
| "4 426.433027 MBA 2.574037 N \n", |
| "... ... ... ... ... \n", |
| "9994 388.723233 MBA 2.390249 N \n", |
| "9995 444.194746 MBA 2.796605 N \n", |
| "9996 495.460110 MBA 2.545674 N \n", |
| "9997 373.848377 MBA 2.160961 N \n", |
| "9998 632.144398 MBA 2.839917 N \n", |
| "\n", |
| "[9999 rows x 8 columns]" |
| ], |
| "text/html": [ |
| "\n", |
| " <div id=\"df-4e05872f-1261-4bd8-9251-a967afaa1b32\">\n", |
| " <div class=\"colab-df-container\">\n", |
| " <div>\n", |
| "<style scoped>\n", |
| " .dataframe tbody tr th:only-of-type {\n", |
| " vertical-align: middle;\n", |
| " }\n", |
| "\n", |
| " .dataframe tbody tr th {\n", |
| " vertical-align: top;\n", |
| " }\n", |
| "\n", |
| " .dataframe thead th {\n", |
| " text-align: right;\n", |
| " }\n", |
| "</style>\n", |
| "<table border=\"1\" class=\"dataframe\">\n", |
| " <thead>\n", |
| " <tr style=\"text-align: right;\">\n", |
| " <th></th>\n", |
| " <th>near_earth_object</th>\n", |
| " <th>absolute_magnitude</th>\n", |
| " <th>eccentricity</th>\n", |
| " <th>inclination</th>\n", |
| " <th>moid_ld</th>\n", |
| " <th>object_class</th>\n", |
| " <th>semi_major_axis_au_unit</th>\n", |
| " <th>hazardous_flag</th>\n", |
| " </tr>\n", |
| " </thead>\n", |
| " <tbody>\n", |
| " <tr>\n", |
| " <th>0</th>\n", |
| " <td>N</td>\n", |
| " <td>3.40</td>\n", |
| " <td>0.076009</td>\n", |
| " <td>10.594067</td>\n", |
| " <td>620.640533</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.769165</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>1</th>\n", |
| " <td>N</td>\n", |
| " <td>4.20</td>\n", |
| " <td>0.229972</td>\n", |
| " <td>34.832932</td>\n", |
| " <td>480.348639</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.773841</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>2</th>\n", |
| " <td>N</td>\n", |
| " <td>5.33</td>\n", |
| " <td>0.256936</td>\n", |
| " <td>12.991043</td>\n", |
| " <td>402.514639</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.668285</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>3</th>\n", |
| " <td>N</td>\n", |
| " <td>3.00</td>\n", |
| " <td>0.088721</td>\n", |
| " <td>7.141771</td>\n", |
| " <td>443.451432</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.361418</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>4</th>\n", |
| " <td>N</td>\n", |
| " <td>6.90</td>\n", |
| " <td>0.190913</td>\n", |
| " <td>5.367427</td>\n", |
| " <td>426.433027</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.574037</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>...</th>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9994</th>\n", |
| " <td>N</td>\n", |
| " <td>15.10</td>\n", |
| " <td>0.160610</td>\n", |
| " <td>2.311731</td>\n", |
| " <td>388.723233</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.390249</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9995</th>\n", |
| " <td>N</td>\n", |
| " <td>13.60</td>\n", |
| " <td>0.235174</td>\n", |
| " <td>7.657713</td>\n", |
| " <td>444.194746</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.796605</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9996</th>\n", |
| " <td>N</td>\n", |
| " <td>14.30</td>\n", |
| " <td>0.113059</td>\n", |
| " <td>2.459643</td>\n", |
| " <td>495.460110</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.545674</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9997</th>\n", |
| " <td>N</td>\n", |
| " <td>15.10</td>\n", |
| " <td>0.093852</td>\n", |
| " <td>3.912263</td>\n", |
| " <td>373.848377</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.160961</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9998</th>\n", |
| " <td>N</td>\n", |
| " <td>13.00</td>\n", |
| " <td>0.071351</td>\n", |
| " <td>3.198839</td>\n", |
| " <td>632.144398</td>\n", |
| " <td>MBA</td>\n", |
| " <td>2.839917</td>\n", |
| " <td>N</td>\n", |
| " </tr>\n", |
| " </tbody>\n", |
| "</table>\n", |
| "<p>9999 rows × 8 columns</p>\n", |
| "</div>\n", |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4e05872f-1261-4bd8-9251-a967afaa1b32')\"\n", |
| " title=\"Convert this dataframe to an interactive table.\"\n", |
| " style=\"display:none;\">\n", |
| " \n", |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", |
| " width=\"24px\">\n", |
| " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n", |
| " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n", |
| " </svg>\n", |
| " </button>\n", |
| " \n", |
| " <style>\n", |
| " .colab-df-container {\n", |
| " display:flex;\n", |
| " flex-wrap:wrap;\n", |
| " gap: 12px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert {\n", |
| " background-color: #E8F0FE;\n", |
| " border: none;\n", |
| " border-radius: 50%;\n", |
| " cursor: pointer;\n", |
| " display: none;\n", |
| " fill: #1967D2;\n", |
| " height: 32px;\n", |
| " padding: 0 0 0 0;\n", |
| " width: 32px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert:hover {\n", |
| " background-color: #E2EBFA;\n", |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", |
| " fill: #174EA6;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert {\n", |
| " background-color: #3B4455;\n", |
| " fill: #D2E3FC;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert:hover {\n", |
| " background-color: #434B5C;\n", |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", |
| " fill: #FFFFFF;\n", |
| " }\n", |
| " </style>\n", |
| "\n", |
| " <script>\n", |
| " const buttonEl =\n", |
| " document.querySelector('#df-4e05872f-1261-4bd8-9251-a967afaa1b32 button.colab-df-convert');\n", |
| " buttonEl.style.display =\n", |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", |
| "\n", |
| " async function convertToInteractive(key) {\n", |
| " const element = document.querySelector('#df-4e05872f-1261-4bd8-9251-a967afaa1b32');\n", |
| " const dataTable =\n", |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", |
| " [key], {});\n", |
| " if (!dataTable) return;\n", |
| "\n", |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", |
| " + ' to learn more about interactive tables.';\n", |
| " element.innerHTML = '';\n", |
| " dataTable['output_type'] = 'display_data';\n", |
| " await google.colab.output.renderOutput(dataTable, element);\n", |
| " const docLink = document.createElement('div');\n", |
| " docLink.innerHTML = docLinkHtml;\n", |
| " element.appendChild(docLink);\n", |
| " }\n", |
| " </script>\n", |
| " </div>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 31 |
| } |
| ], |
| "source": [ |
| "beam_df = beam_df.drop(['diameter', 'albedo', 'diameter_sigma'], axis='columns', inplace=False)\n", |
| "ib.collect(beam_df)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "v03ABuXJKEmv" |
| }, |
| "source": [ |
| "### Normalize the data" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "a3PojL3WBqgE" |
| }, |
| "source": [ |
| "Normalize the numerical columns so that they can be used to train a model. To standarize the data, you can subtract the mean and divide by the standard deviation. This process is also known as finding the [z-score](https://developers.google.com/machine-learning/data-prep/transform/normalization#z-score).\n", |
| "This step improves the performance and training stability of the model during training and inference.\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "sZ2_gB8wENF1" |
| }, |
| "source": [ |
| "First, retrieve both the numerical columns and the categorical columns." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "vsWY8xW5d_Wn" |
| }, |
| "outputs": [], |
| "source": [ |
| "numerical_cols = beam_df.select_dtypes(include=np.number).columns.tolist()\n", |
| "categorical_cols = list(set(beam_df.columns) - set(numerical_cols))" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 587 |
| }, |
| "id": "PD_DTxPCP4hs", |
| "outputId": "16fede03-f67e-4c26-8714-fd3fc6892109" |
| }, |
| "outputs": [ |
| { |
| "output_type": "stream", |
| "name": "stderr", |
| "text": [ |
| "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in double_scalars\n", |
| " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n" |
| ] |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_55302fa5950ce6ceb9f99ff9a168097a\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_55302fa5950ce6ceb9f99ff9a168097a\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_55302fa5950ce6ceb9f99ff9a168097a\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| " absolute_magnitude eccentricity inclination moid_ld \\\n", |
| "306 -1.570727 -0.062543 -0.278518 0.373194 \n", |
| "310 -1.631718 -1.724526 -0.736389 1.087833 \n", |
| "546 -1.753698 1.028793 1.415303 -0.339489 \n", |
| "635 -1.875678 0.244869 0.005905 0.214107 \n", |
| "701 -3.278451 -1.570523 2.006145 1.542754 \n", |
| "... ... ... ... ... \n", |
| "9697 0.807888 -1.151809 -0.082944 -0.129556 \n", |
| "9813 1.722740 0.844551 -0.583247 -1.006447 \n", |
| "9868 0.807888 -0.207399 -0.784665 -0.462136 \n", |
| "9903 0.868878 0.460086 0.092258 -0.107597 \n", |
| "9956 0.746898 -0.234132 -0.161116 -0.601379 \n", |
| "\n", |
| " semi_major_axis_au_unit \n", |
| "306 0.357201 \n", |
| "310 0.344233 \n", |
| "546 0.139080 \n", |
| "635 0.367559 \n", |
| "701 0.829337 \n", |
| "... ... \n", |
| "9697 -0.533538 \n", |
| "9813 -0.677961 \n", |
| "9868 -0.539794 \n", |
| "9903 0.071794 \n", |
| "9956 -0.664887 \n", |
| "\n", |
| "[9999 rows x 5 columns]" |
| ], |
| "text/html": [ |
| "\n", |
| " <div id=\"df-f727aadc-8e99-4a1b-b999-93e65a3d6d02\">\n", |
| " <div class=\"colab-df-container\">\n", |
| " <div>\n", |
| "<style scoped>\n", |
| " .dataframe tbody tr th:only-of-type {\n", |
| " vertical-align: middle;\n", |
| " }\n", |
| "\n", |
| " .dataframe tbody tr th {\n", |
| " vertical-align: top;\n", |
| " }\n", |
| "\n", |
| " .dataframe thead th {\n", |
| " text-align: right;\n", |
| " }\n", |
| "</style>\n", |
| "<table border=\"1\" class=\"dataframe\">\n", |
| " <thead>\n", |
| " <tr style=\"text-align: right;\">\n", |
| " <th></th>\n", |
| " <th>absolute_magnitude</th>\n", |
| " <th>eccentricity</th>\n", |
| " <th>inclination</th>\n", |
| " <th>moid_ld</th>\n", |
| " <th>semi_major_axis_au_unit</th>\n", |
| " </tr>\n", |
| " </thead>\n", |
| " <tbody>\n", |
| " <tr>\n", |
| " <th>306</th>\n", |
| " <td>-1.570727</td>\n", |
| " <td>-0.062543</td>\n", |
| " <td>-0.278518</td>\n", |
| " <td>0.373194</td>\n", |
| " <td>0.357201</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>310</th>\n", |
| " <td>-1.631718</td>\n", |
| " <td>-1.724526</td>\n", |
| " <td>-0.736389</td>\n", |
| " <td>1.087833</td>\n", |
| " <td>0.344233</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>546</th>\n", |
| " <td>-1.753698</td>\n", |
| " <td>1.028793</td>\n", |
| " <td>1.415303</td>\n", |
| " <td>-0.339489</td>\n", |
| " <td>0.139080</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>635</th>\n", |
| " <td>-1.875678</td>\n", |
| " <td>0.244869</td>\n", |
| " <td>0.005905</td>\n", |
| " <td>0.214107</td>\n", |
| " <td>0.367559</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>701</th>\n", |
| " <td>-3.278451</td>\n", |
| " <td>-1.570523</td>\n", |
| " <td>2.006145</td>\n", |
| " <td>1.542754</td>\n", |
| " <td>0.829337</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>...</th>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9697</th>\n", |
| " <td>0.807888</td>\n", |
| " <td>-1.151809</td>\n", |
| " <td>-0.082944</td>\n", |
| " <td>-0.129556</td>\n", |
| " <td>-0.533538</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9813</th>\n", |
| " <td>1.722740</td>\n", |
| " <td>0.844551</td>\n", |
| " <td>-0.583247</td>\n", |
| " <td>-1.006447</td>\n", |
| " <td>-0.677961</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9868</th>\n", |
| " <td>0.807888</td>\n", |
| " <td>-0.207399</td>\n", |
| " <td>-0.784665</td>\n", |
| " <td>-0.462136</td>\n", |
| " <td>-0.539794</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9903</th>\n", |
| " <td>0.868878</td>\n", |
| " <td>0.460086</td>\n", |
| " <td>0.092258</td>\n", |
| " <td>-0.107597</td>\n", |
| " <td>0.071794</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9956</th>\n", |
| " <td>0.746898</td>\n", |
| " <td>-0.234132</td>\n", |
| " <td>-0.161116</td>\n", |
| " <td>-0.601379</td>\n", |
| " <td>-0.664887</td>\n", |
| " </tr>\n", |
| " </tbody>\n", |
| "</table>\n", |
| "<p>9999 rows × 5 columns</p>\n", |
| "</div>\n", |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f727aadc-8e99-4a1b-b999-93e65a3d6d02')\"\n", |
| " title=\"Convert this dataframe to an interactive table.\"\n", |
| " style=\"display:none;\">\n", |
| " \n", |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", |
| " width=\"24px\">\n", |
| " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n", |
| " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n", |
| " </svg>\n", |
| " </button>\n", |
| " \n", |
| " <style>\n", |
| " .colab-df-container {\n", |
| " display:flex;\n", |
| " flex-wrap:wrap;\n", |
| " gap: 12px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert {\n", |
| " background-color: #E8F0FE;\n", |
| " border: none;\n", |
| " border-radius: 50%;\n", |
| " cursor: pointer;\n", |
| " display: none;\n", |
| " fill: #1967D2;\n", |
| " height: 32px;\n", |
| " padding: 0 0 0 0;\n", |
| " width: 32px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert:hover {\n", |
| " background-color: #E2EBFA;\n", |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", |
| " fill: #174EA6;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert {\n", |
| " background-color: #3B4455;\n", |
| " fill: #D2E3FC;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert:hover {\n", |
| " background-color: #434B5C;\n", |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", |
| " fill: #FFFFFF;\n", |
| " }\n", |
| " </style>\n", |
| "\n", |
| " <script>\n", |
| " const buttonEl =\n", |
| " document.querySelector('#df-f727aadc-8e99-4a1b-b999-93e65a3d6d02 button.colab-df-convert');\n", |
| " buttonEl.style.display =\n", |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", |
| "\n", |
| " async function convertToInteractive(key) {\n", |
| " const element = document.querySelector('#df-f727aadc-8e99-4a1b-b999-93e65a3d6d02');\n", |
| " const dataTable =\n", |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", |
| " [key], {});\n", |
| " if (!dataTable) return;\n", |
| "\n", |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", |
| " + ' to learn more about interactive tables.';\n", |
| " element.innerHTML = '';\n", |
| " dataTable['output_type'] = 'display_data';\n", |
| " await google.colab.output.renderOutput(dataTable, element);\n", |
| " const docLink = document.createElement('div');\n", |
| " docLink.innerHTML = docLinkHtml;\n", |
| " element.appendChild(docLink);\n", |
| " }\n", |
| " </script>\n", |
| " </div>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 33 |
| } |
| ], |
| "source": [ |
| "# Get the numerical columns.\n", |
| "beam_df_numericals = beam_df.filter(items=numerical_cols)\n", |
| "\n", |
| "# Standarize DataFrames with only the numerical columns.\n", |
| "beam_df_numericals = (beam_df_numericals - beam_df_numericals.mean())/beam_df_numericals.std()\n", |
| "\n", |
| "ib.collect(beam_df_numericals)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "qdNILsajFvex" |
| }, |
| "source": [ |
| "Next, convert the categorical columns into one-hot encoded variables to use during training.\n" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "Ngoxg0rSywVd" |
| }, |
| "outputs": [], |
| "source": [ |
| "def get_one_hot_encoding(df: pd.DataFrame, categorical_col:list) -> pd.DataFrame:\n", |
| " beam_df_categorical= beam_df[categorical_col]\n", |
| " # Get unique values.\n", |
| " with dataframe.allow_non_parallel_operations():\n", |
| " unique_classes = pd.CategoricalDtype(ib.collect(beam_df_categorical.unique(as_series=True)))\n", |
| " # Use `str.get_dummies()` to get the one-hot encoded representation of the categorical columns.\n", |
| " beam_df_categorical = beam_df_categorical.astype(unique_classes).str.get_dummies()\n", |
| " # Add a column name prefix to the newly created categorical columns.\n", |
| " beam_df_categorical = beam_df_categorical.add_prefix(f'{categorical_col}_')\n", |
| "\n", |
| " return beam_df_categorical" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "source": [ |
| "for categorical_col in categorical_cols:\n", |
| " beam_df_categorical = get_one_hot_encoding(df=beam_df, categorical_col=categorical_col)\n", |
| " beam_df_numericals = beam_df_numericals.merge(beam_df_categorical, left_index = True, right_index = True)\n", |
| "ib.collect(beam_df_numericals)" |
| ], |
| "metadata": { |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 602 |
| }, |
| "id": "k9rvtWqHf6Qw", |
| "outputId": "b8d8ae57-6dba-45b4-e7ae-e4b14084eede" |
| }, |
| "execution_count": null, |
| "outputs": [ |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_6b2563c7f661bc0fc5729c2577d6f232\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_6b2563c7f661bc0fc5729c2577d6f232\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_6b2563c7f661bc0fc5729c2577d6f232\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_6fa896083b128ad99059af69a3d7fc7e\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_6fa896083b128ad99059af69a3d7fc7e\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_6fa896083b128ad99059af69a3d7fc7e\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_6339347de9805da541eba53abaee2d5e\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_6339347de9805da541eba53abaee2d5e\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_6339347de9805da541eba53abaee2d5e\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_1af5b908898a1e5949dcc20549f650eb\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_1af5b908898a1e5949dcc20549f650eb\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_1af5b908898a1e5949dcc20549f650eb\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| " absolute_magnitude eccentricity inclination moid_ld \\\n", |
| "0 -5.657067 -0.867596 0.426645 0.540537 \n", |
| "12 -3.583402 -0.756931 1.364340 0.238610 \n", |
| "47 -3.400432 -0.912290 -0.211925 1.136060 \n", |
| "381 -2.363599 0.271412 -0.078826 0.535299 \n", |
| "515 -2.729540 1.469775 0.799915 -0.602881 \n", |
| "... ... ... ... ... \n", |
| "9146 0.563927 -0.508757 -0.327512 -0.637391 \n", |
| "9657 1.478779 0.487849 -0.637779 -0.648240 \n", |
| "9704 0.380957 -0.238383 0.443053 0.670490 \n", |
| "9879 1.295809 -0.442966 -0.698505 -0.494818 \n", |
| "9980 0.746898 -1.455992 -0.849144 0.592902 \n", |
| "\n", |
| " semi_major_axis_au_unit near_earth_object_N near_earth_object_Y \\\n", |
| "0 0.130649 1 0 \n", |
| "12 -0.187375 1 0 \n", |
| "47 0.691182 1 0 \n", |
| "381 0.712755 1 0 \n", |
| "515 -0.014654 1 0 \n", |
| "... ... ... ... \n", |
| "9146 -0.820638 1 0 \n", |
| "9657 -0.468778 1 0 \n", |
| "9704 0.587128 1 0 \n", |
| "9879 -0.662602 1 0 \n", |
| "9980 -0.022726 1 0 \n", |
| "\n", |
| " near_earth_object_nan object_class_AMO object_class_APO ... \\\n", |
| "0 0 0 0 ... \n", |
| "12 0 0 0 ... \n", |
| "47 0 0 0 ... \n", |
| "381 0 0 0 ... \n", |
| "515 0 0 0 ... \n", |
| "... ... ... ... ... \n", |
| "9146 0 0 0 ... \n", |
| "9657 0 0 0 ... \n", |
| "9704 0 0 0 ... \n", |
| "9879 0 0 0 ... \n", |
| "9980 0 0 0 ... \n", |
| "\n", |
| " object_class_CEN object_class_IMB object_class_MBA object_class_MCA \\\n", |
| "0 0 0 1 0 \n", |
| "12 0 0 1 0 \n", |
| "47 0 0 1 0 \n", |
| "381 0 0 1 0 \n", |
| "515 0 0 1 0 \n", |
| "... ... ... ... ... \n", |
| "9146 0 0 1 0 \n", |
| "9657 0 0 1 0 \n", |
| "9704 0 0 1 0 \n", |
| "9879 0 0 1 0 \n", |
| "9980 0 0 1 0 \n", |
| "\n", |
| " object_class_OMB object_class_TJN object_class_nan hazardous_flag_N \\\n", |
| "0 0 0 0 1 \n", |
| "12 0 0 0 1 \n", |
| "47 0 0 0 1 \n", |
| "381 0 0 0 1 \n", |
| "515 0 0 0 1 \n", |
| "... ... ... ... ... \n", |
| "9146 0 0 0 1 \n", |
| "9657 0 0 0 1 \n", |
| "9704 0 0 0 1 \n", |
| "9879 0 0 0 1 \n", |
| "9980 0 0 0 1 \n", |
| "\n", |
| " hazardous_flag_Y hazardous_flag_nan \n", |
| "0 0 0 \n", |
| "12 0 0 \n", |
| "47 0 0 \n", |
| "381 0 0 \n", |
| "515 0 0 \n", |
| "... ... ... \n", |
| "9146 0 0 \n", |
| "9657 0 0 \n", |
| "9704 0 0 \n", |
| "9879 0 0 \n", |
| "9980 0 0 \n", |
| "\n", |
| "[9999 rows x 22 columns]" |
| ], |
| "text/html": [ |
| "\n", |
| " <div id=\"df-14de458f-3e4b-4c07-92f0-da47fc6c0334\">\n", |
| " <div class=\"colab-df-container\">\n", |
| " <div>\n", |
| "<style scoped>\n", |
| " .dataframe tbody tr th:only-of-type {\n", |
| " vertical-align: middle;\n", |
| " }\n", |
| "\n", |
| " .dataframe tbody tr th {\n", |
| " vertical-align: top;\n", |
| " }\n", |
| "\n", |
| " .dataframe thead th {\n", |
| " text-align: right;\n", |
| " }\n", |
| "</style>\n", |
| "<table border=\"1\" class=\"dataframe\">\n", |
| " <thead>\n", |
| " <tr style=\"text-align: right;\">\n", |
| " <th></th>\n", |
| " <th>absolute_magnitude</th>\n", |
| " <th>eccentricity</th>\n", |
| " <th>inclination</th>\n", |
| " <th>moid_ld</th>\n", |
| " <th>semi_major_axis_au_unit</th>\n", |
| " <th>near_earth_object_N</th>\n", |
| " <th>near_earth_object_Y</th>\n", |
| " <th>near_earth_object_nan</th>\n", |
| " <th>object_class_AMO</th>\n", |
| " <th>object_class_APO</th>\n", |
| " <th>...</th>\n", |
| " <th>object_class_CEN</th>\n", |
| " <th>object_class_IMB</th>\n", |
| " <th>object_class_MBA</th>\n", |
| " <th>object_class_MCA</th>\n", |
| " <th>object_class_OMB</th>\n", |
| " <th>object_class_TJN</th>\n", |
| " <th>object_class_nan</th>\n", |
| " <th>hazardous_flag_N</th>\n", |
| " <th>hazardous_flag_Y</th>\n", |
| " <th>hazardous_flag_nan</th>\n", |
| " </tr>\n", |
| " </thead>\n", |
| " <tbody>\n", |
| " <tr>\n", |
| " <th>0</th>\n", |
| " <td>-5.657067</td>\n", |
| " <td>-0.867596</td>\n", |
| " <td>0.426645</td>\n", |
| " <td>0.540537</td>\n", |
| " <td>0.130649</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>12</th>\n", |
| " <td>-3.583402</td>\n", |
| " <td>-0.756931</td>\n", |
| " <td>1.364340</td>\n", |
| " <td>0.238610</td>\n", |
| " <td>-0.187375</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>47</th>\n", |
| " <td>-3.400432</td>\n", |
| " <td>-0.912290</td>\n", |
| " <td>-0.211925</td>\n", |
| " <td>1.136060</td>\n", |
| " <td>0.691182</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>381</th>\n", |
| " <td>-2.363599</td>\n", |
| " <td>0.271412</td>\n", |
| " <td>-0.078826</td>\n", |
| " <td>0.535299</td>\n", |
| " <td>0.712755</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>515</th>\n", |
| " <td>-2.729540</td>\n", |
| " <td>1.469775</td>\n", |
| " <td>0.799915</td>\n", |
| " <td>-0.602881</td>\n", |
| " <td>-0.014654</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>...</th>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9146</th>\n", |
| " <td>0.563927</td>\n", |
| " <td>-0.508757</td>\n", |
| " <td>-0.327512</td>\n", |
| " <td>-0.637391</td>\n", |
| " <td>-0.820638</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9657</th>\n", |
| " <td>1.478779</td>\n", |
| " <td>0.487849</td>\n", |
| " <td>-0.637779</td>\n", |
| " <td>-0.648240</td>\n", |
| " <td>-0.468778</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9704</th>\n", |
| " <td>0.380957</td>\n", |
| " <td>-0.238383</td>\n", |
| " <td>0.443053</td>\n", |
| " <td>0.670490</td>\n", |
| " <td>0.587128</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9879</th>\n", |
| " <td>1.295809</td>\n", |
| " <td>-0.442966</td>\n", |
| " <td>-0.698505</td>\n", |
| " <td>-0.494818</td>\n", |
| " <td>-0.662602</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9980</th>\n", |
| " <td>0.746898</td>\n", |
| " <td>-1.455992</td>\n", |
| " <td>-0.849144</td>\n", |
| " <td>0.592902</td>\n", |
| " <td>-0.022726</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " </tbody>\n", |
| "</table>\n", |
| "<p>9999 rows × 22 columns</p>\n", |
| "</div>\n", |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-14de458f-3e4b-4c07-92f0-da47fc6c0334')\"\n", |
| " title=\"Convert this dataframe to an interactive table.\"\n", |
| " style=\"display:none;\">\n", |
| " \n", |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", |
| " width=\"24px\">\n", |
| " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n", |
| " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n", |
| " </svg>\n", |
| " </button>\n", |
| " \n", |
| " <style>\n", |
| " .colab-df-container {\n", |
| " display:flex;\n", |
| " flex-wrap:wrap;\n", |
| " gap: 12px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert {\n", |
| " background-color: #E8F0FE;\n", |
| " border: none;\n", |
| " border-radius: 50%;\n", |
| " cursor: pointer;\n", |
| " display: none;\n", |
| " fill: #1967D2;\n", |
| " height: 32px;\n", |
| " padding: 0 0 0 0;\n", |
| " width: 32px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert:hover {\n", |
| " background-color: #E2EBFA;\n", |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", |
| " fill: #174EA6;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert {\n", |
| " background-color: #3B4455;\n", |
| " fill: #D2E3FC;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert:hover {\n", |
| " background-color: #434B5C;\n", |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", |
| " fill: #FFFFFF;\n", |
| " }\n", |
| " </style>\n", |
| "\n", |
| " <script>\n", |
| " const buttonEl =\n", |
| " document.querySelector('#df-14de458f-3e4b-4c07-92f0-da47fc6c0334 button.colab-df-convert');\n", |
| " buttonEl.style.display =\n", |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", |
| "\n", |
| " async function convertToInteractive(key) {\n", |
| " const element = document.querySelector('#df-14de458f-3e4b-4c07-92f0-da47fc6c0334');\n", |
| " const dataTable =\n", |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", |
| " [key], {});\n", |
| " if (!dataTable) return;\n", |
| "\n", |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", |
| " + ' to learn more about interactive tables.';\n", |
| " element.innerHTML = '';\n", |
| " dataTable['output_type'] = 'display_data';\n", |
| " await google.colab.output.renderOutput(dataTable, element);\n", |
| " const docLink = document.createElement('div');\n", |
| " docLink.innerHTML = docLinkHtml;\n", |
| " element.appendChild(docLink);\n", |
| " }\n", |
| " </script>\n", |
| " </div>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 35 |
| } |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "rVdSIyCB0spw" |
| }, |
| "source": [ |
| "### Run the pipeline\n", |
| "\n", |
| "This section combines the previous steps into a full pipeline implementation, and then visualizes the preprocessed data.\n", |
| "\n", |
| "Note that the only standard Apache Beam method invoked here is the `pipeline` instance. The rest of the preprocessing commands are based on native pandas methods that are integrated with the Apache Beam DataFrame API." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": null, |
| "metadata": { |
| "id": "ndaSNond0v8Q", |
| "colab": { |
| "base_uri": "https://localhost:8080/", |
| "height": 651 |
| }, |
| "outputId": "b265e915-e649-44e4-a31a-95ac85c0ebf6" |
| }, |
| "outputs": [ |
| { |
| "output_type": "stream", |
| "name": "stderr", |
| "text": [ |
| "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in double_scalars\n", |
| " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n" |
| ] |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_cb06c945824aa1bb68aa31ad7e601b74\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_cb06c945824aa1bb68aa31ad7e601b74\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_cb06c945824aa1bb68aa31ad7e601b74\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_fb923f80fecb72b4fa55e5cfdba16d23\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_fb923f80fecb72b4fa55e5cfdba16d23\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_fb923f80fecb72b4fa55e5cfdba16d23\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_3f4b1a0f483cd017e004e11816a91d3b\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_3f4b1a0f483cd017e004e11816a91d3b\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_3f4b1a0f483cd017e004e11816a91d3b\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "text/plain": [ |
| "<IPython.core.display.HTML object>" |
| ], |
| "text/html": [ |
| "\n", |
| " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n", |
| " <div id=\"progress_indicator_fce8902eccbfaa17e32ba0c7c242ccec\">\n", |
| " <div class=\"spinner-border text-info\" role=\"status\"></div>\n", |
| " <span class=\"text-info\">Processing... collect</span>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "display_data", |
| "data": { |
| "application/javascript": [ |
| "\n", |
| " if (typeof window.interactive_beam_jquery == 'undefined') {\n", |
| " var jqueryScript = document.createElement('script');\n", |
| " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", |
| " jqueryScript.type = 'text/javascript';\n", |
| " jqueryScript.onload = function() {\n", |
| " var datatableScript = document.createElement('script');\n", |
| " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", |
| " datatableScript.type = 'text/javascript';\n", |
| " datatableScript.onload = function() {\n", |
| " window.interactive_beam_jquery = jQuery.noConflict(true);\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_fce8902eccbfaa17e32ba0c7c242ccec\").remove();\n", |
| " });\n", |
| " }\n", |
| " document.head.appendChild(datatableScript);\n", |
| " };\n", |
| " document.head.appendChild(jqueryScript);\n", |
| " } else {\n", |
| " window.interactive_beam_jquery(document).ready(function($){\n", |
| " \n", |
| " $(\"#progress_indicator_fce8902eccbfaa17e32ba0c7c242ccec\").remove();\n", |
| " });\n", |
| " }" |
| ] |
| }, |
| "metadata": {} |
| }, |
| { |
| "output_type": "execute_result", |
| "data": { |
| "text/plain": [ |
| " absolute_magnitude eccentricity inclination moid_ld \\\n", |
| "0 -5.657067 -0.867596 0.426645 0.540537 \n", |
| "12 -3.583402 -0.756931 1.364340 0.238610 \n", |
| "47 -3.400432 -0.912290 -0.211925 1.136060 \n", |
| "381 -2.363599 0.271412 -0.078826 0.535299 \n", |
| "515 -2.729540 1.469775 0.799915 -0.602881 \n", |
| "... ... ... ... ... \n", |
| "9146 0.563927 -0.508757 -0.327512 -0.637391 \n", |
| "9657 1.478779 0.487849 -0.637779 -0.648240 \n", |
| "9704 0.380957 -0.238383 0.443053 0.670490 \n", |
| "9879 1.295809 -0.442966 -0.698505 -0.494818 \n", |
| "9980 0.746898 -1.455992 -0.849144 0.592902 \n", |
| "\n", |
| " semi_major_axis_au_unit near_earth_object_N near_earth_object_Y \\\n", |
| "0 0.130649 1 0 \n", |
| "12 -0.187375 1 0 \n", |
| "47 0.691182 1 0 \n", |
| "381 0.712755 1 0 \n", |
| "515 -0.014654 1 0 \n", |
| "... ... ... ... \n", |
| "9146 -0.820638 1 0 \n", |
| "9657 -0.468778 1 0 \n", |
| "9704 0.587128 1 0 \n", |
| "9879 -0.662602 1 0 \n", |
| "9980 -0.022726 1 0 \n", |
| "\n", |
| " near_earth_object_nan object_class_AMO object_class_APO ... \\\n", |
| "0 0 0 0 ... \n", |
| "12 0 0 0 ... \n", |
| "47 0 0 0 ... \n", |
| "381 0 0 0 ... \n", |
| "515 0 0 0 ... \n", |
| "... ... ... ... ... \n", |
| "9146 0 0 0 ... \n", |
| "9657 0 0 0 ... \n", |
| "9704 0 0 0 ... \n", |
| "9879 0 0 0 ... \n", |
| "9980 0 0 0 ... \n", |
| "\n", |
| " object_class_CEN object_class_IMB object_class_MBA object_class_MCA \\\n", |
| "0 0 0 1 0 \n", |
| "12 0 0 1 0 \n", |
| "47 0 0 1 0 \n", |
| "381 0 0 1 0 \n", |
| "515 0 0 1 0 \n", |
| "... ... ... ... ... \n", |
| "9146 0 0 1 0 \n", |
| "9657 0 0 1 0 \n", |
| "9704 0 0 1 0 \n", |
| "9879 0 0 1 0 \n", |
| "9980 0 0 1 0 \n", |
| "\n", |
| " object_class_OMB object_class_TJN object_class_nan hazardous_flag_N \\\n", |
| "0 0 0 0 1 \n", |
| "12 0 0 0 1 \n", |
| "47 0 0 0 1 \n", |
| "381 0 0 0 1 \n", |
| "515 0 0 0 1 \n", |
| "... ... ... ... ... \n", |
| "9146 0 0 0 1 \n", |
| "9657 0 0 0 1 \n", |
| "9704 0 0 0 1 \n", |
| "9879 0 0 0 1 \n", |
| "9980 0 0 0 1 \n", |
| "\n", |
| " hazardous_flag_Y hazardous_flag_nan \n", |
| "0 0 0 \n", |
| "12 0 0 \n", |
| "47 0 0 \n", |
| "381 0 0 \n", |
| "515 0 0 \n", |
| "... ... ... \n", |
| "9146 0 0 \n", |
| "9657 0 0 \n", |
| "9704 0 0 \n", |
| "9879 0 0 \n", |
| "9980 0 0 \n", |
| "\n", |
| "[9999 rows x 22 columns]" |
| ], |
| "text/html": [ |
| "\n", |
| " <div id=\"df-43e93125-8ade-4324-bff6-6c06d8f90ea1\">\n", |
| " <div class=\"colab-df-container\">\n", |
| " <div>\n", |
| "<style scoped>\n", |
| " .dataframe tbody tr th:only-of-type {\n", |
| " vertical-align: middle;\n", |
| " }\n", |
| "\n", |
| " .dataframe tbody tr th {\n", |
| " vertical-align: top;\n", |
| " }\n", |
| "\n", |
| " .dataframe thead th {\n", |
| " text-align: right;\n", |
| " }\n", |
| "</style>\n", |
| "<table border=\"1\" class=\"dataframe\">\n", |
| " <thead>\n", |
| " <tr style=\"text-align: right;\">\n", |
| " <th></th>\n", |
| " <th>absolute_magnitude</th>\n", |
| " <th>eccentricity</th>\n", |
| " <th>inclination</th>\n", |
| " <th>moid_ld</th>\n", |
| " <th>semi_major_axis_au_unit</th>\n", |
| " <th>near_earth_object_N</th>\n", |
| " <th>near_earth_object_Y</th>\n", |
| " <th>near_earth_object_nan</th>\n", |
| " <th>object_class_AMO</th>\n", |
| " <th>object_class_APO</th>\n", |
| " <th>...</th>\n", |
| " <th>object_class_CEN</th>\n", |
| " <th>object_class_IMB</th>\n", |
| " <th>object_class_MBA</th>\n", |
| " <th>object_class_MCA</th>\n", |
| " <th>object_class_OMB</th>\n", |
| " <th>object_class_TJN</th>\n", |
| " <th>object_class_nan</th>\n", |
| " <th>hazardous_flag_N</th>\n", |
| " <th>hazardous_flag_Y</th>\n", |
| " <th>hazardous_flag_nan</th>\n", |
| " </tr>\n", |
| " </thead>\n", |
| " <tbody>\n", |
| " <tr>\n", |
| " <th>0</th>\n", |
| " <td>-5.657067</td>\n", |
| " <td>-0.867596</td>\n", |
| " <td>0.426645</td>\n", |
| " <td>0.540537</td>\n", |
| " <td>0.130649</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>12</th>\n", |
| " <td>-3.583402</td>\n", |
| " <td>-0.756931</td>\n", |
| " <td>1.364340</td>\n", |
| " <td>0.238610</td>\n", |
| " <td>-0.187375</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>47</th>\n", |
| " <td>-3.400432</td>\n", |
| " <td>-0.912290</td>\n", |
| " <td>-0.211925</td>\n", |
| " <td>1.136060</td>\n", |
| " <td>0.691182</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>381</th>\n", |
| " <td>-2.363599</td>\n", |
| " <td>0.271412</td>\n", |
| " <td>-0.078826</td>\n", |
| " <td>0.535299</td>\n", |
| " <td>0.712755</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>515</th>\n", |
| " <td>-2.729540</td>\n", |
| " <td>1.469775</td>\n", |
| " <td>0.799915</td>\n", |
| " <td>-0.602881</td>\n", |
| " <td>-0.014654</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>...</th>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " <td>...</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9146</th>\n", |
| " <td>0.563927</td>\n", |
| " <td>-0.508757</td>\n", |
| " <td>-0.327512</td>\n", |
| " <td>-0.637391</td>\n", |
| " <td>-0.820638</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9657</th>\n", |
| " <td>1.478779</td>\n", |
| " <td>0.487849</td>\n", |
| " <td>-0.637779</td>\n", |
| " <td>-0.648240</td>\n", |
| " <td>-0.468778</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9704</th>\n", |
| " <td>0.380957</td>\n", |
| " <td>-0.238383</td>\n", |
| " <td>0.443053</td>\n", |
| " <td>0.670490</td>\n", |
| " <td>0.587128</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9879</th>\n", |
| " <td>1.295809</td>\n", |
| " <td>-0.442966</td>\n", |
| " <td>-0.698505</td>\n", |
| " <td>-0.494818</td>\n", |
| " <td>-0.662602</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " <tr>\n", |
| " <th>9980</th>\n", |
| " <td>0.746898</td>\n", |
| " <td>-1.455992</td>\n", |
| " <td>-0.849144</td>\n", |
| " <td>0.592902</td>\n", |
| " <td>-0.022726</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>...</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " <td>1</td>\n", |
| " <td>0</td>\n", |
| " <td>0</td>\n", |
| " </tr>\n", |
| " </tbody>\n", |
| "</table>\n", |
| "<p>9999 rows × 22 columns</p>\n", |
| "</div>\n", |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-43e93125-8ade-4324-bff6-6c06d8f90ea1')\"\n", |
| " title=\"Convert this dataframe to an interactive table.\"\n", |
| " style=\"display:none;\">\n", |
| " \n", |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", |
| " width=\"24px\">\n", |
| " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n", |
| " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n", |
| " </svg>\n", |
| " </button>\n", |
| " \n", |
| " <style>\n", |
| " .colab-df-container {\n", |
| " display:flex;\n", |
| " flex-wrap:wrap;\n", |
| " gap: 12px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert {\n", |
| " background-color: #E8F0FE;\n", |
| " border: none;\n", |
| " border-radius: 50%;\n", |
| " cursor: pointer;\n", |
| " display: none;\n", |
| " fill: #1967D2;\n", |
| " height: 32px;\n", |
| " padding: 0 0 0 0;\n", |
| " width: 32px;\n", |
| " }\n", |
| "\n", |
| " .colab-df-convert:hover {\n", |
| " background-color: #E2EBFA;\n", |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", |
| " fill: #174EA6;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert {\n", |
| " background-color: #3B4455;\n", |
| " fill: #D2E3FC;\n", |
| " }\n", |
| "\n", |
| " [theme=dark] .colab-df-convert:hover {\n", |
| " background-color: #434B5C;\n", |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", |
| " fill: #FFFFFF;\n", |
| " }\n", |
| " </style>\n", |
| "\n", |
| " <script>\n", |
| " const buttonEl =\n", |
| " document.querySelector('#df-43e93125-8ade-4324-bff6-6c06d8f90ea1 button.colab-df-convert');\n", |
| " buttonEl.style.display =\n", |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", |
| "\n", |
| " async function convertToInteractive(key) {\n", |
| " const element = document.querySelector('#df-43e93125-8ade-4324-bff6-6c06d8f90ea1');\n", |
| " const dataTable =\n", |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", |
| " [key], {});\n", |
| " if (!dataTable) return;\n", |
| "\n", |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", |
| " + ' to learn more about interactive tables.';\n", |
| " element.innerHTML = '';\n", |
| " dataTable['output_type'] = 'display_data';\n", |
| " await google.colab.output.renderOutput(dataTable, element);\n", |
| " const docLink = document.createElement('div');\n", |
| " docLink.innerHTML = docLinkHtml;\n", |
| " element.appendChild(docLink);\n", |
| " }\n", |
| " </script>\n", |
| " </div>\n", |
| " </div>\n", |
| " " |
| ] |
| }, |
| "metadata": {}, |
| "execution_count": 36 |
| } |
| ], |
| "source": [ |
| "# Specify the location of the source CSV file.\n", |
| "source_csv_file = 'gs://apache-beam-samples/nasa_jpl_asteroid/sample_10000.csv'\n", |
| "\n", |
| "# Initialize the pipeline.\n", |
| "p = beam.Pipeline(InteractiveRunner())\n", |
| "\n", |
| "# Create a deferred Apache Beam DataFrame with the contents of the CSV file.\n", |
| "beam_df = p | beam.dataframe.io.read_csv(source_csv_file)\n", |
| "\n", |
| "# Drop irrelevant columns and columns with missing values.\n", |
| "beam_df = beam_df.drop(['spk_id', 'full_name','diameter', 'albedo', 'diameter_sigma'], axis='columns', inplace=False)\n", |
| "\n", |
| "# Get numerical columns and columns with categorical values.\n", |
| "numerical_cols = beam_df.select_dtypes(include=np.number).columns.tolist()\n", |
| "categorical_cols = list(set(beam_df.columns) - set(numerical_cols))\n", |
| "\n", |
| "# Normalize the numerical values.\n", |
| "beam_df_numericals = beam_df.filter(items=numerical_cols)\n", |
| "beam_df_numericals = (beam_df_numericals - beam_df_numericals.mean())/beam_df_numericals.std()\n", |
| "\n", |
| "\n", |
| "# One-hot encode the categorical values.\n", |
| "for categorical_col in categorical_cols:\n", |
| " beam_df_categorical= get_one_hot_encoding(df=beam_df, categorical_col=categorical_col)\n", |
| " beam_df_numericals = beam_df_numericals.merge(beam_df_categorical, left_index = True, right_index = True)\n", |
| "\n", |
| "ib.collect(beam_df_numericals)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "id": "xZvJTqa3XKI_" |
| }, |
| "source": [ |
| "## Process the full dataset with the distributed runner\n", |
| "The previous section demonstrates how to build and execute the pipeline locally using the interactive runner.\n", |
| "This section demonstrates how to run the pipeline on the full dataset by switching to a distributed runner. For this example, the pipeline runs on [Dataflow](https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline)." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "source": [ |
| "PROJECT_ID = \"<my-gcp-project>\"\n", |
| "REGION = \"us-central1\"\n", |
| "TEMP_DIR = \"gs://<my-bucket>/tmp\"\n", |
| "OUTPUT_DIR = \"gs://<my-bucket>/dataframe-result\"" |
| ], |
| "metadata": { |
| "id": "dDBYbMEWbL4t" |
| }, |
| "execution_count": null, |
| "outputs": [] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "These steps process the full dataset, `full.csv`, which contains approximately one million rows. To materialize the deferred DataFrame, these steps also write the results to a CSV file instead of using `ib.collect()`.\n", |
| "\n", |
| "To switch from an interactive runner to a distributed runner, update the pipeline options. The rest of the pipeline steps don't change." |
| ], |
| "metadata": { |
| "id": "Qk1GaYoSc9-1" |
| } |
| }, |
| { |
| "cell_type": "code", |
| "source": [ |
| "# Specify the location of the source CSV file (the full dataset).\n", |
| "source_csv_file = 'gs://apache-beam-samples/nasa_jpl_asteroid/full.csv'\n", |
| "\n", |
| "# Build a new pipeline that runs on Dataflow.\n", |
| "p = beam.Pipeline(DataflowRunner(),\n", |
| " options=beam.options.pipeline_options.PipelineOptions(\n", |
| " project=PROJECT_ID,\n", |
| " region=REGION,\n", |
| " temp_location=TEMP_DIR,\n", |
| " # To speed up the demo, disable autoscaling.\n", |
| " autoscaling_algorithm='NONE',\n", |
| " num_workers=10))\n", |
| "\n", |
| "# Create a deferred Apache Beam DataFrame with the contents of the CSV file.\n", |
| "beam_df = p | beam.dataframe.io.read_csv(source_csv_file)\n", |
| "\n", |
| "# Drop irrelevant columns and columns with missing values.\n", |
| "beam_df = beam_df.drop(['spk_id', 'full_name','diameter', 'albedo', 'diameter_sigma'], axis='columns', inplace=False)\n", |
| "\n", |
| "# Get numerical columns and columns with categorical values.\n", |
| "numerical_cols = beam_df.select_dtypes(include=np.number).columns.tolist()\n", |
| "categorical_cols = list(set(beam_df.columns) - set(numerical_cols))\n", |
| "\n", |
| "# Normalize the numerical values. \n", |
| "beam_df_numericals = beam_df.filter(items=numerical_cols)\n", |
| "beam_df_numericals = (beam_df_numericals - beam_df_numericals.mean())/beam_df_numericals.std()\n", |
| "\n", |
| "\n", |
| "# One-hot encode the categorical values. \n", |
| "for categorical_col in categorical_cols:\n", |
| " beam_df_categorical= get_one_hot_encoding(df=beam_df, categorical_col=categorical_col)\n", |
| " beam_df_numericals = beam_df_numericals.merge(beam_df_categorical, left_index = True, right_index = True\n", |
| "\n", |
| "# Write the preprocessed dataset to a CSV file.\n", |
| "beam_df_numericals.to_csv(os.path.join(OUTPUT_DIR, \"preprocessed_data.csv\"))" |
| ], |
| "metadata": { |
| "id": "1XovR0gKbMlK" |
| }, |
| "execution_count": null, |
| "outputs": [] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "Submit and run the pipeline." |
| ], |
| "metadata": { |
| "id": "a789u4Yecs_g" |
| } |
| }, |
| { |
| "cell_type": "code", |
| "source": [ |
| "p.run().wait_until_finish()" |
| ], |
| "metadata": { |
| "id": "pbUlC102bPaZ" |
| }, |
| "execution_count": null, |
| "outputs": [] |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "Wait while the pipeline job runs." |
| ], |
| "metadata": { |
| "id": "dzdqmzKzTOng" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "## What's next \n", |
| "\n", |
| "This tutorial demonstrated how to analyze and preprocess a large-scale dataset with the Apache Beam DataFrames API. You can now train a model on a classification task using the preprocessed dataset.\n", |
| "\n", |
| "To learn more about how to get started with classifying structured data, see \n", |
| "[Structured data classification from scratch](https://keras.io/examples/structured_data/structured_data_classification_from_scratch/).\n", |
| "\n", |
| "To continue learning, find another dataset to use with the Apache Beam DataFrames API processing. Think carefully about which features to include in your model and how to represent them.\n" |
| ], |
| "metadata": { |
| "id": "UOLr6YgOOSVQ" |
| } |
| }, |
| { |
| "cell_type": "markdown", |
| "source": [ |
| "## Resources\n", |
| "\n", |
| "* [Beam DataFrames overview](https://beam.apache.org/documentation/dsls/dataframes/overview) - An overview of the Apache Beam DataFrames API.\n", |
| "* [Differences from pandas](https://beam.apache.org/documentation/dsls/dataframes/differences-from-pandas) - Reviews the differences between Apache Beam DataFrames and Pandas DataFrames, as well as some of the workarounds for unsupported operations.\n", |
| "* [10 minutes to Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html) - A quickstart guide to the Pandas DataFrames.\n", |
| "* [Pandas DataFrame API](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html) - The API reference for the Pandas DataFrames.\n", |
| "* [Data preparation and feature training in ML](https://developers.google.com/machine-learning/data-prep) - A guideline about data transformation for ML training." |
| ], |
| "metadata": { |
| "id": "nG9WXXVcMCe_" |
| } |
| } |
| ], |
| "metadata": { |
| "colab": { |
| "collapsed_sections": [], |
| "provenance": [] |
| }, |
| "kernelspec": { |
| "display_name": "Python 3", |
| "name": "python3" |
| }, |
| "language_info": { |
| "name": "python" |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 0 |
| } |
