| //// |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| //// |
| [[HopGuiPipelines]] |
| :imagesdir: ../../assets/images |
| :page-pagination: |
| :description: Getting Started (6/8): Pipelines are one of the two main file types in Hop. Learn how to add transforms and connect them through hops before your run, preview and debug your first pipeline. |
| |
| = Pipelines |
| |
| In xref:getting-started/hop-concepts.adoc[Concepts], we walked through pipelines, transforms and hops. |
| Let’s remember: |
| |
| * A pipeline consists of a chain of transforms that read, process or write data. |
| Transforms are connected by hops. |
| Hops have a direction but can't create loops, which effectively makes Pipelines DAGs (Directed Acyclic Graphs). |
| * A transform is each basic operation in your pipeline. |
| Each transform performs an operation on the data: reads from a source, operates on data in the pipeline or writes to to a target. |
| * A hop in a pipeline passes data between transforms. |
| |
| == Create a Pipeline |
| |
| There are two ways to create a pipeline. |
| |
| * click on the **New** option on the Horizontal toolbar and select the Pipeline option. |
| |
| image:getting-started/gs-hop-gui-create-pipeline-1.png[] |
| |
| * File -> New -> Pipeline  |
| image:getting-started/gs-hop-gui-create-pipeline-2.png[] + |
| image:getting-started/gs-hop-gui-create-pipeline-3.png[] |
| |
| Your new pipeline is created. |
| You’ll see the dialog below. |
| |
| image:getting-started/gs-hop-gui-create-pipeline-4.png[width=65%] |
| |
| == Add and connect Transforms |
| |
| === Add Transforms |
| |
| Now you are ready to add the first transform. |
| Click anywhere in the pipeline canvas, the area where you’ll see the image below. |
| |
| image:getting-started/gs-hop-gui-create-pipeline-5.png[] |
| |
| After clicking you will see the context dialog. |
| This is the dialog you'll use every time you need to select transforms to add to your pipeline. |
| |
| image:getting-started/gs-hop-gui-create-pipeline-6.png[] |
| |
| Use the search box in this dialog to find the transforms you need. |
| Click or use the arrow keys and hit enter to add a transform to your pipeline. |
| |
| For now, add a xref:pipeline/transforms/rowgenerator.adoc[Generate Rows] and a xref:pipeline/transforms/addsequence.adoc[Add Sequence] transform to your pipeline. |
| |
| image:getting-started/gs-hop-gui-create-pipeline-add-two-transforms.png[] |
| |
| TIP: check the xref:pipeline/transforms.adoc[full list of transforms]. |
| There are 186 transforms available in Hop 0.99, but you'll soon get to know the most commonly used ones. |
| |
| === Create a Hop |
| |
| There are a number of ways to create a hop: |
| |
| * **shift-drag**: while holding down the shift key on your keyboard. |
| Click on a transform, while holding down your primary mouse button, drag to the second transform. |
| Release the primary mouse button and the shift key. |
| * **scroll-drag**: scroll-click on a transform , while holding down your mouse’s scroll button, drag to the second transform. |
| Release the scroll button. |
| * click on a transform in your pipeline to open the context dialog (the dialog you opened in the '**click anywhere**' step). |
| Click the 'Create hop' image:getting-started/icons/HOP.svg[Create hop,25px,align="bottom"] button and select the transform you want to create the hop to. |
| |
| image:getting-started/gs-hop-gui-create-pipeline-create-hop.png[] |
| |
| == Run your pipeline |
| |
| Running a pipeline to see how it performs can be done in one of the following tasks: |
| |
| * Using the Run icon. |
| |
| image:getting-started/gs-hop-gui-run-pipeline-1.png[] |
| |
| * Select Run and click on Start Execution from the toolbar. |
| |
| image:getting-started/gs-hop-gui-run-pipeline-2.png[] |
| |
| * Press F8 |
| |
| You will see the Run Options dialog. |
| |
| image:getting-started/gs-hop-gui-run-pipeline-3.png[] + |
| |
| image:getting-started/gs-hop-gui-run-pipeline-4.png[] |
| |
| TIP: a 'local' runtime configuration is created when you first start Hop Gui. |
| Check the available xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[runtime configurations] for other engines to run your pipelines on. |
| |
| Make sure your configuration is selected and hit Launch. |
| |
| You'll see green check marks in the upper right corner of the transforms when a pipeline ran successfully. |
| |
| image:getting-started/gs-hop-gui-run-pipeline-5.png[] |
| |
| After each run, the execution results are displayed in the panel on the bottom of your window. |
| The Execution Results contains two tabs: |
| |
| * transform metrics |
| * logging |
| |
| The Transform Metrics tab displays metrics per transform. |
| |
| image:getting-started/gs-hop-gui-run-pipeline-6.png[] |
| |
| The logging tab displays the log of the pipeline according to the log level chosen on execution. |
| |
| image:getting-started/gs-hop-gui-run-pipeline-7.png[] |
| |
| TIP: for more detailed information, check out the xref:pipeline/run-preview-debug-pipeline.adoc[Run, Preview and Debug a Pipeline] page. |
| |
