| <?xml version="1.0" encoding="UTF-8"?> |
| <project version="4"> |
| <component name="StudySettings"> |
| <StudyTaskManager> |
| <option name="VERSION" value="14" /> |
| <option name="myUserTests"> |
| <map /> |
| </option> |
| <option name="course"> |
| <EduCourse> |
| <option name="authors"> |
| <list> |
| <StepikUserInfo> |
| <option name="firstName" value="Henry" /> |
| <option name="id" value="48485817" /> |
| <option name="lastName" value="Suryawirawan" /> |
| </StepikUserInfo> |
| </list> |
| </option> |
| <option name="compatible" value="true" /> |
| <option name="courseMode" value="Course Creator" /> |
| <option name="createDate" value="1557823043901" /> |
| <option name="customPresentableName" /> |
| <option name="description" value="This course provides a series of katas to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" /> |
| <option name="environment" value="" /> |
| <option name="fromZip" value="false" /> |
| <option name="id" value="54530" /> |
| <option name="index" value="-1" /> |
| <option name="instructors"> |
| <list> |
| <option value="48485817" /> |
| </list> |
| </option> |
| <option name="language" value="JAVA 8" /> |
| <option name="languageCode" value="en" /> |
| <option name="name" value="Beam Katas - Java" /> |
| <option name="public" value="true" /> |
| <option name="sectionIds"> |
| <list /> |
| </option> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="type" value="pycharm11 JAVA 8" /> |
| <option name="updateDate" value="1560936271000" /> |
| <option name="items"> |
| <list> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="85639" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Introduction" /> |
| <option name="position" value="1" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1559325015000" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229506" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Hello Beam" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325015000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p> Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow. </p> <p> Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. </p> <p> To learn more about Apache Beam, refer to <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>. </p> <p> <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam". </p> <br> <div class="hint"> Hardcoded input can be created using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Create.html"> Create</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory"> "Creating a PCollection from in-memory data"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713723" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Hello Beam" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/intro/hello/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1552" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="pipeline.apply(Create.of("Hello Beam"))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/intro/hello/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/intro/hello/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/intro/hello/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936162000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="85640" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Core Transforms" /> |
| <option name="position" value="2" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1559325050000" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229507" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Map" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325026000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p> ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection. </p> <p> <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713724" /> |
| <option name="index" value="1" /> |
| <option name="name" value="ParDo" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1752" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, OutputReceiver<Integer> out) { out.output(number * 10); } }))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936166000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p> <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by whitespace (" "). </p> <br> <div class="hint"> You can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html"> OutputReceiver</a> multiple times in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a>. </div> <div class="hint"> If you're using Beam version before v2.5.0, you can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.WindowedContext.html#output-OutputT-"> DoFn.ProcessContext.output(..)</a> multiple times in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>. </div> </html>" /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713725" /> |
| <option name="index" value="2" /> |
| <option name="name" value="ParDo OneToMany" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1777" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(ParDo.of(new DoFn<String, String>() { @ProcessElement public void processElement(@Element String sentence, OutputReceiver<String> out) { String[] words = sentence.split(" "); for (String word : words) { out.output(word); } } }))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936169000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> MapElements can be used to simplify DoFn that maps an element to another element (one to one). </p> <p> <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713726" /> |
| <option name="index" value="3" /> |
| <option name="name" value="MapElements" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1776" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply( MapElements.into(TypeDescriptors.integers()) .via(number -> number * 5) )" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936172000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> FlatMapElements can be used to simplify DoFn that maps an element to multiple elements (one to many). </p> <p> <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713727" /> |
| <option name="index" value="4" /> |
| <option name="name" value="FlatMapElements" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1835" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply( FlatMapElements.into(TypeDescriptors.strings()) .via(sentence -> Arrays.asList(sentence.split(" "))) )" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560791586000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229508" /> |
| <option name="index" value="2" /> |
| <option name="name" value="GroupByKey" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325029000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p> GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/KV.html"> KV</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html"> GroupByKey</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey"> "GroupByKey"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713728" /> |
| <option name="index" value="1" /> |
| <option name="name" value="GroupByKey" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2025" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input .apply(MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word))) .apply(GroupByKey.create())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936177000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229509" /> |
| <option name="index" value="3" /> |
| <option name="name" value="CoGroupByKey" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325032000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p> CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the toString() representation of the WordsAlphabet model. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> CoGroupByKey</a>, <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/TupleTag.html"> TupleTag</a>, and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGbkResult.html"> CoGbkResult</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey"> "CoGroupByKey"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713729" /> |
| <option name="index" value="1" /> |
| <option name="name" value="CoGroupByKey" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2418" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="TupleTag<String> fruitsTag = new TupleTag<>(); TupleTag<String> countriesTag = new TupleTag<>(); MapElements<String, KV<String, String>> mapToAlphabetKv = MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word)); PCollection<KV<String, String>> fruitsPColl = fruits.apply("Fruit to KV", mapToAlphabetKv); PCollection<KV<String, String>> countriesPColl = countries .apply("Country to KV", mapToAlphabetKv); return KeyedPCollectionTuple .of(fruitsTag, fruitsPColl) .and(countriesTag, countriesPColl) .apply(CoGroupByKey.create()) .apply(ParDo.of(new DoFn<KV<String, CoGbkResult>, String>() { @ProcessElement public void processElement( @Element KV<String, CoGbkResult> element, OutputReceiver<String> out) { String alphabet = element.getKey(); CoGbkResult coGbkResult = element.getValue(); String fruit = coGbkResult.getOnly(fruitsTag); String country = coGbkResult.getOnly(countriesTag); out.output(new WordsAlphabet(alphabet, fruit, country).toString()); } }));" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936180000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229510" /> |
| <option name="index" value="4" /> |
| <option name="name" value="Combine" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325044000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Simple combine operations, such as sums, can usually be implemented as a simple function. </p> <p> <b>Kata:</b> Implement the summation of numbers using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html"> Combine.globally(SerializableFunction)</a>. </p> <br> <div class="hint"> Implement the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html#apply-InputT-"> SerializableFunction.apply</a> method that performs the summation of the Iterable. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines"> "Simple combinations using simple functions"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713730" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Simple Function" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1923" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="@Override public Integer apply(Iterable<Integer> input) { int sum = 0; for (int item : input) { sum += item; } return sum; }" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936184000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account. </p> <p> <b>Kata:</b> Implement the average of numbers using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html"> Combine.CombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html"> Combine.CombineFn</a> class that counts the average of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines"> "Advanced combinations using CombineFn"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713731" /> |
| <option name="index" value="2" /> |
| <option name="name" value="CombineFn" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1962" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="class Accum implements Serializable { int sum = 0; int count = 0; @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Accum accum = (Accum) o; return sum == accum.sum && count == accum.count; } @Override public int hashCode() { return Objects.hash(sum, count); } } @Override public Accum createAccumulator() { return new Accum(); } @Override public Accum addInput(Accum accumulator, Integer input) { accumulator.sum += input; accumulator.count++; return accumulator; } @Override public Accum mergeAccumulators(Iterable<Accum> accumulators) { Accum merged = createAccumulator(); for (Accum accumulator : accumulators) { merged.sum += accumulator.sum; merged.count += accumulator.count; } return merged; } @Override public Double extractOutput(Accum accumulator) { return ((double) accumulator.sum) / accumulator.count; }" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936188000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations. </p> <p> <b>Kata:</b> Implement the summation of BigInteger using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combine"> "Combine"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713732" /> |
| <option name="index" value="3" /> |
| <option name="name" value="BinaryCombineFn" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2125" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="@Override public BigInteger apply(BigInteger left, BigInteger right) { return left.add(right); }" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936191000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn Lambda</h2> <p> BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations. </p> <p> Since Beam v2.13.0, you can also use lambda or method reference in order to create the BinaryCombineFn. </p> <p> <b>Kata:</b> Implement the summation of BigInteger using lambda or method reference. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableBiFunction.html"> SerializableBiFunction</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combine"> "Combine"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="750324" /> |
| <option name="index" value="4" /> |
| <option name="name" value="BinaryCombineFn Lambda" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1922" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Combine.globally(BigInteger::add))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936195000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p> After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn. </p> <p> <b>Kata:</b> Implement the sum of scores per player using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html"> Combine.perKey</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html"> Combine.perKey(GlobalCombineFn)</a>. </div> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection"> "Combining values in a keyed PCollection"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713733" /> |
| <option name="index" value="5" /> |
| <option name="name" value="Combine PerKey" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2155" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Combine.perKey(new SumIntBinaryCombineFn()))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="1" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2295" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="@Override public Integer apply(Integer left, Integer right) { return left + right; }" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936199000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229511" /> |
| <option name="index" value="5" /> |
| <option name="name" value="Flatten" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325047000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p> Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html"> Flatten</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#flatten"> "Flatten"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713734" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Flatten" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2040" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="PCollectionList.of(words1).and(words2) .apply(Flatten.pCollections())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936202000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229512" /> |
| <option name="index" value="6" /> |
| <option name="name" value="Partition" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325050000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p> Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections. </p> <p> Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html"> Partition</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#partition"> "Partition"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713735" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Partition" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/partition/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1966" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input .apply(Partition.of(2, (PartitionFn<Integer>) (number, numPartitions) -> { if (number > 100) { return 0; } else { return 1; } }))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/partition/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936206000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237989" /> |
| <option name="index" value="7" /> |
| <option name="name" value="Side Input" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1560791406453" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Input</h2> <p> In addition to the main input PCollection, you can provide additional inputs to a ParDo transform in the form of side inputs. A side input is an additional input that your DoFn can access each time it processes an element in the input PCollection. When you specify a side input, you create a view of some other data that can be read from within the ParDo transform’s DoFn while processing each element. </p> <p> Side inputs are useful if your ParDo needs to inject additional data when processing each element in the input PCollection, but the additional data needs to be determined at runtime (and not hard-coded). Such values might be determined by the input data, or depend on a different branch of your pipeline. </p> <p> <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/View.html"> View</a> to create PCollectionView of citiesToCountries. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a> that accepts <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withSideInputs-org.apache.beam.sdk.values.PCollectionView...-"> side input</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="754085" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Side Input" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2716" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="citiesToCountries.apply(View.asMap())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="1" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2914" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="persons.apply(ParDo.of(new DoFn<Person, Person>() { @ProcessElement public void processElement(@Element Person person, OutputReceiver<Person> out, ProcessContext context) { Map<String, String> citiesToCountries = context.sideInput(citiesToCountriesView); String city = person.getCity(); String country = citiesToCountries.get(city); out.output(new Person(person.getName(), city, country)); } }).withSideInputs(citiesToCountriesView))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936210000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237990" /> |
| <option name="index" value="8" /> |
| <option name="name" value="Side Output" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1560791445676" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Output</h2> <p> While ParDo always produces a main output PCollection (as the return value from apply), you can also have your ParDo produce any number of additional output PCollections. If you choose to have multiple outputs, your ParDo returns all of the output PCollections (including the main output) bundled together. </p> <p> <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.MultiOutputReceiver.html"> MultiOutputReceiver</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withOutputTags-org.apache.beam.sdk.values.TupleTag-org.apache.beam.sdk.values.TupleTagList-"> .withOutputTags</a> to output multiple tagged-outputs in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo.</a> </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs"> "Additional outputs"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="754087" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Side Output" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2253" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="numbers.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, MultiOutputReceiver out) { if (number <= 100) { out.get(numBelow100Tag).output(number); } else { out.get(numAbove100Tag).output(number); } } }).withOutputTags(numBelow100Tag, TupleTagList.of(numAbove100Tag)))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936215000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237991" /> |
| <option name="index" value="9" /> |
| <option name="name" value="Branching" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1560791458069" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Branching</h2> <p> You can use the same PCollection as input for multiple transforms without consuming the input or altering it. </p> <p> <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying each number by 5 and the other transform is multiplying each number by 10. </p> <br> <div class="hint"> Refer to the Beam Design Your Pipeline Guide <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection"> "Multiple transforms process the same PCollection"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="754088" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Branching" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/branching/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1994" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply("Multiply by 5", MapElements.into(integers()).via(num -> num * 5))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="1" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2175" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply("Multiply by 10", MapElements.into(integers()).via(num -> num * 10))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/branching/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936219000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237192" /> |
| <option name="index" value="10" /> |
| <option name="name" value="Composite Transform" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1560431460000" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Composite Transform</h2> <p> Transforms can have a nested structure, where a complex transform performs multiple simpler transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms). These transforms are called composite transforms. Nesting multiple transforms inside a single composite transform can make your code more modular and easier to understand. </p> <p> To create your own composite transform, create a subclass of the PTransform class and override the expand method to specify the actual processing logic. You can then use this transform just as you would a built-in transform from the Beam SDK. For the PTransform class type parameters, you pass the PCollection types that your transform takes as input, and produces as output. Within your PTransform subclass, you’ll need to override the expand method. The expand method is where you add the processing logic for the PTransform. Your override of expand must accept the appropriate type of input PCollection as a parameter, and specify the output PCollection as the return value. </p> <p> <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts numbers from comma separated line and then multiplies each number by 10. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/PTransform.html"> PTransform</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms"> "Composite transforms"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="750323" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Composite Transform" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/composite/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1929" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="@Override public PCollection<Integer> expand(PCollection<String> input) { return input .apply(ParDo.of(new DoFn<String, Integer>() { @ProcessElement public void processElement(@Element String numbers, OutputReceiver<Integer> out) { Arrays.stream(numbers.split(",")) .forEach(numStr -> out.output(Integer.parseInt(numStr))); } })) .apply(MapElements.into(integers()).via(number -> number * 10)); }" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/composite/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560791618000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237765" /> |
| <option name="index" value="11" /> |
| <option name="name" value="DoFn Additional Parameters" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="0" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>DoFn Additional Parameters</h2> <p> In addition to the element and the OutputReceiver, Beam will populate other parameters to your DoFn’s @ProcessElement method. Any combination of these parameters can be added to your process method in any order. </p> <div> <ul> <li> <b>Timestamp</b>: To access the timestamp of an input element, add a parameter annotated with @Timestamp of type Instant </li> <li> <b>Window</b>: To access the window an input element falls into, add a parameter of the type of the window used for the input PCollection. </li> <li> <b>PaneInfo</b>: When triggers are used, Beam provides a PaneInfo object that contains information about the current firing. Using PaneInfo you can determine whether this is an early or a late firing, and how many times this window has already fired for this key. </li> <li> <b>PipelineOptions</b>: The PipelineOptions for the current pipeline can always be accessed in a process method by adding it as a parameter. </li> </ul> </div> <p> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#other-dofn-parameters"> "Accessing additional parameters in your DoFn"</a> section for more information. </p> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753154" /> |
| <option name="index" value="1" /> |
| <option name="name" value="DoFn Additional Parameters" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560699463688" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="85641" /> |
| <option name="index" value="3" /> |
| <option name="name" value="Common Transforms" /> |
| <option name="position" value="3" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1559325072000" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229513" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Filter" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325056000" /> |
| <option name="unitId" value="202038" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p> <b>Kata:</b> Implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a> and only output the intended element. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713736" /> |
| <option name="index" value="1" /> |
| <option name="name" value="ParDo" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1752" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(ParDo.of( new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, OutputReceiver<Integer> out) { if (number % 2 == 1) { out.output(number); } } }) )" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936224000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html"> Filter</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html"> Filter.by(...)</a>. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713737" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Filter" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1718" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Filter.by(number -> number % 2 == 0))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936227000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229514" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Aggregation" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1559325072000" /> |
| <option name="unitId" value="202039" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> <b>Kata:</b> Count the number of elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html"> Count</a>. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713738" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Count" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1707" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Count.globally())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936231000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> <b>Kata:</b> Compute the sum of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Sum.html"> Sum</a>. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713739" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Sum" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1709" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Sum.integersGlobally())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936235000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p> <b>Kata:</b> Compute the mean/average of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Mean.html"> Mean</a>. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713740" /> |
| <option name="index" value="3" /> |
| <option name="name" value="Mean" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1709" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Mean.globally())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936238000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Min</h2> <p> <b>Kata:</b> Compute the minimum of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Min.html"> Min</a>. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713741" /> |
| <option name="index" value="4" /> |
| <option name="name" value="Min" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1709" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Min.integersGlobally())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936242000" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Max</h2> <p> <b>Kata:</b> Compute the maximum of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Max.html"> Max</a>. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713742" /> |
| <option name="index" value="5" /> |
| <option name="name" value="Max" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1709" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(Max.integersGlobally())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936246000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237992" /> |
| <option name="index" value="3" /> |
| <option name="name" value="WithKeys" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="updateDate" value="1560791491864" /> |
| <option name="unitId" value="-1" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>WithKeys</h2> <p> <b>Kata:</b> Convert each fruit name into a KV of its first letter and itself, e.g. <code>apple => KV.of("a", "apple")</code> </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithKeys.html"> WithKeys</a>. </div> <div class="hint"> If using a lambda in Java 8, <code>withKeyType(TypeDescriptor)</code> must be called on the result PTransform. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="754089" /> |
| <option name="index" value="1" /> |
| <option name="name" value="WithKeys" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1875" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input .apply(WithKeys.<String, String>of(fruit -> fruit.substring(0, 1)) .withKeyType(strings()))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936249000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="88010" /> |
| <option name="index" value="4" /> |
| <option name="name" value="IO" /> |
| <option name="position" value="4" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1560431425000" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237187" /> |
| <option name="index" value="1" /> |
| <option name="name" value="TextIO" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1560431430000" /> |
| <option name="unitId" value="209563" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>TextIO Read</h2> <p> When you create a pipeline, you often need to read data from some external source, such as a file or a database. Likewise, you may want your pipeline to output its result data to an external storage system. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms. </p> <p> To read a PCollection from one or more text files, use TextIO.read() to instantiate a transform and use TextIO.Read.from(String) to specify the path of the file(s) to be read. </p> <p> <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html"> TextIO</a> and its corresponding <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html#read--"> TextIO.read()</a> method. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data"> "Reading input data"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="750317" /> |
| <option name="index" value="1" /> |
| <option name="name" value="TextIO Read" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="countries.txt"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="countries.txt" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/io/textio/read/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1615" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="TextIO.read().from(FILE_PATH)" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="1" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1855" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input.apply(MapElements.into(strings()).via(String::toUpperCase))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/io/textio/read/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936253000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237188" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Built-in IOs" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1560431436000" /> |
| <option name="unitId" value="209564" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Built-in I/Os</h2> <p> Beam SDKs provide many out of the box I/O transforms that can be used to read from many different sources and write to many different sinks. </p> <p> See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms. </p> </html>" /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="750319" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Built-in IOs" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/io/builtinios/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/io/builtinios/Task.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936257000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="88156" /> |
| <option name="index" value="5" /> |
| <option name="name" value="Windowing" /> |
| <option name="position" value="5" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1560698891352" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237760" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Adding Timestamp" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="0" /> |
| <option name="unitId" value="210092" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Adding Timestamp - ParDo</h2> <p> Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need timestamps, you must add them to your PCollection’s elements. </p> <p> You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that outputs new elements with timestamps that you set. </p> <p> <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html#outputWithTimestamp-T-org.joda.time.Instant-"> OutputReceiver.outputWithTimestamp</a> method to assign timestamp to the element. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements"> "Adding timestamps to a PCollection’s elements"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753142" /> |
| <option name="index" value="1" /> |
| <option name="name" value="ParDo" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2249" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="events.apply(ParDo.of(new DoFn<Event, Event>() { @ProcessElement public void processElement(@Element Event event, OutputReceiver<Event> out) { out.outputWithTimestamp(event, event.getDate().toInstant()); } }))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560698905262" /> |
| </EduTask> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Adding Timestamp - WithTimestamps</h2> <p> Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need timestamps, you must add them to your PCollection’s elements. </p> <p> You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that outputs new elements with timestamps that you set. </p> <p> <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithTimestamps.html"> WithTimestamps</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements"> "Adding timestamps to a PCollection’s elements"</a> section for more information. </div> </html>" /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753143" /> |
| <option name="index" value="2" /> |
| <option name="name" value="WithTimestamps" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2223" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="events.apply(WithTimestamps.of(event -> event.getDate().toInstant()))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560698907450" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237761" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Fixed Time Window" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="0" /> |
| <option name="unitId" value="210093" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Fixed Time Window</h2> <p> Windowing subdivides a PCollection according to the timestamps of its individual elements. Transforms that aggregate multiple elements, such as GroupByKey and Combine, work implicitly on a per-window basis — they process each PCollection as a succession of multiple, finite windows, though the entire collection itself may be of unbounded size. </p> <p> In the Beam model, any PCollection (including unbounded PCollections) can be subdivided into logical windows. Each element in a PCollection is assigned to one or more windows according to the PCollection’s windowing function, and each individual window contains a finite number of elements. Grouping transforms then consider each PCollection’s elements on a per-window basis. GroupByKey, for example, implicitly groups the elements of a PCollection by key and window. </p> <div> Beam provides several windowing functions, including: <ul> <li>Fixed Time Windows</li> <li>Sliding Time Windows</li> <li>Per-Session Windows</li> <li>Single Global Window</li> </ul> </div> <p> The simplest form of windowing is using fixed time windows. A fixed time window represents a consistent duration, non overlapping time interval in the data stream. </p> <p> <b>Kata:</b> Please count the number of events that happened based on fixed window with 1-day duration. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#fixed-time-windows"> "Fixed time windows"</a> section for more information. </div> </html>" /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753144" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Fixed Time Window" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2906" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="events .apply(Window.into(FixedWindows.of(Duration.standardDays(1)))) .apply(Count.perElement())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560698912954" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="88157" /> |
| <option name="index" value="6" /> |
| <option name="name" value="Triggers" /> |
| <option name="position" value="6" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1560923505422" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237762" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Event Time Triggers" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1560923508379" /> |
| <option name="unitId" value="210094" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Event Time Triggers</h2> <p> When collecting and grouping data into windows, Beam uses triggers to determine when to emit the aggregated results of each window (referred to as a pane). If you use Beam’s default windowing configuration and default trigger, Beam outputs the aggregated result when it estimates all data has arrived, and discards all subsequent data for that window. </p> <p> You can set triggers for your PCollections to change this default behavior. Beam provides a number of pre-built triggers that you can set: </p> <div> <ul> <li>Event time triggers</li> <li>Processing time triggers</li> <li>Data-driven triggers</li> <li>Composite triggers</li> </ul> </div> <p> Event time triggers operate on the event time, as indicated by the timestamp on each data element. Beam’s default trigger is event time-based. </p> <p> The AfterWatermark trigger operates on event time. The AfterWatermark trigger emits the contents of a window after the watermark passes the end of the window, based on the timestamps attached to the data elements. The watermark is a global progress metric, and is Beam’s notion of input completeness within your pipeline at any given point. AfterWatermark.pastEndOfWindow() only fires when the watermark passes the end of the window. </p> <p> <b>Kata:</b> Given that events are being generated every second, please implement a trigger that emits the number of events count within a fixed window of 5-second duration. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 5-second duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0 with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--"> discarding accumulation mode</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753145" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Event Time Triggers" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1905" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardSeconds(5))) .triggering(AfterWatermark.pastEndOfWindow()) .withAllowedLateness(Duration.ZERO) .discardingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560923517000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237763" /> |
| <option name="index" value="2" /> |
| <option name="name" value="Early Triggers" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1560923523075" /> |
| <option name="unitId" value="210095" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Early Triggers</h2> <p> Triggers allow Beam to emit early results, before all the data in a given window has arrived. For example, emitting after a certain amount of time elapses, or after a certain number of elements arrives. </p> <p> <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day duration, please implement an early trigger that emits the number of events count immediately after new element is processed. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-"> withEarlyFirings</a> to set early firing triggers. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0 with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--"> discarding accumulation mode</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html>" /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753146" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Early Triggers" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="resources/log4j2.xml"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="false" /> |
| <option name="name" value="resources/log4j2.xml" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1970" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardDays(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings( AfterProcessingTime.pastFirstElementInPane())) .withAllowedLateness(Duration.ZERO) .discardingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560923531000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="237764" /> |
| <option name="index" value="3" /> |
| <option name="name" value="Window Accumulation Mode" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1560923537697" /> |
| <option name="unitId" value="210096" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Window Accumulation Mode</h2> <p> When you specify a trigger, you must also set the the window’s accumulation mode. When a trigger fires, it emits the current contents of the window as a pane. Since a trigger can fire multiple times, the accumulation mode determines whether the system accumulates the window panes as the trigger fires, or discards them. </p> <p> <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day duration, please implement an early trigger that emits the number of events count immediately after new element is processed in accumulating mode. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/windowing/Window.html#accumulatingFiredPanes--"> accumulatingFiredPanes()</a> to set a window to accumulate the panes that are produced when the trigger fires. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-"> withEarlyFirings</a> to set early firing triggers. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="753147" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Window Accumulation Mode" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java" /> |
| <option name="text" value="" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="1972" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardDays(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings( AfterProcessingTime.pastFirstElementInPane())) .withAllowedLateness(Duration.ZERO) .accumulatingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java" /> |
| <option name="text" value="public class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java" /> |
| <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560923544000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| <Section> |
| <option name="courseId" value="54530" /> |
| <option name="customPresentableName" /> |
| <option name="id" value="85642" /> |
| <option name="index" value="7" /> |
| <option name="name" value="Examples" /> |
| <option name="position" value="7" /> |
| <option name="stepikChangeStatus" value="Up to date" /> |
| <option name="updateDate" value="1557824624000" /> |
| <option name="items"> |
| <list> |
| <Lesson> |
| <option name="customPresentableName" /> |
| <option name="id" value="229515" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Word Count" /> |
| <option name="stepikChangeStatus" value="Content changed" /> |
| <option name="updateDate" value="1557824624000" /> |
| <option name="unitId" value="202040" /> |
| <option name="items"> |
| <list> |
| <EduTask> |
| <option name="customPresentableName" /> |
| <option name="descriptionFormat" value="HTML" /> |
| <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p> <b>Kata:</b> Create a pipeline that counts the number of words. </p> <p> Please output the count of each word in the following format: </p> <pre> word:count ball:5 book:3 </pre> <br> <div class="hint"> Refer to your katas above. </div> </html> " /> |
| <option name="feedbackLink"> |
| <FeedbackLink> |
| <option name="link" /> |
| <option name="type" value="STEPIK" /> |
| </FeedbackLink> |
| </option> |
| <option name="id" value="713743" /> |
| <option name="index" value="1" /> |
| <option name="name" value="Word Count" /> |
| <option name="record" value="-1" /> |
| <option name="status" value="Unchecked" /> |
| <option name="stepikChangeStatus" value="Info and Content changed" /> |
| <option name="files"> |
| <map> |
| <entry key="src/org/apache/beam/learning/katas/examples/wordcount/Task.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list> |
| <AnswerPlaceholder> |
| <option name="hints"> |
| <list /> |
| </option> |
| <option name="index" value="0" /> |
| <option name="initialState" /> |
| <option name="initializedFromDependency" value="false" /> |
| <option name="length" value="6" /> |
| <option name="offset" value="2075" /> |
| <option name="placeholderDependency" /> |
| <option name="placeholderText" value="TODO()" /> |
| <option name="possibleAnswer" value="input .apply(FlatMapElements.into(TypeDescriptors.strings()) .via(line -> Arrays.asList(line.split(" ")))) .apply(Count.perElement()) .apply(ParDo.of(new DoFn<KV<String, Long>, String>() { @ProcessElement public void processElement( @Element KV<String, Long> element, OutputReceiver<String> out) { out.output(element.getKey() + ":" + element.getValue()); } }))" /> |
| <option name="selected" value="false" /> |
| <option name="status" value="Unchecked" /> |
| <option name="studentAnswer" /> |
| <option name="useLength" value="false" /> |
| </AnswerPlaceholder> |
| </list> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="src/org/apache/beam/learning/katas/examples/wordcount/Task.java" /> |
| <option name="text" value="class Task { //put your task here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="true" /> |
| </TaskFile> |
| </value> |
| </entry> |
| <entry key="test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java"> |
| <value> |
| <TaskFile> |
| <option name="answerPlaceholders"> |
| <list /> |
| </option> |
| <option name="highlightErrors" value="true" /> |
| <option name="name" value="test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java" /> |
| <option name="text" value="public class Test { // put your test here }" /> |
| <option name="trackChanges" value="true" /> |
| <option name="trackLengths" value="true" /> |
| <option name="visible" value="false" /> |
| </TaskFile> |
| </value> |
| </entry> |
| </map> |
| </option> |
| <option name="updateDate" value="1560936261000" /> |
| </EduTask> |
| </list> |
| </option> |
| </Lesson> |
| </list> |
| </option> |
| </Section> |
| </list> |
| </option> |
| </EduCourse> |
| </option> |
| </StudyTaskManager> |
| </component> |
| </project> |
