blob: 9fd879f9fa38516da20047bfe6ed83126390cd30 [file] [view]
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
### The single global window
By default, all data in a `PCollection` is assigned to the single global window, and late data is discarded. If your data set is of a fixed size, you can use the global window default for your `PCollection`.
You can use the single global window if you are working with an unbounded data set (e.g. from a streaming data source) but use caution when applying aggregating transforms such as `GroupByKey` and `Combine`. The single global window with a default trigger generally requires the entire data set to be available before processing, which is not possible with continuously updating data. To perform aggregations on an unbounded `PCollection` that uses global windowing, you should specify a non-default trigger for that `PCollection`.
If your `PCollection` is limited (the size is fixed), you can assign all the elements to one global window by specifying explicitly.
{{if (eq .Sdk "go")}}
The first argument that we specify is the `scope` and from the library of `window` we choose which one we need, in this case a `NewGlobalWindows` is necessary. And the last argument is a `PCollection` that contains the elements to which window is applied.
```
globalWindowedItems := beam.WindowInto(s,
window.NewGlobalWindows(),
input)
```
{{end}}
{{if (eq .Sdk "java")}}
The first step is to apply the `Window` transformation. And we write what type of `PCollection` has by calling the `into` method. The argument will be an object of the desired **window-class**.
```
PCollection<String> input = ...;
PCollection<String> batchItems = input.apply(
Window.<String>into(new GlobalWindows()));
```
{{end}}
{{if (eq .Sdk "python")}}
The first step is to apply the `WindowInto` conversion, and from the `window` library we choose which one we need, in this case we need `GlobalWindows`.
```
from apache_beam import window
global_windowed_items = (
input | 'window' >> beam.WindowInto(window.GlobalWindows()))
```
{{end}}
If you do not specify any windows, a single global window will be automatically applied.
### Playground exercise
`CombineFn` : This function allows you to perform operations such as counting, summing, or finding the minimum or maximum element within a global window.
`GroupByKey` : This function groups elements by a key, and allows you to apply a beam.CombineFn to each group of elements within a global window.
`Map` : This function allows you to apply a user-defined function to each element within a global window.
`Filter` : This function allows you to filter elements based on a user-defined condition, within a global window.
`FlatMap` : This function allows you to apply a user-defined function to each element within a global window and output zero or more elements.
These functions can be easily composed together to create complex data processing pipelines. Additionally, it's also possible to create your own custom functions to perform specific operations within a global window.
You can apply the functions on the playground example. You can supplement with **filter** and **count**.
{{if (eq .Sdk "go")}}
Import dependency:
```
"strings"
"github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats"
"github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/filter"
```
Modify code:
```
filtered := applyTransform(s, input)
counted := stats.CountElms(s, filtered)
```
Filter function:
```
func applyTransform(s beam.Scope, input beam.PCollection) beam.PCollection {
return filter.Exclude(s, input, func(element string) bool {
return strings.HasPrefix(strings.ToLower(element), "w")
})
}
```
{{end}}
{{if (eq .Sdk "java")}}
Modify code:
```
batchItems.apply(Filter.by(element -> element.toLowerCase().startsWith("w"))).apply(Count.globally());
```
{{end}}
{{if (eq .Sdk "python")}}
Modify code:
```
(p | beam.Create(['Hello Beam','It`s windowing'])
| 'window' >> beam.WindowInto(window.GlobalWindows())
| 'filter' >> beam.Filter(lambda element: element.lower().startswith("h"))
| 'count' >> beam.combiners.Count.Globally()
| 'Log words' >> Output())
```
{{end}}