Merge branch 'main' of https://github.com/apache/incubator-wayang-website into main
diff --git a/.asf.yaml b/.asf.yaml
index 4df00ed..9dd5a89 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -32,7 +32,7 @@
# Disable wiki for documentation
wiki: false
# Disable issue management
- issues: false
+ issues: true
# Disable projects for project management boards
projects: false
enabled_merge_buttons:
diff --git a/Gemfile.lock b/Gemfile.lock
index 1c0b1f0..3665246 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -10,19 +10,9 @@
em-websocket (0.5.2)
eventmachine (>= 0.12.9)
http_parser.rb (~> 0.6.0)
- ethon (0.12.0)
- ffi (>= 1.3.0)
eventmachine (1.2.7)
ffi (1.14.2)
forwardable-extended (2.6.0)
- html-proofer (3.18.6)
- addressable (~> 2.3)
- mercenary (~> 0.3)
- nokogumbo (~> 2.0)
- parallel (~> 1.3)
- rainbow (~> 3.0)
- typhoeus (~> 1.3)
- yell (~> 2.0)
http_parser.rb (0.6.0)
i18n (1.8.9)
concurrent-ruby (~> 1.0)
@@ -57,18 +47,9 @@
rb-fsevent (~> 0.10, >= 0.10.3)
rb-inotify (~> 0.9, >= 0.9.10)
mercenary (0.4.0)
- nokogiri (1.11.1-arm64-darwin)
- racc (~> 1.4)
- nokogiri (1.11.1-x86_64-darwin)
- racc (~> 1.4)
- nokogumbo (2.0.4)
- nokogiri (~> 1.8, >= 1.8.4)
- parallel (1.20.1)
pathutil (0.16.2)
forwardable-extended (~> 2.6)
public_suffix (4.0.6)
- racc (1.5.2)
- rainbow (3.0.0)
rb-fsevent (0.10.4)
rb-inotify (0.10.1)
ffi (~> 1.0)
@@ -79,19 +60,15 @@
ffi (~> 1.9)
terminal-table (2.0.0)
unicode-display_width (~> 1.1, >= 1.1.1)
- typhoeus (1.4.0)
- ethon (>= 0.9.0)
unicode-display_width (1.7.0)
- yell (2.2.2)
PLATFORMS
universal-darwin-20
DEPENDENCIES
coderay (~> 1.1.0)
- html-proofer
jekyll (~> 4.2)
jekyll-asciidoc (~> 2.1.1)
BUNDLED WITH
- 2.2.9
+ 2.2.27
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..1b1d42c
--- /dev/null
+++ b/README.md
@@ -0,0 +1,2 @@
+# Wayang Website
+Apache Wayang is the API for big data and data processing across multiple data processing frameworks. Wayang enables data engineers, data scientists and data operations to use multiple platforms at the same time. The repo contains the website for [Apache Wayang (incubating)](https://wayang.apache.org)
diff --git a/_data/features.yml b/_data/features.yml
index 3813ce5..052220c 100644
--- a/_data/features.yml
+++ b/_data/features.yml
@@ -15,38 +15,38 @@
# limitations under the License.
#
-- title: Cross-platform
+- title: Cross platform enablement
icon: fas fa-bezier-curve
- description-short: Run a single data analytic task on top of any set of data processing platforms.
+ description-short: Run data analytics tasks on top of any set of data processing platforms.
description: |
- The most salient feature of Apache Wayang is its cross-platform optimizer. Besides deciding the best processing platform to run any incoming task, Apache Wayang can run a single task on multiple processing platforms. Overall, it applies an extensible set of graph transformations to a Apache Wayang plan to find alternative execution plans. Then, it compares all execution plans by using a platform-specific cost model. Cost functions can either be given or learned, and are parameterized with respect to the underlying hardware (e.g., number of computing nodes for distributed operators).
+ The most salient feature of Apache Wayang is its cross platform optimizer. Besides deciding the best processing platform to run any incoming task, Apache Wayang can run a single task on multiple processing platforms. Overall, it applies an extensible set of graph transformations to the Apache Wayang plan to find alternative better execution plans. After it compares all execution plans by using a platform-specific cost model. Cost functions can either be given or learned, and are parameterized with respect to the underlying hardware (e.g., number of computing nodes for distributed operators).
-- title: High-Efficiency
+- title: High Efficiency
icon: fa fa-clock
- description-short: It selects the best available data processing platform for any incoming query.
+ description-short: Apache Wayang selects the best available data processing platform for any incoming query.
description: |
- Apache Wayang provides a number of optimized operators and novel query optimization process that allows it to efficiently deal with big (as well as small) datasets. Furthermore, as its data processing abstraction is based on UDFs, Apache Wayang lets applications expose semantic properties about their functions, optimization hints (e.g., numbers of iterations), constraints (e.g., physical collocation of operators), and alternative plans. The optimizer then uses those artifacts where available in a best-effort approach.
+ Apache Wayang provides a number of optimized operators and novel query optimization processes that allows to efficiently deal with big (as well as small) datasets. Furthermore, as the data processing abstraction is based on UDFs, Apache Wayang lets applications expose semantic properties about their functions, optimization hints (e.g., numbers of iterations), constraints (e.g., physical collocation of operators), and alternative plans. The optimizer then uses those artifacts where available in a best-effort approach.
- title: Flexibility
icon: fa fa-puzzle-piece
description-short: User defined functions (UDFs) as first-class citizens, enabling extensibility and adaptability.
description: |
- Apache Wayang provides a set of Wayang operators, which applications use to specify their tasks, as well as a set of execution operators, which processing platform provide to run application tasks. The key aspect is that Apache Wayang provides a flexible operator mapping structure allowing developers to add, modify, or delete mappings among Wayang and execution operators. As a result, developers can also add or remove Wayang and execution operators.
+ Apache Wayang provides a set of operators, which applications use to specify their tasks. The key aspect of Apache Wayang is the providing of a flexible operator mapping structure; allowing developers to add, modify, or delete mappings among Wayang and execution operators. As a result, developers can also add or remove Wayang and execution operators.
-- title: Ease-of-Use
+- title: Easy to use
icon: fas fa-child
- description-short: A simple interface that allows developers to focus only on the logics of their application.
+ description-short: Apache Wayang provides a simple interface to enable developers to focus on the logics of their applications.
description: |
Apache Wayang exposes a simple Java API to developers whereby they can implement their tasks. Developers focus on the logics of their tasks rather than on low-level details specific to data processing platforms. The figure of the SGD plans above shows the Wayang plan for a scalable gradient descent implementation: we clearly see that this tedious implementation task is now much easier!
- title: Cost Saving
icon: fa fa-piggy-bank
- description-short: Fast development of data analytic applications.
+ description-short: Fast development of data analytics applications, reduces sunken costs and enables innovation
description: |
- Users do not have to know the intricacies of the underlying platforms: they focus on the logic of their application only. This not only speeds up the development of applications, but also it is no longer a must to be an expert in big data infrastructures. Apache Wayang takes care of how and on which data processing platforms to deploy your applications.
+ Users do not have to know the intricacies of the underlying platforms: they focus on the logic of their application only. This not only speeds up the development of applications, but also it is no longer a "must have" to be an expert in big data infrastructures. Apache Wayang takes care of how and on which data processing platforms fits best for the given tasks to deploy applications.
- title: Open Source
icon: fa fa-code-branch
- description-short: All code is on GitHub under Apache License.
+ description-short: All code is on GitHub under Apache Software License 2.0.
description: |
- Apache Wayang has been open source from its very beginnings and will keep being open source until its very endings. Feel free to download it, try it, and contribute to it. Help us to make it better!
+ Apache Wayang has been open source from its very beginnings and will keep being open source until its very endings. Feel free to download, try, and contribute to the project. Help us to make Apache Wayang even better!
diff --git a/_data/navigation.yml b/_data/navigation.yml
index 1af20c6..1f59c20 100644
--- a/_data/navigation.yml
+++ b/_data/navigation.yml
@@ -27,6 +27,10 @@
url: /community
external: false
+- title: Download
+ url: /download
+ external: false
+
- title: Documentation
url: /documentation
external: false
@@ -38,14 +42,20 @@
- title: Apache
sublinks:
- title: Apache Software Foundation
- url: http://www.apache.org/foundation/how-it-works.html
+ url: https://www.apache.org/foundation/how-it-works.html
external: true
- - title: Apache License
- url: http://www.apache.org/licenses/
+ - title: License
+ url: https://www.apache.org/licenses/
+ external: true
+ - title: Events
+ url: https://www.apache.org/events/current-event
+ external: true
+ - title: Security
+ url: https://www.apache.org/security/
external: true
- title: Sponsorship
- url: http://www.apache.org/foundation/sponsorship.html
+ url: https://www.apache.org/foundation/sponsorship.html
external: true
- title: Thanks
- url: http://www.apache.org/foundation/thanks.html
+ url: https://www.apache.org/foundation/thanks.html
external: true
diff --git a/_includes/footer.html b/_includes/footer.html
index 6b80ede..48232fc 100644
--- a/_includes/footer.html
+++ b/_includes/footer.html
@@ -30,12 +30,49 @@
Apache Wayang is an effort undergoing Incubation at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
</p>
<p class="text-center">
- Copyright © 2021 The Apache Software Foundation.<br />
+ Copyright © 2022 The Apache Software Foundation.<br />
Licensed under the Apache License, Version 2.0.<br />
Apache, the Apache Feather logo, and the Apache Incubator project logo are trademarks of The Apache Software Foundation.
</p>
</div>
</div>
+ <div class="row justify-content-md-center w-100">
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://incubator.apache.org">
+ Apache Incubator
+ </a>
+ </div>
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://www.apache.org">
+ About ASF
+ </a>
+ </div>
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://www.apache.org/licenses/">
+ License
+ </a>
+ </div>
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://www.apache.org/events/current-event">
+ Events
+ </a>
+ </div>
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://www.apache.org/security/">
+ Security
+ </a>
+ </div>
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://www.apache.org/foundation/sponsorship.html">
+ Sponsorship
+ </a>
+ </div>
+ <div class="col-md-3">
+ <a class="btn btn-link w-100" href="https://www.apache.org/foundation/thanks.html">
+ Thanks
+ </a>
+ </div>
+ </div>
</div>
</nav>
</footer>
diff --git a/_layouts/home.html b/_layouts/home.html
index 7242616..53e4ce9 100644
--- a/_layouts/home.html
+++ b/_layouts/home.html
@@ -97,6 +97,22 @@
</div>
</div>
</section>
+<section class="container-fluid p-0 mt-n3 mb-n3" style="background-color:#ffffff !important ">
+ <div class="container">
+ <div class="d-flex justify-content-center">
+ <p class="display-4 p-2">
+ {{ page.section-4.title }}
+ </p>
+ </div>
+ <div class="d-flex justify-content-center">
+ <p class="lead p-2">
+ <strong>
+ {{ page.section-4.text }}
+ </strong>
+ </p>
+ </div>
+ </div>
+</section>
<script src="https://cdn.jsdelivr.net/npm/particles.js@2.0.0/particles.min.js"></script>
<script>
diff --git a/_layouts/publication.html b/_layouts/publication.html
index 2d3c122..4c6e849 100644
--- a/_layouts/publication.html
+++ b/_layouts/publication.html
@@ -33,6 +33,13 @@
<i class="far fa-file-pdf"></i> Download
</a>
{% else %}
+ {% if page.link-paper contains "youtube" %}
+ <iframe width="560" height="315"
+ src="{{page.link-paper}}"
+ allowfullscreen>
+ </iframe>
+ </br>
+ {% endif %}
<a href="{{ page.link-paper}}" class="btn btn-outline-info">
<i class="fas fa-external-link-alt"></i> Go External Link
</a>
diff --git a/_publications/2021-08-13-boss.md b/_publications/2021-08-13-boss.md
new file mode 100644
index 0000000..becbc79
--- /dev/null
+++ b/_publications/2021-08-13-boss.md
@@ -0,0 +1,31 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+layout: publication
+title: >
+ Workshop BOSS@VLDB 2021
+subtitle: >
+ Apache Wayang: A Big Data Cross-Platform System
+link-name: Workshop BOSS'21 at VLDB'21
+img-thumb: assets/img/screenshot/bossworkshop.jpg
+authors: Zoi Kaoudi, Bertty Contreras-Rojas and Rodrigo Pardo-Meza
+year: 2021
+month: 08
+day: 16
+link-paper: https://www.youtube.com/watch?v=jgD_1tPn9sc
+link-external: true
+---
+The following video shows a hands-on tutorial of Apache Wayang presented in the BOSS workshop 2021. The workshop, which is held in conjunction with the top-tier conference in data management (VLDB), gives a deep-dive introduction to active open-source systems and this year was focused on the interoperability among big data systems. Our tutorial, after giving a brief overview of Wayang’s goal and architecture, focuses on live demonstration of various Wayang features. First, we give a WordCount example where the basic API is explained. Then, via K-means and SGD examples we show how a user can use iterations and how she can benefit by letting Wayang selecting the platforms to execute these tasks. Finally, we show how one can easily switch from batch processing to streaming via the WordCount task.
diff --git a/assets/css/home.css b/assets/css/home.css
index 5a714d0..7f38f1e 100644
--- a/assets/css/home.css
+++ b/assets/css/home.css
@@ -188,7 +188,20 @@
color: #a00e5a;
}
+.community h3 {
+ color: #a00e5a;
+ border-bottom: 1px solid #ff8724;
+}
+.community blockquote {
+ border-left: 0.3em solid #a00e5a;
+ border-top: 1px solid #a00e5a;
+ border-bottom: 1px solid #a00e5a;
+ border-right: 1px solid #a00e5a;
+ padding-top: 0.5em;
+ padding-left: 0.5em;
+ color: #a00e5a;
+}
+
.community li::marker {
color: #ff8724 !important;
-
}
diff --git a/assets/img/screenshot/bossworkshop.jpg b/assets/img/screenshot/bossworkshop.jpg
new file mode 100644
index 0000000..376fe4c
--- /dev/null
+++ b/assets/img/screenshot/bossworkshop.jpg
Binary files differ
diff --git a/documentation.md b/documentation.md
index d0b7c40..46d5f17 100644
--- a/documentation.md
+++ b/documentation.md
@@ -188,43 +188,41 @@
public static void main(String[] args){
// Settings
- String inputUrl = "file:/tmp.txt";
+ String inputUrl = "file:/tmp.txt";
// Get a plan builder.
WayangContext wayangContext = new WayangContext(new Configuration())
.withPlugin(Java.basicPlugin())
.withPlugin(Spark.basicPlugin());
JavaPlanBuilder planBuilder = new JavaPlanBuilder(wayangContext)
- .withJobName(String.format("WordCount (%s)", inputUrl))
+ .withJobName(String.format("WordCount (%s)", inputUrl))
.withUdfJarOf(WordcountJava.class);
// Start building the Apache WayangPlan.
- Collection<Tuple2<String, Integer>> wordcounts = planBuilder
+ Collection<Tuple2<String, Integer>> wordcounts = planBuilder
// Read the text file.
- .readTextFile(inputUrl).withName("Load file")
+ .readTextFile(inputUrl).withName("Load file")
// Split each line by non-word characters.
- .flatMap(line -> Arrays.asList(line.split("\\W+")))
+ .flatMap(line -> Arrays.asList(line.split("\\W+")))
.withSelectivity(10, 100, 0.9)
- .withName("Split words")
+ .withName("Split words")
// Filter empty tokens.
- .filter(token -> !token.isEmpty())
+ .filter(token -> !token.isEmpty())
.withSelectivity(0.99, 0.99, 0.99)
- .withName("Filter empty words")
+ .withName("Filter empty words")
// Attach counter to each word.
- .map(word -> new Tuple2<>(word.toLowerCase(), 1)).withName("To lower case, add counter")
+ .map(word -> new Tuple2<>(word.toLowerCase(), 1)).withName("To lower case, add counter")
// Sum up counters for every word.
.reduceByKey(
Tuple2::getField0,
- (t1, t2) -> new Tuple2<>(t1.getField0(), t1.getField1() + t2.getField1())
+ (t1, t2) -> new Tuple2<>(t1.getField0(), t1.getField1() + t2.getField1())
)
- .withCardinalityEstimator(new DefaultCardinalityEstimator(0.9, 1, false, in -> Math.round(0.01 </li>
-<li>
-in[0])))
- .withName("Add counters")
+ .withCardinalityEstimator(new DefaultCardinalityEstimator(0.9, 1, false, in -> Math.round(0.01 * in[0])))
+ .withName("Add counters")
// Execute the plan and collect the results.
.collect();
diff --git a/download.md b/download.md
new file mode 100644
index 0000000..45531a0
--- /dev/null
+++ b/download.md
@@ -0,0 +1,63 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+layout: community
+title: "Download"
+---
+Be sure to verify your downloads by [these procedures](https://www.apache.org/info/verification) using these [KEYS](https://downloads.apache.org/incubator/wayang/KEYS) for any Apache release.
+
+# Current Releases (Incubating)
+
+## 0.3.1 (incubating) [[Source Release]](https://www.apache.org/dyn/closer.lua/incubator/wayang/0.6.0/apache-wayang-0.6.0-incubating-source-release.zip) [[SHA512]](https://downloads.apache.org/incubator/wayang/0.6.0/apache-wayang-0.6.0-incubating-source-release.zip.sha512) [[ASC]](https://downloads.apache.org/incubator/wayang/0.6.0/apache-wayang-0.6.0-incubating-source-release.zip.asc)
+This is the first release of Apache Wayang (incubating).
+
+### Frameworks supported
+This release supports the following frameworks:
+
+- Apache Flink v1.7.1
+- Apache Giraph v1.2.0-hadoop2
+- GraphChi v0.2.2 (only available with scala 11.x)
+- Java Streams (version depends on the java version)
+- JDBC-Template
+- Postgres v9.4.1208 (Implementation JDBC-Template)
+- Apache Spark v3.1.2 (scala 12.x) and v2.4.8 (scala 11.x)
+- SQLite3 v3.8.11.2 (implementation JDBC-Template)
+
+> NOTE: depending on the scala version the list of the supported platforms available could be different.
+
+### New Features
+
+- support M1 ARM architecture
+
+### Incompatible changes
+
+- NONE
+
+### Miscellaneous changes
+
+- [WAYANG-3](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-3) Unification of logs
+- WAYANG-[[32](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-32), [33](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-33), [35](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-35), [38](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-38)] License issues removed
+
+### Known Issues
+
+- [WAYANG-23](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-23) Iteration Memory Issue
+
+### Bug Fixes
+
+- [WAYANG-12](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-12) Issues on the compilation
+- [WAYANG-40](https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-40) Different version of platform depending on the scala version
+
+# Previous Releases
diff --git a/index.md b/index.md
index a81dc5c..b2c8d50 100644
--- a/index.md
+++ b/index.md
@@ -18,15 +18,22 @@
title: "Home"
section-1:
title: |
- THE CROSS-PLATFORM </br>
- DATA PROCESSING SYSTEM
+ The API for big data </br>
+ __________ </br>
+ cross-platform processing
text: |
- Apache Wayang is a system designed to fully support cross-platform data processing: It enables users to run data analytics over multiple data processing platforms. For this, it provides an abstraction on top of existing platforms in order to run data analytic tasks on top of any set of platforms. As a result, users can focus on the logics of their applications rather on the intricacies of the underlying platforms.
+ Apache Wayang is a API first system designed to fully support cross-platform data processing: Wayang enables users to run data analytics over multiple data processing platforms, whithout changing the native code. </br>
+ Wayang provides an abstraction on top of existing platforms in order to run data processing and analytic tasks on top of any set of platforms. As a result, users can focus on the logics of their applications rather on the intricacies of the underlying platforms.
section-2:
title: Turning shadows into a show
text: |
Read more on how Apache Wayang converts the light and shadows of data processing platforms to amazing theatre for you.
section-3:
title: Features
- text: How we move the string for you
+ text: How we move the strings for you
+section-4:
+ title: Why is Apache Wayang faster as other, modern frameworks?
+ text: |
+ Apache Wayang uses internal optimization patterns to detect the best possible combination of computation and nodes. We know, just adding more nodes into a cluster doesn't mean more speed; each additional node has several tradeoffs, be it shuffle or communication bottlenecks.
+ Apache Wayang understands the UDF's and optimizes the function for the underlying processing platform. It also uses small JVM instances to reduce the operational overhead when processing a reduced number of data points.
---
