license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. layout: about title: “About” img-architecture: path: assets/img/architecture.png title: Architecture of Apache Wayang img-plan: path: assets/img/plan.png title: SGD plans

Apache Wayang has a three-layer data processing abstraction that sits between user applications and data processing platforms, such as Hadoop and Spark. The figure below depicts the Apache Wayang architecture: (i) an application layer that models all application-specific logic; (ii) a core layer that provides the intermediate representation between applications and processing platforms; and (iii) a platform layer that embraces the underlying processing platforms. Overall, the input of an application layer comprises the logical operators provided by users (or generated by a declarative query parser) and the output is a physical plan (WayangPlan). The WayangPlan is then passed to the core layer where cross-platform optimizations take place to produce an execution plan (ExecutionPlan).

Notice that, in contrast to DBMSs, Apache Wayang decouples physical and execution levels. This separation allows applications to express physical plans in terms of algorithmic needs only, without being tied to a particular processing platform. The salient features of Apache Wayang are cross-platform task execution, high-performance, flexibility, and ease-of-use.