Presently, the following APIs are made private in SparkR that are used by the kernel to provide a form of communicate suitable for use as an interpreter:
SparkR only has an init()
method that connects to the backend service for R and creates a SparkContext instance. That I am aware, there is no other way to currently use SparkR. Because of this, a new method labelled sparkR.connect()
is used that retrieves the existing port under the environment variable EXISTING_SPARKR_BACKEND_PORT. This method is located in sparkR.R
and is exported via the following:
export("sparkR.connect")
SparkR low-level methods to communicate with the backend were marked private, but are used to communicate with our own bridge. If you need to use these invoke them with
SparkR:::isInstanceOf SparkR:::callJMethod SparkR:::callJStatic SparkR:::newJObject SparkR:::removeJObject SparkR:::isRemoveMethod SparkR:::invokeJava
org.apache.spark.api.r.RBackend
is marked as limited access to the package scope of org.apache.spark.api.r
- To circumvent, use a reflective wrapping under `org.apache.toree.kernel.interpreter.r.ReflectiveRBackend`