<!--- Licensed to the Apache Software Foundation (ASF) under one -->
<!--- or more contributor license agreements.  See the NOTICE file -->
<!--- distributed with this work for additional information -->
<!--- regarding copyright ownership.  The ASF licenses this file -->
<!--- to you under the Apache License, Version 2.0 (the -->
<!--- "License"); you may not use this file except in compliance -->
<!--- with the License.  You may obtain a copy of the License at -->

<!---   http://www.apache.org/licenses/LICENSE-2.0 -->

<!--- Unless required by applicable law or agreed to in writing, -->
<!--- software distributed under the License is distributed on an -->
<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
<!--- KIND, either express or implied.  See the License for the -->
<!--- specific language governing permissions and limitations -->
<!--- under the License. -->

# iOS TVM RPC

This folder contains iOS RPC app that allows us to launch an rpc server on a iOS
device. You will need XCode and an iOS device to use this.

## Table of Contents
* [Building](#building)
    * [Building TVM runtime and custom DSO loader plugin](#building-tvm-runtime-and-custom-dso-loader-plugin)
    * [Building iOS TVM RPC application](#building-ios-tvm-rpc-application)
* [Workflow](#workflow)
    * [Standalone RPC](#standalone-rpc)
    * [iOS RPC App with proxy](#ios-rpc-app-with-proxy)
    * [iOS RPC App with tracker](#ios-rpc-app-with-tracker)
* [Communication without Wi-Fi and speed up in case of slow Wi-Fi](#communication-without-wi-fi-and-speed-up-in-case-of-slow-wi-fi)

## Building
### Building TVM runtime and custom DSO loader plugin
While iOS platform itself doesn't allow us to run an unsigned binary, there is a
partial ability to run JIT code on real iOS devices. While application is
running under debug session, system allows allocating memory with write and
execute permissions (a debugger requirement). So we can use this feature to
implement the `tvm.rpc.server.load_module` PackedFunc, used to load code over
RPC. For this purpose we use custom version of `dlopen` function which doesn't
check signature and permissions for module loading.  This custom `dlopen`
mechanic is integrated into TVM RPC as plugin and registered to execution only
inside iOS RPC application.

The custom implementation of `dlopen` and other functions from `dlfcn.h` header are placed in separate repository,
and will be downloaded automatically during cmake build for iOS.

Also, it is necessary to build `libtvm_runtime.dylib` for our iOS device. The
iOS TVM RPC application will be linked with this library.

Run the build using the following commands:
```shell
export DEVELOPER_DIR=/Applications/Xcode.app  # iOS SDK is part of Xcode bundle. Have to set it as default Dev Env
cmake ..
  -DCMAKE_BUILD_TYPE=Debug
  -DCMAKE_SYSTEM_NAME=iOS
  -DCMAKE_SYSTEM_VERSION=14.0
  -DCMAKE_OSX_SYSROOT=iphoneos
  -DCMAKE_OSX_ARCHITECTURES=arm64
  -DCMAKE_OSX_DEPLOYMENT_TARGET=14.0
  -DCMAKE_BUILD_WITH_INSTALL_NAME_DIR=ON
  -DUSE_IOS_RPC=ON  # to enable build iOS RPC application from TVM project tree
  -DUSE_METAL=ON    # to enable Metal runtime

cmake --build . --target custom_dso_loader tvm_runtime
```

### Building iOS TVM RPC application
Before start, please run [init_proj.py](./init_proj.py) to update XCode developer metadata:
```shell
python3 init_proj.py --team_id XXXXXXXXXX --tvm_build_dir "/path/to/tvm/ios/build/folder"
```
You can get value of your `team_id` in the following ways:
- **You have registered Apple Developer Profile**. In this case you developer
  Team ID available at https://developer.apple.com/account/#/membership
- You are using your local developer profile. In this case, leave `XXXXXXXXXX`
  in the command instead of substituting a Team ID. Then open `tvmrpc.xcodeproj`
  by using XCode, click on the project name (`tvmrpc`) on the left panel. Then
  select target `tvmrpc`. At the bottom of this panel go to `Signing &
  Capabilities` tab and in the field `Team` select your local developer profile
  (`Your Name (Personal Team)`).

  On the first run of the application you may see message `Could not launch
  "tvmrpc"` in the XCode and message `Untrusted Developer` on your device. In
  this case it will be necessary to check the certificate. Open
  `Settings -> General -> Device Management -> Apple Development: <your_email>
  -> Trust "Apple Development: <your_email>"` and click `Trust`. After than you
  should rerun your application in the XCode.

After this step, open `tvmrpc.xcodeproj` by using XCode, build the App and
install the App on the phone.

## Workflow
Due to security restriction of iOS10. We cannot upload dynamic libraries to the
App and load it from sandbox.  Instead, we need to build a list of libraries,
pack them into the app bundle, launch the RPC server and connect to test the
bundled libraries.  For more on the approach we use to work around this
limitation, please take a look into section
[Building TVM runtime and custom DSO loader integration](#building-tvm-runtime-and-custom-DSO-loader-plugin).

The test script [tests/ios_rpc_test.py](tests/ios_rpc_test.py) and
[tests/ios_rpc_mobilenet.py](tests/ios_rpc_mobilenet.py) are good templates for
demonstrating the workflow.

We have three different modes for iOS RPC server:
- [Standalone RPC](#standalone-rpc): In this mode RPC server open port on the device and listening. Then
  client connects to the server directly without any mediators.
- [iOS RPC application with Proxy](#ios-rpc-app-with-proxy): RPC server and RPC client communicates through
  `rpc_proxy`. The RPC server on iOS device notify `rpc_proxy` which was run on
  host machine about itself and wait for incoming connections. Communications
  between client and server works through `rpc_proxy`.
- [iOS RPC application with Tracker](#ios-rpc-app-with-tracker): RPC server registered in the `rpc_tracker`
  and client connects to the RPC server through `rpc_tracker`.

### Standalone RPC
Start RPC server on your iOS device:
- Push on the `Connect` button.

After that you supposed to see something like this in the app on the device:
```
IP: <device_ip>
Port: <rpc_server_port>
```

Printed `IP` is the IP address of your device and `PORT` is the number of port
which was open for RPC connection. Next you should use them for connect your RPC
client to the server.

Let's check that direct RPC connection works and we can upload a library with
model and execute it on the device. For this purpose we will use
[ios_rpc_test.py](tests/ios_rpc_test.py). Run it:
```shell
python3 tests/ios_rpc_test.py --host <device_ip> --port <rpc_server_port> --mode "standalone"
```
This will compile TVM IR to shared libraries (CPU and Metal) and run vector
addition on your iOS device. You are supposed to see something like this:
```
Metal: 0.000338692 secs/op
CPU: 0.000219308 secs/op
```

### iOS RPC App with proxy
Start the RPC proxy by running in a terminal:
```shell
python3 -m tvm.exec.rpc_proxy --host 0.0.0.0 --port 9090
```

On success, you should see something like this:
```
INFO:root:RPCProxy: client port bind to 0.0.0.0:9090
INFO:root:RPCProxy: Websock port bind to 8888
```
Connect your iOS device to the RPC proxy via the iOS TVM RPC application. Set
the `Address` and `Port` fields to the address and port of the RPC tracker
respectively. Select mode `Proxy` and push `Connect` button. In success the
text on the button will be changed to `Disconnect` and `Disconnected` in the top
of the screen will be changed to `Connected`.
On RPC proxy side you can see the next message in a log:
```
INFO:root:Handler ready TCPSocketProxy:<iPhone IP address>:server:iphone
```
Then we can check that RPC connection works and we can upload a library with
model and execute it on the target device. For this purpose we will use
[ios_rpc_test.py](tests/ios_rpc_test.py). Run it:
```shell
python3 tests/ios_rpc_test.py --host <host_ip_address> --port 9090 --mode "proxy"
```
The output should be the same as it was in previous section.

### iOS RPC App with tracker
First start an RPC tracker using
```shell
python3 -m tvm.exec.rpc_tracker --host 0.0.0.0 --port 9190
```
On success, you should see something like this:
```
INFO:RPCTracker:bind to 0.0.0.0:9190
```
Connect your iOS device to the RPC tracker via the iOS TVM RPC applcation. Set
the `Address` and `Port` fields to the address and port of the RPC tracker
respectively. Select mode `Tracker` and push `Connect` button. In success the
text on the button will be changed to `Disconnect` and `Disconnected` in the top
of the screen will be changed to `Connected`. On the host side you can check the
connect by the following command:
```shell
python3 -m tvm.exec.query_rpc_tracker --port 9190
```
You are supposed to see something like this:
```
Tracker address 127.0.0.1:9190

Server List
----------------------------
server-address  key
----------------------------
192.168.1.57:9190       server:iphone
----------------------------

Queue Status
------------------------------
key      total  free  pending
------------------------------
iphone   1      1     0
------------------------------
```

Then we can check that RPC connection works and we can upload a library with
model and execute it on the target device. For this purpose we will use
[ios_rpc_test.py](tests/ios_rpc_test.py). Run it:
```shell
python3 tests/ios_rpc_test.py --host <host_ip_address> --port 9190 --mode "tracker"
```
The output will be the same as in section
[Standalone RPC](#standalone-rpc).

## Communication without Wi-Fi and speed up in case of slow Wi-Fi
Connection to the RPC server through `usbmux` can be used then you have slow,
unstable or don't have any Wi-Fi connection. `usbmux` is used for binding local
TCP port to port on the device and transfer packages between these ports by USB
cable.

First of all you should install `usbmux` to your system. You can do it with
brew:
```shell
brew install usbmuxd
```
After that you can use `iproxy` program for binding ports. You can use it for
all described workflows. Let's take a look how it works for
[Standalone RPC](#standalone-rpc).

First, start RPC server on your iOS device. You may see something like this in
the app on the device:
```
IP: unknown
Port: <rpc_server_port>
```
**Note.** Here `IP: unknown` because there was no Internet connection on the iOS
device.
Printed `Port` is the port of the RPC server on your iOS device. We will use it
in binding ports. Run `iproxy`, specify local port which should be used for
communication with device and the printed port on the device:
```shell
iproxy <local_port>:<rpc_server_port>
```
After this command you should see something like this:
```
Creating listening port <local_port> for device port <rpc_server_port>
waiting for connection
```
Now we can check that RPC connection through `usbmux` works and we can upload a
library with model and execute it on the device. For this purpose we will use
[ios_rpc_test.py](tests/ios_rpc_test.py). Run it:
```shell
python3 tests/ios_rpc_test.py --host 0.0.0.0 --port <local_port> --mode standalone
```
The output should be the same as in all previous runs.