This folder contains iOS RPC app that allows us to launch an rpc server on a iOS device. You will need XCode and an iOS device to use this.
While iOS platform itself doesn‘t allow us to run an unsigned binary, there is a partial ability to run JIT code on real iOS devices. While application is running under debug session, system allows allocating memory with write and execute permissions (a debugger requirement). So we can use this feature to implement the tvm.rpc.server.load_module PackedFunc, used to load code over RPC. For this purpose we use custom version of dlopen function which doesn’t check signature and permissions for module loading. This custom dlopen mechanic is integrated into TVM RPC as plugin and registered to execution only inside iOS RPC application.
The custom implementation of dlopen and other functions from dlfcn.h header are placed in separate repository, and will be downloaded automatically during cmake build for iOS.
Also, it is necessary to build libtvm_runtime.dylib for our iOS device. The iOS TVM RPC application will be linked with this library.
Run the build using the following commands:
export DEVELOPER_DIR=/Applications/Xcode.app # iOS SDK is part of Xcode bundle. Have to set it as default Dev Env cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_SYSTEM_NAME=iOS -DCMAKE_SYSTEM_VERSION=14.0 -DCMAKE_OSX_SYSROOT=iphoneos -DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 -DCMAKE_BUILD_WITH_INSTALL_NAME_DIR=ON -DUSE_IOS_RPC=ON # to enable build iOS RPC application from TVM project tree -DUSE_METAL=ON # to enable Metal runtime cmake --build . --target custom_dso_loader tvm_runtime
Before start, please run init_proj.py to update XCode developer metadata:
python3 init_proj.py --team_id XXXXXXXXXX --tvm_build_dir "/path/to/tvm/ios/build/folder"
You can get value of your team_id in the following ways:
You have registered Apple Developer Profile. In this case you developer Team ID available at https://developer.apple.com/account/#/membership
You are using your local developer profile. In this case, leave XXXXXXXXXX in the command instead of substituting a Team ID. Then open tvmrpc.xcodeproj by using XCode, click on the project name (tvmrpc) on the left panel. Then select target tvmrpc. At the bottom of this panel go to Signing & Capabilities tab and in the field Team select your local developer profile (Your Name (Personal Team)).
On the first run of the application you may see message Could not launch "tvmrpc" in the XCode and message Untrusted Developer on your device. In this case it will be necessary to check the certificate. Open Settings -> General -> Device Management -> Apple Development: <your_email> -> Trust "Apple Development: <your_email>" and click Trust. After than you should rerun your application in the XCode.
After this step, open tvmrpc.xcodeproj by using XCode, build the App and install the App on the phone.
Due to security restriction of iOS10. We cannot upload dynamic libraries to the App and load it from sandbox. Instead, we need to build a list of libraries, pack them into the app bundle, launch the RPC server and connect to test the bundled libraries. For more on the approach we use to work around this limitation, please take a look into section Building TVM runtime and custom DSO loader integration.
The test script tests/ios_rpc_test.py and tests/ios_rpc_mobilenet.py are good templates for demonstrating the workflow.
We have three different modes for iOS RPC server:
rpc_proxy. The RPC server on iOS device notify rpc_proxy which was run on host machine about itself and wait for incoming connections. Communications between client and server works through rpc_proxy.rpc_tracker and client connects to the RPC server through rpc_tracker.Start RPC server on your iOS device:
Connect button.After that you supposed to see something like this in the app on the device:
IP: <device_ip> Port: <rpc_server_port>
Printed IP is the IP address of your device and PORT is the number of port which was open for RPC connection. Next you should use them for connect your RPC client to the server.
Let's check that direct RPC connection works and we can upload a library with model and execute it on the device. For this purpose we will use ios_rpc_test.py. Run it:
python3 tests/ios_rpc_test.py --host <device_ip> --port <rpc_server_port> --mode "standalone"
This will compile TVM IR to shared libraries (CPU and Metal) and run vector addition on your iOS device. You are supposed to see something like this:
Metal: 0.000338692 secs/op CPU: 0.000219308 secs/op
Start the RPC proxy by running in a terminal:
python3 -m tvm.exec.rpc_proxy --host 0.0.0.0 --port 9090
On success, you should see something like this:
INFO:root:RPCProxy: client port bind to 0.0.0.0:9090 INFO:root:RPCProxy: Websock port bind to 8888
Connect your iOS device to the RPC proxy via the iOS TVM RPC application. Set the Address and Port fields to the address and port of the RPC tracker respectively. Select mode Proxy and push Connect button. In success the text on the button will be changed to Disconnect and Disconnected in the top of the screen will be changed to Connected. On RPC proxy side you can see the next message in a log:
INFO:root:Handler ready TCPSocketProxy:<iPhone IP address>:server:iphone
Then we can check that RPC connection works and we can upload a library with model and execute it on the target device. For this purpose we will use ios_rpc_test.py. Run it:
python3 tests/ios_rpc_test.py --host <host_ip_address> --port 9090 --mode "proxy"
The output should be the same as it was in previous section.
First start an RPC tracker using
python3 -m tvm.exec.rpc_tracker --host 0.0.0.0 --port 9190
On success, you should see something like this:
INFO:RPCTracker:bind to 0.0.0.0:9190
Connect your iOS device to the RPC tracker via the iOS TVM RPC applcation. Set the Address and Port fields to the address and port of the RPC tracker respectively. Select mode Tracker and push Connect button. In success the text on the button will be changed to Disconnect and Disconnected in the top of the screen will be changed to Connected. On the host side you can check the connect by the following command:
python3 -m tvm.exec.query_rpc_tracker --port 9190
You are supposed to see something like this:
Tracker address 127.0.0.1:9190 Server List ---------------------------- server-address key ---------------------------- 192.168.1.57:9190 server:iphone ---------------------------- Queue Status ------------------------------ key total free pending ------------------------------ iphone 1 1 0 ------------------------------
Then we can check that RPC connection works and we can upload a library with model and execute it on the target device. For this purpose we will use ios_rpc_test.py. Run it:
python3 tests/ios_rpc_test.py --host <host_ip_address> --port 9190 --mode "tracker"
The output will be the same as in section Standalone RPC.
Connection to the RPC server through usbmux can be used then you have slow, unstable or don't have any Wi-Fi connection. usbmux is used for binding local TCP port to port on the device and transfer packages between these ports by USB cable.
First of all you should install usbmux to your system. You can do it with brew:
brew install usbmuxd
After that you can use iproxy program for binding ports. You can use it for all described workflows. Let's take a look how it works for Standalone RPC.
First, start RPC server on your iOS device. You may see something like this in the app on the device:
IP: unknown Port: <rpc_server_port>
Note. Here IP: unknown because there was no Internet connection on the iOS device. Printed Port is the port of the RPC server on your iOS device. We will use it in binding ports. Run iproxy, specify local port which should be used for communication with device and the printed port on the device:
iproxy <local_port>:<rpc_server_port>
After this command you should see something like this:
Creating listening port <local_port> for device port <rpc_server_port> waiting for connection
Now we can check that RPC connection through usbmux works and we can upload a library with model and execute it on the device. For this purpose we will use ios_rpc_test.py. Run it:
python3 tests/ios_rpc_test.py --host 0.0.0.0 --port <local_port> --mode standalone
The output should be the same as in all previous runs.