This is a Unity (2020.3.8f1) Plugin to use MediaPipe (0.8.5).

To use this plugin, you need to build native libraries for the target platforms (Desktop/UnityEditor, Android, iOS). If you'd like to build them on your machine, below commands/tools/libraries are required (not required if you use Docker).

• Python >= 3.9.0
• Bazel >= 3.7.2, (< 4.0.0 for iOS)
• GCC/G++ >= 8.0.0 (Linux, macOS)
• NuGet

• Linux Desktop (tested on ArchLinux)
• Android
• iOS
• macOS (CPU only)
• Windows 10 (CPU only, experimental)

*1: crashes sometimes when the graph exits.

Run commands at the project root if not specified otherwise.
Also note that you need to build native libraries for Desktop CPU or GPU to run this plugin on UnityEditor.

• Docker for Linux (experimental)
• Linux
• Docker for Windows (experimental)
• Windows
• macOS

1. Build a Docker image

   docker build -t mediapipe_unity:latest . -f docker/linux/x86_64/Dockerfile
   
   # Above command may fail depending on glibc version installed to your host machine.
   # cf. https://serverfault.com/questions/1052963/pacman-doesnt-work-in-docker-image
   #
   # In that case, apply a patch.
   #
   #   git apply docker/linux/x86_64/glibc.patch
   
   # You can specify MIRROR_COUNTRY to increase download speed
   docker build --build-arg RANKMIRROS=true --build-arg MIRROR_COUNTRY=FR,GB -t mediapipe_unity:latest . -f docker/linux/x86_64/Dockerfile

2. Run a Docker container

   # Run with `Packages` directory mounted to the container
   docker run --mount type=bind,src=$PWD/Packages,dst=/home/mediapipe/Packages \
       -it mediapipe_unity:latest

3. Run build command inside the container

   # Build native libraries for Desktop CPU.
   # Note that you need to specify `--opencv=cmake`, because OpenCV is not installed to the container.
   python build.py build --desktop cpu --opencv=cmake -v
   
   # Build native libraries for Desktop GPU and Android
   python build.py build --desktop gpu --android arm64 --opencv=cmake -v

If the command finishes successfully, required files will be installed to your host machine.

1. Install OpenCV and FFmpeg (optional)

   By default, it is assumed that OpenCV 3 is installed under /usr (e.g. /usr/lib/libopencv_core.so).
   If your version or path is different, please edit third_party/opencv_linux.BUILD and WORKSPACE.

   For example, if you use ArchLinux and opencv3-opt, OpenCV 3 is installed under /opt/opencv3.
   In this case, your WORKSPACE will look like this.

   new_local_repository(
       name = "linux_opencv",
       build_file = "@//third_party:opencv_linux.BUILD",
       path = "/opt/opencv3",
   )

   On the other hand, if you use Ubuntu, probably OpenCV's shared libraries is installed under /usr/lib/x86_64-linux-gnu/.
   In that case, your opencv_linux.BUILD would be like this.

   cc_library(
       name = "opencv",
       srcs = glob(
           [
               "lib/x86_64-linux-gnu/libopencv_core.so",
               "lib/x86_64-linux-gnu/libopencv_calib3d.so",
               "lib/x86_64-linux-gnu/libopencv_features2d.so",
               "lib/x86_64-linux-gnu/libopencv_highgui.so",
               "lib/x86_64-linux-gnu/libopencv_imgcodecs.so",
               "lib/x86_64-linux-gnu/libopencv_imgproc.so",
               "lib/x86_64-linux-gnu/libopencv_video.so",
               "lib/x86_64-linux-gnu/libopencv_videoio.so",
           ],
       ),
       ...
   )

2. Install Bazelisk and NuGet, and ensure you can run them

   bazel --version
   nuget

3. Install numpy

   pip install numpy --user
   
   # or
   # pip3 install numpy --user

4. (Optional) Install Android SDK and Android NDK, and set environment variables

   # bash
   export ANDROID_HOME=/path/to/SDK
   # ATTENTION!: Currently Bazel does not support NDK r22, so use NDK r21 instead.
   export ANDROID_NDK_HOME=/path/to/ndk/21.4.7075529

5. Run build command

1. Switch to windows containers

   Note that Hyper-V backend is required (that is, Windows 10 Home is not supported).

2. Build a Docker image

   docker build -t mediapipe_unity:windows . -f docker/windows/x86_64/Dockerfile

   This process will hang when MSYS2 is being installed.
   If this issue occurs, remove C:\ProgramData\Docker\tmp\hcs*\Files\$Recycle.Bin\ manually (hcs* is random name).
   cf. docker/for-win#8910

3. Run a Docker container

   Rem Run with `Packages` directory mounted to the container
   Rem Specify `--cpus` and `--memory` options according to your machine.
   docker run --cpus=16 --memory=8192m ^
       --mount type=bind,src=%CD%\Packages,dst=C:\mediapipe\Packages ^
       -it mediapipe_unity:windows

4. Run build command inside the container

   python build.py build --desktop cpu --include_opencv_libs -v

If the command finishes successfully, required files will be installed to your host machine.

1. Switch to Linux containers

   Note that you cannot build native libraries for Desktop with Linux containers.

2. Build a Docker image

   Rem You may skip applying a patch depending on your machine settings.
   Rem See https://serverfault.com/questions/1052963/pacman-doesnt-work-in-docker-image for more details.
   git apply docker/linux/x86_64/pacman.patch
   
   docker build -t mediapipe_unity:linux . -f docker/linux/x86_64/Dockerfile

3. Run a Docker container

   Rem Run with `Packages` directory mounted to the container
   Rem Specify `--cpus` and `--memory` options according to your machine.
   docker run --cpus=16 --memory=8192m ^
       --mount type=bind,src=%CD%\Packages,dst=/home/mediapipe/Packages ^
       -it mediapipe_unity:linux

4. Run build command inside the container

   python build.py build --android arm64 -v

If the command finishes successfully, required files will be installed to your host machine.

1. Follow mediapipe's installation guide and install MSYS2, Python, Visual C++ Build Tools 2019, WinSDK and Bazel (step1 ~ step6).

2. Install Opencv

   By default, it is assumed that OpenCV 3.4.10 is installed under C:\opencv.
   If your version or path is different, please edit third_party/opencv_windows.BUILD and WORKSPACE.

3. Install NuGet, and ensure you can run them

   bazel --version
   nuget

4. Install numpy

   pip install numpy --user

5. Set PYTHON_BIN_PATH

   set PYTHON_BIN_PATH=C:\path\to\python.exe

   When the path includes space characters (e.g. C:\Program Files\Python39\python.exe), it's reported that build command will fail.
   In that case, install python to another directory as a workaround (it's unnecessary to set the path to %PATH%, but don't forget to install numpy for the new Python).

6. Run build command

   python build.py build --desktop cpu --include_opencv_libs -v

You cannot build native libraries for Android on Windows 10, so use Docker for Windows instead.

1. Install Homebrew

2. Install OpenCV 3 and FFmpeg (optional)

   brew install opencv@3
   brew uninstall --ignore-dependencies glog

3. Install Python

   brew install python
   
   # Python version must be >= 3.9.0
   sudo ln -s -f /usr/local/bin/python3.9 /usr/local/bin/python
   pip3 install --user six numpy

4. Install Bazelisk and NuGet

   brew install bazelisk
   # Note that you need to specify bazel version if you'd like to build for iOS
   # See https://github.com/bazelbuild/bazelisk for more details.
   #
   #  e.g. export USE_BAZEL_VERSION=3.7.2
   
   brew install nuget

5. (Optional) Install Xcode

6. (Optional) Install Android SDK and Android NDK, and set environment variables

   # bash
   export ANDROID_HOME=/path/to/SDK
   # ATTENTION!: Currently Bazel does not support NDK r22, so use NDK r21 instead.
   export ANDROID_NDK_HOME=/path/to/ndk/21.4.7075529

7. Run build command

# Required files (native libraries, model files, C# scripts) will be built and installed.

# Build for Desktop with GPU enabled.
python build.py build --desktop gpu -v

# If you've not installed OpenCV locally, you need to build OpenCV from sources for Desktop.
python build.py build --desktop gpu --opencv=cmake -v

# If FFmpeg is installed, you can build OpenCV with FFmpeg.
python build.py build --desktop gpu --opencv=cmake --opencv_deps=ffmpeg -v

# Build for Desktop with GPU disabled, and copy OpenCV shared libraries to `Packages`.
# If you use Windows 10, you would run this command.
python build.py build --desktop cpu --include_opencv_libs -v

# Build for Desktop, Android, and iOS
python build.py build --desktop cpu --android arm64 --ios arm64 -v

Run python build.py --help and python build.py build --help for more details.

Select Mediapipe/Samples/Scenes/DesktopDemo and play.

If you'd like to run graphs on CPU, uncheck Use GPU from the inspector window.

OpenCV's path may not be configured properly.

If you're sure the path is correct, please check on Load on startup in the plugin inspector, click Apply button, and restart Unity Editor.
Some helpful logs will be output in the console.

If you encounter an error like below and you use OpenGL Core as the Unity's graphics APIs, please try Vulkan.

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000_mediapipe/mediapipe/gpu/gl_context_egl.cc:261)

If you set an environment variable GLOG_v before loading native libraries (e.g. libmediapipe_c.so), MediaPipe will output verbose logs to log files (e.g. Editor.log, Player.log).

void OnEnable() {
    // see https://github.com/google/glog#setting-flags
    System.Environment.SetEnvironmentVariable("GLOG_v", "2");
}

You can also setup Glog so that it writes logs to files.

using System.IO;

void OnEnable() {
    var logDir = Path.Combine(Application.persistentDataPath, "Logs");

    if (!Directory.Exists(logDir)) {
      Directory.CreateDirectory(logDir);
    }

    Glog.Initialize("MediaPipeUnityPlugin", logDir);
}

void OnDisable() {
    Glog.Shutdown();
}

• Prepare API Documents
• Implement cross-platform APIs to send images to MediaPipe
• use CVPixelBuffer on iOS
• Box Tracking (on Windows)
• Objectron
• KNIFT

MIT

Note that some files are distributed under other licenses.

• MediaPipe (Apache Licence 2.0)