j20001970 / gdmp Goto Github PK

This is 2 weeks newer than 0.91 1st Feb 2023 releaes

Add mediapipe tasks face blendshapes graph

Hi!
Thanks for your work. I am trying to use your project for my game and have faced with some issues. May I ask you to show a tutorial with usage of your project for pose tracking? Thanks a lot!

Best regards,
Anton

Framework

• Try out mediapipe::api2

Vision Tasks

• HolisticLandmarker

🐧 Linux

• Pipewire backend for CameraHelper
• Build with Freedesktop SDK (Flatpak) in CI

🍎 macOS

• macOS CPU build
• CameraHelper backend using either OpenCV or AVFoundation (maybe share code with iOS?)

🌐 Web

• Web CPU build (#23)

Multiple Architectures

• Android x86_64
• Linux arm64
• Windows arm64
• macOS arm64

OpenCV

• Avoid linking entire libraries to reduce size

If freetype, libpng, zlib, graphite, harfbuzz is system-provided instead of builtin when building Godot editor, loading GDMP library will cause immediate crash on said editor builds.

The issue primarily happens on Linux-based environments, not sure if the same also occur on Windows.

This is a problem because unlike official Godot editor that has everything builtin, distribution packagers might configure Godot to use system library for some modules (e.g. Arch, Flatpak), and users of these builds will run into such issue.

While the root cause is unknown yet, I open this issue for future reference.

Related issues:

1. j20001970/GDMP-demo#17 (comment)
2. flathub/org.godotengine.Godot#153

Hi I finally got the library build and I got simple camera test done.

But as soon as I want to test one of the examples from the Test project it always crashes on:
\GDMP\tasks\vision\vision_task.cc:7

With following error:

For building the library I swapped over to 4.2 branch of godot-cpp I would guess this is already the problem.

Versions of everything:
Godot 4.2.2 Stable
bazel 7.1.1
opencv 3.4.10 < tried to use the exact same version as in the BUILDING readme, I don't know if I can use the later one.
Microsoft Visual Studio 2019

Do you have any Idea what I could try and test or what is the problem maybe?

I get these errors when I try to run setup.py. Thank you in advance for your great work.

H:\GDMP>setup.py --godot-binary "C:\Users\Sereja\Desktop\Godot_v4.0.1-stable_win64.exe"
Dumping Extension API
Godot Engine v4.0.1.stable.official.cacf49999 - https://godotengine.org

Traceback (most recent call last):
File "H:\GDMP\setup.py", line 139, in
generate_bindings(api_json_path)
File "H:\GDMP\setup.py", line 31, in generate_bindings
import binding_generator as bg
ModuleNotFoundError: No module named 'binding_generator'

H:\GDMP>setup.py --extension-api-json "H:\GDMP\extension_api.json"
Traceback (most recent call last):
File "H:\GDMP\setup.py", line 139, in
generate_bindings(api_json_path)
File "H:\GDMP\setup.py", line 31, in generate_bindings
import binding_generator as bg
ModuleNotFoundError: No module named 'binding_generator'

System: Windows 10
Python: 3.11.8

I am currently following the build instruction for Android and found multiple errors:

ERROR: An error occurred during the fetch of repository 'local_execution_config_python':
...
Error in fail: Repository command failed
INFO: Could not find files for the given pattern(s).
ERROR: C:/users/user/downloads/ownprojects/gdmp/mediapipe/WORKSPACE:544:14: fetching local_python_configure rule //external:local_execution_config_python: Traceback (most recent call last):
...
Error in fail: Repository command failed
INFO: Could not find files for the given pattern(s).
ERROR: Analysis of target '//GDMP/android:GDMP' failed; build aborted: Repository command failed
INFO: Could not find files for the given pattern(s).

I searched for clues that might cause this error, and it indicates that Bazel fails to find the local Python binary, but the Python binary referenced if from setup.py, which should be valid. Here are the python path and the build args from the code for reference

python_path: C:\\Users\\User\\Downloads\\ownprojects\\GDMP\\venv\\Scripts\\python.exe
build_args: ['C:\\ProgramData\\chocolatey\\bin\\bazelisk.EXE', 'build', '--action_env', 'PYTHON_BIN_PATH=C:\\\\Users\\\\User\\\\Downloads\\\\ownprojects\\\\GDMP\\\\venv\\\\Scripts\\\\python.exe', '-c', 'dbg', '--config=android', '--copt=-fPIC', '--cpu=arm64-v8a', '//GDMP/android:GDMP']

Do you have any clue how to solve this issue? Many thanks!

Following up of this journey of having GDMP on iOS device. Not sure if this is a Godot issue or the library issue, but the mediapipe tasks are not starting.

i've built the --type release --arch arm64 library and imported it to the GDMP-demo project. Added the messages for privacy permissions (Camera, photo and mic access). The app is able to be installed and opened, and i can navigate to the Hand Landmarker task, but upon clicking the Open camera button, nothing happens.

I went and check the process. I suspect it might be an issue about requesting permission with the camera helper, which did not show up, hence no _permission_result() callback and the program cannot proceed.

FYI, I am using iPad Air 5th generation for testing

Originally posted by @WaterBoiledPizza in #32 (comment)

@j20001970
Is it feasible to upgrade to the latest MediaPipe v0.10.10 so the community could take advantage of the latest feature :

Holistic

https://chuoling.github.io/mediapipe/solutions/holistic.html

I notice you have no mention of building for Mac or Web. Is there a reason for this or are you not testing these platforms?

hi I get this error when attempting to build gdmp:

ERROR: /media/project/taskdisk/24.01/GDMP/mediapipe/GDMP/desktop/BUILD:1:10: error loading package 'GDMP': at /home/project/.cache/bazel/_bazel_project/ab1d178f36bd2d9a03b46953fa4f356a/external/hedron_compile_commands/refresh_compile_commands.bzl:60:6: Unable to find package for @hedron_compile_commands_pip//:requirements.bzl: The repository '@hedron_compile_commands_pip' could not be resolved: Repository '@hedron_compile_commands_pip' is not defined. and referenced by '//GDMP/desktop:GDMP'
ERROR: Analysis of target '//GDMP/desktop:GDMP' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.060s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
    currently loading: GDMP
    Fetching repository @local_config_cc; starting

I am not sure how to resolve, any advice would be appreciated.

current system gentoo linux 64 bit with month-old software updates.

thanks.

Do check out this new project, perhaps the more official way to generate the GDExtension C# Bindings

Tested on c++ GDExtension:

• Examples/DodgeTheCreeps/DodgeTheCreeps.gdextension

dotnet publish DodgeTheCreeps/Source/DodgeTheCreeps.csproj 
-property:GenerateFullPaths=true 
-consoleloggerparameters:NoSummary 
-p:NativeLib=Shared 
--use-current-runtime -o DodgeTheCreeps/Build

• Examples/Summator/Summator.gdextension

dotnet publish Summator/Source/Summator.csproj 
-property:GenerateFullPaths=true 
-consoleloggerparameters:NoSummary 
-p:NativeLib=Shared 
--use-current-runtime -o Summator/Build

Is there a specific version of godot-cpp that should be used?
#14 (comment)

Is there necessary to have TWO repos. one for Godot3 and the other for Godot4? Since both have different GDExtension technology

There have been lots of GDExtension activities in Godot-cpp last one to two weeks
Do consider updating. Await feedback

The latest release does for me as it uses OpenCV v.4.0.7 or requires libopencv_core.so.407, while I have openCV v4.9.0 installed.

Build steps I used:

git submodule update --init --recursive --depth=1
./setup.py --godot-binary ./Godot_v4.2.2-stable_linux.x86_64
./build.py desktop --type release --output output --arch x86_64

Following the build steps as described leads to the following output:

./build.py desktop --type release --output output --arch x86_64
Starting local Bazel server and connecting to it...
ERROR: Traceback (most recent call last):
        File "/home/matter/.cache/bazel/_bazel_matter/57a186cbfd2843e4219f9d5290032761/external/build_bazel_rules_apple/apple/internal/rule_support.bzl", line 217, column 36, in <toplevel>
                deps_cfg = apple_common.multi_arch_split,
Error: 'apple_common' value has no field or method 'multi_arch_split'
ERROR: error loading package 'mediapipe/gpu': at /home/matter/.cache/bazel/_bazel_matter/57a186cbfd2843e4219f9d5290032761/external/build_bazel_rules_apple/apple/ios.bzl:33:5: at /home/matter/.cache/bazel/_bazel_matter/57a186cbfd2843e4219f9d5290032761/external/build_bazel_rules_apple/apple/internal/ios_rules.bzl:75:5: initialization of module 'apple/internal/rule_support.bzl' failed
ERROR: /home/matter/projects/GDMP/mediapipe/GDMP/gpu/BUILD:1:11: errors encountered resolving select() keys for //GDMP/gpu:gpu_helper
ERROR: Analysis of target '//GDMP/desktop:GDMP' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.086s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED:
    Fetching repository @@bazel_tools~cc_configure_extension~local_config_cc; starting
    Fetching repository @@npm; starting
    Fetching repository @@com_google_protobuf; starting
    Fetching repository @@yarn; starting
    Fetching repository @@build_bazel_rules_android; starting
    Fetching repository @@rules_java; starting
    Fetching repository @@com_google_absl; starting
    Fetching repository @@com_google_mediapipe_hand_landmarker_task; starting ... (10 fetches)
Traceback (most recent call last):
  File "/home/matter/projects/GDMP/./build.py", line 188, in <module>
    cmd()
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/bazel', 'build', '-c', 'opt', '--copt=-DMESA_EGL_NO_X11_HEADERS', '--copt=-DEGL_NO_X11', '--copt=-fPIC', '//GDMP/desktop:GDMP']' returned non-zero exit status 1.

Hi! I saw your project from a This week in Godot post. I'm working on a project that relies on face tracking data, and was planning on adding mediapipe support.

What are the difficulties in adding Windows support? I can take a look at tackling that.

Symptom: deleting a graph object will break Godot's rendering.

• On desktop linux, it become unresponsive while other systems like audio are still working, but no error message observed in editor or stderr.
• On Android, it's the same except the screen become blank and produced a EGL_BAD_DISPLAY error in logcat.

The current workaround is to not deleting the graph object, but definitively not the best way as every created graph will pile up memory usage and probably slow down performance until restart.

I have no idea how to further debugging this for now, probably require someone who is more experienced to troubleshoot the issue.

Hi!

thank you for the project, it is very interesting, but can you pls describe in more detail what I should do in step number 5. I am not very familiar with android that's why it is not obvious to me.

Thank you in advance,
Best