de-code / python-tf-bodypix Goto Github PK

View Code? Open in Web Editor NEW

124.0 3.0 19.0 271 KB

A Python implementation of the bodypix model.

License: MIT License

Makefile 6.36% Python 92.82% Dockerfile 0.64% Shell 0.18%

python-tf-bodypix's People

Contributors

Stargazers

Watchers

Forkers

aneeshjose benbatya alanmeeson ashishpatel26 tanaymeh satoshirobatofujimoto pandinosaurus gmontamat dcu-imai tenfleques gorogorov mrriahi ckoshien ninisoe1 princeon blaine141 pinto0309 cansik

python-tf-bodypix's Issues

Can not download the model

Thanks for this great repository. I face a weird problem and could not solve it. I run the Python API code, When it wants to download the model an error is raised. By default, the BodyPixModelPaths.MOBILENET_FLOAT_50_STRIDE_16 is download. After downloading the model architecture in the "download_model" function, it can not open it as a JSON file. The error is:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

After simple checking, I found the error comes from the following code in the "download_model" function.

with open(local_model_json_path, 'r', encoding='utf-8') as model_json_fp: model_json = json.load(model_json_fp)

I think the modle.json file which download is not correct. Can you help me how to fix it?
Thanks a lot for your help

Predict on batch

Is there a way to perform batch prediction to leverage GPU? I believe it is a functionality in the JS version, but I am not sure how to do that in this python version.

SavedModel file does not exist

I'm trying to implement bodypix into ComfyUI
I'm having issue of error
"SavedModel file does not exist"

Cannot get it run on windows machine

AutoFrame feature for software

This app is really cool. It's got 100% of the functionality I need on Linux to replicate Zoom's BG subtraction on Mac/Win.
Now, I want a feature where the face mask is used to crop the output image from ReplaceBackgroundApp.get_output_image() and then rescale that to the output image size.

So the face would be "tracked" regardless of where it goes in the frame ala AutoFrame from Nvidia (https://youtu.be/GRFjfGH87Dk?t=55)

Assign this to me

Convert model to tflite model

I want to run this code on raspberry pi 4. In the first step, I should convert this model to tflite model. I use the following link https://www.tensorflow.org/lite/convert/index to convert the model but it raises the following error.
OSError: SavedModel file does not exist at: ./models/mobilenet-float16-stride16/{saved_model.pbtxt|saved_model.pb}

I also use the following code for model conversion
python -m tf_bodypix \ convert-to-tflite \ --model-path \ "https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/model-stride16.json" \ --optimize \ --quantization-type=float16 \ --output-model-file "./mobilenet-float16-stride16.tflite"
it works and the model convert to tflite but when I want to use it in tensorflow lite based on this link https://www.tensorflow.org/lite/guide/inference, I found that the input model dimension is array([1, 1, 1, 3], dtype=int32). Can you help me to solve this problem?

auto-track sample implementation

@de-code I got inspired to split auto-track into a separate mode and just add it to this project.
I like the effect a lot with my 120deg FOV Spedal camera because I can move around the room and it (mostly) tracks my face which is very convenient for long boring meetings. I know that we talked about adding it to layered-vision but it just was easiest to add the additional mode in this project.

Enjoy!
Ben

auto_track.patch.txt

Part names

When I specify the part names ['left_upper_arm_front', 'right_upper_arm_front'], the mask highlights the back of the sitting lady's arms. When I switch to ['left_upper_arm_back', 'right_upper_arm_back'], the mask highlights the front of the arms instead. Same issue for lower arms.

How hard would it be to port the multi-person functions?

The original TensorFlowJS code includes net.segmentMultiPerson and net.segmentMultiPersonParts, which allow code to more easily distinguish people when they appear in an image together.

I started looking at the code to see if this is something I could port, but I just don't understand TensorFlow or the existing implementations well enough to figure out how.

Please consider this for a future enhancement.

Thanks much.

Could you add some images into the README?

get pose IndexError (BUG)

using the code from repo readme, arises error on get_pose function as follow:

~/.venvs/tf/lib/python3.8/site-packages/tf_bodypix/bodypix_js_utils/multi_person/decode_pose.py in getDisplacement(edgeId, point, displacements)
     33     # LOGGER.debug('point=%s, edgeId=%s, numEdges=%s', point, edgeId, numEdges)
     34     return Vector2D(
---> 35         y=displacements[point.y, point.x, edgeId],
     36         x=displacements[point.y, point.x, numEdges + edgeId]
     37     )

IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

OS: Linux base
Python: 3.8
Installation method: pip install tf-bodypix[all]
clean environment installation: false

Incorrect path concatenation

Using with Windows, the "os.path.join(model_base_path, weights_manifest_path)" in line 81 of download.py generates a wrong URL with "" separator.

Changing to "model_base_path+"/"+weights_manifest_path" corrected the problem

ValueError: 'size' must be a 1-D int32 Tensor

Getting a Value error on some images while not in some.

Entire Traceback:

Traceback (most recent call last):
  File "main.py", line 297, in <module>
    res = run_face_id(face_obj, img)
  File "main.py", line 28, in run_face_id
    text5 = face_obj.check_background()
  File "main.py", line 219, in check_background
    result = self.bodypix_model.predict_single(self.img)
  File "/home/vardan/face_id/lib/python3.7/site-packages/tf_bodypix/model.py", line 293, in predict_single
    model_input_image, padding = self.get_padded_and_resized(image, model_input_size)
  File "/home/vardan/face_id/lib/python3.7/site-packages/tf_bodypix/model.py", line 287, in get_padded_and_resized
    model_input_size.width
  File "/home/vardan/face_id/lib/python3.7/site-packages/tf_bodypix/bodypix_js_utils/util.py", line 79, in pad_and_resize_to
    resized = tf.image.resize([padded], [target_height, target_width])[0]
  File "/home/vardan/face_id/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/home/vardan/face_id/lib/python3.7/site-packages/tensorflow/python/ops/image_ops_impl.py", line 1546, in resize_images_v2
    skip_resize_if_same=False)
  File "/home/vardan/face_id/lib/python3.7/site-packages/tensorflow/python/ops/image_ops_impl.py", line 1226, in _resize_images_common
    raise ValueError('\'size\' must be a 1-D int32 Tensor')
ValueError: 'size' must be a 1-D int32 Tensor

The size of one of the images in question was (145, 226, 3) by width, height, and channels. Is there a minimum size requirement that may be causing it?

Could you add segmentMultiPersonParts into result of model?

Could not download josn file

There is an error occured when I run "bodypix_model = load_model(download_model(
BodyPixModelPaths.MOBILENET_RESNET50_FLOAT_STRIDE_32".
"Expectation: URL fetch failure on https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/resnet50/float/model-stride32.json:404 Not Found

Add ability to play looping videos in replace-background mode

Right now, only PIL is supported for the --background argument.
Change ReplaceBackgroundApp.get_image_source() check if the video extension is a video and use a video loading API (maybe scikit-video or ffmpeg-python?) to lazy load frames and return them as the background image

Part Segmentation Object - pose detection

Hi,

On the bodypix github page, the javascript bodypix version shows a keypoint skeleton which seems to be an allPoses field returned from the segmentPerson() call. Is it possible to get these pose locations from the python api? (ie a list of keypoints with their (x,y) locations)

graph execution error

Hello,

When I change the model to MOBILENET_FLOAT_100_STRIDE_8, I received the error below. I only used the sample code in the repo and only changed the model. The MOBILENET_FLOAT_75_STRIDE_16 and MOBILENET_FLOAT_50_STRIDE_16 were working. I appreciate your help in advance.

Users/takas/bodyPartSeg/bodypix/main.py
Traceback (most recent call last):
  File "c:\Users\takas\bodyPartSeg\bodypix\main.py", line 23, in <module>
    result = bodypix_model.predict_single(image_array)
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tf_bodypix\model.py", line 437, in predict_single
    tensor_map = self.predict_fn(model_input_image)
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tf_bodypix\model.py", line 134, in __call__
    return self.predict_fn(
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tf_bodypix\model.py", line 87, in __call__
    result = self.wrapped(*args, **kwargs)
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\polymorphic_function\monomorphic_function.py", line 1474, in __call__
    return self._call_impl(args, kwargs)
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\wrap_function.py", line 243, in _call_impl
    return super(WrappedFunction, self)._call_impl(
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\polymorphic_function\monomorphic_function.py", line 1492, in _call_impl
    return self._call_with_flat_signature(args, kwargs, cancellation_manager)
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\polymorphic_function\monomorphic_function.py", line 1541, in _call_with_flat_signature
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\polymorphic_function\monomorphic_function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\polymorphic_function\monomorphic_function.py", line 378, in call
    outputs = execute.execute(
  File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tensorflow\python\eager\execute.py", 
line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:

Detected at node 'MobilenetV1/Conv2d_7_depthwise/BatchNorm/batchnorm_1/add_1/conv' defined at (most recent call last):
    File "c:\Users\takas\bodyPartSeg\bodypix\main.py", line 14, in <module>
      bodypix_model = load_model(download_model(
    File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tf_bodypix\model.py", line 608, in 
load_model
      predict_fn = load_model_and_get_predict_function(model_path)
    File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tf_bodypix\model.py", line 579, in 
load_model_and_get_predict_function
      return load_using_tfjs_graph_converter_and_get_predict_function(model_path)
    File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tf_bodypix\model.py", line 564, in 
load_using_tfjs_graph_converter_and_get_predict_function
      tf_fn = tfjs_graph_converter.api.graph_to_function_v2(graph)
    File "C:\Users\takas\bodyPartSeg\bodypix\bodypix\lib\site-packages\tfjs_graph_converter\api.py", line 
420, in graph_to_function_v2
      return wrapped_import.prune(
Node: 'MobilenetV1/Conv2d_7_depthwise/BatchNorm/batchnorm_1/add_1/conv'
Current kernel implementation does not support dilations, received [1 2 2 1]
         [[{{node MobilenetV1/Conv2d_7_depthwise/BatchNorm/batchnorm_1/add_1/conv}}]] [Op:__inference_pruned_766]