wunderdogsw / go-23-app Goto Github PK

In case we need to recognize gestures, e.g. raising your hands in the air, research how this may be achieved

Research, play around to create spheres, cylinders, and cones in the browser on a canvas element

Check how these can be animated

Grand one shapes are:

• Sphere
• Cone
• Cylinder

Cover the detected person with all or some types of shapes so head, torso and limbs are visualized with these.

Create a proof of concept where Grandone objects (sphere, cone, cylinder) grow in some kind of biological way.

This would require investigating algorithms for growth in the natural world: e.g. trees, fungi, crystals, etc. and how this could be applied in THREE.js.

No need to combine it with pose detection for this PoC, just to show some growing shapes somewhere on the screen.

Goal: see how it can help us implement some augmented reality

If we would like to create an AR game, e.g. where someone has to catch or avoid shapes, how could this be implemented?

Render and move as many basic objects (shpere, cone, cylinder) as possible and see what or if there's a limit.

The venue will be pretty dark, so we want to make sure that TensorFlow models or other AI can recgonize faces/hands/bodies also in darker lighting conditions

Please test it out to see how well it performs and report the results of your findings, we may need a spotlight

investigate if multiple persons can be detected

Investigate how to visualize the person on screen (w/ tensorflow) to help user to see him/herself in relation to the scene.

User story

As a Grandone attendee
I want to see colorful cylinders instead of my arms
So that I could have fun moving them around

Acceptance criteria

Given that the app is running and I've allowed the camera

When I stand in front of the camera
Then I should see myself in the video feed with cylinders instead of arms
And the cylinders should be colored in one of the event's colors

When I move my arms
The cylinders should move in the video feed according to the movement of my arms

Implementation suggestions

Try to use Tensorflow pose detection in combination with Three JS

For some reason, hand recognition is not always working in Safari in this code base

The online demo does seem to work, so not sure if this is some Svelte problem or misuse of the TensorFlow code

The below error is produced, but not sure if this is the issue:

[Error] Error: Requested texture size [0x0] is invalid.
validateTextureSize — webgl_util.ts:205
createAndConfigureTexture — gpgpu_util.ts:57
acquireTexture — texture_manager.ts:78
uploadToGPU — backend_webgl.ts:1153
getTexture — backend_webgl.ts:683
fromPixels — FromPixels.ts:82
kernelFunc — engine.ts:644
(anonymous function) — engine.ts:710
scopedRun — engine.ts:478
runKernelFunc — engine.ts:706
f2 — operation.ts:51
(anonymous function) — hand-pose-detection.esm.js:17:17918
(anonymous function) — engine.ts:467
scopedRun — engine.ts:478
(anonymous function) — hand-pose-detection.esm.js:17:17897
(anonymous function) — hand-pose-detection.esm.js:17:1840
(anonymous function) — hand-pose-detection.esm.js:17:882
Promise
T — hand-pose-detection.esm.js:17:643
(anonymous function) — CircleHands.svelte:60
asyncFunctionResume
(anonymous function) — CircleHands.svelte:77
asyncFunctionResume
(anonymous function) — CircleHands.svelte:84
asyncFunctionResume
map
(anonymous function) — index.mjs:1953
flush — index.mjs:1191
promiseReactionJob

	(anonymous function) (CircleHands.svelte:104)
	asyncFunctionResume
	(anonymous function)
	promiseReactionJobWithoutPromise
	promiseReactionJob

Check how Physijs works with ThreeJS for fun and in case we want to go for the gamified concept: https://github.com/chandlerprall/Physijs

Learn how well ARl.js works and how we may use it

Try if it's possible to have the objects cast shadows to each other.

shadow should be full black: 0x000000

One of the ideas was to turn many people into cones, but this may be CPU intensive

Please test body/posture recognition for many people and see how well it performs on mobile

People will be using their mobile phones, so we need to make sure AR can run on various devices