lumalabs / luma-web-examples Goto Github PK

the cloud point change to the splats effect is awesome! but i don't konw how to get these code.Anyone can help??

I was expecting to find the implementation in this repo. Thanks! Looking forward to contribute.

I mean, that I can get genie's interface in my application to obtain text to 3d files, stl/obj/glb, etc.

Hi,

I am using the Luma WebGL library in one of my projects. I have noticed that on some older devices the website becomes really slow once the Luma splat has been rendered. I believe it is because of the number of particles that are rendered since when I upload a higher resolution video the lag becomes worse.

Do you have a way of easily reducing the number of particles for example by providing a parameter such as particleDensity to LumaSplatsThree? In case there isn't such a functionality, do you plan to add one or is there a way to achieve this in another way?

Does LumaSplatsThree support loading external splat files ?

E.g. I crop unnecessary points in https://playcanvas.com/supersplat/editor and want to import edited splat back to Lumalabs Js library. Is that posible?

I find that in the example of the offical luma website I can remove the camera by clicking somewhere, can I achieve this using the webgl library?

attempting to compile the demo scripts and getting this error ~/luma-web-examples/node_modules/@lumaai/luma-web/misc/inline-worker-plugin.d.ts' not found. doesn't look like the misc directory is even in the package. wondering if this is something I need to manually create

Hi all,

I just successfully created capture by calling Luma AI API and got it shown in my Google browser through given url, but I do not find any api call can help with downloading this capture to my local machine, is there any api doc can assist in that?

Thanks!

Is there support for user-defined masks within the LumaWebGL renderer? i.e. can we use a given mesh to define a group of points in the splat that are part of the same group & then toggle the visibility of that group in a Three.js scene?

Congrats for the great work!

I did run some captures in WebXR with my Quest3 and I was surprised it runs! It's smooth but get slow when looking at detailed objects in the scene (main subject).

Is there a way to optimize the splat or add some options in the viewer to make the rendering faster for Mixed reality devices?

• Subsample the number of splats?
• Reduce the volume of the scene? (remove parts)
• Other optimization?

Thanks!

Hi, maybe noob question is it possible to use luma with webgpu? Trying threejs WebGPURenderer but luma fails to render with

Cannot read properties of undefined (reading 'isWebGL2')
    at R.onBeforeRender (@lumaai_luma-web.js?v=e24c7599:917:27)

Branch and code link: https://github.com/xleepy/threejs-app-sandbox/blob/webgpu-api/src/App.tsx#L17

Noticed this runtime log in Vercel. Crashes some clients using Splats.

Seems URL is not polyfilled?

I have a 3D Gaussian Splat output as a .ply file, generated from the original code accompanying the 3DGS publication. Is it possible to load that .ply into the LumaWebGL Renderer?

Default raycaster doesn't detect splat mesh.

This feature would be great for camera navigation tools like click to move. It's implemented in example below, so shouldn't be hard to add to this library.

Example: HOLLYWOOD

  useEffect(() => {
    Object.values(materials).forEach((material) => {
      material.depthTest = false;
      material.needsUpdate = true;
    })

    Object.values(nodes).forEach((node) => {
      node.renderOrder = 9000;
    })
  }, [materials, nodes])

I've tried stuff like this to no avail.

Hi!

When installing npm i @lumaai/luma-web aside [email protected] I got the following error:

Could not resolve dependency: npm WARN peer three@"^0.157.0" from @lumaai/[email protected]

This is because your package.json (https://www.npmjs.com/package/@lumaai/luma-web?activeTab=code) :

  "peerDependencies": {
    "@types/three": "^0.157.2",
    "three": "^0.157.0"
  }

Can you update it to "three": ">=0.157.0" or "three": "*"

For thoses who cannot wait, bad workaround could be npm i @lumaai/luma-web --force

Thanks!

Hi,

I tried installing luma webGL viewer in React 19 RC.

I got error with npm package:

 ⨯ node_modules\react-reconciler\cjs\react-reconciler.development.js (498:1) @ $$$reconciler
 ⨯ TypeError: Cannot read properties of undefined (reading 'ReactCurrentOwner')
 
 var StaticMask = LayoutStatic | PassiveStatic | RefStatic;
   497 |
>498 | var ReactCurrentOwner = ReactSharedInternals.ReactCurrentOwner;
            ^
  499 | function getNearestMountedFiber(fiber) { ...

What is the solution here? Seems like it's not fixable in this repo? What should I do to avoid this?

I've been experimenting with integrating splats from the lumalabs app into a threejs experience, and it's amazing! I found a splat editor online and would love to use this to edit my splats. It looks like I need to upload a file, and just provide a link to the capture as in the webgl library. I was wondering if there's a way to get the file from the luma app downloaded so I can edit it before importing it to my threejs app?

Thanks!

Hello.
I am enjoying using Luma AI.
I am posting this issue because I have tried the WebGL library for Luma AI, but I have encountered some problems.

I wrote code based on three.js and the Luma WebGL example, but when I ran the code, nothing was rendered in the browser and I got a lot of errors in the console on the browser.
Below is the full text of the errors.

Uncaught TypeError: Cannot read properties of undefined (reading 'equals')
    at setBlending (three.module.js:23355:19)
    at Object.setMaterial (three.module.js:23382:6)
    at WebGLRenderer.renderBufferDirect (three.module.js:29149:10)
    at renderObject (three.module.js:29945:11)
    at renderObjects (three.module.js:29914:6)
    at renderScene (three.module.js:29777:41)
    at WebGLRenderer.render (three.module.js:29593:5)
    at animate (main.js:31:12)

I think it might be a Luma AI WebGL issue, since the problem did not occur with the normal three.js example. Any advice would be appreciated.

Here is some information about my environment

Code

code is here

package.json

{
  "name": "luma-app-vanila",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "preview": "vite preview"
  },
  "devDependencies": {
    "vite": "^5.0.0"
  },
  "dependencies": {
    "@lumaai/luma-web": "^0.1.14",
    "three": "^0.159.0"
  }
}

Node.js version

$ bun run node -v
v20.5.1

Package Management Tools

$ bun --version
1.0.7

LumaSplatsThree git version?

(This information is output to the browser console)

LumaSplatsThree git version #8f3367e

This is my unsuccessful attempt:

https://glitch.com/edit/#!/guttural-navy-lentil?path=index.html%3A14%3A9

Is there any basic example I can look at? Thanks

I am getting this error when trying to use Luma splats with React Three Fiber

luma-web.module.js:60 Uncaught TypeError: this.material.updateUniformsAndDefines is not a function
    at a.onBeforeRender (luma-web.module.js:60:354411)
    at renderObject (three.module.js:29444:1)
    at renderObjects (three.module.js:29434:1)
    at renderScene (three.module.js:29303:1)
    at WebGLRenderer.render (three.module.js:29121:1)

I am following the example you have exactly here

Any idea what might be causing this error?

how would one go about raycasting the splat?

I am trying to get the 3D position of a mouse click inside of the LumaSplats mesh.

This is part of the code that I have written:

  function getClicked3DPoint(evt: any) {
    // evt.preventDefault();
    const canvasPosition = canvasRef.current?.getBoundingClientRect();
    console.log('WTF', scene.children)

    mousePosition.x = ((evt.clientX - canvasPosition!.left) / canvasPosition!.width) * 2 - 1;
    mousePosition.y = -((evt.clientY - canvasPosition!.top) / canvasPosition!.height) * 2 + 1;

    rayCaster.setFromCamera(mousePosition, camera); //cameraRef.current!);
    let intersects = rayCaster.intersectObjects(scene.children, true);

    if (intersects.length > 0)
        console.log(intersects[0].point)
        return intersects[0].point;
  }

      <Canvas
        ref={canvasRef}
        frameloop={frameLoop}
        gl={{
          antialias: false,
        }}
        camera={{
          position: [0, 0, 0],
        }}
      >
          <group
            dispose={null}>
            <lumaSplats
              onPointerMissed={() => console.log("you clicked, but hit nothing at all")}
              onClick={(e) => {
                console.log(e.point) // this is the coordinates for the event, eg. mouse click
                getClicked3DPoint(e)
              }}
              onLoad={() => {
                setIsLoading(false);

                if (camera360Ref.current) {
                  camera360Ref.current.object.position.x = -5;
                  camera360Ref.current.object.position.y = 0.4;
                  camera360Ref.current.object.position.z = 7;
                }
              }}
              loadingAnimationEnabled={true}
              semanticsMask={LumaSplatsSemantics.ALL}
              // onBeforeRender={}
              enableThreeShaderIntegration={false}
              particleRevealEnabled={true}
              source={src}
              position={[0, 0, 0]}
              scale={5}
              rotation={[0, 0, 0]}
            />

            <CustomLine start={[0,0,0]} end={[1,1,0]} />
          </group>
     </Canvas>

I have created the following luma capture: https://lumalabs.ai/capture/71f64429-3872-417a-a67c-a8b6c6ab16d3

As you can see it is a capture of a normal room. What I want to achieve is to draw lines on the splat. For this, I want to click on the canvas, this will cast a ray to the first intersecting part of the splat. I will take the coordinates of this intersection and then by clicking on another part of the splat I want to draw a line between these two points.

However, most of the times the event on onPointerMissed is being activated and thus no intersecting point is found. Do you have any idea why this is happening and what I can do in order to fix this?

Hello, I am trying to play around with the new luma library. However, when I try to load a model the fps drops a lot and the website pretty much becomes unusable. I have the following code:

import { Object3DNode, extend } from '@react-three/fiber';
import { LumaSplatsThree } from "@lumaai/luma-web";

// Make LumaSplatsThree available to R3F
extend( { LumaSplats: LumaSplatsThree } );

declare module '@react-three/fiber' {
  interface ThreeElements {
    lumaSplats: Object3DNode<LumaSplatsThree, typeof LumaSplatsThree>
  }
}

import { LumaSplatsSemantics } from "@lumaai/luma-web";
import React from 'react';
import './LumaSplatsReact';

export const LumaModel = () => {
	return <>
		<lumaSplats
			semanticsMask={LumaSplatsSemantics.ALL}
            // onBeforeRender={}
            enableThreeShaderIntegration= {false}
			source='https://lumalabs.ai/capture/ca9ea966-ca24-4ec1-ab0f-af665cb546ff'
			position={[1, 0, 0]}
			scale={2}
			rotation={[0, 0, 0]}
		/>
	</>;
}

import React, { useRef, useState } from 'react';
import {OrbitControls, PerspectiveCamera} from '@react-three/drei';
import { Camera, Scene, WebGLRenderer } from "three";
import { OrbitControls as OrbitControlsStdLib } from 'three-stdlib';

export type DemoProps = {
	renderer: WebGLRenderer,
	scene: Scene,
	camera: Camera,
	controls: OrbitControlsStdLib,
}

export const ModelViewer = (props: {
	modelReactFn: React.FC<{}> | null,
}) => {
	let controlsRef = useRef<OrbitControlsStdLib | null>(null);

	return <>
		<PerspectiveCamera />
		<OrbitControls
			ref={controlsRef}
			autoRotate={false}
			autoRotateSpeed={0.5}
			enableDamping={false}
            keys={{
                LEFT: 'ArrowLeft', //left arrow
                UP: 'ArrowUp', // up arrow
                RIGHT: 'ArrowRight', // right arrow
                BOTTOM: 'ArrowDown' // down arrow
            }}
			makeDefault
		/>
		{props.modelReactFn && <props.modelReactFn/>}
	</>
}

And I am trying to load the model onto a canvas like this:

                                    <Canvas shadows={false} 
                                        camera={{
                                            position: [-6, 7, 7],
                                        }}>
                                            <ModelViewer
                                                key={0}
                                                modelReactFn={LumaModel}
                                            />
                                    </Canvas>

Can you point me into exactly why my website is slowing so much? Because from your demo I can see that it is indeed possible to load the model and still have a good website performance.

Thanks in advance!

First of all, thanks for this great library and docs! Two quick questions:

• Could you implement better z-depth based occlusion for multiple splats in the same scene? Might not be the simplest feature since the whole renderer/rasterizer projects transparent ellipsoids onto the camera plane. But since the color rendering is so good, blending ellipsoids z-depth with the same transparency blending could lead to a pretty correct z-depth for the splat - which would therefore lead to correct depth-fighting for multi-splats scenes, which is atm pretty terrible.
• Will the LumaSplatsWebGL/LumaSplatsThree libraries themselves be open-sourced at some point?

I have given it a try and created an online Luma SPLAT Viewer using this library and three.js r159.

It currently only supports URL loading and it does work when using Luma URLs, like this:

• https://lumalabs.ai/capture/ca9ea966-ca24-4ec1-ab0f-af665cb546ff

The issue can be seen in the attached picture below when I use a URL like this:

• https://raw.githubusercontent.com/GitHubDragonFly/GitHubDragonFly.github.io/main/viewers/examples/legobrick.splat

The library is treating this URL as a "folder", or a location that hosts multiple files, and is trying to get 1 bin and 3 json files out of it.

If I try to use the WebGL Gaussian Splat Viewer then it works, and the following URL can just be entered in a browser:

• https://antimatter15.com/splat/?url=https://raw.githubusercontent.com/GitHubDragonFly/GitHubDragonFly.github.io/main/viewers/examples/legobrick.splat

This legobrick.splat file is just a lego brick made of points and was created from a ply file which was converted to splat by using the above mentioned WebGL Gaussian Splat Viewer.

The library also does not seem to be able to handle local blob URLs either, created with URL.createObjectURL( splat_file ) , which is another issue.

I might be doing something wrong all together but just looking for a second opinion.

Hello,
Thank you for this library, gaussian splatting are progressing fast !

It seems that this repository lacks a proper mention of the licence applied. Could you specify it in the README and add a LICENSE file so that we know if it is opensource or not ?

The licence of the code can be different from the terms and conditions of the Luma service of course.

I have a working project using a Luma splat URL, but I can't make it work with any of the downloaded files (.ply, .luma, .splat)

source can be either of:

• URL to a capture on lumalabs.ai
• path to a luma splats file or folder containing a luma splats artifacts

Am I doing something wrong?