Git Product home page Git Product logo

opendronemap / nodeodm Goto Github PK

View Code? Open in Web Editor NEW
206.0 23.0 149.0 13.12 MB

A lightweight REST API to access aerial image processing engines such as ODM or MicMac

Home Page: https://github.com/OpenDroneMap/NodeODM/blob/master/docs/index.adoc

License: GNU Affero General Public License v3.0

JavaScript 94.02% CSS 1.64% HTML 3.02% Python 0.35% Shell 0.81% Dockerfile 0.14% Batchfile 0.01%
drones photogrammetry api

nodeodm's Introduction

NodeODM

Build Status

NodeODM is a standard API specification for processing aerial images with engines such as ODM. The API is used by clients such as WebODM, CloudODM and PyODM. This repository contains a performant, production-ready reference implementation written in NodeJS.

image

Getting Started

We recommend that you setup NodeODM using Docker.

  • From the Docker Quickstart Terminal (Windows / OSX) or from the command line (Linux) type:
docker run -p 3000:3000 opendronemap/nodeodm
  • If you're on Windows/OSX, find the IP of your Docker machine by running this command from your Docker Quickstart Terminal:
docker-machine ip

Linux users can connect to 127.0.0.1.

  • Open a Web Browser to http://<yourDockerMachineIp>:3000
  • Load some images
  • Press "Start Task"
  • Go for a walk :)

If the computer running NodeODM is using an old or 32bit CPU, you need to compile OpenDroneMap from sources and setup NodeODM natively. You cannot use docker. Docker images work with CPUs with 64-bit extensions, MMX, SSE, SSE2, SSE3 and SSSE3 instruction set support or higher. Seeing a Illegal instruction error while processing images is an indication that your CPU is too old.

Building docker image

If you need to test changes as a docker image, you can build easily as follows:

docker build -t my_nodeodm_image --no-cache .

Run as follows:

docker run -p 3000:3000 my_nodeodm_image &

Testing alternative ODM images through NodeODM

In order to test alternative ODM docker images in NodeODM, you will need to change the dockerfile for NodeODM to point to your ODM image. For example if you built an alternate ODM image as follows:

docker build -t my_odm_image --no-cache .

Then modify NodeODM's Dockerfile to point to the new ODM image in the first line:

FROM my_odm_image
MAINTAINER Piero Toffanin <[email protected]>

EXPOSE 3000
...

Then build the NodeODM image:

docker build -t my_nodeodm_image --no-cache .

Finally run as follows:

docker run -p 3000:3000 my_nodeodm_image &

Running rootless

  • A rootless alternative to Docker is using Apptainer. In order to run NodeODM together with ClusterODM in rootless environments, for example on HPC, we need a rootless alternative to Docker, and that's where Apptainer comes in to play. From the Linux command line, cd into the NodeODM folder and run the following commands to host a NodeODM instance:
apptainer build --sandbox node/ apptainer.def
apptainer run --writable node/ 

apptainer build --sandbox requires you to have root permission to build this apptainer container. Make sure someone with root permission build this for you. You will need to build this apptainer container if you want to work with ClusterODM on the HPC. Check for ClusterODM for more instructions on using SLURM to set it up.

An apptainer.def file can be built directly from the dockerfile as needed:

pip3 install spython
spython recipe Dockerfile &> apptainer.def

API Docs

See the API documentation page.

Some minor breaking changes exist from version 1.x to 2.x of the API. See migration notes.

Run Tasks from the Command Line

You can use CloudODM to run tasks with NodeODM from the command line.

Using an External Hard Drive

If you want to store results on a separate drive, map the /var/www/data folder to the location of your drive:

docker run -p 3000:3000 -v /mnt/external_hd:/var/www/data opendronemap/nodeodm

This can be also used to access the computation results directly from the file system.

Using GPU Acceleration

Since ODM has support for GPU acceleration you can use another base image for GPU processing. You need to use the opendronemap/nodeodm:gpu docker image instead of opendronemap/nodeodm and you need to pass the --gpus all flag:

docker run -p 3000:3000 --gpus all opendronemap/nodeodm:gpu

The GPU implementation is CUDA-based, so will only work on NVIDIA GPUs.

If you have an NVIDIA card, you can test that docker is recognizing the GPU by running:

docker run --rm --gpus all nvidia/cuda:10.0-base nvidia-smi

If you see an output that looks like this:

Fri Jul 24 18:51:55 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82       Driver Version: 440.82       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

You're in good shape!

See https://github.com/NVIDIA/nvidia-docker and https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker for information on docker/NVIDIA setup.

Windows Bundle

NodeODM can run as a self-contained executable on Windows without the need for additional dependencies (except for ODM which needs to be installed separately). You can download the latest nodeodm-windows-x64.zip bundle from the releases page. Extract the contents in a folder and run:

nodeodm.exe --odm_path c:\path\to\ODM

Run it Natively

If you are already running ODM on Ubuntu natively you can follow these steps:

  1. Install Entwine: https://entwine.io/quickstart.html#installation

  2. Install node.js, npm dependencies, 7zip and unzip:

sudo curl --silent --location https://deb.nodesource.com/setup_6.x | sudo bash -
sudo apt-get install -y nodejs python-gdal p7zip-full unzip
git clone https://github.com/OpenDroneMap/NodeODM
cd NodeODM
npm install
  1. Start NodeODM
node index.js

You may need to specify your ODM project path to start the server:

node index.js --odm_path /home/username/OpenDroneMap

If you want to start node ODM on a different port you can do the following:

node index.js --port 8000 --odm_path /home/username/OpenDroneMap

For other command line options you can run:

node index.js --help

You can also specify configuration values via a JSON file:

node index.js --config config.default.json

Command line arguments always take precedence over the configuration file.

Run it using PM2

The app can also be run as a background process using the pm2 process manager, which can also assist you with system startup scripts and process monitoring.

To install pm2, run (using sudo if required):

npm install pm2 -g

The app can then be started using

pm2 start processes.json

To have pm2 started on OS startup run

pm2 save
pm2 startup

and then run the command as per the instructions that prints out. If that command errors then you may have to specify the system (note that systemd should be used on CentOS 7). Note that if the process is not running as root (recommended) you will need to change /etc/init.d/pm2-init.sh to set export PM2_HOME="/path/to/user/home/.pm2", as per these instructions

You can monitor the process using pm2 status.

Test Mode

If you want to make a contribution, but don't want to setup OpenDroneMap, or perhaps you are working on a Windows machine, or if you want to run automated tests, you can turn test mode on:

node index.js --test

While in test mode all calls to OpenDroneMap's code will be simulated (see the /tests directory for the mock data that is returned).

Test Images

You can find some test drone images here.

What if I need more functionality?

NodeODM is meant to be a lightweight API. If you are looking for a more comprehensive solution to drone mapping, check out WebODM, which uses NodeODM for processing.

Contributing

Make a pull request for small contributions. For big contributions, please open a discussion first. Please use ES6 syntax while writing new Javascript code so that we can keep the code base uniform.

Roadmap

See the list of wanted features.

nodeodm's People

Contributors

bibekjoshisrijan avatar binarymason avatar bluecamel avatar daedroza avatar dbaldwin avatar frankmarazita avatar holycowmp3 avatar kauly avatar kikislater avatar lfwill avatar matthewberryman avatar mhupfauer avatar mojodna avatar nghi01 avatar oaydogmus avatar ooglely avatar pepperlk avatar peterburlakov avatar pierotofy avatar rrowlands avatar sean-kenops avatar smathermather avatar spatialhast avatar theoway avatar timic3 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nodeodm's Issues

Progress Reporting API

There should be a "progress" value reported back in the task info. That way we don't have to guess it from looking at the console output in client applications.

locale not supported

I had an issue when running ./configure.sh install on ubuntu xenial. The error was hidden, as the configure script did complete successfully.

Installing OpenSfM Dependencies
Traceback (most recent call last):
  File "/usr/bin/pip", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 215, in main
    locale.setlocale(locale.LC_ALL, '')
  File "/usr/lib/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
Installing Ecto Dependencies
Traceback (most recent call last):
  File "/usr/bin/pip", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 215, in main
    locale.setlocale(locale.LC_ALL, '')
  File "/usr/lib/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

After running ./run.sh foo I got an error from python not being able to import appsettings (the exact error is gone from my shell, and I cannot reproduce since I fixed it).

For others, the fix is running:

export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
sudo dpkg-reconfigure locales

Select LC_ALL, this may take some minutes, but for ubuntu it solves the issue.

Then run ./configure install again. Afterwards the import error was gone, and ./run.sh foo worked fine.

Chunked upload API

For very large datasets, it would be a good addition to have an alternative method to upload images, via chunks. If a connection is bad, the upload via a single POST request is not very feasible.

Return odm_orthophoto.png

It has been asked to provide odm_orthophoto.png in the list of assets by default.

This needs some thought, as there has to be a compromise between speed and availability of assets.

Add option to skip postprocessing

Perhaps a user is not interested in tiles and the potree output, in which case we should skip it.

It might be interesting to move the tiling / potree logic to a worker on WebODM, since it's mostly just time consuming, but not memory intensive.

Failed at Orthophoto building

I took the photos with my iphone5s. I have attached below the link to my data set.
https://drive.google.com/drive/folders/0B7tV_P5gZivFVU1FbHE1Mjk1Tkk?usp=sharing
I have also tried with other data sets i.e. from DJI Phantom 3 which worked well and results were good.
So I feel may be my iphone datasets doesn't have good overlay. But I have tried this data sets with Agisoft photoscan which could create dense cloud. My objective is to compare both the dense cloud generated from Agisoft Photoscan and WebODM.
So I will be glad if you could help..:-)

I have pasted below the error log..
Building objmodel:
Saving model... Building objmodel:
Saving model... done.
Whole texturing procedure took: 96.698s
[INFO] Running ODM Texturing Cell - Finished
[INFO] Running ODM Georeferencing Cell
[DEBUG] None
[WARNING] No coordinates file. Generating coordinates file: /var/www/data/64e26d38-86c4-4a3a-b234-246a5319e9d8/odm_georeferencing/coords.txt
[WARNING] Could not generate coordinates file. Ignore if there is a GCP file. Error: Error in OdmExtractUtm:
No valid GPS position found
For more detailed information, see log file.

[WARNING] Georeferencing failed. Make sure your photos have geotags in the EXIF or you have provided a GCP file.
[INFO] Running ODM Georeferencing Cell - Finished
[INFO] Running ODM Orthophoto Cell
[DEBUG] running /code/build/bin/odm_orthophoto -inputFile /var/www/data/64e26d38-86c4-4a3a-b234-246a5319e9d8/odm_texturing/odm_textured_model.obj -logFile /var/www/data/64e26d38-86c4-4a3a-b234-246a5319e9d8/odm_orthophoto/odm_orthophoto_log.txt -outputFile /var/www/data/64e26d38-86c4-4a3a-b234-246a5319e9d8/odm_orthophoto/odm_orthophoto.png -resolution 20.0 -outputCornerFile /var/www/data/64e26d38-86c4-4a3a-b234-246a5319e9d8/odm_orthophoto/odm_orthophoto_corners.txt
[INFO] OpenDroneMap app finished - Sat May 13 18:38:45 2017
PDAL: No type found for undefined dimension ID.

starting of server failed

Hello
I'm trying to run node-OpenDroneMap natively. When I run this command

node index.js --odm_path /home/wedevgroup/WebODM_natif/WebODM , I get this message

error: Error during startup: Could not load list of options from OpenDroneMap. Is OpenDroneMap installed in /home/wedevgroup/WebODM_natif/WebODM? Make sure that OpenDroneMap is installed and that --odm_path is set properly: Unexpected token T in JSON at position 0

Can someone help me to fix this problem ?
thank!

Syntax Errors. Please have a look or check if versions are okay?

nvidia@tegra-ubuntu:~/NodeODM$ node index.js --test
/home/nvidia/NodeODM/libs/S3.js:89
}, (2 ** file.retries) * 1000);
^

SyntaxError: Unexpected token *
at createScript (vm.js:56:10)
at Object.runInThisContext (vm.js:97:10)
at Module._compile (module.js:549:28)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
at Module.require (module.js:504:17)
at require (internal/module.js:20:19)
at Object. (/home/nvidia/NodeODM/libs/Task.js:33:12)
nvidia@tegra-ubuntu:/NodeODM$ nodejs -v
v6.17.1
nvidia@tegra-ubuntu:
/NodeODM$ npm -v
6.9.0
nvidia@tegra-ubuntu:~/NodeODM$

Can't launch node-OpenDroneMap

Hi,
After succesful achieved all the processes to launch node-OpenDroneMap locally , When I trie to launch: node index.js
this error occurs:
`home/user/OpenDroneMap-0.2/node-OpenDroneMap/libs/Task.js:37
constructor(uuid, name, done, options = []){
^

SyntaxError: Unexpected token =
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/home/user/OpenDroneMap-0.2/node-OpenDroneMap/libs/TaskManager.js:25:12)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)
`
Please help me.
Thanks.

How to get other output files in odm_orthophoto?

let allPaths = ['odm_orthophoto/odm_orthophoto.tif', 'odm_orthophoto/odm_orthophoto.mbtiles',

In this line I see that this is the default list of outputs to be compressed as "All Assets", however it is missing some other files such as odm_orthophoto/odm_orthophoto_corners.txt and odm_orthophoto/odm_orthophoto_log.txt.

Is there a way to specify this outputs list somewhere?

Troubles to run NodeODM node

Hello,

I have an issue and any help would be welcome. When i am trying to run the command docker run -ti -p 3000:3000 opendronemap/nodeodm. It seems that it connects to the server. The output is:
info: Authentication using NoTokenRequired
info: Listening on 0.0.0.0:6367 UDP for progress updates
info: No tasks dump found
info: Checking for orphaned directories to be removed...
info: Server has started on port 3000

but when i enter localhost:3000 in a web browser i don't observe anything.

I would be grateful to anyone for any help. It's urgent

Heartbeat checking

It would be good to add the ability to create a task and optionally use calls to /info as a “heartbeat” check for verifying that a client hasn’t abandoned a task.

Ground Control Points

Hi, this is just a question.
I noticed that the folder that the Ground Control Points are loaded into is called GPC.
I just wanted to know if this is correct (Ground Point Correction ???) or a typo?
Thanks so much

Add config to set a static callback

We already have the option to specify a custom callback URL for a new task. We should also have a callback option at the entire node level (so that it's the same for each task and doesn't need to be passed for each task).

Interpolation method

G'day Guys,

Love the work you are doing! not an issue but a suggestion, would it be possible to add bicublic interpolation as a method for the DEM, DSM, 3D Mesh and Orthomosaic creation??

and are you guys leveraging GDAL in this project??

[Improvement] Save settings file in all.zip

Would be great to save the settings/options for processing as a simple text file in the all.zip that can be downloaded once processing is complete. This way it can be easily traced what settings worked well an which didn't.

Make S3 uploads more resilient

Currently if a S3 upload fails, the task will fail. We should make the upload process robust to failure by attempting multiple times if uploads fail.

Add job timeout

Would be cool to be able to set a maximum runtime for a task, in the remote case that it gets stuck.

Expose ODM version

We should read /code/VERSION and expose it through the API. Otherwise we don't know what version we are running.

Invalid configuration file config-default.json error

Getting error when attempting to run in Docker. Builds w/o errors.
$ docker run --rm -it --name node-odm-001 gei/opendronemap:nodeodm-nocuda-opencv-3.3.1-amd64.debian-stretch
Invalid configuration file config-default.json

Been through the json file, nothing seems out of kilter. ODM builds and runs fine.

Cancelled does not kill process

When a task is cancelled the underlying process keeps living. We end up losing track of the process and all its logs even though its still running.

How to maximize available nodes

I'm running three node (each 4 core and 8GB RAM). It seems the load was not distributed to all available nodes. Am I missing some configuration?

node-0
node-1
node-2

Cannot access processing node after update

There's no template here, so I'll try to get everything. I'm using WebODM on windows 10, version 0.7.0. Also using WebODM on 16.04 Ubuntu Linux LTS, as an easy way to interface with its processing node from windows. My issue is that with the latest upgrade, the processing node running on the Linux box on port 3000 has stopped responding. Manual starts and stops of WebODM has not resolved the issue. Here is my start script, which is in /etc/init.d

[code]
#!/bin/sh
cd /home/kameron/WebODM
./webodm.sh start
docker run -p 3000:3000 -v /mnt/sdb1:/var/www/data opendronemap/node-opendronemap
#docker run -p 3000:3000 opendronemap/node-opendronemap
[/code]

I have been successful at starting WebODM through the console with no errors, but the node isn't connecting over the network and looking through the start script, I think this is the best place to ask this.

Task Run Fail

I pulled the docker image and built it locally and I keeps getting the same output..

/code/run.sh: line 6:    21 Illegal instruction     (core dumped) python $RUNPATH/run.py "$@"

Image import

Would be nice to import images from a git/zip url.

Work with ODM docker

The document says we can specify the path or ODM using --odm_path parameter. What if I am using ODM as docker? Is it possible to have node-OpenDroneMap work with an ODM docker? Thanks!

Enhancement for log time

Raised as enhancement as advised: Would it be possible to allow the rolling log in the WebODM GUI to be configurable or to show System/Server Time? i.e My server is set to BST (GMT +1) but the output shows GMT or BST -1.

faff6876-2919-11e7-974d-b565513ef967

change rename calls to move (so that volumes can be mounted)

When running using the node container within a docker webodm installation, I'm having issues after uploading images where jobs are failing because of the lack of disk space. where the processing node is filling up after a 2nd or third job that the docker volume this container it's using is filling up and is capping out at 10GB.

I have the web-odm containers all running on a centos 7.4 docker host. The underlying docker host has plenty of Disk space - 500GB where I've allocated majority of this to /var/lib/docker.

I'm using the images that are pulled from docker hub.

To get around this I've tried exposing /var/www/data and /var/www/tmp directories on the node container to the underlying host, but get the following message

EXDEV: cross-device link not permitted, rename 'tmp/1ce8eb3d-ce5e-43bf-89d8-d22b92a54f3c' -> 'data/1ce8eb3d-ce5e-43bf-89d8-d22b92a54f3c/images'

however the man for the rename function would indicate that whatever is happening after I upload the images does not like this.

EXDEV  oldpath and newpath are not on the same mounted filesystem.
              (Linux permits a filesystem to be mounted at multiple points,
              but rename() does not work across different mount points, even
              if the same filesystem is mounted on both.)

so any suggestions on how to get around this would make my life a lot easier or to expand the node volume?

node-uuid depreciated error

$ npm install

npm install error:
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/nodemon/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
Removing intermediate container 48d312d65960

npm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
npm WARN deprecated [email protected]: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN prefer global [email protected] should be installed with -g
npm WARN prefer global [email protected] should be installed with -g

Revised node-uuid to uuid in package.json
$ npm install -g coffeescript jshint

docker run throws node-uuid missing error
added node-uuid with uuid, now runs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.