Git Product home page Git Product logo

openiss / openiss Goto Github PK

View Code? Open in Web Editor NEW
22.0 7.0 21.0 83.79 MB

OpenISS -- a unified multimodal motion data delivery framework.

License: Apache License 2.0

Makefile 1.39% Shell 3.98% C 50.26% C++ 5.39% GLSL 0.22% Java 32.12% HTML 2.20% JavaScript 3.46% Dockerfile 0.08% Python 0.90%
iss kinect artists media linux osx windows siggraph web-services vfx wrappers nuitrack motion-capture library

openiss's Introduction

OpenISS

CII Best Practices

OpenISS is a motion capture data aggregation and delivery framework for VFX that has library instances abstracting various middleware and cameras for many application types.

Applications include AI+Art, performing arts, interactive film, facial animation, NUI applications using OpenGL, Processing, TensorFlow, Keras, and MARF.

The OpenISS framework provides a uniform abstraction layer over libfreenect, libfreenect2, librealsense, OpenNI2, NiTE2, NuiTrack, Azure SDK, PCL, and ROS and has REST and SOAP web services. There are different front-end modules as well based on OpenGL, Processing, and web browsers (JS).

The public GitHub updates are done periodically from private forks where a lot of integration / contributions happen.

History

OpenISS is inspired by the development of ISSv1, ISSv2, ISSv3 and MultiCamTk++, but is being built from scratch using C/C++ and various wrappers and uses similar or same open-source libraries, middleware and toolkits for sensors and creative coding. Various wrappers are being developed. See background on the inspirational ISS below. OpenISS API is likewise for the first time made it possble to access Kinect 1 and 2 as well as OpenCV as REST and SOAP services for creative near-realtime online broadcasting. OpenISS is poised to be the core replacement for ISSv2's pipeline. It also serves as an educational tool for graduate and undergraduate students in computer vision, computation arts, pattern recognition, AI, machine learning, and game development. It is designed to be portable.

  • OpenISS component development began in 2016 in C
  • Expanded to C++ and Java since and ongoing

Contributors

Current

  • Kyle Ross, Unity plugin and related rigging
  • Konstantinos (Kosta) Psimoulis: Web Services lead (REST and SOAP), SimpleOpenNIRS/liblrealsense2 support
  • Jonathan Llewellyn: actvity/action recognition, VFX pipeline
  • Dominique Cartier, point clouds integration and object/surface reconstruction

Recent

  • Jashanjot Singh: Gesture and Hand Tracking Framework, NiTE2/NuiTrack, and ROS support, SWIG support for Java/Processing
  • Haotao (Eric) Lai: Person Re-identifcation, Green Screening, device abstracion (Kinects 1 and 2, RealSense D435), TensorFlow/Keras/Python abstractions; Kinect 2 web service
  • Yiran (Bernie) Shen: Facial Recognition and data (landmarks and expressions) provider framework abstraction over OpenFace, OpenCV, including TensortFlow for VFX
  • Yuhao Mao: Art and AI, Sound Visualization, Style Transfer
  • Chao Wang: Style Transfer, Magenta support, motion-based audio support, finger tracking
  • Zihao Song: Singnal Processing for Audio and Image Processing effects and MARF

Web Services

  • See here for more documentation.

Early C API and Linux Builds

  • See detailed list here.

  • Updates for Linux and C in EL6 (CentOS 6.x), CSI230-101 Fall 2017 course students teams:

    • Calum Phillips, Rosser Martinez, Matthew Roy
    • Alex Rader, Cory Smith, Nicholas Robbins
  • Original build automation contributors for Linux in EL6 (CentOS 6.x), CSI230-101 Fall 2016 course students teams:

    • Brian Baron, Colin Brady, Robert Gentile
    • Gabriel Pereyra, Justin Mulkin, Duncan Carrol, Lucas Spiker

Build Instructions

Docker

  • We are working on our Docker images here

Ubuntu

  • Please use cmake
  • More documentation to follow

MacOS

  • Please use cmake
  • More documentation to follow

EL7

EL7 (RHEL, CentOS, Scientific Linux) are as of September 2018 default and preferred build, so the development effort focuses around this platform, but it is known to run on macOS and Ubuntu.

EL6

EL6 is now considered legacy.

Notice, EL6 requires a newer kernel for proper USB3 and NVIDIA support. If you prefer to install dependencies manually, you can follow the scripts referenced here (build.sh and el6.sh) and repeat their relevant steps one by one.

Install:
		yum install git
		git clone https://github.com/OpenISS/OpenISS.git
		git submodule update --init --recursive
		Navigate to OpenISS/src
		Run the command:
			make deps

		This will install all of the 3rd party
		dependencies

Compile:	
		cd OpenISS/src
		make

Clean:		
		Navigate to OpenISS/src
		Run the command:
			make clean

		Removes the installed dependencies

TODO

Build Support

  • OS X (mostly there)
  • Windows (partially there for some components)

Wrappers

  • Java (in progress)
  • C++ (mostly there)
  • Processing (in progress)
  • Max (in progress)
  • PureData (in progress)
  • Python (started)
  • ...

Background on the ISSv1, ISSv2, ISSv3

OpenISS’s core goals are to enable achieving something akin to the original ISS below in an open educational setting. Then, to build custom applications based on it, using OpenISS API as a core.

Academic Publications

Videos

References and Acknowledgements

openiss's People

Contributors

alex-free avatar alexrader avatar amirali-shirkh avatar cjs937 avatar colin-brady avatar dependabot[bot] avatar g-abinad avatar innagta avatar jashanj0tsingh avatar kostapsimoulis avatar laihaotao avatar maeghtesadi avatar navidgol avatar nrgs avatar paulpalmieri avatar person124 avatar rngbob avatar rossermartinez avatar smokhov avatar tgpx31 avatar thanasibab avatar theebz95 avatar vgkanri avatar yasminec avatar zephix109 avatar zerolrs avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

openiss's Issues

RQ-OPENNI2

Enable OpenNI2 skeleton data collection in EL6 Linux. Write/port a C app that illustrates it by visually drawing skeletons of captured subjects.
While libfreenect itself can be built using stock cmake and gcc 4.4 on EL6, enabling building OpenNI2 driver requires cmake3 and 4.8+ gcc from devtools-2. See a related examples here.

     wget -O /etc/yum.repos.d/slc6-devtoolset.repo \
         http://linuxsoft.cern.ch/cern/devtoolset/slc6-devtoolset.repo
     yum install devtoolset-2-gcc-c++
     scl enable devtoolset-2 bash
     cmake3 ... -D...OPENNI2...=ON -L ..

RQ-DOC

Describe your documentation, achievements, limitations, progress to date, experience, future work, bugs, etc. in doc/CSI230F17.md

RQ-IMG-DRV-GL

Implement one effect wrapper that uses that the issimgdrv and a /dev/urandom as a source.

typo in the code.

in src/api/java/openiss/utils/OpenISSImageDriver.java
lines 137-144 there is logical error
looks like the line 139 should be: int height = image_1.getHeight();
please validate and fix if needed.

below is the original code:

        // check height and width
        int width = image_1.getWidth();
        int height = image_2.getHeight();

        // check equal size
        if(width != image_2.getWidth() || height != image_2.getHeight()) {
            throw new IllegalArgumentException("dimensions are not equal.");
        }

RQ-MIC-INPUT

Use microphone input based VFX sample to alter graphics being displayed
based on sound processing similar to the above.

Arch: arch.sh

Similarly to el6.sh, or #17 or #16 -- create an arch.sh dependencies script and make proper integrations with Makefile and build.sh

RQ-GL-VFX-PIPELINE

Complete the OpenGL pipeline profile with OpenGL as many samples as possible from ogl, openFrameworks, opencv.

[CI] add Ubuntu builds

With or without #16, it'd needed to test Ubuntu as well as we are getting more Ubuntu deployments now.

p5.js

Both Processing and Magenta rely on these to render in the browser.

RQ-FAKENECT-NET

Integrate libfreenect’s fakenect to replay Kinect videos and test with the OSC by having a single receiver process and two sender processes – either two fakenect senders, or one real and fake, or two ore more real senders.

ROS

Look into the ROS project and any suitable integrations.

Java: need to port wrappers

  • for libfreenect and libfreenect2
  • for Processing a-la OpenKinect-for-Processing and SimpleOpenNI
  • OpenCV has built, it need to unify export with libfeenect*
  • ofx and others later to see what they export

[C][driver] to build proper driver for kernels on UEFI requires signed code

RQ-OSC

There are two conceptual components to this part.
They will communicate over UDP-based OSC protocol using sockets.
Start with a C implementation of OSC (partial), that is based on TinyOSC that is already a submodule of OpenISS:
https://github.com/mhroth/tinyosc
but embedded into our project.
The purpose to separate skeleton capture and rendering into different computers possibly running Kinect sensors.
One computer will act as a collector and render graphics based on skeleton information from two or more computers.
(a) Using tinyosc, implement issoscsender and issoscreceiver processes with a simple pro- tocol
(b) If skeleton data are available from either libfreenct or libfreenect2 (via OpenNI2 that both have), try capturing real data and sending it over and drawing a skeleton in an OpenGL window. It is OK to re-use an appropriately licensed open source example for this purpose.
(c) OSC protocol messages would look like /player 1 /left-hand /x 1 /y 2 /z 3 etc.

RQ-OFX-VFX-PIPELINE

Complete the sample openFrameworks pipeline profile with as many samples as possible that are non-OpenGL, but related to video, image, etc. processing.

EL7: el7.sh

Akin to el6.sh, we need el7.sh with proper mods elsewhere.

RQ-WEB-CAM

Instead of Kinect, use a web cam for color image processing with OpenCV. Apply any related examples from OpenCV or openFrameworks using our VFX API.

Ubuntu: ubuntu.sh

Akin to el6.sh, we will need ubuntu.sh and the corresponding mods to build.sh and Makefile.

RQ-IMG-DRV

  1. Location should be src/api/c/issimagedrv/ instead of src/api/c/quiz3/.
  2. Augment OpenISS’s build.sh/el6.sh/Makefile as necessary (target(s), option(s)) to install and configure the driver and its Kernel devel dependencies.

OS X: osx.sh

Please create osx branch off master. We need a script similar to scripts/dependencies/el6.sh, but osx.sh to install all required dependencies to build OpenISS submodules and itself. Not urgent. You would figure out what is needed based on #14.

RQ-OPENCV-VFX-PIPELINE

Complete the OpenCV pipeline profile with as many samples as possible that are non-OpenGL, but related to video, image, etc. processing.

OS X: test manual build

Folks, please test see if you are able to rebuild the submodules of OpenISS (libfreenect, libfreenect2, ogl, opencv, tinyosc, openFrameworks) on OS X manually or using their build instructions/scripts Try running their examples. Report progress here. This is NOT urgent.

RQ-MULTI-KINECT

Allow two or more Kinects collecting data simultaneously, and pumping it it to a single collector process. This is a multiprocess application.
See a sample design here: https://doi.org/10.1145/2945078.2945131 that was based on libfreenect2 and Protonect

RQ-SOUND

Locate a library/sample preferably in C or worst case in C++ that can read sound files and play them back. Implement a function of OpenISS that could load and play the specified file either as a subprocess or as an idle call (called from the idle callback when animation is done), such that audio data, such as spectrum or amplitude be available to the VFX to generate sound-based visualizations.

openFrameworks has some sound processing in it; check the samples.

RQ-FAKENECT-FORK

  1. Test and run the recording and replay tools in fakenect-record, and the corresponding driver
    library (see documentation of libfreenect on their GitHub front page).
  2. First, use the dataset provided ISSv2-fakenect-record in one or more instances of your process.
  3. Record at least two datasets from the lab movements 1 minute each, one per team.
  4. Write a simple test app that fork()s two sub-processes that playback the recorded content from each data set concurrently.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.