As an application developer, I want to be able to stream audio efficiently to Lava processes from a microphone attached to my PC.

Conditions

• A process called Microphone or StreamingMicrophone or something like that exists.
• The process can accept parameters for the device id, sampling rate, bit format, etc.
• A PyProcessModel for the process exists that will start recording audio using an existing, well-designed audio library such as PyAudio. The selected audio library must work on Ubuntu, and ideally should also work on Windows and Mac. Properly configuring the audio library may be considered out of scope for this issue, although it should not be excessively difficult to do so.
• The PyProcessModel should implement an appropriate buffering algorithm, such as pushing audio chunks into a non-blocking queue or ring buffer.
• When run on a standard x86 CPU, the process model should not use more than 10% CPU time on a single core, and no audio samples should be dropped.

Add a automatically generated API documentation.

In order to keep the repository small, we should start using LFS for large files and images.

User story

As a Lava user, I want to be able to use various input sources that are relevant for neuromorphic applications.

Conditions of satisfaction

• A Lava Process that reads clock-based audio from files is available.
• A Lava Process that reads clock-based RGB images from files is available.
• A Lava Process that reads clock-based RGBD images (captured with RealSense camera) from files is available.
• A Lava Process that reads clock-based audio from sensors is available.
• A Lava Process that reads clock-based RGB images from sensors is available.
• A Lava Process that reads clock-based RGBD images (captured with RealSense camera) from sensors is available.
• A Lava Process that reads event-based audio (captured with Inivation cochlea) from files is available.
• A Lava Process that reads event-based images (captured with Inivation retina) from files is available.
• A Lava Process that reads event-based images (captured with Prophesee retina) from files is available.
• A Lava Process that reads event-based audio (captured with Inivation cochlea) from sensors is available.
• A Lava Process that reads event-based images (captured with Inivation retina) from sensors is available.
• A Lava Process that reads event-based images (captured with Prophesee retina) from sensors is available.

Stories

• Lava Process for clock-based audio (files + sensor)
• Lava Process for clock-based RGB images (files + sensor)
• Lava Process for clock-based (RealSense) RGBD images (files + sensor)
• Lava Process for event-based (Inivation) audio (files + sensor)
• Lava Process for event-based (Inivation) images (files + sensor)
• Lava Process for event-based (Prophesee) images (files + sensor)

User story

As a Lava user, I want to be able to interface with ROS.

Conditions of satisfaction

• Users can subscribe to ROS topics and receive from it within Lava.
• Users can publish to ROS topics and send to it within Lava.

At the moment, instantiating a RealSense Process in the mode that reads from files, with the align_depth_to_bgr parameter set to True throws a NotImplementedError.

If this feature is to be supported, it needs to be implemented.
If not, a comprehensive error message should clearly state that this is not possible.

As a roboticist, I want to be able to launch a Lava process on my local machine which will connect to a Lava process running on a remote machine. The two processes will exchange data of a pre-defined format, e.g. numpy arrays, at each timestep.

Conditions of Satisfaction:

• When launched, the client process will attempt to initiate a connection to the server process. If the connection fails within a given time period, the client process will try again up to a certain number of times, or indefinitely. Thus, the client process will effectively wait until the server process is available. The client process may either block the rest of the process graph until a connection is made, or allow the rest of the process graph to continue unaffected.
• When launched, the server process will listen on a given port / network address for client connections. The server process may either block until a connection is made, or allow the rest of the process graph to continue unaffected.
• When the client process successfully connects to the server process, it will send a pre-defined message format, consisting of e.g. a numpy array and it will also receive a pre-defined message format on each timestep.
• The above behaviors should be implemented by a client PyProcessModel and a server PyProcessModel. In the future, it may be possible to add additional process models such as a CProcessModel or an NcProcessModel.

Additional Comments

This feature request is created on request and a prototype will be implemented by Ronan Sangouard and @tim-shea.