Git Product home page Git Product logo

eulersson / taconez Goto Github PK

View Code? Open in Web Editor NEW
0.0 1.0 0.0 109.38 MB

Sound warfare against your upper neighbour's high heel walks. AI IoT Raspberry Pi powered solution to detect, analyse, plot and react against them.

License: Apache License 2.0

CMake 3.14% C 7.72% Python 8.51% Dockerfile 4.42% Shell 5.76% Jinja 0.51% Jupyter Notebook 56.46% CSS 0.52% TypeScript 6.96% JavaScript 0.20% PureBasic 3.26% MDX 2.55%
annoying iot neighbours ai

taconez's Introduction

Warning

The development of the project is on hold, so it's in a WIP state. If you want to keep track of the process check its GitHub Project.

taconez

Taconez Logo

Sound warfare against your upper neighbour's high heel walks.

AI IoT Raspberry Pi powered solution to detect, analyse, plot and react against discomforting sounds, by bouncing them back to their source.

Motivation

My neighbour upstairs decided to walk on high heels. I expressed my discomfort and she took it as a personal attack, rejecting all kind of mediation. Since I received insults I decided to use the mirror strategy to see if she realizes how unpleasant it becomes: to bounce back her sounds.

Progress

To see the progress and development check out its GitHub Project.

Quickstart

This obviously has to be deployed to Raspberry Pis, automated provisioning and deployment exists and there's this wiki page is dedicated to it.

But you can simulate the stack locally with Docker Compose! On your host you will need Docker, Docker Compose and PulseAudio.

# Clone the repository
git clone https://github.com/eulersson/taconez.wiki.git

# Make sure all the submodules (third party dependencies) are installed:
git submodule update --init --recursive

# Run a pulse audio server on your host.
brew install pulseaudio
brew service start pulseaudio

# Configure your PulseAudio sound server
pactl load-module module-native-protocol-tcp # allows processes to speak to audio server via TCP
pactl list sinks short # see your sound output devices
pactl set-default-sink <sink-name> # set your default output device
pactl list sources short # see your sound input devices
pactl set-default-source <sink-name> # set your default input device

# Start listening for sounds and bouncing them back upon detection.
docker-compose up

Then you can follow the wiki's Development Workflow.

Description

This project is composed of various pieces:

Module Purpose
Sound Detector Discriminate and pick up specific sounds using AI.
Playback Distributor Route the detected sound and bounce it back from multiple speakers on multiple locations.
Sound Player Play any given sound when told by the distributor.
Journal Web App Visualize the sound occurrences, monitor them and replay them back to your neighour. Useful to have an objective overview. If case gets to lawyers it might be useful.

Architecture

Architecture Diagram

Technology

This is intended to be run in a group of Raspberry Pi

Tool Purpose Why (Reason)
TensorFlow (Python) Audio detection. Provides even a higher level API (Keras) that simplifies machine learning workflows. Also provides a lightweight runtime (TF Lite) for running only inference (feeding the input and getting the output on a saved model).
Next.js (JavaScript) Data visualization web application. Allows splitting a React application between server and client components (full-stack framework), allowing you to dispatch the database queries from the server and return a rendered React component with all the data displayed, encapsulating database interaction only server-side. Allows writing REST-style API endpoints that can be listening to the sound detector.
PulseAudio (C language) Sound server. Allows using TCP to comunicate with the host audio sound server from the container. I wanted thin containers without complex sound libraries and a low level language was suitable for such simple modules.
InfluxDB Noise occurrence log. A time-series database seems very appropiate considering the nature of the data (timestamped). Chosen in favour of Prometheus because it supports string data types and it's PUSH-based instead of PULL-based. We don't want to lose occurrences!
NFS Sharing a volume with the audio data. The database should not get bloated with binary data. Once the audio file is producted, it gets hashed it and stored in the NFS-shared file system so it can get played by the sound-player.
ZeroMQ Communication between audio detector and playback receivers. Instead of having to implement an API, since it's only one instruction, it's simpler to use a PUB-SUB pipeline in the fashion of a queue. The detector places an element and all playback receivers react playing back the sound. A full queue broker installation might be overkill for a simple IoT communication channel.
Docker Containerization. Protects against underlaying operating system components that might get updated and break the app.
Ansible Provisioning and deployment. Automates some base configuration installation on new Raspberry Pi hosts, such as the sound setup.

Technology: Raspberry Pi, Figma, Ansible, Docker, Docker Compose, InfluxDB, Next.js, React, Storybook, Jest, Tailwind CSS, Framer Motion, Python, TensorFlow, C, CMake, PulseAudio, Unity (C Testing Library), cJSON.

Documentation

If you are only interested in the provisioning and deployment of the architecture:

Document Explains How To...
Provisioning & Deployment Flash a new card, plug into a Raspberry Pi and provision it as master or slave and deploy the project
Raspberry Pi Sound Setup Connect multiple Bluetooth speakers, create a combined sink of various speakers, set up microphone.

Further information on how to get a development environment and how the issues of running container sound on the host (be it a MacBook or a Raspberry Pi):

Document Explains How To...
Docker Container Sound Offer host audio I/O to the container through PulseAudio's TCP interface.
Development Workflow Develop the application container on a non-Pi host with keeping the LSP IDE features and still be able to use the microphone and speakers through PulseAudio.

Other documentation:

Document Explains How To...
Stack Recipes Run hello worlds, proof of concepts and basic examples with the stack tools: InfluxDB, NFS and Ansible.
AI: Transfer Learning Repurpose an already trained classification neural network to classify specific sounds.
Modes of Operation The different ways you can run the sound detection (observing (stealth) mode, fight mode, multiclass YAMNet, binary classifier retrained network...).

Coding Standards

Following Google Style Guides.

taconez's People

Contributors

eulersson avatar

Watchers

 avatar

taconez's Issues

Explore container development

Explore the following ideas:

  • First try to get it to work using VScode DEV Containers.
  • To use Neovim exrc and per-project lua files to specify the language server container to hook into.
  • Ideal world would be to have something like VSCode Containers.
  • See if perhaps installing the development tools (Neovim, config files, etc) in the container might be better or not. Since container CPU and memory might be thinner the UX can drop perhaps?

RPi InfluxDB POC

Check it works on Raspberry Pi:

  • Service starts on restarts.
  • You can write an entry with Python.
  • You can read an entry with JavaScript (little NodeJS app).

Provisioning Sound Detector (Ansible)

Provide and document a deployment model for the app.

TODO:

  • Figure out how to run individual tasks from the playbook using the CLI. Maybe using tags?
  • In the host the project files have eulersson:eulersson permissions. See with the posix.synchronize module how they can end up on the target host with anesowa:anesowa permissions. Maybe skipping the archive flag?
  • Use the template module to parametrize the service name.
  • See if PulseAudio stuff can be automated!
  • Upon copying over restart the services (Ansible handler), the service should build the container and run it!
  • Document. Also change the Development workflow section since now instead of rsync or volume binding the ansible playbook could be used.
  • Set default volumes to 100%
  • host vars have an array to iterate over all entries and set

Distribution to be aware of all the master and slaves

It can be useful to know how many players do we have connected.

See where is the most convenient way to store it, I would say into a file somewhere? Or what other kind of registry is worth using for storing persistent state otherwise? For now use a file.

Explore an interactive way to prompt the user to choose a pa source

The ansible deployment also configures what recording source is going to be used. This is usually written down in vars. But it would make it more user friendly to present it as an interactive prompt. Specially because this step is critical!

Here's an example on how this can be done:

- name: Get list of numbers from foo
  command: foo
  register: foo_output
  changed_when: false
  check_mode: no

- name: Extract numbers from foo output
  set_fact:
    numbers_list: "{{ foo_output.stdout_lines }}"

- name: Display options to user and save preference
  block:
    - name: Prompt user to choose a number
      pause:
        prompt: "Please choose a number from the list: {{ numbers_list }}"
        echo: yes
      register: user_choice

    - name: Save preference for chosen number
      command: bar "{{ user_choice.user_input }}"
      changed_when: false
      check_mode: no
  when: numbers_list | length > 0

Choose the most meaningful detection

This is to be done with the actual system that gives you scores and not with a transference learnt neural network.

You have a top_class_name and a list of predictions.

From that build a list with the class names and do not include there the ones to be ignored.

Grab the first occurrence and use it as a suffix.

TF Sample: Train macOS & Run RPi

Play a preroll sound before bouncing the sound back

When a sound is detected and bounced back to the annoying neighbour we need to be clear that the neighbours knows this sound is produced from her, that's why adding a preroll sound can make it easier: "Your sounds are disturbing!!! ".

For that it will be the playback distributor the one in charge to pick one preroll sound and hand it to all connected sound players. It should be picked randomly based on the current time.

Then the player module needs to first play the selected preroll and then the sound.

Configure deployment secrets

It might be worth only if you share the home LAN with people you don't trust, otherwise the influx DB service for instance could be reached and compromised.

Investigate a solution to centralize logs

Explore if we can use the docker logs, how long they stay for, if we can use a logging driver.

As a learning exercise Grafana would be a suitable choice?

For an IoT project this might be clugging the small powers of the Raspberry Pis...

We could parametrize Grafana installation in the ansible playbook so at least I still learn it and don't discard it for the simple fact of "it would consume too much power".

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.