The goal of this lab is to develop a video doorbell with WebRTC and Python (aiortc) on a Raspberry Pi.

You will need:

• A Raspberry Pi with Wifi and storage card
• A supported Webcam such as the Logitech C505e
• A (virtual) machine to run the signaling server on
• Breadboard, jumper wires, a push button and 1kOhm resistor (see here)

For the initial installation of the Raspberry Pi you will need a monitor, keyboard and mouse.

The system to develop should work as follows:

• When the push button is pressed, the aiortc WebRTC implementation starts a video call with the signaling server.
• Ideally, a user should receive a message (SMS, Telegram, ...) with the link to the video call.
• Using the URL the user can connect to the video call with a browser.
• The user can talk with the visitor and quit the call at the end.

See here for installation instructions. You will need a monitor, keyboard and mouse as well as probably a micro-HDMI adapter (for the RPi 4 model).

Change the user password and enable SSH.

Then you can connect to the RPi through SSH without the need for a monitor, keyboard or mouse.

You can reuse the signaling server developed in the VMM-WebRTC-lab, without any modifications. Just deploy it on a (virtual) machine that is accessible from the Raspberry Pi. You also have to install the required packages:

sudo apt install python3-flask python3-flask-socketio

SocketIO has different protocol version, which may be incompatible. Therefore you have to check the installed version:

$ apt search socketio
Sorting... Done
Full Text Search... Done
python3-flask-socketio/focal,now 4.2.1-1 all [installed]
  Socket.IO integration for Flask applications

python3-socketio/focal,now 4.4.0-2 all [installed,automatic]
  python3 implementation of the Socket.IO realtime client and server

python3-socketio-client/focal 0.6.5-0.2 all
  socket.io-client library for Python3

Here, python3-socketio version 4.4.0-2 is installed. The client on the Raspberry Pi has to use a compatible version.

Ideally you will develop the WebRTC client in Python directly on the Raspberry Pi. We recommend using VS Code with a remote SSH connection to the RPi.

Start VS Code on your laptop and create a remote connection to the RPi by clicking on the red icon "Open a Remote Window" all on the bottom left of VS Code.

The WebRTC client is a Python 3 program running on the Raspberry Pi. It uses aiortc, which is a Python implementation of WebRTC, as well as SocketIO to communicate with the signaling server.

Install the Python packages aiortc and SocketIO:

python3 -m pip install aiortc
python3 -m pip install "python-socketio[asyncio_client]==4.6.1"

Here the SocketIO version is fixed to 4.6.1, which is compatible with the version 4.4.0-2 used by the server.

The overall structure of the WebRTC client is:

Loop forever:
  1. Wait until keypress (to be replaced later by the pushbutton press event).
  2. Connect to the signaling server.
  3. Join a conference room with a random name (send 'create' signal with room name).
  4. Wait for response. If response is 'joined' or 'full', stop processing and return to the loop. Go on if response is 'created'.
  5. Send a message (SMS, Telegram, email, ...) to the user with the room name. Or simply start by printing it on the terminal. 
  6. Wait (with timeout) for a 'new_peer' message. If timeout, send 'bye' to signaling server and return to the loop. 
  7. Wait (with timeout) for an 'invite' message. If timemout, send 'bye to signaling server and return to the loop. 
  8. Acquire the media stream from the Webcam.
  9. Create the PeerConnection and add the streams from the local Webcam.
  10. Add the SDP from the 'invite' to the peer connection.
  11. Generate the local session description (answer) and send it as 'ok' to the signaling server.
  12. Wait (with timeout) for a 'bye' message. 
  13. Send a 'bye' message back and clean everything up (peerconnection, media, signaling).

As a starting point, use the examples provided by aiortc, in particular the apprtc example and the videostream-cli example. You can also try the webcam example, which should work out of the box but is harder to adapt (the server runs on the RPi in this example).

The following sections provide some details for the steps above.

In a first step, you have to develop the signaling between the RPi client on the RPi and the Flask server. Follow the SocketIO examples to create a AsyncClient. Since the server uses a self-signed certificate, set ssl_verify=False, to allow the client to establish the connection.

Here is one difficulty that you have to overcome: SocketIO is callback oriented, such that a specific callback handler is called for each message type. However, the client has to wait simultaneously for different messages. E.g., after sending a 'create' message, the response may be 'created', 'joined' or 'full'. This is difficult to handle with SocketIO.

Here is how you can implement this behavior:

• aiortc uses the Python asynchoronous library asyncio.
• Write a function that creates a asyncio Queue.
• For each message type ('created, 'joined', ...), the function create a SocketIO message handler. The handler adds the received message to the Queue.
• The main program can then wait for a new message on the queue (queue.get()) and handle the message depending on the type.
• If you want to limit the waiting time for a new message, you can wrap the queue.get() in an asyncio.wait_for() function.

The Raspberry Pi has to handle several media streams:

• The Webcam (with built-in microphone) generates two streams: a video stream and and audio stream. They have to be transmitted over the peer connection.
• The remote peer sends and audio stream: it has to be send to the audio output (headphone connected to USB or the audio jack.)

Outgoing streams On the Raspberry Pi (Linux), the Webcam device is /dev/video0. See the aiortc Media sources for an example. You have to create a MediaPlayer for this stream. Use a small image size (320x240), otherwise the transmission will be very slow. You have to add the video track of the this MediaPlayer to the peerConnection.

Another MediaPlayer is required for the audio track. The device for this stream is 'default' and the format is 'pulse' (see aiortc). You have to add the audio track of this player to the peerConnection, too.

Incoming stream You will also have to handle the incoming audio stream. The aiortc videostream-cli example uses a MediaRecoder for the incoming stream. The MediaRecorder can write the stream to a file, but also to the audio output. Use 'default' as file name and 'alsa' as format. You should hear the incoming audio stream on the headphones.

Here is the second difficulty that you have to handle. Aiortc does not yet implement trickle ICE. The RPi client therefore won't generate ICE candidates and won't handle remote ICE candidates correctly. Without trickle ICE, WebRTC has to wait until the ICE agent has gathered all local candidates. It can then include them in the SDP (offer or answer) and send them to the peer.

Therefore you have to do the following:

In the RPi client

• Wait until an 'invite' is received (see overall structure above, step 7.)
• Get the SDP from the invite message.
• Use pc.setRemoteDescription to register the SDP.
• Use pc.createAnswer to generate an answer SDP.
• Use pc.setLocalDescription(answer) to register the local SDP.
• Very important: then re-read the answer using pc.localDescription. Only this SDP will contain the ICE candidates!

In the Javascript client (webrtcclient.js)

• When a 'joined' message is received:
  • use pc.createOffer to generate an offer SDP.
  • Use pc.setLocalDescriptionto register the local SDP. This will start the ICE agent.
  • Do not yet send and 'invite' message to the remote peer. Just wait.
• When a local pc.oncicecandidate event is received:
  • Do not send the candidate to the remote peer. Just ignore it.
  • If the candidate is empty, the ICE agent has finished. In this case:
    • Use pc.localDescription to get the offer SDP. It should contain the ICE candidates.
    • Send an 'invite' message to the remote peer with the offer SDP.

The JavaScript client will therefore not use trickle ICE but send all ICE candidates directly in the invite message.

The push button simulates the doorbell button. When the button is pushed, the Raspberry Pi should establish a new video call, with a random string as call ID (room identifier).

Follow the tutorial to wire up the push button and program the GPIO pins on the Raspberry Pi.

The Raspberry Pi should inform the user when somebody rings the door.

Find a way to send a message to the user (SMS, Telegram, email, ...) with the URL to connect to the video call.

You have to provide the following elements to the professor:

• A link to the GitHub repository with your final code.
• A short video (maximum 10 minutes) in which you:
  • give a demonstration of your system,
  • explain how you sent a message to the user
  • show how you structured the code of the RPi client and how you implemented the difficult parts.

All students will vote and the best project will get an additional 0.5 on the grade.