humandynamics / openbadge Goto Github PK

License: MIT License

Python 8.42% Shell 0.16% Makefile 1.97% C 34.15% Jupyter Notebook 0.14% CMake 0.11% Stata 3.25% C++ 48.97% Dockerfile 0.06% Game Maker Language 2.75%

openbadge's Introduction

OpenBadge project

Welcome to the Open Badge project, an open-source framework and toolkit for measuring and shaping face-to-face social interactions using either custom hardware devices or smart phones, and real-time web-based visualizations. Open Badges is a modular system that allows researchers to monitor and collect interaction data from people engaged in real-life social settings. For more information, please read our main publications (1,2,3) ,visit our wiki, or join our discussion forum.

Additional repositories that are part of this project:

https://github.com/mitmedialab/rhythm-badge-programmer - schematics, vector files and instructions for making a programming rig
https://github.com/HumanDynamics/openbadge-hub-py - a python based hub (base station) for pulling data from badges
https://github.com/HumanDynamics/openbadge-hub - a Javascript/Cordova app that communicates with the badges and stores the data to a server
https://github.com/HumanDynamics/openbadge-server - the backend server for the app
https://github.com/HumanDynamics/openbadge-analysis - Python (2.7) package for loading and pre-processing badge data
https://github.com/HumanDynamics/openbadge-analysis-examples - Examples showing how to use the analysis package

Docker wrapper

The nRF working environment is now available as a Docker container. The following commands have been tested under Ubuntu linux, but should work under iOS as well (with some modifications, especially to the device names).

Important notes:

The badge must be connected to the programmer before starting the docker (using docker-compose run)
Docker will mount the project directory under /app inside the container, so the container can see the latest changes (no need to build the container after every code change)

Commands:

docker-compose build : builds the docker container "nrf". You must run this command at least once before calling docker-compose run.
docker-compose run nrf : where is a command to run inside the container.
- When calling make (e.g. make badge_03v6_noDebug flashUnlock flashErase flashS130 flashAPP), the script will automatically change the working directory to the data_collector directory. You can see the different make options by calling: "docker-compose run nrf make"
docker-compose run nrf bash : opens an interactive bash shell
./programming_loop_docker.sh [test/prod] : a utility used for programming a batch of badges. It will compile once, and then enters a loop. Each time a new badge is placed, it will be automatically programmed and its MAC address will be written to the macs.log file. For most purposes, use the "prod" command variables. The "test" option turns on serial output and other testing features and is not recommended for production.

If you are using a raspberry pi to run this code, please use the raspberry pi compose file (-f docker-compose-raspi.yml). For example docker-compose -f docker-compose-raspi.yml build.

Firmware testing tools

The framework includes two tools for testing firmware code

Badge terminal. The utility opens a connections to a badge and allows you to run simple commands. To run, use the following docker command: docker-compose run nrf terminal <MAC ADDRESS>
Integration tests. These are tests designed for testing basic badge functionality. To run the tests, use the following docker command: docker-compose run nrf run_all_tests <MAC ADDRESS>

Note that in order to run the integration tests, you'll need to have a badge connected with a serial interface. A regular J-Link debugger/programmer does not include these. We recommend using a nRF51 dev-kit as a programmer for this purpose (see our wiki/documentations on how to make one).

Badge testing procedure

The badge firmware includes a tester mode that can be used for testing new badges. To compile, use the badge_03v6_tester flag (or badge_03v4_tester, for older hardware). For example:

make badge_03v6_noDebug flashUnlock flashErase  flashS130 flashAPP

Or, when using docker:

docker-compose run nrf make badge_03v6_noDebug flashUnlock flashErase  flashS130 flashAPP

Once programmed, the badge uses the LEDs to indicate the status of the test:

Blink red LED several times, one for each test (EEPROM, Flash, etc)
Microphone test. When it starts, both green and red LEDs will turn on It then looks for quiet->noise->quiet->noise pattern, each section must be at least 200 ms long. This must be completed within 10 seconds. In a quiet place, simply say "ahhhh" twice, with pause in between. Every time the noise passed the threshold, the LEDs will turn off
Accelerometer test. When it starts, both green and red LEDs will turn on. Simply shake the badge
Once done, the green LED will turn off for several seconds and then turn off

Example for known issues with badge manufacturing:

LEDs not turning on - might be a problem with the LEDs themselves
When microphone test starts, both LEDs will momentarily turn on and then off This might indicate a problem in the microphone filter or amplifier circuit (the microphone saturates and noise level says high)

openbadge's People

Contributors

Stargazers

Watchers

Forkers

asmk1 chrisong88 abhimnc techboycr patcon timwis bernal-r zackpi michaelhopfengaertner dominikgrothe dmorden gitter-badger kitspace-forks petershan1119 eltongjata startuplabjyu despar49

openbadge's Issues

BadgeFramework has potential bug in OpenBadge get_status

get_status gets seconds and milliseconds, but in two different commands. Since several milliseconds will elapse between these two commands, the milliseconds might belong to the next seconds.

For example, if the time for the first command is 1111.999, and it takes 20 ms to call the next command, the next command will get the millisecond for 1112.019, which is wrong

Host project on community website for easier ordering

Hi again @OrenLederman!

A few of us have been talking and meeting to work on this. (We keep notes fwiw)

We came across some websites that we were hoping to use so that it's simpler for the next group to walk this path. They allow for project sharing, and essentially one-click ordering for others who want to do the same fabrication.

PCBWay: Shared Projects
seeed studio: Project Hub
Elecrow: Cooperated Designers (seems more profit-motivated and top-down than others)
EDIT: Upverter

tl;dr - Would you be opposed to our setting up a presences for openbadge on a community platform like these ones?

Here are some comments -- 🔍 facts, ❤️ feelings, and 💡 ideas -- related to this discussion:

community platforms: Meaning community side of PCBWay, seeed studios, etc.

General

🔍 The openbadge license is MIT, and so we're technically free to do anything.
❤️ I feel preference to work in a way that aligns with your expectations and goals.
❤️ I feel eager to make it simpler for others to start projects with openbadge.
❤️ I feel eager to actively contribute to having a presence on community platforms, so that I can help foster a larger peer community of collaborators.
❤️ I feel open to letting your team manage this, if you have strong feelings on it.
🔍 The community platforms don't seem to offer accounts for team management.
💡 We could share an single account.
🔍 seeed studio has some options for teams/contributors that we could use accordingly.

Profit-sharing and incentivization on platforms

🔍 The platforms seem to offer financial incentives (paid or discount).
❤️ I feel uninterested in being personally enriched by hosting a project on these platforms.
❤️ I feel that the incentive could be used to promote something community-minded.
💡 We could enabled "monetization", but use it to give community projects discounts/credits.
💡 Alternatively, we could try to disable any monetization on the account.
❤️ I feel that monetization should be disabled if it in any way increases the price of regular orders.
🔍 I've tweeted at PCBWay to discover options for commission.
🔍 The community platforms have referral programs, eg PCBWay's.
1. 💡 We could use the above-mentioned shared account to receive referral bonuses from doc/readme links, and hold it for community uses.

Anyhow, I look forward to hearing your thoughts! Hopefully this is a helpful format for this discussion :)

cc: @kevinphys (pls feel free to add your own thoughts! 😄 )

Summary

Took some comments from rest of issue for summary:

🔍 I get asked a lot of questions about deployment and data analysis. (Oren)
❤️ I feel it's hard to get started with no hardware background. (Jörg)
🔍 I can contribute data analysis skills. (Jörg)
🔍 Many people are requesting badges. (Oren)
❤️ I feel that monetization of selling badges is possible. (Oren)
❤️ I feel that money could support project dev within community. (Oren)
💡 Design an easy-to-use badge firmware "burning" tool. (Oren)
🔍 I am too busy to take lead on new things. (Oren)
🔍 Students might be available to help. (Oren)
💡 Let's take actions to finalize renaming the project to Rhythm Badge. (Oren)
🔍 New firmware rewrite will be released soon. (Oren)
🔍 I can help build badges for fellow researchers. (Bjoern)
🔍 Oren can't sell badges as member of MIT. (Oren)
🔍 A worldwide capacitor shortage makes ordering harder, as stock changes rapidly.
🔍 Testing a programming new badges takes time. (Oren)
🔍 Initial support and analysis of new teams takes time. (Oren)
🔍 sgilbert92 needs to order a couple hundred badges. (Stephen)
🔍 Badges will soon be tested in classrooms. (Stephen)
🔍 Badges will soon be tested in assisted living facilities. (Stephen)
❤️ I feel that we need a better place to ask questions and create discussions. (Oren)
❤️ Questions are often in email right now, and need a better place. (Oren)

Summarized up to #109 (comment)

Use the "Value" column when available, instead of description

EDIT: Oops wrong repo.

~~In Eagle, it seems that the "description" column actually contains uneditable and less useful information like "CAPACITOR, European symbol" (see below screenshot).~~

On the other hand, the "value" column has more useful information that 1clickbom expects to be in description. Also, the "value" column can be edited very easily. Also, "value" is the changeable column when using the "assembly variant" feature, and it's nice that each assembly can be saved in the schematic files. So someone working on a board could create multiple variants, perhaps some with CPL identifiers, etc.

~~Thoughts on using the "value" field instead of description when it is present? Just to streamline the instructions and remove an "open and edit" step :)~~

replace on/off switch with one that has a button on the side

The new (small) switch seems to get turned off sometimes when the badge falls or moves. Couldn't validate it. But if that's not the case, we just sometimes don't push it all the way up, and then the badge turns off.

Scanner code should use max instead of mean when aggregating multiple RSSI readings

Our tests show that it's more stable

Better TZ handling in Python code

Currently, the badge expects (and returns) UTC time. However, the server code saves the datetime in local time, without time zone information.

While the code converts to and from UTC, it is better to use pytz (and tzlocal), and save the time zone as part of the date. Or, alternatively, save all dates in UTC

https://julien.danjou.info/blog/2015/python-and-timezones

http://stackoverflow.com/questions/13218506/how-to-get-system-timezone-setting-and-pass-it-to-pytz-timezone

Discuss making more use of Octoparts' Common Parts Library

I'm still figuring out how this usually works, but is the common parts library something that was considered while building? Basically, using components from the list helps ensure stable stock and cheapest prices.

https://octopart.com/common-parts-library

https://www.open-electronics.org/find-all-the-electronic-components-of-your-dreams-on-common-parts-library-octopart/

Updating the image of the badge to newest version

Hi @OrenLederman,

Since the last email that I sent to you, I have been able to order the badge through pcbway.com. It will take around one month for the entire process. The notes that organized by @patcon are becoming a huge help to this point.

In the past couple of days, I got some emails from them asking for several issues, such as whether they did the right things on soldering the switch. These questions were probably very easy questions for you.

What to do with the extra wielding points?
If this part is soldered correctly?
If this part is soldered correctly?

Honestly, I thought that giving them enough information such as the schematics, BOM, and centroid, everything will go smoothly.

As I don't have any experience with this hardware engineering, so answering this questions is complicated for me.

So the first things that I do as a beginner is to compare the picture of the rhythm badge that I order with the image of the rhythm badge showed in the readme.md, to see if the soldered in the right position and direction.

However, It seems that the picture of the rhythm badge showed in the readme.md is different with a different position and different looks of the components. Is it the older version? If yes, perhaps, It would be better if you could update the image with the newest version, so people (like me), can easily validate our artifacts through visual.

Thank you very much.

Beacons don't keep the badge_id and group_id when turned off

It appears that the ids are kept in RAM but never saved to the EEPROM

Integration tests don't work on raspberry pi

Badge audio sending sometimes sends old data

See log below
(Note - in this version of the hub we use the last chunk's data to determine the last timestamp. Need to change this as a temporary fix)

2016-10-09 13:55:59,170 - INFO - Requesting device H0HBQ2TD1R from server...
2016-10-09 13:55:59,196 - INFO - [D6:B8:C6:BA:9B:2B] Connecting to D6:B8:C6:BA:9B:2B
2016-10-09 13:56:01,000 - INFO - [D6:B8:C6:BA:9B:2B] Connected
2016-10-09 13:56:01,001 - INFO - [D6:B8:C6:BA:9B:2B] Starting audio recording
2016-10-09 13:56:01,139 - INFO - [D6:B8:C6:BA:9B:2B] Got time ack
2016-10-09 13:56:01,140 - INFO - [D6:B8:C6:BA:9B:2B] Badge datetime was: 1476035761,23
2016-10-09 13:56:01,140 - INFO - [D6:B8:C6:BA:9B:2B] Starting proximity scans
2016-10-09 13:56:01,349 - INFO - [D6:B8:C6:BA:9B:2B] Got time ack
2016-10-09 13:56:01,349 - INFO - [D6:B8:C6:BA:9B:2B] Badge datetime was: 1476035761,225
2016-10-09 13:56:01,349 - INFO - [D6:B8:C6:BA:9B:2B] Requesting data since 1476035721 603
2016-10-09 13:56:04,640 - ERROR - [D6:B8:C6:BA:9B:2B] failed pulling data
2016-10-09 13:56:04,640 - ERROR - [D6:B8:C6:BA:9B:2B] 1
2016-10-09 13:56:04,641 - ERROR - [D6:B8:C6:BA:9B:2B] Device disconnected
2016-10-09 13:56:04,641 - INFO - [D6:B8:C6:BA:9B:2B] Disconnecting from D6:B8:C6:BA:9B:2B
2016-10-09 13:56:04,641 - INFO - Errors pulling data. Saving data is anything was received
2016-10-09 13:56:04,641 - INFO - Chunks received: 7
2016-10-09 13:56:04,641 - INFO - saving chunks to file
2016-10-09 13:56:04,641 - DEBUG - Chunk timestamp: 1476035721.603, Voltage: 2.875, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,642 - DEBUG - Chunk timestamp: 1476035727.303, Voltage: 2.875, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,642 - DEBUG - Chunk timestamp: 1476035733.003, Voltage: 2.875, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,642 - DEBUG - Chunk timestamp: 1476035738.703, Voltage: 2.875, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,643 - DEBUG - Chunk timestamp: 1476035744.403, Voltage: 2.875, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,643 - DEBUG - Chunk timestamp: 1476035345.388, Voltage: 2.893, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,643 - DEBUG - Chunk timestamp: 1475973397.361, Voltage: 2.780, Delay: 50, Samples in chunk: 114
2016-10-09 13:56:04,643 - INFO - done writing
2016-10-09 13:56:04,643 - DEBUG - Setting last badge audio timestamp to 1475973397 361
Traceback (most recent call last):
File "./badge_hub.py", line 331, in
pull_devices(mgr, args.start_recording)
File "./badge_hub.py", line 229, in pull_devices
dialogue(b, activate_audio, activate_proximity)
File "./badge_hub.py", line 125, in dialogue
bdg.set_audio_ts(last_chunk.ts, last_chunk.fract)
File "/home/orenled/Documents/OpenBadge/src/nRF_hub/badge.py", line 302, in set_audio_ts
raise ValueError('Trying to update last_audio_ts with old value')
ValueError: Trying to update last_audio_ts with old value

Battery compartment requires manual soldering

Solder paste on the ground contact of the battery compartment doesn't re-flow well, probably because the battery holder itself acts a giant heat-sink. To fix that, we need to manually add solder to that area.

Other than that, the current design is good - it keep that contact event is the battery moves around.

An alternative design would be a very large contact, slightly larger than the area of the bottom part of the battery. With this design, even if you left the battery from its side, it will still keep the contact.

Cleanup SDK Path in Makefile

It sill has a reference to sdk 8, but we are using 12. Note that everything still works when running inside docker

Docker won't work with JLinkExe 6

Might be missing a library. See - https://wiki.segger.com/Could_not_open_J-Link_shared_library#Linux

Rename project with consistent rhythm namespace

Oren mentioned that a rename of the project was overdue: #109 (comment)

Ok, so doing a quick inventory.

random note: website is http://www.rhythm.mit.edu/rhythm-badges/

Name	Proposed Name	Description	Status
`openbadge`	`rhythm-badge`	Main repo and badge hardware files.	✅
`rhythm-rtc`	---
`rhythm-server`	`rhythm-server-js`
`openbadge-server`	`rhythm-server-py`
`openbadge-hub-py`	`rhythm-hub-py`
`openbadge-hub`	`rhythm-hub-mobile`	Cordova mobile app. (aka RoundTable)
`openbadge-analysis`	`rhythm-badge-analysis`
`openbadge-analysis-examples`	`rhythm-badge-analysis-examples`
`openbadge-analysis-test-badge`	`rhythm-badge-analysis-test-badge`
`openbadge-rfduino`	`rhythm-badge-rfduino`		💤
`rhythm-cradle`	---
`rhythm-landing-page`	---
`rhythm-meeting-mediator`	---
`rhythm-client`	---
`rhythm-dashboard`	---

I've still got some other summary I'm hoping to do, like getting active/inactive status, and which prices work together, but curious to hear more about how the ecosystem works :)

Once we have this sorted, I'm happy to put together some PRs to get the name change finalized and ready to go

Badge sends wrong audio chunks when requesting chunk from a future date

2016-10-08 16:31:25,715 - INFO - [D6:B8:C6:BA:9B:2B] Connecting to D6:B8:C6:BA:9B:2B
2016-10-08 16:31:27,125 - INFO - [D6:B8:C6:BA:9B:2B] Connected
2016-10-08 16:31:27,126 - INFO - [D6:B8:C6:BA:9B:2B] Starting audio recording
2016-10-08 16:31:27,265 - INFO - [D6:B8:C6:BA:9B:2B] Got time ack
2016-10-08 16:31:27,265 - INFO - [D6:B8:C6:BA:9B:2B] Badge datetime was: 1475958687,161
2016-10-08 16:31:27,266 - INFO - [D6:B8:C6:BA:9B:2B] Starting proximity scans
2016-10-08 16:31:27,405 - INFO - [D6:B8:C6:BA:9B:2B] Got time ack
2016-10-08 16:31:27,405 - INFO - [D6:B8:C6:BA:9B:2B] Badge datetime was: 1475958687,312
2016-10-08 16:31:27,405 - INFO - [D6:B8:C6:BA:9B:2B] Requesting data since 1475962378 682
2016-10-08 16:31:27,685 - INFO - [D6:B8:C6:BA:9B:2B] finished reading data
2016-10-08 16:31:27,685 - INFO - [D6:B8:C6:BA:9B:2B] Requesting scans since 1475958378
2016-10-08 16:31:28,106 - INFO - [D6:B8:C6:BA:9B:2B] finished reading data
2016-10-08 16:31:28,106 - INFO - [D6:B8:C6:BA:9B:2B] Disconnecting from D6:B8:C6:BA:9B:2B
2016-10-08 16:31:28,106 - INFO - Successfully pulled data
2016-10-08 16:31:28,107 - INFO - Chunks received: 1
2016-10-08 16:31:28,107 - INFO - saving chunks to file
2016-10-08 16:31:28,107 - DEBUG - CSV: Chunk timestamp: 1475958683.841, Voltage: 2.79061579704, Delay: 50, Samples in chunk: 72
2016-10-08 16:31:28,107 - INFO - done writing
2016-10-08 16:31:28,107 - DEBUG - Setting last badge audio timestamp to 1475958683 841
2016-10-08 16:31:28,107 - INFO - Proximity scans received: 5
2016-10-08 16:31:28,107 - INFO - saving proximity scans to file
2016-10-08 16:31:28,107 - DEBUG - SCAN: scan timestamp: 1475958411.000, voltage: 2.75999999046, Devices in scan: 0
2016-10-08 16:31:28,108 - DEBUG - SCAN: scan timestamp: 1475958471.000, voltage: 2.78999996185, Devices in scan: 0
2016-10-08 16:31:28,108 - DEBUG - SCAN: scan timestamp: 1475958531.000, voltage: 2.78999996185, Devices in scan: 0
2016-10-08 16:31:28,108 - DEBUG - SCAN: scan timestamp: 1475958591.000, voltage: 2.78999996185, Devices in scan: 0
2016-10-08 16:31:28,108 - DEBUG - SCAN: scan timestamp: 1475958651.000, voltage: 2.78999996185, Devices in scan: 0
2016-10-08 16:31:28,108 - DEBUG - Setting last badge proximity timestamp to 1475958651
2016-10-08 16:31:30,110 - INFO - Reading devices from file: device_macs.txt
2016-10-08 16:31:30,111 - DEBUG - D6:B8:C6:BA:9B:2B
2016-10-08 16:31:30,111 - INFO - Scanning for devices...

Badges sometimes won't turn on?

Remember the problem with the badges sometimes not turning on (until you discharge the large capacitor)? It's something that we mostly saw after programming the badges.

I did a small test - turned off the badge, connected the battery's VCC to the GND side of the capacitor and then turned on the badge - it won't start. conncet the VCC and GND of the 100uF and then it works. Do the same for the 4.7nF capacitor on the SWD pin and then it work. Short circuiting VCC and GND in the programming pads also works. But shortcircuiting the pads next to the battery does..

I tested it on two 3v4 badges, with the same results. Sometimes you need to keep the connection between Battery's VCC and the GND of the 100uF capacitor connected for a couple of seconds.

This problem doesn't seem to happen in the 3v3 badges, by the way, but may require more testing. Some badges seem more sensitive to this than others. Might be a difference between different batteries too.

Also, it doesn't seem to happen on the 3v4 badges if there is no RTC module installed. Hummm.

Maybe worth checking whether we can recreate the problem if we keep the RTC and remove the large capacitor.

I'm adding this to the spreadsheet and I'll open a bug in the github project (so we have a documentation).

We should revisit this after we are done with the scans.

possible causes:

capacitor is too big
something withe RTC or it's resistors cause the same affect that
maybe it's just the RTC alone.. worth looking at teh signals it's sending when this problem appears (maybe it's doing soething that keeps the nRF51 module "awake"
should try a badge with RTC and smaller capacitor

Oren

New chipset on BMD-200

BMD-200 is no longer shipped with nrf51422, but instead with nrf51822. Changes in the Makefile (JLINK_OPTIONS) required.

Mobile app marks new meetings a "completed"

Scanner code should distinguish between "member" badges and beacons

We'll do this my splitting the range between them . Member badges have id < 16000. Beacons use id >= 16000

Badge sometimes sends way too much scan data (fw2.0, maybe master?)

There appears to be an issue where the badge will send way more scan chunks than it needs to.

For instance, in this log:

(Start of test, first chunk to be sent is stored to RAM chunk 0, then to EXT chunk 475) 

SCANNER: Saving scan results. 1 devices seen
  C:0
    bdg ID#0400, rssi -68, count 13
SCANNER: Done saving results.  used 1 chunks.
STORER: writing SCAN chunk 0 to EXT chunk 475

(... some time passes...)

SCANNER: Saving scan results. 1 devices seen
  C:3 <- Most recent scan goes to RAM Chunk 3
    bdg ID#0400, rssi -73, count 16
SCANNER: Done saving results.  used 1 chunks.
STORER: writing SCAN chunk 3 to EXT chunk 478 <- So the most recent chunk is here, 478.
  Read battery: 2842mV (raw: 2844mV).
  Read battery: 2842mV (raw: 2844mV).
� Read battery: 2843mV (raw: 2844mV).
  Read battery: 2843mV (raw: 2844mV).
�Read battery: 2843mV (raw: 2844mV).
  Read battery: 2841mV (raw: 2837mV).
SCANNER: Started scan.
SENDER: Got REQSCANS request.
SENDER: sending scans since: s:P c:475 <- Correct first chunk to send.
SENDER: sent s:P c:475 n:1
SENDER: sent s:P c:476 n:1
SENDER: sent s:P c:477 n:1
SENDER: sent s:P c:478 n:1 <- Should stop sending after this one.
SENDER: sent s:P c:479 n:1 <- It continues sending.
SENDER: sent s:P c:480 n:1
SENDER: sent s:P c:481 n:1
SENDER: sent s:Q c:0 n:1
SENDER: sent s:Q c:1 n:1
SENDER: sent s:Q c:2 n:1
SENDER: sent s:Q c:3 n:1
SENDER: sent s:Q c:4 n:1
SENDER: sent s:Q c:5 n:1
SENDER: sent s:Q c:6 n:1
SENDER: sent null header.  REQSCANS complete.

We get a bunch of duplicate scan chunks (scan chunks from RAM that have already been stored in EXT) as well as some scan chunks past what have been stored.

Still running down the cause of this bug. I'm not sure if it effects both master and fw2.0 or just fw2.0.

Clarifications before fabricating badges

First off, thank you so so much for making this repo public and keeping it up-to-date. I've had such energizing conversations with people about working with the artifacts of your project. ❤️

Context

I'm just getting ready to order some badges in unofficial collaboration with @shuyanglin of PDIS, the Taiwanese government's digital innovation department created by Digital Minister Audrey Tang.

We're currently getting ready to order a pilot batch of 15 sociometric badges and 3 BLE beacons. We'll later order 85 more, for use at a civic hackathon and other loose experiments. The pilot and larger scale experiment will likely have two separate goals:

to take certain online processes into physical space (pilot), and
to help the civic tech community better understand itself (larger experiment)

Disclaimer

I don't have hardware experience myself. I'm eagerly collaborating with people like @kevinphy of Taipei Hackerspace. Having said that, I accept full responsibility for any stupid questions I ask in this issue :)

Questions

Circuit Board Fabrication

Are the newest files exports always the ones you'd recommend? (I will be assuming "yes" with my following links and questions, but please advise if incorrect. I can imagine considerations of stability vs bleeding edge, but I'm not clear how to account for that.)
Will we need to order/build our own "programmer"? Could we manage without? Is this all the details for doing that?
What is the difference between Badge_03_v6 and Badge_03_v6_LIS2DH12_breakout? Which is recommended? These are the facts as I see them in relation to this question:
- 🔍 All the export dirs in v6 have BOMs, but v6_LIS2DH12 lacks a BOM
- 🔍 The files were all committed at the same time, so no clues there.
- 🔍 all the BOMs in v6 exports actually have the LIS2DH12 component (3-axis accelerometer) included (example)
Will Badge_03v2-Prog (assuming this is the programmer device), work with more recent v6 badges?

Misc.

Which BLE beacons would you recommend?

We'll be happy to contribute back what we learn by expanding on the documentation in this repo!

Note

I emailed David Shrier before arriving in Taiwan, in relation to my hopes of running a sociometric badge experiment. He responded quickly and graciously, and tried to point me in the right direction, but it seemed the direction was away from the badges, which is the primary interest. Apologies for any confusion this second outreach causes :)

BadgeFramework has issues with protocol definitions

The types of most fields in the implementation is signed, but the actual type is signed. This might create issues with some of the data, at some point.

Python Badge server - hci0 sometimes halts

Might be a RasPi issue rather than our code, but it's worth adding a watchdog that resets the hci is scan is stuck or communication with a badge is not responsive.

Badge sometimes send old chunks or empty chunks

I noticed that the badge sometimes send data older than the timestamp we ask for. In fact, it seems to does that when:

After the badge reset and recording hasn't been started yet it will send an empty chunk (timestamp = 0)
After the badge stopped recording, it will send a date from some recent chunk, even if it's earlier than the time requested (see example log below)

Be careful if you decide to change anything in this mechanism.. if it's not something very obvious we'll might need to just keep it as an open bug

One workaround, is to start recording before pulling new data. In fact,t hat is what we do with the phone hub, which might explain why we haven't noticed this before.

Example:

2016-08-28 11:16:27,789 - INFO - Resetting BLE
2016-08-28 11:16:29,801 - INFO - Done resetting BLE
2016-08-28 11:16:29,801 - INFO - Started
2016-08-28 11:16:29,802 - INFO - Will request data since 1472397389.000000
2016-08-28 11:16:29,802 - INFO - Scanning for devices...
2016-08-28 11:16:29,802 - INFO - Reading whitelist:
2016-08-28 11:16:29,805 - INFO - F9:A0:3E:12:9E:3E
2016-08-28 11:16:32,823 - DEBUG - Found E7:00:28:A7:DB:C2, but not on whitelist. Device info: {'rssi': -68, 'scan_date': datetime.datetime(2016, 8, 28, 11, 16, 32, 822945), 'adv_payload': {'proximity_status': 0, 'sync_status': 0, 'audio_status': 0, 'mac': [231, 0, 40, 167, 219, 194], 'badge_id': 63880, 'voltage': 2.75, 'status_flags': 0, 'project_id': 0}}
2016-08-28 11:16:32,824 - DEBUG - Found F9:A0:3E:12:9E:3E, added. Device info: {'rssi': -78, 'scan_date': datetime.datetime(2016, 8, 28, 11, 16, 32, 823176), 'adv_payload': {'proximity_status': 0, 'sync_status': 1, 'audio_status': 0, 'mac': [249, 160, 62, 18, 158, 62], 'badge_id': 830, 'voltage': 2.7, 'status_flags': 1, 'project_id': 0}}
2016-08-28 11:16:34,826 - DEBUG - Unseen device. Adding to dict: F9:A0:3E:12:9E:3E
2016-08-28 11:16:34,827 - INFO - [F9:A0:3E:12:9E:3E] Connecting to F9:A0:3E:12:9E:3E
2016-08-28 11:16:36,092 - INFO - [F9:A0:3E:12:9E:3E] Connected
2016-08-28 11:16:36,302 - INFO - [F9:A0:3E:12:9E:3E] Got status
2016-08-28 11:16:36,302 - INFO - [F9:A0:3E:12:9E:3E] Badge datetime was: 1472397396,111, Voltage: 2.70615839958
2016-08-28 11:16:36,302 - INFO - [F9:A0:3E:12:9E:3E] Requesting data since 1472397389 802.085
2016-08-28 11:16:36,443 - INFO - [F9:A0:3E:12:9E:3E] finished reading data
2016-08-28 11:16:36,443 - INFO - [F9:A0:3E:12:9E:3E] Requesting scans since 1472397389
2016-08-28 11:16:36,589 - INFO - [F9:A0:3E:12:9E:3E] finished reading data
2016-08-28 11:16:36,589 - DEBUG - [F9:A0:3E:12:9E:3E] Setting last seen audio chunk to 1472396577.994
2016-08-28 11:16:36,589 - INFO - [F9:A0:3E:12:9E:3E] Disconnecting from F9:A0:3E:12:9E:3E
2016-08-28 11:16:36,591 - INFO - Successfully pulled data
2016-08-28 11:16:36,591 - INFO - Chunks received: 1
2016-08-28 11:16:36,591 - INFO - saving chunks to file
2016-08-28 11:16:36,591 - INFO - CSV: Chunk timestamp: 1472396578.994, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 0
2016-08-28 11:16:36,591 - INFO - done writing
2016-08-28 11:16:36,591 - INFO - No proximity scans ready
2016-08-28 11:16:38,594 - INFO - Sleeping...
2016-08-28 11:16:44,600 - INFO - Scanning for devices...
2016-08-28 11:16:44,601 - INFO - Reading whitelist:
2016-08-28 11:16:44,601 - INFO - F9:A0:3E:12:9E:3E
2016-08-28 11:16:47,731 - DEBUG - Found E7:00:28:A7:DB:C2, but not on whitelist. Device info: {'rssi': -72, 'scan_date': datetime.datetime(2016, 8, 28, 11, 16, 47, 731034), 'adv_payload': {'proximity_status': 0, 'sync_status': 0, 'audio_status': 0, 'mac': [231, 0, 40, 167, 219, 194], 'badge_id': 63880, 'voltage': 2.75, 'status_flags': 0, 'project_id': 0}}
2016-08-28 11:16:47,731 - DEBUG - Found F9:A0:3E:12:9E:3E, added. Device info: {'rssi': -76, 'scan_date': datetime.datetime(2016, 8, 28, 11, 16, 47, 731212), 'adv_payload': {'proximity_status': 0, 'sync_status': 1, 'audio_status': 0, 'mac': [249, 160, 62, 18, 158, 62], 'badge_id': 830, 'voltage': 2.67, 'status_flags': 1, 'project_id': 0}}
2016-08-28 11:16:49,733 - INFO - [F9:A0:3E:12:9E:3E] Connecting to F9:A0:3E:12:9E:3E
2016-08-28 11:16:51,002 - INFO - [F9:A0:3E:12:9E:3E] Connected
2016-08-28 11:16:51,142 - INFO - [F9:A0:3E:12:9E:3E] Got status
2016-08-28 11:16:51,142 - INFO - [F9:A0:3E:12:9E:3E] Badge datetime was: 1472397410,932, Voltage: 2.67448687553
2016-08-28 11:16:51,143 - INFO - [F9:A0:3E:12:9E:3E] Requesting data since 1472396577 994
2016-08-28 11:16:56,253 - INFO - [F9:A0:3E:12:9E:3E] finished reading data
2016-08-28 11:16:56,254 - INFO - [F9:A0:3E:12:9E:3E] Requesting scans since 1472397389
2016-08-28 11:16:56,392 - INFO - [F9:A0:3E:12:9E:3E] finished reading data
2016-08-28 11:16:56,392 - DEBUG - [F9:A0:3E:12:9E:3E] Setting last seen audio chunk to 1472396686.322
2016-08-28 11:16:56,393 - INFO - [F9:A0:3E:12:9E:3E] Disconnecting from F9:A0:3E:12:9E:3E
2016-08-28 11:16:56,393 - INFO - Successfully pulled data
2016-08-28 11:16:56,393 - INFO - Chunks received: 20
2016-08-28 11:16:56,394 - INFO - saving chunks to file
2016-08-28 11:16:56,394 - INFO - CSV: Chunk timestamp: 1472396578.994, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,394 - INFO - CSV: Chunk timestamp: 1472396584.694, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,394 - INFO - CSV: Chunk timestamp: 1472396590.395, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,394 - INFO - CSV: Chunk timestamp: 1472396596.095, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,395 - INFO - CSV: Chunk timestamp: 1472396601.796, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,395 - INFO - CSV: Chunk timestamp: 1472396607.496, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,395 - INFO - CSV: Chunk timestamp: 1472396613.197, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,395 - INFO - CSV: Chunk timestamp: 1472396618.897, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,395 - INFO - CSV: Chunk timestamp: 1472396624.597, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,396 - INFO - CSV: Chunk timestamp: 1472396630.321, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,396 - INFO - CSV: Chunk timestamp: 1472396636.021, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,396 - INFO - CSV: Chunk timestamp: 1472396641.721, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114
2016-08-28 11:16:56,396 - INFO - CSV: Chunk timestamp: 1472396647.422, Voltage: 2.67448687553, Delay: 50, Samples in chunk: 114

Proximity scan doesn't handle more than 5 devices well

When 6 or more badges appear in the scan, badge sends incorrect packages to the hub, casing an error. It seems that the first package (20 bytes) is correct - 4 bytes for each of the first 5 devices. The next package is 5 bytes long, where the first 4 appears to contain a valid scan data, and the 5th byte is an error.

Looking at the badge debug logs, it seems that the problem is in storing:
SCANNER: Saving scan results. 6 devices seen
C:8
bdg ID#0000, rssi -98, count 34
bdg ID#DFE4, rssi -63, count 7
bdg ID#5F7D, rssi -65, count 6
bdg ID#2EB6, rssi -76, count 5
bdg ID#2660, rssi -88, count 3
bdg ID#A7F9, rssi -72, count 6
SCANNER: Done saving results. used 1ADV: advertising paused, src 0.
STORER: writing SCAN chunk 8 to EXT chunk 616

The storer should have allocated the 6th device to a second chunk. Instead, it appears to add it to the first chunk. The log shows that only one chunk is being stored.

It seems to be a memory leak of some kind. If you compare the log to a 5 badges log, you can see that the SCANNER:Done line gets cut off:

SCANNER: Saving scan results. 5 devices seen
C:7
bdg ID#0000, rssi -96, count 35
bdg ID#5F7D, rssi -66, count 6
bdg ID#2660, rssi -96, count 2
bdg ID#DFE4, rssi -62, count 7
bdg ID#2EB6, rssi -71, count 5
SCANNER: Done saving results. used 1 chunks.
ADV: advertising paused, src 0.
STORER: writing SCAN chunk 7 to EXT chunk 625

BadgeFramework - sending badge_id and group_number

In badge_protocol.py (the old protocol-implementation) l.114 and 115 the badge_id and group_number is only set, when != 0. So if one of them is 0, the serialized message doesn't contain the "0" anymore, it is just ignored. So my implementation of the old-protocol assumes that both id and group number is transmitted at the same time. And I think the old firmware also assumes it, because it checks for the length of the received message, and this shouldn't match anymore.

Update pick-and-place-files to make fabrication easier

Make sure that your pick-and-place files follow all of the IPC conventions for pin numbering/part orientation, so that nothing needs to be corrected after uploading.

Decide on spacing convention - tabs vs. spaces in the new firmware

New version uses both spaces and tabs. Better to decide on one.

Badge broadcast message doesn't always update

It seems that at least the status bits don't always update. Can't seem to find a specific way to recreate yet

[Support] Question from PCBWay on PCB manufacturing

Hi again! Hopefully this is an easy one :)

We've ordered programmer and badge boards from PCBWay. Programmer boards went off without a hitch. Badge boards had a quality assurance question:

Ref attached screenshot, is it correctly designed as no hole is there for the pointed out pad ?

Attached:

After we give them an answer, I'll ask them for suggestion on how to annotate to future clarity.

Badge tester code does not work in firmware 2.0

The badge keep restarting

temporary fix - use code from the old version

Python Badge server - needs timeout when communicating with badges

If a badge accepts a connection but does not reply, the server currently keeps the connection open. We should add a timeout to protect from these cases

nRF badge skips time

Badge time jumped to the future. This does not seem to be an hardware issue (battery level was within reason)

Data file, date skipped from 2016-03-10 to 2016-04-28
2016-03-10 22:20:34,2.56891489029,250,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
2016-04-28 03:59:43,2.60410547256,250,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1

server log, did not send a new date in between
2016-03-10 22:21:37,858 - INFO - [D1:61:D8:00:57:61] Talking with D1:61:D8:00:57:61
2016-03-10 22:21:39,416 - INFO - [D1:61:D8:00:57:61] Connected
2016-03-10 22:21:39,543 - INFO - [D1:61:D8:00:57:61] Got status
2016-03-10 22:21:39,557 - INFO - [D1:61:D8:00:57:61] Already synced
2016-03-10 22:21:39,560 - INFO - [D1:61:D8:00:57:61] Requesting data...
2016-03-10 22:21:40,837 - INFO - [D1:61:D8:00:57:61] Waiting for more data...
2016-03-10 22:21:41,843 - INFO - [D1:61:D8:00:57:61] Waiting for more data...
2016-03-10 22:21:42,847 - INFO - [D1:61:D8:00:57:61] Waiting for more data...
2016-03-10 22:21:42,851 - INFO - [D1:61:D8:00:57:61] finished reading data
2016-03-10 22:21:42,869 - INFO - [D1:61:D8:00:57:61] disconnected
2016-03-10 22:21:42,872 - INFO - [D1:61:D8:00:57:61] Chunks received: 1
2016-03-10 22:21:42,876 - INFO - [D1:61:D8:00:57:61] saving chunks to file
2016-03-10 22:21:42,879 - INFO - [D1:61:D8:00:57:61] Chunk timestamp: 2016-03-10 22:20:34, Voltage: 2.56891489029, Samples in chunk: 116
2016-03-10 22:21:42,888 - INFO - [D1:61:D8:00:57:61] done writing
2016-03-10 22:21:44,899 - INFO - [None] Reading whitelist:
2016-03-10 22:21:44,904 - INFO - [None] D1:61:D8:00:57:61
2016-03-10 22:21:44,907 - INFO - [None] EC:21:82:A8:0B:59
2016-03-10 22:21:44,911 - INFO - [None] Scanning for devices...
2016-03-10 22:21:49,924 - DEBUG - [None] Found EC:21:82:A8:0B:59, added
2016-03-10 22:21:49,928 - DEBUG - [None] Found D1:61:D8:00:57:61, added
2016-03-10 22:21:49,931 - INFO - [None] Found: 2 devices
2016-03-10 22:21:51,935 - INFO - [None] Communicating with synced devices...
...
2016-03-10 22:21:59,312 - INFO - [D1:61:D8:00:57:61] Talking with D1:61:D8:00:57:61
2016-03-10 22:22:00,825 - INFO - [D1:61:D8:00:57:61] Connected
2016-03-10 22:22:00,963 - INFO - [D1:61:D8:00:57:61] Got status
2016-03-10 22:22:00,967 - INFO - [D1:61:D8:00:57:61] Already synced
2016-03-10 22:22:00,979 - INFO - [D1:61:D8:00:57:61] disconnected
2016-03-10 22:22:00,982 - INFO - [D1:61:D8:00:57:61] No data ready
2016-03-10 22:22:02,988 - INFO - [None] Reading whitelist:
2016-03-10 22:22:03,002 - INFO - [None] D1:61:D8:00:57:61
2016-03-10 22:22:03,016 - INFO - [None] EC:21:82:A8:0B:59
2016-03-10 22:22:03,019 - INFO - [None] Scanning for devices...
2016-03-10 22:22:08,064 - DEBUG - [None] Found EC:21:82:A8:0B:59, added
2016-03-10 22:22:08,078 - DEBUG - [None] Found D1:61:D8:00:57:61, added
2016-03-10 22:22:08,088 - INFO - [None] Found: 2 devices
2016-03-10 22:22:10,093 - INFO - [None] Communicating with synced devices...
...
2016-03-10 22:22:13,499 - INFO - [D1:61:D8:00:57:61] Talking with D1:61:D8:00:57:61
2016-03-10 22:22:15,666 - INFO - [D1:61:D8:00:57:61] Connected
2016-03-10 22:22:15,803 - INFO - [D1:61:D8:00:57:61] Got status
2016-03-10 22:22:15,807 - INFO - [D1:61:D8:00:57:61] Already synced
2016-03-10 22:22:15,810 - INFO - [D1:61:D8:00:57:61] Requesting data...
2016-03-10 22:22:17,088 - INFO - [D1:61:D8:00:57:61] Waiting for more data...
2016-03-10 22:22:18,092 - INFO - [D1:61:D8:00:57:61] Waiting for more data...
2016-03-10 22:22:19,097 - INFO - [D1:61:D8:00:57:61] Waiting for more data...
2016-03-10 22:22:19,101 - INFO - [D1:61:D8:00:57:61] finished reading data
2016-03-10 22:22:19,138 - INFO - [D1:61:D8:00:57:61] disconnected
2016-03-10 22:22:19,142 - INFO - [D1:61:D8:00:57:61] Chunks received: 1
2016-03-10 22:22:19,155 - INFO - [D1:61:D8:00:57:61] saving chunks to file
2016-03-10 22:22:19,159 - INFO - [D1:61:D8:00:57:61] Chunk timestamp: 2016-04-28 03:59:43, Voltage: 2.60410547256, Samples in chunk: 116
2016-03-10 22:22:19,174 - INFO - [D1:61:D8:00:57:61] done writing