Does a CSV file have to be of a specific format? I tried uploading data from an accelerometer that I generated from a Micro:Bit a while ago (data0 (1).csv) and I get the below runtime error.

Unhandled Rejection (Error): Invalid column reference: "d.Value"
Array.Demo._handleDataChange
C:/Users/vpotluri/code/jacdac/sonification/tempSonification/sonification/src/Demo.tsx:101
   98 |    private _handleDataChange = (data: any) => {
   99 |        // setDataTable(data);
  100 | 
> 101 |        let dataSummary = data.rollup({
      | ^  102 |            mean: d => op.mean(d.Value),
  103 |            min: d => op.min(d.Value),
  104 |            max: d => op.max(d.Value),

DataManager.handleDataChange
C:/Users/vpotluri/code/jacdac/sonification/tempSonification/sonification/src/DataManager.ts:66
  63 | 
  64 |    public handleDataChange() {
  65 |        for (let i = 0; i < this._listeners.length; i++) {
> 66 |            this._listeners[i](this.table);
     | ^  67 |        }
  68 |    }
  69 | 

DataManager.loadDataFromText
C:/Users/vpotluri/code/jacdac/sonification/tempSonification/sonification/src/DataManager.ts:57
  54 |        this.table = aq.fromCSV(text, { delimiter, header });
  55 |        console.log(this.table.columns());
  56 |        console.log(this.table);
> 57 |        this.handleDataChange();
     | ^  58 |    }
  59 | 
  60 |    public loadDataFromFile(file: File) {

(anonymous function)
C:/Users/vpotluri/code/jacdac/sonification/tempSonification/sonification/src/DataManager.ts:61
  58 |    }
  59 | 
  60 |    public loadDataFromFile(file: File) {
> 61 |        file.text().then(text => DataManager.instance.loadDataFromText(text));
     | ^  62 |    }
  

63 | 
  64 |    public handleDataChange() {

Originally posted by @venkateshpotluri in #70 (comment)

Currently, I am using booleans to display specific parts of the UI as different radio buttons are selected. This will soon make the demo.tsx file very large and unmanageable. We may want to componentize the UI. mentioning #24 as it is related.

I am putting these directly from our meeting discussion, please feel free to break them down as sub-issues as necessary.

The following features need to be implemented to resolve this issue:

• introduce enum SonificationProperty that indicates which property to sonify. Possible values: Volume, Pitch, SterioPanning, Spatial.
• implement functionality to accept multiple enums and vary audio. Implementing support for Spatial can be delayed for later.
• implement controls in the UI to select these parameters. A simple group of checkboxes should do for a preliminary implementation.

https://github.com/make4all/sonification/blob/d3f63ab7f039588d35c8a0efe1c988261abc0841/package.json#L39

The "sonification" package should be "pure", e.g.: no dependencies. All the dependencies are used for demo pages.

https://github.com/make4all/sonification/blob/d3f63ab7f039588d35c8a0efe1c988261abc0841/src/constents.ts#L9

There are already other enums in sonificationclass.ts so let's move these in that file already.

We need sample CSV files. John offered to look on Kaggle and I am offering to make some using micro:bit

Currently, outputs that extend SonifyFixedDuration do not end once the node is updated. We need to implement a fix where we can keep track of whether a node is currently playing, and if so, cut it off before playing the following node.

https://github.com/make4all/sonification/blob/d3f63ab7f039588d35c8a0efe1c988261abc0841/src/SonificationClass.ts#L1

It makes it easier to keep a match between class and file name.

Based on the discussion on September 15, 2021, here are a tentative list of APIs that need to be exposed.

• play() /Plays the audio/
• pause() /pauses the audio/
• volume(level:Number) /number between 0 and 100 to adjust volume/
• seekTo(durationInMS:Number) /seek to a particular duration of the audio/

This will be picked up once I am done with #5. in the mean time, please review and feel free to comment if you think i've missed something important here @pelikhan

This is linked with #23. The open pull request begins to implement this. I need feedback and help.

https://github.com/make4all/sonification/blob/d3f63ab7f039588d35c8a0efe1c988261abc0841/src/sonification.ts#L2

This file looks like a bunch of experimental code. Ready to delete?

linked to #23 , #6.

I will break this down into sub-issues by Tuesday.

Capitalize

Originally posted by @pelikhan in #75 (comment)

Move the vega specs out of the sonification.ts UI.

https://github.com/make4all/sonification/blob/d3f63ab7f039588d35c8a0efe1c988261abc0841/src/Vega-schema.json#L1

Rule of thumb: use const by default, turn to let if you need the variable to be mutable.

Probably good to turn on eslint who finds these issues.

There are two lingering accessibility issues with the Data Grid:

• Each data point is announced as a clickable element. Is there any eventual functionality we would implement when we click a point? else this may appear confusing to a screen reader user.
• there is an empty clickable element next to each column header.

Both require customizing the rendering of the Data Grid

I think we should rethink the project structure.

I was trying to contribute to the library today and I thought I'd add a few simple stream sonifications. It was harder than I thought to add a page for my code.

Would it be possible to brain storm how we can better support the demo-page model we spoke about last meeting?

This came up in the conversation that @jrthompson33 and I had this morning. Doing this will help us understand if the scheduling supports going back.

The project is missing instruction for developer to build and test locally. This can be added in the README.md or in a CONTRIBUTING.md file.

• setup: how to install the dependencies
• development cycle: how to build and debug the code
• release: how to trigger a release

fetching data from a URL is beyond what this library should do. The user can do that on his side since there are many ways to deal with a failed request.

Originally posted by @pelikhan in #75 (comment)

We may have to implement sonification scheduling similar to the approach mentioned here to support #51. For this to work, I would like to brainstorm on who (component) should be responsible for pulling data and other parameter changes from the UI in frequent intervals and how this should happen.

This is with reference to my update from the meeting this morning about using Vega as a data processor. I had thought that reading the data after applying vega transforms would be a straight-forward call, but the vega.view.data function returns the original dataset, and there seems to be no way to retrieve the modified dataset using this method as the modified dataset does not have a name. this issue on Vega-lite informs that this is not a straight-forward process, and This PR on the Vega editor seems to access the transformed data. I am currently in the process of understanding this PR and the related editor source code, and I will update this issue once I have a solution.

@venkateshpotluri has mentioned the following usability issues with the interface:

• Different steps on the UI are exposed to the screen reader without having to press the continue button.

• the menu button to select the example seems slightly tricky to navigate with a screen reader. I will check if there are better examples of menu buttons similar to this one so that we can evaluate if we can dchange anything about these buttons or use a combo box instead.

• The table that displays the data is confusing to parse with a screen reader.

  • NVDA announces that the table is of 0 rows and 0 columns both before and after data is loaded. These announcements are automatically generated by NVDA from the semantics of the table.
  • Only the first 2 columns of the CSV seem to be annotated as headers. The file contains a total of 6 columns. I am able to browse the data for the remaining 4 columns but I do not hear the name of the corresponding header.
  • #80
  • #81

• Data seems to be loaded even before the continue button is clicked. Not sure if this is expected behavior.

• Aria-label for the file upload button contains that it is a button. This causes duplicate announcements that the file upload is a button. the label "Button for file upload" can be replaced with "choose file" or equivalent.

The readme should start with instructions on how to add sonification to a project. Typically, npm install .... Currently it dives directly in the architecture of the library.

tagging @jrthompson33 and @pelikhan to track.

You probably want to throw an error here

Originally posted by @pelikhan in #75 (comment)

I will need to test if changing the thresholds mid-sonification will update the sonification.

Implement a User Interface (UI) for authoring data streaming sonification. The new UI should allow users to add data handlers to incoming JacDac streams. The UI should also support the customization of parameters and data outputs for each data handler.

Should Datum be an interface instead; there's little value on it being a class and interfaces are generally more flexible to use (also they have 0 runtime overhead).

https://github.com/make4all/sonification/blob/d75ebb2ceb2dbf8d5f8c581418e26016f4375a85/src/sonification/Datum.ts#L15

This will be my next step after #4. I will update details once I pick this work item up.

Once our build can update github pages with the latest working version of the project, demoing this project is much easier since everyone can try.

our ultimate goal with the sonification library is to send it Vega-lite payloads and consume sound player functionality. #6 examines the sonification APIs, but to my understanding, this investigation should be about how to create the vega payload to sonify, and how to integrate the sound player functionality into the interface that displays the data. @jmankoff, @pelikhan and @jamesadevine please add comments if you believe there are missing details of this to-do from the meeting.

As discussed during the meeting on Nov 17, we need to link the sonification to a visualization to make debugging and demoes accessible for everybody.

https://github.com/make4all/sonification/blob/a77f3a60226bb1019be3bc0b76ccb97d445b2eb5/src/sonifier.ts#L3

Eventually remove d3 dependency as we're using a tiny feature of it.

We hear a clicking noise when we stop oscillator nodes corresponding to each point once they are done being played. Adding gain nodes and reducing the volume of each oscillator to 0 in the last 0.015 seconds of playback might remove this click noise. source: Web Audio: the ugly click and the human ear. Though this article does not mention this, we may also have to increase the volume of each oscillator for 0.015 seconds at the beginning of playback to make the transition smooth.

This is in follow up with the issue I am facing, discussed in our sync up meeting on Oct 27. I am trying to use the transpiled sonification.ts in an index.html to be able to test and debug the sonification code. I have raised a pull request in which we attempt to play some sound in the sonification.ts file. there seems to be issues with resolution of imports when the sonification.ts is transpiled to sonification.js and is called in the index.html. Here is a list of things I tried based on various sources:

• switched the target, module and lib to es2015/2019.
• Set the moduleResolution to node.
• used the type="module" parameter in the script tag in index.html.
• Used standardized-audio-context for WebAudio (thank you @jmankoff for helping me find this!).

Things we may need

• to be able to use the ts file in an index.html
• some kind of activation mechanism like a button to invoke WebAudio.
  Thank you @pelikhan for offering to help with this!

There are docs but nothing is referencing to them. Handy to have the docs documentation link as the project web site

https://make4all.github.io/sonification/

The goal is to test the sonification library in a react-vega scenario to understand if the library provides all the necessary endpoints. Perhaps this is best to test after @jrthompson33 fixes #50 in case code changes are necessary to the sonification library. We should iteratively do this with the next attempt after I fix #45.

https://github.com/make4all/sonification/blob/d3f63ab7f039588d35c8a0efe1c988261abc0841/src/SonificationClass.ts#L59

The function is already an instance function of the class so "this" point to the class instance. Remove this: Sonifier, parameter.