scratchfoundation / scratch-audio Goto Github PK

Expected behavior

Please describe what should happen
The stack should un-highlight after running

Actual behavior

Describe what actually happens
The stack stays highlighted after the sound has completed

Steps to reproduce

Explain what someone needs to do in order to see what's described in Actual behavior above
At https://llk.github.io/scratch-gui/ drag out a "move 10 steps" and attach "play sound meow until done" to it.
Run the stack
The cat moves 10 steps and "meow" plays, however the stack remains highlighted after playing

Operating system and browser

Chrome 57, macOS 10.12

We will probably need to re-design the audio engine. This is mainly to separate the life cycle of a SoundPlayer from that of a clone. In 2.0, a sound started by a clone continues after the clone has been deleted (see scratchfoundation/scratch-vm#949), so the re-design is required to support that behavior.

@thisandagain and I discussed a possible new design for the audio engine, which I wanted to document here, because this work is likely to be deferred for another few months.

TL;DR: AudioEngine manages a set of persistent SoundPlayers, one for each sound

Requirements

A target (sprite or clone) needs to be able to:

• Play a sound owned by the parent sprite
• Play multiple sounds at once
• Play one one copy of each sound at a time
  • i.e. multiple copies of one sound can not be played simultaneously - it restarts
• Stop all sounds for this target
• Set and change volume for this target
  • by updating a webaudio node
• Set and change pan for this target
  • by updating webaudio nodes
• Set and change pitch effect (i.e. playback rate) for all sounds currently being played by this target
  • Each sound has a buffer source node that must be explicitly updated

The requirements above are currently being met, but this one is not:

• A sound started by a clone plays to completion, even if the clone is deleted while it plays

Current Audio Engine Design

Here's the current (April 2018) setup of the audio engine:

• There's a single Audio Engine, which manages audioBuffers used by all sprites
• Each target (sprite or clone) owns an Audioplayer, which manages audio effects.
• AudioPlayer creates a new SoundPlayer each time a sound is played.
• SoundPlayer is disposed at some point after the sound ends.
• AudioPlayer is disposed when the target is deleted.

A sprite and its clone share a single activeSoundPlayers list, which is used to

• Restart sounds already playing
• Stop all sounds
• Update the pitch effect

Proposed Re-design

The idea is to reduce the system down to just an AudioEngine that manages a set of SoundPlayers. Each sound created in Scratch is associated with a single persistent SoundPlayer object.

The Audio Engine creates and deletes these SoundPlayers to stay in sync with the VM.

Each SoundPlayer would know if it is currently playing, and when played, restart itself if it is already playing.

The Audio Engine can update a SoundPlayer's parameters (volume, pitch, pan) when the sound starts or while it plays.

When a sound is started by a target, the VM would use the parameters for that target to update the SoundPlayer.

Questions

When the VM updates sound parameters for a target (e.g. using a "set volume" block), the AudioEngine will need a way to update those parameters on all the SoundPlayers currently being played by that target. How should this work? Two ideas:

• The targets themselves (in the sound blocks in the VM) could keep track of the SoundPlayers they are currently using to play sounds.
  • It would be a bit complicated to keep this in sync, because one target can restart a sound that another target started, etc.
• Each SoundPlayer could store the ID of the target that most recently played it.
  • In that case, the question is how to efficiently update all the SoundPlayers for a particular target. Most simply, the AudioEngine would need to step through all of its SoundPlayers, checking each for the target ID. If necessary, the AudioEngine could provide a way to cache the results of this search for each target ID in a map, clearing this cache each time a sound is played, stopped or deleted.

The other big question is how this will affect other users of the AudioEngine, such as the sound library. Hopefully the changes to keep things working would be small.

Instantiating a new AudioEngine should not create a new audio context, because it causes problems like this scratchfoundation/scratch-gui#556

You can something similar to the shared audio context technique that the GUI uses.

Currently we cannot load wav files from existing Scratch projects when they have the adpcm format. Looks like the relevant decoder code from Scratch 2.0 is here

Expected behavior

• Should be able to play a long project with audio

Actual behavior

• Audio quality begins to degrade until eventually it "cuts out" and will not resume audio playback until refreshing the page

Steps to reproduce

• Load project https://llk.github.io/scratch-gui/#100736613
• Press green flag
• Sit through the entire animation 🐱
• Press green flag again

Operating system and browser

Mac OS 10.12.2 Chrome Version 56.0.2924.87 (64-bit)

Just started getting this deprecation warning

[Deprecation] GainNode.gain.value setter smoothing is deprecated and will be removed in M64, around January 2018. Please use setTargetAtTime() instead if smoothing is needed. See https://www.chromestatus.com/features/5287995770929152 for more details.

@ericrosenbaum can you determine if we want the instantaneous behavior (in which case I guess we keep the setter the way it is) or if we want a smoothed behavior (in which case we should switch to using setTargetAtTime)?

I'm adding this to the December milestone just to stay ahead of it, but I do not think (judging from the warning) that it would cause any buggy behavior if we let it be. But I'd like to get a decision on the books this month just so we know where we stand.

There's an old Scratch 1.4 block – I'll call it "MIDI instrument" since that's its op code (midiInstrument:). It's Scratch 1.4's version of the "set instrument" block.

In Scratch 2.0, the block looks like this:

The only visible difference is that one block has the obsolete colour and the other has the Sounds category colour, but they are stored differently in the project files and they do behave differently.

Scratch 2.0 has very few instruments in comparison to 1.4 – only 21 vs. 128! (No factorial.) So, to account for the limited sound options, Scratch 2.0 uses a mapping of 1.4 instrument IDs to 2.0 instrument IDs. (That's defined here.)

However, I think it's not very debatable to say that this isn't the right thing to do. Any old 1.4 projects that used instrument blocks are basically broken by this, and any future (well, current) musical projects are relatively more limited now.

The simple way to handle this would be to use the very same map along with a new sound primitive definition in the VM, but that's only a semi-fix. On the one hand, that'd make those 1.4 projects behave the same way they have for the last three-ish years. On the other hand, the projects still "don't work" – they still don't behave in the same way as their old user-created versions.

Arguably, this isn't even a problem – Scratch 2.0 has already been the current version for three and a half years (even longer when 3.0 is released!). Chances are very few people, if any, still care about the old versions of the projects, or the old instrument options. (I was hardly even around then – I certainly didn't care about those projects!)

But the real best solution is likely to come from the official reason these sounds aren't available anymore. Here's some text from an old post by Lightnin:

Unfortunately, for various reasons, Scratch 2.0 can't use those [MIDI] synthesizers, so we have to build our own synthesizer and musical instrument library.

Here's the limitations he's talking about:

We can only use SoundFonts that are public domain or available under a license that allows us to distribute them freely, such as a Creative Commons license.

We're limited in size. The entire library should be no more than two megabytes. Assuming we include 25 instruments, each instrument should be about 80k in size.

So one problem is file size. At this point I think it's alright to sacrifice a bit more data – remember, connections are always improving and browsers do cache files well!

So the real issue here is licensing, and a lack of the sounds.

I'd be interested in seeing how collecting those sounds might go. At this point, the Scratch community is larger than ever, so gathering the sounds might be easier than it was back in 2013 (there's now 10 times more active users!). I guess one of the problems might be consistency of quality, though – obviously sounds from different people are likely to have subtle recording/accuracy differences. The real best bet would be to have one person, or an organized group of people, record every sample, but 128 instruments (of which quite a few may be more difficult to get! There's quite a lot of synthesizers and such..) is quite a daunting number..

I wonder if there's some MIDI libraries already created online that have licenses that would be compatible with Scratch? That'd probably be the best way to get 128 accurate MIDI instrument samples.. :)

TL;DR revisiting the old MIDI sounds might be nice.

But I'm not the professional audio person here, nor a lawyer, so.. I'm really just bringing this up for discussion!

(edit: I'm assuming scratch-audio is the best place to put this!)
(edit2: This also applies to old drum sounds, I think.)

Some menus on blocks allow a reporter to be dropped onto them, to select an item for example with a variable or a "pick random" block. We could allow an effect to be selected by numeric index.

Ideally, all the audio effects would be applied at the level of each sprite or clone. In audio terms, each sprite or clone would have its own audio effects chain that includes all of the active audio effects, with its own settings for its own sounds.

Unfortunately, some of the audio effects are processor intensive: echo, reverb, fuzz, and robot. Instantiating them all for each of a potentially large number of sprites or clones would quickly cause audio dropouts or other glitches.

Right now, as a workaround, the AudioEngine creates a single instance of each of these four effects, through which all sounds are played. Any sprite or clone can set, change, or clear them, but the results affect all sounds.

(The pitch effect and pan effect are less processor intensive and so are instantiated by the AudioPlayer for each sprite or clone).

Some questions:

• Are there ways to re-implement the echo, reverb, fuzz and robot effects to make them less processor intensive?

• Should we dynamically create and destroy audio effects nodes when they are enabled and disabled by blocks?

• Should we dynamically create and destroy audio effects when a sprite or clone starts or stops playing sounds?

• Should we try a hybrid strategy that simulates having individual effects chains for sprites, using a single one? In that case, the single global effect would be set to the value requested by the sprite that most recently played a sound. I think this would require a data structure to track sounds as they start and end. For example, sprite 1 sets the reverb to 10, and starts a sound, causing global reverb to be set to 10; sprite 2 sets reverb to 50 and starts a sound, causing the global reverb to be set to 50; sprite 2's sound finishes but sprite 1's sound is still playing, so the global reverb is set to 10.

Expected behavior

The play sound block has a menu that should list the names of the sounds in the sprite.

Actual behavior

The menu contains a list of numbers 1 to 10.

Steps to reproduce

Click on the menu on the play sound block.

Operating system and browser

Mac OS 10.11.6 Chrome 56.0.2924.87

Request

A playground which would allow you to test scratch-audio.

Why

So you can test scratch-audio without relying on other modules to be working.

I think we can use a vocoder to create a robotic-sounding effect, perhaps by modifying the implementation here.

Another idea for making the input to the "play note" more intuitive and musical: support both numeric input of MIDI note numbers (as in Scratch 2.0) and also note name/octave notation like "C4" or "G#3" or "Eb2" (similar to "scientific pitch notation). We could also automatically convert valid note names without an octave to the default octave. So these would all play the same note:

Play note 60
Play note C4
Play note C

Expected behavior

When the loudness block is called and the microphone is connected, a value is returned.

Actual behavior

Nothing is returned, and an error is thrown: TypeError: this.mic.mediaStream is undefined

It looks like the error is coming from here:

Steps to reproduce

It looks like this error is spawning from this line:

// If the microphone is set up and active, measure the loudness
if (this.mic && this.mic.mediaStream.active) {

The microphone itself is set shortly before:

navigator.mediaDevices.getUserMedia({audio: true}).then(stream => {
    this.mic = this.audioContext.createMediaStreamSource(stream);

This is definitely weird, since mediaStream is defined in the WebAudio spec. Apparently a bug in Firefox. (I tested a comparable demo which uses createMediaStreamSource to take a look - yep, the mediaStream property is actually missing.)

So, browser specific. But worth looking into. I think you could just store the media stream in a separate property from this.mic:

+ this.audioStream = stream;
  this.mic = this.audioContext.createMediaStreamSource(stream);

~ if (this.mic && this.audioStream.active) {}

Operating system and browser

Firefox 58, 59. Debian (Testing).

We should create a small set of scratch projects that we can use to test audio performance.

Expected behavior

Deleting a clone should not cause a sound played by the parent sprite to stop playing.

Actual behavior

Deleting a clone stops the parent sprite's sounds.

Steps to reproduce

This stack causes the music to stop after 1 second:

from @SillyInventor:

Expected Behavior

Pitch and pan effects should change the frequency and speaker side intensity of instruments respectively.

Actual Behavior

No noticeable change.

Steps to Reproduce

Play noise or note, change pitch, replay noise or note.

Operating System and Browser

Ubuntu 16.0.4, Chrome 56.0.2924.87

Expected behavior

When you click a "play sound until done" block, it should highlight during the duration of the sound.

The block does have the correct timing behavior, waiting until the sound completes to continue to execution of the next block in the stack (so, for example, this stack highlights correctly: repeat 10 [play sound and wait]).

Actual behavior

No highlight appears.

Steps to reproduce

Drag out a "play sound until done" block and click on it

Operating system and browser

Mac OS 10.11.6 Chrome 56.0.2924.87

Expected behavior

Each time you play a sound, a webaudio buffersource node is created. After the sound ends, the node should be garbage collected.

Actual behavior

I'm not totally sure but I think that they can not be garbage collected. We store a reference to each buffer source node in order to set the playback rate while the sound plays (for the pitch effect), so at the very least these references need to be deleted after the sound ends.

Note that we cannot debug this using the Chrome Web Audio Inspector, because its own operation prevents garbage collection of nodes (see here)

Operating system and browser

Mac OS 10.11.6 Chrome 59.0

See scratchfoundation/scratch-gui#257

Even if we use scratch-storage for the actual loading, there will still need to be something to indicate that the sounds are decoded and ready for playing.

Expected behavior

Pressing mode button in https://llk.github.io/scratch-gui/#141331752/ should play a brief sound and switch mode.

Actual behavior

Freezes with warning:

Steps to reproduce

Open https://llk.github.io/scratch-gui/#141331752/ , press green flag, press mode (may have to press for a while to trigger toggle mode).

Operating system and browser

Ubuntu 16.04, chrome 56.0.2924.87

Work from develop as per our Git Flow workflow in other 3.0 repositories.

In addition to the built-in instruments (such as piano, flute, etc), the "set instrument" block could allow you to choose one of the sounds in the sprite. Then, the "play note" block would play this sound at different pitches by changing its playback rate. "Play note 60" (middle C) would play the sound unchanged, "play note 61" would shift it up a semitone, etc. The duration set by the "play note" block would stop the sound if necessary, with a short fade out to prevent clicks. So, for example, you could record your dog barking, and make it bark a melody, like happy birthday.

Early experiments with this idea are promising. Seems like fun.

One concern is that the pitch of the sounds produced in this way will depend on the pitch of the recorded sound, which could be confusing, especially if you are trying to use multiple sounds together, or in combination with the built in instruments.

Design questions:

• What should the menu look like? We'll need two sections of the menu, one for the built-in instruments, and one listing the sounds in the sprite.
• How does selecting an instrument by index work? i.e. currently you can select piano by setting the instrument to 1. How would you select the first sound in the sprite by number?
• One solution would be to modify the "set instrument" block so it has an additional menu, to choose between sprite sounds and instruments.

Expected behavior

It should be possible to stop a sound while it's playing without causing a click.

Actual behavior

We're getting click sounds in several situations when sounds are stopped. This includes stopping sounds using the stop button, using the "stop all sounds" block, stopping and restarting a sound by playing it in a loop, and while navigating the sound library by hovering over sounds.

Steps to reproduce

Click the sounds tab
Click "Add Sound"
Add the sound "C Bass"
Run the following stack:

Things to try

Fortunately, in all these clicky situations, the same stop() function is called, here. We should be able to add a gain node to each bufferSourceNode and schedule a very short ramp down before stopping it.

Operating system and browser

Mac OS 10.11.6 Chrome 58.0

Generally, when you run the "play sound" block multiple times with the same sound selected, it stops and restarts the sound. For example this stack plays a stuttering sound, restarting the sound many times per second, rather than playing many overlapping copies of the sound:

In Scratch 2.0, clones of a sprite share their sounds as if they were all being played by the parent sprite, so that if one clone starts a sound, and another clone then starts it, the sound restarts. So this stack also creates a stuttering sound:

In the current implementation for 3.0, each clone gets its own audioPlayer, so many clones can all play the same sound without restarting each other. But this probably creates performance and memory issues (I think this is the cause of #27, for example).

So probably we should change it so that clones don't have a full audioPlayer. Instead, each sprite has an audioPlayer, and clones would only get a reference to it.

If we do that, how do we handle the audio effects?

Generally, how would we implement this?

In readme.md:

npm install https://github.com/LLK/scratch-audioengine.git

Now it should be:

npm install https://github.com/LLK/scratch-audio.git

We plan to move the instrument and drum blocks to an official scratch extension. Specifically, it will include (at a minimum) these blocks:

play drum
rest
play note
set instrument
change tempo
set tempo
tempo

Existing projects that use these blocks will load and function as before (they will load the extension automatically).

Currently we have a temporary placeholder InstrumentPlayer, which plays notes using an existing soundfont library, soundfont-player. These soundfonts are high quality, but far too large in file size for our purposes, and for the moment they load on demand (i.e. at the moment the 'play note' block or 'set instrument' block runs, if the soundfont has not yet loaded, it loads, causing a delay of a few seconds). Other limitations of this approach include the fact that we cannot route these sounds through all of the audio effects (currently, they are routed through echo, reverb, fuzz and robot, but not pitch or pan), we cannot set the volume, and we cannot sustain the sound longer than the sample duration. All this will change once we decide on a new approach that gives us high quality sounds with a reasonable file size.

Expected behavior

The "set effect fuzz to 100" block should cause sounds to play with a nice fuzzy distortion.

The "set effect robot to 100" block should cause sounds to play with a buzzy robotic quality.

Actual behavior

These blocks have no effect.

Steps to reproduce

Go to https://llk.github.io/scratch-gui/ and create these stacks:

clear audio effects
set effect fuzz to 100
play sound 1

clear audio effects
set effect robot to 100
play sound 1

they should play the meow sound with fuzz and robot effect applied, respectively, but they don't.

Operating system and browser

Mac OS 10.11.6 Chome 56.0.2924.87

Expected behavior

It works without error.

Actual behavior

It prevents scratch from loading and throws an error.

Steps to reproduce

Try to load Scratch-VM/GUI playground in IE

Operating system and browser

Windows 10 IE

Screenshot

Expected behavior

I should be able to set an audio effect on a sprite, then clone the sprite, and the clone should have the same audio effect setting.

Actual behavior

Clones don't inherit audio effect values from their parent sprite.

Steps to reproduce

Create this code and press the green flag:

Clicking on the original sprite in the center will the meow with a high pitch, but clicking on the clone will play the sound without the pitch effect applied.

Operating system and browser

Mac OS 10.11.6 Chrome 59.0

We are missing our required TRADEMARK file as per the boilerplate:
https://github.com/LLK/scratch-boilerplate

Expected behavior

The green flag and stop button both clear the audio effects to 0. If you do these things, for example:

click [set pitch effect to 100]
press green flag
click [change pitch effect by 10]
click [play sound meow]

The meow sound should play with pitch effect set to 10.

Actual behavior

The meow sound plays with pitch effect set to 110. Apparently, the green flag and stop buttons are not completely clearing the sound effect values.

Not that the problem appears only after using "change effect." If you press the green flag and then play the sound, it correctly has the pitch effect set to 0. So the problem is with the stored value used by "change effect" to update it.

Steps to reproduce

See above.

Operating system and browser

Mac OS 10.11.6 Chrome 57

Expected behavior

The sound blocks should play sounds on an iPhone.

Actual behavior

No sound plays.

Steps to reproduce

Visit https://llk.github.io/scratch-gui/ on an iPhone and click on a play sound, play note, or play drum block.

Operating system and browser

iPhone 6, iOS 10.2, Chrome for iOS

Expected behavior

When you change the pan effect, and then play a sound, the sound should start playing at the new pan position.

Actual behavior

The pan position sounds like it is sliding from the old to the new position at the beginning of the sound, even if the pan value was set before the sound started.

Steps to reproduce

Open https://llk.github.io/scratch-gui/

Put on headphones, so you can hear the pan effect clearly.

Drag out three separate blocks:

Play sound 1 (should play the meow sound)

Set effect pan left/right 100

Set effect pan left/right -100

Click on pan 100, then play sound, then pan -100, then play sound.

You should hear the sound play in your right ear, then in your left ear, and so on.

Instead, after the pan value has been changed, at the start of the sound you hear it slide quickly from its previous pan position to the new position.

Operating system and browser

Mac OS 10.11.6 Chrome 56.0.2924.87

Expected behavior

If you use several loudness blocks, they should not cause the loudness analysis code to run multiple times in a single scratch animation frame.

Actual behavior

Each time a loudness reporter is called it runs the loudness analysis.

Steps to reproduce

This program uses a run-without-screen-refresh custom block to check the loudness 100,000 times, and measure the time it takes:

On Scratch 2.0 on my machine, this reports 0.08 seconds, and on Scratch 3.0 it reports about 0.45 seconds.

Notes

One strategy for improving this would be store a timestamp each time the loudness is requested, and also store the loudness value itself. Then, each time, if the current time is long enough after the timestamp (e.g. one frame at 30fps- since we are not measuring the platform's audio buffer duration), then make a new measurement- otherwise just return the stored value. For the program above, this should result in the loudness analysis being done once each time the custom block is run, instead of 100,000 times.

Operating system and browser

Mac OS 10.12.6 Chrome 63

Expected behavior

Loudness works consistently on all supported platforms

Actual behavior

From feedback form:

On an iPad microphone seems to either detect noise in the loudness block or it doesn't.

Steps to reproduce

1. Use the loudness block on iPad
2. ?

Operating system and browser

iPad

This library has a problem of browser compatibility. audio.createStereoPanner() is not surpported by safari.

Currently, we have a prototype implementation that uses the drum sounds from Scratch 2.0. We should consider creating a set of new, higher quality drum sounds.

Expected behavior

Speech recognizer should not stop recognizing speech.

Actual behavior

Sometimes the speech reporter stops updating.

Screenshots from one of the times when it stopped:

Steps to reproduce

Will update if we figure them out.

Operating system and browser

Chrome, Mac

We are using the Tone.js library for several things, but we don't need most of it. Eventually we should transition away from it, which will require recreating some parts of it, including:

• Audio buffer loading
• Audio effects superclass
• The audio effects themselves (panner, delay, reverb, distortion, robotic)
• Utility for calculating a frequency for a midi note
• Utility for calculating frequency ratio for a pitch ratio

The "play sound until done" block should resolve and de-highlight at the time the sound completes. Currently it predicts the end time of the sound using the duration and the playback rate at the time the sound is started. If you use a "set pitch effect" block to change the playback rate after starting the sound, this timing is incorrect.

I'm hoping we will be able to use the onended event of the AudioBufferSourceNode, though this is not directly exposed in the way that we are using ToneJs.

We might want to have a global compressor at the end of the effects chain, where the sound from all the sprites gets combined. This would help prevent clipping by reducing the volume if the sounds get too loud.

Expected behavior

The drum sounds should be loaded like other static assets, using scratch-storage.

Actual behavior

I got them working in a hacky way by loading them from the github repo itself. See here.

Reported during Extensions Internal Playtest (6/16/17).

Expected Behavior

The Speech extension should have working speech synthesis and recognition.

Actual Behavior

@thisandagain's speech recognition never worked:

Speech synthesis worked, but speech recognition never worked

Steps to Reproduce

I am not sure.

Operating System and Browser

We were all looking at https://ericrosenbaum.github.io/scratch-gui/ via Chrome, Mac. This behaviour was seen on @thisandagain's computer.

Expected behavior

If the microphone input for the loudness block is connected and then disconnected, it should automatically attempt to reconnect.

Actual behavior

The mic input stays disconnected.

Steps to reproduce

• Show the monitor for the loudness block
• Put your computer to sleep
• Re-awaken it
• Loudness monitor now displays -1 (default value when no mic is connected)

Operating system and browser

Mac OS 10.12.6 Chrome 61

Expected behavior

To match the behavior of Scratch 2.0, the green flag and stop button should not reset the volume for a target.

Actual behavior

Green flag and stop button currently do reset the volume.

Steps to reproduce

The issue is described with repro here: scratchfoundation/scratch-vm#927

It should be as simple as removing this line:
https://github.com/LLK/scratch-audio/blob/84fdf5c76a41a4960e34282ea96687d9ef19d6aa/src/index.js#L125

Expected behavior

Clicking the stop sign twice with a sound playing, should stop the sound with no errors.

Actual behavior

The first click stops the sound, the second click throws an exception and stuff stops working

Steps to reproduce

Add a click-flag-hat and a play-sound-until-done under it. Click the green flag, click the stop button twice.

Operating system and browser

macOS and Safari 11

Notes

This can be fixed by changing the check in https://github.com/LLK/scratch-audio/blob/develop/src/SoundPlayer.js#L51 to if (this.bufferSource && this.isPlaying).

If possible I guess it should check if the buffer source is playing, but I'm not sure that's possible.

We'll need a set of 18 drum sounds to match the scratch 2.0 play drum block. We can start with the existing sounds:

https://github.com/LLK/scratch-flash/tree/master/src/soundbank/drums

And optionally work on replacing them with higher quality sounds.