Testing modifications to the sound engine

kwx · September 4, 2020, 4:22am

Following up on my previous sound post (thanks @richard for helping with that), I’ve started poking at the sound engine to see how hard it would be to implement some waveforms similar to the 8-bit Atari’s POKEY pseudorandom square waves, and while I found a few likely-looking code locations where I added experiments, I’m getting a bit stuck trying to test them.

For example, I found squareWaveTone in pxt-common-packages/libs/mixer/melody.cpp, which I’d guess is used on actual devices, and a separate JS implementation in pxt/pxtsim/simlib.ts which is presumably for the simulator.

However, if I modify these files and run a local server, it’s not picking up the changes in the emulator. I seem to be getting redirected to https://trg-arcade.userpxt.io/---simulator, and I see a reference in pxt-arcade’s package.json to an optional pxt-arcade-sim package to which I don’t have access. (I did do the pxt linking from pxt-arcade to ../pxt and ../pxt-common-packages.) I’m not sure how to activate the localWebConfig setting in pxt/pxtlib/main.ts, and the redirect still happened after I hardcoded a local sim/simulator.html URL in
pxt-arcade/share/src/components/simulator.ts.

(I can’t test on actual hardware yet, my Adafruit PyGamer is still in transit.)

By the way, I’m really enjoying experimenting with MakeCode Arcade, and this isn’t very urgent, but I got caught up in the fun of trying to figure out how it works behind the scenes

richard · September 4, 2020, 4:09pm

I’m afraid the simulator is the one bit of arcade that is not open source.

kwx · September 4, 2020, 7:10pm

Ah, that explains me not being to find it. Would you still be potentially interested in looking at a contributed patch for the sound code? That’s assuming I can get my idea working on real hardware. I can try to set up a standalone JS demo that’s roughly based on the code in pxt/pxtsim/simlib.ts to show what it would sound like.

richard · September 4, 2020, 7:24pm

Actually, you should be able to make this work by working with maker.makecode.com instead of arcade. The simulator implementation is in pxt-core so you should be able to see your changes when serving locally as long as you have your node_modules linked. The same music.playInstructions trick should work in that editor too.

The maker repo is here: https://github.com/microsoft/pxt-maker

richard · September 4, 2020, 7:27pm

Also, yes! We take PRs. Just make sure you don’t use any copyrighted code in your implementation.

kwx · September 4, 2020, 11:19pm

Thank you, I got it working via pxt-maker.

Here’s a recording (warning: loud, watch your speaker volume) of what it currently sounds like in the simulator, using the same logic I’m working on adding to the native music.cpp: https://drive.google.com/file/d/1zyO7kYi0jcFLQPTHptQ84AoagQWpPBRX/view?usp=sharing

namespace music {
    //% shim=music::queuePlayInstructions
    export function playInstructions(timeDelta: number, buf: Buffer) { }
}

// ww = waveform
// ffFF = frequency, low byte first
// ddDD = duration in milliseconds
// vvVV = volume
// wwWW = end volume
// ggGG = end frequency
// 
//  ww -- ffFF ddDD vvVV wwWW ggGG
const sound: Buffer = hex`
    05 00 b801 0004 0008 0000 dc00 // white noise
    04 00 ffff 0004 0008 0000 ffff // new noise at max frequency
    0f 00 b801 0004 0008 0000 dc00 // square wave, descending 440-220 Hz
    04 00 b801 0004 0008 0000 dc00 // new noise, same descending pitch 
    04 00 dc00 0004 0008 0000 7600 //  ... an octave lower
    04 00 b801 0004 0008 0000 b801 //  ... holding at 440 Hz
    04 00 4000 0004 0008 0000 4000 //  ... holding at 64 Hz
    04 00 0004 0004 0008 0008 0000`; //... large sweep, fixed volume
music.playInstructions(0, sound)

(I added comments to document the sound samples, they’d need to be removed to actually run it. I shared the code at https://makecode.com/_K93JfjPwrep4 for reference.)

What do you think? I’ll look into making a pull request once I’ve had a chance to test it on actual hardware.

jwunderl · September 4, 2020, 11:23pm

These sound fantastic to me

peli · September 5, 2020, 10:30am

Sounds authentic!

kwx · September 5, 2020, 8:41pm

This is fun. I now also added a cycle-based generator, using fixed pseudorandom sequences of length 16/32/64 waves. These have a sound in between pure notes and noise. This is inspired by the Atari POKEY which uses 15/31 length cycles, but power-of-two cycles are easier to implement in native code since the upper 6 bits of tonePosition are already effectively a cycle counter, so this can just use them directly.

(Warning: loud sound effects)

Here’s the corresponding input: https://makecode.com/_JVoa3Peo20as

// ww = waveform
// ffFF = frequency, low byte first
// ddDD = duration in milliseconds
// vvVV = volume
// wwWW = end volume
// ggGG = end frequency
ww -- ffFF ddDD vvVV wwWW ggGG

const sound: Buffer = hex`
0f 00 7003 0004 0004 0000 7003 // 880 Hz square wave
10 00 7003 0004 0008 0000 7003 // 16-cycle
11 00 7003 0004 0008 0000 7003 // 32-cycle
12 00 7003 0004 0008 0000 7003 // 64-cycle
04 00 7003 0004 0008 0000 7003 // pseudorandom

0f 00 b801 0004 0004 0000 b801 // 440 Hz
10 00 b801 0004 0008 0000 b801 // ... sequence as above
11 00 b801 0004 0008 0000 b801
12 00 b801 0004 0008 0000 b801
04 00 b801 0004 0008 0000 b801

0f 00 0004 0004 0004 0008 0000 // Sweep
10 00 0004 0004 0008 0008 0000
11 00 0004 0004 0008 0008 0000
12 00 0004 0004 0008 0008 0000
04 00 0004 0004 0008 0008 0000

richard · September 10, 2020, 4:50pm

@mmoskal you might find this thread interesting! (he implemented the existing synthesizer stuff)

kwx · September 10, 2020, 5:02pm

BTW, my hardware has arrived, but I’m currently stuck on being unable to test my changes, music.playInstructions appears to hang the Adafruit PyGamer even when using the original code. I filed https://github.com/microsoft/pxt-arcade/issues/2423 to follow up on that.

I haven’t gotten hiddmesg or WebUSB working yet in my CLI setup, so I don’t know if it’s reporting any error messages. I’ll poke around some more if I find time.

kwx · September 24, 2020, 1:57am

Sorry about the delay, but I finally got pull requests uploaded:

I used the patched simulator to record the sound samples earlier in this thread, but unfortunately I haven’t been able to test it on actual hardware due to the crash issue I mentioned in my previous comment.

Obviously I’d prefer to make sure this actually works before proceeding further, but I wanted to get an initial upload out for early feedback. What do you think?

kwx · September 28, 2020, 6:57pm

Thanks to @mmoskal’s help (including fixing the crash issue), the new waveforms are now live on the MakeCode Arcade beta server. Here’s an example:

open https://makecode.com/_7CdbiE2q0FkL
press the Edit Code button
wait for the editor to load
edit the URL to add /beta so that it reads https://arcade.makecode.com/beta#editor
run in simulator or upload to a device as usual

Enjoy!

jwunderl · September 28, 2020, 7:04pm

@kwx Here’s a link that loads directly into the beta editor: http://arcade.makecode.com/beta#pub:_7CdbiE2q0FkL (it might bump you back to the home page if you get an update when loading though, but the project should be there still)! For anyone trying this, note that the beta editor is, well, a beta – it contains updates that we’re working on / haven’t released yet, so there will be things we haven’t finished / fully tested yet.

mmoskal · September 29, 2020, 9:14am

Someone could try to put a sound-effect extension together, to make it easier to use, see https://makecode.com/extensions/getting-started

The buffers can be created dynamically and numbers can be set with Buffer.setNumber(). One can also use the literal for base buffer and patch it up, like this:

let buf = hex `10 00 7003 0004 0008 0000 7003`
buf = buf.slice() // make the buffer read-write
buf.setNumber(NumberFormat.UInt16LE, 2, duration)

kwx · September 30, 2020, 7:38am

Thanks for the tip! One small gotcha is that setNumber takes a byte offset, so the duration is at offset 4.

I haven’t looked into the details of making extensions yet, but here’s an experiment with a simple wrapper to make working with buffers a bit easier: https://arcade.makecode.com/beta#pub:_4wPgDTUFMcfX

Example:

controller.A.onEvent(ControllerButtonEvent.Pressed, function () {
    // Use attack/decay/sustain/release for volume
    let s1 = new SoundSequence(4);
    s1.addSound(16, 200, 440, 440, 0, 900);
    s1.addSound(16, 200, 440, 440, 900, 500);
    s1.addSound(16, 300, 440, 440, 500, 500);
    s1.addSound(16, 1600, 440, 440, 500, 0);
    // Second sound at a slightly shifted frequency
    let s2 = new SoundSequence(4);
    s2.addSound(16, 200, 444, 444, 0, 900);
    s2.addSound(16, 200, 444, 444, 900, 500);
    s2.addSound(16, 300, 444, 444, 500, 500);
    s2.addSound(16, 1600, 444, 444, 500, 0);
    s1.play()
    s2.play()
})

kwx · October 5, 2020, 7:07am

Here’s my attempt at making an extension for using the sound effects from Blocks - I’m new to this, so not sure about best practices.

The following demo uses just waveforms available in the stable version:

And here’s a demo using the new waveforms, this needs the beta editor: https://arcade.makecode.com/beta#pub:_exXFsgWdKPcA

LCProCODER · October 5, 2020, 7:13am

Those sound effects are cool!

kwx · October 6, 2020, 5:36pm

I’ve tweaked the API a bit and released v1.0.0 of the extension. It now uses wave numbers (instead of a custom type) to make it possible to store them in variables or arrays. Here’s a demo using this, move the stick left/right to switch between the supported waveforms:

(On the off chance that someone has been using the previous version of the extension, sorry about the incompatible change, you’ll need to update blocks referring to waveforms if you update to the latest version of the sound-effects extension.)

It also adds support for white noise - this is at a fixed pitch and doesn’t react to frequency changes, but it’s available in the current stable version. The new tunable noise needs the beta version of MakeCode Arcade: https://arcade.makecode.com/beta#pub:_JL6P9m5fYg9u

If you want to use it in your own project, click on Advanced / Extensions in the editor’s blocks menu, and copy the extension’s address into the search field:
https://github.com/klausw3/pxt-sound-effects

That’ll add the “Sound Effects” blocks to the editor. The “Cycle” and “Tuned Noise” waveforms only work in the beta editor, the others should work on all versions.

jwunderl · October 6, 2020, 5:58pm

Nice!

One thing, right now our tilemaps don’t work perfectly with extensions - names can conflict if you add the extension after the user already defined their own tilemap. I would suggest either removing the tilemap files from the extension, or moving them to testFiles like this: https://github.com/jwunderl/arcade-tilemap-a-star/commit/ae3d0c9576c4b1d14c410c58d226cf1fa4fe87f9 so the extension doesn’t randomly get errors in projects