SFX in MakeCode Arcade?

I’m poking around a little bit at the internals of MakeCode Arcade. I like the ability to play melodies, but I feel like we’re missing the “pew pew” and “crash bang” types of sound effects that are also typical in arcade games.

Specifically, I’m looking for a way to play WAV file data like what is done in MakeCode for Lego Mindstorms EV3. In Arcade, it looks like music.playInstructions() plays sound from a buffer. Does it work the same way as sounds.play() works in Mindstorms, or is there some other way to hook into the sound system in Arcade to play WAV data?

2 Likes

We still have some work to do around sound. Eventually we would like to have a sound designer similar to the sprite image editor so that kids can “shape” their own sounds.

4 Likes

Brilliant; thanks for the update, @peli. A sound designer would be super cool! A set of stock sounds, though, would be perfectly sufficient to begin with, similar to the stock melodies list.

Both are related: we need an api to shape sounds in order to define those built-ins.

3 Likes

I’d like to play wav files too. I would rank wav/sample support higher than synthesis features. Have you conducted user surveys for this?

For comparison, MIT Scratch offers a rich sample library, import of wavs, recording and playback.

2 Likes

Has there been any progress? Many kids would like this.

3 Likes

The engine already has support for creating basic sound effects from waveforms with frequency and volume sweeps using music.playInstructions(), but this didn’t have a Blocks interface, and was a bit clunky to use from JavaScript since it involves modifying buffers.

I’ve recently added engine code for new waveforms for more sound effect variety, including tunable noise and distorted square waves. This is available in the beta editor but not yet on the main site. There’s more information in this rather long thread. You can try a demo using this link to the beta editor: https://arcade.makecode.com/beta#pub:_JL6P9m5fYg9u

This uses the pxt-sound-effects extension which adds a Blocks interface to make this a bit more accessible:

If you want to use it in your own project, click on Advanced / Extensions in the editor’s blocks panel, and copy the extension’s address into the search field:
https://github.com/klausw3/pxt-sound-effects

The extension works for the current version of MakeCode Arcade, but the new waveforms (tunable noise and cycle 16/32/64) will just produce silence if you’re not using the beta editor.

2 Likes

Just to clarify, my previous answer wasn’t quite what the original poster was asking for. music.playInstructions() uses small buffers of instructions for controlling synthesized waveforms. There’s no support currently for playing samples or WAV data directly, and I’m not convinced that adding that functionality would be a good idea since these use a lot of memory and storage space.

However, I think you can go a long way with synthesized sound effects if there’s enough variety in waveforms. The code I added was intended to sound similar to old 8-bit computers such as the Atari POKEY sound chip, here’s an example what it sounds like. (Warning: loud. Watch your speaker volume.)

I think this isn’t quite comparable. As far as I know MIT Scratch runs on modern computers, while MakeCode Arcade targets comparatively low-powered standalone hardware or emulated versions of that. Sampled audio requires about 22 kBytes for a single second of mono 8-bit audio at 22 kHz, and that’s a lot when some target platforms may only have 96 kByte of RAM available.

Last I checked, I think that LEGO Mindstorms cheats a bit and plays sound effects on the tablet which is controlling the robot, so the samples don’t need to be downloaded onto the hardware platform. The downside of that is that the programs don’t work standalone. (I may be out of date for this, that’s what I saw last time I used LEGO Boost and WeDo.)

4 Likes

micro:bit v2 introduced a new SoundExpression engine that sounds really good.

2 Likes

Out of curiosity, do you have a link to a SoundExpression sample or demo? I’m having trouble finding any details about it other than a brief mention in the announcement. There’s a search result pointing to https://makecode.microbit.org/pkg/microbit-foundation/pxt-microbit-v2-extension, but I get a 404 when trying to access that.

Is this related to the micropython 1.0.1 Audio API? That does appear to be quite powerful, though running user-provided waveform-generating callbacks reliably at 244 Hz seems challenging in a cooperative-multitasking environment.

If you go to https://makecode.microbit.org/beta, scroll down to the bottom of music drawer and try “play giggle”. This is still a recorded WAV file (from piping sound through serial on a microbit v2) until we have a web audio engine.

The CODAL synth engine is at https://github.com/lancaster-university/codal-microbit-v2/blob/master/inc/SoundExpressions.h

@AlexK just released a little RTTL parser library

3 Likes

Thanks, looks like this is the implementation file: https://github.com/lancaster-university/codal-microbit-v2/blob/master/source/SoundExpressions.cpp

If I’m understanding it right this looks like a powerful way to control frequency and volume for effects, but it’s still controlling an underlying synthesizer with sine/sawtooth/triangle/square/noise waveform choices? If that’s the case, I think it would potentially combine very nicely with the cycle noise waveforms I had added which could add more variety in the underlying sounds.

This is awesome, @peli! It’ll make building melodies a lot easier for those of us who know musical notation. Plus, being able to store melodies in variables will give advanced coders a lot more flexibility. Bravo!

P.S. I love your GitHub avatar. It cracks me up every time I see it.

FYI, you can do that already, MakeCode has its own string-based notation. peli@'s extension converts RTTL format into the MakeCode syntax which makes it possible to use pre-existing melodies in that format, but if you’re making melodies from scratch I think the existing MakeCode format may be a bit more powerful since it’s more closely tied to the way the underlying sound engine works.

Here’s an example from a test I was working on. FWIW, it doesn’t currently work great since the different instruments quickly get out of sync, it would need a different approach and not just simple repeated loops.

const hihat = new music.Melody("@0,50,0,0 ~5 c8-240 c8 c c c c c @0,250,0,0 c")
const snare = new music.Melody("@10,75,0,0 ~5 r-240 r g5 r r r g r")
const bass = new music.Melody("@10,120,80,0 ~15 c2-120 d# f f# g f# f d#")
const guitar = new music.Melody("@100,100,160,0 ~15 c4-240 g r r c5 g r r c5 b4 g4 f# e d#-120 e-240")
3 Likes

WHAT! Multi-channel audio! Love it!

Something I hadn’t realized with the built-in melody block is that you can store the melodies as strings and then play them back. @peli 's extension makes that capability much more obvious, albeit using the RTTL notation.

Thanks for sharing that!

2 Likes

That is awesome. If only someone, obviously very smart, were able to make a music editor based on this idea. :wink:

1 Like