# Help with custom audio clips...again

**URL:** <https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384>\
**Category:** Help\
**Tags:** art, music\
**Created:** [August 31, 2024, 6:49pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384 "2024-08-31T18:49:59Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![UnsignedArduino](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/unsignedarduino/32/592_2.png) [@UnsignedArduino](https://forum.makecode.com/u/UnsignedArduino)\
**Post date:** [August 31, 2024, 6:49pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/1 "2024-08-31T18:49:59Z")

</div>

For some reason, I decided to try to play custom audio clips in MakeCode Arcade, completely ripping off of @chembot’s [idea](https://forum.makecode.com/t/custom-audio-clips-possibly-using-queueplayinstructions/28168).

Instead of using SPEAR / complicated audio analysis tools, I decided to use [`scipy`’s `signal.spectrogram`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.spectrogram.html#scipy.signal.spectrogram) function to decompose to sine waves and play, which is a more basic version. You can see the GitHub repo [here](https://github.com/UnsignedArduino/Audio-to-MakeCode-Arcade).

The thing I need help with, or would like the input and ideas of others, is that I’ve tried two methods to play the sine waves and I want to improve the sound quality of the second method… (both only tested in Chrome)

* * *

⁉ The first method, is storing two giant `number[][]` arrays of most prominent frequencies and their respective amplitudes, which actually turned out decently: (although it is not playing at full speed, more so ~92% speed…)

> **[never gonna give you up (number arrays)](https://arcade.makecode.com/S01097-41260-88616-24657)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.

(make sure to interact with the page so sound can play. Also it’s quite laggy for the first few seconds as stuff loads in)

I generated it with the code in commit [`04cb7a76510edee890c0f37265cff0d13940a6d3`](https://github.com/UnsignedArduino/Audio-to-MakeCode-Arcade/tree/04cb7a76510edee890c0f37265cff0d13940a6d3) and setting the period to 25. (20 sounded worse, but I left it as the default accidentally)

* * *

💩 The second method, is using multiple long sound instructions to play the sine waves, which I thought would be better but it sounds much worse in that it’s “wobbly” and has a lot of extra sounds and noises:

> **[never gonna give you up (sound instructions)](https://arcade.makecode.com/S43942-91511-21747-08445)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.

I generated it with the code in commit [`7cee335b09a4ecfd57332e53af4c4897c429f1e4`](https://github.com/UnsignedArduino/Audio-to-MakeCode-Arcade/tree/7cee335b09a4ecfd57332e53af4c4897c429f1e4). You can see that I generate the sound instructions according to the format specified in the [docs](https://arcade.makecode.com/developer/sound).

I was wondering if anyone had any ideas on why the sound instruction method is much worse and if there are any ways to improve the quality. I’ve tried different frequencies of sine waves, which wasn’t much better. Maybe there are some limitations I don’t know about or ways I’m abusing the sound instruction API incorrectly. (honestly, there’s a good chance that there is a simple tweak and I’m just incompetent lol)

@richard or anyone else on the dev team, (sorry for the ping) if you have any ideas or thoughts, I would love to hear them! Especially on my ~~ab~~ use on the sound instruction API / format.

Note: I don’t envision this being used actually in games, as this probably takes up a significant / almost all of the simulator’s power just to play the sine waves, which wouldn’t be very good if you were in a game that also needs to process stuff. This is also just lower quality in general, and it would probably be better to use the Song API and MIDI conversion for games instead. (speaking of which, I should probably get to work on some day… 🤔) And this would probably be a PITA for moderation of games - I’m just trying to see what’s possible with MakeCode Arcade. 😁

---

<div class="post-metadata">

**Author:** ![Brohann](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/brohann/32/17380_2.png) [@Brohann](https://forum.makecode.com/u/Brohann)\
**Post date:** [September 1, 2024, 1:01am UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/2 "2024-09-01T01:01:21Z")

</div>

nice converter!

> **[Bob omb Battlefield](https://arcade.makecode.com/52968-09746-53206-04800)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.

I hope this gets to a point where it can be used for sound effects in games (not background music obviously)

---

<div class="post-metadata">

**Author:** ![Brohann](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/brohann/32/17380_2.png) [@Brohann](https://forum.makecode.com/u/Brohann)\
**Post date:** [September 2, 2024, 2:53am UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/3 "2024-09-02T02:53:39Z")

</div>

I was bored again

> **[Dire Dire Docks](https://arcade.makecode.com/10652-06515-35006-06167)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.

---

<div class="post-metadata">

**Author:** ![richard](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/richard/32/5417_2.png) [@richard](https://forum.makecode.com/u/richard)\
**Post date:** [September 4, 2024, 6:24pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/4 "2024-09-04T18:24:02Z")

</div>

@UnsignedArduino neat project!

first off, the clipping noisy sounds are being caused by rapidly switching the amplitude. you’ll get better results if you smoothly transition from the previous volume to the next for each sample instead of just abruptly changing from one volume to the next (i.e. for each instruction, set the start volume to the end volume of the previous instruction)

secondly, an interesting experiment might be to divide the sound into frequency bands instead of just taking the X highest amplitudes at each point in time.

what i mean by this is to basically divide the spectrum into buckets and then for each bucket choose the highest amplitude sample that is within that frequency range. this is how old school vocoders work. for example, here’s a classic piece of music tech:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/makecode/original/3X/9/5/95b213ea3902672b153bc571020b1d7f96f56480.jpeg)

each of those columns in that image represents a frequency in HZ. these numbers i believe were chosen to cover a pleasing range for the human ear, but you might experiment with different cutoff points for each of the buckets (or programmatically determine them based on the source audio).

I don’t know if that will produce better results, but it’s worth a try!

---

<div class="post-metadata">

**Author:** ![UnsignedArduino](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/unsignedarduino/32/592_2.png) [@UnsignedArduino](https://forum.makecode.com/u/UnsignedArduino)\
**Post date:** [September 5, 2024, 1:10am UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/5 "2024-09-05T01:10:22Z")

</div>

> [@richard](#):
>
> first off, the clipping noisy sounds are being caused by rapidly switching the amplitude. you’ll get better results if you smoothly transition from the previous volume to the next for each sample instead of just abruptly changing from one volume to the next (i.e. for each instruction, set the start volume to the end volume of the previous instruction)

Tried it, produced [slightly better results](https://arcade.makecode.com/S04444-66733-71662-77434) but still the noise is there.

> [@richard](#):
>
> I don’t know if that will produce better results, but it’s worth a try!

The bucket method seemed like a good idea (so I put it in a [branch](https://github.com/UnsignedArduino/Audio-to-MakeCode-Arcade/tree/frequency-buckets)) but this result is…[interesting](https://arcade.makecode.com/S03274-24248-63984-00991).

---

<div class="post-metadata">

**Author:** ![randomuser](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/randomuser/32/32505_2.png) [@randomuser](https://forum.makecode.com/u/randomuser)\
**Post date:** [September 5, 2024, 4:07pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/6 "2024-09-05T16:07:42Z")

</div>

THIS IS SO AWESOMEEE!!! I love custom audio in makecode, this has so many applications! I hope you can get it worked out!

---

<div class="post-metadata">

**Author:** ![UnsignedArduino](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/unsignedarduino/32/592_2.png) [@UnsignedArduino](https://forum.makecode.com/u/UnsignedArduino)\
**Post date:** [September 5, 2024, 4:07pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/7 "2024-09-05T16:07:43Z")

</div>

I also [tried](https://arcade.makecode.com/S43052-79049-97191-10311) generating the sound instructions during the program by taking the list of frequencies and amplitudes and generating a buffer, it sound as bad as the Python-generated hex buffers. So maybe I’m messing something up with sound instructions.

---

<div class="post-metadata">

**Author:** ![richard](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/richard/32/5417_2.png) [@richard](https://forum.makecode.com/u/richard)\
**Post date:** [September 5, 2024, 4:09pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/8 "2024-09-05T16:09:53Z")

</div>

@UnsignedArduino I actually did some experiments of my own last night and I found the frequency bucket method works really well, you just need to make sure that you keep it so that each instruction buffer is dedicated to a single bucket; the wobbliness is caused by big shifts in frequency between steps so if you keep each “thread” of audio within a limited frequency range you get better results. I also don’t have the clipping noise issues in mine

---

<div class="post-metadata">

**Author:** ![richard](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/richard/32/5417_2.png) [@richard](https://forum.makecode.com/u/richard)\
**Post date:** [September 5, 2024, 4:34pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/9 "2024-09-05T16:34:38Z")

</div>

to generate the fft data, I used [sox](https://manpages.org/sox), like so:

```auto
sox input.wav -n stat -freq &> file.dat

```

and here’s the script (in JavaScript):

```javascript
import * as fs from "fs";
import * as path from "path";

const NumberFormat = {
    Int8LE: 1,
    UInt8LE: 2,
    Int16LE: 3,
    UInt16LE: 4,
    Int32LE: 5,
    Int8BE: 6,
    UInt8BE: 7,
    Int16BE: 8,
    UInt16BE: 9,
    Int32BE: 10,
    UInt32LE: 11,
    UInt32BE: 12,
    Float32LE: 13,
    Float64LE: 14,
    Float32BE: 15,
    Float64BE: 16,
};

const source = path.resolve("file.dat");
const data = fs.readFileSync(source, "utf-8").split("\n");
const sourceLength = 39.594271;

const buckets = [
    50,
    159,
    200,
    252,
    317,
    400,
    504,
    635,
    800,
    1008,
    1270,
    1600,
    2016,
    2504,
    3200,
    4032,
    5080,
    7000,
    9000
]

const fft = [];
let spectrum = [];

let maxAmp = 0;

for (const line of data) {
    if (!line.trim()) continue;
    const [rawFreq, rawAmp] = line.split(/\s+/);
    if (!rawFreq || !rawAmp) continue;

    const freq = parseFloat(rawFreq) * 2;
    const amp = parseFloat(rawAmp);

    if (isNaN(freq) || isNaN(amp)) continue;

    if (spectrum.length && freq === 0) {
        fft.push(spectrum);
        spectrum = [];
    }

    spectrum.push({
        freq, amp
    });
}
fft.push(spectrum);

const timePerSample = (sourceLength / fft.length) * 1000;

const numThreads = buckets.length;
const threads = [];

for (let i = 0; i < numThreads; i++) {
    threads.push([]);
}

for (let i = 0; i < fft.length; i++) {
    let prev = 0;
    for (let j = 0; j < buckets.length; j++) {
        const [sample] = pickHighest(fft[i], prev, buckets[j], 1);
        maxAmp = Math.max(sample.amp, maxAmp);
        threads[j].push(sample);
        prev = buckets[j];
    }
}

const bufs = threads.map(t => samplesToBuffer(t));

const out = `
namespace music {
    //% shim=music::queuePlayInstructions
    export function queuePlayInstructions(timeDelta: number, buf: Buffer) { }
}

const instructions = [
${bufs.map(b => ` hex\`${b}\`,`).join("\n")}
]

for (const thread of instructions) {
    music.queuePlayInstructions(100, thread);
}
`

console.log(out);

function pickHighest(spectrum, min, max, num) {
    spectrum = spectrum.filter(s => s.freq >= min && s.freq < max);

    const out = [];

    for (let i = 0; i < num; i++) {
        out.push(spectrum[i]);
    }

    out.sort((a, b) => a.amp - b.amp);

    for (let i = num; i < spectrum.length; i++) {
        const current = spectrum[i];

        if (out[0].amp < current.amp) {
            out.shift();
            out.push(current);
            out.sort((a, b) => a.amp - b.amp);
        }
    }

    return out;
}

function samplesToBuffer(samples) {
    const buf = new Uint8Array(samples.length * 12);

    let prevAmp = 0;
    for (let i = 0; i < samples.length; i++) {
        const { freq, amp } = samples[i];
        const scaledAmp = (amp / maxAmp) * 1023;
        addNote(buf, i * 12, timePerSample, prevAmp, scaledAmp, 3, freq, freq)
        prevAmp = scaledAmp;
    }

    return toHex(buf);
}

function toHex(bytes) {
    let r = ""
    for (let i = 0; i < bytes.length; ++i)
        r += ("0" + bytes[i].toString(16)).slice(-2)
    return r
}

function addNote(sndInstr, sndInstrPtr, ms, beg, end, soundWave, hz, endHz) {
    if (ms > 0) {
        setNumber(sndInstr, NumberFormat.UInt8LE, sndInstrPtr, soundWave)
        setNumber(sndInstr, NumberFormat.UInt8LE, sndInstrPtr + 1, 0)
        setNumber(sndInstr, NumberFormat.UInt16LE, sndInstrPtr + 2, hz)
        setNumber(sndInstr, NumberFormat.UInt16LE, sndInstrPtr + 4, ms)
        setNumber(sndInstr, NumberFormat.UInt16LE, sndInstrPtr + 6, beg)
        setNumber(sndInstr, NumberFormat.UInt16LE, sndInstrPtr + 8, end)
        setNumber(sndInstr, NumberFormat.UInt16LE, sndInstrPtr + 10, endHz);
    }
    return sndInstrPtr
}

function fmtInfoCore(fmt) {
    switch (fmt) {
        case NumberFormat.Int8LE: return -1;
        case NumberFormat.UInt8LE: return 1;
        case NumberFormat.Int16LE: return -2;
        case NumberFormat.UInt16LE: return 2;
        case NumberFormat.Int32LE: return -4;
        case NumberFormat.UInt32LE: return 4;
        case NumberFormat.Int8BE: return -10;
        case NumberFormat.UInt8BE: return 10;
        case NumberFormat.Int16BE: return -20;
        case NumberFormat.UInt16BE: return 20;
        case NumberFormat.Int32BE: return -40;
        case NumberFormat.UInt32BE: return 40;

        case NumberFormat.Float32LE: return 4;
        case NumberFormat.Float32BE: return 40;
        case NumberFormat.Float64LE: return 8;
        case NumberFormat.Float64BE: return 80;
    }
}

function fmtInfo(fmt) {
    let size = fmtInfoCore(fmt)
    let signed = false
    if (size < 0) {
        signed = true
        size = -size
    }
    let swap = false
    if (size >= 10) {
        swap = true
        size /= 10
    }
    let isFloat = fmt >= NumberFormat.Float32LE
    return { size, signed, swap, isFloat }
}

function setNumber(buf, fmt, offset, r) {
    let inf = fmtInfo(fmt)
    if (inf.isFloat) {
        let arr = new Uint8Array(inf.size)
        if (inf.size == 4)
            new Float32Array(arr.buffer)[0] = r
        else
            new Float64Array(arr.buffer)[0] = r
        if (inf.swap)
            arr.reverse()
        for (let i = 0; i < inf.size; ++i) {
            buf[offset + i] = arr[i]
        }
        return
    }

    for (let i = 0; i < inf.size; ++i) {
        let off = !inf.swap ? offset + i : offset + inf.size - i - 1
        buf[off] = (r & 0xff)
        r >>= 8
    }
}

```

not posting the audio sample because it’s copyrighted and I’m too lazy to look up something public domain to use 😛

---

<div class="post-metadata">

**Author:** ![\_jupiter](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.makecode.com/_jupiter/32/28012_2.png) [@\_jupiter](https://forum.makecode.com/u/_jupiter)\
**Post date:** [September 15, 2024, 9:23pm UTC](https://forum.makecode.com/t/help-with-custom-audio-clips-again/30384/10 "2024-09-15T21:23:48Z")

</div>

Everyone is not noticing how we can add voices to our games now