Bulk dump - bulk also got aggressively corrected to build on my machine here !!

Here is some information on SysEx Bulk dumps. This text happens to be discussing a Yamaha PSR, but all Yamaha bulk dumps work the same way.

(We volunteer in the MIDI Association, but actually work for Yamaha so this information is accurate).

By the way we attached the MFC10 manual and SysEx Bulk Dump information starts on page 33. .

Format of the system exclusive messages

The general format for the bulk dump system exclusive messages is:

F0 43 73 57 06 TT SS SS SS ZZ ZZ ZZ ... KK F7

TT, Type of data

09 = One Touch Setting
0A = Song
0C = Chord

SS SS SS, Size

Indicates the number of encoded data bytes in this system exclusive message.

Because the most significant bit of each byte must be zero, each byte only stores 7 bits of the value. The most significant bits of the value are in the first byte. For example, the bytes 02 11 7F indicate the value $88FF.

ZZ ZZ ZZ, Running total or End of section flag

If these bytes are 7F 7F 7F, this indicates the end of this type of data.

Otherwise, these bytes indicate the total number of encoded data bytes in all previous system exclusive messages for this type of data. (The value is stored using the 7-bit method mentioned above.)

..., Data

If this is the end of this section, then there are three data bytes which indicate the total number of encoded data bytes for this type of data. (The value is stored using the 7-bit method mentioned above.)

Otherwise, these bytes encode the bulk dump data. Each pair of data bytes in the system exclusive message encodes one byte of the bulk dump data. The first byte of a pair stores the least significant nybble, the second byte stores the most significant nybble. For example, the bytes 0F 07 indicate the decoded byte 7F.

KK, Checksum

The sum of all bytes from SS SS SS to KK, inclusive, should equal 0, modulo 128 decimal.

Thanks for adding a few more thoughts to the manual which i had read

DSI products helpfully explain how you can rebuild the packed data and i've done this with a few other brands too, having dug a bit further with the MFC dumps it creates a bit more confusion as the header for a type of bulk dump subset can reappear frequently throughout the dump - it transpires there are perhaps some 'useful' parts sent 256 bytes long, then there are a couple other bytes between headers (pre listing the upcoming data !), then 256 again and so forth, all the while it is evident the data is somewhat packed

the difference with the other vendors' data (e.g. DSI / Elektron) is that their packed data is packed in one long single bunch, so visually scanning the quite large files can be tricky when you aren't (yet) sure how to pre-empt what's what from the list being sent

your insights look promising to help me establishing how to better interpret and re-assemble bulk data, i'll give your text a good read through and try once again, many thanks : )

I've decided to attach a breakdown of what's hiding inside the BL3 dump, the largest subset of a total dump, this covers Function Assignments - This is in Hex and is taken from the initialised state

Then Manual only covers the initial header and does not mention any data packing strategies, two of the other bulk dump subsets send the data readable and it's all quite straightforward looking.

However, this dump contains quite a few echoes of the main header element including 16x 0 before each data group, there are also some indications that the last part of one piece of data is sent either side of a partial header 'duplicate' and there seems to be some organisational system at play, but even with the pointers above i'm no closer to cracking this

Now, it may not be wise to examine the data contents as is (i.e. everything after the initial 16x 0 and before the checksum and EOX) but i can't help but think the recurring header elements is a clue that the data is sent in smaller pieces and there's no clues as to how to piece it back together

The above text cites an example of sending 88FF as 02 11 7F, now the maths bit is understood, but it's way less clear how this could possibly apply to the data contents in their entirety, there's no way the repeating header elements are packaged up - there are obviously clues as to how to read this in and around those smaller elements

The size is sent as two bytes 1 34 repeatedly or 1 52 decimal - i'm assuming this is a part of knowing which bytes that follow will be of use - in the other dumps the size is 0 25 and 0 100 (dec) and this corresponds exactly to the data byte count

do i treat 1 52 as 256 + 52 or 128 + 52 to get a tally ? but either way, this tally doesn't square with the bytes that seem to be the useful ones (outside of patterns) - but maybe the count only applies to post-processing, unlike on the other dump formats

Someone at Yamaha must have this info, it isn't such an old product and if the device follows standard data packing methods then perhaps the explanation is just missing that one key aspect to unlock it for me

Any advice welcome

ps: pdf , i've used colours and formatting to tie together elements or to show when a string is seemingly repeated either side of a partial header - interpreting this to reverse engineer what's required is proving a bit of a headache - ironically, it would take far too long to manually program in bulk changes to the parameters to see how the data changes (i.e. what's real content and what's pointers etc)

Hopefully this'll look very obvious to someone who can shed some light on it or suggest a better way to approach this

cheers

I've decided to attach a breakdown of what's hiding inside the BL3 dump, the largest subset of a total dump, this covers Function Assignments - This is in Hex and is taken from the initialised state

Then Manual only covers the initial header and does not mention any data packing strategies, two of the other bulk dump subsets send the data readable and it's all quite straightforward looking.

However, this dump contains quite a few echoes of the main header element including 16x 0 before each data group, there are also some indications that the last part of one piece of data is sent either side of a partial header 'duplicate' and there seems to be some organisational system at play, but even with the pointers above i'm no closer to cracking this

Now, it may not be wise to examine the data contents as is (i.e. everything after the initial 16x 0 and before the checksum and EOX) but i can't help but think the recurring header elements is a clue that the data is sent in smaller pieces and there's no clues as to how to piece it back together

The above text cites an example of sending 88FF as 02 11 7F, now the maths bit is understood, but it's way less clear how this could possibly apply to the data contents in their entirety, there's no way the repeating header elements are packaged up - there are obviously clues as to how to read this in and around those smaller elements

The size is sent as two bytes 1 34 repeatedly or 1 52 decimal - i'm assuming this is a part of knowing which bytes that follow will be of use - in the other dumps the size is 0 25 and 0 100 (dec) and this corresponds exactly to the data byte count

do i treat 1 52 as 256 + 52 or 128 + 52 to get a tally ? but either way, this tally doesn't square with the bytes that seem to be the useful ones (outside of patterns) - but maybe the count only applies to post-processing, unlike on the other dump formats

Someone at Yamaha must have this info, it isn't such an old product and if the device follows standard data packing methods then perhaps the explanation is just missing that one key aspect to unlock it for me

Any advice welcome

ps:Attached a zipped text pdf , i've used colours and formatting to tie together elements or to show when a string is seemingly repeated either side of a partial header - interpreting this to reverse engineer what's required is proving a bit of a headache - ironically, it would take far too long to manually program in bulk changes to the parameters to see how the data changes (i.e. what's real content and what's pointers etc)

Hopefully this'll look very obvious to someone who can shed some light on it or suggest a better way to approach this

cheers

Be aware that user "The MIDI Association" quoted some text about the bulk dump format of the Yamaha PSR-225 (original source: rnhart.net/articles/bulk-dump.htm ), then referred to the MFC10 manual's dump format section and claimed "all Yamaha bulk dumps work the same way".

The PSR-225 dump format might give you some clues, but it looks like the MFC10 dump format is different. For example, the PSR-225 uses three bytes to indicate the size, but the MFC10 uses two bytes to indicate the size. The PSR-225 packs each byte of memory data into two bytes of sysex data (memory hex xy is packed into sysex hex 0y 0x), but it looks like the MFC10 uses a different packing scheme (or else the data bytes would all look like hex 0_).

In your MFC10's Function Memory dump, my guess is that the size of hex 01 34 means decimal 128 + 52. It seems odd that the Function Memory dump size would be interpreted differently that the other kinds of dumps. And the example you posted does have odd repeats in it. Also I noticed at the bottom of page 4, the 7th from last byte is F7 -- the end of sysex -- but the bytes after seem like more of the data that should be inside the sysex.

Maybe something is messing up the data. Here are some wild guesses:

Is your computer is sending the same data it is receiving back out to your MFC10? Maybe the MFC10 is trying to dump and save bulk data at the same time and getting confused which part of memory it's currently looking at.

Have you successfully received long sysex messages on your computer before? Maybe your MIDI software or hardware is dropping some bytes or getting confused and repeating bytes it has in an internal buffer.


Anyway, looking at the first few data bytes, here's an initial guess at the data format:

When it needs to store a status byte, it splits it into two bytes in the sysex (status hex xy to sysex hex 0x 0y).
When it needs to store value bytes, it stores it directly in the sysex.

For example the sysex data bytes 09 00 35 40 40 might represent

Status: hex 90 (note for channel 1)
Note Number: hex 35, decimal 53 (F below middle C)
On Velocity: hex 40, decimal 64 (middle of velocity range)
Off Velocity: hex 40, decimal 64 (middle of velocity range)

Has anyone worked with Alesis QuadraSynth sysex? Several of us QS owners suffer from the disappearance of our User Banks, and are unable to reload them from disc or online file. I am struggling to decipher a correspondence between the parameters contained in a preset program (for example, 00 AcGrandPno) and its sysex file from the Alexis disc. I work manually, having no programming experience as yet. Manufacturer support has ended for these synths from the 90's. Thanks.

Hi

Have you managed to resolve all your questions on the MFC10 bulk data dumps yet.

I have spent the last week working through the Function dump and have written a rough and ready Python program that reads in the dump file and prints out the assignments for the hundred Fnn buttons. So I think I have mastered it and managed to locate start and end of all the useful sections.

The Foot Controller dump is straightforward and just started on Program dump which looks easier than the function one.

Regards J

Jamie: I don't have a Yamaha MFC10, but I enjoy reverse engineering undocumented data formats. Would you consider sharing your results just to satisfy my curiosity?