@Clemens

I do use "hardware" serial, not software serial.

I would be so happy if its just bcs. I handle the MIDI incorrectly! Bcs. then its a simple problem, a bug 40 🙂 And then code CAN be stable... so I really hope tat.

When thats said...

[quotePost id=17732]do not set MIDI_Command. (Your "Reset MIDI_Command back" code is wrong; the last command might have been anything else.)
[/quotePost]

As I at the moment know exactly what codes are sent from the MIDI source..the method I have used with setting it back to 192 after getting sysEx End f.x. should work... in my head.
There is only ONE button I am touching and thats the song-changing turnable knob. And MIDI sends Sensing automatically and when the MIDI sync is set to ON it sends Timing CLock too.
HOWEVER... I still havent figured out WHY it sends SysEx data when I turn the knob fast!?

BUT I read u loud and clear.. better safe than sorry, and if it helps, it will be perfect, so I will try reconstruct my code, so it looks in byte intervals instead.

@Clemens:

Also now with changed code to ranges instead of bit-check, it looses data. E.g. this example should have ended on ProgramChange 27 (song 28) after the song 15. But again there comes a SysEx range of commands & data:

[code type=markup]
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 248 = Real Time Message Byte Timing Clock
23:01:52.989 -> 254 = Real Time Message Byte Active Sensing
23:01:52.989 -> 192 = Command Byte : Program Change
23:01:52.989 -> 14 = Data Byte ~ SONG: 15 updateTFT()
23:01:52.989 -> tft.availableForWrite()==0
23:01:53.300 ->
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
23:01:53.300 -> 240 = Command Byte : SysEx begin
23:01:53.300 -> 65 = Data Byte
23:01:53.300 -> 16 = Data Byte
23:01:53.300 -> 0 = Data Byte
23:01:53.300 -> 0 = Data Byte
23:01:53.300 -> 114 = Data Byte
23:01:53.300 -> 18 = Data Byte
23:01:53.300 -> 0 = Data Byte
23:01:53.300 -> 0 = Data Byte
23:01:53.300 -> 0 = Data Byte
23:01:53.300 -> 8 = Data Byte
23:01:53.300 -> 120 = Data Byte
23:01:53.300 -> 247 = Command Byte : SysEx end
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
23:01:53.300 -> 248 = Real Time Message Byte Timing Clock
[/code]

Latest code:

That SysEx looks valid.
Manufacturer ID (65) is Roland; device ID is 16 = default; model ID is 0 0 114 (Boss RC-505 ???); 18 looks like data set 1; 0 0 0 is the address; 8 is the data; 120 is the checksum (sum of address and data and checksum modulo 128 is zero).
I would guess that this message tells you that button nr. 8 was pressed (or that the value increased by 8, or something like that).

It's possible that some bytes get dropped without any error indication. A possible cause would be other processing (such as the output to the other serial port); try using much less logging.

Martin, your most recent log shows 311 milliseconds elapsed between these two lines:

23:01:52.989 -> tft.availableForWrite()==0

23:01:53.300 -> 

That's enough time for up to 971 MIDI bytes to come in. According to one of the Arduino Serial documentation pages, it looks like the serial receive buffer is 64 bytes.

1. Perhaps you can try changing your updateTFT() function to send fewer TFT commands.

Instead of doing all of this every time in your updateTFT() function:

tft.fillScreen(ST77XX_RED);

tft.setTextColor(ST77XX_BLACK);

tft.setCursor(10, 10);

tft.setTextSize(3); 

tft.print("SONG: ");

tft.setCursor(10+108, 10);

tft.setTextColor(ST77XX_BLACK);

tft.println(incomingMIDI_songNumber);

What happens if you just do this?:

tft.setCursor(118, 10);

tft.println(incomingMIDI_songNumber);

2. I couldn't find documentation about what the tft.availableForWrite() function does. Is it similar to the Ardunio Serial.availableForWrite() function? That seems return the number of free bytes in the write buffer. If the tft.availableForWrite() function is similar, then if you only send TFT commands when the tft.availableForWrite() function returns zero, maybe that means you will always have additional wait time until a write buffer has some free space?

What happens if you remove the if(tft.availableForWrite()==0) condition?

@Clemens and @Bavi_H thank you, I really appreciate your continuous posts. You all the time inspire me to try new things! 🙂

@Clemens.

- Yes its a RC-505 mkII. Have u read somewhere that the code stands for that?
- There were no button pressed, only turned the song knob forth & back fast. IF it should make sence somehow as a shortcut command to the last song when many song-data is sent, I would expect e.g. a distance/subtraction value from 14 to 27, so in this case if the value were 27-14=13 OR 27 then it would have made sence. But 65,16,114, 8 & 120 doesnt make any sence to me. But the definitions u have written above is that consequent rules for all SysEx?
- I have tried both with & without logging. Thats also why I made the easy logical setting "trace" in the beginning of the code, so I could test both with and without.
But I do have seen it fail also without the trace. But its a while ago and the code has changed since, so yes, would be smart to test it again.

@Bavi_H

- Ya I have noticed the jumps in milliseconds too. But Im not sure they are trustworthy! As it seems like all data are coming in chunks in that build in trace logging window of Arduino IDE. I have eralier experimented with timing also, both to log, but also to with purpose put in a wait until alle song changes were finish. I didnt have much success with that too. But it might make more sence now when the code is based on byte ranges! So I think first I need to program my own timer object again and post my own time in the log window, so we can check if it is a true long time the tftf wirting takes or if its just the IDE's chunks of data.
But of course the tft writing takes time no matter what. I have also bought some new & other types of tft's to test if they are faster.
- About the many commands and much writing to the tft, yes I have also simplified that a loot earlier. It helped a little, but still not 100% in all cases. But I will optimize that code again of course. BUT in fact I need to update 11 tft displays with different content! So I need a working base solution which doesnt get affected of how much is written, because else I will just have the problem again when adding 1 more or 2 or 5 or 11.
- About your last question in 1, what would happen would be for each writing the area on the screen would be more and more filled & blurred ending up just being a filled square.
- About your 2. question: tft.availableForWrite() isnt something I have found documented, but I also saw the serial.availableForWrite() and just thought that tft is also serial so it must be based on the same class, so I just tried it and it didnt fail. But I have also not seen it "go wrong" (not true) so I guess it doesnt make any difference if it is there or not. - But ya, I have tried MANY desperate things on my way to fix this. L
- I have also tried tft.flush() but it also didnt seem to solve it, also I have tried to disable other interrupts while writing to the tft with NoInterrupts() and interrupts() to reactivate, but this also didnt seem to fix it.

So ya, I will try my own timer and return to you.
I will also test without the trace and return.

But I really really would like to find the exact course of the problem! And not just "minimizing" by making things take a little longer and just reducing the propability of the lost datas. Becaus if we find the exact reaosn, we would probably also be able to dontrol it somehow.

I would prefer a solution with flush, buffer, ringbuffer, interrupts, better tft, better midi interpretation, pauses, or even using extra dedicated Arduinos or even other small computer devices as a solution where I can add so many tft displays and write so much as i want, without it looses data.

BTW... even I optimize my code more & more to take shorter time, the MIDI source device are still sending obscure SysEx data (probably instead of the last ProgramChange), so I suspect the last issue is located in that, so I have also contacted Roland to get insights on what the SysEx data exactly is.

But I will also test the exact MIDI-data both with a MidiMonitor and compare with my trace windows output to check if they are the same, if exactly the same, the problem is no longer in the code but with interpreting Rolands SysEx.

How annoying this forum. It also says "Sorry, but the system detected your content as spam" if I modify my post. Besides I also have to click in the Im not a robot again & again.
Hope the admins will optimize it, so its not an obstrugle more on the way to get help.

Wow, interesting...
After monitoring bothe with a standalone MIDI-Monitor software on the THRU on the MIDI-breakout shield board AND at the same time log-writing to Arduinos trace-window fronon the MIDI IN myself, I can see that the Arduino-code have not just lost 1 the last ProgramChange, but whole 3! In this attached screendump it shows it have lost both ProgramChange 12, 13 & 19!
So again; it doesnt have nything to do with the MIDI source or the strange SysEx data.

I need to find out WHY & WHERE its loosing the data.

I just searched for "F0 41 10 00 00 72 12", and found a forum post that said that such a SysEx is sent when an RC-505 key is pressed.
I guess 8 identifies the control.

Roland uses this DT1 command format in most of its devices.

More ideas:

1. If you don't need the MIDI Timing Clock messages (hex F8 or decimal 248), it looks like you can tell your Boss RC-505mkII to stop sending them, according to the manuals. Go to MENU > MIDI > SYNC OUT and change it to OFF. That will reduce the overall number of messages the Boss RC-505mkII sends out and maybe help avoid overflowing the Ardunio's incoming buffer.

2. In your thread on Arduino Forum, you got a suggestion to change the size of the incoming buffer by changing a #define line at the beginning of HardwareSerial.h. That sure sounds like a good idea to me, I think you should explore that more. Unfortunately, I don't have an Arduino, so I can't help much with that.

3. If the timestamps in the log aren't reliable, I can't be sure, but based on the timestamps, it seems like the TFT commands are slowing things down. Maybe you can split up the TFT commands into smaller portions and cycle through them every update.

int update_TFT_step = 0; // 0 means no update needed



void updateTFT()

{

  switch(update_TFT_step)

  {

    case 1:

    // Initial TFT commands...

    break;



    case 2:

    // Continue TFT commands...

    break;



    case 3:

    // Final TFT commands...

    break;

  }



  if(update_TFT_step < 3) { ++update_TFT_step; }

  else { update_TFT_step = 0; }

}



void loop()

{

  if(Serial1.available() > 0)

  {

    // Read and if Program Change:

    update_TFT_step = 1;

  }



  if(update_TFT_step > 0)

  {

    updateTFT();

  }

}

Oh, there is a "PAGE 2" in this forum, lol. Ok

@Clemens...: Interesting with the "key pressed", since I dont press any botton at all. Only turns a 360 degree knob.

@Bavi_H

As of 1) Thats the thing I also need the timing clock later, so the solution needs to work with that incoming data too.

As of 2) Yes I have also seen that. But I havent managed to locate that HardwareSerial.h file on my computer! Strangely enough. Both Explorer and Total Commander cannot find it. I must be blind or something obscure must have happened to my setup, bcs. how can the program work if it doesnt exist, lol.
Maybe I just need to reinstall the IDE. - Ya I think I will do that now.

As of 3) I really like the idea! E.g. if I can make one display work per loop cycle, I could just have an array of 11 displays and only update 1 display per cycle! - "Maybe" - because the data will probably continue coming in while the 10 others displays are updated anyway, and then the problem will come anyway with lost data!?

Btw... I have already improved the tft writing proces from 311ms to 9-11ms and right now the program doesnt loose data! 🙂

I also had done that in many examples earlier, but with stil no better result. SO I guess the missing data have been caused by many different things which are step by step solved.

BUT it needs to do several other things with the display and also update whole 11 displays and a bunch of other things... So I expect the problem just becoming worse & worse and coming up again & again to haunt me, if I dont manage to find the exact cause and solve that.
E.g. I would really need to know if its due to a a) serial buffer overrun, b) if its a lack of a serial buffer, c) if the problem should be able to be fixed "on the paper" with call toNoInterrupts() & interrupts(), flush(), ifAvailable() or similar.

After updating with song title also, instead of just a 2 digit number, it now takes 92ms.