Murphy’s Law

“Anything that can go wrong, will go wrong”.

strikes again. Well, the modified version of it:

“Anything that can go wrong, will go wrong, but it will wait until it really, really goes wrong”.

It is always amazing to see that systems having a fundamental flaw, they can work for a long period. Only that on day X my application crashes. And when found the problem, I’m wondering how in the world it was *ever* working with that bug in it :-(.

In I had this kind of experience these days week: my ARM Cortex-M0+ application on the Freedom board was working fine for more than 2 months. Then I added some USB CDC support, along with other things, and then my application crashed with a hard fault :-(. Good: it crashed consistently. Bad: as soon as I tried to minimize the application, the problem disappears. And it happened only with higher ARM GNU gcc optimization levels. Thinking and debugging it, it was clear that it had something to do with interrupt timing and interrupt levels. I was even at the point that it could be silicon problem. It turned out, I was wrong. As (nearly) always: I created the problem. But it took an incredible time to show up. Amazing (and embarrassing).

Cortex-M and Interrupts

The search for the problem root cause was a good occasion to re-read all the notes about FreeRTOS and ARM Cortex interrupt handling. Notably

• An article on FreeRTOS.org how to debug Hard faults
• Another article on FreeRTOS.org about Interrupt levels for an RTOS on Cortex-M

The above two articles really had to sink in. And I realized that my FreeRTOS port was not using the right number of interrupt levels available: It assumed 4 interrupt bits (16 levels) which is fine for the ARM Cortex-M4 used in the Kinetis-K. But the Kinetis-L family (ARM Cortex-M0+) only has 2 bits which means 4 interrupt levels. So configPRIO_BITS in FreeRTOSConfig.h was defined with the wrong number of bits :-(. This is fixed with an additional setting in the Processor Expert FreeRTOS component:

The ARM core/family selection has an impact on the available interrupt levels (Please read again the article about interrupt levels on FreeRTOS.org):

• Lowest Interrupt Priority: This value is informal only: it defines the lowest interrupt level supported by the core (3 for Kinetis-L ARM Cortex-M0+, 15 for Kinetis-K ARM Cortex-M4). Keep in mind that higher numbers mean *lower* priorities for ARM Cortex!
• Library Lowest Interrupt Priority: This is the priority of the RTOS itself (tick timer and performance tick timer). Usually this is as well the lowest priority of the system. That way any interrupt can interrupt the RTOS to avoid interrupt latency.
• Max SysCall Interrupt Priority: Interrupts with priority *higher* (numerically lower!) than this value shall *NOT* ❗ call any RTOS API routines.

So with the above settings, the RTOS will run at level 3 (lowest level), and any interrupts with level 3, 2 and 1 can call RTOS API routines, while the interrupts with level 0 (highest interrupt priority) shall *not* call any RTOS API routines.

Well, while all above are very good changes and fixes, they did not cause my above problem. It was something completely different….

Problem found

After debugging the thing for a very long time, I finally have found the reason in my FreeRTOS Cortex-M0+ port: a register was not correctly restored. The offending code is in port.c, function vOnCounterRestart() at the end of the function: In case of optimizations set, the register R7 was restored instead of R3. To correct this, the highlighted lines below have been added:

__attribute__ ((naked)) void vOnCounterRestart(void) {
  ...
#if __OPTIMIZE_SIZE__ || __OPTIMIZE__
  __asm volatile (
    " pop {r3,pc}                \n" /* start exit sequence from interrupt: r3 and lr where pushed above */
  );
#else
  __asm volatile (
    " pop {r7,pc}                \n" /* start exit sequence from interrupt: r7 and lr where pushed above */
  );
#endif
#endif
}

With the wrong register restored, my application from time to time tried to do a function pointer call with the register R7 (which was zero by chance), and crashed with a hard fault. With that things in place, the crash obviously did not happen any more.

Conclusions

Even if something is wrong, things might still work. Only to bite me later on. So that FreeRTOS bug is fixed and the updated version (V1.201) is available here. So if anyone is running into strange problems with my FreeRTOS port and gcc optimizations enabled, it indeed could be my bug above. I feel bad about it, and it has cause me a lot of grief. Nobody is perfect…

Happy Cortexing 🙂