Troubleshooting Tips for FreeRTOS Thread Aware Debugging in Eclipse

FreeRTOS seems to get more and more popular, and I think as well because more and more debugger and Eclipse IDE vendors add dedicated debugging support for it.

FreeRTOS Threads in Eclipse

FreeRTOS Threads in Eclipse

Outline

This article describes tips and tricks to get FreeRTOS thread aware debugging in Eclipse, organized by debug probe connection/vendor. For thread awareness, the debugger needs to know some well-known symbols of the RTOS. So beside of checking if the thread awareness is turned on, have a look about the debug output. If the symbols are not present, it might be because of linker optimizations or not turning on FreeRTOS features.

P&E (Multilink)

The P&E debug connection has no special setting for FreeRTOS thread aware debugging and does it automatically, so there is no option to turn it on or off.

Thread Aware FreeRTOS Debugging with P&E

Thread Aware FreeRTOS Debugging with P&E

The P&E GDB servers should report this at the beginning in the console:

Successfully initialized kernel symbols.
P&E Sucessfully Initialized kernel symbols

P&E Sucessfully Initialized kernel symbols

The P&E GDB Server console should show

Kernel thread information now available.
Kernel Thread Information available

Kernel Thread Information available

Below is a list of symbols the P&E GDB server is looking for:

Searching for FreeRTOS Kernel Symbols…

Symbol pxCurrentTCB found. (0x20002880)
Symbol pxReadyTasksLists found. (0x20002884)
Symbol xDelayedTaskList1 found. (0x200028E8)
Symbol xDelayedTaskList2 found. (0x200028FC)
Symbol xPendingReadyList found. (0x20002918)
Symbol uxCurrentNumberOfTasks found. (0x20002958)
Symbol uxDeletedTasksWaitingCleanUp found. (0x20002940)
Symbol xSuspendedTaskList found. (0x20002944)
Symbol xSchedulerRunning found. (0x20002964)
Successfully initialized kernel symbols. FreeRTOS analysis enabled.

Segger (J-Link)

FreeRTOS Thread Aware Debugging with J-Link

FreeRTOS Thread Aware Debugging with J-Link

Debugging with Segger J-Link

Debugging with Segger J-Link

Thread aware debugging needs to be turned on for Segger with the following command

-rtos GDBServer/RTOSPlugin_FreeRTOS

That option can be specified in the launch configuration (GNU ARM (now MCU) Eclipse plugins, e.g. in Kinetis Design Studio 3.2.0):

FreeRTOS Thread Awareness option in Launch Configuration

FreeRTOS Thread Awareness option in Launch Configuration

In MCUXpresso IDE v10.0.2, there is a drop-down setting for this in the debug configuration:

MCUXpresso IDE Segger FreeRTOS Setting

MCUXpresso IDE Segger FreeRTOS Setting

Check in the JLinkServer Eclipse console view if the needed symbols are present in the application:

Loading RTOS plugin: GDBServer/RTOSPlugin_FreeRTOS...
RTOS plugin (v1.0) loaded successfully
RTOS plugin initialized successfully.
Received symbol: pxCurrentTCB (0x2000B084)
Received symbol: pxReadyTasksLists (0x2000B088)
Received symbol: xDelayedTaskList1 (0x2000B0EC)
Received symbol: xDelayedTaskList2 (0x2000B100)
Received symbol: pxDelayedTaskList (0x2000B114)
Received symbol: pxOverflowDelayedTaskList (0x2000B118)
Received symbol: xPendingReadyList (0x2000B11C)
Received symbol: xTasksWaitingTermination (0x2000B130)
Received symbol: xSuspendedTaskList (0x2000B148)
Received symbol: uxCurrentNumberOfTasks (0x2000B15C)
Received symbol: uxTopUsedPriority (0x20000038)
Received symbol: uxTopReadyPriority (0x2000B164)
Received symbol: vPortEnableVFP (0x00016968)
All mandatory symbols successfully loaded.
Segger J-Link checking FreeRTOS Symbols

Segger J-Link checking FreeRTOS Symbols

NXP (LinkServer)

Debugging FRDM-K64F with LinkServer (LPC-Link2)

Debugging FRDM-K64F with LinkServer (LPC-Link2)

With MCUXpresso IDE, make sure you connectin in ‘All-Stop’ mode (see MCUXpresso_IDE_FreeRTOS_Debug_Guide.pdf inside the IDE installation folder):

LinkServer in All-Stop Mode

LinkServer in All-Stop Mode for FreeRTOS Thread Awareness

💡 As this setting is configured the first time I connect to the probe, the easiest way to change this is to delete the LinkServer launch configuration and connect/debug again. Note that with MCUXpresso V10.0.2 on using All-Stop mode will interfer with Live Variable view (will not be possible in All-Stop mode)

The above setting is accessible in the launch configuration under ‘Non-stop mode’ under the ‘Main’ sub-tab of the LinkServer debug configuration:

Non-Stop Mode in LPC-Link2 Launch Configuration

Non-Stop Mode in LPC-Link2 Launch Configuration

Make sure the following debug helpers are enabled:

NXP LinkServer Thread Awareness

NXP LinkServer Thread Awareness

With the LinkServer (MCUXpresso IDE) threads are shown like below:

FreeRTOS Thread Awareness with LinkServer

FreeRTOS Thread Awareness with LinkServer

If threads are not shown, increase the Debug Level to 4 in the launch configuration:

LinkServer Debug Level

LinkServer Debug Level

💡 I recommend to switch the debug level back to the default of 2, as it produces a lot of output.

In the Debug messages Console view, check that all the needed symbols are found:

FreeRTOS symbol(s) found:
- "pxReadyTasksLists" 0x200028C0
- "xPendingReadyList" 0x20002950
- "xSuspendedTaskList" 0x20002980
- "xDelayedTaskList1" 0x200028A8
- "xDelayedTaskList2" 0x2000296C
- "pxDelayedTaskList" 0x2000289C
- "pxOverflowDelayedTaskList" 0x2000294C
- "xTasksWaitingTermination" 0x20002924
- "xSchedulerRunning" 0x200028A0
- "uxCurrentNumberOfTasks" 0x20002968
- "pxCurrentTCB" 0x2000293C
- "FreeRTOSDebugConfig" 0x0000474C
FreeRTOS stack backtrace is enabled
LinkServer Debug Messages

LinkServer Debug Messages

Read the MCUXpresso_IDE_FreeRTOS_Debug_Guide.pdf inside the MCUXpresso IDE installation folder.

Make sure you have the following turned on in FreeRTOSConfig.h:

#define configINCLUDE_FREERTOS_TASK_C_ADDITIONS_H    1
#define configUSE_TRACE_FACILITY                     1

Make sure you have “freertos_tasks_c_additions.h” present in your project (You can find a version of it in the NXP SDKs or here on GitHub).
If the FreeRTOSDebugConfig structure is missing (e.g. because of optimizations), make sure it is referenced/used from the appliation, e.g. with the following code in main():

extern uint8_t FreeRTOSDebugConfig[];

int main(void) {
 if (FreeRTOSDebugConfig[0]==0) { /* just use it, so the linker cannot remove FreeRTOSDebugConfig[] */
   for(;;); /* FreeRTOSDebugConfig[0] should always be non-zero, so this should never happen */
 }
 /* other code in main */
}

If xTasksWaitingTermination() is missing, check that you have the following enabled in FreeRTOSConfig.h:

#define INCLUDE_vTaskDelete   1

OpenOCD

For OpenOCD, see FreeRTOS Thread Debugging with Eclipse and OpenOCD.

Debugging with onboard OpenSDA and OpenOCD

Debugging with onboard OpenSDA and OpenOCD

GNU Link Time Optimizations (-lto)

If using the GNU Link Time Optimization, this can inline functions and variable usage, so they are not available any more for the thread awareness. The McuOnEclipse FreeRTOS port has an extra setting (config_LTO_Helper) which prevents symbols for the thread awareness to be optimized. That setting is turned off (no) by default as it can increase code size:

enabling LTO Helper

enabling LTO Helper

Stack and Heap?

If everything fails, a good idea is to increase the task stack size and heap size, as some strange errors can happen if tasks overwrite variables of the kernel. There is an stack overflow hook in FreeRTOS, but this will not be able to catch every overflow.

NXP Thread Awareness Plugins

If using the NXP FreeRTOS plugins (see Better FreeRTOS Debugging in Eclipse), check that you have the following settings in FreeRTOSConfig.h (tips from Dave Nadler):

#define configUSE_PORT_OPTIMISED_TASK_SELECTION 0 // Can cause older FreeRTOS Eclipse plug-in to crash otherwise...
#define configTASK_RETURN_ADDRESS 0 // place 0 task return address on stack to help FreeRTOS-aware debugger (GDB unwind thread stack)
#define configUSE_TRACE_FACILITY 1 // without this, FreeRTOS-aware debug task list misses tasks and shows bogus stack overflows

See for example:

Check the warning signs of the view, e.g. turn on configUSE_TRACE_FACILITY:

FreeRTOS Task List

FreeRTOS Task List

The plugin puts diagnostic information into the ‘FreeRTOS_TAD_logs’ folder, located inside the workspace folder:

FreeRTOS_TAD_logs

FreeRTOS_TAD_logs

The log files can give imporant information what is missing or wrong:

"FreeRTOS Task Aware Debugger for GDB" version 1.0.2 (201704101406)
(c) 2016 NXP Semiconductors, Inc.
==================================
11:16:24.795 INFO: [TadModel] DSF session ID 4 has started.
11:16:26.540 INFO: [TadState] TAD state changed: DEBUG_STARTED -> DEBUG_INIT_0 (SUSPENDED, USER_REQUEST)
11:16:27.734 INFO: [TadState] TAD state changed: DEBUG_INIT_0 -> DEBUG_INIT_1 (RESUMED, USER_REQUEST)
11:16:27.806 INFO: [TadState] TAD state changed: DEBUG_INIT_1 -> DEBUG_SUSPENDED (SUSPENDED, BREAKPOINT)
11:17:14.931 INFO: [TadState] TAD state changed: DEBUG_SUSPENDED -> DEBUG_RESUMED (RESUMED, USER_REQUEST)
11:17:16.654 INFO: [TadState] TAD state changed: DEBUG_RESUMED -> READY (SUSPENDED, SIGNAL)
11:17:29.695 INFO: [Benchmark] Loading data for "Task List" has started.
11:17:30.309 INFO: [FreeRTOS] FreeRTOS macro "configGENERATE_RUN_TIME_STATS" is disabled.
11:17:31.288 ERROR: [MemoryReader] Could not get memory block! Invalid address "0x1ffc0008" and/or "0x13ff8" length of a block!
11:17:32.290 INFO: [FreeRTOS] FreeRTOS macro "configUSE_TRACE_FACILITY" is disabled.
11:17:33.291 ERROR: [TaskFactory] Cannot read information from "xDelayedTaskList1".
11:17:33.999 INFO: [Benchmark] Loading data for "Task List" took 4304 ms.
11:18:42.676 INFO: [TadModel] DSF session ID 4 has ended.

In the above case, the a task had been created with 80kByte stack and this caused a problem with that version of the NXP Task aware plugin. Setting the heap to a smaller value below 64kByte worked as a workaround.

Summary

FreeRTOS thread aware debugging depends on debug connection used. In most cases it might fail because some symbols are not present in the application to help the debugger to show the threads. I hope that this article helps you to find and possibly fix any issues.

Happy Threading 🙂

Links

Advertisements

7 thoughts on “Troubleshooting Tips for FreeRTOS Thread Aware Debugging in Eclipse

  1. Any suggestions for the following error from the segger gdb output?

    Loading RTOS plugin: GDBServer\RTOSPlugin_FreeRTOS…
    RTOS plugin (v1.0) loaded successfully
    RTOS plugin initialized successfully.
    Received symbol: pxCurrentTCB (1FFF14B4)
    Received symbol: pxReadyTasksLists (00000000)
    ERROR: Mandatory symbol pxReadyTasksLists not found.

    Using FreeRTOS9.0 and the latest segger tools on a K20F. It all works fine if I turn of LTO, but I’d rather not. Applications works fine in either case.

    Like

    • It means that the linker somehow has removed pxReadyTasksLists. Are you using the McuOnEclipse FreeRTOS port (as this one has several measures to prevent such linker optimizations)?

      Like

      • We are actually using OPENrtos (FREErtos, no license restrictions but same code) unmodified. What is weird is we hav another project using the same code/FREErtosconfig and it works fine in either case.

        Like

      • Hi Michael,
        interestingly, I’m now able to reproduce that same problem:
        RTOS plugin (v1.0) loaded successfully
        RTOS plugin initialized successfully.
        Received symbol: pxCurrentTCB (0x20000130)
        Received symbol: pxReadyTasksLists (0x00000000)
        ERROR: Mandatory symbol pxReadyTasksLists not found.

        I’m investigating what can be done, so stay tuned….

        Like

  2. posten on behalf of Dave Nadler:
    —————————-
    Hi Erich – Perhaps also worth mentioning..
    Without the following FreeRTOS configuration settings I had erratic task-aware debugging:

    #define configUSE_PORT_OPTIMISED_TASK_SELECTION 0 // Note: 1 causes FreeRTOS Eclipse plug-in to crash…
    #define configTASK_RETURN_ADDRESS 0 // place 0 task return address on stack to help FreeRTOS-aware debugger (GDB unwind thread stack)
    #define configUSE_TRACE_FACILITY 1 // without this, FreeRTOS-aware debug task list misses tasks and shows bogus stack overflows

    See for example:
    http://forum.segger.com/index.php?page=Thread&postID=15403#post15403
    https://sourceforge.net/p/freertos/discussion/382005/thread/e7d139e5/

    Thanks again,
    Best Regards, Dave

    Like

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s