Finding Memory Bugs with Google Address Sanitizer (ASAN) on Microcontrollers

Posted on by

Dangling pointers and memory corruption problems are nasty issues for any developer, and usually hard to find and locate in the code. Luckily Google has developed an open source tool to solve such issues: the Address Sanitizer (ASAN). The tool is available for x86 and other desktop style architectures, including Android and Linux. This article describes how ASAN can be used for an embedded target, e.g. ARM Cortex-M4 or similar.

Catching memory errors on ARM Cortex-M4F with ASAN

Outline

ASAN is able to detect dangling pointer accesses (using a pointer on non-allocated memory or after the memory has been deallocated) or other illegal memory accesses which usually corrupt memory or are hard to find. The ASAN Google Wiki page shows different problems which can be detected. ASAN is part of GCC starting with version 4.8.

ASAN can be easily used with Embedded Linux systems, but it is not available (to my knowledge) for microcontrollers. In this article I present a simple approach how to use ASAN memory checks on an ARM Cortex-M microcontroller. An example project is available on GitHub with the NXP MCUXpresso IDE/SDK using the NXP K22FN512 (ARM Cortex-M4F) and can be easily adapted for any other microcontroller.

Bugs detected with ASAN

The ASAN framework can detect many different problems, from buffer overflow, stack overflow, heap overflow, memory leaks, dangling pointers or accessing uninitialized variables.

In the context of this article I present how to use ASAN to detect ‘Use after free’ (dangling pointer) or Heap Overflow issues.

Below are very simple examples of errors which ASAN can detect: below are examples of a dangling pointer (memory use after free) and accessing memory outside the allocated area:

static int arr[] = {1,2,3};
static int *p = &arr[0];

static void Test1(void) {
  p = malloc(16);
  free(p);
  arr[0] = *p;
  *p = 0;  /* BOOM: access to released memory */
}

static int Test2(void) {
  int i, *p;
  uint8_t c;

  p = malloc(10);
  c = *((uint8_t*)p);
  for(int j=0; j<10; j++) {
    p[j] = j; /* BOOM: accessing beyond allocated memory */
  }
  free(p);
  i = *p; /* BOOM: access to released memory */
  return i+c;
}

How it works

ASAN instruments the code for each pointer access and calls an ASAN library routine to check if that read or write access through a pointer is valid. The same way each memory allocation and free operation gets through a special routine. The instrumentation happens with a compiler option.

To know if an access is valid, ASAN maintains a map of bits for the memory (called ‘shadow’), knowing if memory is valid (allocated) or invalid (poisoned). It is possible to cover each byte with a single bit or going up to several bytes per bit depending on the allocation of memory.

Shadow Memory

If the access is invalid, then an error hook gets called which can be used to to halt the application or write to a log, or anything you want. In the most simplistic version it could be something like this:

Example of error reporting hook

To detect ‘out of bound’ accesses, each memory block has a ‘red zone’ at the beginning and the end’ of the usable memory. Additionally a size information is stored in the header red zone area. The ‘red zones’ are ‘poisoned’ in the ‘shadow’ map.

The read/write accesses are implemented in an efficient way like this:

static void CheckShadow(void *address, size_t kAccessSize, rw_mode_e mode) {
  uint8_t *shadow_address;
  uint8_t shadow_value;

  shadow_address = MemToShadow(address);
  shadow_value = *shadow_address;
  if (shadow_value!=0) { /* fast check: poisoned! */
    if (SlowPathCheck(shadow_value, address, kAccessSize)) {
      ReportError(address, kAccessSize, mode);
    }
  }
}

The implementation includes a configurable ‘Quarantine’ list for free memory blocks, implemented with a ring buffer: that way memory issues can be detected even if right after a ‘free’ there is a new ‘malloc’ and if this would re-use the same memory block.

#if McuASAN_CONFIG_CHECK_MALLOC_FREE
void __asan_free(void *p) {
  /* Poisons shadow values for the entire region and put the chunk of memory into a quarantine queue
   * (such that this chunk will not be returned again by malloc during some period of time).
   */
  size_t size = *((size_t*)(p-sizeof(size_t))); /* get size */
  void *q = p;

  for(int i=0; i<size; i++) {
    PoisonShadowByte1Addr(q);
    q++;
  }
  q = p-McuASAN_CONFIG_MALLOC_RED_ZONE_BORDER; /* calculate beginning of malloc()ed block */
#if McuASAN_CONFIG_FREE_QUARANTINE_LIST_SIZE > 0
  /* put the memory block into quarantine */
  freeQuarantineList[freeQuarantineListIdx] = q;
  freeQuarantineListIdx++;
  if (freeQuarantineListIdx>=McuASAN_CONFIG_FREE_QUARANTINE_LIST_SIZE) {
    freeQuarantineListIdx = 0;
  }
  if (freeQuarantineList[freeQuarantineListIdx]!=NULL) {
    free(freeQuarantineList[freeQuarantineListIdx]);
    freeQuarantineList[freeQuarantineListIdx] = NULL;
  }
#else
  free(q); /* free block */
#endif
}
#endif /* McuASAN_CONFIG_CHECK_MALLOC_FREE */

Using McuASAN

Add the following three files to your project:

  • McuASANconfig.h: configuration header file
  • McuASAN.h: interface to the McuASAN implementation
  • McuASAN.c: implementation with memory hooks, callbacks and shadow map

For each source file which shall be instrumented, add the following option to the compiler settings:

-fsanitize=kernel-address
-fsanitize=kernel-address compiler option

Initialize the ASAN module with:

McuASAN_Init();

Configure ASAN with the defines in McuASANconfig.h. The most important ones are:

  • McuASAN_CONFIG_IS_ENABLED: set to 1 to enable ASAN
  • McuASAN_CONFIG_CHECK_MALLOC_FREE: set to 1 for catching malloc/free problems
  • McuASAN_CONFIG_APP_MEM_START: specify start address of covered memory for shadow map
  • McuASAN_CONFIG_APP_MEM_SIZE: size of memory to be covered in shadow map
  • McuASAN_CONFIG_MALLOC_RED_ZONE_BORDER: size of red zones
  • McuASAN_CONFIG_FREE_QUARANTINE_LIST_SIZE: size of quarantine list
#ifndef McuASAN_CONFIG_IS_ENABLED
  #define McuASAN_CONFIG_IS_ENABLED     (0)
  /*!< 1: ASAN is enabled; 0: ASAN is disabled */
#endif

#ifndef McuASAN_CONFIG_CHECK_MALLOC_FREE
  #define McuASAN_CONFIG_CHECK_MALLOC_FREE  (1)
  /*!< 1: check malloc() and free() */
#endif

#ifndef McuASAN_CONFIG_APP_MEM_START
  #define McuASAN_CONFIG_APP_MEM_START 0x20000000
  /*!< base RAM address */
#endif

#ifndef McuASAN_CONFIG_APP_MEM_SIZE
  #define McuASAN_CONFIG_APP_MEM_SIZE  (64*1024)
  /*!< Memory size in bytes */
#endif

#if McuASAN_CONFIG_CHECK_MALLOC_FREE
#ifndef McuASAN_CONFIG_MALLOC_RED_ZONE_BORDER
  #define McuASAN_CONFIG_MALLOC_RED_ZONE_BORDER  (8)
  /*!< red zone border in bytes around memory blocks. Must be larger than sizeof(size_t)! */
#endif

#ifndef McuASAN_CONFIG_FREE_QUARANTINE_LIST_SIZE
  #define McuASAN_CONFIG_FREE_QUARANTINE_LIST_SIZE  (8)
  /*!< list of free blocks in quarantine until they are released. Use 0 for no list. */
#endif

Configure in the error hook what to do (log a message or halt the application). Below an example implementation:

static void ReportError(void *address, size_t kAccessSize, rw_mode_e mode) {
  McuLog_fatal("ASAN ptr failure: addr 0x%x, %s, size: %d", address, mode==kIsRead?"read":"write", kAccessSize);
  __asm volatile("bkpt #0"); /* stop application if debugger is attached */
}

That’s it :-). See as well the example in GitHub and the links the end of the article.

Debug Session with ASAN

Summary

ASAN is very powerful and can help find subtle memory bugs. It requires instrumentation and a simple module which can be configured for your own needs. Files are available on GitHub.

There is certainly even more the ASAN framework can do. I hope you find this as useful as I do.

Happy Sanitizing 🙂

Links

49 thoughts on “Finding Memory Bugs with Google Address Sanitizer (ASAN) on Microcontrollers”

  1. Hello Erich,

    Another great work!
    A small typo. “McuASAN_CONFIG_APP_MEM_START: size of memory to be covered in shadow map”. It should be “McuASAN_CONFIG_APP_MEM_SIZE” here.

    I wonder what is the overhead of memory overhead of the current prototype? The original paper has about 3.37x. Which factors do you think will make difference on MCU? The original paper protects shadow memory by mapping it unmapped pages. How do you implement this? MPU?

    Thanks!
    Lennon

    Liked by 1 person

    Reply ↓

    • Hi Lennon,
      Thanks for bringing up that copy-paste error, fixed now.

      The impact on the implementation really depends: about which source files are instrumented, and how many pointer accesses are in the code. So if there are few memory allocations and rather few pointer accesses, the impact is really minimal or not noticeable. For code with using lots of pointers a factor of 3-4 is realistic. There is still some room for improvements, e.g. turning on inlining or tweaking the detection code.

      I do not use the MPU here: Memory allocation gets registered with a shadow bit map and then every pointer access gets through a checker if the memory is ok or poisoned.

      Thanks!
      Erich

      Like

      Reply ↓

  2. Hey Erich,

    Very interesting – I wanted to follow up on Lennon’s question about memory and execution speed overhead. I presume that you can disable ASAN when you want to go into production, but I was curious as to what would be the impact of the tool on operations.

    I’ve used Cppcheck (http://cppcheck.sourceforge.net/) for a few years now. It’s a nice tool for checking the source code and has been well maintained.

    Unfortunately I just discovered (when I tried to update it) that the Eclipse repository (bintray.com) has been shut down and I can’t install it on a new instance of Eclipse – I’m inquiring and will update you with the information about how to install.

    myke

    Liked by 1 person

    Reply ↓

  4. Thanks Erich for another great article! Some questions:
    1) To clarify a bit about overhead, do I understand correctly:
    – ASAN uses 1 bit per byte of memory so consumes ~1/8 available memory?
    – generated code adds a check call for each pointer (or array?) reference; can be substantial overhead
    2) Can you clarify (for those of us not using malloc/free) if ASAN traps out-of-bounds array references?
    Thanks Erich!
    Best Regards, Dave

    Like

    Reply ↓

    • Hi Dave,
      yes, I used one bit for each memory byte. That way I can check every access. But you can configure this and say use one bit for a block of 8 bytes for example.
      Yes, the compiler will call a ‘read’ or ‘write’ callback for each pointer read or write (there are 1-Byte, 2-Byte and 4-Byte access callbacks). And yes, this means extra overhead (nothing is for free).
      The current implementation I have for Cortex checks for every pointer access if it is accessing valid memory or not. Valid means it is part of the global initialized memory (zero-out and copy-down) and if it is a valid dynamically allocated memory block. It does check out-of-bounds accesses if the access is reaching poisoned memory. But not if the access is accessing another valid memory (e.g. valid other variables). So if you are not using dynamic memory it will catch memory accesses outside your defined memory area. But not within (as they could be perfectly valid).

      Like

      Reply ↓

  5. Pingback: Finding Memory Bugs with Google Address Sanitizer (ASAN) on Microcontrollers @McuOnEclipse « Adafruit Industries – Makers, hackers, artists, designers and engineers!

  6. Thanks Erich for your great work.

    I have some questions:

    1) Our system use some customized memory allocators, osAllocte()/osFree()/heapMalloc()/heapFree(). How to replace them with __asan_malloc()/__asan_free() ?

    2) Can ASAN traps out-of-bounds array on stack?

    Thanks Erich!
    Best Regards, Kaka

    Liked by 1 person

    Reply ↓

    • Hi Kaka,
      thank you!
      1) just let your memory allocators/handlers call __asan_malloc() and __asan_free(). If you have your own customized versions, simply call the asan one from them.
      2) Yes, if your stacks are allocated through __asan_malloc().

      Erich

      Like

      Reply ↓

  7. Is there any chance of implementing detection of dangling pointers to stack variables. Like this:

    int *p;
    void foo()
    {
    int i[5]; // error: should be static
    p = i; // p will be dangling
    }

    Liked by 1 person

    Reply ↓

    • Sorry, I just realized that you are setting global variable “p” to the address of local variable “i” and that’s your question. It’s true that the value of “p” will be dangling is used outside of method “foo” after it is set to &i[0] in “foo”.

      That would be a nice bug to catch, although I expect it to be more of a bug implemented by a new user. I’m saying that because I have seen people struggle with bugs like this because chances are the values set to i[] in “foo” would not change for a while (depending on the stack implementation and timing/stack usage in calls to other methods).

      Anyway, sorry about the original response – it is a very good question.

      Liked by 1 person

      Reply ↓

  8. I’m using an arm mcu in an embedded environment with no OS. Looking at the assembly code, my above code calls __asan_stack_malloc_1(). But I see no documentation of what this function should do. With gcc/intel/linux, it works fine (subsequent use of *p is detected).

    Liked by 1 person

    Reply ↓

    • Hi Jon,
      back to your original question:
      >>Is there any chance of implementing detection of dangling pointers to stack variables.
      Not sure what you man with ‘any chance’: it is technically certainly possible. You just would have to fill in the code for the hooks called by the compiler. I have used -fsanitize=kernel-address which turns on a subset of hooks. You would have to use -fsanitize=address for the general hooks.

      About __asan_stack_malloc_1(): I did not find much information about it: it gets called for your example if using the -fsanitize=address option at the beginning of the function. It passes a size and an address.

      I hope this helps,
      Erich

      Liked by 1 person

      Reply ↓

  9. I looked and couldn’t find any documentation of __asan_stack_malloc_1 (which is definitely what my example code and -fsantize=address causes a call to) that would help me write a replacement. Is there something out there that I missed?

    Liked by 2 people

    Reply ↓

  10. Some good news – latest versions of gcc (I tried 11.1) and -fanalyzer catch my problem at compile time. clang-tidy also catches it.

    Maybe someday there will be better documentation for implementing hooks for the sanitizer. Or better yet, full support for embedded arm.

    Liked by 2 people

    Reply ↓

  11. Hello,
    I downloaded the McuASAN code from Github and imported it into a simple stack access example. But it gave a false positive. The shadow poison value of arr is always -1 which causes this problem. Does armgcc not support to instrument the stack? I did not found the initialization of the shadow of arr from the disassembly code. Could you please help double-check this? Thanks for your help.
    Here is the example code.

    static void Test(void) {
    int arr[3] = {1,2,3};
    int i = 0;
    int result = 0;
    for(;i<3;i++){
    result += arr[i];
    }
    }

    Liked by 1 person

    Reply ↓

  12. Hi,
    Nice post. Is it possible to have something similar for clang?
    I got this message:
    clang: error: unsupported option ‘-fsanitize=kernel-address’ for target ‘arm-none-unknown-eabi’

    Thanks.

    Liked by 1 person

    Reply ↓

  13. Very nice. I’m currently trying to get it to work on an STm32H7 and I’m seeing errors where I didn’t expect any. Looking at the dissassembly it seems that library functions such as memset are not instrumented. Does this mean that I will also have to compile the library myself with the -fsanitize=kernel-address option? Any way to circumvent this?

    Liked by 1 person

    Reply ↓

  14. Which version of the gcc did you use?
    I use 8.2.0 and get undefined references to
    undefined reference to `__asan_storeN_noabort’
    undefined reference to `__asan_loadN_noabort’
    undefined reference to `__asan_storeN_noabort’
    undefined reference to `__asan_loadN_noabort’
    undefined reference to `__asan_load8_noabort’
    undefined reference to `__asan_load8_noabort’
    undefined reference to `__asan_store8_noabort’

    which dont exist in your implementation.

    Liked by 1 person

    Reply ↓

  15. How did you find out what the asan functions are supposed to do?
    I am trying to implement the “-fsanitize=address”, but I cannot find out what interface the “__asan_stack_malloc_0” and co. must have and what their purpose is. It compiles with “uptr __asan_stack_malloc_0(uptr size, uptr real_stack)” and “uptr __asan_stack_malloc_0(uptr size)”, but i crashes with both (I found the one from some apple implementation and the other from gcc8.2 sources)

    Liked by 1 person

    Reply ↓

  17. Another thing, which might also affect others: I ran into a recursive infinite loop in CheckShadow(), because the shadow memory segment is also a pointer access and it’s access caused a new CheckShadow() call, which in turn caused a new one and so one. Didnt you have this problem? I solved it by excluding the shadow memory segment from being checked in the outer if-condition of CheckShadow().

    Liked by 1 person

    Reply ↓

What do you think?

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.