MQTT with lwip and NXP FRDM-K64F Board

Posted on April 9, 2017 by Erich Styger

In the area of IoT (Internet of Things), one obvious need is to have a way to send and receive data with an internet protocol. MQTT (or Message Queue Telemetry Transport) is exactly like that: a light-weight Machine-to-Machine communication protocol. With the MQTT protocol a microcontroller (or ‘client’) can send data and/or subscribe to data. For example to the Adafruit.IO:

Adafruit MQTT IO Feed

Overview

In this article I show the basic steps to get MQTT running on the NXP FRDM-K64F board using MCUXpresso IDE, lwip and MQTT. lwip ois a small and open source TCP/IP stack which is widely used. To keep things very simple in this first post, I’m using it in bare-metal (no RTOS) mode with no encryption/security. The principle applies to any IDE/toolchain, as long there is a lwip port available for your board and IDE/toolchain. I’m using the MCUXpresso IDE as it nicely integrates with the MCUXpresso SDK which includes a lwip port for the FRDM-K64F.

I’m using the following software and tools:

MCUXpresso IDE v10.0.0 b344
MCUXpresso SDK for FRDM-K64F v2.2.0
lwip v2.0.0

MQTT

MQTT is a very cool connectivity protocol on top of TCP/IP with a subscribe/publish messaging transport. If you don’t know MQTT, then I recommend have a read here:

MQTT.org web site and overview: http://mqtt.org/
Adafruit article about MQTT: https://learn.adafruit.com/mqtt-adafruit-io-and-you?view=all#why-mqtt
Excellent MQTT article series over at HiveMQ: http://www.hivemq.com/blog/how-to-get-started-with-mqtt
HiveMQ MQTT essentials: http://www.hivemq.com/blog/mqtt-essentials-part-1-introducing-mqtt

Broker

To use MQTT, I need a server or ‘broker’. Eclipse hosts the Mosquitto project (https://mosquitto.org/) which is probably one of the most used MQTT brokers. You should be able to find plenty of tutorials about installing Mosquitto on Linux, Raspberry Pi or even Windows.

Having a local broker is great for starting a MQTT project, as you can easily debug it and you get things up and running with a local broker, and then you can switch to a broker/provider in the internet. I have used the following tutorial to install Mosquitto on Windows 10:

https://sivatechworld.wordpress.com/2015/06/11/step-by-step-installing-and-configuring-mosquitto-with-windows-7/

Running Mosquitto on Windows

Adafruit.io Broker

I love stuff from Adafruit! And they host a very useful MQTT broker, see https://learn.adafruit.com/mqtt-adafruit-io-and-you/overview.

To send and receive data, I have to use a user name with a password (the AIO key below). 💡 If using the MQTT port 1883, information is sent in clear text which is definitely not something I want. Instead, port 8883 with TLS v1.2 shall be used! In this tutorial I’m assuming a local broker behind a firewall so using port 1883 is ok (but only in this case!).

Adafruit IO Settings

MQTT Client Tools

It is very useful to have a client tool installed on the host machine. Have a read at http://www.hivemq.com/blog/seven-best-mqtt-client-tools and choose one. What I am using is the MQTT.fx (http://www.jensd.de/apps/mqttfx/) utility which has a nice GUI to monitor the broker status:

MQTT.fx Broker Status

To use MQTT.fx with Adafruit.io, see https://cdn-learn.adafruit.com/downloads/pdf/desktop-mqtt-client-for-adafruit-io.pdf.

Creating lwip Project

I started my MQTT project from one of the lwip example projects which come with the MCUXpresso SDK:

Import SDK Examples

And the used the bare metal ‘ping’ example as the base:

lwip ping example

The example project used in this article is on GitHub (https://github.com/ErichStyger/mcuoneclipse/tree/master/Examples/MCUXpresso/FRDM-K64F/FRDM-K64F_lwip_lwip_mqtt_bm).

MQTT Drivers

From the ‘sources’ folder of that project, you need

mqtt.h: interface to mqtt.c
mqtt.c: MQTT implementation on top of lwip
mqtt_opts.h: configuration header file

lwip Initialization

The following code is a standard (bare metal) lwip initialization:

Initialize board pins and clocks
Initialize timer for lwip (time_init())
Configure network interface addresses (own (static) address, netmask and gateway)
lwip initialization (lwip_init())

int main(void) {
 struct netif fsl_netif0;
 ip4_addr_t fsl_netif0_ipaddr, fsl_netif0_netmask, fsl_netif0_gw;

 SYSMPU_Type *base = SYSMPU;
 BOARD_InitPins();
 BOARD_BootClockRUN();
 BOARD_InitDebugConsole();
 /* Disable SYSMPU. */
 base->CESR &= ~SYSMPU_CESR_VLD_MASK;

 time_init();

 IP4_ADDR(&fsl_netif0_ipaddr, configIP_ADDR0, configIP_ADDR1, configIP_ADDR2, configIP_ADDR3);
 IP4_ADDR(&fsl_netif0_netmask, configNET_MASK0, configNET_MASK1, configNET_MASK2, configNET_MASK3);
 IP4_ADDR(&fsl_netif0_gw, configGW_ADDR0, configGW_ADDR1, configGW_ADDR2, configGW_ADDR3);

 lwip_init();

 netif_add(&fsl_netif0, &fsl_netif0_ipaddr, &fsl_netif0_netmask, &fsl_netif0_gw, NULL, ethernetif_init, ethernet_input);
 netif_set_default(&fsl_netif0);
 netif_set_up(&fsl_netif0);

 PRINTF("\r\n************************************************\r\n");
 PRINTF(" MQTT Bare Metal example\r\n");
 PRINTF("************************************************\r\n");
 PRINTF(" IPv4 Address : %u.%u.%u.%u\r\n", ((u8_t *)&fsl_netif0_ipaddr)[0], ((u8_t *)&fsl_netif0_ipaddr)[1],
 ((u8_t *)&fsl_netif0_ipaddr)[2], ((u8_t *)&fsl_netif0_ipaddr)[3]);
 PRINTF(" IPv4 Subnet mask : %u.%u.%u.%u\r\n", ((u8_t *)&fsl_netif0_netmask)[0], ((u8_t *)&fsl_netif0_netmask)[1],
 ((u8_t *)&fsl_netif0_netmask)[2], ((u8_t *)&fsl_netif0_netmask)[3]);
 PRINTF(" IPv4 Gateway : %u.%u.%u.%u\r\n", ((u8_t *)&fsl_netif0_gw)[0], ((u8_t *)&fsl_netif0_gw)[1],
 ((u8_t *)&fsl_netif0_gw)[2], ((u8_t *)&fsl_netif0_gw)[3]);
 PRINTF("************************************************\r\n");

 DoMQTT(&fsl_netif0);
 for(;;) {}
 return 0;
}

The MQTT application logic is implemented in DoMQTT() function.

Connect to MQTT Broker

The following code shows how to connect with lwip to the MQTT broker, adopted from the lwip 2.0.2 code.

The address of the mqtt_client_t structure is passed from the caller:

mqtt_client_t mqtt_client;
mqtt_do_connect(&mqtt_client);

which calls mqtt_do_connect():

static void mqtt_do_connect(mqtt_client_t *client) {
 ip4_addr_t broker_ipaddr;
 struct mqtt_connect_client_info_t ci;
 err_t err;

 IP4_ADDR(&broker_ipaddr, configBroker_ADDR0, configBroker_ADDR1, configBroker_ADDR2, configBroker_ADDR3);

 /* Setup an empty client info structure */
 memset(&ci, 0, sizeof(ci));

 /* Minimal amount of information required is client identifier, so set it here */
 ci.client_id = configMQTT_CLIENT_NAME;
 ci.client_user = configMQTT_CLIENT_USER;
 ci.client_pass = configMQTT_CLIENT_PWD;

 /* Initiate client and connect to server, if this fails immediately an error code is returned
   otherwise mqtt_connection_cb will be called with connection result after attempting
   to establish a connection with the server.
   For now MQTT version 3.1.1 is always used */
 err = mqtt_client_connect(client, &broker_ipaddr, MQTT_PORT, mqtt_connection_cb, 0, &ci);

 /* For now just print the result code if something goes wrong */
 if(err != ERR_OK) {
   printf("mqtt_connect return %d\n", err);
 }
}

The example above uses a static client address. The client information structure is filled with a client name, a password and user name (both can be NULL), and then it connects to the broker with a connection callback (mqtt_connection_cb).

Connection Callback

In the connection callback, if the connection gets accepted, it sets up three callback: one for incoming publish callback (mqtt_incoming_publish_cb()), one for incoming data (mqtt_incoming_data_cb()) and one for subscription requests (mqtt_sub_request()):

static void mqtt_connection_cb(mqtt_client_t *client, void *arg, mqtt_connection_status_t status) {
 err_t err;

 if(status == MQTT_CONNECT_ACCEPTED) {
   printf("mqtt_connection_cb: Successfully connected\n");

   /* Setup callback for incoming publish requests */
   mqtt_set_inpub_callback(client, mqtt_incoming_publish_cb, mqtt_incoming_data_cb, arg);

   /* Subscribe to a topic named "subtopic" with QoS level 1, call mqtt_sub_request_cb with result */
   err = mqtt_subscribe(client, "subtopic", 1, mqtt_sub_request_cb, arg);

   if(err != ERR_OK) {
     printf("mqtt_subscribe return: %d\n", err);
   }
 } else {
   printf("mqtt_connection_cb: Disconnected, reason: %d\n", status);

   /* Its more nice to be connected, so try to reconnect */
   mqtt_do_connect(client);
 }
}

Below are example implementation of the three callbacks:

/* The idea is to demultiplex topic and create some reference to be used in data callbacks
 Example here uses a global variable, better would be to use a member in arg
 If RAM and CPU budget allows it, the easiest implementation might be to just take a copy of
 the topic string and use it in mqtt_incoming_data_cb
*/
static int inpub_id;
static void mqtt_incoming_publish_cb(void *arg, const char *topic, u32_t tot_len) {
 printf("Incoming publish at topic %s with total length %u\n", topic, (unsigned int)tot_len);

 /* Decode topic string into a user defined reference */
 if(strcmp(topic, "print_payload") == 0) {
   inpub_id = 0;
 } else if(topic[0] == 'A') {
   /* All topics starting with 'A' might be handled at the same way */
   inpub_id = 1;
 } else {
   /* For all other topics */
   inpub_id = 2;
 }
}

static void mqtt_incoming_data_cb(void *arg, const u8_t *data, u16_t len, u8_t flags) {
 printf("Incoming publish payload with length %d, flags %u\n", len, (unsigned int)flags);

 if(flags & MQTT_DATA_FLAG_LAST) {
   /* Last fragment of payload received (or whole part if payload fits receive buffer
   See MQTT_VAR_HEADER_BUFFER_LEN) */

   /* Call function or do action depending on reference, in this case inpub_id */
   if(inpub_id == 0) {
     /* Don't trust the publisher, check zero termination */
     if(data[len-1] == 0) {
       printf("mqtt_incoming_data_cb: %s\n", (const char *)data);
     }
   } else if(inpub_id == 1) {
     /* Call an 'A' function... */
   } else {
     printf("mqtt_incoming_data_cb: Ignoring payload...\n");
   }
 } else {
   /* Handle fragmented payload, store in buffer, write to file or whatever */
 }
}

static void mqtt_sub_request_cb(void *arg, err_t result) {
 /* Just print the result code here for simplicity,
 normal behaviour would be to take some action if subscribe fails like
 notifying user, retry subscribe or disconnect from server */
 printf("Subscribe result: %d\n", result);
}

Publishing Data

To publish data, below is an example function. Replace the example topic “erichs/f/test” with your own topic.

static void my_mqtt_publish(mqtt_client_t *client, void *arg) {
 const char *pub_payload= "abcd";
 err_t err;
 u8_t qos = 2; /* 0 1 or 2, see MQTT specification */
 u8_t retain = 0; /* No don't retain such crappy payload... */
 err = mqtt_publish(client, "erichs/f/test", pub_payload, strlen(pub_payload), qos, retain, mqtt_pub_request_cb, arg);
 if(err != ERR_OK) {
   printf("Publish err: %d\n", err);
 }
}

A callback (mqtt_pub_request_cb()) is provided. In my case below I print an error code if it fails:

/* Called when publish is complete either with success or failure */
static void mqtt_pub_request_cb(void *arg, err_t result) {
 if(result != ERR_OK) {
   printf("Publish result: %d\n", result);
 }
}

Summary

With MQTT I can publish/subscribe data on the internet. It is light-weight, open source and makes it ideal for any IoT kind of applications. Using it with the FRDM-K64F and lwip took me a while to get things working (I still face some issues in combination with FreeRTOS, so this is likely subject of a follow-up post). Another thing which I have not covered here is the need for secure connections: using an unencrypted MQTT port 1883 is not something to be used for sensitive data. Instead, port 8883 with SSL/TLS should be used. Yet another reason for a follow-up post I think?

Happy MQTTing:-)

46 thoughts on “MQTT with lwip and NXP FRDM-K64F Board”

Luca Gotti on April 11, 2017 at 08:07 said:

Great article Erich, very interested to see the encrypted version …
Which stack are u going to use ? Maybe WolfSSL ?

Regards, Luca

LikeLike

Reply ↓
- Erich Styger on April 11, 2017 at 08:08 said:
  
  Hi Luca,
  I’m in the process of integrating embed TLS. I have concerns about using WolfSSL because of licensing.
  
  LikeLike
  
  Reply ↓
  - Luca Gotti on April 11, 2017 at 08:59 said:
    
    Hi Erich,
    
    Yes, I’ve seen it now the GitHub projetc : surely the mbed TLS has less concerns about licensing ( open source Apache 2.0 or GNU Public License Version 2.0 ) .
    
    Is it the first time are you using mbed TLS or have you already used in some other projects ?
    
    Regards and thanks !
    
    LikeLike
    
    Reply ↓
    - Erich Styger on April 11, 2017 at 10:04 said:
      
      Hi Luca,
      it is the first time I’m using mbed TLS. Still learning it 🙂
      Erich
      
      LikeLike
Roman on April 11, 2017 at 11:10 said:

Hey Erich
Maybe following kickstarter project might be interesting for you (for projects in future 🙂 ): https://www.kickstarter.com/projects/nanomesher/wireless-human-machine-interface
A wireless display which also takes advantage of MQTT.
Regards
Roman

LikeLike

Reply ↓
- Erich Styger on April 11, 2017 at 12:41 said:
  
  Hi Roman,
  indeed, an interesting board and concept!
  Erich
  
  LikeLike
  
  Reply ↓
Luca Gotti on April 11, 2017 at 11:33 said:

Hi Erich,
I wish you to implement it with succes …

( Yes, also because it could be very useful for my projects … (: )

LikeLike

Reply ↓
- Erich Styger on April 11, 2017 at 12:35 said:
  
  Hi Luca,
  thank you :-).
  
  Erich
  
  LikeLike
  
  Reply ↓
henry zhao on April 12, 2017 at 18:03 said:

Hi Erich,
Do you have any code about how to use DHCP with in FreeRTOS? and send broadcast?

Thanks a lot

Henry

LikeLike

Reply ↓
- Erich Styger on April 12, 2017 at 18:42 said:
  
  Hi Henry,
  yes, have a look at https://mcuoneclipse.com/2015/10/28/tutorial-lwip-with-the-freertos-and-the-freescale-frdm-k64f-board/ and the project on GitHub for this article.
  
  Erich
  
  LikeLike
  
  Reply ↓
  - henry zhao on April 12, 2017 at 18:58 said:
    
    thanks Erich,
    I have download the project and read it, I have a question about this:
    
    after “err = dhcp_start(&fsl_netif0); ” success,
    you call “netif_set_up(&fsl_netif0);”
    
    but I found this:
    
    As such, the list of functions that may be called from
    other threads or an ISR is very limited! Only functions
    from these API header files are thread-safe:
    – api.h
    – netbuf.h
    – netdb.h
    – netifapi.h
    – pppapi.h
    – sockets.h
    – sys.h
    
    so, is this problem? because they are run in difference thread
    
    I don’t know is it problem, or doesn’t matter
    
    thank very much
    
    LikeLike
    
    Reply ↓
    - Erich Styger on April 12, 2017 at 20:31 said:
      
      I have not seen any problem, and I have used it as in this project.
      
      LikeLike
  - henry zhao on April 12, 2017 at 20:32 said:
    
    thanks a lot, this is helpful for me.
    
    LikeLike
    
    Reply ↓
Pingback: Introduction to Security and TLS (Transport Security Layer) | MCU on Eclipse
Pingback: Enable Secure Communication with TLS and the Mosquitto Broker | MCU on Eclipse
Pingback: Easter Weekend Apple Juice Brined Pulled Pork Smoked on Beech Wood | MCU on Eclipse
Pingback: Tutorial: Secure TLS Communication with MQTT using mbedTLS on top of lwip | MCU on Eclipse
Pingback: Tuturial: mbedTLS SLL Certificate Verification with Mosquitto, lwip and MQTT | MCU on Eclipse
Josep Casanovas on May 2, 2017 at 16:17 said:

Hi Erich,

Nice post! Very useful.
I have experienced some issues with this mqtt lib.
If I publish MQTT messages with a not too high frequency (10Hz) it works properly.
Log:
mqtt_publish: Publish with payload length 89 to topic “TEST”
mqtt_output_send: tcp_sndbuf: 8760 bytes, ringbuf_linear_available: 97, get 27,4
mqtt_tcp_sent_cb: Calling QoS 0 publish complete callback

If the frequency is higher, appears some problems at write/output part.
Log:
mqtt_publish: Publish with payload length 93 to topic “TEST”
mqtt_publish: Publish with payload length 93 to topic “TEST”
mqtt_publish: Publish with payload length 91 to topic “TEST”

Have you experienced something like this?
Are this lib limited by publish frequency?

Thanks for this post again!

LikeLike

Reply ↓
- Josep Casanovas on May 2, 2017 at 16:25 said:
  
  Sometimes, I have received some different logs which leads to a board’s block (doesn’t publish anymore and doesn’t seems to works–>this leads me to push reset button).
  Log:
  mqtt_cyclic_timer: Timer should not be running in state 0
  mqtt_tcp_recv_cb: client->conn == pcb
  
  If someone has any idea, would be fantastic!
  
  LikeLike
  
  Reply ↓
- Erich Styger on May 2, 2017 at 17:43 said:
  
  Hi Josep,
  thanks :-). I need to look into this. Are you using the latest version from GitHub as the project has been evolved a bit after that article has been published.
  
  LikeLike
  
  Reply ↓
  - Josep Casanovas on May 3, 2017 at 11:10 said:
    
    Ok. I will stay tuned to any update on your project.
    I have the premonition that this could be due to printf via serial that slow down the running code.
    
    LikeLike
    
    Reply ↓
    - Erich Styger on May 3, 2017 at 21:41 said:
      
      Hi Josep,
      yes, printf is very likely the reason for this. I had not a chance to try it out myself with higher frequency. But you could try to switch off the printf output?
      
      LikeLike
    - Erich Styger on May 5, 2017 at 08:56 said:
      
      Hi Josep,
      I created a simple test with publishing at 50 Hz and 100 Hz, and have not faced any problems, even with the debug messages turned on, at least with the latest version I run which is on GitHub.
      But you have to make sure that you call ethernetif_input(netifp); /* Poll the driver, get any outstanding frames */ sys_check_timeouts(); /* Handle all system timeouts for all core protocols */ frequent enough.
      
      LikeLike
Sergey on June 6, 2017 at 16:01 said:

Hello, Erich
Thank you for all of your work regarding Kinetis MCU’s – it’s very helpful and interesting data!

I’ve started with FRDM-K64F devboard and lwip ping example before going further ahead to MQTT protocol. Ping application worked fine at the first glance, however it goes to hard fault after some random time. I’m using an infinite ‘ping local_address’ command in linux console for testing. I found that ENET_ReadFrame function in fsl_enet.c contains memcpy function (line 942) which seems to use a wrong offset during some circumstances thus becomes a reason of memory corruption.
I still cannot find which situation leads to this fail, maybe you’ve come across similar bug during testing?

Best regards, Sergey

LikeLike

Reply ↓
- Erich Styger on June 6, 2017 at 18:13 said:
  
  Hi Sergey,
  I have not seen the failure you describe. But I saw that the TCP driver provided by NXP in the SDK v2.x is not fully thread and RTOS aware. What I ended up is serving TCP from a single RTOS thread (not multiple ones) to solve that problem. In earlier SDK versions it was possible to use a special driver mode to avoid that problem, see the comments section in https://mcuoneclipse.com/2015/10/28/tutorial-lwip-with-the-freertos-and-the-freescale-frdm-k64f-board/
  I hope this helps,
  Erich
  
  LikeLike
  
  Reply ↓
  - Sergey on June 7, 2017 at 11:12 said:
    
    Thanks for the link, I’m going to add FreeRTOS later too.
    
    Though the problem appears even with a baremetal example.
    
    I’ve just found that everything works great if semihosting is disabled. Board has been running during the whole night and still operational.
    Seems that ethernet buffer overflows during processing time-consuming semihosting printf operations or something like that..
    
    I also got some odd behavior during debug session. Firstly I’ve used default CMSIS-DAP debug and it wasn’t possible to exit from debug session without repowering the board. I’ve found your article regarding j-link / openSDA then switched to this firmware and now the problem is gone.
    
    Later on I’ve discovered some CPU freezes after pausing during debug session when using redlib/semi. Only board reset / re-powering was able to fix this issue.
    I’ve tried to switch to newlib and it seems to work ok now. Except for hard fault issues when semihosting is on -)
    
    LikeLike
    
    Reply ↓
    - Erich Styger on June 10, 2017 at 08:04 said:
      
      Did you disable semihosting in the debugger? Semihosting is pretty evil in an embedded environment. Great for a quick demo as you don’t need a terminal or real UART connection, but should not be used in a real application. It might be that the hard fault comes from a stack overflow, so make sure your MSP (main stack pointer size) and space is large enough, as the semihosting interrupt adds to the interrupt load. And yes, it will delay things too.
      
      LikeLike
    - Sergey on June 13, 2017 at 09:42 said:
      
      Stack and heap size has been set to 4Kb during testing.
      Need to check out MSP size, thank you.
      And yes, semihosting is disabled in debugger configuration..
      
      I’m going to use it more carefully now -)
      
      LikeLike
Sergey on June 23, 2017 at 12:47 said:

Hello, Erich

I’ve got some strange issues during debugging sessions. I’m using latest mcu expresso (with NXP toolchain) + frdm-k64f + segger opensda 2.0. I switched to windows 10 after some strange debugger behaviour during work on Ubuntu 17 o/s to make sure it’s just an incompatibility issue.

But I’ve just realized that problem is still here. I’ve tried to use your bare-metal mqtt+lwip example, then modified application level to constantly send a mqtt messages to server. No problems with debugging at all.
Then I just merged this code with a freescale example for evalution of on-board accelerometer. And I found some issues during debugging session. Most of the time after I press ‘run’ button the board hangs with no indication of problem. Even code stepping (step over actually) leads to that issue sometimes. Only board reset is make sense in that case. No hardware faults. I double checked a stack and heap size, but I don’t use FreeRTOS and it seems that heap/stack are still large enough for lwip sockets.
When I’ve commented out some parts of the code debugger just stopped after ‘run’ at some part of tcp/ip code though allowing me to make steps further.

Suddenly I realized that board is works fine after flashing. If debugger only being used to flash the board everything works just great. But if it hangs then only way to make it alive is full repowering via usb connect/disconnect. Also I found there is no multiple GDB stuck in processes.
I’ve tried to h/w reset board many times after succesfull start-up and found no any sight of troubles. So I suppose it’s not some random code glitches but debugging issue.

Have you ever come across that kind of issue?

Best regards, Sergey

LikeLike

Reply ↓
- Erich Styger on June 23, 2017 at 16:04 said:
  
  Hi Sergey,
  I might have seen this, but not sure. In my case the FreeRTOS Kernel Awareness seemed to have some problems if the project was not a FreeRTOS one. In any case, if you have FreeRTOS kernel awareness enabled, you might give it a try to disable it in the Segger settings.
  The other thing is that I had seen some strange things with older SEGGER versions. You might download a new version from segger.com.
  
  I hope this helps,
  Erich
  
  LikeLike
  
  Reply ↓
  - Sergey on June 26, 2017 at 18:43 said:
    
    Hello, Erich
    
    I haven’t installed the FreeRTOS Kernel Awareness. Segger version is the latest one.
    It looks like some h/w breakpoint hasn’t been deleted from CPU after debug session. It could be a reason of unexpected breakpoint call.
    As for board freezes when exiting debugging – still no clear reason of this problem. I’ll try investigate further during project debugging.
    
    Thank you.
    
    LikeLike
    
    Reply ↓
Krasi Gichev on October 28, 2017 at 19:53 said:

Quite informative article – I also found the “built in” mqtt client in recent version of LWIP. I used it but I have a specific corner case where I believe the code needs improvements.
In the “mqtt_incoming_data_cb” callback in your example we can see that the mqtt client delivers data by data, len, and flags arguments.
In the special case where the published data is larger than a single frame or TCP connection buffers, this callback will be executed several times with fragments of the data. This is good but the current API of the mqtt.c/h does not allow the application to “slow” down the transfer rate. This is often needed and on LWIP core level this is implemented – the recv callback marks the beginning of the asynchronous operation and the opposite call of “tcp_recved” marks the end. This mechanism allows to delay the tcp ack and the tcp socket handles the flow control needed in this case.
I have created an issue to the author of the mqtt client in LWIP in github but I cannot attach the link here for some reason – still the suffix after github is “erian747/lwip/issues/1”.
But I would like to know your opinion – beside staying away from large transfers, do you think that there is a better way to solve this special case? I know MQTT defines something like 260Mbytes for maximum message size, so I believe almost all clients would have to implement “fragmented” ASYNC transfers somehow, even those running on i9 with 32+Gbytes of RAM…

LikeLike

Reply ↓
- Erich Styger on October 28, 2017 at 20:45 said:
  
  Interesting, I have not run into that particular case. Simply because to me MQTT (and everyting IoT) is about high latency/small amount of data. So I stay away from generating large sets of data, and rater have it split into small data packages so I don’t run into that problem. Other data exchange methods have to be used imho for larger sets of data.
  
  LikeLike
  
  Reply ↓
Krasi Gichev on October 28, 2017 at 21:46 said:

I can confirm that you are not the first who says this, but I don’t agree. I am still investigating if other protocols (e.g. classic http, ftp) provide something more that the mqtt protocol and broker.
I strongly believe that TCP layer is good for 256Mbytes, and it is also the transport of choice for http, ftp, scp and so on. So problems should be somewhere above it (if those exists).
My idea is that I want to “export” endpoints of microservices that run inside particular device, often embedded but not always.
So, even for small cortex-m project, or esp8266/esp32 at some point one needs to deal with hundred kbytes or several mbytes of data. I mean the OTA firmware update. The classical approach is to setup an FTP server somewhere, place the FW there, than send a “trigger” signal to the device to go and take the file.
Over MQTT one could achieve all of these steps over single protocol – first you publish your new firmware file as retained message to specia topic, than … go for a beer… Some devices will get the “news” because those were subscribed for “ACME/firmware/update” topic and will receive the file. If other devices are configured to do a startup check only they will not “get the news” but when those get restarted later will “subscribe” there and magically (respect to MQTT devs) will receive the update….
So, how is this a bad idea, and what speak against it? Maybe some common sense that “it cannot be so easy” 🙂
With FTP approach you need to embed both MQTT and FTP clients (because FTP is not good for commanding IoT node and other IoT needs). Also, you need to setup the servers and all the infrastructure down to firewall rules to get access.
Isn’t it easier with MQTT only?

LikeLike

Reply ↓
- Erich Styger on October 29, 2017 at 05:28 said:
  
  I don’t say it is a bad idea. And if MQTT would support this: great! But such a functionality is very low on my priority/wish list, and it seems for many others too. As these things get implemented by volunteers, they (including myself) implement what they need if it does not exist. So I feel if you need this, there is nothing which would prevent you to implement this and to contribute this back?
  
  LikeLike
  
  Reply ↓
  - Krasi Gichev on October 29, 2017 at 08:20 said:
    
    Sure, that’s correct – I already made a patch and I am testing. I just wanted to discuss this and get your opinion.
    
    LikeLike
    
    Reply ↓
Mark on November 7, 2017 at 23:07 said:

Hi Eric

I just cloned your Github repository since I wanted to try MQTT but can’t build the project in MCUXpress. Eg, when I have imported FRDM-K64F_lwip_mqtt_bm and try to build it it first says that there is no MCU target configured so I select Generic M4 (there are no other options for Kinetis in the version that I have). When I build it fails with
d:\freescale\mcuxpressoide_10.0.0_344\ide\tools\arm-none-eabi\include\sys\types.h:73:20: error: conflicting types for ‘in_addr_t’
typedef __uint32_t in_addr_t; /* base type for internet address */
^
In file included from ../source/net.c:37:0:
D:\mcuoneclipse\Examples\MCUXpresso\FRDM-K64F\FRDM-K64F_lwip_lwip_mqtt_bm\lwip\src\include/lwip/inet.h:55:15: note: previous declaration of ‘in_addr_t’ was here
typedef u32_t in_addr_t;
^
make: *** [source/net.o] Error 1
make: *** Waiting for unfinished jobs….
Finished building: ../source/lwip_mqtt.c
Finished building: ../source/mqtt.c

I tried a few other projects, including FRDM-K64F_Demo, and all asked for an MCU target. This one built but it didn’t fit into the default memory – is it necessary to set up the MCU target’s memory or do you use linker scripts that can be added instead?

Are the problems due to my MCUXpress version [MCUXpresso IDE v10.0.0 [Build 344] [2017-03-21] ] or some additional plugins that are needed for these project?
Thanks in advance

Regards

Mark

P.S. Did you look into mbedMQTT instead of the the one in LWIP?

LikeLike

Reply ↓
- Erich Styger on November 8, 2017 at 18:04 said:
  
  Hi Mark,
  if you had a message about the MCU target not configured, you probably have not loaded the SDK for the K64F?
  Unfortunately, the project still needs the SDK for some of the project settings.
  Let me know if this helps,
  Erich
  
  LikeLike
  
  Reply ↓
  - Mark on November 8, 2017 at 20:50 said:
    
    Erich
    
    I installed SDK for FRDM-K64F and then I could choose the correct board.
    
    However I still get errors. The first is a conflict between the typedef for in_addr_t in inet.h and in the header used by the IDE. I changed the typedef in inet.h to __uint32_t and then all files compiled.
    
    The remaining errors are linker errors due to missing malloc functions:
    pvPortMaoc(), vPortFree() and cPortGetFreeHeapSize.
    
    I see that lwip_mqtt.c uses malloc() and free() and so do some FreeRTOS files in “Generated code” – FRTOS1.c, queue.c, tasks.c
    
    In fact I tried to build the “Bare-Metal” version so I am wondering why there are FreeRTOS files in it that are causing difficulties? Do I need to exclude some files or do the generated files need to be re-generated somehow?
    If malloc() and free() are required how do the port versions get set?
    
    Thanks
    
    Regards
    
    Mark
    
    P.S. What code size is expected? It is showing about 400k Flash and 192k SRAM at the moment.
    
    LikeLike
    
    Reply ↓
    - Erich Styger on November 9, 2017 at 20:51 said:
      
      Mark,
      the project started as bare-metal one, and later I added FreeRTOS, but I have not changed the project name afterwards (probably I should, but had no bandwidth yet).
      You don’t need to exclude any files, it should work as is on GitHub (but I have pushed a few changes as I made some recent updates for generating gcov information).
      Without any optimizations at all, the code size is the following:
      text data bss dec hex filename
      133720 2220 131004 266944 412c0 FRDM-K64F_lwip_mqtt_bm.axf
      
      LikeLike
    - Mark on November 9, 2017 at 21:18 said:
      
      Erich
      There are two MQTT projects in the repository:
      “FRDM-K64F_lwip_lwip_mqtt_bm” and “FRDM-K64F_lwip_mqtt_FreeRTOS”
      I have been trying “FRDM-K64F_lwip_lwip_mqtt_bm” but it fails with the errors noted in the last post.
      I just tried “FRDM-K64F_lwip_mqtt_FreeRTOS” instead and this one builds. It gives me
      
      Memory region Used Size Region Size %age Used
      PROGRAM_FLASH: 94260 B 1 MB 8.99%
      SRAM_UPPER: 82656 B 192 KB 42.04%
      SRAM_LOWER: 0 GB 64 KB 0.00%
      FLEX_RAM: 0 GB 4 KB 0.00%
      
      text data bss dec hex filename
      94224 36 82620 176880 2b2f0 FRDM-K64F_lwip_mqtt_FreeRTOS.axf
      
      Therefore it looks like the “FRDM-K64F_lwip_lwip_mqtt_bm” target is broke, so I ‘ll try continuing with the FreeRTOS one instead.
      
      Regards
      
      Mark
      
      LikeLike
Andre on May 7, 2018 at 20:13 said:

Very nice project!
Can you tell something about the performance?
Speed, datarate and round trip time?
Thx in advance 🙂

LikeLike

Reply ↓
- Erich Styger on May 8, 2018 at 14:52 said:
  
  I did not perform any specific benchmarks. performance is heavily affectect by your network speed and your MQTT server load/speed.
  
  LikeLike
  
  Reply ↓
Gary Lynch on June 6, 2020 at 23:58 said:

Erich,

Thanks for making this resource available. I want to get my feet wet with MQTT for the first time and your solution is the most thorough I have found. It looks like a pair of NXP FRDM-K64F board clients and a Raspberry Pi are the simplest platform for a newbie to bring up.

I notice the last post is 2 years old. Have you in the interim gained any new insights that will head off any of the glitches I see on here?

LikeLike

Reply ↓
- Erich Styger on June 7, 2020 at 15:59 said:
  
  That project is running as I have created it two years ago. I did not need to touch it, as it worked as expected for me. A possible glitch could be if you upgrade to newer lwip libraries as I remember seeing some issues around it.
  
  LikeLike
  
  Reply ↓