[ESP32 x PlatformIO] How to coexist the framework with Arduino and ESP-IDF

Introduction.

Developing “Mia,” a talking cat-shaped robot that speaks dialect.

https://mia-cat.com

Previously, in this article, I implemented moving AWS IoT-related configuration files from the LittleFS area to the NVS area.

【ESP32 × PlatformIO】設定ファイル（AWS IoT）をLittleFSからNVS領域に移動

現状、設定ファイルをdata配下に置きビルド時はSPIFFS領域に含まれているが、ダウンロードの処理の不具合などでLittleFSのストレージが溢れた際、証明書の読み込みなど失敗する可能性があるので、NVS領域への書き込みと読み取りに変更す...

kazulog.fun

However, as it is vulnerable in terms of security, we tried to implement NVS encryption this time and came across the following description.

https://github.com/espressif/arduino-esp32/issues/9233

Flash Encryption and Secure Boot are not provided by the Arduino IDE and will not be provided in the future orz

So, first, we need to change from the Arduino framework we are using now to the ESP-IDF framework.

Incidentally, the current settings in the platform.ini file are as follows.

; PlatformIO Project Configuration File 
 ; https://docs.platformio.org/page/projectconf.html 
 [env].
 platform = espressif32 
 framework = arduino 
 board = esp32dev

In this issue, we describe how to make framework changes in PlatformIO.

See description on the official Espressif website below.

https://docs.platformio.org/en/latest/frameworks/espidf.html#configuration

Proceed in the following order

• Add espidf to framework in platform.ini
• Add CmakeLists.txt file to root and src directories
• Add sdkconfig.defaults file and add necessary settings so that arduino can also boot
• Add arduino libraries needed to start also esp-idf in arduino library

ESP-IDF Framework Overview

What is the CmakeLists.txt file?

The ESP-IDF framework uses a build system called CMake.

The CMakeLists.txt file instructs the CMake build system how to build the project, and is needed to manage the project configuration and build process when using ESP-IDF.

Projects that use ESP-IDF in PlatformIO typically have the following structure

project_dir/ 
 ├── include/ 
 ├── src/ 
 │ ├── CMakeLists.txt 
 │ └── main.c 
 ├── CMakeLists.txt 
 └── platformio.ini

We can see that there are two CmakeLists.txt files, one in the root directory and one in the src directory. We will look at the purpose of each and how they are described.

CMakeLists.txt in the root directory

Configure settings for the entire project. Typically, this file will contain the following

# The following lines of boilerplate have to be in your project's CMakeLists 
 # in this exact order for cmake to work correctly 
 cmake_minimum_required(VERSION 3.16.0) 
 
 include($ENV{IDF_PATH}/tools/cmake/project.cmake) 
 project(project-name)

project-name is the name of the project to set within CMake, which can be any name. CMake will use this name to generate project-specific variables to manage the build process.

Since PlatformIO project names are used primarily within the PlatformIO environment and CMake project-name is used within the CMake and ESP-IDF build systems, it is not necessary to match PlatformIO project names, but matching them would readability is enhanced.

CMakeLists.txt in src directory

Control the build process for a specific component, in this case the main component of the application. In a minimal setup, the component is registered with the build system as follows

idf_component_register(SRCS "main.c")

These CMakeLists.txt files are necessary for the project to be built correctly using the ESP-IDF build system. If additional components or special build settings are required, the corresponding settings should be added to these files.

sdkconfig.defaults

The platformio.ini file is the main configuration file for the PlatformIO project, defining the project build settings, boards to be used, frameworks, library dependencies, etc.

On the other hand, sdkconfig.defaults is an ESP-IDF-specific configuration file that allows you to predefine various settings for the ESP-IDF framework. This includes system behavior, enabling/disabling of features, performance tuning, etc. It should be placed in the root directory of the project.

Now that we have an overview of the ESP-IDF framework, we will actually replace the current setup.

Update platformio.ini: add espidf to framework

Change the framework part of the platformio.ini file from arduino to espidf.

At first, I thought I had to specify framework = espidf, and the official website also stated so, so I followed that procedure.

In that case, the following work will occur, and it will be very difficult (in fact, it was quite difficult to proceed to the middle of the process)

Arduino library management:.

• In order to use the Arduino libraries, it is necessary to git clone all Arduino libraries under the components folder, which may result in several GB of project file space.

Code rewriting

• If you build with the espidf framework specified, you will get an error for everything arduino library related, so you need to rewrite all the parts where the arduino library is included in arduino.h.

In file included from  src/app_config .cpp:1: 
 src/app_config.h:1:10: fatal error:  FS .h: No such file or directory 
 
 ************************************************************ 
 *  Looking for FS.h dependency? Check our library registry! 
 * 
 * CLI > platformio lib search "header:FS.h" 
 * Web  > https://registry.platformio.org/search?q=header:FS.h 
 * 
 ************************************************************ 
 
     1  |   #include <FS.h> 
       |           ^~~~~ ~
 compilation terminated. 
 Compiling .pio/build/release/src/donwload.cpp.o 
 In file included from  src/button_manager .cpp:1: 
 src/button_manager.h:1:10: fatal error: Arduino.h: No such file or directory 
 
 ***************************************************************** 
 *  Looking for Arduino.h dependency? Check our library registry! 
 * 
 * CLI > platformio lib search "header:Arduino.h" 
 * Web  > https://registry.platformio.org/search?q=header:Arduino.h 
 * 
 ***************************************************************** 
 

However, the following statement says that it is possible to use the Arduino and ESP-IDF frameworks together with PlatformIO. In my case, I noticed this in the middle of the process and ate up a lot of time.

[ env] 
 platform = espressif32 @ 6.7.0 
 framework = arduino, espidf

By specifying this, many existing Arduino libraries can be used directly, eliminating the need for porting and rewriting code.

So, set framework = arduino, espidf.

In fact, the espidf official documentation includes an example (blink, wifiscan) of using espidf and arduino together in platform.ini.

https://github.com/platformio/platform-espressif32/tree/master/examples?utm_source=platformio.org&utm_medium=docs

Creation of CMakeLists.txt (root and src)

Create a CMakeLists.txt file in the root directory of the project. Describe the following contents.

cmake_minimum_required(VERSION 3.16.0) 
 include($ENV{IDF_PATH}/tools/cmake/project.cmake) 
 project(clocky_platformio) #project name

Create another CMakeLists.txt file in the src folder. Write the following contents.

file(GLOB_RECURSE app_sources *.cpp) 
 idf_component_register(SRCS ${app_sources} 
          INCLUDE_DIRS ". " 
                    INCLUDE_DIRS "json" 
                    )

First, specify all .cpp files in the src directory as build targets using wildcards.

• file: CMake’s built-in command to manipulate the file system.
• GLOB_RECURSE: Option to recursively search for file patterns. The directory and its subdirectories are searched recursively to find files matching the specified pattern.
• app_sources: Name of the variable that will store the search results. This variable contains the paths to the files found.

So, the following code will search for all .cpp files in the src directory and its subdirectories and store their paths in the app_sources variable.

file(GLOB_RECURSE app_sources *.cpp)

Also, INCLUDE_DIRS "." to include the header files in the src directory in the include path. Similarly, to include the json subdirectory in the build, add INCLUDE_DIRS “json”.

file(GLOB_RECURSE app_sources *.cpp) 
 idf_component_register(SRCS ${app_sources} 
 INCLUDE_DIRS ". " 
                    INCLUDE_DIRS "json" 
                    REQUIRES esp-idf)

Creating sdkconfig.defaults

ESP-IDF can be initially configured by describing the necessary settings in sdkconfig.defaults.

Create a new sdkconfig.defaults file in the project root directory and include the following

CONFIG_AUTOSTART_ARDUINO=y 
 CONFIG_FREERTOS_HZ=1000 
 CONFIG_MBEDTLS_PSK_MODES=y 
 CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y

CONFIG_AUTOSTART_ARDUINO=y

• Function: Set to automatically start Arduino libraries.
• Rationale: Enabling this setting will allow Arduino sketches (setup() and loop() functions) to run automatically in the ESP-IDF environment. In other words, you no longer need to use the default app_main() function in ESP-IDF, which is necessary to ensure that Arduino libraries and functions work properly when using ESP-IDF and the Arduino framework together.

CONFIG_FREERTOS_HZ=1000

• Function: Set FreeRTOS system tick rate (Hz) to 1000.
• Rationale: The default system tick rate is 100 Hz, but Arduino sketches and libraries generally work assuming a 1 ms precision (1000 Hz). Therefore, this setting ensures that the Arduino code works accurately.

CONFIG_MBEDTLS_PSK_MODES=y

• Function: enable PSK (Pre-Shared Key) mode in mbedTLS.
• Rationale: PSK mode enables encrypted communications using pre-shared keys. To enhance security features, encrypted communication using PSK can be implemented by enabling this setting.

CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y

• Function: enable key exchange by PSK in mbedTLS.
• Rationale: If PSK mode is enabled, key exchange with PSK must also be enabled. This allows secure key exchange using pre-shared keys to establish secure communication.

If TLS/SSL is used for MQTT communication and PSK is used, CONFIG_MBEDTLS_PSK_MODES andCONFIG_MBEDTLS_KEY_EXCHANGE_PSK must be set.

It is also possible to configure sdkconfig with menuconfig

Incidentally, sdkconfig settings can also be configured by opening ESP-IDF’s menuconfig in platformIO as follows

pio run  -t  menuconfig

The following screen will open, and check the setting items.

• CONFIG_AUTOSTART_ARDUINO: “Component config”->”Arduino Configuration”->”Autostart Arduino setup and loop on boot”
• CONFIG_FREERTOS_HZ: “Component config”->”FreeRTOS”->”Tick rate (Hz)

After checking the specified items, when you try to save the file, you can specify the destination file, and when you select it, the settings are reflected in the file.

Of course, if GUI configuration is troublesome, the sdkconfig file can be modified directly.

LittleFS linker error support

LittleFS linker error occurred when building with the above settings

Linking .pio/build/debug/firmware_debug.elf 
 /Users/ky/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/bin/.. /lib/gcc/xtensa-esp32-elf/8.4.0/... /... /... /.../xtensa-esp32-elf/bin/ld: .pio/build/debug/src/main.cpp.o:(.literal._Z11setupCommonv+0x24): undefined reference to  `fs ::LittleFSFS::begin(bool, char const*, unsigned char, char const*)'

If both ESP-IDF and Arduino frameworks are used, the LittleFS library in Arduino may not be linked correctly.

Found someone facing a similar problem.

https://github.com/platformio/platform-espressif32/issues/965

Add esp_littlefs library to Components folder -> Failed

Clone littlefs library

Clone the esp_littlefs library into the components folder of the project.

https://github.com/joltwallet/esp_littlefs.git

mkdir  -p  components 
 cd components 
 git clone  --branch   v1 .14.4 https://github.com/joltwallet/esp_littlefs.git

Correction of CMakeLists.txt

Modify the CMakeLists.txt file in the root of the project to esp_littlefs

Include libraries.

cmake_minimum_required(VERSION 3.16.0) 
 include($ENV{IDF_PATH}/tools/cmake/project.cmake) 
 
 list(APPEND EXTRA_COMPONENT_DIRS "${CMAKE_SOURCE_DIR}/components/esp_littlefs") 
 
 project(clocky_platformio)

When I built with the above, the link error was not resolved.

components/esp_littlefs/src/esp_littlefs.c:12:10: fatal error: littlefs/lfs.h: No such file or directory 
  #include "littlefs/lfs.h" 
           ^~~~~~~~~~~~~~~~ 
 compilation terminated.

Use idf_component.yml to manage esp_littlefs library -> success

In the thread of someone who encountered the same problem earlier, the following was mentioned, so I tried

Add managed components by addingidf_component .yml See as example https://github.com/Jason2866/Arduino_IDF_LittleFS/blob/main/src/idf_component.yml

Createidf_component.yml file:. Create idf_component.yml file in the src folder of the project and add the following contents

dependencies::  ">=4.4 
   idf: ">=4.4" 
   esp_littlefs::
     git: https://github.com/joltwallet/esp_littlefs.git

Modify CMakeLists.txt:. Modify the CMakeLists.txt file in the project root directory and add the managed_components folder to the build system.

cmake_minimum_required(VERSION 3.16.0) 
 include($ENV{IDF_PATH}/tools/cmake/project.cmake) 
 
 list(APPEND EXTRA_COMPONENT_DIRS managed_components) 
 
 project(clocky_platformio)

The generated managed_components folder is due to the specification in idf_component.yml. It is not the components folder that is generated. This is a folder for ESP-IDF’s component management system to automatically download and place dependencies.

By using idf_component.yml, did the ESP-IDF build system automatically manage dependencies and download and deploy the correct version of the required library? I’m not clear on the exact reason.

Prayerfully wait for compile to finish.

Finally, the Compile error is cleared!

Although the build is successfully completed,

I was able to build it successfully!

It must have taken more than 15 hours net to get to this point, right?

As the number of files became larger, it took longer to compile C++, and when the errors were resolved one by one, the compilation proceeded little by little, but if there were errors in later compilations, they would be displayed, so the waiting time was longer and I would repeatedly think, “Okay, this fix will get me next time! But when a compilation error occurred at the end of the compilation, I was disappointed.

But the following error occurred after MQTT communication after Wifi connection.

***ERROR*** A stack overflow in task loopTask has been detected.

One difficulty goes away, another comes back. Click here to read the article on resolving the above error.

【ESP32】ESP-IDFへの移行後に発生したMQTT通信後のスタックオーバーフローエラー対処法

スタックオーバーフローは、タスクが割り当てられたスタックメモリを使い果たしてしまう現象。MQTTメッセージの受信処理やパース処理が原因でスタックオーバーフローが発生することがある。sdkconfigファイルでスタックサイズの設定変更すること...

kazulog.fun