Introduction.
Developing “Mia,” a talking cat-shaped robot that speaks dialect.
Previously, in this article, I implemented moving AWS IoT-related configuration files from the LittleFS area to the NVS area.
However, as it is vulnerable in terms of security, we tried to implement NVS encryption this time and came across the following description.
https://github.com/espressif/arduino-esp32/issues/9233
Flash Encryption and Secure Boot are not provided by the Arduino IDE and will not be provided in the future orz
So, first, we need to change from the Arduino framework we are using now to the ESP-IDF framework.
Incidentally, the current settings in the platform.ini file are as follows.
; PlatformIO Project Configuration File
; https://docs.platformio.org/page/projectconf.html
[env].
platform = espressif32
framework = arduino
board = esp32devIn this issue, we describe how to make framework changes in PlatformIO.
See description on the official Espressif website below.
https://docs.platformio.org/en/latest/frameworks/espidf.html#configuration
Proceed in the following order
- Add espidf to framework in platform.ini
- Add CmakeLists.txt file to root and src directories
- Add sdkconfig.defaults file and add necessary settings so that arduino can also boot
- Add arduino libraries needed to start also esp-idf in arduino library
ESP-IDF Framework Overview
What is the CmakeLists.txt file?
The ESP-IDF framework uses a build system called CMake.
The CMakeLists.txt file instructs the CMake build system how to build the project, and is needed to manage the project configuration and build process when using ESP-IDF.
Projects that use ESP-IDF in PlatformIO typically have the following structure
project_dir/
├── include/
├── src/
│ ├── CMakeLists.txt
│ └── main.c
├── CMakeLists.txt
└── platformio.iniWe can see that there are two CmakeLists.txt files, one in the root directory and one in the src directory. We will look at the purpose of each and how they are described.
CMakeLists.txt in the root directory
Configure settings for the entire project. Typically, this file will contain the following
# The following lines of boilerplate have to be in your project's CMakeLists
# in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16.0)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(project-name)project-name is the name of the project to set within CMake, which can be any name. CMake will use this name to generate project-specific variables to manage the build process.
Since PlatformIO project names are used primarily within the PlatformIO environment and CMake project-name is used within the CMake and ESP-IDF build systems, it is not necessary to match PlatformIO project names, but matching them would readability is enhanced.
CMakeLists.txt in src directory
Control the build process for a specific component, in this case the main component of the application. In a minimal setup, the component is registered with the build system as follows
idf_component_register(SRCS "main.c")These CMakeLists.txt files are necessary for the project to be built correctly using the ESP-IDF build system. If additional components or special build settings are required, the corresponding settings should be added to these files.
sdkconfig.defaults
The platformio.ini file is the main configuration file for the PlatformIO project, defining the project build settings, boards to be used, frameworks, library dependencies, etc.
On the other hand, sdkconfig.defaults is an ESP-IDF-specific configuration file that allows you to predefine various settings for the ESP-IDF framework. This includes system behavior, enabling/disabling of features, performance tuning, etc. It should be placed in the root directory of the project.
Now that we have an overview of the ESP-IDF framework, we will actually replace the current setup.
Update platformio.ini: add espidf to framework
Change the framework part of the platformio.ini file from arduino to espidf.
At first, I thought I had to specify framework = espidf, and the official website also stated so, so I followed that procedure.
In that case, the following work will occur, and it will be very difficult (in fact, it was quite difficult to proceed to the middle of the process)
Arduino library management:.
- In order to use the Arduino libraries, it is necessary to
git cloneall Arduino libraries under thecomponentsfolder, which may result in several GB of project file space.
Code rewriting
- If you build with the espidf framework specified, you will get an error for everything arduino library related, so you need to rewrite all the parts where the arduino library is included in
arduino.h.
In file included from src/app_config .cpp:1:
src/app_config.h:1:10: fatal error: FS .h: No such file or directory
************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI > platformio lib search "header:FS.h"
* Web > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************
1 | #include <FS.h>
| ^~~~~ ~
compilation terminated.
Compiling .pio/build/release/src/donwload.cpp.o
In file included from src/button_manager .cpp:1:
src/button_manager.h:1:10: fatal error: Arduino.h: No such file or directory
*****************************************************************
* Looking for Arduino.h dependency? Check our library registry!
*
* CLI > platformio lib search "header:Arduino.h"
* Web > https://registry.platformio.org/search?q=header:Arduino.h
*
*****************************************************************
However, the following statement says that it is possible to use the Arduino and ESP-IDF frameworks together with PlatformIO. In my case, I noticed this in the middle of the process and ate up a lot of time.
[ env]
platform = espressif32 @ 6.7.0
framework = arduino, espidfBy specifying this, many existing Arduino libraries can be used directly, eliminating the need for porting and rewriting code.
So, set framework = arduino, espidf.
In fact, the espidf official documentation includes an example (blink, wifiscan) of using espidf and arduino together in platform.ini.
Creation of CMakeLists.txt (root and src)
Create a CMakeLists.txt file in the root directory of the project. Describe the following contents.
cmake_minimum_required(VERSION 3.16.0)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(clocky_platformio) #project nameCreate another CMakeLists.txt file in the src folder. Write the following contents.
file(GLOB_RECURSE app_sources *.cpp)
idf_component_register(SRCS ${app_sources}
INCLUDE_DIRS ". "
INCLUDE_DIRS "json"
)First, specify all .cpp files in the src directory as build targets using wildcards.
file: CMake’s built-in command to manipulate the file system.GLOB_RECURSE: Option to recursively search for file patterns. The directory and its subdirectories are searched recursively to find files matching the specified pattern.app_sources: Name of the variable that will store the search results. This variable contains the paths to the files found.
So, the following code will search for all .cpp files in the src directory and its subdirectories and store their paths in the app_sources variable.
file(GLOB_RECURSE app_sources *.cpp)Also, INCLUDE_DIRS "." to include the header files in the src directory in the include path. Similarly, to include the json subdirectory in the build, add INCLUDE_DIRS “json”.
file(GLOB_RECURSE app_sources *.cpp)
idf_component_register(SRCS ${app_sources}
INCLUDE_DIRS ". "
INCLUDE_DIRS "json"
REQUIRES esp-idf)
Creating sdkconfig.defaults
ESP-IDF can be initially configured by describing the necessary settings in sdkconfig.defaults.
Create a new sdkconfig.defaults file in the project root directory and include the following
CONFIG_AUTOSTART_ARDUINO=y
CONFIG_FREERTOS_HZ=1000
CONFIG_MBEDTLS_PSK_MODES=y
CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=yCONFIG_AUTOSTART_ARDUINO=y
- Function: Set to automatically start Arduino libraries.
- Rationale: Enabling this setting will allow Arduino sketches (setup() and loop() functions) to run automatically in the ESP-IDF environment. In other words, you no longer need to use the default app_main() function in ESP-IDF, which is necessary to ensure that Arduino libraries and functions work properly when using ESP-IDF and the Arduino framework together.
CONFIG_FREERTOS_HZ=1000
- Function: Set FreeRTOS system tick rate (Hz) to 1000.
- Rationale: The default system tick rate is 100 Hz, but Arduino sketches and libraries generally work assuming a 1 ms precision (1000 Hz). Therefore, this setting ensures that the Arduino code works accurately.
CONFIG_MBEDTLS_PSK_MODES=y
- Function: enable PSK (Pre-Shared Key) mode in mbedTLS.
- Rationale: PSK mode enables encrypted communications using pre-shared keys. To enhance security features, encrypted communication using PSK can be implemented by enabling this setting.
CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y
- Function: enable key exchange by PSK in mbedTLS.
- Rationale: If PSK mode is enabled, key exchange with PSK must also be enabled. This allows secure key exchange using pre-shared keys to establish secure communication.
If TLS/SSL is used for MQTT communication and PSK is used, CONFIG_MBEDTLS_PSK_MODES andCONFIG_MBEDTLS_KEY_EXCHANGE_PSK must be set.
It is also possible to configure sdkconfig with menuconfig
Incidentally, sdkconfig settings can also be configured by opening ESP-IDF’s menuconfig in platformIO as follows
pio run -t menuconfigThe following screen will open, and check the setting items.
CONFIG_AUTOSTART_ARDUINO: “Component config”->”Arduino Configuration”->”Autostart Arduino setup and loop on boot”CONFIG_FREERTOS_HZ: “Component config”->”FreeRTOS”->”Tick rate (Hz)
After checking the specified items, when you try to save the file, you can specify the destination file, and when you select it, the settings are reflected in the file.
Of course, if GUI configuration is troublesome, the sdkconfig file can be modified directly.
LittleFS linker error support
LittleFS linker error occurred when building with the above settings
Linking .pio/build/debug/firmware_debug.elf
/Users/ky/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/bin/.. /lib/gcc/xtensa-esp32-elf/8.4.0/... /... /... /.../xtensa-esp32-elf/bin/ld: .pio/build/debug/src/main.cpp.o:(.literal._Z11setupCommonv+0x24): undefined reference to `fs ::LittleFSFS::begin(bool, char const*, unsigned char, char const*)'If both ESP-IDF and Arduino frameworks are used, the LittleFS library in Arduino may not be linked correctly.
Found someone facing a similar problem.
https://github.com/platformio/platform-espressif32/issues/965
Add esp_littlefs library to Components folder -> Failed
Clone littlefs library
Clone the esp_littlefs library into the components folder of the project.
https://github.com/joltwallet/esp_littlefs.git
mkdir -p components
cd components
git clone --branch v1 .14.4 https://github.com/joltwallet/esp_littlefs.gitCorrection of CMakeLists.txt
Modify the CMakeLists.txt file in the root of the project to esp_littlefs
Include libraries.
cmake_minimum_required(VERSION 3.16.0)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
list(APPEND EXTRA_COMPONENT_DIRS "${CMAKE_SOURCE_DIR}/components/esp_littlefs")
project(clocky_platformio)When I built with the above, the link error was not resolved.
components/esp_littlefs/src/esp_littlefs.c:12:10: fatal error: littlefs/lfs.h: No such file or directory
#include "littlefs/lfs.h"
^~~~~~~~~~~~~~~~
compilation terminated.Use idf_component.yml to manage esp_littlefs library -> success
In the thread of someone who encountered the same problem earlier, the following was mentioned, so I tried
Add managed components by addingidf_component
.ymlSee as example https://github.com/Jason2866/Arduino_IDF_LittleFS/blob/main/src/idf_component.yml
Createidf_component.yml file:.
Create idf_component.yml file in the src folder of the project and add the following contents
dependencies:: ">=4.4
idf: ">=4.4"
esp_littlefs::
git: https://github.com/joltwallet/esp_littlefs.gitModify CMakeLists.txt:.
Modify the CMakeLists.txt file in the project root directory and add the managed_components folder to the build system.
cmake_minimum_required(VERSION 3.16.0)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
list(APPEND EXTRA_COMPONENT_DIRS managed_components)
project(clocky_platformio)The generated managed_components folder is due to the specification in idf_component.yml. It is not the components folder that is generated. This is a folder for ESP-IDF’s component management system to automatically download and place dependencies.
By using idf_component.yml, did the ESP-IDF build system automatically manage dependencies and download and deploy the correct version of the required library? I’m not clear on the exact reason.
Prayerfully wait for compile to finish.
Finally, the Compile error is cleared!
Although the build is successfully completed,
I was able to build it successfully!
It must have taken more than 15 hours net to get to this point, right?
As the number of files became larger, it took longer to compile C++, and when the errors were resolved one by one, the compilation proceeded little by little, but if there were errors in later compilations, they would be displayed, so the waiting time was longer and I would repeatedly think, “Okay, this fix will get me next time! But when a compilation error occurred at the end of the compilation, I was disappointed.
But the following error occurred after MQTT communication after Wifi connection.
***ERROR*** A stack overflow in task loopTask has been detected.One difficulty goes away, another comes back. Click here to read the article on resolving the above error.
