Introduction.
Currently, config.json, device.cert.pem, device.private.key, root.ca.pem, etc. are placed under data. (shadow.json is different because it performs writing)
Current problem: Communication impossible if LittleFS is broken
For example, if the LittleFS storage space is overflowed due to a problem in the download process, the reading of certificates may fail, and there is a possibility that it will become impossible to read the certificates. It is better to write certificates and server connection information in the NVS area and read them from there.
Refer to the following article and modify it to write certificates and configuration files in the NVS area. However, the reference article uses the ESP-IDF framework, but this time PlatformIO is used, so that part is different.
https://simplyexplained.com/blog/esp-idf-store-aws-iot-certificates-in-nvs-partition/
The reading of settings in the program should also be changed to reading nvs where it is currently reading from littlefs.
Overall Policy
- Move various configuration files from the data area (so that they are not included in the SPIFFS area at build time)
- Create nvs.csv file, generate certs.bin file containing configuration file information based on the csv file, and write data to the nvs area at build time
- Establish communication by reading certificate data from certs.bin instead of from LittleFS during MQTT communication and OTA updates.
Generate and write nvs.bin file
NVS Features and Namespaces
NVS (Non-Volatile Storage) stands for Non-Volatile Storage, a key value database stored in flash memory, used in ESP-IDF to store Wi-Fi authentication information and RF calibration data.
The default NVS partition can store 16 KB of data. This is enough to store certificates and private keys, so there is no need for a new custom partition map.
NVS also uses namespaces. A sort of “folder” within the NVS partition containing key/value items. This prevents conflicts between apps, third-party components, and ESP-IDFs.
Creating NVS CSV files
First, if the configuration files are left under the data directory, they will be included in the SPIFFS area at build time, so create a new directory (certificates) directly under the root directory and move the configuration files you want to include in NVS into it.
my_project/
├── certificates/
│ ├── config.json
│ ├── device.cert.pem
│ ├── device.private.key
│ ├── root.ca.pem
│ └── nvs.csv
├── include/
├── lib/
├── src/
│ └── main.c
├── platformio.ini
└── ...Create an NVS CSV file (nvs.csv) in the certificates directory that defines the configuration files and certificates that have been moved.
The first entry in the CSV file must always be a namespace entry, so the type of the first line of the CSV should be namespace. Also, the encoding and value of namespace should be left empty.
key,type,encoding,value
certs,namespace,, config,file,string,certificates/config.json
config,file,string,certificates/config.json
device_cert,file,string,certificates/device.cert.pem
device_key,file,string,certificates/device.private.key
root_ca,file,string,certificates/root.ca.pemCreating flash_nvs.py script
Create scripts to create and flush NVS partitions.
generate_nvs.py: generate_nvs_partition() function
- Run the
nvs_partition_gen.pyscript to generate the binary filecertificates/certs.binbased on the contents defined incertificates/nvs.csv. nvs_partition_gen.pyis a command line tool created by Espressif and included in ESP-IDF.
import os
import subprocess
Script import Import
Import("env")
def generate_nvs_partition(source, target, env)
print("Generating NVS partition... ")
command = [
"python",.
os.path.join(os.getenv('IDF_PATH'), 'components', 'nvs_flash', 'nvs_partition_generator', 'nvs_partition_gen.py'),.
"generate",.
"certificates/nvs.csv", " certificates/nvs.csv ".
"certificates/nvs.bin", "certificates/nvs. bin ".
"0x5000"
]
result = subprocess.run(command, check=True)
if result.returncode != 0 :.
print("Error: NVS partition generation failed.")
else: print("NVS partition generated.
print("NVS partition generated successfully.")
AddPreAction("upload", generate_nvs_partition)flash_nvs.py: flash_nvs_partition() function
- Flush the generated
certificates/certs.binto the device’s 0x9000 address (= write the data to flash memory).
import os
import subprocess
Script import Import
Import("env")
def flash_nvs_partition(source, target, env)
print("Flashing NVS partition... ")
bin_path = os.path.join(env['PROJECT_DIR'], 'certificates', 'certs.bin')
command = [
"python", "-m", "esptool",.
"--chip", "esp32",.
"--port", env['UPLOAD_PORT'],.
"--baud", str(env['UPLOAD_SPEED']),
"write_flash", "0x9000", bin_path
]
try:.
result = subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.)
print("NVS partition flashed successfully.")
except subprocess.CalledProcessError as e:.
print(f"Error: NVS partition flashing failed. Error message: {e.stderr.decode()}")
env.AddPostAction("upload", flash_nvs_partition)Incidentally, the current partition table has the following settings: nvs: An area from offset 0x09000 to 0x5000 bytes (20,480 bytes) is allocated, which is sufficient for storing normal configuration files and certificates.
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x09000, 0x5000,
otadata, data, ota, 0x0e000, 0x2000,
app0, app, ota_0, 0x010000,0x390000,
app1, app, ota_1, 0x400000,0x390000,
spiffs, data, spiffs, 0x790000,0x794000,Update platformio.ini
Update the platformio.ini file and add the python code generated earlier as extra_scripts: pre:generate_nvs.py, post:flash_nvs.py.
The generate_nvs_partition function should be run as a pre-script since the NVS partition binary file needs to be generated before the build process. This ensures that all necessary resources are in place during the build.
The flash_nvs_partition function is executed as a POST script.
After the upload is complete, flashing the binary file of the generated NVS partition to the device ensures that the application code and NVS data are present in flash memory.
[env]
platform = espressif32
framework = arduino
board = esp32dev
lib_deps =
earlephilhower/ESP8266Audio@^1.9.7
extra_scripts = pre:generate_nvs.py, post:flash_nvs.py
board_build.filesystem = littlefs
board_build.partitions = clocky.csv
board_upload.flash_size = 16MB
monitor_speed = 115200
upload_speed = 115200
build_unflags =
-std=gnu++11
build_flags =
-DPIOENV="${PIOENV}"Confirmation of certs.bin file generation operation
Below is a command to build in the debug environment, upload the firmware and NVS partition, and start the serial monitor.
platformio run -e debug --target upload --target monitorAfter a successful build, verify that the certs.bin file is generated in the certificates directory.
We successfully confirmed that certs.bin (20KB) was generated in the certificates directory before the build as shown below.
generate_nvs_partition(["upload"], [".pio/build/debug/firmware_debug.bin"])
Generating NVS partition...
Creating NVS binary with version: V2 - Multipage Blob Support Enabled
Created NVS binary: ===> /Users/ky/dev/clocky/clocky_platformio/certificates/certs. bin
NVS partition generated successfully.
Looking for upload port...
....
Writing at 0x002f1146 ... (97 %)
Writing at 0x002f6479 ... ( 98 %)
Writing at 0x002fbc01.. . ( 100 %)
Wrote 3074432 bytes (1236860 compressed) at 0x00010000 in 114.0 seconds (effective 215.7 kbit/s )...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...
flash_nvs_partition(["upload"], [".pio/build/debug/firmware_debug.bin"])
Flashing NVS partition...
Using command: python -m esptool --chip esp32 --port /dev/cu .wchusbserial1440 --baud 115200 write_flash 0x9000 /Users/ky/dev/c locky/clocky_ platformio/certificates/certs.bin
esptool.py v4 .7.0
Serial port /dev/cu .wchusbserial1440
Connecting............
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00009000 to 0x0000 dfff. ..
Compressed 20480 bytes to 3370...
Writing at 0x00009000 ... (100 %)
Wrote 20480 bytes (3370 compressed) at 0x00009000 in 0.5 seconds (effective 345 .5 kbit/s )...
Hash of data verified.Compressed 20480 bytes to 3370… shows the process by which esptool compresses the data to be sent when flushing the file and writes it to flash memory. 20480 bytes (20KB) of data is compressed to 3370 bytes (about 3KB). Although the compressed data will be used when writing to flash memory, 20480 bytes will actually be occupied on the flash memory. Since the size of the NVS partition is exactly 0x5000 bytes (20,480 bytes), this is exactly the size of the NVS partition.
However, this is a very marginal size, and even a small increase in data will quickly result in insufficient capacity, so consider changing the size of nvs to 0x6000 bytes (24,576 bytes) to give a little more leeway.
However, in that case, the ESP32 partition offset usually needs to be placed in multiples of 0x10000, so the app0 partition offset needs to be 0x20000instead of 0x11000, which leaves quite a gap (I want to put as much data as possible in the SPIFFS area). (We want to put as much data as possible in the SPIFFS area), so we will leave it as it is.
Reading configuration files from nvs.bin
Now that nvs.bin has been flashed to ESP32, the next step is to implement the part that reads data from nvs.bin.
Implement NVS initialization and read functions
Create header and source files that initialize and read NVS. Include the necessary ESP-IDF headers.
- #include <nvs_flash.h>
- #include <nvs.h>
// nvs_helper.h
#pragma once
#include <nvs_flash.h>
#include <nvs.h>
#include <Arduino.h>
#include <tl/expected.hpp>
#include <memory>
class NVSHelper {
public:.
static tl::expected<void, String> initNVS();
static tl: :expected<String, String> readNVS(const char* namespace_name, const char* key);
};- Initialize NV S: Initialize NVS by adding the
initNVSfunction. Erase flash memory if necessary. - Reading from NVS: Add
readNVSfunction to read the NVS value from the specified key; open the NVS handle, get the size of the value, and read the value.
#include "nvs_helper.h"
tl::expected<void, String> NVSHelper::initNVS() {
esp_err_t err = nvs_flash_init ();
if (err == ESP_ERR_NVS_NO_FREE_PAGES || err == ESP_ERR_NVS_NEW_VERSION_FOUND) {
ESP_ERROR_CHECK(nvs_flash_erase());
err = nvs_flash_init ();
}
if (err != ESP_OK) {
return tl::make_unexpected("Failed to initialize NVS"); }
}
return {} ; }
}
tl::expected<String, String> NVSHelper::readNVS(const char* namespace_name, const char* key) {
nvs_handle_t my_handle;
esp_err_t err = nvs_open (namespace_name, NVS_READONLY, &my_handle);
if (err != ESP_OK) {
return tl::make_unexpected("Failed to open NVS handle"); }
}
size_t required_size; }
err = nvs_get_str (my_handle, key, NULL , &required_size);
if (err != ESP_OK) {
nvs_close(my_handle);
return tl::make_unexpected("Failed to get size of NVS value");
}
std::unique_ptr<char[]> buf(new char[required_size]);
err = nvs_get_str (my_handle, key, buf.get(), &required_size);
if (err != ESP_OK) {
nvs_close(my_handle);
return tl::make_unexpected("Failed to get NVS value");
}
nvs_close(my_handle);
return String(buf.get()); }
}Fix the part that reads data from NVS
The part that used to call the configuration file from LittleFS will be modified to call it from NVS using the NVSHelper::readNVS function.
For example, the following appconfig.cpp used to be called from LittleFS as follows.
// src/app_config.cpp
#include "app_config.h"
tl::expected<void, String> AppConfig::load() {
File file = LittleFS.open("/config.json", "r");
if ( !file) {
LittleFS.end();
return tl::make_unexpected("Failed to open config file for reading");
}
const size_t size = file.size();
std::unique_ptr<char[]> buf(new char[size]);
file.readBytes(buf.get(), size);
const String jsonStr(buf.get(), size);
file.close();
auto result = deserializeAppConfig(jsonStr);
if (result) {
_config = result.value();
return {} ;
} else {
return tl::make_unexpected(result.error()); }
}
}Change as follows
// src/app_config.cpp
#include "app_config.h"
#include "nvs_helper.h"
tl::expected<void, String> AppConfig::load() {
auto init_result = NVSHelper::initNVS();
if ( !init_result) {
return tl::make_unexpected(init_result.error());
}
auto read_result = NVSHelper::readNVS("certs", "config");
if ( !read_result) {
return tl::make_unexpected(read_result.error()); }
}
const String jsonStr = read_result.value();
auto result = deserializeAppConfig(jsonStr);
if (result) {
_config = result.value();
return {} ;
} else {
return tl::make_unexpected(result.error()); }
}
}OTA update, the part that reads the rootCA is fixed as well (LittleFS to NVS).
// src/ota_update.cpp
#include "nvs_helper.h"
String readRootCA() {
auto init_result = NVSHelper::initNVS();
if ( !init_result) {
Serial.println(init_result.error());
return String();
}
auto read_result = NVSHelper::readNVS("certs", "root_ca");
if ( !read_result) {
Serial.println(read_result.error());
return String();
}
return read_result.value(); }
}Also modified the part of main.cpp that communicates MQTT after initial startup.
// src/main.cpp
void setup(){
// The following strings do not free memory because they are used when re-creating
auto rootCAResult = NVSHelper::readNVS("certs", "root_ca");
auto clientCertResult = NVSHelper: :readNVS("certs", "device_cert");
auto privateKeyResult = NVSHelper: :readNVS("certs", " device_key"); auto
auto mqttClient = std::make_unique<MqttClient>(mqttConfig.host,
mqttConfig.port,
const_cast<char*>(rootCAResult.value().c_str()),
const_cast<char*>(clientCertResult.value().c_str()),
const_cast<char*>(privateKeyResult.value(). c_str ( ) ),
deviceInfo.id); }
}operation check
Build ESP32 and see if it can read the certificate and other configuration files and keys from the NVS area and communicate with MQTT at startup.
16: 59:22.451 > [ 51900][V][ssl_client.cpp:68] start_ssl_client(): Starting socket
16: 59:22.478 > [ 51924][V][ssl_client.cpp:146] start_ssl_client(): Seeding the random number generator
16: 59:22.487 > [ 51933][V][ssl_client.cpp:155] start_ssl_client(): Setting up the SSL/TLS structure...
16:59: 22.497 > [ 51943][V][ssl_client.cpp:178] start_ssl_client(): Loading CA cert
16: 59:22.507 > [ 51953][V][ssl_client.cpp:234] start_ssl_client(): Loading CRT cert
16: 59:22.516 > [ 51963][V][ssl_client.cpp:243] start_ssl_client(): Loading private key
16: 59:22.528 > [ 51974][V][ssl_client.cpp:254] start_ssl_client(): Setting hostname for TLS session...
16: 59:22.536 > [ 51982][V][ssl_client.cpp:269] start_ssl_client(): Performing the SSL/TLS handshake...
16:59:23.733 > [ 53179][D][ssl_client.cpp:282] start_ssl_client(): Protocol is TLSv1 .2 Ciphersuite is TLS-ECDHE-RSA-WITH-AES-128 -GCM-SHA 256
16: 59:23.744 > [ 53191][D][ssl_client.cpp:284] start_ssl_client(): Record expansion is 29
16: 59:23.750 > [ 53197][V][ssl_client.cpp:290] start_ssl_client(): Verifying peer X. 509 certificate...
16:59: 23.759 > [ 53205][V][ssl_client.cpp:298] start_ssl_client(): Certificate verifying.
16:59:23.766 > [ 53212][V][ssl_client.cpp:313] start_ssl_client(): Free internal heap after TL S153048
16: 59:23.774 > [ 53220][V][ssl_client.cpp:369] send_ssl_data(): Writing HTTP request with 143 bytes...
16: 59:23.876 > connected to AWS IoT
16:59:23. 876 > MQTTPubSubClient ::subscribe: $aws/things/device_id/shadow/get/acceptedThe operation was confirmed safely.
However, as it is now, certificates and private keys can be read by users if they want to (dumpable by esptool.py, etc.), so it is better to keep them secret, and it is also necessary to encrypt and enable secure boot for device certificates in the NVS area, which will be addressed in the future.
