- Introduction.
- Need for NVS encryption
- Creation of NVS Key Partition
- Enable NVS encryption in sdkconfig
- NVS encryption key generation
- NVS partition creation and encryption
- Detected overlap at address: 0x8000 for file error response
- Update flash script for NVS partition
- Enable encryption during NVS initialization
- operation check
Introduction.
Developing “Mia,” a talking cat-shaped robot that speaks dialect.
Previously, in this article, I implemented moving AWS IoT-related configuration files from the LittleFS area to the NVS area.
However, as it is vulnerable in terms of security, we will attempt to implement NVS encryption in this case.
Also, as it stands, Flash Encryption and Secure Boot are not provided by the Arduino IDE and there are no plans to provide them in the future.
https://github.com/espressif/arduino-esp32/issues/9233
Therefore, as a preliminary step, the following implementation was made so that PlatformIO can handle not only Arduino but also ESP-IDF framework.
When the preliminaries are finally completed, we can begin NVS encryption.
Need for NVS encryption
Without NVS encryption, anyone with physical access to the flash chip could change, erase, or add key/value pairs.
Once NVS encryption is enabled, key/value pairs cannot be changed or added and recognized as valid pairs without knowing the corresponding NVS encryption key.
The details of NVS encryption for ESP32 are described below on the official EspressIf website.
Click here for an example
https://github.com/espressif/esp-idf/tree/c7bbfaee/examples/security/flash_encryption
However, since ESP32 is defined in the platform.ini file in the PlatformIO framework this time, we will proceed with the implementation in accordance with that file.
Creation of NVS Key Partition
First, a new NVS key partition is created as a dedicated partition for storing the encryption key required to use the NVS encryption function of ESP32.
By doing so, the nvs_flash_init API will automatically store the generated NVS encryption key in the NVS key partition. when NVS encryption is enabled, the nvs_flash_init API function will find the first NVS key partition, i.e., the data type and a partition of the nvs_keys subtype. The API function then automatically generates and stores the nvs key in that partition. A new key is generated and stored only if the corresponding key partition is empty.
So, it is necessary to specify data for Type and nvs_keys for Subtype as the NVS key partition.
Partition table (partition.csv)
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
nvs_keys, data, nvs_keys, 0xe000, 0x2000, # Generate nvs key partition
otadata, data, ota, 0x10000, 0x2000,
app0, app, ota_0, 0x20000, 0x390000,
app1, app, ota_1, 0x410000, 0x390000,
spiffs, data, spiffs, 0x800000, 0x800000Enable NVS encryption in sdkconfig
PlatformIO cannot use CONFIG_NVS_ENCRYPTION build flag
https://esp32.com/viewtopic.php?t=38039
Arduino IDE does not support flash encryption and secure boot
https://esp32.com/viewtopic.php?t=10029
The above is the reason why I made framework compatible with espidf in the following article.
To add settings directly to the sdkconfig file, check or add the following settings
# Enable Flash encryption
CONFIG_SECURE_FLASH_ENC_ENABLED=y
# Enable Flash encrypti on mode
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT=y # In development mode
# CONFIG_SECURE_FLASH_ENCRYPTION_MODE_RELEASE=y # For release mode
# Enable NVS encryption
CONFIG_NVS_ENCRYPTION=yNVS encryption key generation
Creating an Encrypted NVS Partition
The encrypt command in nvs_partition_gen.py can be used to generate NVS encryption keys and create encrypted NVS partitions.
Execute the following command
python nvs_partition_gen.py encrypt sample_singlepage_blob.csv sample_encr.bin 0x3000 --keygen --keyfilesample_keys.bin
This will generate an encrypted NVS partition ( sample_encr.bin ) from the specified CSV file and simultaneously generate the encryption key ( sample_keys.bin) under the keys/ directory.
# extra_script.py
import os
import subprocess
from SCons.Script import Import
Import("env")
# Generate NSS encryption key
def generate_nvs_key(source, target, env)
print("Generating NVS encryption key... ")
key_file = "nvs_keys.bin"
csv_file = "certificates/nvs.csv"
bin_file = "certificates/encrypted_nvs_partition.bin"
size = "0x5000"
command = [
"python",.
os.path.join(os.getenv('IDF_PATH'), 'components', 'nvs_flash', 'nvs_partition_generator', 'nvs_partition_gen.py'),.
"encrypt",.
csv_file,
bin_file,
size,.
"--keygen", key_file, size, csv_file, bin_file, size, csv_file
"--keyfile", key_file
]
result = subprocess.run(command, check=True)
if result.returncode != 0 :.
print("Error: NVS encryption key generation failed.")
else: print("NVS encryption key generation failed.
print("NVS encryption key generated successfully.")
# Generate NVS encryption key
env.AddPreAction("upload", generate_nvs_key)I ran this command and confirmed that nvs_keys.bin was generated under the keys/ directory.
NVS partition creation and encryption
Next, encrypt the NVS partition using the generated NVS encryption key, using the encrypt and –inputkey commands, passing the generated NVS encryption key as the –inputkey argument.
# generate and encrypt NVS partition
def generate_encrypted_nvs_partition(source, target, env)
print("Generating and encrypting NVS partition... ")
csv_file = "certificates/nvs.csv"
bin_file = "certificates/encrypted_nvs_partition.bin"
key_file = "keys/nvs_keys.bin"
size = "0x5000"
command = [
"python",.
os.path.join(os.getenv('IDF_PATH'), 'components', 'nvs_flash', 'nvs_partition_generator', 'nvs_partition_gen.py'),.
"encrypt",.
csv_file,
bin_file,
size,.
"--inputkey", key_file
]
result = subprocess.run(command, check=True)
if result.returncode != 0 :.
print("Error: NVS partition generation and encryption failed.")
else: print("NVS partition generated")
print("NVS partition generated and encrypted successfully.")
# Generate encrypted NVS partition using NVS encryption key
env.AddPreAction("upload", generate_encrypted_nvs_partition)Install cryptography package
Note that the nvs_partition_gen.py script depends on the cryptography package, so install it (if you get a warning). In the case of under virtual environment, see below.
python -m venv venv
source venv/bin/activate
pip install cryptographyAdd cryptography to requirements.txt.
Detected overlap at address: 0x8000 for file error response
When built, the following error appeared
esptool write_flash: error: argument <address> <filename>: Detected overlap at address: 0x8000 for file: /Users/ky/dev/clocky/ clocky_platformio/.pio/build/debug/partitions.binI googled and found that someone else had encountered the same error, even though the partition table should not have caused the overlap.
They have two solutions.
- Increase
CONFIG_PARTITION_TABLE_OFFSEToption from 0x8000 -> 0x10000 CONFIG_LOG_BOOTLOADER_LEVELVERBOSEto INFO
Once generated, try outputting the .bin file along with its size
~/dev/clocky/clocky_platformio/.pio/build/debug feat-nvs-encryption ● ? ⍟16 ls -la * .bin ✔ 3933 12:27:49
-rw-r--r-- 1 ky staff 36192 6 3 10:30 bootloader.bin
-rw-r--r-- 1 ky staff 2973488 6 3 10:31 firmware_debug.bin
-rw-r--r-- 1 ky staff 8192 6 3 10:30 ota_data_ini tial .bin
-rw-r--r-- 1 ky staff 3072 6 3 10:30 partitions.binThe size of the bootloader.bin built with verbose is 36192 bytes, larger than the usual 0x8000 - 0x1000 = 28672 bytes.
So, if the offset initions.bin starts at the default 0x8000, it overlaps with the end of the bootloader, causing an overlap error.
If CONFIG_PARTITION_TABLE_OFFSET is changed to 0x10000, the partition table will be placed at the address 0x10000. In this case, the space allocated to the boot loader is 0x10000 - 0x1000 = 60KB = 61,440 bytes > 36192 bytes, a sufficient amount.
Also, need to change the offset of the partition table
The partitions set in the partition table are followed by the following two.
- Boot loader: 0x1000 to 0x8000 (typically 32KB)
- Partition table: 0x8000 to 0x9000 (4KB)
Therefore, nvs is modified by offsetting the sum of bootloader.bin and partitions.bin. Since the initial offset is shifted, the subsequent partitions are also shifted by that amount.
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs , 0x11000 , 0x5 000
nvs_keys, data, nvs_keys , 0x16000 , 0x2000 , encrypted
otadata, data, ota , 0x18000 , 0x2000,
app0, app, ota_0 , 0x20000 , 0x390 000 ,
app1, app, ota_1 , 0x410000 , 0x390 000 ,
spiffs, data, spiffs, 0x7A0000 , 0x86 0000I changed to the above settings and rebuilt, and the build started successfully.
generate_nvs_partition(["upload"], [".pio/build/debug/firmware_debug.bin"])
Generating NVS partition with encryption...
Created encryption keys: ===> /Users/ky/dev/clocky/clocky_platformio/keys/nvs_keys .bin
Creating NVS binary with version: V2 - Multipage Blob Support Enabled
Created NVS binary: ===> /Users/ky/dev/clocky/clocky_platformio/certificates/certs. bin
NVS partition generated successfully.
Looking for upload port...
Auto-detected: /dev/cu .wchusbserial1420
Uploading .pio/build/debug/firmware_debug.bin
esptool.py v4 .5.1
Serial port /dev/cu .wchusbserial1420
Connecting....
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT , Dual Core, 240MHz, VRef calibration in efuse , CodingScheme None
Crystal is 40MHz
MAC: d8 :13:2a:25:3d:9c
Uploading stub...
Running stub...
Stub running...
Configuring flash size...Update flash script for NVS partition
Compared to flash_nvs_partition() created in the previous article here, the offset of the nvs area has changed from 0x09000 to 0x11000, so the start position of writing has been modified.
# extra_script.py
import os
import subprocess
from SCons.Script import Import
Import("env")
def flash_nvs_partition(source, target, env)
print("Flashing NVS partition with encryption... ")
bin_path = os.path.join(env['PROJECT_DIR'], 'certificates', 'certs.bin')
command = [
"python", "-m", "esptool",.
"--chip", "esp32",.
"--port", env['UPLOAD_PORT'],.
"--baud", str(env['UPLOAD_SPEED']),
"write_flash", "0x11000", bin_path,.
]
try:.
result = subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.)
print("NVS partition flashed successfully.")
except subprocess.CalledProcessError as e:.
print(f"Error: NVS partition flashing failed. Error message: {e.stderr.decode()}")
env.AddPostAction("upload", flash_nvs_partition)
Enable encryption during NVS initialization
The initNVS function for NVS initialization can be left as is.
This function initializes NVS and, if necessary, erases and reinitializes the NVS partition.
nvs_flash_init automatically handles encryption if NVS encryption is enabled. No additional processing is required since the necessary key generation and settings are done internally during initialization.
#include "nvs_helper.h"
tl::expected<void, String> NVSHelper::initNVS() {
esp_err_t err = nvs_flash_init ();
if (err == ESP_ERR_NVS_NO_FREE_PAGES || err == ESP_ERR_NVS_NEW_VERSION_FOUND) {
ESP_ERROR_CHECK(nvs_flash_erase());
err = nvs_flash_init ();
}
if (err != ESP_OK) {
return tl::make_unexpected("Failed to initialize NVS"); }
}
return {} ; }
}operation check
The following log was displayed when I built it
- NVS partition is encrypted: “I (1662) nvs: NVS partition “nvs” is encrypted.”
- NVS key generation: “I (1463) nvs: NVS key partition empty, generating keys”
- Flushed successfully: “NVS partition flashed successfully.”
looking (feeling) safe
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
20:35: 01.547 > I (1789) flash_encrypt: bootloader encrypted successfully
20:35: 01.547 > I (1838) flash_encrypt: partition table encrypted and loaded successful ly
20:35:01. 547 > I (1839) flash_encrypt: Encrypting partition 1 at offset 0x16000 (length 0x2000 )...
20: 35:01.547 > I (1936) flash_encrypt: Done encrypting
20:35:01. 547 > I (1936 ) flash_encrypt: Encrypting partition 2 at offset 0x18000 (length 0x2000 )...
20: 35:01.547 > I ( 2003) flash_encrypt: Done encrypting
20:35: 01.547 > I (2003 ) esp_image: segment 0: paddr=00020020 vaddr=3f400020 size=18518ch (1593740) map
20: 35:01.547 > I (2582) esp_image: segment 1: paddr=001a51b4 vaddr=3ffb0000 size=049ech ( 18924)20:35:01.547 > I (2589) esp_image: segment 2: paddr=001a9ba8 vaddr=40080000 size=06470h ( 25712)20:35:01.547 > I (2598) esp_image: segment 3: paddr=001b0020 vaddr=400d0020 size=136bach (1272748) map
20 :35:01.547 > I (3059) esp_image: segment 4: paddr=002e6bd4 vaddr=40086470 size=0f320h ( 622 40 )20:35:01.547 > I (3082) flash_encrypt: Encrypting partition 3 at offset 0x20000 (length 0x390000 )...
20: 35:34.033 > I (40553) flash_encrypt: Done encrypting
20:35:34. 036 > E (40553) esp_image: image at 0x410000 has invalid magic byte (nothing flashed here?)
20:35:34.044 > I (40555) efuse: BURN BLOCK0
20:35: 34.052 > I ( 40570) efuse: BURN BLOCK0 - OK (all write block bits are set)
20:35:34.058 > I (40570) flash_encrypt: flash encryption completed
20:35:34. 063 > I (40573) boot: Resetting with flash encryption enabled...
20:35:34. 072 > ets Jul 29 2019 12:21:46
20:35:34. 072 >20:35:34. 072 > rst :0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
20:35: 34.078 > configsip: 0 , SPIWP:0xee
20:35:34. 080 > clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
20: 35:34.086 > mode:DIO, clock div:2
20:35:34. 089 > load:0x3fff00b8,len:10988
20:35:34 .091 > ho 0 tail 12 room 4
20:35:34. 091 > load:0x40078000,len:21276
20:35:34. 094 > load:0x40080400,len:3836
20:35:34. 097 > entry 0x40080 69c
20:35:34. 167 > I (56) boot: ESP-IDF 4.4.7 2nd stage bootloader
20:35:34 .167 > I (56 ) boot: compile time 20:31:54
20:35:34 .167 > I (56) boot: Multicore bootloader
20:35: 34.167 > I ( 60) boot: chip revision: v3.1
20: 35:34.167 > I (64) boot.esp32: SPI Speed : 40MHz
20:35:34. 167 > I (69) boot.esp32: SPI Mode : DIO
20:35:34. 167 > I (73) boot.esp32: SPI Flash Size : 16MB
20:35:34. 167 > I (78) boot: Enabling RNG early entropy source...
20:35:34. 167 > I (83) boot: Partition Table
20:35:34. 167 > I ( 87) boot: ## Label Usage Type ST Offset Length
20:35:34. 167 > I ( 94) boot: 0 nvs WiFi data 01 02 00011000 00005000
20:35:34. 167 > I (102) boot: 1 nvs_keys NVS keys 01 04 00016000 00002000
20: 35:34.167 > I ( 109) boot: 2 otadata OTA data 01 00 00018000 00002000
20:35:34. 171 > I (117) boot: 3 app0 OTA app 00 10 00020000 00390000
20:35: 34.177 > I (124) boot: 4 app1 OTA app 00 11 00410000 00390000
20:35: 34.185 > I (132) boot: 5 spiffs Unknown data 01 82 007a0000 00860000
20:35: 34.194 > I (139) boot: End of partition table
20:35:34 .197 > I (144) esp_image: segment 0: paddr=00020020 vaddr=3f400020 size=18518ch (1593740) map
20:35:34.792 > I (748) esp_image: segment 1: paddr=001a51b4 vaddr=3ffb0000 size=049ech ( 18924) load
20:35:34.801 > I (756) esp_image: segment 2: paddr=001a9ba8 vaddr=40080000 size=06470h ( 25712) load
20:35:34.811 > I (767) esp_image: segment 3: paddr=001b0020 vaddr=400d0020 size=136bach (1272748) map
20:35:35.287 > I (1243) esp_image: segment 4: paddr=002e6bd4 vaddr=40086470 size=0f320h ( 62240 ) load
20:35:35.325 > I (1281) boot: Loaded app from partition at offset 0x20000
20:35: 35.330 > I (1281 ) boot: Checking flash encryption...
20:35: 35.333 > I (1281 ) flash_encrypt: flash encryption is enabled (3 plaintext flashes left)
20:35: 35.341 > I (1288) boot: Disabling RNG early entropy source...
20:35:35. 347 > I (1305) cpu_start: Multicore app
20: 35:35.353 > I (1305 ) cpu_start: Pro cpu up.
20:35: 35.355 > I (1306) cpu_start: Starting app cpu, entry point is 0x400820d8
20:35: 35.364 > I (0) cpu_start: App cpu up.
20:35: 35.370 > I (1324) cpu_start: Pro cpu start user code
20:35: 35.373 > I (1324 ) cpu_start: cpu freq: 160000000
20:35: 35.378 > I (1324 ) cpu_start: Application information
20:35: 35.384 > I (1329) cpu_start: Project name: clocky_platformio
20:35: 35.389 > I (1334) cpu_start: App version: 0. 2.0-64-g45f6856-dirty
20:35: 35.395 > I ( 1341) cpu_start: Compile time: Jun 3 2024 20:30:50
20:35:35.400 > I (1347) cpu_start: ELF file SHA256: 792351b48ecf68fc...
20:35:35.408 > I (1353) cpu_start: ESP-IDF: 4. 4.7
20:35:35. 411 > I (1358) cpu_start: Min chip rev: v0.0
20:35:35. 417 > I ( 1363) cpu_start: Max chip rev: v3.99
20:35:35. 422 > I (1368 ) cpu_start: Chip rev: v3.1
20:35:35. 425 > I (1373) heap_init: Initializing. RAM available for dynamic allocation
20:35:35. 433 > I (1380) heap_init: At 3FFAE6E0 len 00001920 (6 KiB ): DRAM
20: 35:35.439 > I (1386) heap_init: At 3FFBA370 len 00025C90 (151 KiB ): DRAM
20: 35:35.447 > I (1392) heap_init: At 3FFE0440 len 00003AE0 (14 KiB ): D/IRAM
20:35:35.453 > I (1399) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB ): D/IRAM
20: 35:35.459 > I (1405) heap_init: At 40095790 len 0000A870 (42 KiB ): IRAM
20: 35:35.467 > I (1413) spi_flash: detected chip: generic
20:35:35. 470 > I (1416) spi_flash: flash io: dio
20:35: 35.475 > W (1420) flash_encrypt: flash encryption mode is DEVELOPMENT (not secure)
20:35:35. 481 > I (1430) gpio: GPIO[27]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 0| Pulldo wn : 0|Intr:020:35:35.492 > I (1438) cpu_start: Starting scheduler on PRO CPU.
20:35:35.497 > I (0) cpu_start: Starting scheduler on APP CPU.
<spanstyle="background-color: rgb(251, 243, 219);">20:35:35.503 > I (1463) nvs: NVS key partition empty, generating keys</span>
20:35:35. 685 > I (1650) nvs: NVS partition I (1654) gpio: GPIO[27]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 0| Pulldo wn : 0|Intr:520 :35:35.697 > Starting
20:35: 35.697 > After Starting - Available heap size: 252572 bytes
<spanstyle="background-color: rgb(251, 243, 219);">20:35: 35.703 > I (1662) nvs: NVS partition "nvs" is encrypted.</span>But then, when initializing NVS and calling NVS area data (certificates, etc.), ESP_ERR_NVS_CORRUPT_KEY_PART error occurred.)
10:08:46.929 > E (1508) nvs: Failed to read NVS security cfg: [0x1117] (ESP_ERR_NVS_CORRUPT_KEY_PART)
10:08:46.938 > Failed to initialize NVS
10:08:46. 943 > E (1528) nvs: Failed to read NVS security cfg: [0x1117] (ESP_ERR_NVS_CORRUPT_KEY_PART)
10:08:46.951 > failed to load app config: Failed to initialize NVS
10:08:46. 954 > Guru Meditation Error: Core 1 panic' ed (LoadProhibited). Exception was unhandled.
Exception was unha ndled .
10:08:46.962 > Core 1 register dump:.
10:08:46.965 > PC : 0x400d7046 PS : 0x00060c30 A0 : 0x800e5ea6 A1 : 0x3ffc1d70
10:08:46.973 > A2 : 0x3ffb4a20 A3 : 0x00000001 A4 : 0x0000008b A5 : 0x00000000
10:08:46.979 > A6 : 0x00000001 A7 : 0x0001c200 A8 : 0x3ffb5774 A9 : 0x3ffc1d50
10:08:46.987 > A10 : 0x00000000 A11 : 0x400e44b0 A12 : 0x00000002 A13 : 0x00000000
10:08:46.996 > A14 : 0x3ffb7034 A15 : 0x3ffb368c SAR : 0x0000000a EXCCAUSE: 0x0000001c
10:08:47.004 > EXCVADDR: 0x00000000 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xffffffff Then, when it is started for the second or subsequent times, an INVALID HEADER ERROR appears next.
14: 31:46.606 > rst :0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
14:31:46.651 > invalid header: 0x26c1 fbcaIf ESP32 is set to encryption even though the text is not encrypted and is in plain text, an invalid header error will occur when trying to read the plain text in encrypted mode.
A separate article will be written on this response.
It’s not so simple.
