方言を話すおしゃべり猫型ロボット『ミーア』をリリースしました（こちらをクリック）

[ESP32] OTA update: Firmware update using MQTT and AWS IoT device shadow

2024.04.17
This article can be read in about 43 minutes.

We are implementing an OTA update feature to allow developers to remotely update firmware when new features are released after the product is available to users.

In this article, we have implemented “firmware update function on a single device”.

In addition, in this article, we described a function that notifies users when a new firmware is uploaded by a developer.

[Mia] ESP32 OTA update function implementation: Firmware update app notification
A new Firmware binary is uploaded to the AWS s3 firmware directory with the version specified by the developer.The version of each user's firmware is compared to the latest version uploaded by the developer, and if they are different, the Flutter app is notified that the new firmware is available for installation.
kazulog.fun
2024.05.11

This time, as the final implementation of the OTA update function, we will describe the part that “Downloads the new Firmware when the user requests installation of the new Firmware.”

This will complete the implementation of the OTA update function.

Implementation steps

  1. Server-side (Go) steps :
    • Update the database user table to reflect the new Firmware version.
    • Update AWS IoT Device Shadow desired to the new version. The communication format used by AWS IoT Device Shadow is JSON.
  2. Device side (ESP32) operations :
    • Receive desired state updates from AWS IoT via MQTT.
    • Deserialize the received data (JSON format) and detect the difference between it and the current state (repoted).
    • If there is a difference (= firmware update is required), update via OTA.
    • Update the Device Shadow’s reported state to report successful updates to the cloud.

Server side (Go)

Update DB and Device Shadow Desired

Database update

  • UpdateUser(h.db, user)Call a function to update user information in the database. Specifically, access the Database and update the User table with the UPDATE statement.

Go structures → Protocol Buffers

  • ToDeviceConfigA function is a method defined in the Go language that converts data Userfrom a structure instance to a structure. The structure is a data structure based on the Protocol Buffers definition, and is used for the desired settings of AWS IoT device shadows.pb.DeviceConfigpb.DeviceConfig
  • UserThe fields of the structure are read by mapping the column values ​​in the FirmwareVersiondatabase table.userfirmware_version
Go
// user.go
type User struct {
	// 他のフィールド...
	FirmwareVersion   string	  `db:"firmware_version" json:"firmware_version"`
}

func (u *User) ToDeviceConfig() pb.DeviceConfig {
	return pb.DeviceConfig{
		FirmwareVersion:         &u.FirmwareVersion,
	}
}

Device shadow updates :

  • The communication format used by AWS IoT Device Shadow is JSON, so h.sm.UpdateShadow()the function converts the Protocol Buffers format data to JSON and updates the desired state of AWS IoT Device Shadow.
Go
// user_handler.go
type UserHandler struct {
	db *sqlx.DB
	sm ShadowManager
}

func (h *UserHandler) HandleUpdateUser(c echo.Context) error {
	uid := c.Get("uid").(string)
	user := &User{}
	if err := c.Bind(user); err != nil {
		return err
	}
	user.UID = uid

	// DB更新
	updatedUser, err := UpdateUser(h.db, user)
	if err != nil {
		c.Logger().Errorf("failed to update user: %v", err)
		return echo.NewHTTPError(http.StatusInternalServerError, "failed to update user")
	}

	// デバイスIDが存在するとき
	if updatedUser.DeviceID.Valid {
		dc := updatedUser.ToDeviceConfig()
		desired := &pb.PublishShadowDesired{
			State: &pb.ShadowDesiredState{
				Desired: &pb.ShadowDesired{
					Config: &dc,
				},
			},
		}
		// DeviceShadowを反映
		if err = h.sm.UpdateShadow(updatedUser.DeviceID.String, desired); err != nil {
			c.Logger().Errorf("failed to update device shadow: %v", err)
			return echo.NewHTTPError(http.StatusInternalServerError, "failed to update device shadow")
		}
	}
	return c.JSON(http.StatusOK, updatedUser)
}

Device side (ESP32)

overall flow

First, let’s look at the overall flow. Personally, I was confused about the desired state of Device Shadow and the desired settings inside the device, so I’ve organized it and described it.

AWS IoT Device Shadow desiredStates

The AWS IoT Device Shadow desiredstate indicates the desired state that the device should achieve. This is configured on the server side (cloud) and sent to the device. This state is managed in the cloud even when the device is offline and synced when the device comes back online.

Settings inside the devicedesired

Settings inside a device desiredare the internal parameters and settings that the device uses to actually perform operations. This is a setting that is updated based on the state of his Device Shadow received desired, but directly affects the behavior of the device. The device desiredactually performs operations based on these internal settings and reportedreflects the results in its state.

Synchronization and update process

Sync :

  • desiredThe device receives the Device Shadow state from the cloud .
  • Update the desiredinternal settings of the device based on the received state.desired

execution :

  • desiredThe device adjusts its behavior based on the updated internal settings.

report :

  • reportedReport the adjusted actual state to the cloud as the device state.

Update desired state of device shadow via MQTT communication

  • The MQTT client receives a new message and the configured onMessagecallback function is called.
  • desiredDevices subscribe to state updates from AWS IoT using the MQTT protocol . New desiredstate data $aws/things/THING_NAME/shadow/update/deltais sent through the topic (example)
C++
// src/mqtt_client.cpp
void MqttClient::onMessage(const std::function<void(const String &topic, const String &payload)> &callback) {
  client->onMessage([callback](const String &topic, const String &payload) {
    Serial.println("MQTTPubSubClient::onMessage: " + topic + " " + payload);
    callback(topic, payload);
  });
}

Parsing MQTT messages (topic and payload)

  • MQTT messages deltaare sent from a device shadow topic and contain a state reporteddifferent from the current state of the device.desired
  • The payload is typically JSON-formatted data that contains information about the state of the device and represents desiredthe state or states of a particular device shadow .reported

topic :$aws/things/XXXXXXXXXXXXXXXXXXXXXXXX/shadow/update/delta

  • This is a delta topic for the Device Shadow of a specific device (Thing). where is XXXXXXXXXXXXXXXXXXXXXXXXthe device Thing Name (or Thing ID)

Payload

  • version: Device shadow version number. Increases with each shadow update.
  • timestamp: UNIX timestamp when the device shadow was last updated.
  • state: desiredIndicates the new state of the device. In this example, configinside the object phrase_typethere is a key called , and its value is "hakata". This indicates a parameter related to the device’s configuration or operation, and the device must update its operation according to this new value.
  • metadata: desiredContains additional information about each state’s fields. This phrase_typeincludes the last updated timestamp.
JSON
// MQTTメッセージのペイロード例
{
  "version": 1859,
  "timestamp": 1713327942,
  "state": {
    "config": {
      "phrase_type": "hakata"
    }
  },
  "metadata": {
    "config": {
      "phrase_type": {
        "timestamp": 1713250306
      }
    }
  }
}

Deserialization of JSON format data (payload)

  • onMessageIn the callback, a corresponding handler function ( handleShadowGetAcceptedor handleShadowUpdateDelta) is selected and executed based on the received topic. In this case, it’s an update, so handleShadowUpdateDelta it’s a function.
C++
// src/sync_shadow.cpp

tl::expected<void, String> SyncShadow::start() {
  // MQTTクライアント初期化
  client->init();

  // TopicごとにHandlerを呼び出す
  this->client->onMessage([this](const String &topic, const String &payload) {
    Logger &logger = Logger::getInstance();

    if (topic == topicGetAccepted) {
      if (auto result = handleShadowGetAccepted(payload); !result) {
        logger.error(ErrorCode::SHADOW_FAILED, result.error());
      }
    } else if (topic == topicUpdateDelta) {
      if (auto result = handleShadowUpdateDelta(payload); !result) {
        logger.error(ErrorCode::SHADOW_FAILED, result.error());
      }
    }
  });
}

// デバイスシャドウの更新があった場合に呼ばれる
tl::expected<void, String> SyncShadow::handleShadowUpdateDelta(const String &payload) {
  auto resultDelta = deserializeSubscribeShadowUpdateDelta(payload);
  if (!resultDelta) {
    return tl::make_unexpected("can not deserialize delta shadow: " + resultDelta.error());
  }
  // 現在の設定
  const auto currentConfig = this->reported.config;
  // 更新された設定
  const auto deltaConfig = resultDelta->state->config;

  // configに変更がない場合は終了
  if (!isDeviceConfigChanged(deltaConfig, currentConfig)) {
    Serial.println("no change: skip");
    return {};
  }
  // 更新された設定を反映
  updateDeviceConfig(*this->desired.config, *resultDelta->state->config);
  return {};
}

The received data (JSON format) is deserialized on the device side. This deserializeDeviceConfiguses functions to DeviceConfigcreate objects from JSON data. In the case of the payload of the MQTT message mentioned earlier, resultDelta is SubscribeShadowUpdateDeltaa structure and the following result is obtained.

ShellScript
Phrase Type: hakata

updateDeviceConfig(*this->desired.config, *resultDelta->state->config): This function updates the current device desiredsettings ( this->desired.config) based on the newly received delta ( ).resultDelta->state->config

At this point, the internal settings of the device have been updated desiredbased on the received AWS IoT Device Shadow state.desired

Next, change the current settings to match the desired settings inside the device.

Difference detection between device desired settings and current settings

getDeviceConfigDiffUse a function to detect if there is a difference between the device’s current configuration ( currentConfig) and the desired configuration ( ) and return an object representing that difference.desiredConfigDeviceConfig

C++
// src/json/device_config.cpp
// 現在の設定とdesiredの設定の差分を取得する
std::optional<DeviceConfig> getDeviceConfigDiff(const std::optional<DeviceConfig> &desiredConfig, const std::optional<DeviceConfig> ¤tConfig) {
  DeviceConfig diff;
  bool hasDiff = false;

  // desiredの値がcurrentと異なる場合にdiffに設定する
  auto checkAndAssign = [&hasDiff](const auto &desired, const auto ¤t, auto &diffField) {
    if (desired.has_value()) {
      if (!current.has_value() || *desired != *current) {
        diffField = desired;
        hasDiff = true;
      }
    }
  };
  
  checkAndAssign(desiredConfig->firmware_version, currentConfig->firmware_version, diff.firmware_version);

  if (hasDiff) {
    return diff;
  } else {
    return std::nullopt;
  }
}

bool isDeviceConfigChanged(const std::optional<DeviceConfig> &desiredConfig, const std::optional<DeviceConfig> ¤tConfig) {
  if (desiredConfig->firmware_version.has_value() && desiredConfig->firmware_version != currentConfig->firmware_version) {
    return true;
  }
  // その他のフィールド
  // ...
  // すべてのフィールドが同じ場合
  return false;
}

Firmware update determination and OTA update execution

Wi-Fi connection settings

  • When the system boots, a Wi-Fi connection is first established.

Process inside SyncShadow’s start method

  • SyncShadow::start()The callback configured onMessagein , allows the device to automatically receive state updates from AWS IoT desiredand update the device’s internal desired configuration accordingly.

Role of the applyAndReportConfigUpdates function

  • applyAndReportConfigUpdates()The function is called periodically ( within the function) to detect configuration and configuration differences loop()within the device , update the device configuration if necessary, and send the results to the state of the AWS IoT Device Shadow. reflect ondesiredreportedreported
  • This function operates under the assumption that the SyncShadowinternal settings managed by the class desiredhave already been updated, and adjusts the actual device behavior based on those settings.
C++
// src/main.cpp

void setup() {
	// Wi-Fi接続の設定
	setupWiFi(ssid, password);
	
	// システムの起動ログ
	Serial.println("System initialization...");
	
	// SyncShadowインスタンスの取得
	SyncShadow &syncShadow = SyncShadow::getInstance();
	
	// SyncShadowの開始
	if (auto result = syncShadow.start(); !result) {
	    Serial.println("Failed to start SyncShadow: " + result.error());
	} else {
	    Serial.println("SyncShadow started successfully.");
	}
		
	// デバイス設定の変更を適用し、デバイスシャドウに通知する
	void applyAndReportConfigUpdates() {
	  Logger &logger = Logger::getInstance();
	  SyncShadow &syncShadow = SyncShadow::getInstance();
	
	  auto desiredConfig = syncShadow.getDesiredConfig();
	  auto reportedConfig = syncShadow.getReportedConfig();
	
	  auto optConfigDiff = getDeviceConfigDiff(desiredConfig, reportedConfig);
	  // 変更がない場合はスキップ
	  if (!optConfigDiff) {
	    return;
	  }
	  logger.debug("device config changed");
	
	  // 変更のあった設定
	  auto configDiff = optConfigDiff.value();
	  logger.debug("config diff: " + serializeDeviceConfig(configDiff));
	
	  // ファームウェアバージョンの変更があった場合
	  if (configDiff.firmware_version.has_value()) {
	    logger.debug("Firmware version changed: " + desiredConfig->firmware_version.value());
	    // ファームウェアアップグレード開始
	    if (auto result = upgradeFirmware(desiredConfig->firmware_version.value()); !result) {
	      logger.error(ErrorCode::OTA_UPDATE_FAILED, "failed to upgrade firmware: " + result.error());
	    return;
	    } else {
	      logger.info("Firmware upgrade successful");
	    }
	  }
  }
}

void loop() {
	// デバイスシャドウが更新されていた場合の処理
  applyAndReportConfigUpdates();
}

If the previous getDeviceConfigDifffunction returns a difference list and the firmware versions are different, call the do_firmware_upgrade function to start installing the desired firmware version.

The upgradeFirmware function takes the new firmware version (the firmware version listed in device shadow’s desired field) as an argument, generates the complete firmware download URL, and executes the download. A detailed explanation regarding this is below.

Change the function from void type to type and return an tl::expected<void, String>empty object on success . tl::expectedThis allows subsequent methods to inherit the success value, allowing them to report changes to the device’s shadow state.

C++
// src/ota_update.cpp
tl::expected<void, String> upgradeFirmware(const String &firmwareVersion) {
    Serial.println("OTA firmware upgrade start");
    if (WiFi.status() != WL_CONNECTED) {
        Serial.println("Wi-Fi not connected, attempting to connect...");
        connectWiFi();
        if (WiFi.status() != WL_CONNECTED) {
            return tl::make_unexpected("Failed to connect to Wi-Fi");
        }
    }

    // ルートCA証明書を読み込む
    String cert = readRootCA();
    if (cert.length() == 0) {
        return tl::make_unexpected("Failed to load root CA certificate");
    }

    esp_err_t err = nvs_flash_init();
    if (err == ESP_ERR_NVS_NO_FREE_PAGES || err == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        esp_err_t erase_err = nvs_flash_erase();
        if (erase_err != ESP_OK) {
            return tl::make_unexpected("Failed to erase NVS partition");
        }
        err = nvs_flash_init();
    }
    if (err != ESP_OK) {
        return tl::make_unexpected("Failed to initialize NVS");
    }

    // Firmware Update URLを組み立てる
    String firmwareUpdateUrl = String(FIRMWARE_UPGRADE_URL) + firmwareVersion + ".bin";

    Serial.println("Starting OTA task");
    esp_http_client_config_t config = {
        .url = firmwareUpdateUrl.c_str(),
        .cert_pem = cert.c_str(),
        .event_handler = _http_event_handler,
        .keep_alive_enable = true,
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);
    // 念の為に通信タイムアウトを10分に設定。
    esp_err_t ret = esp_http_client_set_timeout_ms(client, 600000);
    // otaアップデート実施
    esp_err_t result = esp_https_ota(&config);

    if (result == ESP_OK) {
        Serial.println("Firmware upgrade succeeded");
        esp_restart();
        return {};
    } else {
        String error_message = "Firmware upgrade failed: " + String(result);
        return tl::make_unexpected(error_message);
    }
}

Reporting successful update

  1. Update settings : updateDeviceConfig(*this->reported.config, config)Use functions to update internally held reported.config(reported settings) with new settings. Overwrite the device’s current settings with the latest changes.
  2. Update device shadow : this->report()Calls the method to reflect updated settings in the device shadow. This method internally communicates with the AWS IoT Device Shadow service using MQTT and reportedsynchronizes the state to the cloud shadow.
  3. Retrieve and save reported configuration : syncShadow.getReported()Retrieves the current reported configuration with and PersistentShadow::getInstance().save(reported)saves this configuration to local persistent storage. This ensures that the latest settings are retained even after the device is restarted or powered off.
C++
// src/main.cpp
// デバイス設定の変更を適用し、デバイスシャドウに通知する
void setup() {
	// 既存のコード
	// ...
	
	// デバイス設定をデバイスシャドウに通知
  if (auto result = syncShadow.reportConfig(configDiff); !result) {
    logger.error(ErrorCode::CONFIG_REPORT_FAILED, "failed to report config: " + result.error());
    return;
  }
  const auto reported = syncShadow.getReported();
  if (auto result = PersistentShadow::getInstance().save(reported); !result) {
    logger.error(ErrorCode::IO_FAILED, "failed to save app state: " + result.error());
    return;
  }
}
C++
// src/sync_shadow.cpp
// デバイスシャドウの状態を報告
tl::expected<void, String> SyncShadow::report() const {
  if (!isWiFiConnected()) {
    return {};
  }
  const auto publishShadowUpdate = PublishShadowUpdate{.state = reported};
  const String payload = serializePublishShadowUpdate(publishShadowUpdate);
  if (client->publish(topicUpdate, payload)) {
    return {};
  } else {
    return tl::make_unexpected("Failed to publish shadow update");
  }
}

tl::expected<void, String> SyncShadow::reportConfig(const DeviceConfig &config) {
  updateDeviceConfig(*this->reported.config, config);
  if (auto result = this->report(); !result) {
    return tl::make_unexpected("failed to report config: " + result.error());
  }
  return {};
}

Operation confirmation

Confirmed operation when updating firmware version from 1.0.4 to 1.0.5.

First, if a firmware update is available, a message will appear in the app, and users can choose to accept the update by clicking on the message.

When you click install, the app first sends an API request to the server for a firmware update.

ShellScript
flutter: Request updateUser: {"firmware_version":"1.0.5"}

Device Shadow Update : Set new firmware version as desired state in AWS IoT device shadow.

desired firmware_version has been changed to 1.0.5.

ShellScript
clocky_api_local  | 2024/04/17 11:50:00 received listen request from user: [USER_ID]
clocky_api_local  | 2024/04/17 11:50:00 received listen firmware update request from user: [USER_ID]
clocky_api_local  | 2024/04/17 11:50:00 received listen request from deviceId: [DEVICE_ID]
clocky_api_local  | 2024/04/17 11:50:00 sent message to user: 1
clocky_api_local  | subscribing to topic: $aws/things/[DEVICE_ID]/shadow/update/accepted
clocky_api_local  | {"time":"2024-04-17T11:50:00.754083024Z","id":"","remote_ip":"[IP_ADDRESS]","host":"[HOST_IP]:8080","method":"GET","uri":"/app/weather","user_agent":"Dart/3.3 (dart:io)","status":200,"error":"","latency":166399359,"latency_human":"166.399359ms","bytes_in":0,"bytes_out":233}
JSON
// device shadow
{
  "state": {
    "desired": {
      "config": {
        "firmware_version": "1.0.5"
      }
    },
    "reported": {
      "config": {
        "firmware_version": "1.0.4"
      }
    },
  }
}

MQTT communication : ESP32 detects updates to the device shadow’s desired state and receives changes from the MQTT topic.

ShellScript
MQTTPubSubClient::onMessage: $aws/things/[DEVICE_ID]/shadow/update/delta {"version":1242,"timestamp":[TIMESTAMP],"state":{"config":{"firmware_version":"1.0.5"}},"metadata":{"config":{"firmware_version":{"timestamp":[TIMESTAMP]}}}}
{"timestamp":[TIMESTAMP],"message":"device config changed","level":"DEBUG","thingName":"[DEVICE_ID]","freeHeap":[FREE_HEAP],"usedBytes":[USED_BYTES]}
MQTTPubSubClient::publish: mia/things/[DEVICE_ID]/logs {"timestamp":[TIMESTAMP],"message":"device config changed","level":"DEBUG","thingName":"[DEVICE_ID]","freeHeap":[FREE_HEAP],"usedBytes":[USED_BYTES]}
{"timestamp":[TIMESTAMP],"message":"config diff: {"config":{"firmware_version":"1.0.5"}}","level":"DEBUG","thingName":"[DEVICE_ID]","freeHeap":[FREE_HEAP],"usedBytes":[USED_BYTES]}
MQTTPubSubClient::publish: mia/things/[DEVICE_ID]/logs {"timestamp":[TIMESTAMP],"message":"config diff: {"config":{"firmware_version":"1.0.5"}}","level":"DEBUG","thingName":"[DEVICE_ID]","freeHeap":[FREE_HEAP],"usedBytes":[USED_BYTES]}
{"timestamp":[TIMESTAMP],"message":"Firmware version changed: 1.0.5","level":"DEBUG","thingName":"[DEVICE_ID]","freeHeap":[FREE_HEAP],"usedBytes":[USED_BYTES]}
MQTTPubSubClient::publish: mia/things/[DEVICE_ID]/logs {"timestamp":[TIMESTAMP],"message":"Firmware version changed: 1.0.5","level":"DEBUG","thingName":"[DEVICE_ID]","freeHeap":[FREE_HEAP],"usedBytes":[USED_BYTES]}

Firmware download and installation : Download and install new firmware

ShellScript
OTA firmware upgrade start
Starting OTA task
HTTP_EVENT_ON_CONNECTED
HTTP_EVENT_HEADER_SENT
HTTP_EVENT_ON_HEADER, key=x-amz-id-2, value=[AMAZON_ID]
HTTP_EVENT_ON_HEADER, key=x-amz-request-id, value=[REQUEST_ID]
HTTP_EVENT_ON_HEADER, key=Date, value=[DATE]
HTTP_EVENT_ON_HEADER, key=Last-Modified, value=[MODIFIED_DATE]
HTTP_EVENT_ON_HEADER, key=ETag, value="[ETAG_VALUE]"
HTTP_EVENT_ON_HEADER, key=x-amz-server-side-encryption, value=AES256
HTTP_EVENT_ON_HEADER, key=Accept-Ranges, value=bytes
HTTP_EVENT_ON_HEADER, key=Content-Type, value=application/octet-stream
HTTP_EVENT_ON_HEADER, key=Server, value=AmazonS3
HTTP_EVENT_ON_HEADER, key=Content-Length, value=[CONTENT_LENGTH]
HTTP_EVENT_ON_DATA, len=[DATA_LENGTH]
HTTP_EVENT_ON_DATA, len=[DATA_LENGTH]
HTTP_EVENT_ON_DATA, len=[DATA_LENGTH]
....
HTTP_EVENT_ON_DATA, len=[DATA_LENGTH]
HTTP_EVENT_DISCONNECTED
Total data received before disconnect: [TOTAL_BYTES] bytes
HTTP_EVENT_DISCONNECTED
Total data received before disconnect: [TOTAL_BYTES] bytes
Firmware upgrade succeeded

Reboot device : The device will automatically reboot after the firmware installation is complete.

Success Notification : Notify apps and servers of successful updates.

Logs sent to AWS CloudWatch

ShellScript
{
    "timestamp": [TIMESTAMP],
    "message": "Firmware version changed: 1.0.5",
    "level": "DEBUG",
    "thingName": "[DEVICE_ID]",
    "freeHeap": [FREE_HEAP],
    "usedBytes": [USED_BYTES]
}

Confirmed that Device Shadow reported was also successfully changed to 1.0.5.

It was completed successfully.

in conclusion

When the user accepts the update, the app sends an API request to the server, and the server reflects the changes in the AWS IoT device shadow. The device detects this change via MQTT and includes a section to download and upgrade the firmware.

After that, it will take a few minutes to download the firmware, so I would like to display the progress in the app while it is downloading. Described in a separate article.

スポンサーリンク

コメント

Copied title and URL