[Mia] ESP32 OTA update function implementation: Firmware update app notification

Introduction

Last time, in this article, we implemented OTA updates for ESP32 only on the device side. This time, we will implement a part that notifies users of the app when a developer uploads new firmware.

Firmware update app notification

overall flow

The developer specifies the version and uploads a new firmware binary to the AWS s3 firmware directory.

Compare each user’s Firmware version (=value of the firmware_version column in the User table of the database) with the latest Firmware version uploaded by the developer, and if they are different, the Flutter app will display “New Firmware (Firmware version) can be installed.” Notify.

Server (Go) side implementation

Manage user firmware versions (columns and structures)

Store the new Firmware version in the User table.

• Column name :firmware_version
• Type :VARCHAR
• Length : Depends on the structure of the version number, but generally a range VARCHAR(10)of VARCHAR(20)is often sufficient.
• Initial value: 1.0.0

$ cd scripts/migrations
$ migrate create -ext sql -format 2006010215 add_firmware_version_to_users

Up and down sql files will be created, so write them as below.

// 2024040407_add_firmware_version_to_users.up.sql
ALTER TABLE users 
  ADD COLUMN firmware_version VARCHAR(20) DEFAULT '1.0.0';

// 2024040407_add_firmware_version_to_users.down.sql
ALTER TABLE users
  DROP COLUMN firmware_version;

I applied the migration and added the fimware_version column to the User table.

UserAdd field to structFirmwareVersion

usersfirmware_versionFields that correspond to columns in the table . Allow column data to be read and written Userthrough the structure .firmware_version

// user_db.go
type User struct {
	ID                         int                  `db:"id" json:"id"`
	UID                        string               `db:"uid" json:"uid"`
	DeviceID                   types.NullString     `db:"device_id" json:"device_id"`
	// ... その他のフィールド ...
	SleepTransitionTime        types.NullInt64      `db:"sleep_transition_time" json:"sleep_transition_time"`
	HealthDataIntegrationStatus bool                `db:"healthdata_integration_status" json:"healthdata_integration_status"`
	FirmwareVersion            string      　　　　　 `db:"firmware_version" json:"firmware_version"` // ファームウェアバージョン
}

Add firmware update availability notification to notification.proto file

Define a notification message to notify the user that new firmware is available, and define an RPC method to use this new message type.

// notification.proto

syntax = "proto3";

package protos;
option go_package = "github.com/EarEEG-dev/clocky_be/pb";

import "shadow.proto";

// Existing definition...
message StreamRequest {
  int32 userId = 1; // user id
}

message FirmwareUpdateAvailableResponse {
  int32 userId = 1;
  string message = 2;
  string new_firmware_version = 3;　　// New firmware version available
}

service NotificationService {
  rpc Listen(StreamRequest) returns (stream StreamResponse);
  // New method for firmware update notification
  rpc ListenFirmwareUpdates(StreamRequest) returns (stream FirmwareUpdateAvailableResponse);
}

.protostreamServer streaming RPC can be implemented by using keywords on the server response side in the message definition of the file .

This allows the client to send a request once and then continuously send information by “pushing” the data to the client each time there is a change on the server side. We use server streaming RPC to push new firmware updates to the client as soon as they become available on the server side.

Click here for more information on Server Streaming RPC.

Once you have written the messages and rpc methods in the .proto file, protocuse the command to .protogenerate Go source code from the file.

Added Firmware Version comparison logic to gRPC Listen method

Implement the new Firmware version notification part using the ListenFirmwareUpdates function created by compiling the rpc method created in the .proto file.

Add logic to keep the new Firmware version fixed, and Listenin the method, compare the new Firmware version with the user’s current version and send a notification.

This time, I directly wrote the new firmware version in the function with const newFirmwareVersion = “v1.0.1”, but in the future I will prepare a table to store the new firmware information and call it from the database.

// grpc_server.go

type server struct {
	pb.UnimplementedHelloServiceServer
	pb.UnimplementedNotificationServiceServer
	grpcServer *GRPCServer
}

func (s *server) ListenFirmwareUpdates(req *pb.StreamRequest, stream pb.NotificationService_ListenFirmwareUpdatesServer) error {
	uid = req.UserId

	log.Printf("received listen firmware update request from user: %s", uid)

	// Retrieve user information from DB to get firmware version
	user, err := GetUser(s.grpcServer.db, uid)
	if err != nil {
		return err
	}
	currentVersion := user.FirmwareVersion 
	
	// TODO: write directly but make it readable from DB
    const newFirmwareVersion = "1.0.1"
    
    response := &pb.FirmwareUpdateAvailableResponse{
        UserId:  req.UserId,
        Message: "Your firmware is up to date.",
        NewFirmwareVersion: currentVersion,
    }

    if newFirmwareVersion != currentVersion {
        response.Message = "NEW_FIRMWARE_AVAILABLE"
        response.NewFirmwareVersion = newFirmwareVersion
    }

    if err := stream.Send(response); err != nil {
		return status.Errorf(codes.Internal, "failed to send firmware update response: %v", err)
	}	

    <-stream.Context().Done()
    return stream.Context().Err()
}

Add firmware update availability notification to notification.proto file

Next, let’s move on to the application side implementation.

On the Flutter app side, we notification.protoneed to generate Dart code based on the same file and create a stub to communicate with the server side. It is important to use the same protobuf definition on the server and client sides. This allows both sides to communicate using the same protocol and exchange information in the correct data format.

// proto/notification.proto
syntax = "proto3";

package protos;
option go_package = "github.com/EarEEG-dev/clocky_be/pb";

message StreamRequest {
  int32 userId = 1; // user id
}

message FirmwareUpdateAvailableResponse {
  int32 userId = 1;
  string message = 2;
  string new_firmware_version = 3;
}

service NotificationService {
  rpc ListenFirmwareUpdates(StreamRequest) returns (stream FirmwareUpdateAvailableResponse);
}

notification.protoSince the file protosis in a directory and I want to output the generated file to lib/grpc, use the command below to .protogenerate Dart code from the file.

protoc --dart_out=grpc:lib/grpc -Iprotos protos/notification.proto

protocWhen you generate Dart client code using a compiler, three types of files are mainly generated: *.pb.dart, *.pbjson.dartand .*.pbgrpc.dart

• .pb.dartFile: Contains the Dart class definition for protobuf messages. Provides serialization and deserialization of messages and access to data within messages.
• .pbjson.dartFile: Provides JSON encoding and decoding logic. Used to convert protobuf messages to JSON format, or from JSON to protobuf messages. It is mainly used when JSON format is required to interface with an API, but gRPC usually does not use it directly.
• .pbgrpc.dartFile: Contains the Dart class definition for the gRPC service client. .protoFor each service defined in the file, a corresponding client-side stub is generated. These stubs can be used to call RPC methods defined on the server. Contains the functions necessary for gRPC communication, such as calling methods, sending requests, receiving responses, and managing streaming communication.

This time we will use the method in the .pbgrpc.dartfile class . The method is used by the client to call the corresponding RPC on the server .NotificationServiceClientlistenFirmwareUpdateslistenFirmwareUpdatesListenFirmwareUpdates

// lib/grpc/notification.pbgrpc.dart
$grpc.ResponseStream<$0.FirmwareUpdateAvailableResponse> listenFirmwareUpdates($0.StreamRequest request, {$grpc.CallOptions? options}) {
  return $createStreamingCall(_$listenFirmwareUpdates, $async.Stream.fromIterable([request]), options: options);
}

Added method to listen for firmware updates to gRPC client

The current GrpcServiceclass contains methods, but the method for listening to firmware updates is undefined, so add it sayHello.listenNotifications

StreamRequestCreate an instance, userIdconfigure it and build the request. Then call to listen to the stream client.listenFirmwareUpdates(request)from the server . Use keywords to provide data from this stream as a stream to the caller of the method. This method allows clients to receive firmware update information related to a specified user ID in real time.FirmwareUpdateAvailableResponseyield*

// services/grpc_service.dart
class GrpcService {
  final ClientChannel channel;

  GrpcService({required String host, required int port})
      : channel = ClientChannel(
          host,
          port: port,
          options: const ChannelOptions(
          // Use secure connection in production environment
            credentials: ChannelCredentials.insecure(),
          ),
        );

  // Existing methods...

  // Added method for listening for firmware updates
  Stream<FirmwareUpdateAvailableResponse> listenFirmwareUpdates(
      int userId) async* {
    final headers = await headersProvider();
    final client = NotificationServiceClient(channel,
        options: CallOptions(
          metadata: headers,
        ));
    final request = StreamRequest()..userId = userId;
    yield* client.listenFirmwareUpdates(request);
  }
}

Define StreamProvider

Since the response from gRPC is a type, wrap it Streamin order to display the data reactively on the UI .StreamProvider

// services/grpc_service.dart
final firmwareUpdateProvider =
    StreamProvider<FirmwareUpdateAvailableResponse>((ref) {
  final grpcService = ref.watch(grpcServiceProvider);

  final user = ref.read(userProvider);
  if (user?.uid == null) {
    return const Stream.empty();
  }
  return grpcService.listenFirmwareUpdates(user!.id!);
});

Firmware update message displayed on home screen

firmwareUpdateProviderGet data:watch Update the Text widget based on data provided by this provider.

If there is a firmware update, a text message will be displayed with the firmware version listed, and if the user’s firmware is already up to date, the notification will not be displayed.

// lib/screens/home/home_tab.dart

import 'package:flutter/material.dart';
import 'package:flutter_riverpod/flutter_riverpod.dart';

class HomeTab extends ConsumerStatefulWidget {
  const HomeTab({super.key});

  @override
  _HomeTabState createState() => _HomeTabState();
}

class _HomeTabState extends ConsumerState<HomeTab> {
  
  @override
  Widget build(BuildContext context) {
    final firmwareUpdateAsyncValue = ref.watch(firmwareUpdateProvider);

    return DebugContainer(
      child: Center(
        child: firmwareUpdateAsyncValue.when(
          data: (response) {
            if (response.message == "NEW_FIRMWARE_AVAILABLE") {
              return Text("新しいファームウェア（${response.newFirmwareVersion}）がダウンロード可能です");
            } else {
              // Nothing is displayed if firmware is up-to-date
              return SizedBox.shrink();
            }
          },
          loading: () => CircularProgressIndicator(),
          error: (error, stack) => Text("エラーが発生しました: $error"),
        ),
      ),
    );
  }
}

Operation confirmation

In this way, I was able to successfully display a notification at the top of the home screen when a new firmware version was available.

The server-side logs also show that requests are being made from the app to see if there is a firmware update.

clocky_api_local  | 2024/04/10 10:57:37 received listen firmware update request from user: XXXXXXXXXXXXXXXXXXXXXX
clocky_api_local  | 2024/04/10 10:57:38 sent message to user: 1

Next, instead of just displaying the firmware update notification as text, we will make it a message bar that can be clicked, and when clicked, an alert dialog will be displayed asking whether to install the new firmware version.

Once the user clicks install, we update Deviceshadow and implement the part that actually downloads the new firmware to the ESP32 via MQTT communication.