[ESP32 x Flutter] How to send Wi-Fi scan results in chunks via BLE

Introduction.
Confirmation of current code
Shorten JSON key names to reduce JSON size
1. operation check
Transmission in multiple messages by data division
operation check

Introduction.

In a previous development article on Mia, a cat-shaped talking robot, we described a function that implements a Wi-Fi network search function in ESP32 and sends it to the Flutter application via Bluetooth.

At first, the scanned Wi-Fi list was displayed without error, but at some point the following error appeared.

The log of PlatformIO (ESP32) showed the following error.

The size of the data you are trying to set is 724 bytes, but the maximum size allowed by the ESP32 BLE characteristics is 600 bytes, and the data is too large to be sent over BLE.

ShellScript

BLEServer connected
Scanning WiFi networks...
Scan complete.
Available networks:
Size of data to send: 724 bytes
[ 68429][E][BLECharacteristic.cpp:662] setValue(): Size 724 too large, must be no bigger than 600

Confirmation of current code

Currently, here is the code that sends the Wi-Fi scan results from ESP32 to the Flutter app via Bluetooth communication: the Wi-Fi scan results are serialized into a JSON format string and sent in batches.

C++

// src/BLEManager.cpp

// Serialisation of Wi-Fi network lists into JSON format strings.
String serializeNetworks(const std::vector &networks) {
  DynamicJsonDocument doc(1024);
  JsonArray array = doc.to();

  for (const auto &network : networks) {
    JsonObject obj = array.createNestedObject();
    obj["ssid"] = network.ssid;
    obj["rssi"] = network.rssi;
    obj["encryptionType"] = network.encryptionType;
  }

  String result;
  serializeJson(doc, result);
  return result;
}

void BLEManager::CharacteristicCallbacks::onWrite(
    BLECharacteristic *pCharacteristic) {
  std::string value = pCharacteristic->getValue();

  if (value.length() > 0) {
    std::string ssid;
    std::string password;
    std::string response;

    // Processing Wi-Fi network list requests
    if (value == "request_wifi_list") {
      // Scan for available Wi-Fi networks
      std::vector networks = scanNetworks();

      // Serialisation of scan results
      String networkListJson = serializeNetworks(networks);
      Serial.printf("networkListJson: %sn", networkListJson.c_str());

      // Log output of the size of the transmitted data.
      Serial.printf("Size of data to send: %d bytesn", networkListJson.length());

      // Serialised list transmitted via BLE
      pCharacteristic->setValue(networkListJson.c_str());
      pCharacteristic->notify();
      Serial.println("WiFi network list sent via BLE");

      return;
    }
  }
}

On the receiving end, Flutter uses the fluter_blue_plus plugin, and the default MTU (Maximum Transmission Unit: a value defining the maximum data size exchanged between BLE devices) on Android devices is 512 bytes as automatically requested; on iOS and macOS the MTU is automatically negotiated.

https://pub.dev/packages/flutter_blue_plus

So I think there are two ways to handle this case.

(1) Split large data into multiple messages and send them
(2) Reduce JSON size by shortening key names

I will do both. First, let’s start with a simple 2.

Shorten JSON key names to reduce JSON size

Currently, when sending a scanned Wifi list via BLE, the list structure is as shown below and the key names are rather long.

ShellScript

{"ssid":"XXXXXXX-X-XXXXX","rssi":-56,"encryptionType":3}

In the part where the Wi-Fi network list is serialized into a JSON format string, shorten the key name by changing ssid tos, rssi tor, encryptionType toe, etc.

C++

// src/BLEManager.cpp
// Serialisation of Wi-Fi network lists into JSON format strings.
String serializeNetworks(const std::vector &networks) {
  DynamicJsonDocument doc(1024);
  JsonArray array = doc.to();

  for (const auto &network : networks) {
    JsonObject obj = array.createNestedObject();
    obj["s"] = network.ssid;
    obj["r"] = network.rssi;
    obj["e"] = network.encryptionType;
  }

  String result;
  serializeJson(doc, result);
  return result;
}

Since the JSON key names have been changed, the code on the Flutter side (app) needs to be changed accordingly. Specifically, the factory method fromJson in the WifiNetwork class and the parts that use it will update the JSON key names to match the new shorter keys.

Dart

// lib/services/ble_service.dart
// Class that holds Wi-Fi network information.
class WifiNetwork {
  final String ssid;
  final int rssi;
  final int encryptionType;

  WifiNetwork(
      {required this.ssid, required this.rssi, required this.encryptionType});

  factory WifiNetwork.fromJson(Map json) {
    return WifiNetwork(
      ssid: json['ssid'] as String,
      rssi: json['rssi'] as int,
      encryptionType: json['encryptionType'] as int,
    );
  }
}

operation check

The Wifi list, which had exceeded 600 bytes and resulted in a BLE transmission error, was successfully sent after being reduced to 479 bytes by shortening the JSON key.

ShellScript

execute 1 minute action
Scanning WiFi networks...
12
Scan complete.
Available networks:
SSID / RSSI / Encryption
Size of data to send: 479 bytes
WiFi network list sent via BLE

However, this is not a fundamental solution, and if the Wi-Fi scan result is close to 100 in a commercial facility or other situation where a lot of Wi-Fi is flying around, then the BLE transmit size oversize error cannot be resolved.

So it is necessary to implement BLE split transmission.

Transmission in multiple messages by data division

PlatformIO (ESP32): Send in chunks

Create a function to send the WiFi network list in chunks.

Whenever data reaches the BLE maximum payload size (typically 512 bytes), data is sent through the BLE to start a new chunk.

C++

// Added to BLEManager.h
void sendDataInChunks(const std::string& data, unsigned int chunkSize);

// Implementation added to BLEManager.cpp
void BLEManager::sendDataInChunks(const std::string& data, unsigned int chunkSize) {
    size_t numChunks = data.length() / chunkSize + (data.length() % chunkSize != 0);
    for (size_t i = 0; i setValue(chunk);
        _pCharacteristic->notify();
        delay(100);
    }
}

void BLEManager::CharacteristicCallbacks::onWrite(BLECharacteristic *pCharacteristic) {
    std::string value = pCharacteristic->getValue();

    if (value == "request_wifi_list") {
        std::vector networks = scanNetworks();
        String networkListJson = serializeNetworks(networks);
        Serial.printf("networkListJson: %sn", networkListJson.c_str());
        Serial.printf("Size of data to send: %d bytesn", networkListJson.length());

        // Data is sent in chunks, where 512 is the BLE payload size.
        sendDataInChunks(networkListJson.c_str(), 512);
        Serial.println("WiFi network list sent via BLE in chunks");
    }
}

Add CCCD to Descriptor defined as part of BLE characteristics

When performing split transmissions, the notification (Notify) function of BLE can be used to send data to the application as soon as the ESP32 is ready for data. To achieve this, the “Client Characteristic Configuration Descriptor (CCCD)” must be set for the characteristics.

http://marchan.e5.valueserver.jp/cabin/comp/jbox/arc212/doc21201.html

C++

// BLEManager.hに追加
#include 

// BLEManager.cppに実装を追加
void BLEManager::initialize() {
  // Initialize the BLE Device
  BLEDevice::init("ミーア V1");

  // Create the BLE Server
  _pServer = std::unique_ptr(BLEDevice::createServer());
  _pServer->setCallbacks(&serverCallbacks);

  // Create the BLE Service
  BLEService *pService = _pServer->createService(SERVICE_UUID);

  // Create a BLE Characteristic
  _pCharacteristic = std::unique_ptr(
      pService->createCharacteristic(
          CHARACTERISTIC_UUID,
          BLECharacteristic::PROPERTY_READ |
              BLECharacteristic::PROPERTY_WRITE |
              BLECharacteristic::PROPERTY_NOTIFY |
              BLECharacteristic::PROPERTY_INDICATE));
              
  // Create a BLE Descriptor (CCCD)
  _pCharacteristic->addDescriptor(new BLE2902());
  _pCharacteristic->setCallbacks(&characteristicCallbacks);
  pService->start();
}

Flutter (app): reconstruct from segmented data

Notification Settings

Call setNotifyValue(true) on the characteristic to allow it to receive data from the BLE device asynchronously. This allows the Flutter app to be notified each time the device sends new data and receives the data immediately.
After enabling notification of a characteristic (after calling setNotifyValue(true )), the lastValueStream can be subscribed to in order to receive changes in the data sent from that characteristic in real time.

Reference to the Last Value Stream section of the flutter_blue_plus package

https://pub.dev/packages/flutter_blue_plus

How to reconstruct a split transmission

Add the received data to the StringBuffer and check if the data starts with a JSON start ( [ ) and ends with an end ( ]). If all chunks are received and a valid JSON string is formed, it is parsed and converted into a list of WifiNetwork objects.

Dart

// lib/services/ble_service.dart
import 'dart:async';
import 'dart:convert';
import 'package:flutter_blue_plus/flutter_blue_plus.dart';
import 'package:your_app/models/wifi_network.dart';

class BLEService {
  FlutterBluePlus flutterBlue = FlutterBluePlus.instance;
  final String serviceUUID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx";
  final String characteristicUUID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx";

  Future<List> requestWifiList(BluetoothDevice device) async {
    final completer = Completer<List>();
    List wifiList = [];
    StringBuffer buffer = StringBuffer();
    bool isStartChunk = true;
    bool isEndChunk = false;

    try {
      // デバイスに接続
      await device.connect();

      // サービスを探索
      List services = await device.discoverServices();
      BluetoothService service =
          services.firstWhere((s) => s.uuid == clockyServiceUUID);
      BluetoothCharacteristic characteristic =
          service.characteristics.firstWhere((c) => c.uuid == wifiUuid);

      // 通知を有効にする
      await characteristic.setNotifyValue(true);

      // Wi-Fi リスト要求を送信
      await characteristic.write(utf8.encode("request_wifi_list"));

      // ストリームを購読し、受信したデータをバッファリング
      Stream<List> stream = characteristic.lastValueStream;
      stream.listen((data) {
        String decodedData = utf8.decode(data);
        print('Received chunk: $decodedData');

        if (isStartChunk) {
          if (decodedData.startsWith('[')) {
            buffer.write(decodedData);
            isStartChunk = false;
            isEndChunk = false;
          }
        } else {
          if (decodedData.endsWith(']')) {
            buffer.write(decodedData);
            isEndChunk = true;
          } else {
            buffer.write(decodedData);
          }
        }

        if (isEndChunk) {
          String jsonData = buffer.toString();
          print("Received JSON data: $jsonData");

          try {
            final jsonResponse = jsonDecode(jsonData) as List;
            wifiList = jsonResponse.map((networkJson) {
              final networkMap = networkJson as Map;
              return WifiNetwork.fromJson(networkMap);
            }).toList();
            
            if (!completer.isCompleted) {
              buffer.clear();
              completer.complete(wifiList);
            }
          } catch (e) {
            print('Error parsing JSON data: $e');
          }
          isStartChunk = true;
          isEndChunk = false;
        }
      });

      // ストリームが完了したらリストを返す
      return completer.future;
    } catch (e) {
      print('Error occurred while requesting Wi-Fi list: $e');
      rethrow;
    }
  }
}

Completer and use and error avoidance during multiple rescanning

Completer Objectives

Completer is used to notify the outside world that an asynchronous operation has been completed. In this case, Future is used to complete and return the Wi-Fi network list to the caller after all data has been correctly received and parsed.
If there is no Completer, there is no way to verify that all data has arrived, the process proceeds with incomplete data, and the app does not show a list of Wifi scan results.

Error avoidance during multiple rescanning

If the “Find Wi-Fi” button is pressed again after the Completer has been completed once, an error occurs when trying to call. complete( ) again on a Completer that has already been completed. This is because once a Completer is completed (i.e., the .complete( ) method is called), it is not possible to call .complete() on the same Completer again.
To prevent this, the state of the Completer should be checked and completed only if it is incomplete. Also, create a new Completer instance at the start of each scan operation to make it independent of the state of the previous operation.

Dart

Future<List> requestWifiList(BluetoothDevice device) async {
  // スキャンごとに新しいCompleterを作成
  final completer = Completer<List>();
  
  try {
    // ... (デバイスへの接続コードなど)

    // ストリームリスナーコールバック
    Stream<List> stream = characteristic.lastValueStream;
    stream.listen((data) {
      String decodedData = utf8.decode(data);

      // ... (チャンク処理とJSONデータ解析のコード)

      if (isEndChunk) {
        // 完全なJSONデータを処理

        // 解析前にJSONデータを検証
        try {
          // Completerが完了していないことを確認
          if (!completer.isCompleted) {
            buffer.clear();
            completer.complete(wifiList);
          }
        } catch (e) {
          print('Error parsing JSON data: $e');
        }
      }
    });
    
    // CompleterのFutureを返す
    return completer.future;
  } catch (e) {
    print('Error occurred while requesting Wi-Fi list: $e');
    rethrow;
  }
}

operation check

PlatformIO (ESP32)

Even in the case of data exceeding 512 bytes, it is successfully transmitted.

ShellScript

Scanning WiFi networks...
13
Scan complete.
Available networks:
SSID / RSSI / Encryption
Size of data to send: 720 bytes
WiFi network list sent via BLE in chunks

flutter (sound)

It can be seen that the Wi-Fi list can be reconstructed as JSON data only when a valid JSON string is formed for the Wi-Fi scan results sent in segments from the ESP32.

ShellScript

flutter: Received chunk: P-337H-G","rssi":-87,"encryptionType":4},{"ssid":"********-W","rssi":-88,"encryptionType":3},<br>flutter: Received chunk: [{"ssid":"********-W","rssi":-64,"encryptionType":3},{"ssid":"********-W","rssi":-71,"encryptionType":3},{"ssid":"AP_sta<br>flutter: Received chunk: ff_3F_2.4","rssi":-88,"encryptionType":4},{"ssid":"********-W","rssi":-89,"encryptionType":4}]<br>flutter: Received JSON data: [{"ssid":"********-W","rssi":-64,"encryptionType":3},{"ssid":"********-W","rssi":-71,"encryptionType":3},{"ssid":"********-W","rssi":-79,"encryptionType":3}]

The Wifi list was successfully displayed on the app screen as well.

When the number of Wi-Fi scans exceeds the screen height, the number is scrolled down. It was confirmed that the number of Wi-Fi is also displayed successfully when the “Find Again” button is clicked.