▶️Introduction
The Camera SDK is a PPG recording module that you can use in combination with the FibriCheck Cloud SDK. A FibriCheck measurement contains PPG data. To obtain this data, the Camera SDK communicates with the native iOS/Android camera layer, processes this data, and returns an object to submit to our backend for analysis. Multiple properties and listeners can be adjusted/attached for improving the visualization/customization of the process.
The different phases of a FibriCheck measurement are:
Finger detection Check for the presence of a finger on the camera. You can set the timeout to 0 to skip this phase. By default, this value is set to
-1
which means that it keeps checking until a finger has been detected.Pulse detection Check if a pulse is present. When no pulse has been detected for 10 seconds, the calibration phase will start.
Calibration When performing a measurement, a baseline needs to be calculated. When this baseline has been calculated, the calibration is ready and recording can start.
Recording During the recording phase, the Camera SDK algorithm calculates the PPG data by processing the mobile device's camera feed.
Processing When the recording is finished, some additional processing needs to be done on the measurement. When done, a measurement object is presented via the onMeasurementProcessed event.
Installation
Set the correct permissions
The recording makes use of the device's camera. So you'll need to make sure that your application has access to the camera.
Depending on the operating system, you will need to make changes to the project's configuration file.
Add this to the AndroidManifest.xml
file:
<uses-permission android:name="android.permission.CAMERA" />
For more information regarding Android permissions, check the official Android documentation.
Next to modifying the project configuration, your app will also have to ask the user to allow using the camera:
To ask for the correct permissions we use the permission_handler
package.
Add the following snippet to your Podfile
:
post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
target.build_configurations.each do |config|
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
'$(inherited)',
## dart: PermissionGroup.camera
'PERMISSION_CAMERA=1',
]
end
end
end
Then in your code you can request the camera
permission:
var status = await Permission.camera.status;
if (status.isDenied) {
PermissionStatus requestResult
requestResult = await Permission.camera.request()
if (requestResult.isGranted) {
// Request has been granted
}
}
For more details, you can take a look at the permission_handler documentation
Install the Camera SDK
In your project, you can add the package below to the pubspec.yaml
file. Replace {TOKEN}
with the personal access token you've received from FibriCheck.
flutter_fibricheck_sdk:
git:
url: https://{TOKEN}@github.com/fibricheck/flutter-camera-sdk
ref: v1.0.0
Perform your first measurement
In this paragraph, we explain how you can perform a measurement
The Camera SDK provides a widget that has the following structure:
FibriCheckView(
fibriCheckViewProperties:
FibriCheckViewProperties(
flashEnabled: true,
lineThickness: 4,
...,
),
onCalibrationReady: () => debugPrint("Flutter onCalibrationReady"),
onFingerDetected: () => debugPrint("Flutter onFingerDetected"),
onFingerDetectionTimeExpired: () => debugPrint("Flutter onFingerDetectionTimeExpired"),
onFingerRemoved: () => debugPrint("Flutter onFingerRemoved"),
onHeartBeat: (heartbeat) => debugPrint("Flutter onHeartBeat $heartbeat"),
onMeasurementFinished: () => debugPrint("Flutter onMeasurementFinished"),
onMeasurementProcessed: (measurement) => debugPrint("Flutter onMeasurementProcessed $measurement"),
onMeasurementStart: () => debugPrint("Flutter onMeasurementStart"),
onMovementDetected: () => debugPrint("Flutter onMovementDetected"),
onPulseDetected: () => debugPrint("Flutter onPulseDetected"),
onPulseDetectionTimeExpired: () => debugPrint("Flutter onPulseDetectionTimeExpired"),
onSampleReady: (ppg, raw) => debugPrint("Flutter onSampleReady $ppg $raw"),
onTimeRemaining: (seconds) => debugPrint("Flutter onTimeRemaining $seconds"),
),
You can use the FibriCheckView
exported from the package:camera_sdk/fibri_check_view.dart
package to perform a measurement and hook up sdk.postMeasurement
to post the data returned from the camera to the backend in the onMeasurementProcessed
event.
Before taking a measurement, you need to check if you are entitled to perform a measurement. This can be achieved by invoking
sdk.canPerformMeasurement
. If you try to execute a measurement when you are not entitled, aNoActivePrescriptionError
will be thrown. So make sure you've Activated a Prescription.It is highly recommended to provide the camera SDK version as a second argument, as shown in the example.
import 'package:camera_sdk/fibri_check_view.dart';
import 'dart:convert';
import 'package:camera_sdk/fibri_check_view.dart';
import 'package:flutter/material.dart';
import 'package:flutter_fibricheck_sdk/flutter_fibricheck_sdk.dart';
import 'package:flutter_fibricheck_sdk/measurement.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:wakelock/wakelock.dart';
import '../0_design_system/fc_colors.dart';
import 'widgets/fc_metrics.dart';
import 'widgets/fc_title.dart';
class CameraScreen extends StatefulWidget {
const CameraScreen({super.key, required this.sdk});
final FLFibriCheckSdk sdk;
@override
State<CameraScreen> createState() => _CameraScreenState();
}
class _CameraScreenState extends State<CameraScreen> {
Future<void>? _requestCameraPermission;
bool _hasCameraPermission = false;
bool _measurementFinished = false;
String _timeRemaining = "-";
String _heartBeat = "-";
String _status = "Place your finger on the camera";
Measurement? _measurement;
@override
initState() {
super.initState();
_requestCameraPermission = _requestCameraPermissionImpl();
}
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
backgroundColor: FCColors.brokenWhite,
appBar: AppBar(
backgroundColor: FCColors.green,
title: const Text('Camera'),
),
body: Column(
children: [
Expanded(
child: FutureBuilder(
future: _requestCameraPermission,
builder: (context, snapshot) {
if (snapshot.connectionState == ConnectionState.waiting) {
return const DemoTitleWidget(title: "Requiring camera permission");
}
if (!_hasCameraPermission) {
return const DemoTitleWidget(title: "Camera permission not granted");
}
return Column(
children: [
DemoTitleWidget(title: _status),
Container(
decoration: const BoxDecoration(
border: Border(
top: BorderSide(color: FCColors.lightGray, width: 1),
bottom: BorderSide(color: FCColors.lightGray, width: 1),
),
),
height: 200,
child: FibriCheckView(
fibriCheckViewProperties: FibriCheckViewProperties(
flashEnabled: true,
lineThickness: 4,
),
onCalibrationReady: () => {
debugPrint("Flutter onCalibrationReady"),
setState(() {
_status = "Recording heartbeat...";
}),
},
onFingerDetected: () => {
Wakelock.enable(),
debugPrint("Flutter onFingerDetected"),
setState(() {
_status = "Detecting pulse...";
}),
},
onFingerDetectionTimeExpired: () => debugPrint("Flutter onFingerDetectionTimeExpired"),
onFingerRemoved: () => {
Wakelock.disable(),
debugPrint("Flutter onFingerRemoved"),
},
onHeartBeat: (heartbeat) => {
debugPrint("Flutter onHeartBeat $heartbeat"),
setState(() {
_heartBeat = heartbeat.toString();
}),
},
onMeasurementFinished: () => {
debugPrint("Flutter onMeasurementFinished"),
setState(() {
_status = "Measurement finished!";
}),
},
onMeasurementProcessed: (measurement) async => {
await _onMeasurementFinished(measurement),
debugPrint("Flutter onMeasurementProcessed $measurement"),
if (Navigator.canPop(context)) Navigator.pop(context),
},
onPulseDetected: () => {
debugPrint("Flutter onPulseDetected"),
setState(() {
_status = "Calibrating...";
}),
},
onTimeRemaining: (seconds) => {
debugPrint("Flutter onTimeRemaining $seconds"),
setState(() {
_timeRemaining = seconds.toString();
}),
},
),
),
DemoMetricsWidget(timeRemaining: _timeRemaining, heartBeat: _heartBeat),
],
);
},
),
),
],
),
),
);
}
Future<void> _requestCameraPermissionImpl() async {
var result = await Permission.camera.request();
setState(() {
_hasCameraPermission = result.isGranted;
});
}
Future _onMeasurementFinished(String measurementString) async {
var mCreationData = MeasurementCreationData.fromCameraSdk(measurementString);
await widget.sdk.postMeasurement(mCreationData, "v0.0.1");
}
}
In rare cases, it can occur that the motion sensors don't provide the correct data. Because the algorithm requires motion sensor data to be available, an onMeasurementError
event will be thrown in this case. Look here for more information.
Device Requirements
A full FibriCheck measurement consists of an on-device data acquisition step and a cloud data analysis step, performed by an AI algorithm. The on-device algorithms extracts a raw measurement from the the camera feed.
The use of the SDK has no significant impact on storage or memory usage of the device. The frames from the camera feed are processed on-the-fly by memory-efficient algorithms.
During the measurement there will be a significant CPU usage caused by the on-the-fly processing of the camera feed. However these processing algorithms also power the FibriCheck application which is broadly available on low-end and high-end smartphones. In this respect, we don't expect any performance issues by using the Camera SDK's in your application.
The following table lists the required minimum mobile operating system versions, and minimum framework versions:
Android
Android KitKat (4.4) API Level 19
iOS
iOS 11 (Sept. 17)
Important Remarks
Camera selection
Modern phones have multiple cameras. The Camera SDK uses the default capture device that is able to record video content.
To guide the user in putting their finger on the correct camera, it's recommended to show the camera output as a "peephole" in the interface at the start of a measurement.
In order to aid the user in using the correct camera lens, you can provide a preview of the relevant camera via the following package: https://pub.dev/packages/camera.
The Camera SDK uses the default camera.
Make sure to catch all measurement errors
An ongoing measurement will stop when a measurement error occurs. Make sure that an onMeasurementError
has been defined. See the onMeasurementError documentation for more information
Framework-specific remarks
Drawing on the JS Thread
When benchmarking the SDK, we noticed that drawing on the JS Thread while taking a measurement caused severe spikes in the processing power. This will results in a bad quality measurement. So when creating a visualisation, for example counting down the seconds that are left in a measurement, make sure you are not drawing on the JS Thread. Either make use of Native Driver or use React Reanimated. When using third party libraries for creating animations, make sure they also offload the drawing from the JS Thread.
Not using Hermes
When benchmarking the SDK, we noticed that Hermes also had a big impact on the performance of low-end devices. So we advice you to enable it if possible. Instructions can be found in their documentation.
Visualizing the ongoing measurement
The SDK emits an onSampleReady
event on each processed frame. The event contains a filtered ppg
value and a raw
measurement value of the latest received video frame.
You can use these values to visualize the PPG graph to the user during the measurement. Depending on the lightning conditions, the variance of the ppg
values can be very low during most part of the measurement (decimal values between -1 and +1). Make sure to apply appropriate scaling to the graph to correctly visualize the PPG measurement to the user. Avoid visualizing an apparently flat line.
Last updated