Comment on page
▶
Introduction
The Camera SDK is a PPG recording module that you can use in combination with the FibriCheck Cloud SDK. A FibriCheck measurement contains PPG data. To obtain this data, the Camera SDK communicates with the native iOS/Android camera layer, processes this data, and returns an object to submit to our backend for analysis. Multiple properties and listeners can be adjusted/attached for improving the visualization/customization of the process.
The different phases of a FibriCheck measurement are:
- 1.Finger detection Check for the presence of a finger on the camera. You can set the timeout to 0 to skip this phase. By default, this value is set to
-1which means that it keeps checking until a finger has been detected. - 2.Pulse detection Check if a pulse is present. When no pulse has been detected for 10 seconds, the calibration phase will start.
- 3.Calibration When performing a measurement, a baseline needs to be calculated. When this baseline has been calculated, the calibration is ready and recording can start.
- 4.Recording During the recording phase, the Camera SDK algorithm calculates the PPG data by processing the mobile device's camera feed.
- 5.Processing When the recording is finished, some additional processing needs to be done on the measurement. When done, a measurement object is presented via the onMeasurementProcessed event.
The Camera SDK is currently available for Flutter, React Native, Cordova, iOS and Android
The recording makes use of the device's camera. So you'll need to make sure that your application has access to the camera.
Depending on the operating system, you will need to make changes to the project's configuration file.
Android
iOS
Add this to the
AndroidManifest.xml file:<uses-permission android:name="android.permission.CAMERA" />
Add this to the
Info.plist file:<key>NSCameraUsageDescription</key>
<string>Your own description of the purpose</string>
Next to modifying the project configuration, your app will also have to ask the user to allow using the camera:
Flutter
React Native
Android
iOS
Cordova
Add the following snippet to your
Podfile:post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
target.build_configurations.each do |config|
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
'$(inherited)',
## dart: PermissionGroup.camera
'PERMISSION_CAMERA=1',
]
end
end
end
Then in your code you can request the
camera permission:var status = await Permission.camera.status;
if (status.isDenied) {
PermissionStatus requestResult
requestResult = await Permission.camera.request()
if (requestResult.isGranted) {
// Request has been granted
}
}
import {request, PERMISSIONS} from 'react-native-permissions';
const [camera, setCamera] = useState(false);
useEffect(() => {
if (Platform.OS === 'ios') {
request(PERMISSIONS.IOS.CAMERA).then(result => {
setCamera(result === 'granted');
});
} else {
request(PERMISSIONS.ANDROID.CAMERA).then(result => {
setCamera(result === 'granted');
});
}
}, []);
As you can see in the snippet, we used the
react-native-permissions library. In order to use this library, add these values to your Podfile:target 'MyAppName' do
# react-native-permissions permission handlers
permissions_path = '../node_modules/react-native-permissions/ios'
pod 'Permission-Camera', :path => "#{permissions_path}/Camera.podspec"
end
Make sure you execute
pod install and restart the metro bundler after applying these changes!The following snippet from the Android documentation on requesting runtime permissions shows how to request the
CAMERA permission: if (ContextCompat.checkSelfPermission(
context, Manifest.permission.CAMERA) ==
PackageManager.PERMISSION_GRANTED) {
// You can use the API that requires the permission.
performAction(...);
} else if (shouldShowRequestPermissionRationale(...)) {
// In an educational UI, explain to the user why your app requires this
// permission for a specific feature to behave as expected, and what
// features are disabled if it's declined. In this UI, include a
// "cancel" or "no thanks" button that lets the user continue
// using your app without granting the permission.
showInContextUI(...);
} else {
// You can directly ask for the permission.
// The registered ActivityResultCallback gets the result of this request.
requestPermissionLauncher.launch(
Manifest.permission.CAMERA);
}
The following snippet has been extracted from the Apple Documentation and shows how to check if the user has given permission to use the camera:
var isAuthorized: Bool {
get async {
let status = AVCaptureDevice.authorizationStatus(for: .video)
// Determine if the user previously authorized camera access.
var isAuthorized = status == .authorized
// If the system hasn't determined the user's authorization status,
// explicitly prompt them for approval.
if status == .notDetermined {
isAuthorized = await AVCaptureDevice.requestAccess(for: .video)
}
return isAuthorized
}
}
func setUpCaptureSession() async {
guard await isAuthorized else { return }
// Set up the capture session.
}
No specific action needed for the Cordova SDK. The SDK will automatically ask permission to the user if the permissions are configured correctly in the previous step.
Flutter
React Native
Native (Android / iOS)
Cordova
In your project, you can add the package below to the
pubspec.yaml file. Replace {TOKEN} with the personal access token you've received from FibriCheck. flutter_fibricheck_sdk:
git:
url: https://{TOKEN}@github.com/fibricheck/flutter-camera-sdk
ref: v1.0.0
In your project, if you are using
yarn or npm you need to create a file called .npmrc at the root level of your project and add these lines. Replace {AUTH_TOKEN} with the personal access token you've received from FibriCheck@fibricheck:registry=https://npm.pkg.github.com
@extrahorizon:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${AUTH_TOKEN}
Alternatively, this file can be added/edited in your home directory and it will be applied to all projects.
using npm:
npm install @fibricheck/react-native-camera-sdk
using yarn:
yarn add @fibricheck/react-native-camera-sdk
You will receive access to the Native Camera SDK repository. This repository will contain the source files that you can include directly in your project.
To install the Camera SDK for Cordova, you will receive an authentication token to access the private npm package hosted on GitHub Packages.
To include the
cordova-camera-sdk package in your application, follow these steps:Add a
.npmrc file to the root of your project with the following content:@fibricheck:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${AUTH_TOKEN}
Replace
${AUTH_TOKEN} with a valid authentication token.
Next, add the package to your project using npm or yarnnpm install --save @fibricheck/cordova-fibricheck-camera-sdk
// OR
yarn install --save @fibricheck/cordova-fibricheck-camera-sdk
After completing the above steps, the FibriCheck Camera SDK is available in your code in the
cordova.plugins object: const { FibriCheckCordovaSDK: fc } = cordova.plugins;
For iOS, the minimal deployment target is iOS 13. Make sure to add the following configuration to the
config.xml, where the value must be >=13:<platform name="ios">
<preference name="deployment-target" value="13.0" />
</platform>
In this paragraph, we explain how you can perform a measurement
Flutter
React Native
Android
iOS
Cordova
The Camera SDK provides a widget that has the following structure:
FibriCheckView(
fibriCheckViewProperties:
FibriCheckViewProperties(
flashEnabled: true,
lineThickness: 4,
...,
),
onCalibrationReady: () => debugPrint("Flutter onCalibrationReady"),
onFingerDetected: () => debugPrint("Flutter onFingerDetected"),
onFingerDetectionTimeExpired: () => debugPrint("Flutter onFingerDetectionTimeExpired"),
onFingerRemoved: () => debugPrint("Flutter onFingerRemoved"),
onHeartBeat: (heartbeat) => debugPrint("Flutter onHeartBeat $heartbeat"),
onMeasurementFinished: () => debugPrint("Flutter onMeasurementFinished"),
onMeasurementProcessed: (measurement) => debugPrint("Flutter onMeasurementProcessed $measurement"),
onMeasurementStart: () => debugPrint("Flutter onMeasurementStart"),
onMovementDetected: () => debugPrint("Flutter onMovementDetected"),
onPulseDetected: () => debugPrint("Flutter onPulseDetected"),
onPulseDetectionTimeExpired: () => debugPrint("Flutter onPulseDetectionTimeExpired"),
onSampleReady: (ppg, raw) => debugPrint("Flutter onSampleReady $ppg $raw"),
onTimeRemaining: (seconds) => debugPrint("Flutter onTimeRemaining $seconds"),
),
You can use the
FibriCheckView exported from the package:camera_sdk/fibri_check_view.dart package to perform a measurement and hook up sdk.postMeasurement to post the data returned from the camera to the backend in the onMeasurementProcessed event.- Before taking a measurement, you need to check if you are entitled to perform a measurement. This can be achieved by invoking
sdk.canPerformMeasurement. If you try to execute a measurement when you are not entitled, aNoActivePrescriptionErrorwill be thrown. So make sure you've Activated a Prescription. - It is highly recommended to provide the camera SDK version as a second argument, as shown in the example.
import 'package:camera_sdk/fibri_check_view.dart';
import 'dart:convert';
import 'package:camera_sdk/fibri_check_view.dart';
import 'package:flutter/material.dart';
import 'package:flutter_fibricheck_sdk/flutter_fibricheck_sdk.dart';
import 'package:flutter_fibricheck_sdk/measurement.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:wakelock/wakelock.dart';
import '../0_design_system/fc_colors.dart';
import 'widgets/fc_metrics.dart';
import 'widgets/fc_title.dart';
class CameraScreen extends StatefulWidget {
const CameraScreen({super.key, required this.sdk});
final FLFibriCheckSdk sdk;
@override
State<CameraScreen> createState() => _CameraScreenState();
}
class _CameraScreenState extends State<CameraScreen> {
Future<void>? _requestCameraPermission;
bool _hasCameraPermission = false;
bool _measurementFinished = false;
String _timeRemaining = "-";
String _heartBeat = "-";
String _status = "Place your finger on the camera";
Measurement? _measurement;
@override
initState() {
super.initState();
_requestCameraPermission = _requestCameraPermissionImpl();
}
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
backgroundColor: FCColors.brokenWhite,
appBar: AppBar(
backgroundColor: FCColors.green,
title: const Text('Camera'),
),
body: Column(
children: [
Expanded(
child: FutureBuilder(
future: _requestCameraPermission,
builder: (context, snapshot) {
if (snapshot.connectionState == ConnectionState.waiting) {
return const DemoTitleWidget(title: "Requiring camera permission");
}
if (!_hasCameraPermission) {
return const DemoTitleWidget(title: "Camera permission not granted");
}
return Column(
children: [
DemoTitleWidget(title: _status),
Container(
decoration: const BoxDecoration(
border: Border(
top: BorderSide(color: FCColors.lightGray, width: 1),
bottom: BorderSide(color: FCColors.lightGray, width: 1),
),
),
height: 200,
child: FibriCheckView(
fibriCheckViewProperties: FibriCheckViewProperties(
flashEnabled: true,
lineThickness: 4,
),
onCalibrationReady: () => {
debugPrint("Flutter onCalibrationReady"),
setState(() {
_status = "Recording heartbeat...";
}),
},
onFingerDetected: () => {
Wakelock.enable(),
debugPrint("Flutter onFingerDetected"),
setState(() {
_status = "Detecting pulse...";
}),
},
onFingerDetectionTimeExpired: () => debugPrint("Flutter onFingerDetectionTimeExpired"),
onFingerRemoved: () => {
Wakelock.disable(),
debugPrint("Flutter onFingerRemoved"),
},
onHeartBeat: (heartbeat) => {
debugPrint("Flutter onHeartBeat $heartbeat"),
setState(() {
_heartBeat = heartbeat.toString();
}),
},
onMeasurementFinished: () => {
debugPrint("Flutter onMeasurementFinished"),
setState(() {
_status = "Measurement finished!";
}),
},
onMeasurementProcessed: (measurement) async => {
await _onMeasurementFinished(measurement),
debugPrint("Flutter onMeasurementProcessed $measurement"),
if (Navigator.canPop(context)) Navigator.pop(context),
},
onPulseDetected: () => {
debugPrint("Flutter onPulseDetected"),
setState(() {
_status = "Calibrating...";
}),
},
onTimeRemaining: (seconds) => {
debugPrint("Flutter onTimeRemaining $seconds"),
setState(() {
_timeRemaining = seconds.toString();
}),
},
),
),
DemoMetricsWidget(timeRemaining: _timeRemaining, heartBeat: _heartBeat),
],
);
},
),
),
],
),
),
);
}
Future<void> _requestCameraPermissionImpl() async {
var result = await Permission.camera.request();
setState(() {
_hasCameraPermission = result.isGranted;
});
}
Future _onMeasurementFinished(String measurementString) async {
var mCreationData = MeasurementCreationData.fromCameraSdk(measurementString);
await widget.sdk.postMeasurement(mCreationData, "v0.0.1");
}
}
The Camera SDK provides a React Native Component that you can integrate quickly in your application.
<RNFibriCheckView
style={{ flex: 1, backgroundColor: '#ffffff' }}
onFingerDetected={() => console.log('finger detected')}
onFingerRemoved={() => console.log('finger removed')}
onCalibrationReady={() => console.log('calibration ready')}
onMeasurementFinished={() => console.log('measurement finished')}
onMeasurementStart={() => console.log('measurement recording started')}
onFingerDetectionTimeExpired={() =>
console.log('finger detection time expired')
}
onPulseDetected={() => console.log('pulse detected')}
onPulseDetectionTimeExpired={() =>
console.log('pulse detection time is expired')
}
onMovementDetected={() => console.log('movement detected')}
onHeartBeat={(heartRate) => console.log(`current heart rate: ${heartRate}`)}
onTimeRemaining={(seconds) => console.log(`time remaining: ${seconds}`)}
onMeasurementError={(error) => console.log(`measurement error occured: ${error}`)}
onMeasurementProcessed={(data) =>
console.log('measurement processed and ready to send!');
}
/>
You can use the
RNFibriCheckView exported from the @fibricheck/react-native-camera-sdk package to perform a measurement and hook up sdk.postMeasurement to post the data returned from the camera to the backend in the onMeasurementProcessed event.- Before taking a measurement, you need to check if you are entitled to perform a measurement. This can be achieved by invoking
sdk.canPerformMeasurement. If you try to execute a measurement when you are not entitled, aNoActivePrescriptionErrorwill be thrown. So make sure you've Activated a Prescription. - It is highly recommended to provide the camera sdk version as a second argument, as shown in the example.
- The measurement context is prefilled here, but needs to be completed by the user. The best practice is to add this at a later stage with
sdk.updateMeasurementContext. More information about this can be found in the Measurement Structure.
import client from '@fibricheck/javascript-sdk';
import { RNFibriCheckView } from '@fibricheck/react-native-camera-sdk';
import { useEffect, useState } from 'react';
const sdk = client({
consumerKey: '',
consumerSecret: '',
});
await sdk.authenticate({
token: '',
tokenSecret: '',
});
const App = () => {
const [cameraData, setCameraData] = useState();
const [context, setContext] = useState({
symptoms: [
'lightheaded',
'confused'
],
activity: 'sleeping',
});
async function postMeasurement(cameraData, context) {
const measurement = {
...cameraData,
context
};
await sdk.postMeasurement(measurement, RNFibriCheckView.version);
}
return (
<RNFibriCheckView
onHeartBeat={(heartRate) => {
console.log('heartRate', heartRate);
}}
onTimeRemaining={(seconds) => {
console.log('onTimeRemaining', seconds);
}}
onMeasurementProcessed={async (cameraData) => {
console.log('onMeasurementProcessed', cameraData);
setCameraData(cameraData);
}}
onFingerDetected={() => {
console.log('finger detected');
}}
/>
);
};
The
FibriChecker class is the main class that you have to use to implement FibriCheck in your application.
import com.qompium.fibrichecker.FibriChecker;
import com.qompium.fibrichecker.listeners.FibriListener;
import com.qompium.fibrichecker.measurement.MeasurementData;
public class MeasurementView extends LinearLayout {
linearLayout = new LinearLayout(context);
linearLayout.setLayoutParams(new LinearLayout.LayoutParams(LinearLayout.LayoutParams.MATCH_PARENT, LinearLayout.LayoutParams.MATCH_PARENT));
linearLayout.setOrientation(LinearLayout.HORIZONTAL);
fibriChecker = new FibriChecker.FibriBuilder(context.getCurrentActivity(), linearLayout).build();
fibriChecker.setFibriListener(new FibriListener() {
@Override public void onSampleReady(final double ppg, double raw) {
// Invoked multiple times during the measurement
// Contains data to update visualisations in your views
}
@Override public void onFingerDetected() {
// Invoked when the finger is placed correctly
}
@Override public void onFingerRemoved(double y, double v, double stdDevY) {
// Invoked when the finger is removed
}
@Override public void onCalibrationReady() {
}
@Override public void onHeartBeat(int value) {
}
@Override public void timeRemaining(int seconds) {
}
@Override public void onMeasurementFinished() {
}
@Override public void onMeasurementStart() {
}
@Override public void onFingerDetectionTimeExpired() {
}
@Override public void onPulseDetected() {
}
@Override public void onPulseDetectionTimeExpired() {
}
@Override public void onMovementDetected() {
}
@Override public void onMeasurementProcessed(MeasurementData measurementData) {
}
@Override public void onMeasurementError(String message) {
}
});
fibriChecker.start();
return linearLayout;
}
The iOS Native SDK is written in Objective-C. Newer iOS applications are typically developed in Swift. It's possible to use Objective-C files and Swift files in the same project. See the Apple documentation for more info.
The following code snippet is a working example that will show the heart rate, calculated using the Native SDK, on the main screen:
import SwiftUI
@main
struct test_fc_appApp: App {
var body: some Scene {
WindowGroup {
ContentView()
}
}
}
struct ContentView: View {
@State var heartRate : UInt = 0
var body: some View {
VStack {
Image(systemName: "globe")
.imageScale(.large)
.foregroundColor(.accentColor)
Text("HeartRate: " + String(heartRate))
Button("Measurement", action: {
let viewSelf = self
DispatchQueue.global(qos: .userInitiated).async {
var threadDone = false
func handleError(_: String?) -> Void {
print("There was a measurement error")
threadDone = true
}
func handleMeasurementStart() -> Void {
print("Measurement started")
}
func handleMeasurementFinished() -> Void {
print("Measurement finished")
threadDone = true
}
func handleHeartRate(hr: UInt) -> Void {
print("Received HeartRate " + String(hr))
DispatchQueue.main.async {
print("Update HeartRate")
viewSelf.heartRate = hr
}
}
func handleMovementDetected() -> Void {
print("Movement Detected")
}
func handlePulseDetection() -> Void {
print("Pulse Detected")
}
let fc = FibriChecker()
fc.onMeasurementError = handleError
fc.onMeasurementStart = handleMeasurementStart
fc.onMovementDetected = handleMovementDetected
fc.onMeasurementFinished = handleMeasurementFinished
fc.onHeartBeat = handleHeartRate
fc.onPulseDetected = handlePulseDetection
fc.startMeasurement()
while !threadDone {
usleep(1)
}
}
})
}
.padding()
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
const { FibriCheckCordovaSDK: fc } = cordova.plugins;
// Step 1: initialise the SDK
await fc.initialise();
// Set listeners for events (full list available in docs)
fc.onMeasurementStart(() => {
console.log("Measurement Started")
});
fc.onSampleReady((ppg: number, raw: number) => {
console.log("PPG Sample Received");
// use ppg value to visualize a graph
});
fc.onHeartBeat((hr: number) => {
console.log(`Heartbeat ${hr}`);
});
fc.onMeasurementProcessed((data) => {
console.log("Measurement Processed");
// data object should be sent to FibriCheck Cloud
});
await fc.startMeasurement();
In rare cases, it can occur that the motion sensors don't provide the correct data. Because the algorithm requires motion sensor data to be available, an
onMeasurementError event will be thrown in this case. Look here for more information. A full FibriCheck measurement consists of an on-device data acquisition step and a cloud data analysis step, performed by an AI algorithm. The on-device algorithms extracts a raw measurement from the the camera feed.
The use of the SDK has no significant impact on storage or memory usage of the device. The frames from the camera feed are processed on-the-fly by memory-efficient algorithms.
During the measurement there will be a significant CPU usage caused by the on-the-fly processing of the camera feed. However these processing algorithms also power the FibriCheck application which is broadly available on low-end and high-end smartphones. In this respect, we don't expect any performance issues by using the Camera SDK's in your application.
The following table lists the required minimum mobile operating system versions, and minimum framework versions:
Platform | Minimum OS (and framework) |
|---|---|
Android | Android KitKat (4.4)
API Level 19 |
iOS | iOS 11 (Sept. 17) |
Modern phones have multiple cameras. The Camera SDK uses the default capture device that is able to record video content.
To guide the user in putting their finger on the correct camera, it's recommended to show the camera output as a "peephole" in the interface at the start of a measurement.
Flutter
React Native
In order to aid the user in using the correct camera lens, you can provide a preview of the relevant camera via the following package: https://pub.dev/packages/camera.
The Camera SDK uses the default camera.
There is a library react-native-vision-camera that is able to select the correct lens. The following snippet provides an example of how this can be implemented:
import { Camera, useCameraDevices } from 'react-native-vision-camera';
import styled from 'styled-components/native';
const SCREEN_HEIGHT = Dimensions.get('window').height;
const SCREEN_WIDTH = Dimensions.get('window').width;
const radiusBasis = Number(
Math.round(SCREEN_HEIGHT / (SCREEN_HEIGHT <= 800 ? 6 : 8)),
);
const CameraContainer = styled.View`
flex: 1;
align-items: center;
justify-content: center;
max-height: ${radiusBasis}px;
`;
const CameraContent = styled.View`
overflow: hidden;
border-radius: 100px;
`;
const devices = useCameraDevices('wide-angle-camera');
const device = devices.back;
export const FindYourLens = () => {
return (
<CameraContainer>
<CameraContent>
<Camera
style={...}
device={device}
preset={'vga-640x480'}
/>
</CameraContent>
</CameraContainer>
);
};
An ongoing measurement will stop when a measurement error occurs. Make sure that an
onMeasurementError has been defined. See the onMeasurementError documentation for more information React Native
When benchmarking the SDK, we noticed that drawing on the JS Thread while taking a measurement caused severe spikes in the processing power. This will results in a bad quality measurement. So when creating a visualisation, for example counting down the seconds that are left in a measurement, make sure you are not drawing on the JS Thread. Either make use of Native Driver or use React Reanimated. When using third party libraries for creating animations, make sure they also offload the drawing from the JS Thread.
When benchmarking the SDK, we noticed that Hermes also had a big impact on the performance of low-end devices. So we advice you to enable it if possible. Instructions can be found in their documentation.
The SDK emits an
onSampleReady event on each processed frame. The event contains a filtered ppg value and a raw measurement value of the latest received video frame. You can use these values to visualize the PPG graph to the user during the measurement. Depending on the lightning conditions, the variance of the
ppg values can be very low during most part of the measurement (decimal values between -1 and +1). Make sure to apply appropriate scaling to the graph to correctly visualize the PPG measurement to the user. Avoid visualizing an apparently flat line. Last modified 3mo ago