▶️Introduction

The Camera SDK is a PPG recording module that you can use in combination with the FibriCheck Cloud SDK. A FibriCheck measurement contains PPG data. To obtain this data, the Camera SDK communicates with the native iOS/Android camera layer, processes this data, and returns an object to submit to our backend for analysis. Multiple properties and listeners can be adjusted/attached for improving the visualization/customization of the process.

The different phases of a FibriCheck measurement are:

Finger detection Check for the presence of a finger on the camera. You can set the timeout to 0 to skip this phase. By default, this value is set to -1 which means that it keeps checking until a finger has been detected.
Pulse detection Check if a pulse is present. When no pulse has been detected for 10 seconds, the calibration phase will start.
Calibration When performing a measurement, a baseline needs to be calculated. When this baseline has been calculated, the calibration is ready and recording can start.
Recording During the recording phase, the Camera SDK algorithm calculates the PPG data by processing the mobile device's camera feed.
Processing When the recording is finished, some additional processing needs to be done on the measurement. When done, a measurement object is presented via the onMeasurementProcessed event.

The Camera SDK is currently available for Flutter, React Native, Cordova, iOS and Android

Installation

Set the correct permissions

The recording makes use of the device's camera. So you'll need to make sure that your application has access to the camera.

Depending on the operating system, you will need to make changes to the project's configuration file.

Add this to the AndroidManifest.xml file:

<uses-permission android:name="android.permission.CAMERA" />

For more information regarding Android permissions, check the official Android documentation.

Add this to the Info.plist file:

<key>NSCameraUsageDescription</key>
<string>Your own description of the purpose</string>

For more information regarding these iOS permissions, check the official iOS documentation.

Next to modifying the project configuration, your app will also have to ask the user to allow using the camera:

To ask for the correct permissions we use the permission_handler package.

Add the following snippet to your Podfile:

post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)

    target.build_configurations.each do |config|
      config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
        '$(inherited)',    

        ## dart: PermissionGroup.camera
        'PERMISSION_CAMERA=1',
      ]
    end  
  end
end

Then in your code you can request the camera permission:

var status = await Permission.camera.status;
if (status.isDenied) {
    PermissionStatus requestResult
    requestResult = await Permission.camera.request()
    
    if (requestResult.isGranted) {
        // Request has been granted
    }
}

For more details, you can take a look at the permission_handler documentation

import {request, PERMISSIONS} from 'react-native-permissions';  

const [camera, setCamera] = useState(false);

useEffect(() => {
    if (Platform.OS === 'ios') {
      request(PERMISSIONS.IOS.CAMERA).then(result => {
        setCamera(result === 'granted');
      });
    } else {
      request(PERMISSIONS.ANDROID.CAMERA).then(result => {
        setCamera(result === 'granted');
      });
    }
  }, []);

As you can see in the snippet, we used the react-native-permissions library. The installation guide can be found in the readme of the library's repository.

The following snippet from the Android documentation on requesting runtime permissions shows how to request the CAMERA permission:

if (ContextCompat.checkSelfPermission(
        context, Manifest.permission.CAMERA) ==
        PackageManager.PERMISSION_GRANTED) {
    // You can use the API that requires the permission.
    performAction(...);
} else if (shouldShowRequestPermissionRationale(...)) {
    // In an educational UI, explain to the user why your app requires this
    // permission for a specific feature to behave as expected, and what
    // features are disabled if it's declined. In this UI, include a
    // "cancel" or "no thanks" button that lets the user continue
    // using your app without granting the permission.
    showInContextUI(...);
} else {
    // You can directly ask for the permission.
    // The registered ActivityResultCallback gets the result of this request.
    requestPermissionLauncher.launch(
            Manifest.permission.CAMERA);
}

The following snippet has been extracted from the Apple Documentation and shows how to check if the user has given permission to use the camera:

var isAuthorized: Bool {
    get async {
        let status = AVCaptureDevice.authorizationStatus(for: .video)
        
        // Determine if the user previously authorized camera access.
        var isAuthorized = status == .authorized
        
        // If the system hasn't determined the user's authorization status,
        // explicitly prompt them for approval.
        if status == .notDetermined {
            isAuthorized = await AVCaptureDevice.requestAccess(for: .video)
        }
        
        return isAuthorized
    }
}

func setUpCaptureSession() async {
    guard await isAuthorized else { return }
    // Set up the capture session.
}

Install the Camera SDK

In your project, you can add the package below to the pubspec.yaml file. Replace {TOKEN} with the personal access token you've received from FibriCheck.

   flutter_fibricheck_sdk:
     git:
       url: https://{TOKEN}@github.com/fibricheck/flutter-camera-sdk
       ref: v1.0.0

To install the Camera SDK, you will need to have access to the Camera SDK git repository.

In your project, if you are using yarn or npm you need to create a file called .npmrc at the root level of your project and add these lines. Replace {AUTH_TOKEN} with the personal access token you've received from FibriCheck

@fibricheck:registry=https://npm.pkg.github.com
@extrahorizon:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${AUTH_TOKEN}

Alternatively, this file can be added/edited in your home directory and it will be applied to all projects.

Explanation from GitHub on how to add your token can be found here.

using npm:

npm install @fibricheck/react-native-camera-sdk

using yarn:

yarn add @fibricheck/react-native-camera-sdk

To install the Camera SDK for Cordova, you will receive an authentication token to access the private npm package hosted on GitHub Packages.

To include the cordova-camera-sdk package in your application, follow these steps:

Add a .npmrc file to the root of your project with the following content:

@fibricheck:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${AUTH_TOKEN}

Replace ${AUTH_TOKEN} with a valid authentication token. Next, add the package to your project using npm or yarn

npm install --save @fibricheck/cordova-fibricheck-camera-sdk 
// OR
yarn install --save @fibricheck/cordova-fibricheck-camera-sdk

After completing the above steps, the FibriCheck Camera SDK is available in your code in the cordova.plugins object:

const { FibriCheckCordovaSDK: fc } = cordova.plugins;

For iOS, the minimal deployment target is iOS 13. Make sure to add the following configuration to the config.xml, where the value must be >=13:

<platform name="ios">
    <preference name="deployment-target" value="13.0" />
</platform>

Perform your first measurement

In this paragraph, we explain how you can perform a measurement

The Camera SDK provides a widget that has the following structure:

    FibriCheckView(
        fibriCheckViewProperties: 
            FibriCheckViewProperties(
                flashEnabled: true,
                lineThickness: 4,
                ...,
                ),
        onCalibrationReady: () => debugPrint("Flutter onCalibrationReady"),
        onFingerDetected: () => debugPrint("Flutter onFingerDetected"),
        onFingerDetectionTimeExpired: () => debugPrint("Flutter onFingerDetectionTimeExpired"),
        onFingerRemoved: () => debugPrint("Flutter onFingerRemoved"),
        onHeartBeat: (heartbeat) => debugPrint("Flutter onHeartBeat $heartbeat"),
        onMeasurementFinished: () => debugPrint("Flutter onMeasurementFinished"),
        onMeasurementProcessed: (measurement) => debugPrint("Flutter onMeasurementProcessed $measurement"),
        onMeasurementStart: () => debugPrint("Flutter onMeasurementStart"),
        onMovementDetected: () => debugPrint("Flutter onMovementDetected"),
        onPulseDetected: () => debugPrint("Flutter onPulseDetected"),
        onPulseDetectionTimeExpired: () => debugPrint("Flutter onPulseDetectionTimeExpired"),
        onSampleReady: (ppg, raw) => debugPrint("Flutter onSampleReady $ppg $raw"),
        onTimeRemaining: (seconds) => debugPrint("Flutter onTimeRemaining $seconds"),
    ),

You can use the FibriCheckView exported from the package:camera_sdk/fibri_check_view.dart package to perform a measurement and hook up sdk.postMeasurement to post the data returned from the camera to the backend in the onMeasurementProcessed event.

Before taking a measurement, you need to check if you are entitled to perform a measurement. This can be achieved by invoking sdk.canPerformMeasurement. If you try to execute a measurement when you are not entitled, a NoActivePrescriptionError will be thrown. So make sure you've Activated a Prescription.
It is highly recommended to provide the camera SDK version as a second argument, as shown in the example.

import 'package:camera_sdk/fibri_check_view.dart';
import 'dart:convert';

import 'package:camera_sdk/fibri_check_view.dart';
import 'package:flutter/material.dart';
import 'package:flutter_fibricheck_sdk/flutter_fibricheck_sdk.dart';
import 'package:flutter_fibricheck_sdk/measurement.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:wakelock/wakelock.dart';

import '../0_design_system/fc_colors.dart';
import 'widgets/fc_metrics.dart';
import 'widgets/fc_title.dart';

class CameraScreen extends StatefulWidget {
  const CameraScreen({super.key, required this.sdk});

  final FLFibriCheckSdk sdk;

  @override
  State<CameraScreen> createState() => _CameraScreenState();
}

class _CameraScreenState extends State<CameraScreen> {
  Future<void>? _requestCameraPermission;

  bool _hasCameraPermission = false;
  bool _measurementFinished = false;

  String _timeRemaining = "-";
  String _heartBeat = "-";
  String _status = "Place your finger on the camera";
  Measurement? _measurement;

  @override
  initState() {
    super.initState();
    _requestCameraPermission = _requestCameraPermissionImpl();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        backgroundColor: FCColors.brokenWhite,
        appBar: AppBar(
          backgroundColor: FCColors.green,
          title: const Text('Camera'),
        ),
        body: Column(
          children: [
            Expanded(
              child: FutureBuilder(
                future: _requestCameraPermission,
                builder: (context, snapshot) {
                  if (snapshot.connectionState == ConnectionState.waiting) {
                    return const DemoTitleWidget(title: "Requiring camera permission");
                  }

                  if (!_hasCameraPermission) {
                    return const DemoTitleWidget(title: "Camera permission not granted");
                  }
                  return Column(
                    children: [
                      DemoTitleWidget(title: _status),
                      Container(
                        decoration: const BoxDecoration(
                          border: Border(
                            top: BorderSide(color: FCColors.lightGray, width: 1),
                            bottom: BorderSide(color: FCColors.lightGray, width: 1),
                          ),
                        ),
                        height: 200,
                        child: FibriCheckView(
                          fibriCheckViewProperties: FibriCheckViewProperties(
                            flashEnabled: true,
                            lineThickness: 4,
                          ),
                          onCalibrationReady: () => {
                            debugPrint("Flutter onCalibrationReady"),
                            setState(() {
                              _status = "Recording heartbeat...";
                            }),
                          },
                          onFingerDetected: () => {
                            Wakelock.enable(),
                            debugPrint("Flutter onFingerDetected"),
                            setState(() {
                              _status = "Detecting pulse...";
                            }),
                          },
                          onFingerDetectionTimeExpired: () => debugPrint("Flutter onFingerDetectionTimeExpired"),
                          onFingerRemoved: () => {
                            Wakelock.disable(),
                            debugPrint("Flutter onFingerRemoved"),
                          },
                          onHeartBeat: (heartbeat) => {
                            debugPrint("Flutter onHeartBeat $heartbeat"),
                            setState(() {
                              _heartBeat = heartbeat.toString();
                            }),
                          },
                          onMeasurementFinished: () => {
                            debugPrint("Flutter onMeasurementFinished"),
                            setState(() {
                              _status = "Measurement finished!";
                            }),
                          },
                          onMeasurementProcessed: (measurement) async => {
                            await _onMeasurementFinished(measurement),
                            debugPrint("Flutter onMeasurementProcessed $measurement"),
                            if (Navigator.canPop(context)) Navigator.pop(context),
                          },
                          onPulseDetected: () => {
                            debugPrint("Flutter onPulseDetected"),
                            setState(() {
                              _status = "Calibrating...";
                            }),
                          },
                          onTimeRemaining: (seconds) => {
                            debugPrint("Flutter onTimeRemaining $seconds"),
                            setState(() {
                              _timeRemaining = seconds.toString();
                            }),
                          },
                        ),
                      ),
                      DemoMetricsWidget(timeRemaining: _timeRemaining, heartBeat: _heartBeat),
                    ],
                  );
                },
              ),
            ),
          ],
        ),
      ),
    );
  }

  Future<void> _requestCameraPermissionImpl() async {
    var result = await Permission.camera.request();
    setState(() {
      _hasCameraPermission = result.isGranted;
    });
  }

  Future _onMeasurementFinished(String measurementString) async {
    var mCreationData = MeasurementCreationData.fromCameraSdk(measurementString);
    await widget.sdk.postMeasurement(mCreationData, "v0.0.1");
  }
}

The Camera SDK provides a React Native Component that you can integrate quickly in your application.

<RNFibriCheckView
    style={{ flex: 1, backgroundColor: '#ffffff' }}
    onFingerDetected={() => console.log('finger detected')}
    onFingerRemoved={() => console.log('finger removed')}
    onCalibrationReady={() => console.log('calibration ready')}
    onMeasurementFinished={() => console.log('measurement finished')}
    onMeasurementStart={() => console.log('measurement recording started')}
    onFingerDetectionTimeExpired={() =>
        console.log('finger detection time expired')
    }
    onPulseDetected={() => console.log('pulse detected')}
    onPulseDetectionTimeExpired={() =>
        console.log('pulse detection time is expired')
    }
    onMovementDetected={() => console.log('movement detected')}
    onHeartBeat={(heartRate) => console.log(`current heart rate: ${heartRate}`)}
    onTimeRemaining={(seconds) => console.log(`time remaining: ${seconds}`)}
    onMeasurementError={(error) => console.log(`measurement error occured: ${error}`)}
    onMeasurementProcessed={(data) =>
        console.log('measurement processed and ready to send!');
    }
/>

You can use the RNFibriCheckView exported from the @fibricheck/react-native-camera-sdk package to perform a measurement and hook up sdk.postMeasurement to post the data returned from the camera to the backend in the onMeasurementProcessed event.

Before taking a measurement, you need to check if you are entitled to perform a measurement. This can be achieved by invoking sdk.canPerformMeasurement. If you try to execute a measurement when you are not entitled, a NoActivePrescriptionError will be thrown. So make sure you've Activated a Prescription.
It is highly recommended to provide the camera sdk version as a second argument, as shown in the example.
The measurement context is prefilled here, but needs to be completed by the user. The best practice is to add this at a later stage with sdk.updateMeasurementContext. More information about this can be found in the Measurement Structure.

import client from '@fibricheck/javascript-sdk';
import { RNFibriCheckView } from '@fibricheck/react-native-camera-sdk';
import { useEffect, useState } from 'react';

const sdk = client({
  consumerKey: '',
  consumerSecret: '',
});

await sdk.authenticate({
  token: '',
  tokenSecret: '',
});

const App = () => {
  const [cameraData, setCameraData] = useState();
  const [context, setContext] = useState({
    symptoms: [
      'lightheaded',
      'confused'
    ],
    activity: 'sleeping',
  });
  
  async function postMeasurement(cameraData, context) {
    const measurement = {
      ...cameraData,
      context
    };
    await sdk.postMeasurement(measurement, RNFibriCheckView.version); 
  }
  
  return (
    <RNFibriCheckView
      onHeartBeat={(heartRate) => {
        console.log('heartRate', heartRate);
      }}
      onTimeRemaining={(seconds) => {
        console.log('onTimeRemaining', seconds);
      }}
      onMeasurementProcessed={async (cameraData) => {
        console.log('onMeasurementProcessed', cameraData);
        setCameraData(cameraData);
      }}
      onFingerDetected={() => {
        console.log('finger detected');
      }}
    />
  );
};

The FibriChecker class is the main class that you have to use to implement FibriCheck in your application.

import com.qompium.fibrichecker.FibriChecker;
import com.qompium.fibrichecker.listeners.FibriListener;
import com.qompium.fibrichecker.measurement.MeasurementData;


public class MeasurementView extends LinearLayout {

    linearLayout = new LinearLayout(context);
    linearLayout.setLayoutParams(new LinearLayout.LayoutParams(LinearLayout.LayoutParams.MATCH_PARENT, LinearLayout.LayoutParams.MATCH_PARENT));
    linearLayout.setOrientation(LinearLayout.HORIZONTAL);


    fibriChecker = new FibriChecker.FibriBuilder(context.getCurrentActivity(), linearLayout).build();
    fibriChecker.setFibriListener(new FibriListener() {

      @Override public void onSampleReady(final double ppg, double raw) {
        // Invoked multiple times during the measurement
        // Contains data to update visualisations in your views
      }

      @Override public void onFingerDetected() {
        // Invoked when the finger is placed correctly
      }

      @Override public void onFingerRemoved(double y, double v, double stdDevY) {
        // Invoked when the finger is removed 
      }

      @Override public void onCalibrationReady() {
      }

      @Override public void onHeartBeat(int value) {
      }

      @Override public void timeRemaining(int seconds) {
      }

      @Override public void onMeasurementFinished() {
      }

      @Override public void onMeasurementStart() {
      }

      @Override public void onFingerDetectionTimeExpired() {
      }

      @Override public void onPulseDetected() {
      }

      @Override public void onPulseDetectionTimeExpired() {
      }

      @Override public void onMovementDetected() {
      }

      @Override public void onMeasurementProcessed(MeasurementData measurementData) {
      }

       @Override public void onMeasurementError(String message) {

      }
    });

    fibriChecker.start();

    return linearLayout;
}

The iOS Native SDK is written in Objective-C. Newer iOS applications are typically developed in Swift. It's possible to use Objective-C files and Swift files in the same project. See the Apple documentation for more info.

The following code snippet is a working example that will show the heart rate, calculated using the Native SDK, on the main screen:

import SwiftUI

@main
struct test_fc_appApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

struct ContentView: View {
    @State var heartRate : UInt = 0
    var body: some View {
        VStack {
            Image(systemName: "globe")
                .imageScale(.large)
                .foregroundColor(.accentColor)
            Text("HeartRate: " + String(heartRate))
            Button("Measurement", action: {
                let viewSelf = self
                
                DispatchQueue.global(qos: .userInitiated).async {
                    
                    var threadDone = false

                    func handleError(_: String?) -> Void {
                        print("There was a measurement error")
                        threadDone = true
                    }

                    func handleMeasurementStart() -> Void {
                        print("Measurement started")
                    }

                    func handleMeasurementFinished() -> Void {
                        print("Measurement finished")
                        threadDone = true
                    }

                    func handleHeartRate(hr: UInt) -> Void {
                        print("Received HeartRate " + String(hr))
                        DispatchQueue.main.async {
                            print("Update HeartRate")
                            viewSelf.heartRate = hr
                        }
                    }

                    func handleMovementDetected() -> Void {
                        print("Movement Detected")
                    }
                    
                    func handlePulseDetection() -> Void {
                        print("Pulse Detected")
                    }
                
                    let fc = FibriChecker()
                    
                    fc.onMeasurementError = handleError
                    fc.onMeasurementStart = handleMeasurementStart
                    fc.onMovementDetected = handleMovementDetected
                    fc.onMeasurementFinished = handleMeasurementFinished
                    fc.onHeartBeat = handleHeartRate
                    fc.onPulseDetected = handlePulseDetection

                    fc.startMeasurement()
                    
                    while !threadDone {
                        usleep(1)
                    }
                }     
            })
        }
        .padding()
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

const { FibriCheckCordovaSDK: fc } = cordova.plugins;

// Step 1: initialise the SDK
await fc.initialise();

// Set listeners for events (full list available in docs)
fc.onMeasurementStart(() => {
    console.log("Measurement Started")
});

fc.onSampleReady((ppg: number, raw: number) => {
    console.log("PPG Sample Received");
    // use ppg value to visualize a graph
});

fc.onHeartBeat((hr: number) => {
    console.log(`Heartbeat ${hr}`);
});

fc.onMeasurementProcessed((data) => {
    console.log("Measurement Processed");
    // data object should be sent to FibriCheck Cloud
});

await fc.startMeasurement();

In rare cases, it can occur that the motion sensors don't provide the correct data. Because the algorithm requires motion sensor data to be available, an onMeasurementError event will be thrown in this case. Look here for more information.

Device Requirements

A full FibriCheck measurement consists of an on-device data acquisition step and a cloud data analysis step, performed by an AI algorithm. The on-device algorithms extracts a raw measurement from the the camera feed.

The use of the SDK has no significant impact on storage or memory usage of the device. The frames from the camera feed are processed on-the-fly by memory-efficient algorithms.

During the measurement there will be a significant CPU usage caused by the on-the-fly processing of the camera feed. However these processing algorithms also power the FibriCheck application which is broadly available on low-end and high-end smartphones. In this respect, we don't expect any performance issues by using the Camera SDK's in your application.

The following table lists the required minimum mobile operating system versions, and minimum framework versions:

Platform

Minimum OS (and framework)

Android

Android KitKat (4.4) API Level 19

iOS

iOS 11 (Sept. 17)

Important Remarks

Camera selection

Modern phones have multiple cameras. The Camera SDK uses the default capture device that is able to record video content.

To guide the user in putting their finger on the correct camera, it's recommended to show the camera output as a "peephole" in the interface at the start of a measurement.

In order to aid the user in using the correct camera lens, you can provide a preview of the relevant camera via the following package: https://pub.dev/packages/camera.

The Camera SDK uses the default camera.

There is a library react-native-vision-camera that is able to select the correct lens. The following snippet provides an example of how this can be implemented:

import { Camera, useCameraDevices } from 'react-native-vision-camera';
import styled from 'styled-components/native';

const SCREEN_HEIGHT = Dimensions.get('window').height;
const SCREEN_WIDTH = Dimensions.get('window').width;

const radiusBasis = Number(
  Math.round(SCREEN_HEIGHT / (SCREEN_HEIGHT <= 800 ? 6 : 8)),
);

const CameraContainer = styled.View`
  flex: 1;
  align-items: center;
  justify-content: center;
  max-height: ${radiusBasis}px;
`;

const CameraContent = styled.View`
  overflow: hidden;
  border-radius: 100px;
`;

const devices = useCameraDevices('wide-angle-camera');
const device = devices.back;

export const FindYourLens = () => {
  return (
    <CameraContainer>
      <CameraContent>
        <Camera
            style={...}
            device={device}
            preset={'vga-640x480'}
        />
      </CameraContent>
    </CameraContainer>
  );
};

Make sure to catch all measurement errors

An ongoing measurement will stop when a measurement error occurs. Make sure that an onMeasurementError has been defined. See the onMeasurementError documentation for more information

Framework-specific remarks

Drawing on the JS Thread

When benchmarking the SDK, we noticed that drawing on the JS Thread while taking a measurement caused severe spikes in the processing power. This will results in a bad quality measurement. So when creating a visualisation, for example counting down the seconds that are left in a measurement, make sure you are not drawing on the JS Thread. Either make use of Native Driver or use React Reanimated. When using third party libraries for creating animations, make sure they also offload the drawing from the JS Thread.

Not using Hermes

When benchmarking the SDK, we noticed that Hermes also had a big impact on the performance of low-end devices. So we advice you to enable it if possible. Instructions can be found in their documentation.

Visualizing the ongoing measurement

The SDK emits an onSampleReady event on each processed frame. The event contains a filtered ppg value and a raw measurement value of the latest received video frame.

You can use these values to visualize the PPG graph to the user during the measurement. Depending on the lightning conditions, the variance of the ppg values can be very low during most part of the measurement (decimal values between -1 and +1). Make sure to apply appropriate scaling to the graph to correctly visualize the PPG measurement to the user. Avoid visualizing an apparently flat line.

PreviousChangelog NextEvents

Last updated 1 year ago