CN113907733A

CN113907733A - Bonaxi AI

Info

Publication number: CN113907733A
Application number: CN202010666177.0A
Authority: CN
Inventors: 江禺霏
Original assignee: Sichuan Ruilan Zhibo Technology Co ltd
Current assignee: Sichuan Ruilan Zhibo Technology Co ltd
Priority date: 2020-07-10
Filing date: 2020-07-10
Publication date: 2022-01-11

Abstract

The invention provides a non-contact Bonashi AI for measuring vital signs based on videos. The technique uses remote photoplethysmography (rPPG) for Heart Rate Variability (HRV) and psychological pressure measurements. It invokes a color camera or infrared camera (more than 8 bits) running on any terminal to perform valid signal extraction on the face of the detected person. After a series of analysis processes, measurement initialization is successful, a plurality of feature points are generated on the face of a detected person, remote photoplethysmography (rPPG) is used for drawing data through the scanned feature points, the face of the detected person is slightly changed, so that slight change of pulse is found, and a piece of data is generated to be displayed on a dynamic wave line. The heart rate value is measured once the peak value and the valley value are generated once each drawing, and the heart rate values detected by face recognition due to the fact that the face recognition is easily influenced by the outside (light, a shelter, and the distance) may be different, and more accurate heart rate values are generated through multiple times of measurement and correction.

Description

Bonaxi AI

Technical Field

The present invention relates generally to the field of artificial intelligence. More specifically, the present invention relates to a contactless, video-based bonari AI for vital sign measurements. The technique uses remote photoplethysmography (rPPG) for Heart Rate Variability (HRV) and psychological pressure measurements.

Background

At present, most people begin to pursue a healthy life style, whether the heart rate is normal or not can reflect the health condition of the human body in real time, the change of the heart rate of the people is often known, and the health state of the human body can be kept by adjusting diet, living habits, sports and the like.

The existing professional heart rate measuring equipment mainly comprises two types: the first type of heart rate measuring equipment generally consists of two parts, one part is a special heart rate measuring device and can be tightly bound to the chest, and the chest has few chances to touch and is stable; the other part is a watch which is worn on the hand, and the main function is to read the data of the heart rate measuring equipment in front of the chest. The equipment of second kind survey rhythm of heart is the bracelet of measuring the rhythm of heart, wears on the wrist, and the state of wearing at ordinary times can not be very tight for comfortable, when surveying the rhythm of heart, needs skin and stable are hugged closely to the window of surveying the rhythm of heart, needs the static ten seconds of people, and the rhythm of heart just can be measured to tight dial plate in other one hand.

Whether the first heart rate measuring equipment or the second heart rate measuring equipment is used, when the detecting equipment for contacting the detected person needs to be in direct contact with the skin and the first heart rate measuring equipment is used, the additional detecting equipment needs to be tightly tied to the chest. The second type of heart rate measuring equipment requires the hand to press the equipment tightly to perform the measurement. In the detection process, the detection equipment is often separated from the detected person because the detection equipment is not tightly bound or the human hand is loosened, so that the detection is interrupted, and the detection data is inaccurate. Therefore, the user experience is not good for any mode of detection equipment, each detection equipment needs to add extra cost or action in the detection process, and the detection data is prone to be inaccurate.

Measuring Heart Rate Variability (HRV) is currently widely used in the medical and psychological fields. The development of other fields of measurement of HRV is also driven by the increasing improvements of healthcare and fitness tracking equipment in everyday life. In most studies, HRV variables vary with changes in stress due to different causes and are directly associated with many diseases. We can measure HRV by a number of different devices, from complex high-precision multi-sensing HRV measurement devices (ECG) to simple single-sensing devices (e.g. PPG, smart watch, IR sensor) etc., almost all known HRV measurement devices need to be attached to the subject's body, and these contacts not only inconvenience the measurement to some extent but also easily cause discomfort to the subject. Therefore, a high-availability and reliable method of contactless measurement of HRV is lacking. From the current situation, it is necessary to carry corresponding devices to obtain HRV measurement data, and then transmit the data to a smart phone or a computer client in a wireless manner or a copy manner for analysis and processing, so that there are many inconveniences in implementing HRV measurement in daily life.

Disclosure of Invention

The invention aims to innovate the prior art and provides a method for carrying out biological feature recognition on the face of a user through video streaming, extracting effective signals and carrying out psychological stress and HRV measurement by utilizing autonomously developed remote photoplethysmography (rPPG). The method can effectively reduce extra cost and action required in the heart rate detection process, is simple in measurement mode, high-efficiency and reliable in measurement result, and can improve user experience.

From the accuracy of heart rate monitoring, more than 90% of the solutions on the market today are implemented based on OpenCV. Some face detection methods developed from OpenCV technology have accuracy lower than 90%, and are susceptible to light and have poor stability, which are common problems in application products in the market at present. The influence is more obvious when vibration and light change occur. The measurement accuracy is not high, the HRV cannot be accurately generated, and an accurate result cannot be provided. A breakthrough solution is provided for a multi-layer AI framework independently developed by Bonash AI. It comprises-linear algebra/matrix manipulation/decompression/factorization/machine learning/mean clustering/minimum angle regression/linear regression/perceptron/PCA/LDA/nearest neighbor search/support vector machine/basic nerve/network definition/back propagation/Bayesian network signal processing/filtering/FFT/denoising/convolution/independent component analysis/deep learning/simple auto encoder/convolution nerve/deep circulation network/Lstm/RBM/deep signal neural network, etc. Bonaxi AI builds the entire mathematical back-end without any open source or third party development, provides a solution for controlling the entire bottom layer of each algorithm and model, which can support the most common operating systems and processing architectures (CPU, GPU, DSP, ARM, FPGA).

The OpenCV is a cross-platform computer vision library issued based on BSD license (open source), and can run on Linux, Windows, Android, and Mac OS operating systems. OpenCV is written in C + + language, and its main interface is also C + + language, but a large amount of C language interfaces are still reserved.

From the viewpoint of the accuracy of face detection, bonito AI possesses signal analysis and image processing functions, and pre-processes the detection data of the face of the subject as a key part of the whole measurement process. The face detection precision of the bona xi AI is more than or equal to 95 percent, and the precision of other products is less than or equal to 90 percent. The biological feature face recognition precision of the bona xi AI is more than or equal to 98 percent, and the biological feature face recognition precision can be not limited under the same condition. Other products only support face recognition. Borna chii can use autonomously developed remote photoplethysmography (rPPG) to perform accurate measurements of psychological stress and HRV. The measurement precision error is less than or equal to 2 bpm.

Bonaxi AI can be applied to any smartphone/tablet/personal computer, does not need to touch the skin during measurement, and can be used by subjects of any age and skin color. It can be run off-line without the need for networking. Psychological stress testing for bona chi AI is based on Baevsky and us/european stress index leveling criteria (global acceptance). Which runs as part of the SDK, provides a clear pressure level display based on the same video analytics solution.

The bona xi AI utilizes a webcam, a color camera of a mobile phone or an infrared camera (more than 8 bits) running on any terminal to perform correct signal acquisition for the facial biometric identification of the subject. The method mainly comprises the following steps when a terminal camera is called for shooting: face detection, face tracking, motion compensation, illumination normalization (removal of external light sources), selection of skin regions of interest, resulting in a photoplethysmography signal (mild reflection intensity).

The biometric identification is directly based on the correlation of the heartbeat interval change with the physiological phenomenon. It was proposed by russian space research experts in 1950, and was first completed in space. In 1961, non-invasive measures of human partial function including measures of cortisol levels (mood swings, muscle problems, etc.), mental stress, weight estimates, heart failure, hypertension, distress, cognitive and memory performance, were performed by the autonomic system, demonstrating that HRV is associated with obesity, CVA sudden death risk, inflammatory markers, cognitive ability, insulin sensitivity, visceral fat, aerobic exercise levels, fatigue and overtraining cues.

The biometric identification mainly comprises: fingerprint identification, retina identification, iris identification, gait identification, vein identification, face identification and the like. Compared with other recognition methods, the face recognition has the characteristics of directness, friendliness and convenience, so that a user has no psychological barrier to the face recognition. The present invention preferably employs face recognition for authenticating unique features as well as personal authentication, special identification (a person portrays from a known few features), general identification, and the like.

Essentially, the data obtained by biometric identification is unclean, the data needs to be analyzed and processed by bona chi AI, the main steps are to detect abnormal values, reduce noise of signals and remove artifacts caused by rapid intensity changes (eliminate abnormal values) to generate an optimal signal approximate value, namely, reconstruct an Electrocardiogram (ECG) signal, and perform signal spectrum analysis by remote photoplethysmography (rPPG), which can be used for signal reconstruction of Heart Rate Variability (HRV).

The signal processing can improve the value, accuracy and stability of the final result of the data, making it a key tool in data preparation. Signal processing can eliminate outliers and unnecessary components, thereby creating a more compact, optimized data set. The signal noise reduction processing can only reserve effective data for modeling and analysis, and the analysis accuracy is improved.

If the initialization for starting measurement is successful (acquiring measurement data), a plurality of feature points are generated on the face, and the data is drawn through the scanned feature points. Remote photoplethysmography (rPPG) can find a slight change of pulse by a slight change of face and generate a piece of data to be displayed on a dynamic wave line according to a mark point of the face color of a detected person. The heart rate value is measured once the peak value and the valley value are generated once each drawing, and the heart rate values detected by face recognition due to the fact that the face recognition is easily influenced by the outside (light, a shelter, and the distance) may be different, and more accurate heart rate values are generated through multiple times of measurement and correction.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a block diagram illustrating a system for measuring psychological stress and HRV using Bonahsi AI in accordance with an exemplary embodiment of the present invention;

FIG. 2 is a schematic diagram of a method for pre-processing facial information of a subject using a mobile phone camera according to the present invention;

FIG. 3 is a schematic diagram of the present invention for analyzing and processing the acquired data (unclean);

FIG. 4 is a schematic structural diagram of a method for measuring mental stress and HRV using Bonahsi AI according to the present invention;

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The embodiment of the invention provides a method for measuring psychological pressure and Heart Rate Variability (HRV) based on mobile phone video stream, the selected intelligent mobile phone for measuring HRV is provided with a color camera or an infrared camera (more than 8 bits), the mobile phone not only can be used as a normal mobile phone, but also can realize non-contact psychological pressure and HRV measurement in daily life, and the trouble of carrying special measuring equipment is avoided.

To facilitate understanding of the present embodiment, the method for measuring Heart Rate Variability (HRV) and psychological stress using bonarih AI disclosed in the present embodiment will be described in detail first.

As shown in fig. 1, the bonsai AI calls a mobile phone video stream to perform biometric recognition on a subject, thereby obtaining facial information of the subject (see fig. 2). Image signal analysis and processing are performed by using an artificial intelligence technique of bona chi AI, and a more accurate data set is acquired (refer to fig. 3). The data set is analyzed and processed by using autonomously developed plethysmography (rPPG), so that the aim of measuring the psychological pressure and the Heart Rate Variability (HRV) is fulfilled.

As shown in fig. 2, a method for acquiring facial information of a subject by a mobile phone camera according to an embodiment of the present invention includes: the specific area of the face of the subject is selected as a measuring area, an application system generates an initial data set by data input/modeling, anomaly detection, classification, prediction, regression, statistical analysis, time sequence and computer vision, and the Bonashi AI can perform signal preprocessing according to the initial data set, and the steps comprise face recognition, face tracking, motion compensation, illumination compensation (external source removal) and selection of a face effective area. The bona-chi AI system performs data adjustment/signal processing/missing data removal/normative data/principal component analysis/independent component analysis/filtered data/verification data, and performs model building/data verification/parametric transformation/model updating after the processing is completed.

The detailed implementation process is that the code layer is only limited to the packaging of the bottom code calling specific algorithm in the so library, all logic codes are specifically posted later, and two so files at the bottom are specifically called in the bottom algorithm of the so files, so as to achieve the detection purpose. {

System.loadlibrary(“binahhrv”)

System.loadlibrary(“video_hr”)

}

Opening the camera initialization sdk successfully generates feature points on the face and stores them in a byte array for the underlying so file call and calculating the heart rate value

byte[]data＝newbyte[]{-34，-64，23，11，0，0，0，0，44，0，0，0，-60，6，0，0，0，0，111，116，115，83，-126，-85，-41，-120，-36，-36，87，25，30，-36，-101，28，-35，-105，-39，-101，92，89，-40，24，-38，-105，91，88，91，-39，-108，33，-37，-70，74，35，114，115，95，101，120，112，111，114，116，95，102，111，114，101，97，99，104，83，-122，-21，-5，0，0，0，121，24，0，0，92，0，0，0，51，8，-128，28，-60，-31，28，102，20，1，61，-120，67，56，-124，-61，-116，66，-128，7，121，120，7，115，-104，113，12，-26，0，15，-19，16，14，-12，-128，14，51，12，66，30，-62，-63，29，32，0，0，3，0，0，0，19，4，-63，-120，1，65，4，71，-128，0，0，0，0，0，0，0}

When facial occlusion renders the measurable points unable to satisfy the measurement, the measurement is prompted to be invalid and a msg value is returned.

Message msg＝new Message()；

Msg.obj＝bmp2；

Msg.what＝3；

mSDKResponeHandler.sendMessage(msg)

The accuracy of the detection result is affected by the external factors (light, face shielding, distance) in the detection process. The subject should select a place with good light to detect the heart rate and HRV. During detection, the test subject should keep the mobile phone still, the face is completely positioned in the detection area, the distance between the test subject and the mobile phone is kept at 0.3-0.5 m as the optimal distance, and the longest distance is not more than 1.5 m. The face can not be shielded or can not be accurately detected. (the subject should be removed and examined with glasses or other coverings).

Fig. 3 is a specific operation flow of signal analysis and processing, which can eliminate abnormal values and unnecessary components, thereby creating a more compact and optimized data set. The data denoising processing can only reserve effective data for modeling and analysis, and the analysis accuracy is improved. The processing operation of the signal processing includes, but is not limited to, filtering/fast fourier transform algorithm/hadamard/denoising/convolution/independent component analysis. The signal processing simplifies the data quantity and provides convenient data for machine learning and artificial intelligence. The method accelerates the data processing speed, obviously reduces false alarm, and compares the data with a medical oximeter and a Bonash remote photoplethysmography (PPG) in real time based on a data set and a use case, wherein the error is less than or equal to 2bpm in 1000+ testers, and the accuracy rate is kept between 80 and 99 percent.

Wherein the machine learning includes but is not limited to k-means clustering/minimum angle regression/linear regression/perceptron model/principal/element analysis/linear discriminant analysis/nearest neighbor algorithm/support vector machine/basic neural network/basic definition/back propagation artificial neural network/bayesian network, etc.

Image signal processing simplifies data input by using a mathematical operation method (ML algorithm) which has been verified and used for centuries. The specific steps of forming the training model by using the ML algorithm after the signal processing are to select a proper training algorithm/scene creation to create feature extraction/scene creation for a classifier/data segmentation required for parallel training preparation/activation. The numerical data set is converted into a 'waveform' after signal processing so that the data is embodied as a manifold image for analysis. The specific code layer is the function developed by the code layer (fig. 2) described above to determine the function to perform the signal analysis and processing.

Fig. 4 is a schematic diagram of a method for measuring psychological stress and HRV using bonarih AI according to an embodiment of the present invention. Bonaxi AI utilizes autonomously developed remote optical plethysmography (rPPG) techniques to analyze, score, and save data from the acquired dataset, including but not limited to large-scale formatted data/CSV files/digital medical images and communications/databases/images/videos/structured/unstructured/semi-structured data. All stage models are saved through scoring. And (4) repeating the above operations on the new test data, and finally presenting a test result on an application interface, wherein the test result is more accurate and credible if the analysis result is objected and can be repeatedly measured for 2-3 times. The specific algorithm calls so Algorithm save (ndk:: basic.) as follows

Operation code executed by processor

.text：00000000005C3060 STP X29，X30，[SP，#0x90+var_s0]

.text：00000000005C3084 STUR X8，[X29，#var_28]

.text：00000000005C3088 BL ._ZN2cv5utils5trace7details6RegionC2ERKNS3_21Locatio；CODE XREF：cv：：Algorithm：：save(std：：_ndk1：：basic_string<char，std：：_ndk1：：char_traits<char>，std：：_ndk1：：all ocator<char>>const&)+58↑j

The pseudo code is as follows

After a series of processing, when the detection result meets the condition, a final heart rate value is generated and transmitted back to the application layer

Claims

1. A bonari AI for measuring psychological stress and Heart Rate Variability (HRV) in a non-contact manner (based on video streaming), which is characterized in that a color camera running on any terminal is used for carrying out biological feature recognition on the face of a user and extracting effective signals, a complete data set is generated through serial analysis and processing, and the acquired data set is analyzed and processed by using independently developed remote photoplethysmography (rPPG) so as to achieve the purpose of measuring the psychological stress and the HRV.

2. The mental stress test of bona chi AI according to claim 1 is based on Baevsky and us/european stress index rating measures (global approval) that operate as part of SDK based on the same video analytics solution for clear stress level display.

3. The method of claim 1, wherein the bonsai AI is capable of being run off-line during the measurement without being networked.

4. The bonsai AI support color or infrared cameras (more than 8 bits) running on any terminal, including but not limited to smart phones, tablets, according to claim 1.

5. The bonarichiai of claim 1 invoking the end-point camera to perform biometric recognition and signal extraction and signal pre-processing of the subject by face recognition, face tracking, motion compensation, illumination compensation (external source removal), selection of valid regions of the face to produce photoplethysmography signals (mild reflex intensity).

6. The serial analysis process of acquired signals according to claim 5, characterized by the elimination of outliers and unnecessary components, thus creating a more compact and optimized data set, whose main steps are outlier detection, signal noise reduction and removal of artifacts due to rapid intensity changes (elimination of outliers).

7. According to claim 6, the measurement initialization is successful, a plurality of feature points are generated on the face, and the data is drawn through the scanned feature points.

8. Remote photoplethysmography (rPPG) can detect slight changes in pulse and generate heart rate values by slight facial changes based on labeled points of the subject's facial color.