CN114366061A

CN114366061A - Heart rate measuring method, computer program product, storage medium and electronic device

Info

Publication number: CN114366061A
Application number: CN202111672314.2A
Authority: CN
Inventors: 王立华
Original assignee: Beijing Kuangshi Technology Co Ltd; Beijing Megvii Technology Co Ltd
Current assignee: Beijing Kuangshi Technology Co Ltd; Beijing Megvii Technology Co Ltd
Priority date: 2021-12-31
Filing date: 2021-12-31
Publication date: 2022-04-19

Abstract

The application relates to the technical field of signal processing, and provides a heart rate measuring method, a computer program product, a storage medium and an electronic device. The heart rate measuring method comprises the following steps: acquiring a signal sequence generated according to the reflected light signal; determining a plurality of reference fragments in the signal sequence; calculating a correlation coefficient between each reference segment and each corresponding delay segment in the plurality of reference segments, and calculating a heart rate cycle corresponding to the reference segment according to the correlation coefficient to obtain a plurality of heart rate cycles; and calculating the heart rate corresponding to the signal sequence according to the plurality of heart rate cycles. The method has the advantages of strong noise resistance and high measurement precision.

Description

Heart rate measuring method, computer program product, storage medium and electronic device

Technical Field

The invention relates to the technical field of signal processing, in particular to a heart rate measuring method, a computer program product, a storage medium and electronic equipment.

Background

Common methods for measuring heart rate include electrocardiography (Electro Cardio Graph, abbreviated as ECG) and photoplethysmography (Photo pulse Graph, abbreviated as PPG). The ECG method detects an electrical signal of the heart by placing two biopotential electrodes to a specific portion of the human body, and further calculates the heart rate from the electrical signal. The ECG method has high reliability of measurement results, but generally has high equipment cost and complicated operation, and is not suitable for daily use. The PPG method uses a light source to emit measuring light to irradiate a human body part to be measured, and uses a photoelectric sensor to convert reflected light into an electric signal, and further calculates the heart rate through the electric signal. PPG methods can use some relatively simple devices (e.g. watch, bracelet) for heart rate measurement, but the measurement results are susceptible to noise interference, resulting in a significant drop in measurement accuracy.

Disclosure of Invention

An object of the embodiments of the present application is to provide a heart rate measuring method, a computer program product, a storage medium, and an electronic device, so as to improve the above technical problems.

In order to achieve the above purpose, the present application provides the following technical solutions:

in a first aspect, an embodiment of the present application provides a heart rate measurement method, including: acquiring a signal sequence generated according to the reflected light signal; the reflected light signal is a signal formed by reflecting a measuring light ray emitted by a light source by a human body part to be measured, the signal sequence is composed of a plurality of sampling points, and the signal value of each sampling point represents the brightness of the reflected light signal at the sampling time corresponding to the sampling point; determining a plurality of reference fragments in the signal sequence; each reference segment corresponds to a different time, and the reference segment is a sequence segment which takes the corresponding time as a starting time in the signal sequence; calculating a correlation coefficient between each reference segment and each corresponding delay segment in the plurality of reference segments, and calculating a heart rate cycle corresponding to the reference segment according to the correlation coefficient to obtain a plurality of heart rate cycles; each delay segment corresponds to a different delay amount, and the delay segment is a corresponding sequence segment in the signal sequence after delaying the reference segment according to the corresponding delay amount; and calculating the heart rate corresponding to the signal sequence according to the plurality of heart rate cycles.

The method belongs to one PPG method, the heart rate is measured by performing autocorrelation operation on a signal sequence, and the noise resistance of the method is obviously higher than that of the existing method because the influence of noise on a correlation coefficient is small, and the high measurement precision can be always kept. In addition, the heart rate in the method is calculated according to a plurality of heart rate cycles, the reliability is high, and even if a certain heart rate cycle is not accurately calculated, the final heart rate measurement result can be only influenced to a very limited extent.

In an implementation manner of the first aspect, the calculating, according to the correlation coefficient, a heart rate cycle corresponding to the reference segment includes: determining a maximum value of a first correlation coefficient sequence exceeding a first threshold value, and determining a delay amount corresponding to the maximum value as a heart rate period corresponding to the reference segment; the correlation coefficient sequence is formed by arranging the correlation coefficients according to the sequence that the corresponding delay amount is sequentially increased.

In the above implementation, the constraint "exceeding the first threshold" ensures that the selected maximum is a peak of the sequence of correlation coefficients, and the "first" ensures that the amount of delay for that maximum is one heart rate cycle rather than a plurality of heart rate cycles.

In one implementation of the first aspect, there is an overlap between the plurality of reference segments.

In the implementation manner, because the reference segments are allowed to overlap, more reference segments can be cut out from the signal sequence, so that more heart rate cycles can be calculated, and the reliability of the heart rate measurement result is further improved.

In one implementation form of the first aspect, the reference fragment has a total of l-c-s_max+1, each reference fragment corresponding to the interval [0, l-c-s_max]Wherein l is the length of the signal sequence and c is the length of the reference fragment, in the interval [0, l-c-s_max]The interval between any two adjacent time instants is 1.

In the above implementation, by applying the interval [0, l-c-s_max]The time (the interval between adjacent times is 1) is selected densely to calculate the correlation coefficient, so that the calculation result can comprehensively and accurately reflect the periodicity of the signal sequence, and the heart rate can be accurately calculated based on the correlation coefficient subsequentlyAnd (4) calculating.

In one implementation of the first aspect, the time-lapse fragment has s in common_max-s_min+1, each time-lapse segment corresponds to an interval [ s ]_min，s_max]A delay amount; wherein s is_minFor the expected minimum heart rate period, s_maxInterval [ s ] for expected maximum heart rate cycle_min，s_max]The interval between any two adjacent delay time amounts is 1.

In the above implementation, by within the interval s_min，s_max]The time delay amount (the interval of the adjacent time delay amounts is 1) is intensively selected to calculate the correlation coefficient, so that the calculation result can comprehensively and accurately reflect the periodicity of the signal sequence, and the heart rate can be accurately calculated based on the correlation coefficient.

In one implementation manner of the first aspect, the determining a plurality of reference segments in the signal sequence includes: determining a plurality of initial reference fragments in the signal sequence; each initial reference segment corresponds to a different time, and the initial reference segment is a sequence segment which takes the corresponding time as an initial time in the signal sequence; calculating the quality score of each initial reference fragment in the plurality of initial reference fragments to obtain a plurality of quality scores; wherein the quality score characterizes an amount of AC signal contained in the initial reference segment; if the total number of qualified quality scores in the quality scores exceeds a second threshold value, the initial reference segments are the reference segments, or the initial reference segments with qualified quality scores are determined to be the reference segments.

As can be seen from the principles of the PPG method, the heart rate cannot be measured effectively if there is not enough ac component in the PPG signal. In the implementation manner, the subsequent heart rate calculation is performed only when the total number of qualified quality scores exceeds the second threshold (indicating that the quality score of the whole signal sequence is high enough, or the alternating current component in the signal sequence is enough), so that the accuracy of the heart rate measurement result is improved, and the computation amount is saved.

In addition, on the premise that the overall quality of the signal sequence is qualified, all initial reference segments may be used as reference segments for subsequent correlation coefficient calculation, or only the initial reference segments with qualified quality scores may be used as reference segments for subsequent correlation coefficient calculation. The former is favorable for fully utilizing information contained in the signal sequence, and the latter screens sequence segments participating in correlation coefficient calculation through mass fraction, so that the accuracy of heart rate measurement results is improved, and the calculation amount is saved. In one implementation of the first aspect, calculating the quality score of the initial reference segment includes: fitting a straight line satisfying the following condition with a plurality of sampling points in the initial reference segment: the sum of deviation values between a plurality of sampling points in the initial reference segment and the straight line is a minimum value; determining the minimum value as a quality score of the initial reference segment.

In the above implementation, the straight line obtained by fitting may be regarded as a dc component in the initial reference segment, and the deviation amount between the sampling point and the straight line may be regarded as an ac component in the initial reference segment, so that the quality score of the initial reference segment is determined to be consistent with the definition of the quality score according to the sum of the deviation amounts.

In an implementation manner of the first aspect, the calculating a heart rate corresponding to the signal sequence according to the plurality of heart rate cycles includes: carrying out weighted summation on the multiple heart rate cycles to obtain an average heart rate cycle, and calculating the heart rate corresponding to the signal sequence according to the average heart rate cycle; or calculating the heart rate corresponding to each heart rate cycle to obtain a plurality of heart rates, and performing weighted summation on the heart rates to obtain the heart rate corresponding to the signal sequence.

In the implementation mode, the heart rate is calculated by means of weighted summation, and the reliability is high. In particular, if the weights are set reasonably, the influence of inaccurate calculation of individual heart rate cycles on the heart rate measurement result can be effectively weakened.

In one implementation of the first aspect, the weight for weighted summation for each heart rate cycle is determined based on at least one of the following three factors: correlation coefficients corresponding to the heart rate cycles; the heart rate cycles are sized in the plurality of heart rate cycles; the quality score of the reference segment corresponding to the heart rate cycle.

In the implementation mode, the weighting weight of the heart rate period is determined by multiple factors, so that not only is the flexibility of the weighting weight in calculation given, but also the weighting weight can be calculated more reasonably, and the accuracy of heart rate measurement is improved.

In one implementation manner of the first aspect, the acquiring a signal sequence generated from the reflected light signal includes: controlling an area serving as a light source on a display screen to emit the measuring light; acquiring an image sequence generated by an image sensor arranged below the display screen according to the reflected light signal; and determining the brightness of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image to obtain the signal sequence consisting of a plurality of sampling points.

In the implementation manner, the display screen on the terminal device (for example, a mobile phone) used by the user daily is used as the light source, so that the heart rate measurement does not depend on additional devices (for example, a watch and a bracelet), the cost of the heart rate measurement performed by the user is reduced, and the inconvenience caused by wearing the measurement device by the user is avoided. In addition, the above implementation mode also uses the brightness of the image as the signal value of the sampling point, and simply and efficiently realizes the conversion from the two-dimensional image signal to the one-dimensional signal sequence.

In one implementation manner of the first aspect, the determining the brightness of each frame of image in the image sequence as the signal value of the corresponding sampling point of the frame of image includes: and determining the brightness of the central part of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image.

If the area as the light source is located directly above the image sensor, the brightness of the image generated by the image sensor gradually decreases from the center to the periphery of the image, and in order to improve the accuracy of heart rate measurement, the higher the brightness of the image, the better (within a reasonable range), so that only the brightness of the center portion of the image is taken as the signal value of the sampling point, which is advantageous for more accurately calculating the heart rate.

In a second aspect, an embodiment of the present application provides a heart rate measuring apparatus, including: the signal sequence acquisition module is used for acquiring a signal sequence generated according to the reflected light signal; the reflected light signal is a signal formed by reflecting a measuring light ray emitted by a light source by a human body part to be measured, the signal sequence is composed of a plurality of sampling points, and the signal value of each sampling point represents the brightness of the reflected light signal at the sampling time corresponding to the sampling point; a sequence segment determination module for determining a plurality of reference segments in the signal sequence; each reference segment corresponds to a different time, and the reference segment is a sequence segment which takes the corresponding time as a starting time in the signal sequence; the heart rate cycle calculation module is used for calculating correlation coefficients between the reference segments and the corresponding delay segments according to each reference segment in the reference segments, calculating heart rate cycles corresponding to the reference segments according to the correlation coefficients, and obtaining a plurality of heart rate cycles in total; each delay segment corresponds to a different delay amount, and the delay segment is a corresponding sequence segment in the signal sequence after delaying the reference segment according to the corresponding delay amount; and the heart rate calculation module is used for calculating the heart rate corresponding to the signal sequence according to the plurality of heart rate cycles.

In a third aspect, an embodiment of the present application provides a computer program product, which includes computer program instructions, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the method provided in the first aspect or any one of the possible implementation manners of the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the method provided by the first aspect or any one of the possible implementation manners of the first aspect.

In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory in which computer program instructions are stored, and a processor, where the computer program instructions are read and executed by the processor to perform the method provided by the first aspect or any one of the possible implementation manners of the first aspect.

In one implementation manner of the fifth aspect, the apparatus further includes: the display screen and set up the image sensor under the display screen, the signal sequence is obtained to the treater through carrying out following step: controlling an area serving as a light source on the display screen to emit measuring light, and acquiring an image sequence generated by the image sensor according to the reflected light signal; and determining the brightness of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image to obtain the signal sequence consisting of a plurality of sampling points.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 illustrates a structure of an electronic device provided in an embodiment of the present application;

fig. 2 illustrates an operation principle of an electronic device provided by an embodiment of the present application;

fig. 3 shows a flow of a heart rate measurement method provided by an embodiment of the present application;

fig. 4(a) and 4(B) illustrate the calculation principle of the correlation coefficient matrix in the heart rate measurement method provided by the embodiment of the present application;

fig. 5 shows a structure of a heart rate measuring device provided by an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The terms "first," "second," and the like, are used solely to distinguish one entity or action from another entity or action without necessarily being construed as indicating or implying any actual such relationship or order between such entities or actions.

Fig. 1 shows a structure of an electronic device 100 provided in an embodiment of the present application. Referring to fig. 1, the electronic device 100 includes: a processor 110, a memory 120, a display 130, and an image sensor 140, interconnected and in communication with each other via a communication bus 150 and/or other form of connection mechanism (not shown).

Processor 110 generally refers to devices capable of performing data operations. For example, the Processor 110 may be a general-purpose Processor including a Central Processing Unit (CPU), a Micro Control Unit (MCU), a Network Processor (NP), or other conventional processors; the Processor may also be a dedicated Processor, including a Neural-Network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, and a discrete hardware component. Also, when there are a plurality of processors 110, some of them may be general-purpose processors, and the other may be special-purpose processors.

Memory 120 broadly refers to a device capable of data storage. For example, the Memory 120 may be a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an electrically Erasable Programmable Read-Only Memory (EEPROM), and the like.

Display screen 130 broadly refers to a device that can emit light for display of content. For example, the Display panel 130 may be an Organic Light-Emitting Diode (OLED) panel, a Liquid Crystal Display (LCD) panel, or the like.

The image sensor 140 broadly refers to a device capable of receiving an optical signal and converting the optical signal into an image signal. For example, the image sensor 140 may be a Complementary Metal Oxide Semiconductor (CMOS) sensor, a Charge Coupled Device (Charge Coupled Device) sensor, or the like.

The image sensor 140 is disposed under the display screen 130 and integrated inside the electronic device 100, for example, the image sensor 140 may be a part of an off-screen fingerprint identification module or an off-screen camera module. For the electronic device 100 carrying the underscreen fingerprint recognition module or the underscreen camera module, the image sensor 140 in these modules can be multiplexed when measuring the heart rate, so as to save the implementation cost. For example, to fingerprint identification module under the screen, both can carry out fingerprint identification after multiplexing, can carry out the heart rate measurement again.

Referring to fig. 2, the process of the electronic device 100 performing heart rate measurements is briefly described as follows:

first, under the control of the processor 110, a specified area on the display screen 130 emits a measurement light 170. The designated area is used as a light source in the heart rate measurement process, and is not referred to as a light source area 132, and the shape, size, and position of the light source area 132 are not strictly limited, so that the measurement is convenient.

For example, in fig. 2, a gray area on the display screen 130 represents the light source area 132, which is an oval or circular area having an area smaller than or equal to the area of a section of a human finger, and is positioned directly above the image sensor 140.

The measurement light 170 may be selected from monochromatic light, for example, where experiments have shown that green light has a superior effect on measuring heart rate. The display screen 130 is not limited to what is displayed in areas other than the light source area 132 during the measurement, but is preferably clearly distinguishable from the light source area 132. For example, when the light source region 132 displays green, other regions of the display screen 130 may display black, so that on one hand, interference with the light source region 132 may be reduced, and on the other hand, the position of the light source region 132 may also be highlighted on the display screen 130, so that the user can clearly know which portion of the display screen 130 should be touched when measuring the heart rate.

After the user sees the light source region 132 illuminated, the user can press the finger 160 against the light source region 132 to perform a heart rate measurement while keeping the finger 160 as motionless as possible during the measurement. It should be understood that other body parts may be used to make heart rate measurements, and that the finger 160 is merely exemplary. It should be noted that in fig. 2, the finger 160 does not contact the light source region 132, which is only for more clearly plotting the reflection process of the light, and the finger 160 can be close to the light source region 132 during the actual measurement.

The measuring light 170 is reflected by the finger 160 to form a reflected light signal 172, the reflected light signal 172 passes through the display screen 130 and impinges on the image sensor 140 under the screen, and the image sensor 140 converts the reflected light signal 172 into an electrical signal, i.e., an image signal. During heart rate measurement, the image sensor 140 continuously generates image signals forming a sequence of images.

The processor 110 acquires the image sequence from the image sensor 140 and performs further calculations based on the image sequence to obtain the heart rate to be measured, the calculation process being further described in the introduction to fig. 3.

For better measurement, the processor 110 can control the whole display screen 130 (including the light source region 132) to emit light as much as possible at the maximum brightness: for example, the maximum brightness here may be the real maximum brightness of the display screen 130 (when the display screen 130 is in the high brightness state), or may be the maximum brightness of the display screen 130 in the non-high brightness state. For simplicity, the term "true maximum brightness" is understood to mean the limit brightness that the display 130 can display, but the display 130 may be burned out due to the brightness displayed for a long time, so the display 130 usually operates in the non-highlight state, and the maximum brightness that the display 130 can display is the "maximum brightness in the non-highlight state".

In addition, whether the display screen 130 can display with maximum brightness is also related to the characteristics of the image sensor 140: for example, if the display screen 130 adopts the maximum brightness, the image sensor 140 is overexposed, which causes the quality of the generated image signal to be seriously degraded, and the heart rate measurement result will be affected, at this time, the brightness of the display screen 130 must be reduced until the image sensor 140 does not generate the overexposure. The brightness of the display screen 130 may be configured in advance, and the display screen 130 may be directly controlled to perform light emitting display according to the configured brightness during subsequent multiple measurements.

In addition, some display screens 130 support localized brightness adjustments (e.g., individual adjustments to the brightness of light source regions 132 rather than adjusting the brightness of the entire display screen 130), so the above brightness setting logic for display screen 130 may be performed only for light source regions 132, and the analysis is not repeated.

During heart rate measurement, processor 110 and other components may access memory 120, read and/or write data stored therein. In particular, computer program instructions may be stored in the memory 120 and read and executed by the processor 110 to perform the heart rate measurement method steps in the embodiments of the present application, such as the above-mentioned steps of controlling the light source region 132 to emit light, calculating the heart rate from a sequence of images, and the like.

As mentioned above, the current methods for measuring heart rate mainly fall into two categories, namely, ECG method and PPG method, and the above-mentioned method for measuring heart rate of the electronic device 100 belongs to PPG method (the reflected light signal 172 is a PPG signal), but the existing PPG method mostly needs to wear additional devices, such as a watch, a bracelet, and other wearable devices, and these devices are usually provided with LED lamps as light sources. On the one hand, the wearing process of these devices is cumbersome, and on the other hand, the special purchase of these devices also requires a modest cost. However, in the electronic device 100, the light source area 132 on the display 130 emits light, and the image sensor 140 under the display performs photoelectric conversion, so that the electronic device 100 can be completely implemented as a mobile terminal such as a mobile phone for daily use, that is, a user does not need to purchase additional devices specially for measuring the heart rate, and inconvenience caused by wearing the devices is avoided.

It will be appreciated that the configuration shown in FIG. 1 is merely illustrative and that electronic device 100 may include more or fewer components than shown in FIG. 1 or have a different configuration than shown in FIG. 1. And it should be understood that when the configuration of fig. 1 is changed, the step of measuring heart rate is adjusted accordingly, not necessarily in exact correspondence with the steps described above.

For example, the image sensor 140 in the electronic device 100 may be replaced by other types of photosensors, except that the outputs of these photosensors may not necessarily be a two-dimensional image sequence, but may also be a one-dimensional signal sequence, which does not affect the subsequent heart rate calculation, because the method in the embodiment of the present application finally converts the image sequence into a signal sequence for further processing, as will be understood from the following description. The image sensor 140 is employed in the electronic device 100, in order to multiplex the functions of fingerprint recognition or image capture under the screen (both of which would otherwise require the provision of the image sensor 140 under the screen).

Further, the electronic device 100 may not include the display 130, but may use other light sources, such as LED lamps. In other words, at this time, the electronic device 100 and the existing device for measuring the heart rate by using the PPG method are not significantly different in structure, but the way of calculating the heart rate by the processor 110 is different from the existing PPG method (explained later).

As another example, electronic device 100 may not include display 130 and image sensor 140, but only processor 110 and memory 120. For example, the device X includes the display screen 130 and the image sensor 140, but the device X does not process the image sequence locally at X after generating the image sequence, but sends the image sequence to the electronic device 100 through the network for heart rate calculation, and since the electronic device 100 is only responsible for calculating the heart rate according to the image sequence, the display screen 130 and the image sensor 140 are not necessary, and at this time, the electronic device 100 does not need to execute any logic for controlling the display screen 130 to emit light.

Further, the electronic device 100 may also comprise a communication unit (e.g. a network card) for data transmission via the network and the device X.

The electronic device 100 may be implemented as a cell phone, a tablet, a wearable device (e.g., watch, bracelet), a PC, a server, and so on. In different implementations, the electronic device 100 may contain different components: for example, when implemented as a mobile phone, the processor 110, the memory 120, the display 130, and the image sensor 140 may be included (the hardware structure and the process of calculating the heart rate are different from those of the conventional PPG method); when implemented as a watch, the watch may include components such as a processor 110, a memory 120, LED lights, and a photosensor (the hardware structure is similar to the existing PPG method, but the process of calculating the heart rate is different); implemented as a PC, may include components such as a processor 110 and a memory 120 (the PC receives or reads a sequence of images from the memory 120 and calculates a heart rate from the sequence of images).

It should be further noted that although it is mentioned above that the electronic device 100, when implemented as a mobile phone, can save the trouble of wearing the device and save the cost of measuring the heart rate, this does not mean that the electronic device 100 cannot be implemented as a wearable device, because objectively some users really have a need to use the wearable device (for example, some users prefer to watch with a watch for time, and some users prefer to record sports data with a bracelet), the cost of wearing and purchasing the wearable device is acceptable for these users, and the heart rate measurement method in the embodiment of the present application can still improve the heart rate measurement result when applied to the wearable device.

Further, the electronic device 100 is not limited to a single device, and may be a combination of a plurality of devices or a cluster formed by a large number of devices.

Fig. 3 illustrates a flow chart of a heart rate measurement method provided by an embodiment of the present application, which may be executed by the processor 110 in the electronic device 100 (and its possible changes) illustrated in fig. 1. Referring to fig. 3, the method includes:

step S210: a signal sequence generated from the reflected light signal is acquired.

The signal sequence is generated by a reflected light signal, and the reflected light signal is a signal formed by reflecting the measuring light emitted by the light source by the human body part to be measured. The signal sequence may be in the form of a digital signal, and the signal sequence is composed of a plurality of samples, and the coordinate of each sample may be denoted as (x, y), where the x coordinate represents the sampling time of the sample, the y coordinate represents the signal value (or amplitude) of the sample, the signal value represents the brightness of the reflected light signal at the x time, and the difference between the x coordinates of each two adjacent samples is 1, which represents a sampling time interval. The length of the signal sequence may be defined as the number of samples included in the signal sequence, and the waveform of the signal sequence may be defined as the signal value fluctuation pattern of the samples included in the signal sequence.

In some implementations, the signal sequence may be converted from an image sequence. For example, referring to fig. 2, the measurement light 170 emitted from the light source region 132 is reflected by the finger 160 to form a reflected light signal 172, and the reflected light signal 172 is converted into an image signal by the image sensor 140 under the screen. After the finger 160 presses the light source region 132 for a period of time, the image signal generated by the image sensor 140 will form an image sequence of multiple frames, and the brightness of each frame of image in the image sequence is calculated as the signal value of the sample point corresponding to each frame of image, so as to obtain the signal sequence in step S210 (the sequence of sample points is consistent with the sequence of images in the image sequence). In this process, since the information in each frame of image is mapped to a single signal value, the transition from a two-dimensional image sequence to a one-dimensional signal sequence is simply and efficiently achieved.

For a frame of image, its brightness may be defined as the average of the brightness of all or some of the pixels in the image. For example, it may be the average of the brightness of each pixel in the whole image; also for example, it may be a luminance average value of each pixel of a central portion of the image (which may refer to a rectangular area, a circular area, etc. located at a central position of the image), and so on.

The latter definition will be briefly explained: if the light source region 132 is located directly above the image sensor 140, the brightness of the image generated by the image sensor 140 gradually decreases from the center to the periphery of the image, and in order to improve the accuracy of heart rate measurement, the higher the brightness of the image, the better (within a reasonable range), so that only the brightness of the center portion of the image is taken as the signal value of the sampling point, which is beneficial to more accurately calculating the heart rate.

Of course, the image sequence is not necessarily obtained directly from the image sensor 140, and for example, it is also possible that other devices transmit to the current device (referring to the device that is performing step S210) or that the current device reads directly from the local storage space.

In some implementations, the signal sequence can be derived directly based on the reflected light signal without first generating an image sequence. For example, the image sensor 140 in fig. 2 is replaced by other types of photosensors, which may directly or indirectly (requiring analog-to-digital conversion) output a sequence of signals, the signal value of each sample in the sequence representing the brightness of the reflected light signal at the sampling instant.

Of course, the signal sequence is not necessarily obtained directly from the photosensor, for example, it is also possible that other devices transmit to the current device, or that the current device reads directly from the local storage space.

Further, after the signal sequence is obtained, it may be subjected to appropriate pre-processing, and then step S220 is performed. The preprocessing operation may include one or more of filtering, interpolation. Wherein, the filtering can remove some noises in the signal sequence, and the interpolation is mainly used for weakening the influence on the signal sequence caused by uneven intervals between image frames.

Step S220: a plurality of reference fragments in the signal sequence is determined.

Step S230: and calculating a correlation coefficient between the reference segment and the corresponding multiple delay segments aiming at each reference segment in the multiple reference segments, and calculating a heart rate cycle corresponding to the reference segment according to the correlation coefficient to obtain multiple heart rate cycles.

The above two steps are described together. Each reference segment corresponds to a different time instant, and a reference segment may be defined as a sequence segment in the signal sequence starting from its corresponding time instant. Each delay segment corresponds to a different delay amount, and a delay segment can be defined as a corresponding sequence segment in the signal sequence after the reference segment is delayed according to the corresponding delay amount.

In the definitions of reference fragments and delay fragments, the concept of sequence fragments is mentioned, which can be understood as a segment in a signal sequence, each sequence fragment comprising a plurality of consecutive samples in the signal sequence. The length of the sequence segment can be defined as the number of samples contained in the sequence segment, and as can be seen from the above definition, the reference segment and its corresponding delay segment are of equal length. The waveform of a sequence segment can be defined as the signal value fluctuation pattern of the samples contained in the sequence segment.

The definitions of the reference and delay segments can be understood with reference to fig. 4(a), where the white boxes represent the signal sequence, and the length is L, without designating the signal sequence as L. The black boxes indicate the sequence segments in the signal sequence starting at t, and having a length c, which is not denoted as X, and it is obvious that X ═ L [ t: t + c-1], X is a reference segment, and X corresponds to t. The grey boxes indicate the sequence segments in the signal sequence starting at t + s, the length c, which is not denoted as Y, and it is clear that Y is L [ t + s: t + s + c-1], Y is a delay segment corresponding to X, and Y corresponds to s. When the delay amount s takes some value, there may also be parts of X and Y that overlap, i.e. X and Y comprise common samples. In particular, s ≠ 0, since X and Y are the same sequence fragment when s ≠ 0.

The delay segment Y can also be understood as follows: a sequence segment with the length of c and the start time of t is intercepted from L as a reference segment X, and after X is delayed by s (right shift by s), the sequence segment is aligned with a sequence segment with the length of c and the start time of t + s in L (two sequence segments are aligned, that is, the sampling points included in the two sequence segments correspond to each other one by one according to the sequence in the sequence segments), and the sequence segment is a delay segment Y corresponding to X.

Following the notation in fig. 4(a), a correlation coefficient, which is a quantized representation of the correlation between the two, can be calculated for any one of the delay segments Y corresponding to the reference segments X and X. The correlation coefficient may be defined in various ways, such as pearson correlation coefficient, cosine similarity, modified cosine similarity, etc. Taking the pearson correlation coefficient as an example, the calculation formula is:

where cov (X, Y) represents the covariance of X and Y, σ (X) represents the standard deviation of X, and σ (Y) represents the standard deviation of Y. Eta_X,YDenotes the Pearson correlation coefficient, η, between X and Y_X,YIs in the range of [ -1,1 [)]，η_X,Y> 0 denotes a positive correlation of X and Y,. eta_X,YWhen 1 is the strongest, η_X,Y< 0 indicates that X and Y are inversely related, η_X,YThe correlation is strongest when the correlation is-1. Considering only the case of positive correlation, η_X,YThe larger the correlation between X and Y, the more similar the waveform.

Further, since X and Y are both sequence segments in L, i.e., η_X,YIs represented by the correlation between one part of L and another, so η is calculated_X,YBelongs to the autocorrelation operation on L, and η_X,YAlso reflected is the autocorrelation properties of L.

If L is a periodic sequence, the waveform of L in each period is the same, in other words, a reference segment X is cut from L, and delayed to obtain a corresponding delayed segment Y in L, and if the delay amount is closer to the period of L, the waveforms between X and Y are more similar, and the calculated correlation coefficient is larger. Although L may not be a strict periodic sequence in practice, L may still be analyzed following a periodic sequence due to the periodic heartbeat signal contained therein.

The above properties of L provide a theoretical basis for measuring heart rate: trying to find a delay amount, if it can maximize the correlation coefficient calculated between the reference segment X and the delay segment Y, the delay amount can be regarded as the period of the heartbeat signal, referred to as the heart rate period. The heart rate period can be directly converted into the corresponding heart rate (unit: times/minute), and the conversion relation is

Where 60 denotes the number of seconds contained in 1 minute, s denotes the amount of delay, and Δ T denotes the sampling interval of L, and is multiplied by Δ T because s is a value other than time in the digital signal.

To find the delay amount corresponding to the heart rate cycle, a group of candidate delay amounts may be selected, and then correlation coefficients between X and Y obtained under these delay amounts are calculated one by one, and a maximum value among these delay amounts is found, where the delay amount corresponding to the maximum value is the heart rate cycle. The "delay amounts of a batch of candidates" is the delay amount of the "delay pieces" in step S230.

For example, the interval [ s ] may be selected_min，s_max]Total of s in_max-s_min+1 different delay amounts s_min，s_min+1，…，s_max(interval 1 between any two adjacent delay amounts) as the above-mentioned "delay amounts of a batch of candidates". Wherein s is_minIs the minimum of these delay amounts, s_maxIs the maximum of these delay amounts, s_minRepresenting the expected minimum heart rate period, and can be calculated from the expected maximum heart rate, s_maxRepresenting the expected maximum heart rate period, may be calculated from the expected minimum heart rate. For example, a heart rate range of a person (or a heart rate range desired to be measured) is predictedEnclose) is 40-200 times/min, then

Referring to fig. 4(B), the black boxes represent the reference segments X, and the gray boxes represent the distribution range of the delay segments Y corresponding to X in L: if s ═ s_minY is located at the far left of the gray box, if s ═ s_maxThen the position of Y is to the far left of the gray box, and if s takes other values, then the position of Y is in the middle of the gray box. For each position Y, a correlation coefficient can be calculated with X, i.e. s can be calculated_max-s_min+1 correlation coefficients, finding out the maximum value from these correlation coefficients, the delay s corresponding to the maximum value_pIs the heart rate cycle.

However, the heart rate cycle so found is not necessarily correct, because s is as described above_max-s_minThere may be multiple peaks in the +1 correlation coefficients, and the maximum value necessarily corresponds to one of the peaks, but the peak is not necessarily a heart rate cycle, and may be an integer multiple of the heart rate cycle. For example, if the user's true heart rate period happens to be s_minInterval [ s ]_min，s_max]May include s_min、2s_min、3s_minIn terms of the autocorrelation characteristics of the periodic sequence, the correlation coefficient calculated when s takes these values is large, and is not necessarily s ═ s_minMaximum, also possible when s-2 s_minIs maximum.

Thus, in some implementations, the heart rate cycle may also be calculated using the following alternatives: for the above s_max-s_minAnd the +1 correlation coefficients are sorted according to the sequence that the corresponding delay amount is sequentially increased, the first maximum value exceeding the first threshold value is taken from the correlation coefficient sequence formed after the sorting, and then the delay amount corresponding to the maximum value can be used for determining the heart rate period. Here, the constraint "exceeding a first threshold" ensures that the selected maximum is a peak in the sequence of correlation coefficients, and "first" ensures that the amount of delay for that maximum is one heart rate cycle rather than a plurality of heart rate cyclesAnd (4) period.

It should be noted that, if the correlation coefficients are calculated by sequentially obtaining the corresponding delay segments and the reference segment according to the sequence in which the delay amounts sequentially increase, the calculated correlation coefficients are arranged according to the sequence in which the corresponding delay amounts sequentially increase, and no sorting is required. In contrast, if the correlation coefficients are not calculated, the corresponding delay segments and the reference segment are not sequentially obtained according to the sequence of sequentially increasing delay amounts to calculate the correlation coefficients, the calculated correlation coefficients are not arranged according to the sequence of sequentially increasing delay amounts, and thus the heart rate period can be further determined by sorting first.

There is a significant advantage to using the periodicity of L to perform autocorrelation operations to obtain the heart rate: in the case where noise is mixed in L, since the periodicity of L is not significantly changed by the noise, the calculated correlation coefficient is not greatly affected. In other words, even if the values of the correlation coefficients are influenced by noise to some extent, the magnitude relationship between the correlation coefficients is not easily influenced by the noise, so that the position of the maximum value in the correlation coefficients is not changed, that is, the heart rate calculation result is not influenced.

To accurately and seamlessly calculate the heart rate cycle, a dense selection method may be used to select the delay amount corresponding to the delay segment, such as the interval s in the previous example_min，s_max]Each delay amount in the delay line is selected. But this is not the only way to choose the amount of delay:

for example, s can be selected_min，s_min+2，…，s_maxAs the delay amount corresponding to the delay segment. In this way, the interval between adjacent delay amounts is 2, and the total number of the selected delay amounts is less, so that the total number of correspondingly obtained delay segments is also less, and the reduction of the calculation amount of the correlation coefficient is facilitated.

For another example, 1,2, …,200 may be selected as the delay amount corresponding to the delay segment. This approach does not deliberately follow the interval s_min，s_max]In the above-mentioned step (2), the time delay amount is selected,thus, s is not needed_minAnd s_maxThe estimation can be carried out without omitting special heart rate cycles (less than s)_minOr greater than s_maxHeart rate cycle).

Of course, the manner in which the heart rate period is calculated is similar regardless of the amount of delay chosen and will not be repeated. Once the amount of delay has been selected, the corresponding delay segment is determined, with the reference segment already determined.

The following problem of the length l of the signal sequence and the length c of the sequence fragment is explained: both values have no strict requirement, but l cannot be too short, and at least a plurality of heart rate cycles are included, so that the heart rate cycle can be presumed according to the autocorrelation operation. For example, if the signal sequence corresponds to a signal of duration 4s, it is sufficient to satisfy this requirement; as for the value of c, experiments show that too small or too large will adversely affect the heart rate calculation, so the sequence length corresponding to one heart rate cycle can be taken, and the heart rate cycle here can be a common heart rate cycle of ordinary people, for example, a heart rate cycle corresponding to any one of 60 to 90 heart rates.

In the above description of the method for calculating the heart rate cycle, the default reference segment is fixed, or only one reference segment is considered, so that a heart rate cycle corresponding to the reference segment can be calculated. However, there are a plurality of reference segments determined in step S220, and for each reference segment, a corresponding heart rate cycle can be calculated according to the method described above, so that a plurality of heart rate cycles are obtained in step S230.

The plurality of reference segments determined in step S220 may or may not have overlap. For example, sequence segments L [1:50], L [51:100], L [101:150] are 3 reference segments without overlap; the sequence segments L [1:50], L [26:75] and L [51:100] are 3 reference segments with overlapping.

If only the sequence segments which do not overlap in the signal sequence are selected as the reference segments, the total number of the reference segments which can be extracted from the signal sequence is theoretically small, so that the total number of the heart rate cycles calculated in step S230 is relatively small, the reliability of the subsequently obtained heart rate measurement result is relatively low, and the calculation amount for calculating the correlation coefficient is favorably reduced.

If the overlapped sequence segments in the signal sequence are allowed to be selected as the reference segments, the total number of the reference segments that can be extracted from the signal sequence is theoretically large, so that the total number of the heart rate cycles calculated in S230 is relatively large, the reliability of the subsequently obtained heart rate measurement result is relatively high, but the calculation amount of the correlation coefficient is increased to a certain extent.

The following mainly describes the case where there is overlap between reference segments. Since each reference segment corresponds to a time instant, determining a time instant in the signal sequence is equivalent to determining a reference segment in the signal sequence starting at the time instant.

In order to accurately and uninterruptedly calculate the heart rate cycle, similar to the method of densely selecting the delay amount, the time corresponding to the reference segment may also adopt a densely selecting mode: for example, the interval [0, l-c-s_max]Internal l-c-s_max+1 different moments 0,1, …, l-c-s_max(interval 1 between any two adjacent time instants) as reference segment corresponding time instants, so that a total of l-c-s can be obtained_max+1 reference slices (step S220).

Where 0 denotes that the reference segment X (with length c) is truncated from the leftmost side of the signal sequence L, in order to ensure that the end time of the delay segment Y corresponding to X does not exceed the last time in L (i.e. the end of the gray box in fig. 4(B) cannot exceed the end of the white box), X cannot be truncated from the rightmost side of L, but at most from L-c-s_maxAt this time (when the end of the gray box in fig. 4(B) is aligned exactly with the end of the white box).

It should be understood that there are many different ways to select the time corresponding to the reference segment: for example, the interval between adjacent time instants does not have to be 1; also for example, it is not necessary to follow the interval [0, l-c-s ]_max]Selecting; as another example, the times in the signal sequence (between reference fragments obtained at this time) may be chosen randomlyThere may or may not be overlap in the overlap), and so forth.

In addition, the lengths of the plurality of reference segments determined in step S220 may or may not be identical. If the lengths of the two reference segments are different, the lengths of the delay segments corresponding to the two reference segments are also different. Even if the two reference segments have the same length, the delay amounts of the delay segments corresponding to the two reference segments do not have to be exactly the same. However, in the embodiment of the present application, for simplicity, the case where the lengths of all the reference segments are the same and the delay amounts of the delay segments corresponding to each reference are also completely the same is mainly taken as an example.

Further, in some implementations, a correlation coefficient matrix may also be defined. The correlation coefficient matrix includes two dimensions, which are respectively referred to as a first dimension and a second dimension, where the second dimension is a column of the matrix if the first dimension is a row of the matrix, and the second dimension is a row of the matrix if the first dimension is a column of the matrix.

The first dimension of the correlation coefficient matrix comprises a plurality of different time instants which can be arranged according to an increasing order, and each time instant corresponds to a reference segment; the second dimension of the correlation coefficient matrix comprises a plurality of different delay amounts, which may be arranged in an increasing order, each delay amount corresponding to a delay segment.

Without loss of generality, assuming that a certain time in the first dimension is t, a certain delay amount in the second dimension is s, and the correlation coefficient matrix is denoted as R, a value R (s, t) of the correlation coefficient matrix at (s, t) represents: correlation coefficient eta between reference segment X with t as starting time and time delay segment Y with t + s as starting time in signal sequence L_X,Y。

According to the definition of the correlation coefficient matrix, step S230 may also be regarded as a process of calculating the correlation coefficient matrix first, and then calculating a plurality of heart rate cycles according to the correlation coefficient matrix:

for example, if the first dimension is a row of the matrix and the second dimension is a column of the matrix, from the interval [0, l-c + s_max]Taking possible moments as a first dimension, fromInterval [ s ]_min，s_max]Taking the possible delay amount as the second dimension, the correlation coefficient matrix has (l-c-s) in common_max+1)×(s_max-s_min+1) correlation coefficients, calculating the correlation coefficients according to the formula given above, and taking the delay amount corresponding to the first maximum value exceeding the first threshold as the heart rate period for the correlation coefficients in each row of the correlation coefficient matrix, thereby calculating the total l-c-s_max+1 heart rate cycle.

Step S240: and calculating the heart rate corresponding to the signal sequence according to the plurality of heart rate cycles.

Step S240 may implement heart rate calculation by means of weighted summation, for example:

mode 1: first, the plurality of heart rate cycles obtained in step S230 are weighted and summed to obtain an average heart rate cycle, and then the average heart rate cycle is converted into a heart rate corresponding to the signal sequence, that is, a heart rate to be measured.

Mode 2: firstly, each heart rate cycle obtained in step S230 is converted into a corresponding heart rate to obtain a plurality of heart rates, and then the heart rates are weighted and summed to obtain the heart rate corresponding to the signal sequence, that is, the heart rate to be measured.

As for how the weighting of the weighted sum in the mode 1 and the mode 2 is set, a preset value may be adopted, or calculation may be performed according to some factors, which will be described later after the concept of the mass fraction is introduced. In the mode 1 and the mode 2, the heart rate is calculated by means of weighted summation, and the reliability is high. In particular, if the weights are set reasonably, the influence of inaccurate calculation of individual heart rate cycles on the heart rate measurement result can be effectively weakened.

In addition to the weighted summation, step S240 does not exclude other implementations, such as inputting a plurality of heart rate cycles into a model, predicting a heart rate corresponding to a signal sequence using the model, and so on.

The heart rate obtained in step S240 may be used for displaying, playing in voice, printing, storing, and the like, which is not limited in this application.

It should be noted that, only the period of the signal sequence obtained in step S210 is calculated in step S240, and if the signal sequence changes, the calculated heart rate period also changes accordingly. For example, step S210 may continuously acquire a recent signal sequence (e.g., a signal sequence converted from a recently acquired 100-frame image), and output a measured heart rate in real time, so that the user may observe a real-time change of the heart rate.

Brief summary the heart rate measurement method in fig. 3: the method belongs to a PPG method, the heart rate is measured by performing autocorrelation operation on a signal sequence, and the noise resistance of the method is obviously stronger than that of the existing method because the influence of noise on a correlation coefficient is smaller, so that higher measurement accuracy can be always kept. In contrast, the conventional PPG method generally performs peak detection or frequency domain analysis on the PPG signal (i.e. the reflected light signal) directly to perform heart rate measurement, so that the measurement result is easily interfered by noise.

In addition, the heart rate in the method is calculated according to a plurality of heart rate cycles, the reliability is high, and even if a certain heart rate cycle is not accurately calculated, the final heart rate measurement result can be only influenced to a very limited extent.

In the following, on the basis of the above example, the mass fraction of the sequence segments is introduced:

as can be seen from the principles of the PPG method, if there is not enough ac component in the PPG signal (i.e. the reflected light signal), the heart rate cannot be measured efficiently, since the ac component is due to the heartbeat. Therefore, in some implementations, when step S220 is executed, it may be first evaluated whether there is enough ac component in the signal sequence as a whole, and then it is determined whether to perform subsequent heart rate calculation according to the evaluation result, so as to avoid inaccurate or meaningless result of heart rate measurement. Specifically, the method includes (step S220 may be implemented as the following steps a to d):

step a: a plurality of initial reference fragments in the signal sequence is determined.

Each initial reference segment corresponds to a different time, and the initial reference segment may be defined as: and the sequence segment taking the corresponding time as the starting time in the signal sequence. It will be appreciated that the initial reference segment and the reference segment are identical in definition, and thus the manner in which the plurality of initial reference segments are determined may also be referred to previously and will not be repeated.

Of course, as will be understood from the following description, the initial reference segment may not be used as the reference segment, or only a part of the initial reference segment may be used as the reference segment.

Step b: and c, calculating the mass fraction of each initial reference fragment in the step a to obtain a plurality of mass fractions.

The quality score of an initial reference segment characterizes the amount of ac signal contained in the initial reference segment, e.g., the higher the quality score, the more ac signal it contains. Possible ways of calculating the quality score are described later.

Step c: and c, judging whether the total number of the qualified mass fractions exceeds a second threshold value or not for the mass fractions obtained in the step b, if so, executing the step d, otherwise, not executing the step d.

And judging whether one quality score is qualified or not by combining a threshold, wherein the threshold is not called as a third threshold, if the quality score is larger than the third threshold, the quality score is qualified, and if the quality score is not larger than the third threshold, the quality score is not qualified. If the mass fraction of the initial reference segment with enough (the total number exceeds the second threshold) in the signal sequence is qualified, the alternating current component in the signal sequence is enough to meet the basic requirements of the PPG method, so that the subsequent heart rate calculation can be carried out, otherwise, the alternating current component in the signal sequence is very limited, the heart rate measurement cannot be effectively carried out, and the measurement process can be ended.

In order to make the quality score of the initial reference segment representative of the quality of the signal sequence, when the initial reference segment is selected in step a, the selected plurality of initial reference segments may cover the entire signal sequence, and there may be an overlap between the selected plurality of initial reference segments.

By executing the steps a to c, unnecessary heart rate calculation steps can be avoided, so that the accuracy of the heart rate measurement result is improved, and the calculation amount is saved. For the case that the total number of qualified quality scores in step c does not exceed the second threshold, a corresponding prompt message may also be output, e.g. prompting the user to press the finger 160 at the correct position, etc.

In addition, the total number of qualified mass fractions in step c exceeds the second threshold, only indicating that the signal sequence is generally qualified for heart rate measurement, but there may still be a small number of initial reference fragment mass fractions that are not qualified.

Step d: directly regarding the plurality of initial reference segments in step a as the plurality of reference segments in step S220, or determining only the plurality of initial reference segments with qualified quality scores as the plurality of reference segments in step S220.

On the premise that the overall quality of the signal sequence is qualified, all initial reference segments can be used as reference segments for subsequent correlation coefficient calculation, or only the initial reference segments with qualified quality scores can be used as reference segments for subsequent correlation coefficient calculation. The former is favorable for fully utilizing information contained in the signal sequence, and the latter screens sequence segments participating in correlation coefficient calculation through mass fraction, so that the accuracy of heart rate measurement results is improved, and the calculation amount is saved.

For example, 100 initial reference segments are determined in the step a, the second threshold is 80, and assuming that the quality scores of 90 initial reference segments in the step c are qualified, in one implementation, all 100 initial reference segments may be used as reference segments to participate in the subsequent correlation coefficient calculation, and in another implementation, only 90 quality score qualified initial reference segments may be used as reference segments to participate in the subsequent correlation coefficient calculation.

In some implementations, the following calculation is taken for the quality scores of the initial reference segments (the quality scores of each initial reference segment are calculated in the same way):

step b 1: fitting a straight line satisfying the following condition with a plurality of sampling points in the initial reference fragment: the sum of the deviation amounts between the plurality of sample points in the initial reference segment and the straight line is minimized.

Step b 2: the minimum value of the sum of the deviation amounts in step b1 is determined as the quality score of the initial reference segment.

Wherein, the deviation amount between a certain sampling point and a straight line represents the deviation degree of the sampling point and the straight line. For example, the deviation amount of a certain sample point can be defined as:

(1) the distance from the sampling point to the straight line;

(2) the square of the distance of the sample point to the straight line;

(3) the absolute value of the difference between the y coordinate of the corresponding point of the sample point on the straight line and the y coordinate of the sample point, wherein the corresponding point of the sample point on the straight line is: points on the straight line having the same x coordinate as the sampling point;

(4) the square of the difference between the y coordinate of the corresponding point of the sample point on the straight line and the y coordinate of the sample point.

Of course, the amount of deviation can also be defined in other ways and is not listed any more. Next, how to obtain a straight line and a mass score satisfying the conditions by the optimization method will be described by taking the case where the deviation amount is defined in item (2) as an example:

let ax + by + c be 0, where a, b, and c are all parameters of the equation. Normalization of these parameters, i.e. division of both sides of the equation by

The normalized form is obtained: ax + By + C ═ 0, where a²+B²＝1。

The objective function Q to be optimized is set as follows:

wherein (x)_i,y_i) Represents any sample in the initial reference fragment, i is the sample number,

this part is the sample point (x)_i,y_i) The square of the distance to the straight line Ax + By + C is the deviation in step a, which is 0.

To solve for A, B, C, the partial derivative may be calculated for the right side of the objective function, and the following equation may be derived:

A×(E(xx)E(y)-E(xy)E(x))+B×(E(xy)E(y)-E(yy)E(x))＝1

where E () denotes expectation, x denotes a sequence consisting of the x coordinates of all the samples in the initial reference fragment, x sequence for short, y denotes a sequence consisting of the y coordinates of all the samples in the initial reference fragment, y sequence for short, xx denotes the dot product of two x sequences, yy denotes the dot product of two y sequences, and xy denotes the dot product of the x sequence and the y sequence.

Recombination restriction Condition A²+B²The parameter A, B can be found at 1, and C can be calculated, and the detailed calculation process is omitted. Order:

p＝E(xx)E(y)-E(xy)E(x)

q＝E(xy)E(y)-E(yy)E(x)

then there are:

C＝-E(x)×A+E(y)×B

finding A, B, C naturally corresponds to finding an equation where ax + by + c is 0 (if a, b, c can no longer be calculated for the purpose of calculating mass fraction only, this is only a logical understanding). Substituting solved A, B, C into the expression for Q yields:

Q_min＝A²×E(xx)+B²×E(yy)+C²+2AB×E(xy)+2AC×E(x)+2BC×E(y)

Q_minis the quality score of the initial reference segment.

In the process of calculating the mass fraction, the straight line obtained by fitting can be regarded as a direct current component in the initial reference segment, and the deviation amount between the sampling point and the straight line can be regarded as an alternating current component in the initial reference segment, so that the mass fraction of the initial reference segment determined according to the sum of the deviation amounts is consistent with the definition of the mass fraction given in the step b set forth above.

It is pointed out before step S240 is described that the heart rate corresponding to the signal sequence can be calculated by weighted summation, and two weighted summation manners are given, the calculation manner of the weight is described below by way of example in manner 1, and the weight in manner 2 can also be calculated similarly.

Optionally, the weight for weighted summation for each heart rate cycle is determined according to at least one of the following three factors:

(1) correlation coefficient corresponding to heart rate cycle

As can be seen from the foregoing, in an implementation manner, the heart rate cycle may be calculated according to a certain very large correlation coefficient (the maximum value of the first correlation coefficient in the correlation coefficient sequence that exceeds the first threshold), so that the correlation coefficient corresponding to the heart rate cycle is the very large correlation coefficient.

For example, the correlation coefficient corresponding to a certain heart rate cycle belongs to the interval [0.8, 1], and may be mapped to weight 1 according to a preset mapping relationship; the correlation coefficient corresponding to a certain heart rate period belongs to an interval [0.5, 0.8), and can be mapped into a weight of 0.5 according to a preset mapping relation; the correlation coefficient corresponding to a certain heart rate cycle belongs to the interval [ -1, 0.5), and may be mapped to weight 0 according to a preset mapping relationship, and so on. The example summarizes that the correlation coefficient corresponding to the heart rate cycle is mapped to the weight in a stepwise manner, and the correlation coefficient and the weight may have a positive correlation.

(2) Heart Rate periods the size of the plurality of heart rate periods obtained in step S230 is ordered

For example, if a certain heart rate cycle ranks 10% last or 10% first in all heart rate cycles obtained in step S230, the weight is set to 0; if a certain heart rate cycle is 10% -90% of the middle of all heart rate cycles, the weight of the certain heart rate cycle is set to be a value other than 0, and the weight can be calculated according to factors (1) and (3) or through other modes. This example is summarized as excluding outliers in the heart rate cycle, avoiding that some erroneous estimates of the heart rate cycle disturb the heart rate measurement.

(3) Quality fraction of reference segment corresponding to heart rate cycle

As shown in step S230, a heart rate cycle is necessarily calculated according to a reference segment and a plurality of corresponding delay segments, where the reference segment is a reference segment corresponding to the heart rate cycle. Further, if step S220 is implemented by steps a to d, a quality score is calculated for each initial reference segment, and the reference segments are subsets of the initial reference segments, so that the reference segments necessarily also calculate quality scores at this time, and the quality scores can be directly used for the weight calculation here. Alternatively, even if step S220 is not implemented by steps a-d (i.e., the quality scores are not calculated), the quality scores of the reference segments may be calculated separately when determining the weights, in the same manner as described above.

For example, if the quality score of the reference segment corresponding to a certain heart rate period is not qualified, the weight of the reference segment is set to 0; if the quality score of the reference segment corresponding to a certain heart rate cycle is qualified, the weight of the reference segment is set to be 1, or the quality score is mapped to a certain value in the interval (0,1), or the weight is further determined according to the factor (1) and the like.

As an example of determining the weight by considering the above three factors at the same time, first, the factor (3) is considered, if the quality score of the reference segment corresponding to a certain heart rate cycle is qualified, the factor (1) is further considered, if the correlation coefficient corresponding to the heart rate cycle is 0.9, the weight is set to 1 according to the predetermined mapping relationship, then the factor (2) is considered, and if the value of the heart rate cycle is judged to be the last 10% of all the heart rate cycles, the weight is set to 0 again.

The weighting weight of the heart rate period is determined by multiple factors, so that the flexibility of the weighting in calculation is given, the weighting can be calculated more reasonably, and the accuracy of heart rate measurement is improved.

Fig. 5 shows functional modules that may be included in the heart rate measurement device 300 provided in the embodiment of the present application. Referring to fig. 5, the heart rate measuring apparatus 300 includes:

a signal sequence acquisition module 310, configured to acquire a signal sequence generated according to the reflected light signal; the reflected light signal is a signal formed by reflecting a measuring light ray emitted by a light source by a human body part to be measured, the signal sequence is composed of a plurality of sampling points, and the signal value of each sampling point represents the brightness of the reflected light signal at the sampling time corresponding to the sampling point;

a sequence segment determination module 320 for determining a plurality of reference segments in the signal sequence; each reference segment corresponds to a different time, and the reference segment is a sequence segment which takes the corresponding time as a starting time in the signal sequence;

a heart rate cycle calculation module 330, configured to calculate, for each reference segment of the multiple reference segments, a correlation coefficient between the reference segment and the corresponding multiple delay segments, and calculate, according to the correlation coefficient, a heart rate cycle corresponding to the reference segment, so as to obtain multiple heart rate cycles in total; each delay segment corresponds to a different delay amount, and the delay segment is a corresponding sequence segment in the signal sequence after delaying the reference segment according to the corresponding delay amount;

and a heart rate calculating module 340, configured to calculate a heart rate corresponding to the signal sequence according to the plurality of heart rate cycles.

In an implementation manner of the heart rate measuring apparatus 300, the heart rate cycle calculating module 330 calculates the heart rate cycle corresponding to the reference segment according to the correlation coefficient, including: determining a maximum value of a first correlation coefficient sequence exceeding a first threshold value, and determining a delay amount corresponding to the maximum value as a heart rate period corresponding to the reference segment; the correlation coefficient sequence is formed by arranging the correlation coefficients according to the sequence that the corresponding delay amount is sequentially increased.

In one implementation of the heart rate measurement device 300, there is overlap between the plurality of reference segments.

In one implementation of heart rate measurement device 300, the reference fragments have a total of l-c-s_max+1, each reference fragment corresponding to the interval [0, l-c-s_max]Wherein l is the length of the signal sequence and c is the length of the reference fragment, in the interval [0, l-c-s_max]The interval between any two adjacent time instants is 1.

In one implementation of heart rate measuring device 300, the time delay segments have s in common_max-s_min+1, each time-lapse segment corresponds to an interval [ s ]_min，s_max]A delay amount; wherein s is_minFor the expected minimum heart rate period, s_maxInterval [ s ] for expected maximum heart rate cycle_min，s_max]The interval between any two adjacent delay time amounts is 1.

In one implementation of the heart rate measuring device 300, the sequence segment determining module 320 determines a plurality of reference segments in the signal sequence, including: determining a plurality of initial reference fragments in the signal sequence; each initial reference segment corresponds to a different time, and the initial reference segment is a sequence segment which takes the corresponding time as an initial time in the signal sequence; calculating the quality score of each initial reference fragment in the plurality of initial reference fragments to obtain a plurality of quality scores; wherein the quality score characterizes an amount of AC signal contained in the initial reference segment; if the total number of qualified quality scores in the quality scores exceeds a second threshold value, the initial reference segments are the reference segments, or the initial reference segments with qualified quality scores are determined to be the reference segments.

In one implementation of the heart rate measuring device 300, the sequence segment determining module 320 calculates the quality score of the initial reference segment, including: fitting a straight line satisfying the following condition with a plurality of sampling points in the initial reference segment: the sum of deviation values between a plurality of sampling points in the initial reference segment and the straight line is a minimum value; determining the minimum value as a quality score of the initial reference segment.

In one implementation of the heart rate measuring apparatus 300, the calculating module 340 calculates the heart rate corresponding to the signal sequence according to the plurality of heart rate cycles, including: carrying out weighted summation on the multiple heart rate cycles to obtain an average heart rate cycle, and calculating the heart rate corresponding to the signal sequence according to the average heart rate cycle; or calculating the heart rate corresponding to each heart rate cycle to obtain a plurality of heart rates, and performing weighted summation on the heart rates to obtain the heart rate corresponding to the signal sequence.

In one implementation of the heart rate measurement device 300, the weight of the heart rate calculation module 340 to weight and sum each heart rate cycle is determined according to at least one of the following three factors: correlation coefficients corresponding to the heart rate cycles; the heart rate cycles are sized in the plurality of heart rate cycles; the quality score of the reference segment corresponding to the heart rate cycle.

In one implementation of the heart rate measuring device 300, the signal sequence acquiring module 310 acquires a signal sequence generated according to the reflected light signal, including: controlling an area serving as a light source on a display screen to emit the measuring light; acquiring an image sequence generated by an image sensor arranged below the display screen according to the reflected light signal; and determining the brightness of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image to obtain the signal sequence consisting of a plurality of sampling points.

In one implementation of the heart rate measuring apparatus 300, the area as the light source is located right above the image sensor, and the signal sequence obtaining module 310 determines the brightness of each frame of image in the image sequence as the signal value of the corresponding sampling point of the frame of image, including: and determining the brightness of the central part of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image.

The heart rate measuring device 300 provided by the embodiment of the present application, the implementation principle and the technical effects thereof have been introduced in the foregoing method embodiments, and for the sake of brief description, portions of the device embodiments that are not mentioned may refer to corresponding contents in the method embodiments.

Embodiments of the present application further provide a computer-readable storage medium, where computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are read and executed by a processor, the method for measuring a heart rate provided by the embodiments of the present application is executed. For example, the computer-readable storage medium may be embodied as memory 120 in electronic device 100 in FIG. 1.

Embodiments of the present application further provide a computer program product, which includes computer program instructions, and when the computer program instructions are read and executed by a processor, the method for measuring a heart rate provided by the embodiments of the present application is performed.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of heart rate measurement, comprising:

acquiring a signal sequence generated according to the reflected light signal; the reflected light signal is a signal formed by reflecting a measuring light ray emitted by a light source by a human body part to be measured, the signal sequence is composed of a plurality of sampling points, and the signal value of each sampling point represents the brightness of the reflected light signal at the sampling time corresponding to the sampling point;

determining a plurality of reference fragments in the signal sequence; each reference segment corresponds to a different time, and the reference segment is a sequence segment which takes the corresponding time as a starting time in the signal sequence;

calculating a correlation coefficient between each reference segment and each corresponding delay segment in the plurality of reference segments, and calculating a heart rate cycle corresponding to the reference segment according to the correlation coefficient to obtain a plurality of heart rate cycles; each delay segment corresponds to a different delay amount, and the delay segment is a corresponding sequence segment in the signal sequence after delaying the reference segment according to the corresponding delay amount;

and calculating the heart rate corresponding to the signal sequence according to the plurality of heart rate cycles.

2. The method for measuring heart rate according to claim 1, wherein the calculating the heart rate cycle corresponding to the reference segment according to the correlation coefficient includes:

determining a maximum value of a first correlation coefficient sequence exceeding a first threshold value, and determining a delay amount corresponding to the maximum value as a heart rate period corresponding to the reference segment; the correlation coefficient sequence is formed by arranging the correlation coefficients according to the sequence that the corresponding delay amount is sequentially increased.

3. Method of heart rate measurement according to claim 1 or 2, characterized in that there is an overlap between the plurality of reference segments.

4. Heart rate measurement method according to claim 3, wherein the reference fragments have a total of l-c-s_max+1, each reference fragment corresponding to the interval [0, l-c-s_max]Wherein l is the length of the signal sequence and c is the length of the reference fragment, in the interval [0, l-c-s_max]The interval between any two adjacent time instants is 1.

5. Heart rate measurement method according to any one of claims 1-4, wherein the time-lapse fraction shares s_max-s_min+1, each time-lapse segment corresponds to an interval [ s ]_min，s_max]A delay amount; wherein s is_minFor the expected minimum heart rate period, s_maxInterval [ s ] for expected maximum heart rate cycle_min，s_max]The interval between any two adjacent delay time amounts is 1.

6. The method of heart rate measurement according to any one of claims 1-4, wherein the determining a plurality of reference segments in the signal sequence comprises:

determining a plurality of initial reference fragments in the signal sequence; each initial reference segment corresponds to a different time, and the initial reference segment is a sequence segment which takes the corresponding time as an initial time in the signal sequence;

calculating the quality score of each initial reference fragment in the plurality of initial reference fragments to obtain a plurality of quality scores; wherein the quality score characterizes an amount of AC signal contained in the initial reference segment;

if the total number of qualified quality scores in the quality scores exceeds a second threshold value, the initial reference segments are the reference segments, or the initial reference segments with qualified quality scores are determined to be the reference segments.

7. The heart rate measurement method of claim 6, wherein calculating the quality score of the initial reference segment comprises:

fitting a straight line satisfying the following condition with a plurality of sampling points in the initial reference segment: the sum of deviation values between a plurality of sampling points in the initial reference segment and the straight line is a minimum value;

determining the minimum value as a quality score of the initial reference segment.

8. The method according to any one of claims 1-7, wherein the calculating a heart rate corresponding to the signal sequence from the plurality of heart rate cycles comprises:

carrying out weighted summation on the multiple heart rate cycles to obtain an average heart rate cycle, and calculating the heart rate corresponding to the signal sequence according to the average heart rate cycle; or,

and calculating the heart rate corresponding to each heart rate cycle to obtain a plurality of heart rates, and performing weighted summation on the heart rates to obtain the heart rate corresponding to the signal sequence.

9. A method as claimed in claim 8, wherein the weighting for weighted summation of each heart rate cycle is determined in dependence on at least one of the following three factors:

correlation coefficients corresponding to the heart rate cycles;

the heart rate cycles are sized in the plurality of heart rate cycles;

the quality score of the reference segment corresponding to the heart rate cycle.

10. The method of any one of claims 1-9, wherein acquiring a sequence of signals generated from reflected light signals comprises:

controlling an area serving as a light source on a display screen to emit the measuring light;

acquiring an image sequence generated by an image sensor arranged below the display screen according to the reflected light signal;

and determining the brightness of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image to obtain the signal sequence consisting of a plurality of sampling points.

11. The method of claim 10, wherein the area as a light source is located directly above the image sensor, and the determining the brightness of each frame of image in the image sequence as the signal value of the corresponding sampling point of the frame of image comprises:

and determining the brightness of the central part of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image.

12. A computer program product comprising computer program instructions which, when read and executed by a processor, perform the method of any one of claims 1 to 11.

13. A computer-readable storage medium having stored thereon computer program instructions which, when read and executed by a processor, perform the method of any one of claims 1-11.

14. An electronic device, comprising: a memory having stored therein computer program instructions which, when read and executed by the processor, perform the method of any of claims 1-11.

15. The electronic device of claim 14, wherein the device further comprises: the display screen and set up the image sensor under the display screen, the signal sequence is obtained to the treater through carrying out following step: controlling an area serving as a light source on the display screen to emit measuring light, and acquiring an image sequence generated by the image sensor according to the reflected light signal; and determining the brightness of each frame of image in the image sequence as the signal value of the sampling point corresponding to the frame of image to obtain the signal sequence consisting of a plurality of sampling points.