JP6765678B2 - Pulse wave detector and pulse wave detection program - Google Patents

Description

The present invention relates to a pulse wave detection device and a pulse wave detection program for detecting a human pulse wave.

Detection of pulse waves is extremely important for understanding the physiological situation of human beings. In detecting the pulse wave, first, a rectangular evaluation area is set on the screen of the moving image, the subject's face is seated so as to enter the evaluation area, and the stationary face is photographed in the moving image. When detecting pulse waves indoors, sunlight entering through a window is used as a light source. When the obtained moving image is separated into each component of R component, G component, and B component and averaged, a fluctuation with a pulse wave is obtained. Each of these components contains a pulse wave signal weighted according to the light absorption characteristics of hemoglobin, and when ICA (Independent Component Analysis) or the like is performed on this, the pulse wave is generated. can get.

The pulse wave is obtained from the moving image in this way because the volume of blood vessels changes with the heartbeat of the subject, and the optical distance through which sunlight passes through the skin changes, which is the change in the reflected light from the face. Because it appears as.

When the pulse wave is detected outdoors, for example, when the pulse wave detection device is mounted on the vehicle to detect the pulse wave of the driver (subject), the change in the brightness of the ambient light becomes a disturbance factor and the pulse wave is detected. Becomes difficult. Patent Documents 1 and 2 disclose a technique for removing the influence of a change in the brightness of ambient light in detecting a pulse wave. According to the technique, even when the brightness changes due to the subject moving in a vehicle or the like, the pulse wave can be detected satisfactorily.

Japanese Unexamined Patent Publication No. 2016-193021 WO2016-159150

However, when the subject moves in a vehicle or the like, the ambient light changes frequently. Such a change in ambient light is accompanied by not only a change in brightness but also a change in color. Since the pulse wave signal is measured from the color change of the face, the color change of the ambient light becomes a large disturbance, and there is a problem that it is difficult to detect the pulse wave even by the techniques of Patent Documents 1 and 2.

The present invention has been made to solve the above-mentioned problems, and provides a pulse wave detection device and a pulse wave detection program capable of detecting a pulse wave by removing the influence of a color change of ambient light. It is an object.

In order to achieve this object, the pulse wave detection device of the present invention refers to a photographing means for photographing an area including the skin of a subject for at least a predetermined time and an image of the skin area photographed by the photographing means. A strong / weak region measuring means for identifying a strong region and a weak region, a strong region measuring means for measuring a signal of an image taken by the photographing means for a region specified as a region having a strong pulse wave by the strong / weak specifying means, and the above-mentioned A weak region measuring means for measuring a signal of an image taken by the photographing means and a signal of a weak pulse region measured by the weak region measuring means for a region specified as a region where the pulse wave is weak by the strength specifying means. Based on the above, the correction means for correcting the signal in the strong region of the pulse wave measured by the strong region measuring means and the output means for outputting the pulse wave signal corrected by the correction means are provided.

Further, the pulse wave detection program of the present invention has an image acquisition function for acquiring an image obtained by capturing an image of a region including the subject's skin for at least a predetermined time, and an image of the skin region acquired by the image acquisition function. A strength identification function that identifies a strong region and a weak region, and a strong region measurement function that measures a signal of an image acquired by the image acquisition function for a region specified as a region with a strong pulse wave by the strength identification function. A weak region measurement function that measures the signal of the image acquired by the image acquisition function for a region specified as a region where the pulse wave is weak by the strength identification function, and a weak region measurement function of the pulse wave measured by the weak region measurement function. Based on the signal of, the computer realizes a correction function that corrects the signal in the strong region of the pulse wave measured by the strong region measurement function and an output function that outputs the pulse wave signal corrected by the correction function. It is something that makes you.

According to the pulse wave detection device and the pulse wave detection program of the present invention, the signal in the strong pulse wave region (target region) is corrected based on the signal in the weak pulse wave region (reference region), and this is corrected. Output as a signal. Therefore, there is an effect that the pulse wave can be detected by removing the influence of the color change of the ambient light without requiring a background (color checker or skin color background) for color correction. Also, since the skin color in the region where the pulse wave is strong (target region) and the skin color in the region where the pulse wave is weak (reference region) are similar, it is possible to correct the pulse wave signal with a simpler algorithm than usual. effective. That is, there is an effect that the pulse wave signal can be corrected in a short time.

It is a schematic diagram which shows the appearance of the pulse wave detection device. (A) is a block diagram showing an electrical configuration of a pulse wave detection device, (b) is a diagram schematically showing a detection region table, and (c) is a diagram schematically showing strong region data. It is a representation figure. It is a figure which represented the area of an image schematically. It is a flowchart of a main process. It is a flowchart of an area acquisition process. It is a figure for demonstrating the main process and the area acquisition process, (a) is a schematic diagram which showed the image acquired from a camera, (b) is a face image extracted from the image of (a). It is a schematic diagram shown, and (c) is a figure which showed the frequency spectrum of the G value in the face image existing in the image history memory. It is a figure for demonstrating the area acquisition process, (a) is a figure which showed typically the face area, (b) is a figure which shows typically the waveform of the time transition of G value in each face area. FIG. 5C is a diagram schematically showing a frequency spectrum of a G value in a face image stored in an image history memory. (A) is a diagram schematically showing the G value at the peak frequency, (b) is a diagram schematically showing a strong region, and (c) is a diagram schematically showing a weak region. It is a figure. It is a flowchart of pulse wave output processing. It is a figure which showed the time transition of the value of a strong region value memory and the time transition of the value of a strong region value memory after correction when the color of ambient light is changed periodically. (A) is a flowchart of a region acquisition process in the second embodiment, (b) is a schematic diagram showing a strong region, and (c) is a schematic diagram showing a weak region.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. First, an outline of the pulse wave detection device 1 will be described with reference to FIG. FIG. 1 is a schematic view showing the appearance of the pulse wave detection device 1. The pulse wave detection device 1 is a device for acquiring a pulse wave signal of the subject H from a color change in the skin region of the face of the subject H and detecting the pulse wave. The pulse wave detection device 1 is for inputting a control unit 2 that controls each part of the pulse wave detection device 1, a camera 3, a display of the detected pulse wave, and an instruction from the user to the pulse wave detection device 1. It has a display unit 4.

The camera 3 is a device for photographing the subject H and acquiring the image, and is provided at a height at which the subject H faces the seated subject H and the vicinity of the subject's face can be photographed. The camera 3 acquires an image every 1/30 second and transmits it to the control unit 2. The image acquisition interval by the camera 3 is not limited to 1/30 second, and may be 1/30 second or more or 1/30 second or less depending on the processing speed of the camera 3 and the control unit 2. good.

The display unit 4 includes an LCD 10 for displaying the acquired pulse wave signal and the like, and a touch panel 11 for inputting an instruction from the user to the control unit 2 (see FIG. 2), and is a pulse wave detection device. It is provided on the upper part of 1.

The control unit 2 decomposes the image near the face of the subject H acquired from the camera 3 into an RGB color space, and from the intensity (signal) of the G component, the region where the pulse wave is strong in the subject H (target region). ) And the region where the pulse wave is weak (reference region). Then, after correcting the intensity of the G component in the region where the pulse wave is strong with the intensity of the G component in the region where the pulse wave is weak, the pulse wave signal is acquired and output to the LCD 10 of the display unit 4. Hereinafter, the intensity of the G component is referred to as "G value", the region where the pulse wave is strong is referred to as "strong region SA", and the region where the pulse wave is weak is referred to as "weak region WA".

Next, the electrical configuration of the pulse wave detection device 1 will be described with reference to FIGS. 2 and 3. FIG. 2A is a block diagram showing an electrical configuration of the pulse wave detection device 1, FIG. 2B is a diagram schematically showing a detection region table 6b, and FIG. 2C is a diagram. , Is a diagram schematically showing strong region data, and FIG. 3 is a diagram schematically showing regions A1 to A1088 of the acquired image.

The control unit 2 has a CPU 5, a hard disk drive (HDD) 6, and a RAM 7, which are connected to the input / output ports 9 via the bus line 8, respectively. A camera 3 and a display unit 4 are connected to the input / output ports 9, respectively. The CPU 5 is an arithmetic unit that controls each unit connected by the bus line 8.

The HDD 6 is a rewritable non-volatile storage device that stores a program executed by the CPU 5, fixed value data, and the like, and stores the control program 6a and the detection area table 6b. When the control program 6a is executed by the CPU 5, the main process of FIG. 4 is executed. The detection area table 6b is a data table in which the strong area SA and the weak area WA are stored in association with the face image of the subject H. The detection area table 6b will be described with reference to FIGS. 2 (b) and 2 (c) and FIG.

As shown in FIG. 2B, the face image data 6b1, the strong region data 6b2, and the weak region data 6b3 are provided in the detection region table 6b, and are stored in association with each other. In the face image data 6b1, the face images FP1, FP2, ..., Which are images of the skin region of the face in the subject H, acquired by the camera 3 are stored.

The strong region data 6b2 stores the strong region data SA1, SA2, ... Showing the strong region SA (see FIG. 8B) in the subject H corresponding to the face images FP1, FP2, ... Will be done. Here, the data structure of the strong region data SA1 will be described with reference to FIGS. 2 (c) and 3.

As shown in FIG. 2C, the strong region data SA1 stores the regions A1 to A1088 corresponding to the strong region SA in the subject H of the face image FP1. As shown in FIG. 3, the areas A1 to A1088 are obtained by dividing the image acquired by the camera 3 into 34 in the vertical direction and 32 in the horizontal direction, for a total of 1088, in order from the upper left area of the image. Areas A1, A2, ..., A1088. For example, in FIG. 2C, the strong region data SA1 stores regions A175, A176, A177 ... As the strong region SA. Hereinafter, when the regions A1, A2, ..., A1088 are not distinguished, they are referred to as "region An". Although the image is divided into 1088, the image is not limited to this, and the image may be divided into 1088 or more, or 1088 or less.

Return to FIG. 2 (b). In the weak region data 6b3, the weak region data WA1, WA2, ... Showing the information of the weak region WA (see FIG. 8C) of the subject H corresponding to the face images FP1, FP2, ... Be remembered. The weak region data WA1, WA2, ... Also store the region An corresponding to the weak region WA (see FIG. 8C), respectively. Hereinafter, when the strong region data SA1, SA2, ... Are not distinguished, they are referred to as "strong region data SAm", and when the weak region data WA1, WA2, ... Are not distinguished, they are referred to as "weak region data WAm". ..

Since the region An determined to be the strong region SA or the weak region WA is stored for the strong region data SAm and the weak region data WAm, the positions of the pixels in the image determined to be the strong region SA or the weak region WA are stored. The amount of data required for the strong region data SAm and the weak region data WAm can be suppressed as compared with the case where is stored. As a result, the processing time required for the reading process (FIGS. 4 and S5) and the storage process (FIGS. 5 and S33) of the strong region data SAm and the weak region data WAm is also shortened, so that the response of the pulse wave detection device 1 is improved. ..

Return to FIG. 2 (a). The RAM 7 is a memory for the CPU 5 to rewritably store various work data, flags, and the like when the control program 6a is executed. The image history memory 7a and the skin of the face of the subject H acquired from the camera 3 A face image memory 7b for storing an image of an area, a strong area memory 7c, a weak area memory 7d, a strong area value memory 7e, a weak area value memory 7f, a strong / weak ratio memory 7g, and a correction value memory 7h. Are provided respectively.

The image history memory 7a is a memory in which images acquired from the camera 3 are stored in chronological order. Further, the image history memory 7a stores images for up to 30 seconds. The image history memory 7a is configured to store images for a maximum of 30 seconds, but the present invention is not limited to this, and an image of 30 seconds or more may be stored, or an image of 30 seconds or less may be stored. May be remembered.

The strong area memory 7c is a memory in which the area An determined to be the strong area SA is stored from the image stored in the image history memory 7a, and the weak area memory 7d is from the image stored in the image history memory 7a. , A memory in which an area An determined to be a weak area WA is stored. Since the data structures of the strong region memory 7c and the weak region memory 7d are the same as those of the strong region data SA1 (FIG. 2C), the description thereof will be omitted.

The strong region value memory 7e is a memory in which the average value of the G values of the regions An (that is, the strong region SA) stored in the strong region memory 7c in the image stored in the face image memory 7b is stored, and is a weak region. The value memory 7f is a memory in which the average value of the G values in the area An (that is, the weak area WA) stored in the weak area memory 7d of the image stored in the face image memory 7b is stored. The pulse wave detection device 1 acquires a pulse wave signal based on the value of the strong region value memory 7e. At that time, if it is determined that the value of the weak area value memory 7f includes disturbance due to the color change of the ambient light, the value of the strong area value memory 7e is corrected based on the value of the weak area value memory 7f. The pulse wave signal is acquired from the value obtained.

The strength ratio memory 7g is a memory used for correcting the value of the strong region value memory 7e, and is a memory for storing the ratio of the G value between the strong region SA and the weak region WA, and the correction value memory 7h is the strong region value. This is a memory in which a correction parameter value for correcting the value of the memory 7e is stored.

Next, the process executed by the CPU 5 of the control unit 2 will be described with reference to FIGS. 4 to 10. FIG. 4 is a flowchart of the main process. The main process is performed immediately after the power of the pulse wave detection device 1 is turned on.

The main process first acquires an image from the camera 3 and adds it to the image history memory 7a (S1). Since an image is acquired from the camera 3 every 1/30 second, when a new image is acquired, the image is added to the image history memory 7a. When an image is stored in the image history memory 7a for 30 seconds and a new image is acquired from the camera 3, the oldest image in the image history memory 7a is deleted, and the newly acquired image is the image history. It is added to the memory 7a.

After the processing of S1, the skin region of the face of the subject H is extracted from the image acquired by the processing of S1 and stored in the face image memory 7b (S2). Here, a method of extracting the face portion of the subject H from the image will be described with reference to FIGS. 6A and 6B.

FIG. 6A is a schematic view showing an image acquired from the camera 3, and FIG. 6B is a schematic view showing a face image extracted from the image of FIG. 6A. As shown in FIG. 6A, the image acquired from the camera 3 is composed of a partial image A composed of the skin region of the face and a partial image B composed of other hair, clothes, and the like. .. Since the pulse wave is detected in the partial image A, the partial image A is extracted, the partial image B for the partial image B is invalidated, and the partial image B is stored in the face image memory 7b. Hereinafter, the image composed of the partial image A is referred to as a "face image". As a method for extracting the partial image A, known image recognition techniques such as threshold value processing for separating by pixel values, edge detection, and pattern recognition are appropriately applied.

Return to FIG. After the processing of S2, it is confirmed whether or not the face image of the face image data 6b1 of the detection area table 6b, which is close to the face image of the face image memory 7b, exists (S3). If there is no face image of face image data 6b1 that is close to the value of the face image memory 7b (S3: No), the subject H detects the pulse wave for the first time with the pulse wave detection device 1, or the pulse wave. However, there is a case where the face image of the face image memory 7b and the face image of the face image data 6b1 do not match due to the lapse of time from the detection or the like. In such a case, the area acquisition process (S4) is performed. The area acquisition process will be described with reference to FIGS. 5 to 8.

FIG. 5 is a flowchart of the area acquisition process. The area acquisition process first confirms whether or not an image for 30 seconds exists in the image history memory 7a (S20). This is because in the area acquisition process, the strong area SA and the weak area WA are acquired from the time transition of the G value of the image stored in the image history memory 7a, so that a certain time or more (30 seconds in the present embodiment). ) Minutes of images are needed.

When an image for 30 seconds exists in the image history memory 7a (S20: Yes), the peak frequency BPMa of the pulse frequency band in the face image acquired from the value of the image history memory 7a is acquired (S21). The process of S21 will be described with reference to FIGS. 6 (b) and 6 (c).

FIG. 6C is a diagram showing a frequency spectrum of the G value in the face image stored in the image history memory 7a. In the process of S21, first, a face image is acquired from each of the images stored in the image history memory 7a (see FIG. 6B). The method of acquiring the face image is the same as the processing method of S2 described above. Then, the average value of the G values in all the pixels of each face image is acquired. As a result, the time transition of the G value in the face image stored in the image history memory 7a is acquired.

By performing a discrete Fourier transform on the time transition of the G value by FFT or the like, the frequency spectrum of the G value in the face image stored in the image history memory 7a is acquired. In the frequency spectrum of the G value, the peak frequency BPMa, which is the peak value in the pulse frequency band (0.67 to 2.50 Hz) indicating the frequency band of the pulse, is acquired (FIG. 6 (c)). This peak frequency BPMa is estimated to be the peak frequency of the pulse rate in the subject H. In the process of S25 or less described later, for each region An in the face image, the region An having a large frequency spectrum value of the peak frequency BPMa is determined to be the strong region SA, and the other regions An are determined to be the weak region WA. The pulse frequency band is not limited to the range of 0.67 to 2.50 Hz, and even if a range wider than 0.67 to 2.50 Hz is set depending on the distribution of the frequency spectrum intensity of the G value. It is good, and a narrow range may be set.

Return to FIG. After the processing of S21, the area An of the face image acquired from the image history memory 7a is acquired, and the area An is set to the face areas F1, F2, ..., Fj in order (S22). Hereinafter, when the face areas F1, F2, ..., Fj are not distinguished, they are referred to as "face area Fi". Here, the acquisition process of the face region Fi in the process of S22 will be described with reference to FIG. 7A.

FIG. 7A is a diagram schematically showing the face region Fi. As shown in FIG. 7 (a), in the process of S22, the face image acquired by the process of S21 is first divided into regions An, and then a partial image A composed of the skin region of the face (FIG. 6 (b)). Acquire the area An corresponding to (see). Then, the acquired regions An are face regions F1, F2, ..., Respectively (FIG. 7 (a)). In the example of FIG. 7A, the area A175 is the face area F1, the area A176 is the face area F2, and the area A177 is the face area F3, ...

Return to FIG. After the processing of S22, the time transition FGi of the G value in each face area Fi is acquired from the image stored in the image history memory 7a (S23). Specifically, for each image stored in the image history memory 7a, all the G values in the pixels corresponding to the face area Fi are acquired, and the average value thereof is calculated. By arranging the average value of the G value of the face area Fi for each image in the order in which the images are acquired, the time transition FGi of the G value in the face area Fi is acquired. This is done for all of the face area Fi. Then, after the processing of S23, the frequency spectrum SFGi of the G value is acquired by performing the discrete Fourier transform of the time transition FGi of the G value by FFT or the like (S24). Here, the time transition FGi of the G value acquired in the process of S23 and the frequency spectrum SFGi of the G value acquired in the process of S24 will be described with reference to FIGS. 7 (b) and 7 (c).

FIG. 7 (b) is a diagram schematically showing the waveform of the time transition FGi of the G value in each face region Fi, and FIG. 7 (c) is a diagram schematically showing the frequency spectrum SFGi of the G value in each face region Fi. It is a figure shown as a target. As shown in FIG. 7B, even in the same subject H, the time transition FGi of the corresponding G value differs depending on the position of the face region Fi. Further, as shown in FIG. 7C, also in the frequency spectrum SFGi of the G value, there are a face region Fi in which the peak of the frequency spectrum is remarkable and a face region Fi in which the peak of the frequency spectrum is not remarkable. This is because the region where the skin color change (change in G value) occurs and the region where the skin color change does not occur are mixed.

Generally, as the volume of blood vessels changes with the heartbeat of subject H, the optical distance through which ambient light passes through the skin changes, and the reflected light changes, resulting in a change in skin color. .. That is, the skin color change occurs in the region where the pulse wave is strong (strong region SA), while the skin color change does not occur so much in the region where the pulse wave is weak (weak region WA). Therefore, in the present embodiment, attention is paid to the value of the frequency spectrum SFGi of the G value, which indicates the strength of the color change of the skin, and the acquisition of the face region Fi corresponding to each of the strong region SA and the weak region WA is obtained. Do.

Return to FIG. After the processing of S24, the G value PFGi at the peak frequency BPMa acquired in S21 is acquired from the G value frequency spectrum SFGi (S25). Then, the maximum value maxPFG in the acquired total G value PFGi is acquired (S26). As described above, since the peak frequency BPMa is a frequency estimated to be the peak frequency of the pulse rate in the subject H, the G value PFGi at the peak frequency BPMa is a value indicating the intensity of the pulse wave signal in the face region Fi. Will be done. In the processing after S27, the face region Fi is divided into a strong region SA and a weak region WA by comparing the G value PFGi with the maximum value maxPFG.

After the processing of S26, the counter variable i is initialized with 1 (S27). After the processing of S27, it is confirmed whether the G value PFGi is larger than the value obtained by multiplying the maximum value maxPFG by 0.5 (S28). When the G value PFGi is larger than the maximum value maxPFG multiplied by 0.5 (S28: Yes), the face region Fi is determined to be the strong region SA, so the corresponding region An is added to the strong region memory 7c. (S29). On the other hand, when the G value PFGi is equal to or less than the maximum value maxPFG multiplied by 0.5 (S28: No), the face region Fi is determined to be the weak region WA, so the corresponding region An is set to the weak region memory 7d. Add (S30). The value to be multiplied by the maximum value maxPFG is not limited to 0.5, and 0.5 or more may be set according to the difference between the G value PFGi and the maximum value maxPFG, or 0.5. The configuration may be set as follows.

Here, the processes of S28 to S30 will be described with reference to FIG. FIG. 8 (a) is a diagram schematically showing the G value PFGi at the peak frequency BPMa, FIG. 8 (b) is a diagram schematically showing the strong region SA, and FIG. 8 (c) is a diagram. , Is a diagram schematically showing a weak region WA. For the sake of explanation, FIG. 8A shows a face region Fi having a G value PFGi equal to or less than the maximum value maxPFG multiplied by 0.3 by shading, and a G value PFGi multiplying the maximum value maxPFG by 0.3. Face area Fis that are larger than the above value and less than or equal to the maximum value maxPFG multiplied by 0.6 are indicated by diagonal lines, and face area Fis in which the G value PFGi is larger than the maximum value maxPFG multiplied by 0.6 are shown by filling. Further, in FIGS. 8B and 8C, the face region Fi corresponding to the strong region SA and the weak region WA, respectively, is represented by diagonal lines.

By the processing of S28, the face region Fi in which the G value PFGi is larger than the value obtained by multiplying the maximum value maxPFG by 0.5 is determined to be the strong region SA. Therefore, in FIG. 8A, the G value PFGi is larger than the maximum value maxPFG multiplied by 0.5, that is, the face region Fi represented by the diagonal line and the face region Fi represented by the fill are strong regions. The area An which is determined to be SA and corresponds to the face area Fi is added to the strong area memory 7c. On the other hand, in FIG. 8A, the G value PFGi is equal to or less than the maximum value maxPFG multiplied by 0.5, that is, the shaded face region Fi is determined to be the weak region WA, and the face region Fi is determined to be the weak region WA. The area An corresponding to is added to the weak area memory 7d.

The peak frequency BPMa and the positions of the strong region SA and the weak region WA vary greatly among individuals. Therefore, in the present embodiment, the strong region SA and the weak region WA are divided based on the maximum value maxPFG of the G value at the peak frequency BPMa estimated to be the peak frequency of the pulse rate of the subject H. It is possible to appropriately divide the strong region SA and the weak region WA in consideration of such individual differences for each person H.

Return to FIG. After the processing of S29 and S30, 1 is added to the counter variable i (S31). Then, it is confirmed whether the counter variable i is larger than the number j of the face area Fi (S32). When the counter variable i is the number j or less of the face area Fi (S32: No), the process of S28 or less is repeated.

On the other hand, when the counter variable i is larger than the number j of the face area Fi (S32: Yes), the division of the strong area and the weak area by the processing of S28 to S30 is completed for all the face area Fi, so that the face. The values of the image memory 7b, the strong area memory 7c, and the weak area memory 7d are added to the memory areas of the face image data 6b1, the strong area data 6b2, and the weak area data 6b3 in the detection area table 6b (S33).

After the processing of S33, "area acquisition completed" is returned to the main processing (S34), the area acquisition processing is terminated, and the process returns to the main function. On the other hand, in the process of S20, if the image for 30 seconds does not exist in the image history memory 7a (S20: No), "area not acquired" is returned to the main function (S35), and the area acquisition process is terminated. , Return to the main function.

Return to FIG. After the area acquisition process in S4, it is confirmed whether the area acquisition process returns the area acquisition completion (S6). When the area acquisition completion is returned from the area acquisition process (S6: Yes), the strong area and the weak area are normally acquired by the area acquisition process, so the processes after S7 are performed. On the other hand, when the area acquisition process returns the area acquisition failure (S6: No), a sufficient number of images are not acquired in the image history memory 7a, and the strong area SA and the weak area WA are not acquired. Therefore, by repeating the process of S1 again, the process after S7 is waited until the image for 30 seconds is accumulated in the image history memory 7a.

In the processing of S3, when the face image data 6b1 that is close to the value of the face image memory 7b exists (S3: Yes), the value of the strong region data 6b2 corresponding to the face image data 6b1 is obtained from the detection area table 6b. , The value of the weak area data 6b3 is acquired and stored in the face image memory 7b and the strong area memory 7c, respectively (S5).

When the area acquisition completion is returned from the area acquisition process after the process of S5 or in the process of S6 (S6: Yes), the area An stored in the strong area memory 7c (that is, from the value of the face image memory 7b) (that is, The G value of the pixel corresponding to the strong region SA) is acquired (S7), and the average value of the G values is stored in the strong region value memory 7e (S8). After the processing of S8, the G value of the pixel corresponding to the area An (that is, the weak area WA) stored in the weak area memory 7d is acquired from the value of the face image memory 7b (S9), and the average value of the G values is obtained. Is stored in the weak area value memory 7f (S10).

After the processing of S10, the pulse wave output processing is performed (S11). The pulse wave output processing will be described with reference to FIG. The pulse wave output process is a process for correcting disturbances such as ambient light included in the value of the strong region value memory 7e based on the value of the weak region value memory 7f and outputting to the LCD 10.

FIG. 9 is a flowchart of pulse wave output processing. First, it is confirmed whether the pulse wave output processing is immediately after the start of the main processing of FIG. 4 (S50). If it is immediately after the start of the main processing (S50: Yes), it is the initialization timing of the pulse wave output processing, so the value of the strong region value memory 7e is set to the initial value P0 of the strong region value (S51), and the weak region. The value of the value memory 7f is set to the initial value N0 of the weak region value (S52). Further, 0 is set in the strength ratio memory 7g (S53).

On the other hand, if it is not immediately after the start of the main process (S50: No), the processes of S51 to S53 are skipped. After the processing of S50 and S53, the difference between the value of the strong region value memory 7e and the initial value P0 is set in the difference value ΔP (S54), and the difference value ΔN is set to the weak region value memory 7f and the initial value N0. Set the difference (S55).

After the processing of S55, it is confirmed whether the absolute value of the difference value ΔN is larger than the difference in the vertical movement of the G value due to the pulse wave (“5” in this embodiment) (S56). The value of the difference in the vertical movement of the G value due to the pulse wave, which is compared with the absolute value of the difference value ΔN, is not necessarily limited to 5, but is a value sufficiently larger than the difference in the vertical movement of the G value. If there is, it may be 5 or more, or 5 or less.

When the absolute value of the difference value ΔN is larger than the difference in the vertical movement of the G value due to the pulse wave (S56: Yes), the current weak region value memory 7f has a disturbance due to a color change of ambient light or the like. It is determined that the value of the strong region value memory 7e, which is included and described later, needs to be corrected by the processing of S59 and S61. In such a case, the correction parameter value for correcting the value of the strong region value memory 7e is calculated from the ratio of the G value between the strong region SA and the weak region WA and the difference value ΔN by the processing of S57 to S61. The value of the strong region value memory 7e is corrected.

First, it is confirmed whether the value of the strength ratio memory 7g is 0 (S57). When the value of the strength ratio memory 7g is 0 (S57: Yes), that is, when the processing of S57 is executed for the first time after the main processing is started, 0 is set in the strength ratio memory 7g by the processing of S53. Therefore, a value obtained by dividing the difference value ΔP by the difference value ΔN, that is, the strength ratio of the G value between the strong region SA and the weak region WA is set in the strength ratio memory 7g (S58).

On the other hand, in the processing of S57, when the value of the strength ratio memory 7g is not 0 (S57: No), the strength ratio of the G value between the strong region SA and the weak region WA is already set in the strength ratio memory 7g. Therefore, the process of S58 is skipped. Then, after the processing of S57 and S58, the correction parameter value obtained by multiplying the correction value memory 7h by the value of the strength ratio memory 7g and the difference value ΔN is calculated and set in the correction value memory 7h (S59). That is, since the skin colors of the strong region SA and the weak region WA are similar, a simple calculation method (algorithm) of multiplying the strength ratio of the G value of the strong region SA and the weak region WA by the difference value ΔN. Therefore, the correction parameter value for the value of the strong region value memory 7e can be calculated. As a result, the value of the strong region value memory 7e can be corrected in a short time.

In the processing of S56, when the absolute value of the difference value ΔN is equal to or less than the difference in the vertical movement of the G value due to the pulse wave (S56: No), the difference between the current weak region value memory 7f and the initial value N0. Is small, it is determined that the disturbance due to the color change of the ambient light is not included, and it is determined that the correction by the processing of S59 and S61 is not necessary. Therefore, 0 is set in the correction value memory 7h (S60).

Then, after the processing of S59 and S60, the value of the strong region value memory 7e is subtracted from the value of the correction value memory 7h to correct the value of the strong region value memory 7e, and this value is used as a pulse wave signal. Output to LCD 10 (S61).

Here, the correction of the strong region value memory 7e in the processing of S59 to S61 will be described with reference to FIG. FIG. 10 is a graph showing the time transition of the value of the strong region value memory 7e and the time transition of the value of the corrected strong region value memory 7e when the ambient light is periodically changed in color. Graph G1 is a time transition of the value of the strong region value memory 7e, and graph G2 is a time transition of the value of the strong region value memory 7e after correction.

As shown in graph G1, when the ambient light is periodically changed in color, a waveform in which a rectangular wave having a large amplitude and a waveform having a small amplitude are combined is output as a time transition of the value of the strong region value memory 7e. To. A square wave having a large amplitude corresponds to a waveform due to a color change of ambient light, and a waveform having a small amplitude corresponds to a waveform due to a pulse wave signal. A pulse wave is detected by detecting a waveform generated by a pulse wave signal. However, since the amplitude of the square wave due to the color change of the ambient light is sufficiently larger than the amplitude of the waveform due to the pulse wave signal, the amount of change in the G value is the waveform of the pulse wave signal, especially before and after the color change of the ambient light. It becomes larger than the amplitude of, and the transition of the pulse wave signal cannot be detected accurately.

On the other hand, the time transition of the value of the strong region value memory 7e after the correction is only the waveform due to the pulse wave signal because the rectangular wave due to the change in the ambient light is corrected as shown in the graph G2. As a result, even when the color of the ambient light changes, the time transition of the value of the corrected strong region value memory 7e does not change, so that the pulse wave can be detected stably.

Return to FIG. After the processing of S61, the pulse wave output processing is finished, and the process returns to the main processing of FIG. After the pulse wave output process of S11, the process of S1 or less is repeated again.

As described above, according to the pulse wave detection device 1 in the present embodiment, the ratio of the value of the strong region value memory 7e and the value of the weak region value memory 7f is multiplied by the value based on the weak region value memory 7f. As a result, a correction parameter value for correcting the value of the strong region value memory 7e for detecting the pulse wave is calculated, and the value of the strong region value memory 7e is corrected by the correction parameter value. As a result, the disturbance due to the color change of the ambient light is removed from the value of the strong region value memory 7e without the need for a background (color checker or skin color background) for correcting the color change of the ambient light, and the pulse wave. Can be detected. Further, since the correction parameter value for the value of the strong region value memory 7e is calculated by a simple calculation method (algorithm), the value of the strong region value memory 7e can be corrected in a short time, and the pulse wave detection device 1 Response is improved.

Next, a second embodiment of the present invention will be described with reference to FIG. In the first embodiment described above, in the area acquisition process of FIG. 5, the image of the image history memory 7a is divided into the face area Fi, and the G value PFGi of the peak frequency BPMa of the frequency spectrum SFGi in the face area Fi is maximum. The face region Fi larger than the value of the value maxPFG multiplied by 0.5 was designated as the strong region SA, while the face region Fi equal to or less than the value obtained by multiplying the maximum value maxPFG by 0.5 was designated as the weak region WA. On the other hand, in the area acquisition process of the pulse wave detection device 1 in the second embodiment, the strong area SA and the weak area SA and the weak area are based on the specific part of the face in the value of the face image memory 7b acquired from the camera 3. Divide into WA. In the second embodiment, the same parts as those in the first embodiment described above are designated by the same reference numerals, and the description thereof will be omitted.

11 (a) is a flowchart of the region acquisition process of the pulse wave detection device 1 in the second embodiment, FIG. 11 (b) is a schematic diagram showing a strong region SA, and FIG. 11 (c) is a schematic diagram. It is a schematic diagram which shows the weak region WA. The origin O in FIGS. 11 (b) and 11 (c) indicates the position of the lower end of the image. In the area acquisition process in the second embodiment, first, the height h1 (see FIG. 11B) between the position of the lower end of the image and the position of the upper end of the face is acquired from the value of the face image memory 7b (S80). , The height h2 between the position of the lower end of the image and the position of the mouth (see FIG. 11B) is acquired (S81). Then, the region An corresponding to h2 to h1 in the face portion is acquired from the value of the face image memory 7b and stored in the strong region memory 7c (S82).

After the processing of S82, the height h3 (see FIG. 11C) between the position of the lower end of the image and the position of the lower end of the chin is acquired from the value of the face image memory 7b (S83), and the position of the lower end of the image is obtained. , The height h4 with respect to the position of the lower end of the face (see FIG. 11C) is acquired (S84). Then, from the value of the face image memory 7b, the area An corresponding to h4 to h3 in the face portion is acquired and stored in the weak area memory 7d (S85).

After the processing of S85, the values of the face image memory 7b, the strong area memory 7c, and the weak area memory 7d are applied to the memory areas of the face image data 6b1, the strong area data 6b2, and the weak area data 6b3 in the detection area table 6b. Addition (S86), return "area acquisition completed" to the main process (S87), and end the area acquisition process.

In general, the portion of the face above the mouth is defined as the region where the pulse wave is strongly detected, and the portion below the chin is defined as the region where the pulse wave is weakly detected. Therefore, in the second embodiment, h1 to h2 in the face portion, that is, the region An corresponding to the area above the mouth in the face is determined to be the strong region SA and is stored in the strong region memory 7c. On the other hand, the region An corresponding to h3 to h4 in the face portion is determined to be the weak region WA and is stored in the weak region memory 7d. Therefore, since the strong region SA and the weak region WA can be divided from the value of the face image memory 7b acquired from the camera 3, the pulse wave can be detected immediately after the start of the main process, and the pulse wave detection device can be used. The response of 1 is improved.

Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and it is easy that various improvements and changes can be made without departing from the spirit of the present invention. It can be inferred from.

In the above embodiment, the control program 6a is executed by the pulse wave detection device 1. However, the present invention is not necessarily limited to this, and the control program 6a may be stored in a personal computer, a smartphone, a tablet terminal, or the like to execute the control program 6a.

In the above embodiment, the pulse wave detection device 1 is configured to output the detected pulse wave signal to the LCD 10. However, the present invention is not necessarily limited to this, and the detected pulse wave signal may be stored in the HDD 6, and the detected pulse wave signal is transmitted to an external server, PC, or the like by a communication device (not shown). It may be configured to be performed.

Further, the pulse wave detection device 1 may be mounted on a vehicle such as an automobile, and the pulse wave signal detected by the pulse wave detection device 1 may be analyzed to monitor the physical condition of the driver (target person H). If it is determined that the driver is not in good physical condition, an alarm sound may be output to notify the surroundings, or the vehicle may communicate with the central center and the central center may rescue the driver. It may be a configuration for transmitting a request, a configuration for stopping the vehicle, or a configuration for controlling the vehicle.

In the above embodiment, when the face image acquired from the camera 3 is decomposed into the RGB color space, the pulse wave is detected by the change in the G value. However, the present invention is not limited to this, and a pulse wave may be detected by a change in the Q value (chromaticity: cold color system) when the face image is decomposed into the YIQ color space. Since the Q value is a color component that is resistant to changes in brightness, it is possible to detect pulse waves that are resistant to changes in brightness by using the Q value for detecting pulse waves. Further, a color component other than the G value in the RGB color space or a color component other than the Q value in the YIQ color space may be used, or a pulse wave may be used by using the color component in the color space other than the RGB and YIQ color spaces. It may be configured to detect.

In the above embodiment, the pulse wave is detected by the face image acquired from the camera 3. However, the present invention is not limited to this, and the pulse wave may be detected by an image of a part of the human body in which the pulse wave is strongly detected other than the face.

In the above embodiment, the area acquisition process is configured as a separate process in the first embodiment and the second embodiment. However, the present invention is not necessarily limited to this, and the area acquisition processing of the first embodiment and the second embodiment may be used properly according to the situation or the like. For example, when the image for 30 seconds is not stored in the image history memory 7a immediately after the execution of the main function, the area acquisition process in the second embodiment is performed from the value of the face image memory 7b to the strong area SA and the weak area WA. Is acquired, and the pulse wave output processing of S11 is performed based on the acquired strong region SA and weak region WA.

On the other hand, the area acquisition process in the first embodiment is executed as a background process, and when the image for 30 seconds is stored in the image history memory 7a, the strong area SA and the weak area WA are acquired, and thereafter. Performs the pulse wave output processing of S11 based on the strong region SA and the weak region WA acquired in the region acquisition processing in the first embodiment. As a result, the pulse wave can be detected immediately after the main processing is executed, and the strong region SA and the weak region WA, which are acquired in the region acquisition processing in the first embodiment after that, in consideration of the individual difference for each subject H. The pulse wave can be detected based on the above.

1 Pulse wave detection device 3 Camera (photographing means)
6a Control program (pulse wave detection program)
6b Detection area table (pulse wave storage means)
BPMa Peak frequency S3 Image detection means S7, S8 Strong region measurement means S9, S10 Weak region measurement means S21 Frequency acquisition means S23 Time transition calculation means S24 Frequency spectrum calculation means S25 Predetermined component value acquisition means S28 to S30 Strength identification means S33, S86 Additional storage means S59 Part of correction means S61 Part of correction means, output means S81 Mouth detection means S83 Jaw detection means

Claims (11)

A means of photographing the area including the subject's skin for at least a predetermined time, and
With respect to the image of the skin area taken by the photographing means, the strength identifying means for identifying the strong pulse wave region and the weak pulse wave region, and
A strong region measuring means for measuring a signal of an image taken by the photographing means with respect to a region specified as a region having a strong pulse wave by the strength specifying means.
A weak region measuring means for measuring a signal of an image taken by the photographing means for a region specified as a region where a pulse wave is weak by the strength specifying means, and a weak region measuring means.
Based on the signal of the weak region of the pulse wave measured by the weak region measuring means, the correction means for correcting the signal of the strong region of the pulse wave measured by the strong region measuring means, and the correction means.
A pulse wave detection device including an output means for outputting a pulse wave signal corrected by the correction means. With respect to the image taken by the photographing means, the peak frequency acquisition means for acquiring the peak frequency of the pulse rate, and
A time transition calculation means for calculating the time transition of a predetermined component for each decomposition region obtained by decomposing an image captured by the photographing means,
For each of the decomposition regions, a frequency spectrum calculating means for calculating the frequency spectrum of the time transition of the predetermined component calculated by the time transition calculating means, and
Each of the decomposition regions is provided with a predetermined component value acquisition means for acquiring the value of the predetermined component at the peak frequency acquired by the peak frequency acquisition means among the frequency spectra calculated by the frequency spectrum calculation means.
The strength specifying means is characterized in that each decomposition region is specified into a strong region and a weak region of the pulse wave based on the value of the predetermined component acquired by the predetermined component value acquiring means. The pulse wave detection device according to claim 1. The pulse wave detection device according to claim 2, wherein the predetermined component is a G component in an RGB color space. The pulse wave detection device according to claim 2, wherein the predetermined component is a Q component in the YIQ color space. A pulse wave storage means for storing a strong pulse wave region and a weak pulse wave region in association with an image,
It is provided with an image detecting means for detecting an image similar to the image taken by the photographing means from the images stored in the pulse wave storage means.
The strength specifying means identifies a region with a strong pulse wave stored in the pulse wave storage means as a region with a strong pulse wave in association with the image detected by the image detecting means, while the image detecting means The invention according to any one of claims 1 to 4, wherein a weak pulse wave region stored in the pulse wave storage means is specified as a weak pulse wave region in association with the detected image. Pulse wave detector. The pulse wave detection device according to any one of claims 1 to 5, wherein the region including the skin of the subject photographed by the photographing means is a region including the face of the subject. A mouth detecting means for detecting a portion of the subject's mouth in the area including the subject's face photographed by the photographing means is provided.
The pulse wave detecting device according to claim 6, wherein the strength specifying means identifies a region in which the pulse wave is strong based on a portion of the mouth detected by the mouth detecting means. A jaw detecting means for detecting a jaw portion of the subject in the area including the face of the subject photographed by the photographing means is provided.
The pulse wave detection according to claim 6 or 7, wherein the strength specifying means identifies a region where the pulse wave is weak based on the jaw portion detected by the jaw detecting means. apparatus. A pulse wave storage means for storing a strong pulse wave region and a weak pulse wave region in association with an image,
It is provided with an additional storage means for additionally storing a strong region and a weak region of the pulse wave specified by the strength / weakness specifying means in the pulse wave storage means in association with the image designated as the specific target. The pulse wave detection device according to any one of claims 1 to 8. The photographing means is for photographing the passengers of the vehicle.
The pulse wave detection device according to any one of claims 1 to 9, further comprising a monitoring means for monitoring the physical condition of the occupant using the pulse wave signal output by the output means. An image acquisition function that acquires images of the area including the subject's skin for at least a predetermined period of time,
With respect to the image of the skin area acquired by the image acquisition function, a strength identification function for identifying a strong pulse wave region and a weak pulse wave region, and a strength identification function
A strong region measurement function that measures the signal of the image acquired by the image acquisition function for a region specified as a region with a strong pulse wave by the strength identification function, and
A weak region measurement function that measures the signal of the image acquired by the image acquisition function for a region specified as a region where the pulse wave is weak by the strength identification function, and a weak region measurement function.
Based on the signal in the weak region of the pulse wave measured by the weak region measurement function, the correction function for correcting the signal in the strong region of the pulse wave measured by the strong region measurement function and
A pulse wave detection program characterized in that a computer realizes an output function that outputs a pulse wave signal corrected by the correction function.

Images