JP6102433B2 - Pulse wave detection program, pulse wave detection method, and pulse wave detection device - Google Patents

Description

  The present invention relates to a pulse wave detection program, a pulse wave detection method, and a pulse wave detection device.

  Techniques for detecting blood volume fluctuations, so-called pulse waves, from images taken by a subject are known. For example, a conventional technique has been proposed in which a subject's face is imaged with a video camera, a concentration is calculated from the imaging data, and a heart rate and a respiration rate are measured.

JP-A-2005-218507

  By the way, the pulse wave is detected from the change in luminance of the image of the skin portion accompanying the change in blood flow of the capillary blood vessels. The luminance change related to the pulse wave is very small. For this reason, in detecting the pulse wave, in order to accurately detect the luminance change related to the pulse wave, it is desired to measure the luminance change widely using the skin portion of the face. On the other hand, when the luminance change measurement area is a wide area of the face, the background around the face included in the image is included in the measurement area, and the luminance change becomes large due to the influence of the background, and the pulse wave is detected. Accuracy may be reduced.

  In view of this, it is conceivable to perform face detection on each frame image to identify a face area, and to measure a change in luminance from the measurement area using the central portion of the face area as a measurement area. For example, it is conceivable that the central portion narrowed down to the center of the face area by the distance or ratio from the face frame that is the outer edge of the face area is used as the measurement area.

  However, even when the measurement region is narrowed down to the center portion depending on the distance or ratio from the face frame, the pulse wave detection accuracy may be reduced. For example, depending on the orientation of the face of the subject, the measurement region may include a background, and the pulse wave detection accuracy is reduced.

  FIG. 18 is a diagram for explaining an example of a change in the measurement region depending on the face orientation of the subject. In the example of FIG. 18, images of a state where the subject is directly facing, a state where the subject is facing up, a state where the subject is facing right, and a state where the subject is facing down are shown. In the example of FIG. 18, the measurement area 201 is a central portion that is narrowed down to the center of the face area by the distance or ratio from the face frame 200 that is the outer edge of the face area including the eyes, nose, mouth, cheek, and forehead of the subject's face. .

  In the image with the subject facing upward, the measurement region 201 includes the nose of the subject, the outer edge of the eye, and the mouth. In addition, in the image with the subject facing right, the measurement area 201 includes a background and a mouth. Further, in the image in a state where the subject faces downward, the measurement area 201 includes a nose, eyes, and a mouth. When such a measurement region 201 is used for pulse wave detection, there are cases where the luminance of the nose changes, the blinking of the eyes, the background becomes noise, and the detection accuracy of the heart rate is lowered.

  An object of one aspect is to provide a pulse wave detection program, a pulse wave detection method, and a pulse wave detection device that can suppress a decrease in detection accuracy of a pulse wave.

  According to one aspect of the present invention, the pulse wave detection program causes a computer to execute processing for detecting a position of a predetermined part of the face of the subject in the image from an image of the subject taken by the imaging device. The pulse wave detection program causes a computer to execute a process of detecting a rotation angle of the face direction of the subject with respect to a predetermined direction from an image taken by the imaging device. The pulse wave detection program causes the computer to execute processing for calculating the position of the pulse wave measurement region of the subject's face from the position of the predetermined part and the detection result of the rotation angle. The pulse wave detection program causes a computer to execute processing for measuring a pulse wave from the calculated measurement region.

  Advantageous Effects of Invention According to one aspect of the present invention, it is possible to suppress a decrease in pulse wave detection accuracy.

FIG. 1 is a block diagram showing a functional configuration of the pulse wave detection device. FIG. 2A is a diagram illustrating an example of a histogram of each region of an image. FIG. 2B is a diagram illustrating an example of a histogram of each region of the image. FIG. 2C is a diagram illustrating an example of a histogram of each region of the image. FIG. 2D is a diagram illustrating an example of a histogram of each region of the image. FIG. 3 is a diagram for explaining an arrangement pattern of facial parts. FIG. 4 is a diagram for explaining a change in the size of the region depending on the horizontal direction of the face of the subject in the image. FIG. 5 is a diagram illustrating an example of the spectrum of each signal of the R component and the G component multiplied by the weighting factor. FIG. 6 is a diagram for explaining a change in the size of the region depending on the vertical direction of the face of the subject in the image. FIG. 7 is a diagram for explaining a change in the size of the region due to a change in the vertical direction of the face of the subject. FIG. 8 is a diagram illustrating the passband. FIG. 9 is a diagram illustrating an example of the spectrum of each signal of the G signal and the R signal. FIG. 10 is a diagram illustrating an example of a spectrum of each signal of the R component multiplied by the G component and the correction coefficient k. FIG. 11 is a diagram illustrating an example of a spectrum after calculation. FIG. 12 is a block diagram illustrating a functional configuration of the measurement unit. FIG. 13 is a diagram illustrating an example of a state in which the background is included in the measurement region. FIG. 14 is a diagram illustrating an example of a result of extracting pixels in the passband. FIG. 15 is a flowchart showing the procedure of the calibration process. FIG. 16 is a flowchart showing the procedure of the detection process. FIG. 17 is a diagram for describing an example of a computer that executes a pulse wave detection program. FIG. 18 is a diagram for explaining an example of a change in the measurement region depending on the face orientation of the subject.

  Hereinafter, embodiments of a pulse wave detection program, a pulse wave detection method, and a pulse wave detection device disclosed in the present application will be described in detail with reference to the drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

[Configuration of pulse wave detector]
FIG. 1 is a block diagram showing a functional configuration of the pulse wave detection device. A pulse wave detection device 10 shown in FIG. 1 is a device that detects a pulse wave of a subject using an image taken by the subject without bringing a measuring instrument into contact with the living body under ambient light such as sunlight or room light. It is. The “pulse wave” here refers to an index representing fluctuations in blood volume, that is, increase or decrease in blood flow, and includes so-called heart rate and heart rate waveform. Examples of the pulse wave detection device 10 include a mobile terminal device such as a smartphone, a PDA (Personal Digital Assistant), and a mobile phone. One aspect of the pulse wave detection device 10 is realized by causing a smartphone to pre-install or install an application program for detecting a pulse wave and operating the application program. The pulse wave detection device 10 may be an information processing device such as a desktop PC (personal computer), a tablet PC, or a notebook PC.

  As shown in FIG. 1, the pulse wave detection device 10 includes a camera 20, a display unit 21, an input unit 22, a communication I / F (interface) unit 23, a storage unit 24, and a control unit 25. . Note that the pulse wave detection device 10 may include various functional units included in known mobile terminal devices in addition to the functional units illustrated in FIG. 1. For example, the pulse wave detection device 10 may include an antenna, a carrier communication unit that performs communication via a carrier network, a GPS (Global Positioning System) receiver, and the like.

  The camera 20 is an imaging device using an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). For example, the camera 20 can be equipped with three or more light receiving elements such as R (red), G (green), and B (blue). As an implementation example of the camera 20, a digital camera or a Web camera may be connected via an external terminal, or the camera can be used when the camera is mounted from the time of shipment like a portable terminal. . In addition, although the case where the pulse wave detection device 10 has the camera 20 is illustrated here, the pulse wave detection device 10 does not necessarily have the camera 20 when an image can be acquired via a network or a storage device. .

  The display unit 21 is a display device that displays various types of information. Examples of the display unit 21 include display devices such as an LCD (Liquid Crystal Display) and a CRT (Cathode Ray Tube). The display unit 21 displays various information. If it is not necessary to display information through the pulse wave detection device 10, the display unit 21 may be omitted. It can also be displayed on the display unit of another device.

  The input unit 22 is an input device that inputs various types of information. For example, as the input unit 22, an input device such as a mouse, a keyboard, various buttons provided on the pulse wave detection device 10, and a transmissive touch sensor provided on the display unit 21 can be cited. In the example of FIG. 1, since the functional configuration is shown, the display unit 21 and the input unit 22 are separately provided. For example, the display unit 21 and the input unit 22 such as a touch panel are integrally configured. May be.

  The communication I / F unit 23 is an interface that controls communication with other devices. As an aspect of the communication I / F unit 23, a network interface card such as a LAN card can be employed. The communication I / F unit 23 transmits / receives various information to / from other devices via a network (not shown). For example, the communication I / F unit 23 transmits information on the detected pulse wave to the server device.

  The storage unit 24 is a device that stores various types of information. For example, the storage unit 24 includes a semiconductor memory device such as a RAM (Random Access Memory) and a flash memory, and a storage device such as a hard disk, an SSD (Solid State Drive), and an optical disk.

  The storage unit 24 stores an OS (Operating System) executed by the control unit 25 and various programs. Furthermore, the storage unit 24 stores various data used in programs executed by the control unit 25. For example, the storage unit 24 stores calculation formula information 30, calculation parameter information 31, angle parameter information 32, and passband information 33.

  The calculation formula information 30 is data that stores various calculation formulas used for calculation of the pulse wave measurement region of the subject's face. The calculation parameter information 31 is data in which values of various parameters set in various arithmetic expressions are stored.

  Here, the luminance change related to the pulse wave is very small. For this reason, in detecting a pulse wave, it is preferable to measure the luminance change widely using the skin portion of the face in order to accurately detect the luminance change related to the pulse wave. On the other hand, when the luminance change measurement area is a wide area of the face, the background around the face included in the image is included in the measurement area, and the luminance change becomes large due to the influence of the background, and the pulse wave is detected. Accuracy may be reduced.

  FIG. 2A is a diagram illustrating an example of a histogram of each region of an image. In the example of FIG. 2A, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 70A including the eye, nose, mouth, cheek, and forehead. In the example of FIG. 2A, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 70B including the nose and mouth. The rectangular area 70A includes a surrounding background, and the eyes of the subject are reflected. When the image of the rectangular area 70A is used for pulse wave detection, the background or blinking of the eyes may become noise, leading to a decrease in heart rate detection accuracy. The rectangular area 70B shows the mouth of the subject. When this rectangular region 70B is used for detecting a pulse wave, the movement of the mouth may become noise, leading to a decrease in heart rate detection accuracy.

  FIG. 2B is a diagram illustrating an example of a histogram of each region of the image. In the example of FIG. 2B, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 71A including the eye. In the example of FIG. 2B, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 71B including the nose. In the example of FIG. 2B, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 71C including the mouth. In the rectangular area 71A, the eyes of the subject are shown. When the image of the rectangular area 71A is used for pulse wave detection, blinking of the eyes may cause noise and decrease in heart rate detection accuracy. The rectangular area 71A shows the subject's nose. The nose portion may have a large change in luminance due to the way light strikes. When the image of the rectangular area 71B is used for pulse wave detection, the luminance change of the nose portion may become noise and cause a decrease in heart rate detection accuracy. The rectangular area 71C shows the mouth of the subject. When the image of the rectangular area 71C is used for pulse wave detection, the movement of the mouth may become noise, leading to a decrease in heart rate detection accuracy.

  FIG. 2C is a diagram illustrating an example of a histogram of each region of the image. In the example of FIG. 2C, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 72A including the right cheek. In the example of FIG. 2C, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 72B including the left cheek. In the example of FIG. 2C, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 72C including the forehead. The rectangular area 72A to the rectangular area 72C can extract a skin color range, and are preferable areas for detecting a pulse wave.

  FIG. 2D is a diagram illustrating an example of a histogram of each region of the image. In the example of FIG. 2D, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular region 73A of the hair portion of the head. In the example of FIG. 2D, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 73B of the background portion. In the example of FIG. 2D, a histogram of pixel values is shown for each of the R, G, and B components of the rectangular area 73C of the subject's clothes. Each of the rectangular area 73A to the rectangular area 73C is an area where a pulse wave cannot be detected because the skin portion is hardly included.

  For this reason, in this embodiment, pulse waves are measured using the right cheek, left cheek and forehead regions as measurement regions.

By the way, a person has the same arrangement pattern of facial parts such as eyes, ears, nose, mouth, cheeks, and forehead included in the face. For example, the right cheek region and the left cheek region suitable for pulse wave measurement are located in the vicinity between the eyes and the mouth with respect to the vertical direction. The right cheek region is located near the position of the right eye with respect to the horizontal direction. The left cheek region is located near the position of the left eye with respect to the horizontal direction. Further, for example, a forehead region suitable for pulse wave measurement is located above the eye with respect to the vertical direction. The forehead region is located in the vicinity between the right eye and the left eye with respect to the horizontal direction. Therefore, the position of the other part can be estimated from the position of the part of the face of the subject. FIG. 3 is a diagram for explaining an arrangement pattern of facial parts. In the example of FIG. 3, the human face is shown with the eyes, mouth, and left cheek simplified. For example, when the coordinates of the right eye are (x ₁ , y ₁ ), the coordinates of the left eye are (x ₂ , y ₂ ), and the coordinates of the mouth are (x ₃ , y ₃ ), it corresponds to the left cheek. The coordinates (X ₁ , Y ₁ ) and (X ₂ , Y ₂ ) of the two opposing vertices P1 and P2 of the rectangular area can be expressed as the following equations (1) and (2).

(X ₁ , Y ₁ ) = (α · x ₁ + (1−α) x ₂ , β · y ₂ + (1−β) y ₃ ) (1)
(X ₂ , Y ₂ ) = (γ · x ₁ + (1−γ) x ₂ , δ · y ₂ + (1−δ) y ₃ ) (2)

  Here, each person has a difference in the position of the part of the face. Therefore, in Equations (1) and (2), various parameters of α, β, γ, and δ are provided in order to calculate the position of the measurement region corresponding to the subject. For example, when α = 0 and β = 0.9, the vertex P1 is the position of the left eye in the horizontal direction and the position of 90% height of the eyes and mouth in the vertical direction.

  The calculation formula information 30 stores a calculation formula used to calculate a measurement region from a predetermined part of the subject's face. In this embodiment, the right cheek, left cheek, and forehead regions are calculated as the measurement regions from the positions of the eyes and mouth as the predetermined portions. For this reason, the calculation formula information 30 stores calculation formulas for calculating the right cheek, left cheek, and forehead areas as measurement areas from the positions of the eyes and mouth. For example, the arithmetic expression information 30 stores the above expressions (1) and (2). Note that the predetermined part serving as a reference for calculating the measurement region is not limited to the eye and the mouth, and any part such as the eye, nose, mouth, eyebrows, and ears may be used. For example, the eye or nose may be used.

  The calculation parameter information 31 stores parameter values provided in each calculation expression. For example, the calculation parameter information 31 stores values of various parameters α, β, γ, and δ provided in the above equations (1) and (2). In this embodiment, the parameter value of the calculation parameter information 31 is stored by the parameter storage unit 43 described later. However, the parameter value corresponding to the standard arrangement pattern of the facial part is obtained in advance and calculated. The parameter information 31 may be stored.

  The angle parameter information 32 is data that stores parameters for correcting the measurement region in accordance with the orientation of the subject's face in the image.

  Here, the size of each area of the face changes according to the orientation of the face of the subject in the image. For example, the cheek region suitable for pulse wave measurement changes in size in the image according to the orientation of the subject's face. FIG. 4 is a diagram for explaining a change in the size of the region depending on the horizontal direction of the face of the subject in the image. In the example of FIG. 4, a human face is shown simplified by eyes, nose, mouth, right cheek, and left cheek. The example on the left side of FIG. 4 shows a state of facing each other. The example on the right side of FIG. 4 shows a state in which the face is rotating in the horizontal direction. As shown on the left side of FIG. 4, in the state of facing, the right cheek region 80R where the right cheek is observed and the left cheek region 80L where the left cheek is observed have substantially the same size. On the other hand, as shown on the right side of FIG. 4, in the state where the face is rotated in the horizontal direction, the sizes of the right cheek region 80R and the left cheek region 80L change, and one is small and the other is large. In the example of FIG. 4, the size of the right cheek region 80R is increased, and the size of the left cheek region 80L is decreased.

FIG. 5 is a diagram for explaining a change in the size of the region due to a change in the horizontal direction of the subject's face. In the example of FIG. 5, changes in the area of the right cheek region 80R and the area of the left cheek region 80L according to the lateral rotation angle θ _y of the face direction of the subject with respect to the facing state are shown on the upper side. . In the example of FIG. 5, changes in the horizontal width of the right cheek region 80R and the horizontal width of the left cheek region 80L according to the rotation angle θ _y are shown on the lower side. For example, when the shape of the face is regarded as an elliptical sphere, the right cheek region 80R and the left cheek region 80L have a rotation angle θ within a predetermined range from the rotation angle θ _y = 0 as shown in the upper side of FIG. It can be considered that there is a linear relationship between _y and area. In the case of rotation only in the horizontal direction, assuming that the vertical widths of the right cheek region 80R and the left cheek region 80L are constant, the right cheek region 80R and the left cheek region 80L have predetermined rotation angles θ _y = 0. In this range, it can be considered that there is a linear relationship between the rotation angle θ _y and the lateral width.

Therefore, even when the orientation of the subject's face changes in the horizontal direction, the position of the measurement region can be calculated from the rotation angle θ _{y in} the horizontal direction of the face. For example, as in the following formulas (3) and (4), the coordinates (X ₁ , Y ₁ ) of the apex P1 of the right cheek are changed to (X ′ ₁ , Y ′ ₁ ) according to the rotation angle θ _y. It can be expressed that the coordinates (X ₂ , Y ₂ ) of the vertex P2 of the right cheek change to (X ′ ₂ , Y ′ ₂ ).

(X ′ ₁ , Y ′ ₁ ) = (X ₁ , Y ₁ ) (3)
(X ′ ₂ , Y ′ ₂ ) = (κ · X ₁ + (1−κ) · X ₂ , Y ₂ ) (4)

This κ is determined according to the rotation angle θ _y . In the angle parameter information 32, a value of κ is obtained and stored in advance for each rotation angle θ _y as a lateral angle parameter. For example, when the rotation angle θ _y = 0 °, κ = 0, and when the rotation angle θ _y = 30 °, κ = 1. When the rotation angle θ _y exceeds 30 degrees, the right cheek region 80R and the left cheek region 80L have no linear relationship between the rotation angle θ _y and the lateral width. Therefore, it is assumed that the calculation is performed within the range where the rotation angle θ _y is ± 30 degrees.

  Similarly, the measurement area changes with respect to the change in the vertical direction of the subject's face.

  FIG. 6 is a diagram for explaining a change in the size of the region depending on the vertical direction of the face of the subject in the image. In the example of FIG. 6, a human face is shown simplified by eyes, nose, mouth, left cheek, and forehead. When the face orientation of the left cheek region 80L where the left cheek is observed and the forehead region 81 where the forehead is observed shown in FIG. 6 is rotated in a vertical direction called so-called tilt, the size of the region changes.

FIG. 7 is a diagram for explaining a change in the size of the region due to a change in the vertical direction of the face of the subject. In the example of FIG. 7, the upper side, the change in the area of the left cheek region 80L of the area and the amount area 81 corresponding to the rotation angle theta _x longitudinal direction of the face of the subject to directly facing state is shown. In the example of FIG. 7, on the lower side, the change in height of the vertical width of the left cheek regions 80L corresponding to the rotation angle theta _x and forehead region 81 is shown. As shown in the upper side of FIG. 7, the area of the left cheek region 80L decreases as the rotation angle θ _x increases. On the other hand, the area of the forehead region 81 increases in size as the rotation angle θ _x increases. For example, assuming that the shape of the face is an elliptic sphere, the left cheek region 80L and the forehead region 81 have a linear relationship between the rotation angle θ _x and the area within a predetermined range from the rotation angle θ _x = 0. Can be considered.

Thus, it calculates the position of the measurement region from the rotational angle theta _x longitudinal face even when the direction of the face of the subject is changed in the vertical direction. For example, as in the following formulas (5) and (6), the coordinates (X ₁ , Y ₁ ) of the apex P1 of the right cheek are changed according to the rotation angle θ _x which is a change in the vertical direction of the face. _{(X "1, Y" 1} ) to change the coordinates of the vertices of the right cheek P2 (X _2, Y ₂₎ is can be represented when changes _{(X "2, Y" 2} ).

(X ″ ₁ , Y ″ ₁ ) = (X ₁ , Y ₁ ) (5)
(X ″ ₂ , Y ″ ₂ ) = (X ₂ , λ · Y ₁ + (1−λ) · Y ₂ ) (6)

This λ is determined according to the rotation angle θ _x . In the angle parameter information 32, a value of λ is obtained and stored in advance for each rotation angle θ _x as an angle parameter in the vertical direction. For example, when the rotation angle θ _x = 0 °, λ = 0, and when the rotation angle θ _x = 15 °, λ = 1. When the rotation angle θ _x exceeds 15 degrees, the left cheek region 80L and the forehead region 81 have no linear relationship between the rotation angle θ _x and the vertical width in the vertical direction. Therefore, it is assumed that the calculation is performed within the range where the rotation angle θ _x is ± 15 degrees.

  The arithmetic expression information 30 stores an arithmetic expression used for correcting the measurement region in accordance with a change in the orientation of the subject's face. In this embodiment, the measurement areas are the right cheek, left cheek, and forehead. For this reason, the arithmetic expression information 30 stores arithmetic expressions used for correcting the right cheek, left cheek, and forehead areas. For example, the arithmetic expression information 30 stores the above expressions (4) to (6).

The angle parameter information 32 stores parameter values used in the calculation formula for correcting the measurement region. For example, the angle parameter information 32, the above equation (4), (5) and the value of the rotation angle theta _y every κ used in the above equation (5), for each rotational angle theta _x used in (6) The value of λ is stored.

  The passband information 33 is data in which a standard skin color range is stored as a passband for each measurement region. In this embodiment, the right cheek, left cheek, and forehead areas are used as measurement areas. In the passband information 33, standard skin color ranges of the right cheek, left cheek, and forehead areas are used as passbands. Remembered. In the present embodiment, the pass band of the pass band information 33 is stored by the pass band storage unit 44 described later. However, a standard skin color range is obtained in advance and stored in the pass band information 33. It is good as it is.

  The control unit 25 is a device that controls the pulse wave detection device 10. As the control unit 25, an electronic circuit such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array) can be employed. The control unit 25 has an internal memory for storing programs defining various processing procedures and control data, and executes various processes using these. The control unit 25 functions as various processing units by operating various programs. For example, the control unit 25 includes an imaging control unit 40, a face part detection unit 41, a face direction detection unit 42, a parameter storage unit 43, a passband storage unit 44, a calculation unit 45, a measurement unit 46, And an output control unit 47.

  The shooting control unit 40 is a processing unit that controls the camera 20 to control image shooting. For example, the imaging control unit 40 controls the camera 20 to continuously capture images when performing pulse wave detection or calibration. For example, when a predetermined operation for instructing the input unit 22 to start detection of a pulse wave or a predetermined operation for instructing execution of calibration is performed, the imaging control unit 40 controls the camera 20 to perform a predetermined frame rate. Take images continuously with. The frame rate may be any period that can sample a person's pulse wave. In the present embodiment, moving images are taken by the camera 20, and pulse waves are detected from each image taken. Examples of the frame rate include a general moving image shooting period such as 24 fps (frame per second), 30 fps, 60 fps, and the like. Note that the imaging control unit 40 may cause the display unit 21 to display an image captured by the camera 20.

  The face part detection unit 41 is a processing unit that detects a predetermined part of the face from an image taken by the subject with the camera 20. Any method of detecting the face part may be used. In this embodiment, for example, the eyes and mouth are detected as the predetermined part of the face, but other parts may be used. For example, the face part detection unit 41 detects a predetermined part such as a subject's eye, nose, or mouth by executing image processing such as template matching on the image. And the face part detection part 41 detects the position of the predetermined part of the test subject's face in the image. For example, the face part detection unit 41 detects a predetermined position such as the center of gravity or the center position of the predetermined part as the position of the predetermined part.

  The face part detection unit 41 detects the position of the measurement region when performing calibration of various parameters used when calculating the position of the measurement region. In the present embodiment, the face part detection unit 41 detects the positions of the right cheek, left cheek, and forehead regions, for example.

The face direction detection unit 42 is a processing unit that detects a rotation angle of the face direction with respect to a predetermined direction from an image taken by the subject with the camera 20. Any method of detecting the orientation of the face may be used. In this embodiment, the rotation angles θ _x and θ _y of the face direction with respect to the direction in which the subject faces the camera 20 are detected as the face direction with respect to the predetermined direction. For example, in the image, as shown on the left side of FIG. 4, the mouth is located near the center of the left and right eyes when the faces are facing each other. In the image, as shown on the right side of FIG. 4, the mouth is located on the left side of the center of the left and right eyes with the face facing left, and the larger the rotation angle, the greater the deviation from the center. Become. Therefore, for example, the face orientation detection unit 42 detects the lateral rotation angle θ _y of the face orientation from the deviation of the mouth position from the center of the left and right eyes. In addition, for example, the arrangement pattern of various parts such as eyes, nose, mouth and the like with the face facing up is stored, and the rotation angle θ _{x of the} face direction from the change of the arrangement pattern of various parts in the photographed image , Θ _y may be detected. The predetermined direction is not limited to the direction in which the subject faces the camera 20. For example, the rotation angle may be obtained on the basis of the direction in which the subject tilts the face with respect to the camera 20.

  The parameter storage unit 43 is a processing unit that performs calibration according to the subject. For example, when a predetermined operation for instructing the input unit 22 to start detecting a pulse wave is performed, the parameter storage unit 43 prompts the subject to face the camera 20 in order to perform calibration. For example, the parameter storage unit 43 displays a message on the display unit 21 and prompts the subject to face the camera 20. Thereby, the subject in the state of facing the camera 20 is photographed with the camera 20. The face part detection unit 41 described above detects the position of the predetermined part and the position of the measurement region from the captured image. In the present embodiment, the face part detection unit 41 detects, for example, the positions of the eyes and mouth as predetermined parts, and detects the positions of the right cheek, left cheek, and forehead areas as measurement areas.

The parameter storage unit 43 substitutes the position of the predetermined part detected by the face part detection unit 41 and the position of the measurement area into the arithmetic expression calculated from the predetermined part of the face stored in the arithmetic expression information 30. Thus, the values of various parameters provided in each arithmetic expression are calculated. For example, the parameter storage unit 43 includes the coordinates of the right eye (x ₁ , y ₁ ), the coordinates of the left eye (x ₂ , y ₂ ), the coordinates of the mouth (x ₃ , y ₃ ), the vertex P 1 of the left cheek region, The values of various parameters α, β, γ, and δ are calculated by substituting the coordinates (X ₁ , Y ₁ ) and (X ₂ , Y ₂ ) of P2 into the equations (1) and (2). The parameter storage unit 43 stores the calculated parameter values in the calculation parameter information 31. When the face direction detection unit 42 detects the face direction from the arrangement pattern of various parts with the face facing, the parameter storage unit 43 stores the various parts of the subject facing the camera 20. An arrangement pattern may be obtained and stored in the storage unit 24.

  The passband storage unit 44 is a processing unit that sets the passband information 33. For example, when performing calibration according to the subject, the passband storage unit 44 includes, for each measurement region, each pixel included in the measurement region from an image of the subject's face that is captured by the camera 20. To obtain a histogram of pixel values for each wavelength component. Then, the passband storage unit 44 specifies a standard pixel value range for each wavelength region with respect to the peak pixel value for each wavelength component, and a standard pixel value range for each specified wavelength component. Is set in the passband information 33 as a passband.

  FIG. 8 is a diagram illustrating the passband. In the example of FIG. 8, an example in which a histogram of pixel values is obtained for each of the R, G, and B components of the left cheek region 80L is shown. The passband storage unit 44 specifies a standard pixel value range for each of R, G, and B components with reference to the peak pixel value. For example, the passband storage unit 44 specifies a standard pixel value range of a pixel value range having a frequency of a predetermined ratio of the peak frequency with reference to the peak pixel value in the histogram. For example, the passband storage unit 44 specifies a standard pixel value range as a so-called half-value width range that is half the peak frequency. The standard pixel value range may be a range having a predetermined width centered on the peak pixel value.

  The skin color of the subject included in the measurement region varies depending on the shooting environment, the skin tone of the subject himself, and the like. Therefore, the passband storage unit 44 specifies the range of the standard pixel value of the subject's skin color from the actually captured image, and sets it as the passband in the passband information 33, so that the actual subject's skin color is set. The skin part can be extracted.

  The calculation unit 45 calculates the position of the pulse wave measurement region in the image of the subject taken by the camera 20 from the position of the predetermined part of the face detected by the face part detection unit 41. For example, the calculation unit 45 reads, from the calculation formula information 30, formulas that calculate the positions of the right cheek, left cheek, and forehead regions that are measurement regions from the positions of the eyes and mouth of the face. For example, the calculation unit 45 reads the above formulas (1) and (2) from the calculation formula information 30 as formulas for calculating the right cheek region from the positions of the eyes and mouth. In these equations, parameters are provided to adjust the position according to the individual. The value of this parameter is stored in the calculation parameter information 31. For example, the calculation parameter information 31 stores the values of the parameters α, β, γ, and δ in the above formulas (1) and (2).

The calculation unit 45 reads out the parameter value from the calculation parameter information 31 and sets it in an equation, and calculates the position of the measurement region from the position of a predetermined part of the face by performing an operation using the equation. For example, the calculation unit 45 sets the values of the parameters α, β, γ, and δ in the expressions (1) and (2), and calculates the coordinates of the right eye, the left eye, and the mouth detected by the face part detection unit 41. The coordinates (X ₁ , Y ₁ ) and (X ₂ , Y ₂ ) of the rectangular area corresponding to the left cheek are calculated.

The calculation unit 45 corrects the position of the measurement region according to the rotation angle of the face direction detected by the face direction detection unit 42. For example, the calculation unit 45 calculates expressions for correcting the positions of the right cheek, left cheek, and forehead regions according to the horizontal rotation angle θ _y of the face and the vertical rotation angle θ _x of the face. Read from the formula information 30. For example, the calculation unit 45 reads the above expressions (3) and (4) from the arithmetic expression information 30 as expressions for correcting the right cheek region in accordance with the horizontal rotation angle θ _y of the face. Further, the calculation unit 45, as an expression for correcting the area of the right cheek in accordance with the rotation angle theta _x vertical face above formula (5) is read from the arithmetic expression information 30 (6). In these equations, parameters corresponding to the rotation angle θ _y and the rotation angle θ _x are provided. Parameter values corresponding to the rotation angle θ _y and the rotation angle θ _x are stored in the angle parameter information 32. For example, the angle parameter information 32 for each rotation angle theta _y, the above equation (3), stored values of κ used in (4). Further, the angle parameter information 32 for each rotation angle theta _x, the equation (5), stored values of λ used in (6).

The calculation unit 45 reads the values of the parameters corresponding to the rotation angle θ _y and the rotation angle θ _x from the angle parameter information 32, sets the values in an equation, corrects the position of the measurement region by performing an operation using the equation. For example, the calculation unit 45 reads the value of κ corresponding to the rotation angle θ _y in the horizontal direction from the angle parameter information 32 and sets it in the equations (3) and (4). Then, the calculation unit 45 substitutes the coordinates (X ₁ , Y ₁ ), (X ₂ , Y ₂ ) of the shape region corresponding to the right cheek in the equations (3) and (4), and corrects by the horizontal rotation. The coordinates (X ′ ₁ , Y ′ ₁ ) and (X ′ ₂ , Y ′ ₂ ) of the subsequent right cheek shape region are calculated. Further, the calculation unit 45 reads the value of λ corresponding to the rotation angle θ _x in the vertical direction from the angle parameter information 32 and sets it in the equations (5) and (6). Then, the calculation unit 45 substitutes the corrected coordinates (X ′ ₁ , Y ′ ₁ ) in the horizontal direction into (X ₁ , Y ₁ ) and corrects the corrected coordinates (X ′ ₂ , Y ′ ₂ in the horizontal direction). ) Is set to (X ₂ , Y ₂ ), and the coordinates (X ″ ₁ , Y ″ ₁ ) and (X ″ ₂ , Y ″ ₂ ) of the right cheek shape region after correction by vertical rotation are calculated. In addition, about the position of the measurement area, the case where the position in the vertical direction is corrected after correcting the position in the horizontal direction has been described. However, after the position in the vertical direction is corrected, the position in the horizontal direction is corrected. May be.

  The measurement unit 46 is a processing unit that measures a pulse wave from each image captured by the camera 20. The measurement unit 46 measures the pulse wave from the signal of each wavelength component included in the measurement region calculated and corrected by the calculation unit 45 of the image taken by the camera 20. For example, the measurement unit 46 extracts, for each measurement region, each pixel within the passband range of the measurement region in which the wavelength component is stored in the passband information 33 from each pixel included in the measurement region. Then, the measurement unit 46 is a signal in which components in a specific frequency band other than the pulse wave frequency band in which the pulse wave can be taken between the respective wavelength components are canceled out from the extracted representative value signal for each wavelength component of each pixel. Is measured as a pulse wave.

  As one aspect, the measurement unit 46 is a representative value of two wavelength components of the three wavelength components included in the image, that is, the R component, the G component, and the B component of the R component and the G component that have different blood absorption specifications. A waveform is detected using time series data.

  To explain this, there are capillaries on the face surface, and when the blood flow flowing through the blood vessels changes due to the heartbeat, the amount of light absorbed by the bloodstream also changes according to the heartbeat. The brightness that is produced also changes with the heartbeat. Although the amount of change in luminance is small, when the average luminance of the entire face region is obtained, the time-series data of luminance includes a pulse wave component. However, the luminance also changes due to body movement in addition to the pulse wave, and this becomes a noise component of pulse wave detection, so-called body movement artifact. Therefore, a pulse wave is detected with two or more wavelengths having different light absorption characteristics of blood, for example, a G component having a high light absorption characteristic (about 525 nm) and an R component having a low light absorption characteristic (about 700 nm). Since the heart rate is in the range of 30 bpm to 240 bpm when converted to 0.5 Hz to 4 Hz per minute, other components can be regarded as noise components. Assuming that noise does not have wavelength characteristics or is minimal even if it is present, components other than 0.5 Hz to 4 Hz should be equal between the G signal and the R signal. Is different. Therefore, by correcting the sensitivity difference between components other than 0.5 Hz to 4 Hz and subtracting the R component from the G component, the noise component can be removed and only the pulse wave component can be extracted.

  For example, the G component and the R component can be represented by the following formula (7) and the following formula (8). “Gs” in the following equation (7) indicates the pulse wave component of the G signal, “Gn” indicates the noise component of the G signal, and “Rs” in the following equation (8) indicates the R signal. “Rn” indicates the noise component of the R signal. Further, since the noise component has a sensitivity difference between the G component and the R component, the correction coefficient k for the sensitivity difference is expressed by the following equation (9).

Ga = Gs + Gn (7)
Ra = Rs + Rn (8)
k = Gn / Rn (9)

  When the sensitivity difference is corrected and the R component is subtracted from the G component, the pulse wave component S is expressed by the following equation (10). When this is transformed into the formula represented by Gs, Gn, Rs and Rn using the above formula (7) and the above formula (8), the following formula (11) is obtained, and further, the above formula (9 ) To eliminate k and rearrange the equations, the following equation (12) is derived.

S = Ga-kRa (10)
S = Gs + Gn−k (Rs + Rn) (11)
S = Gs− (Gn / Rn) Rs (12)

  Here, the G signal and the R signal have different light absorption characteristics, and Gs> (Gn / Rn) Rs. Therefore, the pulse wave component S from which noise is removed can be calculated by the above equation (12).

  FIG. 9 is a diagram illustrating an example of the spectrum of each signal of the G signal and the R signal. The vertical axis of the graph shown in FIG. 9 indicates the signal intensity, and the horizontal axis indicates the frequency (bpm). As shown in FIG. 9, since the sensitivity of the image sensor differs between the G component and the R component, both signal intensities are different. On the other hand, in both the R component and the G component, there is no change in that noise appears outside the range of 30 bpm to 240 bpm, particularly in a specific frequency band of 3 bpm or more and less than 20 bpm. For this reason, as shown in FIG. 9, the signal intensity corresponding to the designated frequency Fn included in the specific frequency band of 3 bpm or more and less than 20 bpm can be extracted as Gn and Rn. The sensitivity difference correction coefficient k can be derived from these Gn and Rn.

  FIG. 10 is a diagram illustrating an example of a spectrum of each signal of the R component multiplied by the G component and the correction coefficient k. In the example of FIG. 10, the result of multiplying the absolute value of the correction coefficient is shown for convenience of explanation. Also in the graph shown in FIG. 10, the vertical axis indicates the signal strength, and the horizontal axis indicates the frequency (bpm). As shown in FIG. 10, when the spectrum of each signal of the G component and the R component is multiplied by the correction coefficient k, the sensitivity is uniform between the components of the G component and the R component. In particular, the spectrum signal intensity in a specific frequency band is almost the same in most spectrum signals. On the other hand, in the peripheral region 60 of the frequency where the pulse wave is actually included, the signal intensity of the spectrum is not uniform between the components of the G component and the R component.

  FIG. 11 is a diagram illustrating an example of a spectrum after calculation. In FIG. 11, the scale of the signal intensity, which is the vertical axis, is enlarged from the viewpoint of improving the visibility of the frequency band in which the pulse wave appears. As shown in FIG. 11, when the spectrum of the R signal after multiplication by the correction coefficient k is subtracted from the spectrum of the G signal, a signal in which a pulse wave appears due to a difference in light absorption characteristics between the G component and the R component. The noise component is reduced with the strength of the component maintained as much as possible. In this way, it is possible to detect a pulse wave waveform from which only noise components have been removed.

  Next, the functional configuration of the measurement unit 46 will be described more specifically. FIG. 12 is a block diagram illustrating a functional configuration of the measurement unit. As shown in FIG. 12, the measurement unit 46 includes a first calculation unit 51, BPF (Band-Pass Filter) 52R and 52G, extraction units 53R and 53G, and LPF (Low-Pass Filter) 54R and 54G. Have. The measurement unit 46 includes a second calculation unit 55, BPFs 56R and 56G, a multiplication unit 57, a calculation unit 58, and a detection unit 59. In the example of FIGS. 9 to 11, the example in which the pulse wave is detected in the frequency space has been described. However, in FIG. 12, the noise in the time series space is reduced from the viewpoint of reducing the time required for the conversion to the frequency component. The functional structure in the case of detecting a pulse wave by canceling components is shown.

  For each measurement region, the first calculation unit 51 extracts each pixel within the passband range of the measurement region in which the wavelength component is stored in the passband information 33 from each pixel included in the measurement region. And the 1st calculation part 51 calculates an average value as a representative value of the pixel value of each pixel extracted from the measurement area for every R component and G component. The representative value may be other than the average value as long as it is a value that represents the pixel value of each pixel included in the measurement target region. Then, the first calculation unit 51 samples the average values of the R signal and the G signal calculated from each measurement region in a time series over a predetermined time, for example, 1 second, 1 minute, and the like, and the sampled R signal and G The time series data of the signal is output to the function unit at the subsequent stage. For example, the first calculation unit 51 outputs the time series data of the R signal to the BPF 52R and the BPF 56R, and outputs the time series data of the G signal to the BPF 52G and the BPF 56G.

  Each of the BPF 52R, BPF 52G, BPF 56R, and BPF 56G is a band pass filter that passes only signal components in a predetermined frequency band and removes signal components in other frequency bands. These BPF 52R, BPF 52G, BPF 56R, and BPF 56G may be implemented by hardware or may be implemented by software.

  The difference in the frequency band that the BPF passes will be described. The BPF 52R and the BPF 52G pass a signal component in a specific frequency band in which a noise component appears more noticeably than other frequency bands.

  Such a specific frequency band can be determined by comparing with a frequency band that can be taken by a pulse wave. An example of a frequency band that can be taken by a pulse wave is a frequency band of 0.5 Hz to 4 Hz, and a frequency band of 30 bpm to 240 bpm when converted per minute. From this, as an example of the specific frequency band, a frequency band of less than 0.5 Hz and more than 4 Hz that cannot be measured as a pulse wave can be employed. Further, the specific frequency band may partially overlap with the frequency band that can be taken by the pulse wave. For example, it is allowed to overlap with the frequency band that the pulse wave can take in the section of 0.7 Hz to 1 Hz that is difficult to be measured as a pulse wave, and the frequency band of less than 1 Hz and 4 Hz or more is adopted as the specific frequency band. You can also Further, the specific frequency band can be narrowed down to a frequency band in which noise is more noticeable with the frequency band of less than 1 Hz and 4 Hz or more as the outer edge. For example, noise appears more noticeably in a low frequency band lower than a frequency band that can take a pulse wave, rather than a high frequency band that is higher than a frequency band that the pulse wave can take. For this reason, a specific frequency band can also be narrowed down to a frequency band of less than 1 Hz. Further, since there are many differences in the sensitivity of the image sensor of each component in the vicinity of the direct current component where the spatial frequency is zero, the specific frequency band can be narrowed down to a frequency band of 3 bpm or more and less than 60 bpm. Further, the specific frequency band can be narrowed down to a frequency band of 3 bpm to less than 20 bpm in which noise such as flickering of ambient light other than human body movement, for example, blinking or shaking of the body, is likely to appear.

  Here, as an example, the following description will be given assuming that the BPF 52R and the BPF 52G pass a signal component in a frequency band of 0.05 Hz to 0.3 Hz as a specific frequency band. Here, the case where a bandpass filter is used to extract a signal component in a specific frequency band is illustrated, but a low-pass filter is used when a signal component in a frequency band below a certain frequency is extracted. You can also

  On the other hand, the BPF 56R and the BPF 56G pass signal components in a frequency band that can be taken by a pulse wave, for example, a frequency band of 1 Hz to 4 Hz. Hereinafter, a frequency band that can be taken by a pulse wave may be referred to as a “pulse wave frequency band”.

  The extraction unit 53R extracts the absolute intensity value of the signal component in the specific frequency band of the R signal. For example, the extraction unit 53R extracts the absolute intensity value of the signal component in the specific frequency band by executing a multiplication process that powers the signal component in the specific frequency band of the R component. The extraction unit 53G extracts the absolute intensity value of the signal component in the specific frequency band of the G signal. For example, the extraction unit 53G extracts the absolute intensity value of the signal component in the specific frequency band by executing a multiplication process that powers the signal component in the specific frequency band of the G component.

  The LPF 54R and the LPF 54G are low-pass filters that perform a smoothing process for responding to time changes on time-series data of absolute intensity values in a specific frequency band. The LPF 54R and the LPF 54G are the same except that the signal input to the LPF 54R is an R signal and the signal input to the LPF 54G is a G signal. By such smoothing processing, absolute value intensities R′n and G′n in a specific frequency band are obtained.

  The second calculation unit 55 divides the absolute value strength G′n of the specific frequency band of the G signal output by the LPF 54G by the absolute value strength R′n of the specific frequency band of the R signal output by the LPF 54R. G′n / R′n ”is executed. Thereby, a correction coefficient k for the sensitivity difference is calculated.

  The multiplier 57 multiplies the signal component in the pulse wave frequency band of the R signal output by the BPF 56R by the correction coefficient k calculated by the second calculator 55.

  The calculation unit 58 calculates “k * Rs” by subtracting the signal component in the pulse wave frequency band of the G signal output by the BPF 56G from the signal component in the pulse wave frequency band of the R signal multiplied by the correction coefficient k by the multiplication unit 57. -Gs "is executed. The time series data of the signal obtained by such calculation corresponds to the waveform of the pulse wave.

  The detection unit 59 detects the pulse wave of the subject using the pulse wave waveform signal after the calculation. As one aspect, the detection unit 59 outputs time-series data of signals as a pulse wave detection result. As another aspect, the detection unit 59 can also detect a heart rate from a spectrum converted into a frequency component by applying a Fourier transform to the time-series data of the signal.

  Returning to FIG. 1, the measurement unit 46 performs the measurement process shown in FIG. 12, and calculates each pixel included in the measurement target region and stored in the passband information 33 within the passband range of the measurement region. Extract and measure the pulse wave from each extracted pixel. Here, the background may be included in the measurement region depending on the orientation of the face of the subject. However, the measurement unit 46 extracts each pixel within the passband range. Thereby, the measurement part 46 can measure a pulse wave accurately even if the pixel outside the range of the pass band is included in the measurement region.

  FIG. 13 is a diagram illustrating an example of a state in which the background is included in the measurement region. In the example of FIG. 13, the background of the left cheek region 80 </ b> L is included due to the subject's face changing to the left. This background portion has a large luminance change and becomes noise. In the example of FIG. 13, the left cheek region 80L includes a nose. This nose portion also has a large change in luminance due to the way the light hits it, resulting in noise. When the pulse wave is obtained from the left cheek region 80L, the detection accuracy of the pulse wave is reduced due to noise. For example, FIG. 13 shows a histogram of pixel values for each wavelength component of each pixel in the left cheek region 80L. As shown in FIG. 13, the noise is included in the left cheek region 80 </ b> L, so that pixels are distributed over a wide range in the histogram.

  The measurement unit 46 extracts each pixel within the passband range. As a result, the pixel of the noise is not extracted and the pixel of the skin portion is extracted, so that the pulse wave can be measured with high accuracy. FIG. 14 is a diagram illustrating an example of a result of extracting pixels in the passband. The example of FIG. 14 is a result of extracting pixels in the passband from the left cheek region 80L shown in FIG. As shown in FIG. 14, the pixels of the skin portion are extracted except the pixels of the background portion and the nose portion.

  The output control unit 47 outputs the pulse wave measured by the measurement unit 46. For example, the output control unit 47 displays the pulse wave measured by the measurement unit 46 on the display unit 21. Note that the output control unit 47 may transmit information on the pulse wave measured by the measurement unit 46 to another device. For example, the measured pulse wave information may be transmitted to a server that provides an electronic medical record service or a diagnostic service via a network. For example, in a diagnostic service, if a person with high blood pressure has tachycardia, such as 100 bpm or more, it is diagnosed that angina or myocardial infarction is suspected. Or make a diagnosis. As a result, a pulse monitoring service can be provided outside the hospital, for example, at home or at home.

[Process flow]
Subsequently, processing executed by the pulse wave detection device 10 according to the present embodiment will be described. First, the flow of calibration processing for setting various parameters of the arithmetic expression and the pass band by the pulse wave detection device 10 according to the present embodiment will be described. FIG. 15 is a flowchart showing the procedure of the calibration process. This calibration processing is executed, for example, at a timing when a predetermined operation for instructing the input unit 22 to execute calibration is performed.

  As shown in FIG. 15, the shooting control unit 40 controls the camera 20 to start shooting images continuously (S10). The parameter storage unit 43 prompts the subject to face the camera 20 (S11). For example, the parameter storage unit 43 displays a message “Please face the camera 20” on the display unit 21 to prompt the subject to face the camera 20. The face part detection unit 41 detects a face area from the image taken by the camera 20 (S12). For example, the face part detection unit 41 detects the positions of the eyes, mouth position, and right cheek, left cheek, and forehead areas. The face part detection unit 41 determines whether or not a face area has been detected (S13). If the face area cannot be detected (No at S13), the process proceeds to S12 again, and the face area is detected for the next photographed image.

On the other hand, when the face area can be detected (Yes in S13), the face direction detection unit 42 detects the face direction with respect to the directly facing state (S14). The face direction detection unit 42 determines whether or not the face direction can be regarded as a directly facing state (S15). For example, the face direction detection unit 42 determines whether or not the rotation angles θ _x and θ _y of the face direction are within a predetermined allowable angle that can be regarded as a state of facing each other with reference to zero. When it is not in the state of facing (No at S15), the process proceeds to S12 again, and the face area is detected for the next photographed image.

  On the other hand, when it can be regarded as a face-to-face state (S15 affirmative), the parameter storage unit 43 calculates the measurement area stored in the calculation formula information 30 from the eye, mouth position, and right cheek, left cheek, and forehead areas. The values of various parameters provided in each arithmetic expression calculated by (1) are calculated (S16). The parameter storage unit 43 stores the calculated values of various parameters in the calculation parameter information 31 (S17).

  The passband storage unit 44 obtains a histogram of pixel values for each wavelength component from each pixel included in the measurement region for each measurement region from the captured image (S18). For each measurement region, the passband storage unit 44 specifies a standard pixel value range based on the peak pixel value for each wavelength component, and sets the standard pixel value range for each specified wavelength component. The passband information 33 is set as the passband (S19), and the process is terminated.

  Next, the flow of detection processing for detecting a pulse wave by the pulse wave detection device 10 according to the present embodiment will be described. FIG. 16 is a flowchart showing the procedure of the detection process. This detection process is executed, for example, at a timing when a predetermined operation for instructing the input unit 22 to start detecting a pulse wave is performed.

  As shown in FIG. 16, the shooting control unit 40 controls the camera 20 to start shooting images continuously (S50). The face part detection unit 41 detects a face area in the image captured by the camera 20 (S51). For example, the face part detection unit 41 detects the position of the eyes and mouth. The face direction detection unit 42 detects the face direction with respect to the facing state (S52).

The face part detection unit 41 determines whether or not the detected face orientation is within the measurement range (S53). For example, when the detected rotation angle θ _y is within a range of ± 30 degrees and the rotation angle θ _x is within a range of ± 15 degrees, the face part detection unit 41 determines that it is within the measurement range. When the face orientation is not within the measurement range (No at S53), the output control unit 47 outputs a warning to the display unit 21 that the face orientation is not within the measurement range (S54), and the process proceeds to S51 again. Next, the pulse wave is measured for the photographed image.

  On the other hand, when the orientation of the face is within the measurement range (Yes in S53), the calculation unit 45 calculates the pulse wave in the photographed image from the face eye and mouth position detected by the face part detection unit 41. The position of the measurement area is calculated (S55). The calculation unit 45 corrects the position of the measurement region according to the rotation angle of the face direction detected by the face direction detection unit 42 (S56).

  The measurement unit 46 measures the pulse wave from the signal of each wavelength component included in the corrected measurement region of the captured image (S57). For example, the measurement unit 46 extracts, for each measurement region, each pixel within the passband range of the measurement region in which the wavelength component is stored in the passband information 33 from each pixel included in the measurement region. Then, the measurement unit 46 is a signal in which components in a specific frequency band other than the pulse wave frequency band in which the pulse wave can be taken between the respective wavelength components are canceled out from the extracted representative value signal for each wavelength component of each pixel. Is measured as a pulse wave.

  The output control unit 47 determines whether or not the process is finished (S58). For example, when a predetermined operation for instructing the end of processing is performed on the input unit 22, the output control unit 47 determines that the processing has ended (Yes in S <b> 58) and ends the processing. On the other hand, if the predetermined operation for instructing the end of the process is not performed, the output control unit 47 determines that the process is not ended (No in S58) and outputs the pulse wave measurement result by the measurement unit 46 to the display unit 21. (S59), the process proceeds to S51 again, and the pulse wave is measured for the next photographed image.

[Effect of Example 1]
As described above, the pulse wave detection device 10 according to the present embodiment detects the position of a predetermined part of the subject's face in the image from the image of the subject taken by the camera 20. The pulse wave detection device 10 detects a rotation angle of the face direction of the subject with respect to a predetermined direction from the image taken by the camera 20. Then, the pulse wave detection device 10 calculates the position of the measurement region of the pulse wave of the subject's face from the detection result of the position and rotation angle of the predetermined part, and measures the pulse wave from the calculated measurement region. Thereby, since the pulse wave detection device 10 can measure the pulse wave by obtaining the position of the measurement region according to the orientation of the face of the subject, the pulse wave detection accuracy can be suppressed.

  The pulse wave detection device 10 according to the present embodiment detects the position of a predetermined part of the subject's face and the rotation angle of the subject's face each time an image is captured by the camera 20, and the subject's face The position of the pulse wave measurement region is calculated. Thereby, the pulse wave detection device 10 can stably measure the pulse wave even when the orientation of the subject's face changes during imaging.

  In addition, the pulse wave detection device 10 according to the present embodiment is configured so that the pixel value of each wavelength component among the pixels included in the measurement region is pulsed from a pixel in the standard pixel value range stored in the storage unit 24. Measure the waves. As a result, the pulse wave detection device 10 can stably measure the pulse wave even when the pixel value of each wavelength component includes a pixel of noise outside the standard pixel value range.

  Further, the pulse wave detection device 10 according to the present embodiment obtains a histogram of pixel values for each wavelength component from each pixel included in the measurement region of the image in a state where the face of the subject is directly facing. Then, the pulse wave detection device 10 specifies a standard pixel value range based on the peak pixel value for each wavelength component, and stores the standard pixel value range for each specified wavelength component in the storage unit 24. Set. Thereby, since the pulse wave detection apparatus 10 can set the range of the standard pixel value of the actual skin part of the subject, the image of the skin part of the subject can be extracted.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

  In the above embodiment, the case where two types of R signal and G signal are used as the input signal is illustrated, but any type of signal and any number of signals can be input as long as the signals have a plurality of different optical wavelength components. It can be a signal. For example, two signals of any combination among signals having different optical wavelength components such as R, G, B, IR, and NIR can be used, or three or more signals can be used.

  In the above-described embodiments, the case where the calibration for setting various parameters of the arithmetic expression and the passband and the detection of the pulse wave are performed as separate processes has been described, but the disclosed apparatus is not limited thereto. For example, calibration processing may be performed at the beginning of detection processing for detecting a pulse wave.

  Further, in the above-described embodiment, the case has been described in which the orientation of the subject is detected in the calibration process to determine whether or not the subject is facing, but the disclosed apparatus is not limited thereto. For example, after prompting the subject to face the camera 20, various parameter values may be calculated or the passband may be set assuming that the subject is facing the subject after a certain period of time.

[Distribution and integration]
Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific state of distribution / integration of each device is not limited to the one shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, each processing unit of the imaging control unit 40, the face part detection unit 41, the face direction detection unit 42, the parameter storage unit 43, the passband storage unit 44, the calculation unit 45, the measurement unit 46, and the output control unit 47 illustrated in FIG. May be integrated or divided as appropriate. Each processing function performed by each processing unit may be realized in whole or in part by a CPU and a program that is analyzed and executed by the CPU, or may be realized as hardware by wired logic. . In addition, each processing function performed in each processing unit has an arbitrary part of the functional unit in another device, and the functions of the pulse wave detection device 10 described above are achieved by being connected to a network and cooperating. It may be realized.

[Pulse wave detection program]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. In the following, an example of a computer that executes a pulse wave detection program having the same function as that of the above-described embodiment will be described with reference to FIG.

  FIG. 17 is a diagram for describing an example of a computer that executes a pulse wave detection program. As illustrated in FIG. 17, the computer 100 includes an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 includes a CPU 150, a ROM 160, an HDD 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.

  The HDD 170 includes the imaging control unit 40, the face part detection unit 41, the face direction detection unit 42, the parameter storage unit 43, the passband storage unit 44, the calculation unit 45, the measurement unit 46, and the output control unit 47 of the first embodiment. A pulse wave detection program 170a that performs the same function as is stored in advance. The pulse wave detection program 170a may be integrated or separated as appropriate, as with each component of each functional unit shown in FIG. In other words, all data stored in the HDD 170 need not always be stored in the HDD 170, and only data necessary for processing may be stored in the HDD 170.

  Then, the CPU 150 reads the pulse wave detection program 170 a from the HDD 170 and develops it in the RAM 180. Accordingly, as shown in FIG. 17, the pulse wave detection program 170a functions as a pulse wave detection process 180a. The pulse wave detection process 180a expands various data read from the HDD 170 in an area allocated to itself on the RAM 180 as appropriate, and executes various processes based on the expanded various data. The pulse wave detection process 180a includes processing executed by each functional unit shown in FIG. 1 or FIG. 12, for example, processing shown in FIG. 15 and FIG. In addition, each processing unit virtually realized on the CPU 150 does not always require that all the processing units operate on the CPU 150, and only a processing unit necessary for processing needs to be virtually realized.

  Note that the pulse wave detection program 170a is not necessarily stored in the HDD 170 or the ROM 160 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk inserted into the computer 100, so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 100 may acquire and execute each program from these portable physical media. In addition, each program is stored in another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 acquires and executes each program from these. It may be.

DESCRIPTION OF SYMBOLS 10 Pulse wave detection apparatus 20 Camera 24 Memory | storage part 25 Control part 30 Calculation formula information 31 Calculation parameter information 32 Angle parameter information 33 Pass band information 40 Imaging | photography control part 41 Face part detection part 42 Face direction detection part 43 Parameter storage part 44 Pass band Storage unit 45 Calculation unit 46 Measurement unit 47 Output control unit

Claims (6)

On the computer,
Detecting the position of the predetermined part of the face of the subject in the image from the image of the subject imaged by the imaging device;
From the image photographed by the imaging device, a rotation angle of the face direction of the subject with respect to a predetermined direction is detected,
Based on the relationship information indicating the relationship between the position of the predetermined part and the position of the measurement area of the pulse wave of the subject's face, the position of the measurement area is calculated from the position of the predetermined part, Based on the correction information indicating the relationship between the rotation angle of the direction and the position change of the measurement region, the calculated position of the measurement region is corrected according to the detected rotation angle, and according to the orientation of the face of the subject. and calculates the position of the measurement region,
A pulse wave detection program for executing a process of measuring a pulse wave from a measurement region calculated according to the orientation of the face of the subject . Each time an image is taken by the imaging device, the position of a predetermined part of the subject's face and the rotation angle of the orientation of the subject's face are detected, and the position of the pulse wave measurement region of the subject's face is calculated. The pulse wave detection program according to claim 1, wherein: In the processing for measuring the pulse wave, the pixel value of each wavelength component among the pixels included in the measurement region is stored in a storage unit that stores a range of standard pixel values used for pulse wave measurement for each wavelength component. The pulse wave detection program according to claim 1 or 2, wherein a pulse wave is measured from a pixel that falls within a range of stored standard pixel values. On the computer,
Obtain a histogram of pixel values for each wavelength component from each pixel included in the measurement region of the image of the subject's face facing up, and for each wavelength component, a standard pixel value based on the peak pixel value 4. The pulse wave detection program according to claim 3 , further comprising: executing a process of specifying a range and setting a standard pixel value range for each specified wavelength component in the storage unit. 5. Computer
Detecting the position of the predetermined part of the face of the subject in the image from the image of the subject imaged by the imaging device;
From the image photographed by the imaging device, a rotation angle of the face direction of the subject with respect to a predetermined direction is detected,
Based on the relationship information indicating the relationship between the position of the predetermined part and the position of the measurement area of the pulse wave of the subject's face, the position of the measurement area is calculated from the position of the predetermined part, Based on the correction information indicating the relationship between the rotation angle of the direction and the position change of the measurement region, the calculated position of the measurement region is corrected according to the detected rotation angle, and according to the orientation of the face of the subject. and calculates the position of the measurement region,
A pulse wave detection method, comprising: performing a process of measuring a pulse wave from a measurement region calculated according to the orientation of the face of the subject . A face part detection unit that detects a position of a predetermined part of the face of the subject in the image from an image of the subject photographed by the imaging device;
A face direction detection unit that detects a rotation angle of the face direction of the subject with respect to a predetermined direction from an image captured by the imaging device;
Based on the relationship information indicating the relationship between the position of the predetermined part and the position of the measurement area of the pulse wave of the subject's face, the position of the measurement area is calculated from the position of the predetermined part, Based on the correction information indicating the relationship between the rotation angle of the direction and the position change of the measurement region, the calculated position of the measurement region is corrected according to the detected rotation angle, and according to the orientation of the face of the subject. A calculation unit for calculating the position of the measurement area;
A measurement unit that measures a pulse wave from the measurement region calculated by the calculation unit according to the orientation of the face of the subject ;
A pulse wave detection device comprising:

Images