JP6142664B2 - Pulse wave detection device, pulse wave detection program, pulse wave detection method, and content evaluation system - Google Patents

Description

  The present invention relates to a pulse wave detection device, a pulse wave detection program, a pulse wave detection method, and a content evaluation system.

  A technique for detecting blood volume fluctuations, so-called pulse waves, is known. The detection result of the pulse wave can be used as an index for health management, and can also be used as an index indicating the viewer's reaction to the video.

  In general, a measurement instrument attached to a living body is often used for detecting a pulse wave. For example, when a dedicated device is attached to a living body, changes in skin pressure due to blood vessel expansion are collected with a pressure sensor, or blood vessels are irradiated with visible light or infrared light, and the amount of transmitted or reflected light is changed. A method such as detection is adopted.

  However, when a pulse wave is detected using a measuring instrument, the sensor is attached to the living body or the photosensitive surface is brought into close contact with the living body. Therefore, detection is difficult unless the measuring instrument is in contact with the living body. There is a problem that it is troublesome to live daily life with a measuring instrument attached.

  For this reason, in order to detect a pulse wave in a state where the measuring instrument does not contact the living body, a heartbeat measuring method using an image taken by a subject has been proposed. In such a heart rate measuring method, an independent component analysis (ICA) is applied to a signal component of an image obtained by photographing a subject's face by a camera, and then the heart rate is measured by converting the signal into a frequency component. . The aim is to improve the signal-to-noise ratio by applying such independent component analysis.

JP 2011-130996 A JP 2012-239661 A JP 2000-271096 A JP 2002-34936 A

“Non-contact, automated cardiac pulse measurements using video imaging and blind source separation”, OPTICS EXPRESS Vol.18, No.10, 2010

  However, in the above technique, the pulse wave detection accuracy may be reduced as described below.

  That is, in the above heart rate measuring method, when a brightness change other than the pulse wave of the face is reflected on the camera that captures the face of the subject, the brightness change other than the pulse wave of the face is measured from the image captured by the camera. As a result, the measurement accuracy of the heartbeat may decrease. For example, when the subject measures a pulse wave while viewing an image displayed on the screen, the light displayed on the screen is reflected by the face of the subject, and the reflected light enters the camera. For this reason, the screen light reflected by the face is reflected in the image photographed by the camera. When a change in luminance is detected from an image in which the screen light is reflected in this way, the luminance change due to the pulse wave of the face is finer than the luminance change due to the screen light. Changes may be misdetected as heartbeats. In this case, an incorrect heartbeat is displayed or recorded.

  In one aspect, an object of the present invention is to provide a pulse wave detection device, a pulse wave detection program, a pulse wave detection method, and a content evaluation system that can suppress a decrease in detection accuracy of a pulse wave.

  The pulse wave detection device according to one aspect includes a first acquisition unit that acquires an image in which a living body is captured by a camera, and a biological extraction unit that extracts a biological region from the image. Furthermore, the pulse wave detection device includes a first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal that represents each pixel included in the living body region, and a living body from which the component of the frequency band is extracted. A detection unit that detects a pulse wave from the luminance signal of the region. Furthermore, the pulse wave detection device uses the second acquisition unit that acquires luminance information related to display of a predetermined screen and the luminance information related to display of the screen to execute detection of pulse waves by the detection unit. And a detection control unit for controlling the above.

  A decrease in pulse wave detection accuracy can be suppressed.

FIG. 1 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a scene in which pulse wave detection is performed. FIG. 3 is a flowchart illustrating the procedure of the pulse wave detection process according to the first embodiment. FIG. 4 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the second embodiment. FIG. 5 is a flowchart illustrating the procedure of the pulse wave detection process according to the second embodiment. FIG. 6 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the third embodiment. FIG. 7 is a flowchart illustrating the procedure of the pulse wave detection process according to the third embodiment. FIG. 8 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the fourth embodiment. FIG. 9 is a flowchart illustrating the procedure of the pulse wave detection process according to the fourth embodiment. FIG. 10 is a schematic diagram illustrating an example of a computer that executes a pulse wave detection program according to the first to fifth embodiments.

  A pulse wave detection device, a pulse wave detection program, a pulse wave detection method, and a content evaluation system according to the present application will be described below with reference to the accompanying drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

[Configuration of pulse wave detector]
First, the functional configuration of the pulse wave detection device according to the present embodiment will be described. FIG. 1 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the first embodiment. The pulse wave detection device 100 shown in FIG. 1 uses the image obtained by photographing the living body of the subject by the camera 110 without bringing the measuring instrument into contact with the subject under ordinary environmental light such as sunlight or room light. The pulse wave detection process for detecting the pulse wave, that is, the fluctuation of the blood volume is executed.

  Such a pulse wave detection device 100 can be implemented as one aspect by installing a pulse wave detection program in which the above-described pulse wave detection processing is realized as package software or online software in a desired computer.

  Here, the above-described pulse wave detection program can be installed in any device as long as it is a computer having a processor and a memory, but as an example, it can be installed in an information processing device with a display.

  For example, the above-described pulse wave detection program can be installed in a mobile communication terminal such as a smartphone, a mobile phone, or a PHS (Personal Handyphone System). In addition, the above-described pulse wave detection program can be installed not only on mobile communication terminals but also on all mobile terminal devices including digital cameras, tablet terminals, and slate terminals that do not have the ability to connect to mobile communication networks. . Thus, when the above-described pulse wave detection program is installed in a mobile terminal device, the mobile terminal device can function as the pulse wave detection device 100. Furthermore, when a camera is provided as a standard in the mobile terminal device, the camera provided as a standard in the mobile terminal device can also be used for pulse wave detection.

  The pulse wave detection program is not limited to the portable terminal device, and can be installed in the information processing apparatus in general. For example, the above-described pulse wave detection program can be installed in desktop and notebook personal computers. In this case, an image taken by a Web camera connected to a personal computer can be used for pulse wave detection. The pulse wave detection program can also be installed in a television receiver or an external device connected to the television receiver. In this case, an image taken by a camera attached to or incorporated in the television receiver or its external device may be used for pulse wave detection. Furthermore, the pulse wave detection program can be installed in a game machine connected to the television receiver, or the pulse wave detection program can be incorporated as a software module to be executed by the game machine. When a camera is built in such a game machine, an image photographed by the camera can be used for pulse wave detection.

  As shown in FIG. 1, a pulse wave detection device 100 includes a camera 110, an image memory 111, a first acquisition unit 112, a living body extraction unit 113, a first calculation unit 114, and a BPF (Band-Pass Filter). 115 and a detection unit 116. Furthermore, the pulse wave detection device 100 includes a display unit 120, a frame buffer 121, a second acquisition unit 122, and a second calculation unit 124. Furthermore, the pulse wave detection device 100 includes an illuminance sensor 130 and a detection control unit 150.

  Such a pulse wave detection device 100 may include various functional units included in a known computer in addition to the functional units illustrated in FIG. For example, when the pulse wave detection device 100 is implemented as a stationary terminal, it may further include an input device such as a keyboard and a mouse. Moreover, when the pulse wave detection apparatus 100 is mounted as a tablet terminal or a slate terminal, an acceleration sensor or an angular velocity sensor may be further included. When the pulse wave detection device 100 is mounted as a mobile communication terminal, the pulse wave detection device 100 further includes functional units such as an antenna, a wireless communication unit connected to the mobile communication network, and a GPS (Global Positioning System) receiver. It doesn't matter.

  The camera 110 is an imaging device that includes an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). For example, the camera 110 can be equipped with three or more light receiving elements such as R (red), G (green), and B (blue). As an example of mounting the camera 110, a digital camera or a Web camera may be connected via an external terminal. As another implementation example, when a camera is mounted from the time of shipment, such as an in camera or an out camera, the camera can be used. Hereinafter, an image captured by the camera 110 may be referred to as a “camera image”. Here, the case where the pulse wave detection device 100 includes the camera 110 is illustrated, but the pulse wave detection device 100 does not necessarily include the camera 110 when an image can be acquired via a network or a storage device. Also good.

  The display unit 120 is a display device that displays various types of information, for example, an image output from an OS (Operating System) or an application program that operates on the pulse wave detection device 100, or an image supplied from an external device. As an aspect of the display unit 120, a monitor or a display may be employed, or the display unit 120 may be implemented as a touch panel by being integrated with an input device.

  FIG. 2 is a diagram illustrating an example of a scene in which pulse wave detection is performed. As an example, FIG. 2 illustrates an example in which the pulse wave detection device 100 is mounted as a mobile terminal device, and an in-camera installed on the same side as the display unit 120 is used as the camera 110. ing. As shown in FIG. 2, when the user of the pulse wave detection device 100 browses the image displayed on the display unit 120, the user's face faces the screen of the display unit 120. Hereinafter, an image displayed on the display unit 120 may be referred to as a “screen display image”. Thus, when the user views the display image on the screen, the user's face faces not only the display unit 120 but also the camera 110 provided on the same side as the display unit 120. For this reason, it can be estimated that there is a high possibility that the user's face is reflected in the camera image taken while the user is browsing the display image on the screen. In this situation, if the pulse wave can be detected from the camera image in the background where the image is displayed on the display unit 120, the user of the pulse wave detection device 100 can perform health care or can view the still image. And content including videos.

  Here, the above-described pulse wave detection program can be activated when an activation operation is performed via an input device (not shown), and the content is displayed on the display unit 120 as described above. It can also be started in the background when triggered.

  At this time, when the above-described pulse wave detection program operates in the background, photographing by the camera 110 is started in the background while displaying the content on the display unit 120. As a result, a state in which the user browses the content with the face facing the screen of the display unit 120 is photographed as a camera image. Such content may be any kind of display object such as a document, video, video, etc., may be stored in the pulse wave detection device 100, or acquired from an external device such as a Web server. It does not matter if it is In this way, when the content is displayed, the user is likely to gaze at the display unit 120 until the browsing of the content is completed, so that it is possible to detect a camera image showing the user's face, that is, a pulse wave. It can be expected that continuous images will be acquired continuously.

  Further, when the above-described pulse wave detection program is activated by a user's activation operation, a camera image that can easily detect a pulse wave is captured through image display by the display unit 120, audio output from a speaker (not shown), and the like. In this way, it is possible to guide the image capturing operation. For example, the pulse wave detection program activates the camera 110 when activated via an input device. In response to this, the camera 110 starts photographing the subject accommodated in the photographing range of the camera 110. At this time, in the case of photographing the user's face, the pulse wave detection program may display an image captured by the camera 110 on the display unit 120 and display the target position that reflects the user's nose as an aim. it can. As a result, the user's nose can be photographed with the user's nose in the center of the photographing range among facial parts such as the user's eyes, ears, nose and mouth. Thereafter, a camera image obtained by photographing the user's face by the camera 110 is stored in the image memory 111.

  The image memory 111 is a storage device that stores images. As one aspect, each time a picture is taken by the camera 110, a camera image obtained by the picture is stored in the image memory 111. At this time, the image memory 111 may store a moving image encoded by a predetermined compression encoding method, or may store a set of still images showing a user's face. Here, the case where the camera image photographed by the camera 110 is stored is illustrated, but an image received via the network may be stored.

  The first acquisition unit 112 is a processing unit that acquires a camera image. As an aspect, the first acquisition unit 112 acquires a camera image captured by the camera 110. As another aspect, the first acquisition unit 112 may acquire a camera image from an auxiliary storage device such as a hard disk or an optical disk that stores the camera image or a removable medium such as a memory card or a USB (Universal Serial Bus) memory. it can. As a further aspect, the first acquisition unit 112 can also acquire a camera image by receiving from an external device via a network. The first acquisition unit 112 exemplifies the case where the processing is executed using image data such as two-dimensional bitmap data or vector data obtained from an output from an image sensor such as a CCD or CMOS. It is also possible to directly acquire the signal output from, and execute the subsequent processing.

  The living body extracting unit 113 is a processing unit that extracts a living body region from the camera image. As one aspect, the living body extracting unit 113 extracts a living body region based on a predetermined facial part from the camera image acquired by the first acquiring unit 112. For example, the biometric extraction unit 113 performs image processing such as template matching on the camera image, thereby performing facial processing such as a facial organ such as a user's eye, ear, nose, or mouth, and a specific facial part among so-called facial parts. That is, the user's nose is detected. Then, the living body extraction unit 113 extracts a living body region included in a predetermined range from the center with the user's nose as the center. Thereby, an image of a living body region including a face center part such as a user's nose and a part of a cheek located around the nose is extracted as an image used for detecting a pulse wave. Thereafter, the biological extraction unit 113 outputs an image of the biological region extracted from the camera image that is the original image to the first calculation unit 114. Note that the specific facial parts are not limited to the nose, and eyes or a mouth may be selected.

  The first calculation unit 114 is a processing unit that calculates a representative value of pixel values of each pixel included in the living body region. As one aspect, the first calculation unit 114 performs predetermined statistical processing on the pixel value of each pixel included in the biological region extracted by the biological extraction unit 113. For example, when a camera image is picked up by three types of light receiving elements of RGB, the first calculation unit 114 causes the luminance value of the R component, the luminance value of the G component, and the B component of each pixel included in the living body region. The average processing is executed by focusing on the luminance value of the G component among the luminance values of. As an example of such an average process, an average process such as an arithmetic average, a weighted average, or a moving average can be applied. The above statistical processing is not limited to the average processing, and the median value or the mode value may be calculated. Thereby, the average value of the luminance values of the G component possessed by each pixel included in the living body area is calculated as a representative value representing the living body area. In addition, although the case where the luminance value of the G component having the highest light absorption characteristic due to blood is averaged among the three wavelength components of RGB is illustrated here, the luminance values of other wavelength components such as the R component and the B component are averaged. It doesn't matter as you do. Furthermore, the average value of the luminance values of the G component, R component, and B component may be appropriately weighted and added.

  The BPF 115 is a band-pass filter that passes only signal components in a predetermined frequency band and removes signal components in other frequency bands. As one aspect, the BPF 115 extracts time-series data of the representative value of the living body area calculated by the first calculating unit 114, that is, a signal component in a frequency band that can be taken by a pulse wave from the luminance signal of the living body area. Here, as an example of a frequency band that can be taken by a pulse wave, a frequency band of 0.5 Hz to 4 Hz, in other words, a frequency band of 30 bpm (beats per minute) to 240 bpm can be given. Thus, the luminance signal Pi of the living body region that has passed through the BPF 115 includes a luminance change corresponding to the pulse wave. The BPF 115 may be implemented by hardware or may be implemented by software.

  The detection unit 116 is a processing unit that detects a user's pulse wave from the luminance signal Pi of the living body region output by the BPF 115. As an aspect, the detection unit 116 can output the waveform of the luminance signal Pi of the living body region output by the BPF 115 as it is as a pulse waveform. As another aspect, the detection unit 116 performs peak detection of the waveform of the luminance signal Pi, for example, detection of the zero cross point of the differential waveform, etc., every time the amplitude value of the luminance signal Pi of the living body region is output. Can calculate the pulse rate. For example, when a peak is detected from the waveform of the luminance signal Pi of the living body region by the peak detection, the detection unit 116 stores the sampling time in which the peak, that is, the maximum point is detected, in an internal memory (not shown). In addition, the detection unit 116 can calculate the pulse rate by obtaining the time difference from the maximum point n times the predetermined parameter at the time when the peak appears and dividing the time difference by n. In addition, although the case where the pulse rate was detected by the peak interval was illustrated here, the frequency that corresponds to the pulse wave by converting the pulse wave waveform to the frequency component, that is, the frequency that takes a peak in the range of 0.5 Hz to 4 Hz. It is also possible to detect the pulse rate.

  The pulse rate and pulse wave waveform obtained in this way can be output to any output destination including the display unit 120 included in the pulse wave detection device 100. For example, in the case where a diagnostic program for diagnosing the function of the autonomic nerve from fluctuations in pulse rate or pulse period or diagnosing a heart disease or the like from the pulse wave waveform is installed in the pulse wave detection device 100, the diagnostic program Can be the output destination. In addition, a server device that provides a diagnostic program as a Web service can be used as an output destination. Further, a terminal device used by a person concerned of the user who uses the pulse wave detection device 100, such as a caregiver or a doctor, can be used as the output destination. This also enables monitoring services outside the hospital, for example, at home or at home. Needless to say, the measurement result and diagnosis result of the diagnostic program can be displayed on the terminal device of the person concerned including the display unit 120 of the pulse wave detection device 100. Thus, the user's health management can be performed by using the detection result of the pulse wave.

  Here, while a series of camera images includes a luminance change corresponding to the pulse wave of the user's face, there is also room for a luminance change other than the face pulse wave to be reflected. For example, when the user detects a pulse wave while viewing the display image on the screen, the screen light output as the display image on the screen by the display unit 120 is reflected by the user's face and enters the camera 110. However, the reflected light may be reflected in the camera image. As described above, the change in the screen light reflected in the camera image is often detected as a change larger than the luminance change corresponding to the pulse wave, which causes erroneous detection.

  Therefore, the pulse wave detection device 100 according to the present embodiment uses the luminance information on the screen to determine whether or not there is a possibility of erroneous detection when the pulse wave is detected from the camera image. In order to realize this determination, the frame buffer 121, the second acquisition unit 122, the second calculation unit 124, and the illuminance sensor 130 are used by the detection control unit 150.

  The frame buffer 121 is a storage device that stores a display image on the screen. For example, a display image of a screen displayed on the display unit 120 is drawn in the frame buffer 121 as a processing result of an OS or an application program that operates on a processor included in the pulse wave detection device 100. The frame buffer 121 may be realized by allocating a part of the storage area of the main storage device connected to the processor, or a dedicated memory device may be provided.

  The second acquisition unit 122 is a processing unit that acquires luminance information related to screen display. As one aspect, the second acquisition unit 122 acquires the display image of the screen stored in the frame buffer 121 using an API (Application Program Interface) provided by the OS or application program. As another aspect, a function of acquiring a representative value of the luminance of the designated color in the display image stored on the screen stored in the frame buffer 121 is added to the OS or application program. In addition, the second acquisition unit 122 can also acquire luminance information related to screen display by calling a function added to the OS or application program. As a further aspect, the second acquisition unit 122 can also acquire an image captured by the display unit 120 by a camera different from the camera 110 as luminance information related to screen display.

  The second calculation unit 124 is a processing unit that calculates a representative value of pixel values of each pixel included in the display image on the screen. As an aspect, the second calculation unit 124 performs predetermined statistical processing on the pixel value of each pixel included in the display image of the screen acquired by the second acquisition unit 122. For example, when the screen display image has a luminance value corresponding to the RGB color system, the second calculation unit 124 uses the R component luminance value and the G component luminance included in each pixel included in the screen display image. The averaging process is executed by focusing on the luminance value of the G component among the luminance values of the value and the B component. Thereby, the average value of the luminance values of the G component of each pixel included in the display image on the screen is calculated as a representative value representing the display image on the screen. In addition, although the case where the luminance value of the G component having the highest light absorption characteristic due to blood is averaged among the three wavelength components of RGB is illustrated here, the luminance values of other wavelength components such as the R component and the B component are averaged. It doesn't matter as you do. Furthermore, the average value of the luminance values of the G component, R component, and B component may be appropriately weighted and added.

  The illuminance sensor 130 is a sensor that senses ambient brightness. As one aspect, when the pulse wave detection device 100 is mounted as a mobile terminal device, the illuminance sensor provided as standard in the mobile terminal device can be used as it is as a power saving measure or a dark place measure. As another aspect, when the pulse wave detection device 100 is mounted as an information processing device, an illuminance sensor built in or attached to a display or the like can be used. Note that the type of the illuminance sensor 130 may be one using a phototransistor or a photodiode, or an amplifier circuit added to the photodiode.

  The detection control unit 150 is a processing unit that controls whether or not to detect the pulse wave by the detection unit 116 using the luminance information of the screen. As illustrated in FIG. 1, the detection control unit 150 includes an illuminance determination unit 151 and a change amount determination unit 152.

  Among these, the illuminance determination unit 151 is a processing unit that determines whether or not the illuminance detected by the illuminance sensor 130 is greater than or equal to a predetermined threshold value. As an aspect, the illuminance determination unit 151 acquires the illuminance detected by the illuminance sensor 130 every time the camera image stored in the image memory 111 is updated. In addition, the illuminance determination unit 151 determines whether the illuminance detected by the illuminance sensor 130 is equal to or greater than a predetermined threshold. Such threshold determination of illuminance is executed for the purpose of determining whether or not the brightness of ambient ambient light where the user views the display unit 120 affects the detection of the pulse wave.

  For example, if the surroundings are sufficiently bright, it can be estimated that the camera image was shot in a situation where the brightness of the ambient light is large enough to ignore the reflected light on the screen. In this case, it is not always necessary to determine whether a change in screen brightness affects the camera image. Therefore, when the illuminance is equal to or greater than the threshold value, the detection control unit 150 pauses the acquisition of the display image on the screen by the second acquisition unit 122 and causes the first acquisition unit 112 to acquire a camera image. As a result, the load of processing executed by the second acquisition unit 122 and the second calculation unit 124 and the power consumed by the execution of the processing can be reduced, and the pulse wave can be detected using a camera image advantageous for pulse wave detection. Detection can be performed. Here, the case where the acquisition of the display image of the screen by the second acquisition unit 122 is illustrated as an example, but the calculation of the representative value of the display image of the screen by the second calculation unit 124 may be stopped.

  On the other hand, if the surroundings are not sufficiently bright and dark, it can be estimated that the camera image was shot in a situation where the brightness of the ambient light is not so great that the reflected light from the screen can be ignored. In this case, there remains room for a change in screen brightness to adversely affect the camera image. Therefore, when the illuminance is less than the threshold value, the detection control unit 150 causes the second acquisition unit 122 to acquire the screen display image without pausing the screen display image acquisition.

  In addition, although the case where the illuminance determination unit 151 determines whether the illuminance is equal to or higher than the threshold is illustrated here, when the environment where the user browses the screen is an excessively dark place, the pulse wave It is also difficult to capture the corresponding luminance change with the camera image. For this reason, when the illuminance is less than or equal to a threshold value for determining whether the light is excessively dark, the detection of the pulse wave may be stopped by stopping the acquisition of the camera image by the first acquisition unit 112. Good.

  The change amount determination unit 152 is a processing unit that determines whether or not the change in the representative value of the luminance of the display image on the screen output by the second calculation unit 124 is equal to or greater than a predetermined threshold value. As one aspect, when the illuminance determination unit 151 determines that the illuminance is equal to or greater than the threshold, the change amount determination unit 152 performs the following process. For example, the change amount determination unit 152 displays the representative value of the screen display image output by the second calculation unit 124 and the representative value of the display image of the screen acquired from the frame buffer 121 before the display image of the screen. Calculate the difference between and. In this way, the amount of change in the luminance of the display image on the screen is calculated by calculating the difference between the representative values between the frames. Then, the change amount determination unit 152 determines whether or not the change amount of the luminance of the display image on the screen is equal to or greater than a predetermined threshold value. Such threshold determination of the change amount is executed for the purpose of determining whether or not the display image on the screen adversely affects the detection of the pulse wave.

  For example, when the amount of change in luminance of the display image on the screen is larger than the luminance change corresponding to the pulse wave, the possibility that the pulse wave is erroneously detected increases due to the reflected light of the display image on the screen as noise. Therefore, the detection control unit 150 pauses the acquisition of the camera image by the first acquisition unit 112. Accordingly, it is possible to reduce the load of processing executed by the functional units after the first acquisition unit 112 and the power consumed by the execution of the processing, and it is possible to suppress erroneous detection of pulse waves. Here, as an example, the case where the acquisition of the camera image by the first acquisition unit 112 is paused is illustrated, but the processing by any functional unit in the subsequent stage from the first acquisition unit 112 may be paused. . For example, the processing by any of the functional units of the living body extraction unit 113, the first calculation unit 114, the BPF 115, or the detection unit 116 may be suspended, and the detection result of the pulse wave detected by the detection unit 116 is discarded. It is good. At this time, the processing load and power consumption can be further reduced as the processing executed by the preceding functional unit is suspended.

  On the other hand, when the amount of change in luminance of the display image on the screen is smaller than the luminance change corresponding to the pulse wave, the possibility that the reflected light of the display image on the screen becomes noise and the pulse wave is erroneously detected is reduced. Therefore, the detection control unit 150 causes the first acquisition unit 112 to acquire a camera image. As a result, the processing of each functional unit subsequent to the first acquisition unit 112 is executed, and as a result, detection of the pulse wave is executed.

  Each functional unit shown in FIG. 1 can be realized by causing a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like to execute a pulse wave detection program. Each functional unit described above can also be realized by a hard wired logic such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The functional units include the first acquisition unit 112, the living body extraction unit 113, the first calculation unit 114, the BPF 115, the detection unit 116, the second acquisition unit 122, the second calculation unit 124, the detection control unit 150, and the like. .

  For example, a semiconductor memory element can be used for the image memory 111 and the frame buffer 121 described above. For example, examples of the semiconductor memory device include a video random access memory (VRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), and a flash memory. Further, an auxiliary storage device such as a hard disk or an optical disk may be substituted for a memory device such as a semiconductor memory element.

[Process flow]
Next, the process flow of the pulse wave detection device 100 according to the present embodiment will be described. FIG. 3 is a flowchart illustrating the procedure of the pulse wave detection process according to the first embodiment. This pulse wave detection process is a process that is started every time a camera image stored in the image memory 111 is updated, and is repeatedly executed until the camera image is not updated. Note that the pulse wave detection process ends when the end operation is accepted via an input device (not shown) or the like, or when the shooting of the camera image ends when the display of the content ends.

  As illustrated in FIG. 3, when the camera image stored in the image memory 111 is updated, the illuminance determination unit 151 acquires the illuminance detected by the illuminance sensor 130 (step S101). Then, the illuminance determination unit 151 determines whether or not the illuminance acquired in step S101 is equal to or greater than a predetermined threshold (step S102).

  At this time, if the illuminance is equal to or greater than the threshold (Yes in step S102), it can be estimated that the camera image was captured in a situation where the brightness of the ambient light is large enough to ignore the reflected light on the screen. In this case, it is not always necessary to determine whether a change in screen brightness affects the camera image. Therefore, the detection control unit 150 skips steps S103 to S105, and proceeds to step S106.

  On the other hand, if the illuminance is less than the threshold (No in step S102), it can be estimated that the camera image was shot in a situation where the brightness of the ambient light is not so great that the reflected light from the screen can be ignored. In this case, the second acquisition unit 122 acquires the display image of the screen stored in the frame buffer 121 using the API provided by the OS or application program (step S103).

  Subsequently, the second calculation unit 124 performs a predetermined statistical process on the pixel value of each pixel included in the display image of the screen acquired in step S103, thereby obtaining a representative value representing the display image of the screen. Calculate (step S104).

  Then, the change amount determination unit 152 calculates the difference between the representative value of the display image of the screen calculated in step S104 and the representative value of the display image of the screen acquired before the display image of the screen. It is determined whether or not the amount of change in luminance of the displayed image on the screen is greater than or equal to a predetermined threshold (step S105).

  Here, when the amount of change in luminance of the display image on the screen is equal to or greater than the threshold (Yes in step S105), the possibility that the reflected light of the display image on the screen becomes noise and the pulse wave is erroneously detected increases. Therefore, the detection control unit 150 pauses the acquisition of the camera image by the first acquisition unit 112 and ends the process as it is.

  On the other hand, when the amount of change in luminance of the display image on the screen is less than the threshold (No in step S105), the possibility that the reflected light of the display image on the screen becomes noise and the pulse wave is erroneously detected is reduced. Therefore, the detection control unit 150 causes the first acquisition unit 112 to acquire a camera image (step S106).

  Subsequently, the living body extracting unit 113 extracts a living body region from the camera image acquired in Step S106 (Step S107). Thereafter, the first calculation unit 114 calculates the representative value of the pixel value of each pixel included in the biological region extracted in step S107 (step S108).

  Subsequently, the BPF 115 extracts time-series data of representative values of the living body region, that is, a signal component in a frequency band that can be taken by a pulse wave from the luminance signal of the living body region (step S109). Thereafter, the detection unit 116 detects the user's pulse wave from the luminance signal Pi of the living body region output by the BPF 115 (step S110), and ends the process.

[Process flow]
As described above, the pulse wave detection device 100 according to the present embodiment uses the luminance information relating to the display of the screen from the camera image when the living body of the subject who views the display image of the screen is photographed by the camera 110. Controls whether to detect a pulse wave. For this reason, in the pulse wave detection device 100 according to the present embodiment, when the display on the screen adversely affects the camera image, the detection of the pulse wave is suspended while the display on the screen does not adversely affect the camera image. In this case, the pulse wave detection can be continued. Therefore, in the pulse wave detection device 100 according to the present embodiment, pulse wave detection can be executed by focusing on a camera image advantageous for pulse wave detection. Therefore, according to the pulse wave detection device 100 according to the present embodiment, it is possible to suppress a decrease in pulse wave detection accuracy.

  In the first embodiment, the case where the pulse wave detection is controlled according to whether or not the amount of change in the luminance of the display image on the screen is equal to or greater than the threshold is exemplified. The fact that the amount of change in luminance is greater than or equal to the threshold does not necessarily mean that pulse waves cannot be detected. This is because even if the amount of change in the brightness of the display image on the screen is greater than or equal to the threshold, if the brightness of the display image on the screen does not change in the frequency band that can be taken by the pulse wave, the BPF 115 removes the change in the brightness of the display image on the screen. Because it can. Therefore, in the present embodiment, an example will be described in which whether or not to detect the pulse wave is controlled depending on whether or not the luminance of the display image on the screen changes in a frequency band that can be taken by the pulse wave.

  FIG. 4 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the second embodiment. The pulse wave detection device 200 shown in FIG. 4 is different from the pulse wave detection device 100 shown in FIG. 1 in that it has a BPF 220 and a detection control unit 250. In FIG. 4, the same reference numerals are given to functional units that exhibit the same functions as the functional units illustrated in FIG. 1, and descriptions thereof are omitted.

  Similar to the BPF 115, the BPF 220 is a band-pass filter that passes only signal components in a predetermined frequency band and removes signal components in other frequency bands. As one aspect, the BPF 220 extracts time-series data of the representative value of the screen display image calculated by the second calculation unit 124, that is, a signal component in a frequency band that can be taken by the pulse wave from the luminance signal of the screen display image. To do. Here, as an example of a frequency band that can be taken by a pulse wave, a frequency band of 0.5 Hz to 4 Hz, in other words, a frequency band of 30 bpm (beats per minute) to 240 bpm can be given. The BPF 115 may be implemented by hardware or may be implemented by software.

  Compared to the detection control unit 150 illustrated in FIG. 1, the detection control unit 250 has an amplitude value of the luminance signal Li of the display image of the screen output by the BPF 220 that is equal to or greater than a predetermined threshold instead of the change amount determination unit 152. The difference is that it has an amplitude determination unit 252 for determining whether or not there is.

  FIG. 5 is a flowchart illustrating the procedure of the pulse wave detection process according to the second embodiment. Similar to the pulse wave detection process shown in FIG. 3, this pulse wave detection process is started every time the camera image stored in the image memory 111 is updated, and is repeatedly executed until the camera image is not updated. Process. In FIG. 5, steps in which the same processing as the pulse wave detection processing shown in FIG. 3 is executed are assigned the same step numbers, and steps having different processing are assigned different step numbers. .

  As shown in FIG. 5, when the camera image stored in the image memory 111 is updated, the illuminance determination unit 151 acquires the illuminance detected by the illuminance sensor 130 (step S101). Then, the illuminance determination unit 151 determines whether or not the illuminance acquired in step S101 is equal to or greater than a predetermined threshold (step S102).

  At this time, if the illuminance is equal to or greater than the threshold (Yes in step S102), it can be estimated that the camera image was captured in a situation where the brightness of the ambient light is large enough to ignore the reflected light on the screen. In this case, it is not always necessary to determine whether a change in screen brightness affects the camera image. Therefore, the detection control unit 250 skips steps S103 to S202, and proceeds to step S106.

  On the other hand, if the illuminance is less than the threshold (No in step S102), it can be estimated that the camera image was shot in a situation where the brightness of the ambient light is not so great that the reflected light from the screen can be ignored. In this case, the second acquisition unit 122 acquires the display image of the screen stored in the frame buffer 121 using the API provided by the OS or application program (step S103).

  Subsequently, the second calculation unit 124 performs a predetermined statistical process on the pixel value of each pixel included in the display image of the screen acquired in step S103, thereby obtaining a representative value representing the display image of the screen. Calculate (step S104). Thereafter, the BPF 220 extracts time-series data of representative values of the display image on the screen, that is, a signal component in a frequency band that can be taken by the pulse wave from the luminance signal of the display image on the screen (step S201).

  Then, the amplitude determination unit 252 determines whether or not the amplitude value of the luminance signal Li of the display image on the screen from which the signal component of the frequency band that can be taken by the pulse wave is extracted in step S201 is equal to or greater than a predetermined threshold ( Step S202).

  Here, when the amplitude value of the luminance signal Li of the display image on the screen is equal to or greater than the threshold (Yes in step S202), the luminance change of the display image on the screen and the luminance change corresponding to the pulse wave are similar to each other, The possibility that the reflected light of the display image becomes noise and the pulse wave is erroneously detected is further increased. Therefore, the detection control unit 250 pauses the acquisition of the camera image by the first acquisition unit 112 and ends the process as it is.

  On the other hand, when the amplitude value of the luminance signal Li of the display image on the screen is less than the threshold (No in step S202), the luminance change of the display image on the screen and the luminance change corresponding to the pulse wave are not similar to each other, The possibility that the reflected light of the display image on the screen becomes noise and the pulse wave is erroneously detected is reduced. Therefore, the detection control unit 250 causes the first acquisition unit 112 to acquire a camera image (step S106).

  Subsequently, the living body extracting unit 113 extracts a living body region from the camera image acquired in Step S106 (Step S107). Thereafter, the first calculation unit 114 calculates the representative value of the pixel value of each pixel included in the biological region extracted in step S107 (step S108).

  Subsequently, the BPF 115 extracts time-series data of representative values of the living body region, that is, a signal component in a frequency band that can be taken by a pulse wave from the luminance signal of the living body region (step S109). Thereafter, the detection unit 116 detects the user's pulse wave from the luminance signal Pi of the living body region output by the BPF 115 (step S110), and ends the process.

[Effect of Example 2]
As described above, the pulse wave detection device 200 according to the present embodiment determines whether or not to execute the detection of the pulse wave depending on whether or not the luminance of the display image on the screen changes in a frequency band that can be taken by the pulse wave. Control. Therefore, in the pulse wave detection device 200 according to the present embodiment, not only the magnitude of the change in the brightness of the display image on the screen but also the change in the brightness of the display image on the screen and the brightness change of the target pulse wave are similar to each other. The detection of the pulse wave can be paused by focusing on the case. Therefore, in the pulse wave detection device 200 according to the present embodiment, even when the luminance of the display image on the screen greatly changes, the pulse wave detection can be continued if the noise can be removed by the BPF 115 or the like. it can. Therefore, according to the pulse wave detection device 200 according to the present embodiment, it is possible to suppress a decrease in the detection accuracy of the pulse wave and to increase the opportunity for detecting the pulse wave from the camera image.

  In the second embodiment, the luminance change of the display image on the screen and the target pulse are monitored by monitoring the amplitude value of the luminance signal Li of the display image on the screen from which the component of the frequency band that can be taken by the pulse wave is extracted. Although the case where it is determined whether or not the luminance change of the wave is similar to each other is illustrated, it is also possible to determine whether or not the luminance change of both is similar by other methods. For example, in the present embodiment, by calculating the similarity between the waveform of the luminance signal Pi of the living body region from which the components of the frequency bands that can be taken by the pulse waves are extracted and the waveform of the luminance signal Li of the display image on the screen. An example of determining whether or not the luminance change of the display image on the screen is similar to the luminance change of the target pulse wave will be described.

  FIG. 6 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the third embodiment. The pulse wave detection device 300 shown in FIG. 6 is different from the pulse wave detection device 200 shown in FIG. 4 in that it includes a detection control unit 350. Furthermore, the detection control unit 350 illustrated in FIG. 6 is different from the detection control unit 250 illustrated in FIG. 4 in that it includes a similarity calculation unit 351 and a similarity determination unit 352.

  Here, similarly to the description of the second embodiment, in FIG. 6, the same reference numerals are given to the functional units that perform the same functions as the functional units shown in FIG. 4, and the description thereof is omitted. To do. 6 does not show the illuminance sensor 130 and the illuminance determination unit 151 shown in FIGS. 1 and 4, the illuminance sensor 130 and the illuminance determination unit 151 may be further included. Further, the illuminance sensor 130 and the illuminance determination unit 151 can be omitted in the pulse wave detection device 100 or the pulse wave detection device 200 shown in FIGS. 1 and 4.

  The similarity calculation unit 351 is a processing unit that calculates the similarity between the waveform of the luminance signal Pi of the living body region output by the BPF 115 and the waveform of the luminance signal Li of the screen display image output by the BPF 220. . As one aspect, the similarity calculation unit 351 calculates a correlation coefficient between the luminance signal Pi of the living body region and the luminance signal Li of the display image on the screen using the following formula (1), The correlation coefficient is calculated as the similarity. In the following formula (1), “Lm” indicates the average value of the luminance signal Li of the display image on the screen, “Pm” indicates the average value of the luminance signal Pi of the living body region, and “Σ” indicates i = 1 to i = n.

  The similarity determination unit 352 is a processing unit that determines whether the similarity calculated by the similarity calculation unit 351 is a predetermined threshold, for example, “0.2” or more. Here, when the similarity is equal to or greater than the threshold value, the luminance change of the display image on the screen may be superimposed on the pulse wave, and the reflected light of the display image on the screen may become noise and the pulse wave may be erroneously detected. Is expensive. Therefore, the detection control unit 350 pauses the detection of the pulse wave by the detection unit 116. On the other hand, when the similarity is less than the threshold, the luminance change of the display image on the screen is not superimposed on the pulse wave, and the reflected light of the display image on the screen may be noise and the pulse wave may be erroneously detected. The nature is low. Therefore, the detection control unit 350 causes the detection unit 116 to detect a pulse wave from the luminance signal Pi of the living body region.

  FIG. 7 is a flowchart illustrating the procedure of the pulse wave detection process according to the third embodiment. Like the pulse wave detection process shown in FIG. 5, this pulse wave detection process is started every time the camera image stored in the image memory 111 is updated, and is repeatedly executed until the camera image is not updated. Process.

  As illustrated in FIG. 7, when the camera image stored in the image memory 111 is updated, the second acquisition unit 122 uses the API provided by the OS or application program to display the screen stored in the frame buffer 121. A display image is acquired (step S301).

  Subsequently, the second calculation unit 124 performs a predetermined statistical process on the pixel value of each pixel included in the display image of the screen acquired in step S301 to obtain a representative value that represents the display image of the screen. Calculate (step S302). Thereafter, the BPF 220 extracts time-series data of representative values of the display image on the screen, that is, a signal component in a frequency band that can be taken by the pulse wave from the luminance signal of the display image on the screen (step S303).

  On the other hand, the first acquisition unit 112 acquires a camera image stored in the image memory 111 (step S304). Subsequently, the living body extraction unit 113 extracts a living body region from the camera image acquired in step S304 (step S305).

  Subsequently, the first calculation unit 114 calculates the representative value of the pixel value of each pixel included in the living body region extracted in step S305 (step S306). Then, the BPF 115 extracts time-series data of representative values of the living body region, that is, a signal component in a frequency band that can be taken by a pulse wave from the luminance signal of the living body region (step S307).

  After that, the similarity calculation unit 351 obtains the waveform of the luminance signal Li of the display image on the screen from which the component of the frequency band that can be taken in step S303 is extracted, and the component of the frequency band that can be taken in step S307. The degree of similarity is calculated between the extracted waveform of the luminance signal Pi of the living body region (step S308).

  Then, the similarity determination unit 352 determines whether or not the similarity calculated in step S308 is greater than or equal to a predetermined threshold (step S309). Here, when the similarity is equal to or greater than the threshold (Yes in step S309), the luminance change of the screen display image is superimposed on the pulse wave, and the reflected light of the screen display image becomes noise and the pulse wave is erroneously detected. There is a high possibility of being. Therefore, the detection control unit 350 pauses the detection of the pulse wave by the detection unit 116 and ends the process as it is.

  On the other hand, when the similarity is less than the threshold (No in step S309), the luminance change of the display image on the screen is not superimposed on the pulse wave, and the reflected light of the display image on the screen becomes noise and the pulse wave is erroneous. It is unlikely to be detected. Therefore, the detection control unit 350 causes the detection unit 116 to detect a pulse wave from the luminance signal Pi of the living body region. In this case, the detection unit 116 detects the user's pulse wave from the luminance signal Pi of the living body region output by the BPF 115 (step S310), and ends the process.

  In the flowchart shown in FIG. 7, an example in which the processing of step S304 to step S307 is executed after the processing of step S301 to step S303 is illustrated, but the order in which these processing is executed is switched. Can also be executed in parallel.

[Effect of Example 3]
As described above, the pulse wave detection device 300 according to the present embodiment includes the waveform of the luminance signal Pi of the living body region from which the components of the frequency bands that can be taken by the pulse wave are extracted, and the luminance signal Li of the display image on the screen. Whether or not to detect a pulse wave is controlled depending on whether or not the similarity between waveforms is equal to or greater than a threshold value. Therefore, according to the pulse wave detection device 300 according to the present embodiment, it is possible to suppress a decrease in the detection accuracy of the pulse wave and increase the opportunity for detecting the pulse wave from the camera image as in the second embodiment. be able to.

  In the first to third embodiments, the case where the display image of the screen is acquired as the luminance information related to the display of the screen is illustrated. However, the display image of the screen may not necessarily be acquired. For example, in the present embodiment, an example will be described in which an image of a specific region having a specific positional relationship with a biological region among non-biological regions of a camera image is acquired as luminance information related to screen display.

  FIG. 8 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the fourth embodiment. The pulse wave detection device 400 illustrated in FIG. 8 is different from the pulse wave detection device 300 illustrated in FIG. 6 in that it includes a non-biological extraction unit 410 and a second acquisition unit 420. Here, as in the description of the first to third embodiments, in FIG. 8, the same reference numerals are given to the functional units that perform the same functions as the functional units shown in FIG. It will be omitted.

  The non-biological extraction unit 410 is a processing unit that extracts a non-biological region from a camera image. As one aspect, first, the non-living body extraction unit 410 extracts a living body region based on a predetermined facial part from a camera image in the same manner as the living body extraction unit 113 illustrated in FIG. Then, the non-living body extraction unit 410 extracts an area excluding the living body area from the camera image as a non-living area. Thereafter, the non-biological extraction unit 410 outputs an image of the non-biological region extracted from the camera image that is the original image to the second acquisition unit 420.

  The 2nd acquisition part 420 is a process part which acquires the image of the specific area | region which has specific positional relationship between living body area | regions among non-biological area | regions as brightness | luminance information regarding the display of a screen. As an aspect, the second acquisition unit 420 determines the specific area according to the positional relationship with a plurality of face parts included in the living body area in the non-living body area. As an example, a case where a mouth part and a nose part are used for determining a specific region is assumed. In this case, the second acquisition unit 420 has a predetermined shape and range centered or centered at a position away from the mouth part by a predetermined distance from the nose part to the mouth part on the non-biological region of the camera image. It is determined as a specific area. Arbitrary shapes such as ellipses, circles, and polygons can be adopted as the shape of the specific region, and the size of the specific region is within the non-biological region, and any size is adopted if it does not overlap with the biological region. it can. In addition, the distance between the specific area and the living body area is preferably equal to or longer than the length of the neck assumed to be reflected in the camera image. Thereby, it is possible to specify, as the specific area, an area where the user's clothes are highly likely to be reflected among the non-biological areas included in the camera image.

  In this example, the case where a plurality of facial parts are used to identify a specific area is illustrated, but an area of a predetermined size located below the living body area is identified without depending on the positional relationship of the facial parts. It may be specified as an area. Here, the case where the non-living body extraction unit 410 is provided independently of the living body extracting unit 113 in order to extract the non-living body region is illustrated, but the living body region extracted by the living body extracting unit 113 from the camera image is illustrated. It is good also as extracting a non-biological area | region by excluding. Further, the method of extracting the non-living area is not limited to the method of detecting from the living area detected by using the template matching described above, and the area where the skin color is not detected by the skin color detection is extracted as the non-living area. It doesn't matter. Furthermore, here, the case where the specific area is determined with the user's clothes reflected in the camera image as a target has been illustrated, but the specific area can also be determined with hair or the like that is less likely to exhibit a luminance change corresponding to the pulse wave as a target. . In this case, a part of the non-living area existing above the living body area extracted from the face, in other words, from the mouth part to the nose part, for example, an area having a specific color such as black or brown is specified. It can be determined as a region.

  Thus, the image of the specific area acquired by the second acquisition unit 420 is output to the second calculation unit 124. In the second calculation unit 124 to which the image of the specific area is input, the representative value of the pixel value of each pixel included in the image of the specific area is calculated instead of the display image on the screen.

  FIG. 9 is a flowchart illustrating the procedure of the pulse wave detection process according to the fourth embodiment. Like the pulse wave detection process shown in FIG. 7, this pulse wave detection process is started every time a camera image stored in the image memory 111 is updated, and is repeatedly executed until the camera image is not updated. Process. In FIG. 9, the same step numbers are assigned to steps in which the same processing as the pulse wave detection processing shown in FIG. 7 is executed, and different step numbers are assigned to steps having different processing. .

  As shown in FIG. 9, when the camera image stored in the image memory 111 is updated, the non-biological extraction unit 410 acquires the camera image stored in the image memory 111 (step S401). Subsequently, the non-biological extraction unit 410 extracts a non-biological region from the camera image acquired in step S401 (step S402).

  And the 2nd acquisition part 420 determines the image of the specific area | region which has a specific positional relationship with the biological body area | region among the non-biological area | regions extracted by step S402 (step S403). Subsequently, the second calculation unit 124 calculates a representative value representing the specific area by executing predetermined statistical processing on the pixel value of each pixel included in the specific area acquired in step S403 (step S403). S404). Thereafter, the BPF 220 extracts time-series data of the representative value of the specific area, that is, a signal component in a frequency band that can be taken by the pulse wave from the luminance signal of the specific area (step S405).

  On the other hand, the first acquisition unit 112 acquires a camera image stored in the image memory 111 (step S304). Subsequently, the living body extraction unit 113 extracts a living body region from the camera image acquired in step S304 (step S305).

  Subsequently, the first calculation unit 114 calculates the representative value of the pixel value of each pixel included in the living body region extracted in step S305 (step S306). Then, the BPF 115 extracts time-series data of representative values of the living body region, that is, a signal component in a frequency band that can be taken by a pulse wave from the luminance signal of the living body region (step S307).

  After that, the similarity calculation unit 351 obtains the waveform of the luminance signal L′ i in the specific region from which the component of the frequency band that can be taken in step S405 is extracted, and the component of the frequency band that can be taken in step S307. The degree of similarity is calculated between the extracted waveform of the luminance signal Pi of the living body region (step S308).

  Then, the similarity determination unit 352 determines whether or not the similarity calculated in step S308 is greater than or equal to a predetermined threshold (step S309). Here, when the similarity is equal to or greater than the threshold (Yes in step S309), the luminance change of the screen display image is superimposed on the pulse wave, and the reflected light of the screen display image becomes noise and the pulse wave is erroneously detected. There is a high possibility of being. Therefore, the detection control unit 350 pauses the detection of the pulse wave by the detection unit 116 and ends the process as it is.

  On the other hand, when the similarity is less than the threshold (No in step S309), the luminance change of the display image on the screen is not superimposed on the pulse wave, and the reflected light of the display image on the screen becomes noise and the pulse wave is erroneous. It is unlikely to be detected. Therefore, the detection control unit 350 causes the detection unit 116 to detect a pulse wave from the luminance signal Pi of the living body region. In this case, the detection unit 116 detects the user's pulse wave from the luminance signal Pi of the living body region output by the BPF 115 (step S310), and ends the process.

  In the flowchart shown in FIG. 9, the example in which the processes in steps S304 to S307 are performed after the processes in steps S401 to S405 are illustrated is illustrated. However, the order in which these processes are performed is switched. Can also be executed in parallel.

[Effect of Example 4]
As described above, the pulse wave detection device 400 according to the present embodiment uses an image of a specific area having a specific positional relationship with a living body area among non-biological areas of a camera image as luminance information related to screen display. get. For this reason, in the pulse wave detection device 400 according to the present embodiment, even when it is difficult to acquire the actual display image of the screen displayed on the display unit 120, the screen light reflected in the camera image instead. The luminance change can be acquired. Therefore, according to the pulse wave detection device 400 according to the present embodiment, the application range of the pulse wave detection control can be expanded, so that versatility can be improved.

  In this embodiment, the difference between the third embodiment and the third embodiment has been described. However, the first embodiment can be combined with the present embodiment, and the second embodiment and the second embodiment can be combined. You can also.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[Other examples of detection results]
In Examples 1 to 4 described above, the case where the detection result of the pulse wave is used for health management is illustrated, but the detection result of the pulse wave can be used for other purposes besides health management. As an example, it is assumed that the pulse wave detection result is used for content evaluation. In this case, the pulse wave detection devices 100 to 400 associate the content image included in the content displayed on the display unit 120 with the detection result of the pulse wave at the time when the content image is displayed.

  For example, when the content is a moving image, each frame included in the moving image is associated with the pulse rate at the time when the frame is reproduced. As a result, it is possible to specify a frame in which the highest pulse rate is measured, or to evaluate the degree of excitement of the scene based on the level of the pulse rate corresponding to a specific scene or the fluctuation tendency of the pulse rate. For example, when the pulse rate is higher than a predetermined threshold or when the pulse rate tends to increase, it can be evaluated that the user is excited to see the scene.

  When the content is a website, each web page included in the website is associated with the pulse rate at the time when each web page is displayed. Accordingly, by monitoring the popular corner in the website by identifying the web page whose pulse rate is measured at the maximum value, or by identifying the web page whose pulse rate is lower than the average value, the website You can monitor the unpopular corners. In addition, although the case where the web page in one web site is used as the content image is illustrated here, the portal site or the summary site is the content, and each link destination website pasted on the portal site or the summary site is the content image. It is also possible to evaluate popular websites and unpopular websites.

  The content evaluation using the pulse rate can be executed in the pulse wave detection devices 100 to 400, or the association data in which the content image and the detection result of the pulse wave are associated with the plurality of pulse wave detection devices 100 to 100. It is also possible to cause the content evaluation apparatus to which 400 is connected to execute content evaluation. In this way, by evaluating the content with the content evaluation device to which the plurality of pulse wave detection devices 100 to 400 are connected, for example, the evaluation of the popularity or unpopularity of the content or the rise or fall of the content can be aggregated. it can.

[Other implementation examples]
In the first to fourth embodiments described above, the pulse wave detection devices 100 to 400 exemplify the case where the pulse wave detection processing is executed in a stand-alone manner. However, the pulse wave detection devices 100 to 400 may be implemented as a client server system. For example, the pulse wave detection devices 100 to 400 may be implemented as a Web server that provides a pulse wave detection service, or may be implemented as a cloud that provides a pulse wave detection service by outsourcing. As described above, when the pulse wave detection devices 100 to 400 operate as server devices, mobile terminal devices such as smartphones and mobile phones and information processing devices such as personal computers can be accommodated as client terminals. When a camera image is acquired from these client terminals via a network, a pulse wave detection process is executed, and a pulse wave detection result or a diagnosis result diagnosed using the detection result is returned to the client terminal. A wave detection service can also be provided.

[Distribution and integration]
In addition, each component of each illustrated apparatus does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, the first acquisition unit 112, the living body extraction unit 113, the first calculation unit 114, the BPF 115, the detection unit 116, the second acquisition unit 122, the second calculation unit 124, or the detection control unit 150 is connected to the external device of the pulse wave detection device 100. It is also possible to connect via a network. In addition, different devices each have the first acquisition unit 112, the biological extraction unit 113, the first calculation unit 114, the BPF 115, the detection unit 116, the second acquisition unit 122, the second calculation unit 124, or the detection control unit 150. You may make it implement | achieve the function of said pulse-wave detection apparatus 100 by connecting with a network and cooperating.

[Pulse wave detection program]
The various processes described in the above embodiments can be realized by executing a program prepared in advance on a computer such as a portable terminal device, a personal computer, or a workstation. In the following, an example of a computer that executes a pulse wave detection program having the same function as that of the above-described embodiment will be described with reference to FIG.

  FIG. 10 is a schematic diagram illustrating an example of a computer that executes a pulse wave detection program according to the first to fifth embodiments. As illustrated in FIG. 10, the computer 1000 includes an operation unit 1100 a, a speaker 1100 b, a camera 1100 c, a display 1200, and a communication unit 1300. Further, the computer 1000 includes a CPU 1500, a ROM 1600, an HDD 1700, and a RAM 1800. These units 1100 to 1800 are connected via a bus 1400.

  The HDD 1700 includes the first acquisition unit 112, the biological extraction unit 113, the first calculation unit 114, the BPF 115, the detection unit 116, the second acquisition unit 122, the second calculation unit 124, and the detection control unit described in the first embodiment. A pulse wave detection program 1700a that performs the same function as 150 is stored in advance. Further, the HDD 1700 includes the first acquisition unit 112, the biological extraction unit 113, the first calculation unit 114, the BPF 115, the detection unit 116, the second acquisition unit 122, the second calculation unit 124, and the BPF 220 described in the second embodiment. The pulse wave detection program 1700a that exhibits the same function as the detection control unit 250 may be stored in advance. Further, the HDD 1700 includes the first acquisition unit 112, the biological extraction unit 113, the first calculation unit 114, the BPF 115, the detection unit 116, the second acquisition unit 122, the second calculation unit 124, and the BPF 220 described in the third embodiment. The pulse wave detection program 1700a that exhibits the same function as the detection control unit 350 may be stored in advance. The HDD 1700 includes a first acquisition unit 112, a biological extraction unit 113, a first calculation unit 114, a BPF 115, a detection unit 116, a non-biological extraction unit 410, a second acquisition unit 420, a second acquisition unit, and the like. 2 The pulse wave detection program 1700a that exhibits the same functions as those of the calculation unit 124, the BPF 220, and the detection control unit 350 may be stored in advance. The pulse wave detection program 1700a may be appropriately integrated or separated as in the case of each functional unit shown in FIG. 1, FIG. 4, FIG. 6, or FIG. In other words, all the data stored in the HDD 1700 need not always be stored in the HDD 1700, and only the data necessary for processing may be stored in the HDD 1700.

  Then, the CPU 1500 reads the pulse wave detection program 1700a from the HDD 1700 and develops it in the RAM 1800. Accordingly, as shown in FIG. 10, the pulse wave detection program 1700a functions as a pulse wave detection process 1800a. This pulse wave detection process 1800a expands various data read from the HDD 1700 in an area allocated to itself on the RAM 1800 as appropriate, and executes various processes based on the expanded various data. The pulse wave detection process 1800a includes processing executed by each functional unit shown in FIG. 1, FIG. 4, FIG. 6, or FIG. 8, for example, processing shown in FIG. . In addition, each processing unit virtually realized on the CPU 1500 does not always require that all processing units operate on the CPU 1500, and only a processing unit necessary for processing may be virtually realized.

  Note that the pulse wave detection program 1700a is not necessarily stored in the HDD 1700 or the ROM 1600 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk inserted into the computer 1000, so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 1000 may acquire and execute each program from these portable physical media. Each program is stored in another computer or server device connected to the computer 1000 via a public line, the Internet, a LAN, a WAN, etc., and the computer 1000 acquires and executes each program from these. It may be.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary note 1) a first acquisition unit that acquires an image of a living body imaged by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit for acquiring luminance information related to display of a predetermined screen;
A pulse wave detection device, comprising: a detection control unit that controls whether or not to perform pulse wave detection by the detection unit using luminance information related to display of the screen.

(Additional remark 2) The said 2nd acquisition part acquires the display image of the said screen as the luminance information regarding the display of the said screen, The pulse-wave detection apparatus of Additional remark 1 characterized by the above-mentioned.

(Supplementary Note 3) The detection control unit
When the change in the representative value of luminance representing each pixel included in the display image of the screen is equal to or greater than a predetermined threshold, the detection unit does not detect a pulse wave, and the change in the representative value of luminance is less than the threshold The pulse wave detection device according to appendix 2, wherein the pulse wave is detected by the detection unit.

(Appendix 4) The pulse wave detection device
A second filter for extracting a component of a frequency band corresponding to the pulse wave from a luminance signal representing each pixel of the display image of the screen;
The detection control unit
When the amplitude value of the luminance signal of the screen from which the component of the frequency band is extracted is equal to or greater than a predetermined threshold, the detection unit does not detect a pulse wave, and the detection unit when the amplitude value is less than the threshold The pulse wave detection device according to attachment 2, wherein the pulse wave is detected.

(Appendix 5) The pulse wave detection device
A second filter for extracting a component of a frequency band corresponding to the pulse wave from a luminance signal representing each pixel of the display image of the screen;
The detection control unit
The similarity of the waveform is calculated between the luminance signal of the screen from which the components of the frequency band are extracted from each other and the luminance signal of the living body region, and if the similarity is equal to or greater than a predetermined threshold, a pulse wave is applied to the detection unit. 3. The pulse wave detection device according to appendix 2, wherein the pulse wave is detected by the detection unit when the similarity is less than the threshold without being detected.

(Additional remark 6) The said 2nd acquisition part acquires the image of the specific area | region which has a specific positional relationship between the said biological body area | region among the non-biological area | regions of the said image as brightness | luminance information regarding the display of the said screen. 2. The pulse wave detection device according to appendix 1, which is characterized.

(Appendix 7) The detection control unit
When the change in the representative value of luminance representing each pixel included in the specific region is equal to or greater than a predetermined threshold, the detection unit does not detect a pulse wave, and the change in the representative value of luminance is less than the threshold. 7. The pulse wave detection device according to appendix 6, wherein the pulse wave is detected by the detection unit.

(Appendix 8) The pulse wave detection device
A second filter for extracting a component of a frequency band corresponding to the pulse wave from a luminance signal representing each pixel of the specific region;
The detection control unit
The detection unit does not detect a pulse wave when the amplitude value of the luminance signal in the specific region from which the frequency band component is extracted is equal to or greater than a predetermined threshold, and the detection is performed when the amplitude value is less than the threshold. 7. The pulse wave detection device according to appendix 6, wherein the pulse wave is detected by the unit.

(Appendix 9) The pulse wave detection device
A second filter for extracting a component of a frequency band corresponding to the pulse wave from a luminance signal representing each pixel of the display image of the specific region;
The detection control unit
The similarity of the waveform is calculated between the luminance signal of the specific area and the luminance signal of the living body area from which the components of the frequency band are extracted from each other, and when the similarity is equal to or greater than a predetermined threshold, 7. The pulse wave detection device according to appendix 6, wherein the pulse wave is detected by the detection unit when the similarity is less than the threshold value.

(Appendix 10)
Get brightness information about the display of a given screen,
Using the luminance information related to the display of the screen, a process for acquiring an image of a living body captured by a camera, a process for extracting a biological area from the image, and a pulse wave from a luminance signal representative of each pixel included in the biological area Either the process of extracting the component of the frequency band that can be taken by or the process of detecting the pulse wave from the luminance signal of the living body region from which the component of the frequency band is extracted, or each process is executed A pulse wave detection program for executing a process for controlling the above.

(Appendix 11) The computer
Get brightness information about the display of a given screen,
Using the luminance information related to the display of the screen, a process for acquiring an image of a living body captured by a camera, a process for extracting a biological area from the image, and a pulse wave from a luminance signal representative of each pixel included in the biological area Either the process of extracting the component of the frequency band that can be taken by or the process of detecting the pulse wave from the luminance signal of the living body region from which the component of the frequency band is extracted, or each process is executed A pulse wave detection method characterized by executing a process for controlling the above.

(Appendix 12) A content evaluation system having a pulse wave detection device for detecting a pulse wave and a content evaluation device for evaluating content from a detection result of the pulse wave,
The pulse wave detector is
A first acquisition unit that acquires an image of a living body photographed by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit for acquiring luminance information related to display of a predetermined screen;
A detection control unit that controls whether or not to perform detection of a pulse wave by the detection unit, using luminance information related to display of the screen;
A content image included in the content displayed on the screen, and an association unit associating the detection result of the pulse wave at the time when the content image is displayed on the screen;
The content evaluation device includes:
A content evaluation system comprising: an evaluation unit that evaluates the content using a pulse wave detection result associated with the content image.

DESCRIPTION OF SYMBOLS 100 Pulse wave detection apparatus 110 Camera 111 Image memory 112 1st acquisition part 113 Living body extraction part 114 1st calculation part 115 BPF
116 detection unit 120 display unit 121 frame buffer 122 second acquisition unit 124 second calculation unit 130 illuminance sensor 150 detection control unit 151 illuminance determination unit 152 change amount determination unit

Claims (7)

A first acquisition unit that acquires an image of a living body photographed by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit that acquires a display image of the screen as luminance information related to display of the predetermined screen;
When the change in the representative value of luminance representing each pixel included in the display image of the screen is equal to or greater than a predetermined threshold, the detection unit does not detect a pulse wave, and the change in the representative value of luminance is less than the threshold And a detection control unit that causes the detection unit to detect a pulse wave. A first acquisition unit that acquires an image of a living body photographed by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit that acquires a display image of the screen as luminance information related to display of the predetermined screen;
A second filter that extracts a frequency band component corresponding to the pulse wave from a luminance signal representing each pixel of the display image of the screen;
When the amplitude value of the luminance signal of the screen from which the component of the frequency band is extracted is equal to or greater than a predetermined threshold, the detection unit does not detect a pulse wave, and the detection unit when the amplitude value is less than the threshold A detection control unit that detects pulse waves
A pulse wave detection device comprising: A first acquisition unit that acquires an image of a living body photographed by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit that acquires a display image of the screen as luminance information related to display of the predetermined screen;
A second filter that extracts a frequency band component corresponding to the pulse wave from a luminance signal representing each pixel of the display image of the screen;
The similarity of the waveform is calculated between the luminance signal of the screen from which the components of the frequency band are extracted from each other and the luminance signal of the living body region, and if the similarity is equal to or greater than a predetermined threshold, a pulse wave is applied to the detection unit. A detection control unit that causes the detection unit to detect a pulse wave when the similarity is less than the threshold without being detected;
A pulse wave detection device comprising: A first acquisition unit that acquires an image of a living body photographed by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit that acquires an image of a specific area having a specific positional relationship with the living body area among the non-biological areas of the image as luminance information related to display of a predetermined screen;
A pulse wave detection device, comprising: a detection control unit that controls whether or not to perform pulse wave detection by the detection unit using luminance information related to display of the screen. On the computer,
Acquire the display image of the screen as luminance information related to the display of the predetermined screen,
When a change in the representative value of luminance representing each pixel included in the display image on the screen is equal to or greater than a predetermined threshold, processing for obtaining an image in which a living body is photographed by a camera, and extracting a living body region from the image Processing, processing for extracting a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the biological region, or detecting a pulse wave from a luminance signal of the biological region from which the component of the frequency band has been extracted One of the processes is paused, and when the change in the representative value of the luminance is less than the threshold, a process for executing each process is executed . Computer
Acquire the display image of the screen as luminance information related to the display of the predetermined screen,
When a change in the representative value of luminance representing each pixel included in the display image on the screen is equal to or greater than a predetermined threshold, processing for obtaining an image in which a living body is photographed by a camera, and extracting a living body region from the image Processing, processing for extracting a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the biological region, or detecting a pulse wave from a luminance signal of the biological region from which the component of the frequency band has been extracted One of the processes is paused, and when the change in the representative value of the luminance is less than the threshold, a process for executing each process is executed . A content evaluation system having a pulse wave detection device for detecting a pulse wave and a content evaluation device for evaluating content from a detection result of the pulse wave,
The pulse wave detector is
A first acquisition unit that acquires an image of a living body photographed by a camera;
A biological extraction unit that extracts a biological region from the image;
A first filter that extracts a component of a frequency band that can be taken by a pulse wave from a luminance signal representing each pixel included in the living body region;
A detection unit for detecting a pulse wave from a luminance signal of the living body region from which the component of the frequency band is extracted;
A second acquisition unit that acquires a display image of the screen as luminance information related to display of the predetermined screen;
When the change in the representative value of luminance representing each pixel included in the display image of the screen is equal to or greater than a predetermined threshold, the detection unit does not detect a pulse wave, and the change in the representative value of luminance is less than the threshold If the detection control unit causes the detection unit to detect a pulse wave, the content image included in the content displayed on the screen is associated with the detection result of the pulse wave at the time when the content image is displayed on the screen And
The content evaluation device includes:
A content evaluation system comprising: an evaluation unit that evaluates the content using a pulse wave detection result associated with the content image.

Images