JP7340801B2 - Vital data output method, vital data output device and vital sensing system - Google Patents

Description

The present disclosure relates to a vital data output method, a vital data output device, and a vital sensing system.

Patent Document 1 discloses a video conference control method for controlling a video conference photographed by a first camera connected to a first display device. In this video conference control method, the processor determines that in the user environment, the direction of the face with respect to the first camera is farther from the front than the direction of the face with respect to the second camera connected to the second display device, or When the orientation of the face cannot be detected with the first camera and at least one of an input/output event and an application event is detected on the second display device, the video conference is switched to the second camera.

JP2017-108366A

However, if the configuration of Patent Document 1 described above is applied to, for example, a multi-display environment of a user who works while alternately viewing multiple monitors, the following problem may occur. Specifically, in such a multi-display environment, a user may perform operations such as data input while alternately viewing each of a plurality of monitors. For this reason, there is a possibility that the direction of the user's face, which is imaged by each of the plurality of cameras, may not be stable for a certain period of time. Therefore, when an image captured by a camera that captures the user is required in order to non-contactly measure the user's biological information (for example, pulse rate or stress index) in a multi-display environment, the orientation of the user's face is unstable. This poses a problem in that it becomes difficult to measure highly accurate biological information.

The present disclosure has been devised in view of the conventional circumstances described above, and provides a vital data output method and a vital data output device that measure a user's biological information with high precision in an environment where multiple cameras are placed around the user. and a vital sensing system.

The present disclosure is a vital data output method for measuring biometric information of a user with a plurality of cameras arranged around the user, wherein a captured image of the user captured by at least one camera among the plurality of cameras is obtained. and extracting a biosignal including the biometric information of the user based on the captured video, calculating the reliability of the extracted biosignal, and determining whether the calculated reliability is greater than or equal to a predetermined comparison reference value. If the condition is satisfied, the biological information of the user is estimated and output based on the biological signal , and if the condition is not satisfied that the reliability is equal to or higher than the predetermined comparison reference value, the condition is not satisfied. The present invention provides a vital data output method, which stores information of the camera for which the reliability is calculated and obtains a captured image captured by at least one camera among the cameras for which the reliability is not stored.

The present disclosure also provides a vital data output device that is communicably connected to a plurality of cameras, and which outputs biometric information of a user from an image captured by at least one camera among the plurality of cameras. a communication unit that receives a biosignal including a biosignal and a reliability of the biosignal, and a processor that verifies whether or not the reliability received by the communication unit is greater than or equal to a predetermined comparison reference value, the processor estimates and outputs the biometric information of the user based on the biosignal when the reliability satisfies the condition that the reliability is equal to or higher than the predetermined comparison reference value; If the above conditions are not met, the information of the camera for which the reliability was calculated that does not satisfy the condition is stored, and the captured image captured by at least one camera among the cameras that are not stored is obtained. Provides a vital data output device.

The present disclosure also provides a vital sensing system that measures biometric information of a user in which a plurality of cameras are arranged around the user, and at least one camera among the plurality of cameras collects a captured image of the user from a captured image of the user. extracts a biological signal including biological information, calculates the reliability of the extracted biological signal, transmits the biological signal and the reliability to the server, and the server calculates the reliability of the received biological signal. Estimating and outputting the biometric information of the user based on the biosignal when a condition that the reliability is equal to or greater than a predetermined comparison reference value is met, and the condition that the reliability is equal to or greater than the predetermined comparison reference value is not satisfied. In this case, there is provided a vital sensing system that stores information about the camera for which the reliability is calculated and does not satisfy the condition, and acquires a captured image taken by at least one camera among the cameras that are not stored. do.

According to the present disclosure, biometric information of a user can be measured with high precision in an environment where a plurality of cameras are arranged around the user.

An explanatory diagram of a use case example of the vital sensing system according to Embodiment 1 An explanatory diagram of a use case example of the vital sensing system according to Embodiment 1 viewed from above A diagram showing an example of the internal configuration of the vital sensing system according to Embodiment 1. Diagram explaining an example of calculating reliability of color change time series signal Sequence diagram showing an example of a pulse estimation procedure of the vital sensing system according to Embodiment 1 A diagram illustrating an example of a pulse estimation procedure of the vital sensing system according to a modification of Embodiment 1. A diagram illustrating an example of generation of a color change time series signal according to a modification of the first embodiment Internal configuration example of vital sensing system according to Embodiment 2 Sequence diagram showing an example of a pulse estimation procedure of the vital sensing system according to Embodiment 2 A diagram illustrating camera switching in the vital sensing system according to Embodiment 2

(Details leading to the content of Embodiment 1)
Conventionally, there is a video conference control method for controlling a video conference captured by a camera. In this video conference control method, in a multi-display environment, the direction of the face with respect to the first camera connected to the first monitor is farther from the front than the direction of the face with respect to the second camera connected to the second monitor. or when the first camera cannot detect the direction of the face and the presence or absence of a user operation on the second monitor or communication for a predetermined time or more is detected, the video conference is switched to the second camera.

However, when applying the above-mentioned configuration of Patent Document 1 to a multi-display environment where a user works while alternately viewing multiple monitors, the user may perform operations such as data input while alternately viewing each of the multiple monitors. There is a possibility of doing so. As a result, the orientation of the user's face that is imaged by each of the plurality of cameras is not stable over a certain period of time, and there is a possibility that the camera that images the user may be switched in response to a change in the orientation of the user's face. Therefore, when an image captured by a camera that captures the user is required in order to non-contactly measure the user's biological information (for example, pulse rate or stress index) in a multi-display environment, the orientation of the user's face is unstable. This poses a problem in that the user cannot be imaged for a period of time (for example, several seconds or one minute) during which biometric information can be measured, making it difficult to measure biometric information with high precision.

Furthermore, when measuring a user's biometric information, depending on environmental conditions such as lighting arrangement, the measured biometric information may be captured from the user's side (i.e., profile) or diagonally, rather than by using a captured image that captures the user from the front. The accuracy of measuring biological information may be higher if a captured video imaged from another direction is used.

Therefore, in each of the embodiments described below, a vital data output method, a vital data output device, and a vital sensing system that measure a user's biological information with high precision in an environment where multiple cameras are arranged around the user are described. An example will be explained.

EMBODIMENT OF THE INVENTION Hereinafter, each embodiment which specifically disclosed the structure and operation|movement of the vital data output method, the vital data output device, and the vital sensing system which concern on this disclosure is described in detail, referring drawings suitably. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of well-known matters or redundant explanations of substantially the same configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

Additionally, the terms used in the following description are illustrative and not intended to be limiting. For example, the term "camera" includes a captured image captured by the camera and a biological signal (for example, a color change time series signal) included in the captured image. In other words, "camera selection" used in the description of each embodiment refers to the selection of an image taken by this camera and the selection of a biological signal (for example, a color change time series signal) included in the imaged image. include. Furthermore, "estimation of biological information (pulse, heartbeat, stress index, etc.)" used in the following description includes "measurement of biological information (pulse, heartbeat, stress index, etc.)" and is not intended to be limiting.

(Embodiment 1)
An example use case of the vital sensing system 100 according to the first embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is an explanatory diagram of an example use case of the vital sensing system 100 according to the first embodiment. FIG. 2 is an explanatory diagram of a use case example of the vital sensing system 100 according to the first embodiment, viewed from above. In the vital sensing system 100 according to the first embodiment, a terminal device P1 and a camera C1 are connected for wired or wireless communication, and a terminal device P2 and a camera C2 are connected for wired or wireless communication. and sends and receives data.

Note that although FIG. 1 shows an example in which the number of terminal devices and the number of cameras are two each (that is, user H1 is in a dual display environment), the number of terminal devices and the number of cameras are two in the example. but not limited to. The number of terminal devices and the number of cameras may be, for example, three terminal devices and three cameras (that is, a multi-display environment), or may not be the same number. Further, the configuration of the vital sensing system 100 according to the first embodiment may include two or more cameras and two or more terminal devices to which the one or more cameras are communicably connected. Further, the camera may be a camera built into the terminal device.

The terminal device P1 is, for example, a desktop PC (Personal Computer), and is connected to the camera C1 via a network NW so as to be capable of wired or wireless communication. The terminal device P1 receives a captured image of the front of the user H1's face captured by the camera C1, and detects the position of the user H1's face appearing in the received captured image. The terminal device P1 sets an area (hereinafter referred to as a sensing area) for measuring changes in complexion on the face surface for the purpose of extracting pulse data (or heartbeat data) among the detected face positions, and A time-series signal (hereinafter referred to as a color-change time-series signal) indicating a change in the complexion color of the face surface of the user H1 in the sensing region is extracted. The terminal device P1 calculates the reliability for estimating the pulse measurement accuracy of the pulse data based on the waveform of the color change time series signal, and compares the extracted color change time series signal and the calculated reliability. It is transmitted to the server S1 and stored in the memory (see FIG. 3).

Further, the terminal device P1 performs time synchronization with the terminal device P2 in order to match the imaging time of the captured video captured by the camera C1 and the captured video captured by the camera C2. Note that the time synchronization may be performed by receiving time information from the server S1.

The camera C1 is connected to the terminal device P1 for wired or wireless communication. The camera C1 images the user H1 within the viewing angle Ag1, and transmits the captured image of the user H1 to the terminal device P1. Although the camera C1 shown in FIGS. 1 and 2 is shown as being externally attached to the terminal device P1, it may be a camera built into the terminal device P1.

The terminal device P2 is, for example, a notebook PC equipped with a camera C2, and is connected to the server S1 via the network NW so as to be capable of wired or wireless communication. The terminal device P2 receives a captured image of the side face (that is, profile) of the user H1's face captured by the camera C2, and detects the position of the user H1's face appearing in the received captured image. The terminal device P2 sets a sensing area that includes the position of the detected face and measures changes in facial color for the purpose of extracting pulse data, and detects the facial color of the user H1 in the set sensing area. Extract time-series signals showing changes. The terminal device P2 calculates the reliability for estimating the pulse measurement accuracy of the pulse data based on the waveform of the color change time series signal, and sends the extracted color change time series signal and the calculated reliability to the server S1. and stores it in memory (see FIG. 3). Note that the sensing area may be changed depending on the position of the user H1's face.

Further, the terminal device P2 performs time synchronization with the terminal device P1 in order to match the imaging time of the captured video captured by the camera C2 and the captured video captured by the camera C1. Note that the time synchronization may be performed by receiving time information from the server S1.

The camera C2 is provided in the terminal device P2 and captures an image of the user H1 within the viewing angle Ag2. The camera C2 images the user H1 within the viewing angle Ag2, and transmits the captured image of the user H1 to the processor in the terminal device P2 (see FIG. 3).

Note that each of the plurality of terminal devices P1 and P2 is not limited to a PC, and may be, for example, a smartphone, a tablet terminal, or the like. Further, each of the plurality of cameras C1 and C2 may be, for example, a camera included in a smartphone or a tablet terminal, or a camera connected to enable wired or wireless communication.

FIG. 3 is a diagram showing an example of the internal configuration of the vital sensing system 100 according to the first embodiment. The vital sensing system 100 according to the first embodiment is configured to include each of a plurality of terminal devices P1 and P2, each of a plurality of cameras C1 and C2, a server S1, and a network NW. In the example shown in FIG. 3, an example is shown in which the terminal device P1 and the camera C1 are connected to enable wired communication, but it goes without saying that the present invention is not limited to this. Similarly, the method of communication between each of the plurality of terminal devices P1 and P2 and the server S1 via the network NW is not limited to the wireless communication shown in FIG. 3.

Terminal device P1 is communicably connected to camera C1 and server S1. The terminal device P1 transmits the color change time series signal extracted from the captured video imaged by the camera C1 and the calculated reliability to the server S1 via the network NW. The terminal device P1 includes a communication section 20, a processor 21, a memory 22, and a monitor 23.

The communication unit 20 is connected to the communication unit 40 in the camera C1 and the communication unit 10 in the server S1 for wired or wireless communication. The wired communication here includes, for example, a USB (Universal Serial Bus) cable (not shown), an HDMI (registered trademark) (High-Definition Multimedia Interface) cable (not shown), or a LAN (Local Area Network) cable (not shown). The This is the communication used. Furthermore, the wireless communication referred to here is, for example, short-range wireless communication such as Bluetooth (registered trademark) or NFC (registered trademark), or communication via a wireless LAN such as Wi-fi (registered trademark).

The communication unit 20 receives the captured video of the user H1 from the communication unit 40 of the camera C1, stores it in the memory 22, and outputs it to the processor 21. Further, the communication unit 20 calculates the pulse rate of the pulse data of the user H1 based on the color change time series signal indicating the change in complexion of the face surface of the user H1 extracted by the processor 21 and the waveform of the color change time series signal. The reliability for estimating the measurement accuracy is transmitted to the communication unit 10 in the server S1 via the network NW.

The processor 21 is configured using, for example, a CPU (Central Processing unit) or an FPGA (Field Programmable Gate Array), and performs various processing and control in cooperation with the memory 22. Specifically, the processor 21 has a function of detecting the position of the face of the user H1 from the captured video and detecting the color of the user H1 from the sensing area by referring to the program and data held in the memory 22 and executing the program. It realizes functions such as extracting color change time series signals and calculating reliability of extracted color change time series signals.

The processor 21 detects an area where the face of the user H1 is shown from the captured video captured by the camera C1. The processor 21 sets a sensing area for measuring changes in complexion on the surface of the user H1's face among the detected facial areas, and generates a color change time series signal based on changes in the complexion of the user H1 from this sensing area. Extract. The processor 21 calculates the reliability for estimating the pulse measurement accuracy of the pulse data of the user H1 based on the waveform of the color change time series signal. The processor 21 transmits the extracted color change time series signal and the calculated reliability to the server S1 via the network NW, and stores them in the memory 22.

Further, the processor 21 communicates with the processor 31 in the terminal device P2 or the processor 11 in the server S1 so that the imaging time of the captured video captured by the camera C1 and the captured video captured by the camera C2 do not deviate from each other. Perform time synchronization between

The memory 22 includes, for example, a RAM (Random Access Memory) as a work memory used when the processor 21 executes each process, and a ROM (Read Only Memory) that stores programs and data that define the operations of the processor 21. have Data or information generated or acquired by the processor 21 is temporarily stored in the RAM. A program that defines the operation of the processor 21 is written in the ROM. The memory 22 stores the captured image received from the camera C1, the color change time series signal extracted from the captured image, the calculated reliability, and the like.

The monitor 23 is configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence). The monitor 23 displays the extracted color change time series signal. In addition, when receiving the color change time series signal extracted from the terminal device P2 via the network NW, the monitor 23 receives the color change time series signal extracted based on the captured video imaged by the camera C1, and the color change time series signal extracted based on the captured video imaged by the camera C1. The color change time series signal extracted based on the captured image captured by C2 may be displayed side by side. Furthermore, when receiving the pulse estimation result of the user H1 from the server S1, the monitor 23 may also display this pulse estimation result. The user H1 may change the display target (for example, a plurality of color change time series signals, a color change time series signal with higher reliability, a pulse, a stress index, etc.) to be displayed on the monitor 23 through an input operation.

Next, the internal configuration of the terminal device P2 will be explained. The terminal device P2 includes a communication section 30, a processor 31, a memory 32, a monitor 33, and a camera C2.

The communication unit 30 is connected to the communication unit 10 in the server S1 for wired or wireless communication. Note that the communication unit 30 may be communicably connected to the communication unit 20 in the terminal device P1 as well. The wired communication here is communication using, for example, a USB cable (not shown) or a LAN cable (not shown). The wireless communication referred to here is, for example, short-range wireless communication such as Bluetooth (registered trademark) or NFC (registered trademark), or communication via a wireless LAN such as Wi-fi (registered trademark).

The communication unit 30 calculates the pulse rate measurement accuracy of the user H1's pulse data based on the color change time series signal indicating the change in complexion of the face surface of the user H1 extracted by the processor 31 and the waveform of the color change time series signal. , and the reliability for estimating , is transmitted to the communication unit 10 in the server S1 via the network NW.

The processor 31 is configured using, for example, a CPU or an FPGA, and performs various processing and control in cooperation with the memory 32. Specifically, the processor 31 refers to the program and data held in the memory 32 and executes the program to detect the face of the user H1 from the captured video and detect the color change of the user H1 from the sensing area. It realizes functions such as extracting time-series signals and calculating reliability of extracted color change time-series signals.

The processor 31 detects an area where the face of the user H1 is shown from the captured video captured by the camera C2. The processor 31 sets a sensing area for measuring changes in complexion on the surface of the user H1's face among the detected facial areas, and generates a color change time series signal based on changes in the complexion of the user H1 from this sensing area. Extract. The processor 31 calculates the reliability for estimating the pulse measurement accuracy of the pulse data of the user H1 based on the waveform of the color change time series signal. Note that the processor 31 may calculate the reliability for each waveform, or for each predetermined period of time. The processor 31 transmits the extracted color change time series signal and the calculated reliability to the server S1 via the network NW, and stores them in the memory 32.

Note that the processor 31 communicates with the processor 21 in the terminal device P1 or the processor 11 in the server S1 in order to match the imaging time of the captured video captured by the camera C2 with the imaging time of the captured video captured by the camera C2. Perform time synchronization.

The memory 32 includes, for example, a RAM as a work memory used when the processor 31 executes each process, and a ROM that stores programs and data that define the operations of the processor 31. Data or information generated or acquired by the processor 31 is temporarily stored in the RAM. A program that defines the operation of the processor 31 is written in the ROM. The memory 32 stores the captured video input from the camera C2, the color change time series signal extracted from the captured video, the calculated reliability, and the like.

The monitor 33 is configured using, for example, an LCD or an organic EL. The monitor 33 displays the extracted color change time series signal. In addition, when receiving the color change time series signal extracted from the terminal device P1 via the network NW, the monitor 33 receives the color change time series signal extracted based on the captured image captured by the camera C1, and the color change time series signal extracted based on the captured video imaged by the camera C1. The color change time series signal extracted based on the captured image captured by C2 may be displayed side by side. Furthermore, when receiving the pulse estimation result of the user H1 from the server S1, the monitor 33 may also display this pulse estimation result. The user H1 may change the display target (for example, a plurality of color change time series signals, a color change time series signal with higher reliability, a heartbeat, a pulse, a stress index, etc.) to be displayed on the monitor 33 through an input operation.

The camera C2 includes at least a lens (not shown) and an image sensor (not shown). The image sensor is a solid-state imaging device such as a CCD (Charged-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor), and converts an optical image formed on an imaging surface into an electrical signal. Camera C2 outputs the captured image to processor 31.

Next, the camera C1 will be explained. The camera C1 is communicably connected to the terminal device P1, and transmits the captured image to the terminal device P1. The camera C1 includes a communication section 40, a processor 41, a memory 42, and an imaging section 43.

The communication unit 40 is connected to the communication unit 20 in the terminal device P1 so that data can be communicated by wire or wirelessly. The wired communication here is communication using, for example, a USB cable (not shown), an HDMI (registered trademark) cable (not shown), or a LAN cable (not shown). Furthermore, the wireless communication referred to here is, for example, short-range wireless communication such as Bluetooth (registered trademark) or NFC (registered trademark), or communication via a wireless LAN such as Wi-fi (registered trademark). The communication unit 40 transmits the captured video imaged by the imaging unit 43 to the terminal device P1.

The processor 41 is configured using, for example, a CPU or an FPGA, and performs various processing and control in cooperation with the memory 42. The processor 41 stores the inputted captured video captured by the imaging unit 43 in the memory 42, and transmits it to the communication unit 20 in the terminal device P1.

The memory 42 includes, for example, a RAM as a work memory used when the processor 41 executes each process, and a ROM that stores programs and data that define the operations of the processor 41. Data or information generated or acquired by the processor 41 is temporarily stored in the RAM. A program that defines the operation of the processor 41 is written in the ROM. The memory 42 stores captured images captured by the imaging unit 43.

The imaging unit 43 includes at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a CCD or CMOS solid-state imaging device, and converts an optical image formed on an imaging surface into an electrical signal. The imaging unit 43 outputs the captured image to the processor 41.

Next, the internal configuration of the server S1 will be explained. The server S1 is connected to each of the plurality of terminal devices P1 and P2 via the network NW1 so as to be capable of wireless communication. The server S1 includes a communication unit 10, a processor 11, and a memory 12.

The communication unit 10 is connected to enable wireless communication with the communication unit 20 in the terminal device P1 and the communication unit 30 in the terminal device P2. The wireless communication referred to here is, for example, short-range wireless communication such as Bluetooth (registered trademark) or NFC (registered trademark), or communication via a wireless LAN such as Wi-fi (registered trademark).

The communication unit 10 receives the color change time series signal of the user H1 based on the captured video imaged by the camera C1 and the reliability from the communication unit 20 of the terminal device P1. The communication unit 10 also receives the color change time series signal of the user H1 based on the captured video imaged by the camera C2 and the reliability from the communication unit 30 in the terminal device P2. The communication unit 10 outputs these color change time series signals and reliability to the processor 11 and stores them in the memory 12. Furthermore, the communication unit 10 transmits information on the pulse rate, stress index, etc. of the user H1 estimated by the processor 11 to each of the plurality of terminal devices P1 and P2 via the network NW1.

The processor 11 is configured using, for example, a CPU or an FPGA, and performs various processing and control in cooperation with the memory 12. Specifically, the processor 11 refers to the program and data held in the memory 12 and executes the program to detect the face of the user H1 from the captured video and detect the color change of the user H1 from the sensing area. It realizes functions such as extracting time-series signals and calculating reliability of extracted color change time-series signals.

The processor 11 tests the reliability received from the terminal device P1 and the reliability received from the terminal device P2. Specifically, the processor 11 examines and selects the color change time series signal with the highest reliability among each of the received reliability.

In addition, in order to test whether the extracted color change time series signal is a color change time series signal that can estimate the pulse rate with high accuracy, the processor 11 may perform the test using a threshold value. If the highest reliability calculated during the test does not exceed the threshold, the processor 11 does not estimate the pulse and generates an alert to notify the user H1 that pulse data cannot be measured, and It may be transmitted to each of P1 and P2. Note that the threshold value is set in advance and stored in the memory 12. Each of the plurality of terminal devices P1 and P2 displays the received alert on the monitor and notifies the user H1.

The processor 11 estimates the pulse rate of the user H1 based on the color change time series signal selected as a result of the test. The processor 11 stores the pulse rate data of the user H1 as the estimation result in the memory 12. Furthermore, the processor 11 outputs the pulse data of the user H1 to the communication unit 10. The pulse data of the user H1 input to the communication unit 10 is transmitted to each of the plurality of terminal devices P1 and P2 via the network NW1. Note that if the user H1 has previously set a terminal device to display the user H1's pulse data, the pulse rate data estimated by the processor 11 may be transmitted only to the set terminal device.

The processor 11 calculates the stress index of the user H1 based on the estimated pulse data of the user H1. Further, the processor 11 may determine whether the user H1 is present/vacant (absent) based on the color change time series signal and the reliability.

Note that the processor 11 may execute a control instruction to synchronize the respective times of the plurality of terminal devices P1 and P2 so that the imaging times of the captured images captured by each of the plurality of cameras C1 and C2 do not deviate. .

The memory 12 includes, for example, a RAM as a work memory used when the processor 11 executes each process, and a ROM that stores programs and data that define the operations of the processor 11. Data or information generated or acquired by the processor 11 is temporarily stored in the RAM. A program that defines the operation of the processor 11 is written in the ROM. The memory 12 stores the color change time series signal and reliability received from each terminal device, estimated pulse data, stress index, or seat presence/vacancy (absence) information for each user H1.

The network NW1 is communicably connected to the server S1 and each of the plurality of terminal devices P1 and P2.

FIG. 4 is a diagram illustrating an example of calculating the reliability of a color change time-series signal. Table TB1 shown in FIG. 4 is a table that compares captured images captured by each of the plurality of cameras C1 and C2, face positions (that is, face orientations), color change time series signals, and test results.

The captured video is a captured video captured by each of the plurality of cameras C1 and C2. Camera C1 captures an image of the front of user H1, and transmits the captured image to terminal device P1. Camera C2 images the side surface (that is, profile) of user H1, and transmits the captured image to terminal device P2.

The terminal device P1 determines that the face position of the user H1 shown in the received captured image is "front", and sets a sensing area SAr1 for extracting a color change time series signal from the user H1's face. Note that the sensing area SAr1 that is set may be an area in which it is determined that the facial color change of the user H1 is large through execution of image processing. The terminal device P1 extracts the color change time series signal Vt1 from the set sensing area SAr1.

Furthermore, the terminal device P2 determines that the face position of the user H1 shown in the received captured image is the "side face", and sets a sensing area SAr2 for extracting a color change time series signal from the user H1's face. Note that the sensing area SAr2 setting may be an area in which it is determined that the facial color change of the user H1 is large through execution of image processing. The terminal device P2 extracts the color change time series signal Vt2 from the set sensing area SAr2.

Note that each of the color change time series signals Vt1 and Vt2 extracted here is extracted over time T0 or more, which is a time period during which the stress index can be calculated. That is, the imaged video imaged by the camera C1 is imaged over time T0 or more so that the stress index can be calculated. Note that the time T0 for which the stress index can be calculated here is specifically several seconds or one minute, but is not limited to this and may be any time that allows the stress index to be calculated.

The terminal device P1 calculates the reliability of the extracted color change time series signal Vt1. Furthermore, the terminal device P2 calculates the reliability of the extracted color change time series signal Vt2. The method for calculating reliability will be explained below.

The reliability is an index indicating the reliability of the biosignal (that is, the color change time series signal), and is, for example, an index indicating the accuracy, suitability, or periodicity of the biosignal. The reliability is calculated by each of the terminal devices P1 and P2, for example, based on the waveform of the color change time series signal. First, each of the terminal devices P1 and P2 tests whether the period of the waveform of the color change time series signal extracted from the sensing area is within the range of the pulse period, and as a result of the test, the color change time series signal is When the period of the signal waveform is the period of a pulse, the root mean square error of the waveform amplitude and period of the color change time series signal is calculated. Each of the terminal devices P1 and P2 calculates reliability as an index indicating the degree of pulse measurement accuracy of the estimated color change time series signal according to the magnitude of the calculated root mean square error of the amplitude and period. .

Note that the reliability calculation method is not limited to the method described above. The reliability is calculated based on a model learned using each of the waveform data and pulse data of a plurality of color change time series signals learned by machine learning (hereinafter referred to as a learned model). good. The trained model may be stored in the memory of each of the terminal devices P1 and P2, and executed by the processor in the terminal device P1 and P2. Further, the trained model may be generated from learning data stored in the memory 12 in the server S1, and may be transmitted to each of the terminal devices P1 and P2. The trained model is stored in the memory included in each of the terminal devices P1 and P2.

Note that each of the waveform data and pulse rate data of the multiple color change time series signals used to generate the trained model is not limited to user H1 whose biological information (pulse rate, stress index, etc.) is to be measured. The information may be collected from each of a plurality of users who use the sensing system 100. Thereby, the vital sensing system 100 can generate a learned model from more learning data, and therefore can measure the biological information (pulse, stress index, etc.) of the user H1 with high precision.

Furthermore, trained models may be generated for each user. When a trained model for each user is used, each of the plurality of terminal devices P1 and P2 can, for example, adjust the lighting arrangement, natural light, arrangement of other objects, and the user's face at the position where the user H1 works (i.e., the imaging position). It is possible to calculate reliability that takes into account (adjusts) environmental conditions such as features (bruises, scars, skin color). Thereby, the vital sensing system 100 can measure the biological information (pulse, stress index, etc.) of the user H1 with high precision according to the environment in which multiple cameras are arranged around the user H1.

In addition, the reliability may be calculated for each predetermined time or period, or may be calculated for each arbitrary time among the extracted color change time series signals in a predetermined time period.

Although the reliability calculation method has been described above, the method for testing (selecting) which color change time series signal to use among the plurality of color change time series signals is not limited to the example based on reliability. The reliability may be calculated based on the standard deviation, or may be calculated using other statistical estimation methods. Further, the reliability may be replaced with terms such as evaluation value, score, index, etc., for example.

The terminal device P1 transmits the reliability calculated based on the color change time series signal Vt1 and the color change time series signal Vt1 to the server S1 via the network NW1. Further, the terminal device P2 transmits the reliability calculated based on the color change time series signal Vt2 and the color change time series signal Vt2 to the server S1 via the network NW1.

The server S1 tests the highest reliability among the received reliability. In the example shown in FIG. 4, the server S1 selects the color change time series signal Vt2 with the highest reliability as a signal from which the pulse of the user H1 can be estimated (that is, the pulse measurement accuracy is higher than a predetermined accuracy). The server S1 estimates the pulse of the user H1 based on the color change time series signal Vt2, and calculates a stress index based on the estimated pulse.

Here, the number of cycles and time of the color change time series signal required to calculate reliability are smaller (shorter) than the number of cycles and time required to estimate pulse, heartbeat, and the like. As a result, the server S1 can calculate the reliability of the color change time series signal more quickly, and based on the calculated reliability, the server S1 can calculate the reliability of the color change time series signal using the color change time series signal that is calculated to have high pulse measurement accuracy of the pulse data. The biological information (pulse) of the user H1 can be estimated (measured). In this way, by making the number of cycles or the time required to calculate reliability smaller than the time required to estimate pulse rate, heart rate, etc., server S1 can, for example, collect biological information (pulse rate, stress index, etc.) in real time. When outputting, when a threshold value is set for reliability, processing such as selection of color change time series signal used for estimating (measuring) biological information, selection of camera, etc. is performed according to the calculated reliability. Able to execute adaptively and efficiently.

Further, the number of cycles and time of the color change time series signal required to calculate reliability may be larger (longer) than the number of cycles and time required to estimate pulse, heartbeat, and the like. In such a case, the server S1 can improve the reliability calculation accuracy itself. In other words, the server S1 estimates (measures) the biological information (pulse) of the user H1 using the color change time series signal calculated to have high pulse measurement accuracy of the pulse data based on the reliability calculated with high accuracy. ), which is useful, for example, when the pulse and heartbeat of the user H1 are not estimated in real time.

Furthermore, among the color change time series signals, the time period in which the reliability is less than the threshold is limited, and from other color change time series signals whose reliability is higher than the threshold, user H1's pulse, stress index, etc. If it is possible to estimate or calculate the user H1's pulse, stress index, etc., the server S1 may estimate or calculate the user H1's pulse, stress index, etc. from other color change time series signals whose reliability is equal to or higher than the threshold.

FIG. 5 is a sequence diagram showing an example of a pulse estimation procedure of the vital sensing system 100 according to the first embodiment. In FIG. 5, an example will be described in which the server S1 estimates the pulse rate from the color change time series signal with the highest reliability.

First, the processing executed by the terminal device P1 will be explained. The terminal device P1 receives and acquires a captured image of the user H1 from the camera C1 (St1A).

The terminal device P1 detects the face position of the user H1 from the acquired captured image (St2A), and extracts a sensing area from the detected face position to measure a change in complexion on the face surface for the purpose of extracting the pulse. Then, a color change time series signal of the face surface of user H1 in the extracted sensing region is extracted (St4A). Note that when the position of the user H1's face changes in the captured video, the sensing area may be changed according to the position of the user H1's face.

The terminal device P1 calculates reliability indicating the accuracy of pulse measurement of the user H1 based on the waveform of the extracted color change time series signal (St5A). The terminal device P1 transmits the extracted color change time series signal and the calculated reliability to the server S1 via the network NW1 (St6A).

Next, the processing executed by the terminal device P2 will be explained. The terminal device P2 receives and acquires the captured video of the user H1 from the camera C2 (St1B).

The terminal device P2 detects the face position of the user H1 from the acquired captured image (St2B), and extracts a sensing area from the detected face position to measure changes in complexion on the face surface for the purpose of extracting the pulse. (St3B), and extracts a color change time series signal of the face surface of user H1 in the extracted sensing region (St4B). Note that when the position of the user H1's face changes in the captured video, the sensing area may be changed according to the position of the user H1's face.

The terminal device P2 calculates the reliability for estimating the pulse measurement accuracy of the user H1's pulse data based on the waveform of the extracted color change time series signal (St5B). The terminal device P2 transmits the extracted color change time series signal and the calculated reliability to the server S1 via the network NW1 (St6B).

The server S1 verifies the reliability received from the terminal device P1 and the reliability received from the terminal device P2, and selects the color change calculated with the highest reliability as the color change time series signal for estimating the pulse. A time series signal is selected (St7).

The server S1 estimates the pulse rate of the user H1 from the selected color change time series signal (St8).

As described above, the vital sensing system 100 according to the first embodiment measures the pulse (an example of biological information) of the user H1 around whom a plurality of cameras C1 and C2 are arranged. The vital sensing system 100 acquires a captured image of the user H1 captured by at least one camera among the plurality of cameras C1 and C2, and generates a color change time series signal (biological body) containing pulse information of the user based on the captured image. An example of a signal) is extracted, and the reliability of the extracted color change time series signal is calculated. The vital sensing system 100 satisfies the conditions under which the calculated reliability is equal to or higher than a predetermined comparison reference value (for example, the reliability of color change time series signals acquired from other cameras, a preset threshold for reliability, etc.) If the following is satisfied, the user's pulse is estimated (measured) based on the color change time series signal and output.

As a result, the vital sensing system 100 according to the first embodiment measures the biological information (pulse) of the user H1 with higher precision in an environment where the plurality of cameras C1 and C2 are respectively arranged around the user H1. can. In addition, the vital sensing system 100 according to the modification of the first embodiment uses the color change time series signal that has higher reliability, that is, the pulse rate measurement accuracy of the estimated (measured) pulse data, to detect the user H1's biological body. Information (pulse) can be estimated (measured).

Furthermore, in the vital sensing system 100 according to the first embodiment, the time required to calculate reliability is shorter than the time required to estimate (measure) biological information (pulse, stress index, etc.). As a result, the vital sensing system 100 according to the first embodiment can calculate reliability faster than biological information, so when outputting biological information (pulse, stress index, etc.) in real time, for example, a threshold value is set for reliability. or when each of a plurality of cameras is switched to take an image as in the vital sensing system 100a according to Embodiment 2, which will be described later, the estimation (measurement) of biological information is ) and camera selection (switching) can be performed adaptively and efficiently. Therefore, the vital sensing system 100 according to Embodiment 1 uses the color change time series signal calculated to have high pulse rate measurement accuracy of the pulse data based on the calculated reliability to determine the biological information (pulse rate) of the user H1. ) can be estimated (measured).

Furthermore, the biological signal (that is, the color change time series signal) extracted in the vital sensing system 100 according to the first embodiment has periodicity. Here, the number of cycles for which reliability can be calculated based on the biosignal is smaller than the number of cycles for which biometric information (pulse) can be estimated. Thereby, vital sensing system 100 according to Embodiment 1 can calculate reliability from biological signals with fewer cycles. In other words, the vital sensing system 100 can calculate reliability faster than biological information, so for example, when outputting biological information (pulse, stress index, etc.) in real time, when a threshold is set for reliability, When a plurality of cameras are switched to take images as in the vital sensing system 100a according to the second embodiment, the color change time series used for estimating (measuring) biological information is determined according to the calculated reliability. Signal selection, camera selection (switching), etc. can be performed adaptively and efficiently. Therefore, the vital sensing system 100 according to Embodiment 1 uses the color change time series signal calculated to have high pulse rate measurement accuracy of the pulse data based on the calculated reliability to determine the biological information (pulse rate) of the user H1. ) can be estimated (measured).

Further, the time required to calculate the reliability calculated by the vital sensing system 100 according to the first embodiment is longer than the time required to estimate (measure) biological information (pulse, stress index, etc.). In other words, since the vital sensing system 100 according to the first embodiment calculates the reliability over a long period of time, the calculated reliability itself is highly accurate. As a result, the vital sensing system 100 according to the first embodiment can calculate the reliability of the signal used for estimating (measuring) biological information with higher accuracy, and based on this calculated reliability, the vital sensing system 100 The biometric information (pulse) of the user H1 can be estimated (measured) using the color change time series signal calculated to have high pulse rate measurement accuracy.

Furthermore, in the vital sensing system 100 according to the first embodiment, the calculated reliability is less than the comparison reference value (for example, the reliability of the color change time series signal acquired from another camera, threshold, etc.), the biological information (pulse, stress index, etc.) estimated during reliability calculation is not output. As a result, the vital sensing system 100 according to the first embodiment can generate a color change time series signal that is calculated to have high pulse measurement accuracy for pulse data that has been calculated to have low pulse measurement accuracy over a long period of time, for example. The biometric information (pulse) of the user H1 can be estimated (measured) using the .

In addition, the vital sensing system 100 according to the first embodiment includes a color change time series signal extracted based on captured images captured by a camera C1 that captures an image of the user H1 and one or more other cameras (for example, a camera C2). Compare the reliability of each. Thereby, the vital sensing system 100 according to the first embodiment can measure the biological information (pulse, stress index, etc.) of the user H1 with high precision using the color change time series signal with higher reliability.

(Modification of Embodiment 1)
The vital sensing system 100 according to the first embodiment extracts the stress index over a period of time longer than which it is possible to calculate it (specifically, several seconds or one minute) and extracts it into one color change time series signal with the highest reliability. An example of estimating the pulse rate of the user H1 based on the above example has been shown. In the vital sensing system 100 according to the modification of the first embodiment, each of a plurality of color change time series signals extracted over a period of time (specifically, several seconds or one minute) in which a stress index can be calculated is An example in which the reliability is calculated in at least two intervals and the pulse rate of the user H1 is estimated based on the color change time series signal with the highest reliability in a predetermined interval will be described.

With reference to FIGS. 6 and 7, a pulse estimation procedure and a generation example of a color change time series signal of the vital sensing system 100 according to a modification of the first embodiment will be described. FIG. 6 is a sequence diagram showing an example of a pulse estimation procedure of the vital sensing system 100 according to a modification of the first embodiment. FIG. 7 is a diagram illustrating an example of generation of a color change time series signal according to a modification of the first embodiment. Note that the pulse estimation procedure example of the vital sensing system 100 according to the modification of the first embodiment shown in FIG. Almost the same process as the pulse estimation procedure example of the vital sensing system 100 according to the first embodiment is executed. Therefore, the same reference numerals will be used for the same operating procedures or operating procedures as in Embodiment 1, and the description thereof will be omitted. Moreover, the color change time series signal Vt3 shown in FIG. 7 shows an example extracted by the terminal device P1 using the captured video imaged by the camera C1. Similarly, the color change time series signal Vt4 shown in FIG. 7 shows an example extracted by the terminal device P2 using the captured video imaged by the camera C2.

The terminal device P1 calculates the reliability indicating the pulse measurement accuracy of the user H1 for each predetermined time period m1, m2, m3, m4, and m5 based on the waveform of the extracted color change time series signal Vt3 (St51A ). Here, each of the plurality of time periods m1 to m5 shown in FIG. 7 is set by the processor 21, and is based on the periodicity or shape of the waveform of the extracted color change time series signal Vt3. The time period in which the waveform is periodic (that is, the period in which it is estimated to be calculated with high reliability) and the period in which it is not periodic (in other words, the period in which it is estimated to be calculated with low reliability) It is set to be classified.

Note that the interval for calculating the reliability is not limited to the above-mentioned time period, but may be for each predetermined time period or for each predetermined cycle. Furthermore, although FIG. 7 shows an example in which each of the plurality of time periods m1 to m5 is the same time period in each of the color change time series signals Vt3 and Vt4, the interval for calculating reliability is It may be set for each changing time series signal.

The terminal device P1 stores the extracted color change time series signal Vt3 in association with each calculated reliability for each time period, and stores the color change time series signal Vt3 and the time of the color change time series signal Vt3. The reliability calculated for each band is transmitted to the server S1 via the network NW1 (St61A).

The terminal device P2 calculates the reliability indicating the pulse measurement accuracy of the user H1 for each predetermined time period m1, m2, m3, m4, and m5 based on the waveform of the extracted color change time series signal Vt4 (St51B ). Here, each of the plurality of time periods m1 to m5 shown in FIG. 7 is set by the processor 31, and is based on the periodicity or shape of the waveform of the extracted color change time series signal Vt4. The time period in which the waveform is periodic (that is, the period in which it is estimated to be calculated with high reliability) and the period in which it is not periodic (in other words, the period in which it is estimated to be calculated with low reliability) It is set to determine and classify these.

The terminal device P2 stores the extracted color change time series signal Vt4 in association with each calculated reliability for each time period, and stores the color change time series signal Vt4 and the time of the color change time series signal Vt4. The reliability calculated for each band is transmitted to the server S1 via the network NW1 (St61B).

The server S1 determines the reliability for each time period associated with the color change time series signal Vt3 received from the terminal device P1 and the time zone associated with the color change time series signal Vt4 received from the terminal device P2. The color change time series signals for each time period that were calculated to have higher reliability as the color change time series signals for estimating the pulse (color change time series signals shown in Figure 7) were tested. Each of the section signals Vt31 and Vt32 in Vt3 and the section signals Vt41 and Vt42 in the color change time series signal Vt4 are extracted. The server S1 generates one color change time series signal Vt5 from each of the extracted color change time series signals in each time period (St7C). Note that the generated color change time series signal Vt5 has a length of one minute or more so that the stress index can be calculated.

In addition, if the time periods (sections) set for each of the plurality of color change time series signals Vt3 and Vt4 are different and there are overlapping sections according to the test, the server S1 determines the time period (section) with higher reliability. One color change time series signal Vt5 may be generated by preferentially extracting the signal of .

The server S1 estimates the pulse rate of the user H1 based on the generated color change time series signal Vt5 (St8).

As described above, the vital sensing system 100 according to the modification of the first embodiment can select only the signal with higher reliability among the plurality of color change time series signals in an environment where a plurality of cameras are arranged around the user. A color change time series signal can be generated by extracting the color change. As a result, the vital sensing system 100 according to the modification of the first embodiment uses the color change time series signal with high reliability, that is, the pulse measurement precision of the estimated (measured) pulse data, to detect the user H1's biological body. Information (pulse) can be estimated (measured). Furthermore, in measurements that require a color change time series signal for one minute or more, such as stress index, the vital sensing system 100 can detect color change time series signals for one minute or longer even when highly reliable color change time series signals cannot be extracted. By determining the reliability of the color change time series signal from the color change time series signal of several seconds, the biological information (pulse) of the user H1 can be easily measured with high precision.

(Embodiment 2)
The vital sensing system 100 according to the first embodiment has shown an example in which captured images are acquired simultaneously from each of a plurality of cameras that have time synchronization performed. In the vital sensing system 100a according to the second embodiment, a plurality of cameras are each connected to one terminal device so as to be communicable, and an example will be described in which one camera acquires a captured image from each of the plurality of cameras. .

FIG. 8 is a diagram showing an example of the internal configuration of the vital sensing system 100a according to the second embodiment. The internal configuration example of the vital sensing system 100a according to the second embodiment shown in FIG. 8 has almost the same configuration as the internal configuration example of the vital sensing system 100 according to the first embodiment. The same reference numerals are used for the same components as in Embodiment 1, and the description thereof will be omitted. Although the number of terminal devices shown in FIG. 8 is one, it goes without saying that the present invention is not limited to this example. For example, in the vital sensing system 100a according to the second embodiment, each of a plurality of cameras may be communicably connected to one terminal device.

Terminal device P1a in vital sensing system 100a according to the second embodiment is connected to enable wired communication with camera C1, and connected to enable wireless communication with camera C2a and server S1a. The terminal device P1a includes a communication section 20a, a processor 21a, a memory 22a, a monitor 23, and an operation section 24. Note that the terminal device P1a may be communicably connected to three or more cameras.

The communication unit 20a is connected to the camera C1 for wired communication, and is connected for wireless communication to the camera C2a and the server S1a. The communication unit 20a receives a captured video of the user H1 from either the communication unit 40 in the camera C1 or the communication unit 50 in the camera C2a, stores it in the memory 22a, and outputs it to the processor 21. In addition, the communication unit 20a calculates the pulse rate measurement accuracy of the user H1 based on the color change time series signal indicating the change in complexion of the face surface of the user H1 extracted by the processor 21a and the waveform of the color change time series signal. The indicated reliability is transmitted to the communication unit 10 in the server S1a via the network NW.

In addition, in the reliability test calculated based on the color change time series signal executed by the server S1a, if the reliability is less than the reliability threshold set in the server S1a, the communication unit 20a determines whether the reliability information that is less than the threshold is received from the communication unit 10. The communication unit 20a outputs information indicating that the received reliability is less than the threshold to the processor 21a.

The processor 21a detects an area where the face of the user H1 is shown from the captured video received from a preset camera. The processor 21a sets a sensing area for measuring changes in complexion on the face surface of the user H1 among the detected facial areas, and generates a color change time series signal based on changes in the complexion of the user H1 from this sensing area. Extract. The processor 21a calculates reliability indicating the accuracy of pulse measurement of the user H1 based on the waveform of the color change time series signal. The processor 21a transmits the extracted color change time series signal and the calculated reliability to the server S1a via the network NW, and stores the captured image in the memory 22a in association with information about the camera that captured the image. do.

Here, a method for selecting a camera for extracting a color change time-series signal and calculating reliability by the processor 21a in the second embodiment will be described. The processor 21a may preferentially select the camera input from the operation unit 24 and selected (set) by the user H1 as the main camera, or the processor 21a may preferentially select the camera that is input from the operation unit 24 and selected (set) as the main camera by the user H1, or if the direction of the user H1's face reflected in the captured video is the front camera. It may be selected preferentially.

Further, the processor 21a may preferentially select a camera (for example, the camera C1 shown in FIG. 8) that is connected to the terminal device P1a and the plurality of cameras C1 and C2a in a wired communication manner. Thereby, the processor 21a can receive captured video in a raw log (raw data) state, which is data that has not been subjected to compression processing according to the standards of each wireless communication system. Note that if the terminal device P1a has a built-in camera, the processor 21a may select the built-in camera with the highest priority.

Furthermore, based on the information that the reliability is less than the threshold value inputted from the communication unit 20a, the processor 21a controls the camera that captured the imaged video from which the color change time series signal whose reliability is less than the threshold value is extracted. Information is acquired and stored in the memory 22a. The processor 21a selects a camera that acquires a captured image from a current camera (that is, a camera whose reliability has been verified as less than a threshold value), a camera for which reliability calculation and verification have not been performed (that is, an untested camera), and other cameras. switch to the camera. The processor 21a turns on the other switched cameras and starts imaging the user H1. Note that other cameras may be selected by the user H1.

The processor 21a extracts the color change time series signal and calculates the reliability again based on the captured images obtained from other cameras. The processor 21a transmits the extracted color change time series signal and the calculated reliability to the server S1a via the network NW, and also stores the captured image in the memory 22a in association with information of other cameras that have captured the image. to be memorized.

Note that, if the reliability verified by the server S1a for each of all the cameras communicatively connected to the terminal device P1a is less than a threshold value, the processor 21a issues an alert that the installation position of the camera is inappropriate. is generated and output to the monitor 23a. Furthermore, the processor 21a generates a control instruction to reset the information of all cameras that have been verified and whose reliability has been verified to be less than a threshold value, which is stored in the memory 22a. The processor 21a outputs the generated control instruction to the memory 22a to reset the information of the verified camera.

The memory 22a stores information that can identify each of the plurality of cameras C1 and C2a communicably connected to the terminal device P1a, and extracts information from the captured images and captured images received from each of the plurality of cameras C1 and C2a. The color change time series signal and reliability are stored. The memory 22a also stores information about the camera that is input from the operation unit 24 and selected (set) as the main camera.

Furthermore, the memory 22a stores information on verified cameras whose reliability has been calculated and whose reliability has been verified by the server S1a. Note that, when a control instruction to reset the information on the certified camera is input from the processor 21a, the memory 22a resets the information on the certified camera based on this control instruction.

The monitor 23a displays alerts input from the processor 21a. Note that the alert may be output as audio from a speaker (not shown).

The operation unit 24 is, for example, a user interface that detects input operations by the user H1, and is configured using a mouse, a keyboard, a touch panel, or the like. The operation unit 24 accepts selection (setting) and switching of the main camera that images the user H1 based on an input operation by the user H1. The operation unit 24 converts the received input operation from the user H1 into a signal, and outputs the signal to each of the processor 21a and the memory 22a.

The camera C2a is connected to the terminal device P1a so as to be able to communicate wirelessly, and transmits the captured image to the terminal device P1a. The camera C2a includes a communication section 50, a processor 51, a memory 52, and an imaging section 53.

The communication unit 50 is connected to the communication unit 20a of the terminal device P1a so as to be able to wirelessly communicate data. The wireless communication referred to here is, for example, short-range wireless communication such as Bluetooth (registered trademark) or NFC (registered trademark), or communication via a wireless LAN such as Wi-fi (registered trademark). The communication unit 50 transmits the captured video imaged by the imaging unit 53 to the terminal device P1a.

The processor 51 is configured using, for example, a CPU or an FPGA, and performs various processing and control in cooperation with the memory 52. The processor 51 stores the inputted captured video captured by the imaging unit 53 in the memory 52, and transmits it to the communication unit 20a of the terminal device P1a.

The memory 52 includes, for example, a RAM as a work memory used when the processor 51 executes each process, and a ROM that stores programs and data that define the operations of the processor 51. Data or information generated or acquired by the processor 51 is temporarily stored in the RAM. A program that defines the operation of the processor 51 is written in the ROM. The memory 52 stores captured images captured by the imaging unit 53.

The imaging unit 53 includes at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a CCD or CMOS solid-state imaging device, and converts an optical image formed on an imaging surface into an electrical signal. The imaging unit 53 outputs the captured image to the processor 51.

The server S1a according to the second embodiment is connected to the terminal device P1a via the network NW so as to be able to communicate wirelessly. The server S1a includes a communication unit 10a, a processor 11a, and a memory 12a.

The communication unit 10a receives the color change time series signal and reliability from the communication unit 20a in the terminal device P1a. Further, when information indicating that the reliability is less than the threshold is input from the processor 11a, the communication unit 10a transmits this information to the communication unit 20a in the terminal device P1a via the network NW.

The processor 11a tests the reliability of the color change time series signal received from the terminal device P1a and a reliability threshold that is set in advance and stored in the memory 12a. When the received reliability is equal to or higher than the threshold, the processor 11a estimates the pulse of the user H1 from the received color change time series signal, and calculates the stress index of the user H1 based on the estimated pulse data. . The processor 11a also stores the estimated pulse data and stress index in the memory 12a. On the other hand, when the received reliability is less than the threshold, the processor 11a generates information indicating that the reliability is less than the threshold and outputs it to the communication unit 20a.

Here, in the vital sensing system 100a according to the second embodiment, the number of cycles and time of the color change time series signal required to calculate reliability are smaller (shorter) than the number of cycles and time required to estimate pulse rate, etc. good. Thereby, if the calculated reliability is less than the threshold value, the server S1a can immediately switch the camera that images the user H1 from the current camera to another camera.

Furthermore, among the color change time series signals, the time period in which the reliability is less than the threshold is limited, and the pulse, heartbeat, and stress of the user H1 are determined from other color change time series signals whose reliability is higher than the threshold. If the index or the like can be estimated or calculated, the server S1a may estimate or calculate the pulse rate, stress index, etc. of the user H1 from other color change time series signals whose reliability is equal to or higher than the threshold value.

The memory 12a is set in advance and stores the reliability with which the user's pulse can be estimated or the stress index can be calculated. The memory 12a also stores pulse data and stress index estimated by the processor 11a for each user.

Referring to FIG. 9, a pulse estimation procedure of the vital sensing system 100a according to the second embodiment will be described. FIG. 9 is a sequence diagram showing an example of a pulse estimation procedure of the vital sensing system 100a according to the second embodiment. Although FIG. 9 shows an example in which the camera selected (set) by the user H1 is selected as the camera that first images the user H1, it goes without saying that the camera selection method is not limited to this.

The terminal device P1a turns on the camera C1 selected in advance by the user H1 among the plurality of communicably connected cameras C1 and C2a, and starts capturing an image of the user H1 (St10). The terminal device P1a receives and acquires a captured image of the user H1 from the camera C1 (St11).

The terminal device P1a detects the face position of the user H1 from the acquired captured image (St12), and extracts a sensing area from the detected face position in order to measure a change in complexion on the face surface for the purpose of extracting the pulse. Then, a color change time series signal of the face surface of user H1 in the extracted sensing region is extracted (St14). Note that when the position of the user H1's face changes in the captured video, the sensing area may be changed according to the position of the user H1's face.

The terminal device P1a calculates reliability indicating the accuracy of pulse measurement of the user H1 based on the waveform of the extracted color change time series signal (St15). The terminal device P1a transmits the extracted color change time series signal and the calculated reliability to the server S1a via the network NW1 (St16).

The server S1a tests the reliability of the color change time series signal received from the terminal device P1a and the threshold value stored in the memory 12a (St17). In the process of step St17, if the reliability is greater than or equal to the threshold (St17, YES), the server S1a estimates the pulse of the user H1 based on the color change time series signal (St18).

On the other hand, in the process of step St17, if the reliability is less than the threshold (St17, NO), the server S1a generates information indicating that the reliability of the color change time series signal received as the test result is less than the threshold. and transmits it to the terminal device P1a (St19).

Upon receiving the test result, the terminal device P1a outputs the test result to the memory 22a, and also determines the reliability of the captured images captured by each of the cameras communicatively connected to the terminal device P1a. It is determined whether the test has been completed (St20).

If the terminal device P1a has verified all the communicably connected cameras based on the verified camera information stored in the memory 22a (St20, YES), the installation position of the camera is inappropriate. An alert to that effect is generated and output to the monitor 23a (St21). Further, the terminal device P1a resets the information of the verified cameras stored in the memory 22a, and puts all the cameras connected communicably into an untested state (St23). Thereafter, the vital sensing system 100a returns to the process of step St11.

On the other hand, the terminal device P1a, based on the information of the verified cameras stored in the memory 22a, selects the camera that is currently turned on if all the cameras connected for communication have not been verified (St20, NO). Turn it off (St22). Furthermore, the terminal device P1a turns on any one camera that has not been verified (St24). Thereafter, the vital sensing system 100a returns to the process of step St11.

FIG. 10 is a diagram illustrating camera switching in the vital sensing system 100a according to the second embodiment. Note that the color change time series signals shown in FIG. 10 are omitted for clarity of explanation, but the color change time series signals Vt61, Vt62, Vt71, and Vt72 can be used to calculate the stress index. It is a signal that lasts for a period of time (that is, 1 minute) or more, and is actually a discontinuous signal due to the switching time of about 1 second associated with switching between the cameras C1 and C2a.

In the example shown in FIG. 10, the terminal device P1a first turns on the camera C1 at time 0 (zero) and receives the captured image captured by the camera C1. The terminal device P1a stores the color change time series signal Vt61 extracted from the received captured image and the reliability in the memory 22a, and transmits it to the server S1a. Here, the reliability of the calculated color change time series signal Vt61 is lower than the threshold because the waveform shape and cycle are not regular (that is, the amplitude and cycle of the color change time series signal Vt61 have large variations). value.

The server S1a tests the reliability of the received color change time series signal Vt61 and a preset threshold, and generates information indicating that the reliability of the color change time series signal Vt61 is less than the threshold as a test result. and transmits it to the terminal device P1a.

Based on the received test result, the terminal device P1a stores information about the camera C1 in the memory 22a as a certified camera. The terminal device P1a turns off the camera C1, turns on the untested camera C2a that is not stored as a certified camera in the memory 22a at time T1, and starts imaging the user H1 again.

The terminal device P1a receives the captured video captured by the camera C2a. The terminal device P1a stores the color change time series signal Vt71 extracted from the received captured image and the reliability in the memory 22a, and transmits it to the server S1a. Here, the reliability of the calculated color change time series signal Vt71 is lower than the threshold because the waveform shape and cycle are not regular (that is, the amplitude and cycle of the color change time series signal Vt71 have large variations). value.

The server S1a tests the reliability of the received color change time series signal Vt71 and a preset threshold, and generates information indicating that the reliability of the color change time series signal Vt71 is less than the threshold as a test result. and transmits it to the terminal device P1a.

Based on the received test result, the terminal device P1a stores information about the camera C2a in the memory 22a as a certified camera. Here, since all the cameras C1 and C2a communicatively connected to the terminal device P1a have been verified, the terminal device P1a resets the information of the verified cameras stored in the memory 22a. The terminal device P1a turns off the camera C2a, turns on the camera C1 again at time T2, and starts capturing an image of the user H1 again. Furthermore, the terminal device P1a generates an alert indicating that the camera installation position is inappropriate, and outputs it to the monitor 23a.

The terminal device P1a receives the captured video captured by the camera C1. The terminal device P1a stores the color change time series signal Vt62 extracted from the received captured image and the reliability in the memory 22a, and transmits it to the server S1a. Here, the reliability of the calculated color change time series signal Vt62 is lower than the threshold because the waveform shape and cycle are not regular (that is, the amplitude and cycle of the color change time series signal Vt62 have large variations). value.

The server S1a tests the reliability of the received color change time series signal Vt62 and a preset threshold, and generates information indicating that the reliability of the color change time series signal Vt62 is less than the threshold as a test result. and transmits it to the terminal device P1a.

Based on the received test result, the terminal device P1a stores information about the camera C1 in the memory 22a as a certified camera. The terminal device P1a turns off the camera C1, turns on the untested camera C2a that is not stored as a certified camera in the memory 22a at time T3, and starts capturing the image of the user H1 again.

The terminal device P1a receives the captured video captured by the camera C2a. The terminal device P1a stores the color change time series signal Vt72 extracted from the received captured image and the reliability in the memory 22a, and transmits it to the server S1a. Here, the reliability of the calculated color change time series signal Vt72 is higher than the threshold value because the waveform shape and cycle are regular (that is, the variations in the amplitude and cycle of the color change time series signal Vt72 are small). value.

The server S1a tests the reliability of the received color change time series signal Vt72 and a preset threshold value. Since the reliability of the color change time series signal Vt72 is equal to or higher than the threshold as a result of the test, the server S1a continues imaging the user H1 with the camera C2a after time T4 until the calculated reliability becomes less than the threshold. .

As described above, the vital sensing system 100a according to the second embodiment can measure the user's biological information with high precision in an environment where a plurality of cameras are arranged around the user. In addition, the vital sensing system 100a executes imaging of the user H1 using any one of the plurality of cameras, sets and extracts a sensing region, extracts a color change time series signal, and calculates reliability. The processing load on the server S1a and the terminal device P1a can be reduced.

Further, the vital sensing system 100a according to the second embodiment uses the plurality of cameras C1, C2a until the calculated reliability satisfies a condition that is equal to or higher than a predetermined threshold value (that is, a comparison reference value) stored in the memory 12a. The extraction of the color change time series signal based on the imaged video imaged by at least one of the cameras and the calculation of the reliability of the color change time series signal are repeated. Thereby, the vital sensing system 100a according to the second embodiment can measure the biological information (pulse) of the user H1 with high precision in an environment where a plurality of cameras C1 and C2a are arranged around the user H1.

In addition, when the vital sensing system 100a according to the second embodiment does not satisfy the condition that the reliability is equal to or higher than a predetermined threshold value (that is, the comparison reference value), the vital sensing system 100a according to the second embodiment stores information about the camera whose reliability that does not satisfy the condition is calculated. Image capture of the user H1 is performed using at least one camera stored in the memory 12a and not stored (that is, an unverified camera). Thereby, the vital sensing system 100a according to the second embodiment can measure the biological information (pulse) of the user H1 with high precision in an environment where a plurality of cameras C1 and C2a are arranged around the user H1. In addition, the vital sensing system 100a executes imaging of the user H1 using any one of the plurality of cameras, sets and extracts a sensing region, extracts a color change time series signal, and calculates reliability. The processing load on the server S1a and the terminal device P1a can be reduced.

Further, in the vital sensing system 100a according to the second embodiment, when there is no camera that is not stored in the memory 12a (that is, all cameras communicatively connected to the terminal device P1a have been verified). , reset the information of the verified camera stored in the memory 12a. As a result, the vital sensing system 100a according to the second embodiment can efficiently switch each of the plurality of cameras C1 and C2a in an environment where the plurality of cameras C1 and C2a are arranged around the user H1, thereby making the system more reliable. It is possible to obtain a captured image that is calculated with a high degree of accuracy.

Furthermore, the vital sensing system 100a according to the second embodiment tests the reliability of the color change time series signal extracted based on the captured image and a predetermined threshold value stored in the memory 12a. As a result, in the vital sensing system 100a according to the second embodiment, in an environment where a plurality of cameras C1 and C2a are arranged around the user H1, the reliability is set to the threshold (that is, the pulse measurement accuracy of the measured pulse data). The biological information (pulse) of the user H1 can be measured with high precision based on the above color change time series signal (indicating that the measurement accuracy is certain).

Furthermore, the plurality of cameras in the vital sensing system 100a according to the second embodiment capture images of the user H1 for one minute or more. As a result, the vital sensing system 100a according to the second embodiment can image the user H1 for more than the time (1 minute) required to calculate the stress index, and obtain a color change time series signal with reliability equal to or higher than the threshold value. Based on this, biometric information (pulse, stress index, etc.) of the user H1 can be measured with high precision.

Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can come up with various changes, modifications, substitutions, additions, deletions, and equivalents within the scope of the claims, and It is understood that it falls within the technical scope of the present disclosure. Further, each of the constituent elements in the various embodiments described above may be arbitrarily combined without departing from the spirit of the invention.

The present disclosure is useful as a presentation of a vital data output method, a vital data output device, and a vital sensing system that measure a user's biological information with high precision in an environment where a plurality of cameras are arranged around the user.

10, 20, 30, 40, 50 Communication section 11, 21, 31, 41, 51 Processor 12, 22, 32, 42, 52 Memory 23, 33 Monitor 24 Operation section 43, 53 Imaging section C1, C2 Camera H1 User P1 , P2 Terminal device NW Network S1 Server

Claims (12)

A vital data output method for measuring biological information of a user with multiple cameras placed around the user, the method comprising:
obtaining an image of the user captured by at least one camera among the plurality of cameras;
extracting a biosignal including the biometric information of the user based on the captured video;
Calculating the reliability of the extracted biological signal,
Estimating and outputting the biometric information of the user based on the biosignal when the calculated reliability satisfies a condition that is equal to or greater than a predetermined comparison reference value ;
If the reliability does not satisfy the condition that the reliability is equal to or higher than the predetermined comparison reference value, the information of the camera for which the reliability that does not satisfy the condition is calculated is stored, and at least one of the cameras that is not stored is stored. Obtaining images captured by two cameras ,
Vital data output method. Extraction of a biological signal based on a captured image captured by at least one camera among the plurality of cameras and the reliability of the biological signal until the calculated reliability satisfies a condition that the calculated reliability is equal to or higher than the predetermined comparison reference value. Repeat the calculation of
The vital data output method according to claim 1. The time required to calculate the reliability is shorter than the time required to estimate the biological information.
The vital data output method according to claim 1. The biological signal has periodicity,
The number of cycles at which the reliability can be calculated is smaller than the number of cycles at which the biological information can be estimated.
The vital data output method according to claim 1. The time required to calculate the reliability is longer than the time required to estimate the biological information.
The vital data output method according to claim 1. If the calculated reliability does not satisfy a condition that is equal to or greater than the predetermined comparison reference value, the biological information estimated during the calculation of the reliability is not output.
The vital data output method according to claim 5. resetting the information of the stored camera if there is no camera that is not stored;
The vital data output method according to claim 1 . the predetermined comparison reference value is a predetermined threshold;
The vital data output method according to claim 1. The predetermined comparison reference value is the reliability of a biological signal extracted based on captured images captured by one or more other cameras that capture images of the user.
The vital data output method according to claim 1. The plurality of cameras capture images of the user for one minute or more,
The vital data output method according to any one of claims 1 to 9 . A vital data output device that measures a user's biological information and is equipped with a plurality of cameras,
a communication unit that receives a biosignal including the biometric information of the user and a reliability of the biosignal from a captured video captured by at least one of the plurality of cameras;
a processor that tests whether the reliability received by the communication unit is greater than or equal to a predetermined comparison reference value,
The processor estimates and outputs the biological information of the user based on the biological signal when the reliability satisfies a condition that the reliability is equal to or higher than the predetermined comparison reference value, and the processor estimates and outputs the biological information of the user based on the biological signal, and the If the condition of being equal to or greater than the reference value is not satisfied, information about the camera for which the reliability that does not satisfy the condition is calculated is stored, and the captured image is captured by at least one camera among the cameras that are not stored. obtain ,
Vital data output device. A vital sensing system that is communicably connected between multiple cameras and a server,
At least one camera among the plurality of cameras,
Extracting a biosignal including biometric information of the user from a captured video of the user;
Calculating the reliability of the extracted biological signal,
transmitting the biological signal and the reliability to the server;
The server is
estimating and outputting the biometric information of the user based on the biosignal when the received reliability satisfies a condition that is equal to or greater than a predetermined comparison reference value ;
If the reliability does not satisfy the condition that the reliability is equal to or higher than the predetermined comparison reference value, the information of the camera for which the reliability that does not satisfy the condition is calculated is stored, and at least one of the cameras that is not stored is stored. Obtaining images captured by two cameras ,
Vital sensing system.

Images