JP2021153878A - Information processing device and pulse wave calculation method - Google Patents

Abstract

To accurately calculate a pulse wave of a blood vessel from an image of a body surface.SOLUTION: An image acquisition unit 301 acquire a moving image of a body surface. This moving image includes a plurality of frame images. A region-of-interest extraction unit 302 extracts two or more regions of interest including a plurality of pixels from the frame images. A pulse wave calculation unit 303 calculates a plurality of pulse waves corresponding to different positions of the body surface by calculating time series signals indicating changes in the volume of a blood vessel from temporal changes in a pixel value on each pixel in the regions of interest. A degree-of-similarity calculation unit 304 calculates a degree of similarity of at least two pulse waves of the plurality of pulse waves. Specifically, the degree of similarity is calculated on the basis of at least either of a correlation value of the at least two pulse waves of the plurality of pulse waves and a difference value. A pulse wave synthesizing unit 305 synthesizes at least two pulse waves that the pulse wave calculation unit 303 calculates on the basis of the pulse wave similarity. An output control unit 306 causes an output unit to output the synthesized pulse wave.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an information processing device and a pulse wave calculation method.

For example, Patent Document 1 discloses a technique for detecting a pulse wave of a subject by using luminance signals of a plurality of pixels in an image taken by the subject.

Japanese Unexamined Patent Publication No. 2014-073159

In the technique disclosed in Patent Document 1, when pulse waves of different blood vessels are detected in a plurality of pixels, the shape of the pulse waves corresponding to each pixel is different, and the pulse wave cannot be calculated accurately. One aspect of the present disclosure is to provide an information processing device and a pulse wave calculation method that contribute to accurately calculating a pulse wave of a blood vessel from an image obtained by photographing a body surface.

The information processing apparatus according to one aspect of the present disclosure includes a pulse wave calculation unit that calculates a plurality of pulse waves corresponding to different positions on the body surface from a moving image of the body surface, and at least one of the plurality of pulse waves. It includes a similarity calculation unit that calculates the similarity between two pulse waves, and a pulse wave synthesis unit that synthesizes pulse waves based on the similarity.

The pulse wave calculation method according to one aspect of the present disclosure includes a step of calculating a plurality of pulse waves corresponding to different positions on the body surface from a moving image of the body surface, and at least two of the plurality of pulse waves. It includes a step of calculating the similarity of pulse waves and a step of synthesizing pulse waves based on the similarity.

It is a figure which shows an example of the relationship between the information processing apparatus 100 and the living body A. It is a block diagram which shows an example of the electric structure of an information processing apparatus. It is a block diagram which shows an example of the functional structure of the control part of 1st Embodiment. It is a figure which shows an example of the area of interest in a frame image. It is a figure which shows an example of the pixel corresponding to a synthetic candidate pulse wave. It is a flowchart which shows an example of the process executed by the information processing apparatus of 1st Embodiment. It is a figure which shows an example of the relationship between a pixel corresponding to a pulse wave of interest, and a blood vessel. It is a figure which shows an example of a pulse wave. It is a figure which shows an example of a pulse wave. It is a block diagram which shows an example of the functional structure of the control part of the 2nd Embodiment. It is a block diagram which shows an example of the functional structure of the control part of 3rd Embodiment. It is a figure which shows an example of a region of interest. It is a block diagram which shows an example of the functional structure of the control part of 4th Embodiment. It is a figure which shows an example of a pixel corresponding to a pulse wave and a peripheral pixel. It is a block diagram which shows an example of the functional structure of the control part of 5th Embodiment. It is a figure which shows an example of the pulse wave feature amount. It is a flowchart which shows an example of the processing flow executed by the information processing apparatus of 5th Embodiment. It is a block diagram which shows an example of the functional structure of the control part of 6th Embodiment. It is a flowchart which shows an example of the processing flow executed by the information processing apparatus of 6th Embodiment. It is a figure which shows an example of the relationship between the pixel corresponding to the 1st pulse wave and the 2nd pulse wave, and a blood vessel.

(First Embodiment)
The first embodiment will be described. In the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicate description will be omitted.

FIG. 1 is a diagram showing an example of the relationship between the information processing device 100 and the living body A. The information processing device 100 photographs the body surface of the living body A by the imaging unit 101 and calculates the pulse wave of the living body A. A pulse wave is a signal indicating a change in the volume of a blood vessel. Specifically, the color of the body surface to be photographed changes as the volume of blood vessels changes with the beating of the heart. Therefore, the information processing device 100 calculates the pulse wave based on the change in the color of the body surface photographed by the imaging unit 101. For example, the information processing device 100 is a PC (Personal Computer), a smartphone, a tablet, a notebook computer, a dedicated terminal for pulse wave measurement, or the like.

The imaging unit 101 photographs the body surface of the living body A and acquires a moving image of the body surface. Specifically, the imaging unit 101 photographs the body surface of the living body A at a predetermined frame rate (for example, 10 to 300 fps (frames per second)) for a predetermined time (for example, 10 seconds to 60 seconds). The imaging unit 101 stores the acquired moving image of the body surface in the storage unit 201 (see FIG. 2). In the following description, for convenience of explanation, the moving image of the body surface is simply referred to as "moving image". The body surface includes the body surface of the face of the living body A. The imaging unit 101 only needs to be able to acquire a moving image of the body surface, and the arrangement position and type of the imaging unit 101 are not limited. For example, the imaging unit 101 may be an RGB camera or an infrared camera. In the following description, for convenience of explanation, a human being will be illustrated as the living body A, but the living body A may be a living body other than a human being.

FIG. 2 is a block diagram showing an example of the electrical configuration of the information processing apparatus 100. The information processing device 100 includes an imaging unit 101, a storage unit 201, a control unit 202, an output unit 203, and the like.

The storage unit 201 is a storage device capable of storing arbitrary information, and is composed of, for example, a hard disk, an SSD (Solid State Drive), a semiconductor memory, or the like. The storage unit 201 stores a moving image of the body surface acquired by the imaging unit 101. Further, the storage unit 201 stores, for example, a program that causes the control unit 202 to function, data required for the control unit 202 to function, and the like.

The control unit 202 executes various processes according to the programs and data stored in the storage unit 201. The control unit 202 is realized by, for example, a processor such as a CPU (Central Processing Unit).

The output unit 203 outputs the pulse wave of the living body A. For example, the output unit 203 is a display device.

FIG. 3 is a block diagram showing an example of the functional configuration of the control unit 202. The control unit 202 includes an image acquisition unit 301, an area of interest extraction unit 302, a pulse wave calculation unit 303, a similarity calculation unit 304, a pulse wave synthesis unit 305, an output control unit 306, and the like.

The image acquisition unit 301 acquires a moving image from the storage unit 201. The moving image stored in the storage unit 201 includes a plurality of frame images. The frame image is an image of each frame of the moving image.

The region of interest extraction unit 302 extracts one or more regions of interest from the frame image included in the acquired moving image. The region of interest is a part of the body surface region of the frame image and is a region including a plurality of pixels. For example, when the moving image includes the face region of the living body A, the regions of interest are the cheek portion and the forehead portion.

The pulse wave calculation unit 303 calculates a plurality of pulse waves corresponding to different positions on the body surface from the moving image. In the present specification, the pulse wave is a time series indicating a change in blood vessel volume calculated from a time series signal indicating a pixel value (for example, a brightness value) of each frame image with respect to the same position on the body surface. It is a signal. In the present specification, the pixel value is information indicating the brightness of a pixel in a moving image, and is, for example, the pixel value of each pixel for each of R (Red), G (Green), and B (Blue) or each pixel. Is the brightness value of. In the present embodiment, the same position on the body surface is defined as the same pixel in the frame image as long as the positional relationship between the imaging unit 101 and the living body A is constant. Further, in the present embodiment, the different positions on the body surface are assumed to be each pixel in the region of interest. Therefore, in the present embodiment, the pulse wave calculation unit 303 calculates a time-series signal indicating a change in blood vessel volume from a change in pixel value over time for each pixel in the region of interest.

The similarity calculation unit 304 calculates the similarity of at least two pulse waves among the plurality of pulse waves. Specifically, the similarity calculation unit 304 calculates the similarity based on at least one of the correlation value and the difference value of at least two pulse waves among the plurality of pulse waves. In the following description, for convenience of explanation, the similarity is referred to as "pulse wave similarity". Further, in the following description, for convenience of explanation, at least two pulse waves related to one pulse wave similarity are referred to as "attention pulse waves".

The pulse wave synthesis unit 305 synthesizes a pulse wave based on the pulse wave similarity. For example, the pulse wave synthesis unit 305 synthesizes from at least two pulse waves calculated by the pulse wave calculation unit 303 based on the pulse wave similarity. In the following description, for convenience of explanation, at least two pulse waves to be synthesized are referred to as "synthesis candidate pulse waves". Further, in the following description, for convenience of explanation, the pulse wave synthesized by the pulse wave synthesizing unit 305 is referred to as a “synthetic pulse wave”.

The output control unit 306 causes the output unit 203 to output a synthetic pulse wave. Alternatively, the output control unit 306 may transmit the synthetic pulse wave to a device different from the information processing device 100 via the communication means.

Next, the pulse wave corresponding to the pixel in the moving image of the body surface will be described in detail. FIG. 4 shows an example of the region of interest 402 in one frame image 401. When the image acquisition unit 301 extracts the region of interest 402 of the pixels of vertical W × horizontal H from the frame image 401, the pulse wave calculation unit 303 calculates the pulse wave corresponding to each pixel in the region of interest 402. That is, when the image acquisition unit 301 extracts the region of interest 402 of vertical W × horizontal H pixels, the pulse wave calculation unit 303 calculates W × H pulse waves. For example, the region of interest 402 illustrated in FIG. 4 includes 10 vertical × 10 horizontal pixels. Therefore, the pulse wave calculation unit 303 calculates 100 pulse waves corresponding to each pixel in the region of interest 402.

The similarity calculation unit 304 calculates at least two pulse wave similarity among the pulse waves corresponding to each pixel in the region of interest 402. For example, the similarity calculation unit 304 selects the pulse wave corresponding to the pixel 403a in the region of interest 402 and the pulse wave corresponding to the pixel 403b in the region of interest 402 as the pulse wave of interest. Then, the similarity calculation unit 304 calculates the pulse wave similarity between the pulse wave corresponding to the pixel 403a and the pulse wave corresponding to the pixel 403b. In this way, the similarity calculation unit 304 calculates the pulse wave similarity with respect to the pulse waves corresponding to all the pixels in the region of interest 402.

FIG. 5 is a diagram showing an example of pixels corresponding to synthetic candidate pulse waves. When the pulse wave similarity of the pulse wave corresponding to each pixel of the plurality of pixels 501 satisfies the above-mentioned predetermined condition, the pulse wave synthesis unit 305 adds the pulse wave corresponding to the plurality of pixels 501 to the composite pulse wave candidate list. do. Then, the pulse wave synthesis unit 305 synthesizes the pulse waves included in the synthetic pulse wave candidate list (that is, the pulse waves corresponding to the plurality of pixels 501). As a result, the pulse wave synthesis unit 305 synthesizes pulse waves corresponding to a plurality of positions satisfying the above-mentioned predetermined conditions, and calculates the combined pulse wave.

Next, an example of the processing flow executed by the information processing apparatus 100 of the present embodiment will be described. FIG. 6 is a flowchart showing an example of processing executed by the information processing apparatus 100 of the present embodiment. In this example, when the number of frames of the moving image stored in the storage unit 201 exceeds a predetermined number of frames after the imaging unit 101 starts shooting, the control unit 202 starts the process of S601 shown in FIG. do. Further, it is assumed that the synthetic candidate pulse wave has not been determined at the time when the processing of S601 is started.

In S601, the image acquisition unit 301 acquires a moving image of the body surface from the storage unit 201. For example, the image acquisition unit 301 acquires a moving image from the storage unit 201 after a predetermined first time (for example, 30 seconds) has elapsed from the start of shooting from the imaging unit 101. The first time is the photographing time required for the information processing apparatus 100 to calculate the pulse wave from the moving image. Alternatively, the image acquisition unit 301 may acquire a moving image from the storage unit 201 every second time (for example, 1 second) shorter than the first time.

In S602, the region of interest extraction unit 302 extracts the region of interest 402 from each frame image 401 included in the moving image acquired in S601. Then, the region of interest extraction unit 302 outputs the extracted region of interest 402 to the pulse wave calculation unit 303. Alternatively, the region of interest extraction unit 302 may associate the extracted region of interest 402 with the name of the face portion (cheek, forehead, etc.) and output it to the pulse wave calculation unit 303.

Further, the region of interest extraction unit 302 may determine whether or not the frame image 401 includes a face region. For example, the region of interest extraction unit 302 authenticates the face of the living body A from the acquired moving image. Here, the region of interest extraction unit 302 may authenticate the face of the living body A from the moving image by using pattern matching, SVM (Support Vector Machine), a neural network, or the like, and the details of the authentication method do not matter. For example, the region of interest extraction unit 302 extracts the region of interest 402 from a plurality of frame images 401 including the face region continuously in chronological order among the frame images 401 including the face region.

In S603, the pulse wave calculation unit 303 calculates a plurality of pulse waves from the region of interest 402 extracted in S602. Specifically, the pulse wave calculation unit 303 extracts a pixel value from each pixel in the region of interest 402. Then, when the number of frames of the moving image acquired in S601 is equal to or greater than the number of frames corresponding to the predetermined first time, the pulse wave calculation unit 303 describes the pixels in the region of interest 402 over the predetermined first time. Each time, a signal indicating a change in pixel value over time is extracted. The pulse wave calculation unit 303 processes the extracted signal by using a predetermined signal processing method, and calculates the processed result as a pulse wave. For example, the signal processing method may be an independent component analysis, a principal component analysis, a dye component separation, a bandpass filter, a lowpass filter, or the like. Further, the pulse wave calculation unit 303 may normalize the calculated pulse wave. For example, the pulse wave calculation unit 303 normalizes the pulse wave so that the maximum value of the pulse wave is 1 and the minimum value of the pulse wave is -1.

The pulse wave calculation unit 303 associates the calculated pulse wave with the pulse wave identification information and outputs it to the similarity calculation unit 304. The pulse wave identification information is information for identifying a pulse wave, for example, information for identifying a position corresponding to the pulse wave (for example, the coordinate value of a pixel corresponding to the pulse wave in the frame image 401). Alternatively, the pulse wave identification information may be a number different from the coordinate values in the frame image 401 assigned for each position corresponding to the pulse wave.

In S604, the similarity calculation unit 304 selects at least two attention pulse waves from the plurality of pulse waves calculated in S603. In other words, the similarity calculation unit 304 selects at least two attention pulse waves corresponding to different pixels among the plurality of pixels in the region of interest 402.

In S605, the similarity calculation unit 304 calculates the pulse wave similarity of the pulse wave of interest. Specifically, the similarity calculation unit 304 calculates the pulse wave similarity for all combinations of pulse waves included in the pulse wave of interest. In this case, the similarity calculation unit _304, N _{C 2} pieces (= (N × (N- 1) / 2) number) to calculate the similarity of. Here, N is the number of pulse waves included in the pulse wave of interest, which is a natural number. Alternatively, the similarity calculation unit 304 may determine a reference pulse wave from at least two pulse waves included in the pulse wave of interest. Then, the similarity calculation unit 304 may calculate the pulse wave similarity between the determined reference pulse wave and the pulse wave other than the reference pulse wave included in the attention pulse wave. In this case, the similarity calculation unit 304 calculates N-1 similarity.

For example, the similarity calculation unit 304 calculates the correlation coefficient r1 calculated using the following equation (1) as the pulse wave similarity. In the following description, the pulse wave value calculated from the pixel value (for example, the luminance value) of the position p on the body surface at the time point i is referred to as x (p, i). Further, in the following description, the pulse wave signal at time t at the position p on the body surface is referred to as X (p, t). That is, X (p, t) is a vector of x (p, i) (i is an integer), and X (p, t) = {x (p, 0), x (p, 1), ..., x (p, t), ...}. Further, X (p1, t) and X (p2, t) shown in the equation (1) are pulse wave signals of time t corresponding to the position p1 on the body surface and the position p2 on the body surface, respectively. be. For example, x (p1, i) and x (p2, i) indicate pixel values at time point i that correspond to different positions p1 and p2, respectively. For example, time point i is the frame number of the moving image.

ここで、Ｔは動画像の時間長、つまり動画像に含まれるフレーム画像の枚数である。 For example, the expected value E (X (p1, t)) and the expected value E (X (p2, t)) shown in the equation (1) are determined by using the following equations (2) and (3). ..

Here, T is the time length of the moving image, that is, the number of frame images included in the moving image.

Alternatively, the similarity calculation unit 304 may calculate the pulse wave similarity using the correlation function R (τ) calculated using the following equation (4).

The similarity calculation unit 304 may determine τ when the correlation function R (τ) shown in the equation (4) is maximized as the time difference τ _max. Then, the similarity calculation unit 304 may calculate the value R (τ) of the cross-correlation function at the _{time difference τ max as the pulse wave similarity.}

Alternatively, the similarity calculation unit 304 may calculate the pulse wave similarity based on the difference value between the pulse waves of interest. For example, the similarity calculation unit 304 uses the following equation (5) to calculate the reciprocal r2 of the average absolute deviation between the pulse waves of interest corresponding to the position p1 on the body surface and the position p2 on the body surface. It may be calculated as.

The above equations (1), (4), and (5) are examples of equations for calculating the pulse wave similarity. The pulse wave similarity may be an index indicating the similarity between pulse waves corresponding to different positions on the body surface, and is not limited to the correlation value, the cross-correlation value, and the difference value calculated by the above formula.

In S606, the similarity calculation unit 304 stores the pulse wave similarity calculated in S605 and the pulse wave of interest corresponding to the pulse wave similarity in the storage unit 201 in association with each other.

In S607, the similarity calculation unit 304 determines whether or not the pulse wave similarity has been calculated for all the pulse waves. If the pulse wave similarity has not been calculated for all pulse waves (S607: No), at least two new notable pulse waves are selected from the plurality of pulse waves (S608). Then, the control unit 202 continues the process by returning the process to S605. On the other hand, when the pulse wave similarity is calculated for all the pulse waves (S607: Yes), the control unit 202 shifts the process to S609.

In S609, the pulse wave synthesis unit 305 adds a pulse wave having a pulse wave similarity satisfying a predetermined condition to the synthesis candidate pulse wave list. For example, when the pulse wave similarity is equal to or higher than a predetermined threshold value (for example, threshold value = 0.9), the pulse wave synthesizing unit 305 determines that the pulse wave of interest corresponding to the pulse wave similarity satisfies the above-mentioned predetermined condition. judge. Alternatively, in the pulse wave synthesis unit 305, a predetermined number (for example, 10) of attention pulse waves satisfy the above-mentioned predetermined conditions in descending order of the pulse wave similarity among the pulse wave similarity stored in the storage unit 201. May be determined.

In S610, the pulse wave synthesizing unit 305 synthesizes the pulse waves included in the synthesis candidate pulse wave list determined in S609. For example, the pulse wave synthesis unit 305 calculates the average value of the pixel values (for example, the brightness value) at each time point in the synthesis candidate pulse wave determined in S609. The pulse wave synthesizing unit 305 may calculate the average value of the pixel values at each time point by arithmetic mean, geometric mean, or the like. Then, the pulse wave synthesis unit 305 calculates the composite pulse wave based on the time-series signal of the calculated average value.

Alternatively, the pulse wave synthesis unit 305 may weight the pixel value of the synthesis candidate pulse wave according to the pulse wave similarity corresponding to the synthesis candidate pulse wave. In that case, the pulse wave synthesis unit 305 synthesizes the weighted synthetic candidate pulse waves to calculate the synthetic pulse wave.

Alternatively, the pulse wave synthesizing unit 305 may perform a process of removing the phase difference between the synthesis candidate pulse waves and calculate the mode or median value of the synthesis candidate pulse wave. Then, the pulse wave synthesis unit 305 calculates the calculated mode or median time-series signal as a composite pulse wave.

Alternatively, the pulse wave synthesis unit 305 may perform a process of removing the phase difference between the synthesis candidate pulse waves and perform multivariate analysis (principal component analysis, independent component analysis, etc.) on the synthesis candidate pulse wave. Then, the pulse wave synthesis unit 305 may calculate the component obtained by the multivariate analysis as a synthetic pulse wave.

Alternatively, when the similarity calculation unit 304 calculates _{the value R (τ) of the cross-correlation function in the time difference τ max} using the equation (4) as the pulse wave similarity, the pulse wave synthesis unit 305 calculates the time difference τ _max. The pixel value of the synthetic candidate pulse wave may be weighted according to the above. In that case, the pulse wave synthesis unit 305 synthesizes a weighted synthesis candidate pulse wave.

In S611, the output control unit 306 causes the output unit 203 to output the synthetic pulse wave synthesized in S610. Alternatively, the output control unit 306 may transmit the synthetic pulse wave to a device different from the information processing device 100 via the communication means.

When the pulse wave calculation unit 303 calculates the pulse wave corresponding to one pixel, the calculated pulse wave may include noise due to the influence of lighting or the like at the time of shooting. Therefore, by synthesizing the pulse waves corresponding to a plurality of pixels, the noise contained in the pulse waves is smoothed. However, since the pulse waves of different blood vessels have different changes in the volume of the blood vessels, it is not possible to accurately calculate the pulse waves when the pulse waves of different blood vessels are combined. On the other hand, for at least two pulse waves corresponding to different positions, the pulse wave similarity corresponding to the pulse wave of the same blood vessel is higher than the pulse wave similarity corresponding to the pulse wave of different blood vessels. Therefore, the information processing apparatus 100 can improve the calculated signal noise ratio of the pulse wave by synthesizing at least two pulse waves having a relatively high pulse wave similarity by the pulse wave synthesis unit 305.

FIG. 7 shows an example of the relationship between the pixel corresponding to the pulse wave of interest and the blood vessel. For example, it is assumed that the pulse wave calculation unit 303 calculates the pulse waves corresponding to the pixels 701a and the pixels 701b in the region of interest 701, respectively. The two pulse waves corresponding to the pixels 701a and 701b are the pulse waves corresponding to the blood vessel 711. Then, it is assumed that the pulse wave similarity of the two attention pulse waves corresponding to the pixels 701a and the pixel 701b exceeds a predetermined threshold value.

Similarly, it is assumed that the pulse wave calculation unit 303 calculates the pulse waves corresponding to the pixels 702a and the pixels 702b in the region of interest 702, respectively. Here, the two pulse waves corresponding to the 702a and the pixel 702b are the pulse waves corresponding to the blood vessel 711. Then, it is assumed that the pulse wave similarity of the two attention pulse waves corresponding to the pixels 702a and the pixel 702b exceeds a predetermined threshold value. In that case, the pulse wave synthesis unit 305 synthesizes the pulse waves corresponding to the pixels 701a, the pixel 701b, the pixel 702a, and the pixel 702b, and calculates the combined pulse wave.

On the other hand, it is assumed that the pulse wave calculation unit 303 calculates the pulse waves corresponding to the pixels 703a and the pixels 703b in the region of interest 703, respectively. Here, the two pulse waves corresponding to the pixels 703a and the pixel 703b are the pulse waves corresponding to the blood vessel 712. However, for example, the two pulse waves corresponding to pixel 701a and pixel 703a are pulse waves of different blood vessels. Further, it is assumed that the two pulse waves corresponding to the pixels 701a and the pixels 703a are equal to or less than a predetermined threshold value. In that case, the synthetic candidate pulse wave does not include the combination of the pulse wave corresponding to the pixel 701a and the pixel 703a.

Further, it is assumed that the pulse wave calculation unit 303 calculates the pulse waves corresponding to the pixels 704a and the pixels 704b in the region of interest 704, respectively. Here, the two pulse waves corresponding to the pixels 704a and the pixel 704b are the pulse waves corresponding to the blood vessel 712. Then, it is assumed that the pulse wave similarity of the two attention pulse waves corresponding to the pixels 704a and the pixel 704b exceeds a predetermined threshold value. In that case, the pulse wave synthesis unit 305 synthesizes the pulse waves corresponding to the pixels 703a, the pixel 703b, the pixel 704a, and the pixel 704b, and calculates the composite pulse wave.

8A and 8B are diagrams showing an example of a pulse wave. The pulse wave X (701a, t) shown in FIG. 8A indicates a pulse wave corresponding to the pixel 701a shown in FIG. 7. Further, the pulse wave X (702a, t) shown in FIG. 8A indicates the pulse wave X corresponding to the pixel 702a shown in FIG. 7. That is, the pulse wave X (701a, t) and the pulse wave X (702a, t) are pulse waves corresponding to the blood vessel 711 illustrated in FIG. 7.

Here, it is assumed that the pulse wave similarity between the pulse wave X (701a, t) and the pulse wave X (702a, t) satisfies a predetermined condition. Further, as shown in FIG. 8A, it is assumed that the pulse wave X (702a, t) has a delay of a time difference τ1 from the pulse wave X (701a, t). In that case, the pulse wave synthesizing unit 305 synthesizes the pulse wave X (702a, t) with the time difference τ1 and the pulse wave X2 shifted in the time axis direction, and the pulse wave X (701a, t) to combine the pulse wave X (701a, t). calculate. Here, the pulse wave X2 (702a, t) = {x (702a, τ1), x (702a, τ1 + 1), ..., X (702a, τ1 + t), ...}. The time difference τ1 is, for example, the above-mentioned time difference τ _max . That is, the pulse wave synthesis unit 305 performs a process of removing the phase difference between the synthesis candidate pulse waves and synthesizes the pulse waves.

On the other hand, the pulse wave X (703a, t) shown in FIG. 8B shows the pulse wave corresponding to the pixel 703a shown in FIG. 7. Further, the pulse wave X (704a, t) shown in FIG. 8B indicates a pulse wave corresponding to the pixel 704a shown in FIG. 7. That is, the pulse wave X (703a, t) and the pulse wave X (704a, t) are pulse waves corresponding to the blood vessel 712 illustrated in FIG. 7.

Similarly, it is assumed that the pulse wave similarity between the pulse wave X (703a, t) and the pulse wave X (704a, t) satisfies a predetermined condition. Further, as shown in FIG. 8B, it is assumed that the pulse wave X (704a, t) has a delay of a time difference τ2 from the pulse wave X (703a, t). In that case, the pulse wave synthesizing unit 305 synthesizes the pulse wave X3, which is the pulse wave X (704a, t) shifted in the time axis direction by a time difference τ2, and the pulse wave X (703a, t), and synthesizes the pulse wave. Is calculated. Here, the pulse wave X3 (704a, t) = {x (704a, τ2), x (704a, τ2 + 1), ..., X (704a, τ2 + t), ...}. The time difference τ2 is, for example, the above-mentioned time difference τ _max .

As described above, the pulse wave synthesizing unit 305 synthesizes the pulse waves of the same blood vessel to calculate the synthesized pulse wave. As a result, the information processing apparatus 100 can calculate the pulse wave more accurately than when the pulse wave corresponding to a single pixel is output.

(Modification 1) The image acquisition unit 301 may determine the entire frame image 401 as the region of interest 402. In other words, the pulse wave calculation unit 303 may calculate the pulse wave for each pixel in the frame image 401. That is, the pulse wave synthesis unit 305 synthesizes the pulse wave from the moving image. Specifically, the pulse wave synthesis unit 305 synthesizes pulse waves based on the pulse wave similarity calculated from each pixel of the frame image 401.

(Modification 2) An apparatus different from the information processing apparatus 100 may include an imaging unit 101. In that case, the information processing device 100 acquires a moving image generated by the imaging unit 101 from a device different from the information processing device 100 via a network or the like.

(Modification 3) The information processing device 100 may have a function of tracking the living body A. In that case, the information processing device 100 can track the living body A and acquire a moving image of the body surface of the same living body A. As a result, the information processing apparatus 100 can calculate the pulse wave from the same position on the body surface even when the living body A is moving. For example, the imaging unit 101 may move in accordance with the movement of the living body A, and each pixel of the imaging unit 101 may continue to photograph the same position on the body surface of the living body A. Further, for example, the same position of the living body A may be tracked between image frames by image processing. At this time, the same position on the body surface is not necessarily the same pixel in the frame image.

(Second Embodiment)
The second embodiment will be described. In each of the following embodiments, the configurations and processes having substantially the same functions as those of the first embodiment will be referred to by a common reference numeral, and the description thereof will be omitted, and the points different from those of the first embodiment will be described.

FIG. 9 is a block diagram showing an example of the functional configuration of the control unit 202 of the present embodiment. The difference between the control unit 202 illustrated in FIG. 9 and the control unit 202 illustrated in FIG. 3 is that the biological information extraction unit 901 is further provided.

The biological information extraction unit 901 extracts biological information based on the synthetic pulse wave. Biometric information includes at least one of pulse rate, stress level, and blood pressure. In the present embodiment, calculating and estimating biometric information is referred to as "extraction of biometric information".

For example, the biological information extraction unit 901 counts the number of peaks of the synthetic pulse wave and calculates the counted peak number as the pulse rate.

Further, for example, the biological information extraction unit 901 performs a fast Fourier transform on a predetermined time interval in the synthetic pulse wave to calculate a power spectrum. Then, the biological information extraction unit 901 analyzes the power spectrum and calculates the stress level.

Further, for example, the biological information extraction unit 901 performs the synthetic pulse wave twice with respect to time, calculates the acceleration pulse wave, and estimates the blood pressure from the acceleration pulse wave. Further, for example, the biological information extraction unit 901 estimates the blood pressure from the shape of the synthetic pulse wave.

As described above, the synthetic pulse wave has a higher signal-to-noise ratio than the pulse wave corresponding to a single position (for example, a single pixel) on the body surface. Therefore, the biological information extraction unit 901 can extract biological information with higher reliability than the case of extracting biological information based on a pulse wave corresponding to a single position on the body surface.

(Third Embodiment)
The third embodiment will be described. FIG. 10 is a block diagram showing an example of the functional configuration of the control unit 202 of the present embodiment. The difference between the control unit 202 illustrated in FIG. 10 and the control unit 202 illustrated in FIG. 3 is that the region determination unit 1001 is further provided. Further, the control unit 202 of the present embodiment includes a pulse wave calculation unit 1002 instead of the pulse wave calculation unit 303.

The area determination unit 1001 determines a plurality of areas including a plurality of pixels from the frame image included in the moving image acquired by the image acquisition unit 301.

The pulse wave calculation unit 1002 calculates one pulse wave from each region. Specifically, the pulse wave calculation unit 1002 calculates each pulse wave among the plurality of pulse waves based on the statistical value (for example, total value or average value) of the pixel values of the plurality of regions. For example, the pulse wave calculation unit 1002 calculates the statistical value of the pixel value of the region for each frame image, and calculates the signal indicating the temporal change of the calculated statistical value. Then, the pulse wave calculation unit 1002 processes the calculated signal by using a predetermined signal processing method, and calculates the processed result as a pulse wave.

The similarity calculation unit 304 calculates the similarity of at least two pulse waves among the pulse waves corresponding to the plurality of regions calculated by the pulse wave calculation unit 1002. For example, the pulse wave similarity is calculated for the combination of pulse waves in all regions. In this case, the similarity calculation unit _304, N _{C 2} pieces (= (N × (N- 1) / 2) number) to calculate the similarity of. Here, N is the number of regions and is a natural number. Alternatively, the similarity calculation unit 304 may determine a reference pulse wave among the pulse waves in at least two regions. Then, the similarity calculation unit 304 may calculate the pulse wave similarity between the determined reference pulse wave and the pulse wave other than the reference pulse wave. In this case, the similarity calculation unit 304 calculates N-1 similarity.

The pulse wave synthesis unit 305 synthesizes a synthetic pulse wave based on the pulse wave similarity calculated by the similarity calculation unit 304. For example, the pulse wave synthesis unit 305 synthesizes pulse waves in at least two regions set by the region determination unit 1001 based on the pulse wave similarity. In the following description, for convenience of explanation, at least two regions to be synthesized are referred to as "composite candidate regions".

Next, the pixels corresponding to the pulse wave in the moving image of the body surface will be described in detail. FIG. 11 is a diagram showing an example of a region of interest. The region determination unit 1001 determines the region from the region of interest 402. The region 1101, the region 1102, the region 1103, and the region 1104 illustrated in FIG. 11 are examples of regions.

Here, it is assumed that the storage unit 201 stores the pulse wave corresponding to each region in the region of interest 402 and the pulse wave similarity corresponding to the pulse wave in association with each other.

For example, the pulse wave synthesis unit 305 determines a synthesis candidate region including a plurality of regions within the region of interest 402. The pulse wave synthesis unit 305 determines a plurality of synthesis candidate regions from the regions 1101, the region 1102, the region 1103, and the region 1104 based on the pulse wave similarity corresponding to each region. For example, it is assumed that the pulse wave synthesis unit 305 determines the region 1101 and the region 1102 as synthesis candidate regions. In that case, the pulse wave synthesizing unit 305 calculates the synthesized pulse wave as one pulse wave based on the signal indicating the temporal change of the statistical value of the pixel value of the pixels included in the area 1101 and the area 1102.

As a result, the pulse wave synthesizing unit 305 can synthesize a pulse wave of interest having a relatively high pulse wave similarity to calculate the synthesized pulse wave. Therefore, for the same reason as described above, the calculated signal-to-noise ratio of the synthetic pulse wave is improved. Further, for the same reason as described above, the output unit 203 more accurately outputs the pulse wave of one blood vessel than the case of outputting the pulse wave corresponding to a single position (for example, a single pixel) on the body surface. Can be output to.

(Fourth Embodiment)
The fourth embodiment will be described. FIG. 12 is a block diagram showing an example of the functional configuration of the control unit 202 of the present embodiment. The control unit 202 of the present embodiment includes a pulse wave calculation unit 1201 instead of the pulse wave calculation unit 303.

The pulse wave calculation unit 1201 has a plurality of statistical values (for example, total value or average value) of the pixel value of one pixel included in the moving image and the pixel value of at least one pixel adjacent to one pixel. Of the pulse waves, one pulse wave corresponding to the one pixel is calculated. In the following description, for convenience of explanation, at least one pixel adjacent to one pixel corresponding to the pulse wave is referred to as a "peripheral pixel".

FIG. 13 is a diagram showing an example of a pixel corresponding to a pulse wave and a peripheral pixel. For example, the pulse wave calculation unit 1201 calculates statistical values of the pixel values of the pixel 1301 and the pixel values of the eight peripheral pixels 1302 adjacent to the pixel 1301 with respect to the pixel 1301 in the region of interest 402. The pulse wave calculation unit 1201 calculates the pulse wave corresponding to the pixel 1301 based on the signal indicating the temporal change of the statistical value of the pixel 1301.

As illustrated in FIG. 13, it is assumed that the peripheral pixels 1302 of the pixel 1301 are eight pixels surrounding the pixel 1301. Then, when the image acquisition unit 301 extracts the region of interest 402 of vertical W × horizontal H pixels, the pulse wave calculation unit 303 calculates (W-2) × (H-2) pulse waves. For example, the region of interest 402 illustrated in FIG. 13 includes 10 vertical × 10 horizontal pixels. In that case, the pulse wave calculation unit 303 corresponds to each pixel in the region 1303 shown by the broken line in FIG. 13 based on the statistical values of each pixel and the peripheral pixels, 64 (= (10-). 2) × (10-2)) pulse waves are calculated. The peripheral pixels 1302 illustrated in FIG. 13 are eight pixels surrounding the pixel 1301, but the peripheral pixels 1302 may be less than eight pixels. Further, the number of peripheral pixels 1302 may be more than eight pixels. For example, the peripheral pixel 1302 may include pixels further adjacent to the pixels surrounding the pixel 1301.

As a result, the pulse wave calculation unit 1201 can smooth the noise included in the calculated pulse wave as compared with the case where the pulse wave is calculated from the pixel value of one pixel. As a result, the pulse wave calculation unit 1201 can improve the signal noise ratio of the calculated pulse wave as compared with the case of calculating the pulse wave from the pixel value of one pixel, and can calculate the pulse wave more accurately.

(Fifth Embodiment)
The fifth embodiment will be described. FIG. 14 is a block diagram showing an example of the functional configuration of the control unit 202 of the present embodiment. The difference between the control unit 202 illustrated in FIG. 14 and the control unit 202 illustrated in FIG. 3 is that the feature amount extraction unit 1401 is further provided. Further, the control unit 202 of the present embodiment includes a similarity calculation unit 1402 and a pulse wave synthesis unit 1403 in place of the similarity calculation unit 304 and the pulse wave synthesis unit 305.

The feature amount extraction unit 1401 extracts the feature amount indicating the feature of the shape of the pulse wave. In the following description, for convenience of explanation, the feature amount will be referred to as a "pulse wave feature amount". The pulse wave feature amount is a value related to a feature point in the pulse wave, and is, for example, at least one of a period, an amplitude, a frequency, a rising angle, and a spectrum. In the following description, for convenience of explanation, the feature point will be referred to as a "pulse wave feature point". The pulse wave characteristic points are the maximum point, the minimum point, the inflection point, and the like of the pulse wave. Alternatively, the pulse wave feature point may be a signal obtained by differentiating the pulse wave into the first or multiple orders, or a maximum point, a minimum point, an inflection point, or the like of a spectrum obtained by frequency analysis.

The similarity calculation unit 1402 calculates the pulse wave similarity based on the comparison result of at least two pulse wave features.

The pulse wave synthesis unit 1403 synthesizes at least two pulse waves based on the pulse wave similarity calculated by the similarity calculation unit 1402.

FIG. 15 is a diagram showing an example of pulse wave features. In the following description, FIG. 15 shows an example of pulse wave X1501. T11 illustrated in FIG. 15 indicates a time point corresponding to the minimum value a15 in the pulse wave X1501. Further, t12 illustrated in FIG. 15 indicates a time point corresponding to the maximum value a16 in the pulse wave X1501. H illustrated in FIG. 15 shows the difference value between the minimum value a15 at the time point t11 and the maximum value a16 at the time point t12 on the pulse wave X1501. For example, the feature amount extraction unit 1401 calculates h = (a16-a15) illustrated in FIG. 15 as one of the pulse wave feature amounts. In the following description, for convenience of explanation, the difference value between the adjacent minimum point and the maximum point in one pulse wave is referred to as "wave height".

Further, θ illustrated in FIG. 15 indicates the rising angle of the pulse wave X1501 between the time point t11 and the time point t12. For example, the feature amount extraction unit 1401 ^{calculates the rising angle θ = tan -1} ((a16-a15) / (t12-t11)) (rad) illustrated in FIG. 15 as one of the pulse wave feature amounts.

Next, an example of the processing flow executed by the information processing apparatus 100 of the present embodiment will be described. FIG. 16 is a flowchart showing an example of a processing flow executed by the information processing apparatus 100 of the present embodiment. In this example, when the number of frames of the moving image stored in the storage unit 201 exceeds a predetermined number of frames after the imaging unit 101 starts photographing, the control unit 202 of the present embodiment has S601 shown in FIG. ~ S604 processing is executed. Then, when the process of S604 shown in FIG. 6 is executed, the control unit 202 starts the process of S1601 shown in FIG.

In S1601, the feature amount extraction unit 1401 extracts the pulse wave feature amount from the attention pulse wave selected in S604. For example, the feature amount extraction unit 1401 extracts a pulse wave feature point from each of the selected pulse waves of interest. Then, the feature amount extraction unit 1401 extracts the pulse wave feature amount from the attention pulse wave based on the extracted pulse wave feature points.

When a plurality of maximum points and a plurality of minimum points are extracted from one pulse wave, the feature amount extraction unit 1401 extracts a plurality of wave heights h based on the maximum points and the minimum points. For example, when the feature amount extraction unit 1401 extracts 10 sets of combinations of maximum points and minimum points from one pulse wave, 10 wave heights h are calculated.

Further, when the feature amount extraction unit 1401 extracts a plurality of pulse wave feature amounts of the same type, the similarity calculation unit 1402 obtains the statistical values (for example, the average value or the median value) of the extracted plurality of pulse wave feature amounts. It is output as a pulse wave feature. Alternatively, the feature amount extraction unit 1401 outputs the extracted plurality of pulse wave feature amounts to the similarity calculation unit 1402.

Alternatively, the feature amount extraction unit 1401 may associate the pulse wave feature amount with the feature amount identification information and output the feature amount to the similarity calculation unit 1402. The feature amount identification information is information for identifying a pulse wave feature amount, for example, information indicating a time point corresponding to a pulse wave feature amount and information for identifying a pulse wave feature point corresponding to a pulse wave feature amount. The information for identifying the pulse wave feature points is, for example, a number assigned to each pulse wave feature point in one pulse wave.

In S1602, the similarity calculation unit 1402 compares the pulse wave features. Specifically, the similarity calculation unit 1402 compares the pulse wave features of each of the two attention pulse waves. For example, when the feature amount extraction unit 1401 extracts a plurality of pulse wave feature amounts, each pulse wave feature amount is compared.

In S1603, the similarity calculation unit 1402 calculates the pulse wave similarity based on the comparison result of each pulse wave feature amount corresponding to at least two attention pulse waves. Specifically, the similarity calculation unit 1402 calculates the pulse wave similarity based on the comparison result of each pulse wave feature amount corresponding to at least two attention pulse waves. Then, the control unit 202 shifts the processing to S606. Then, the control unit 202 executes the processes of S606 to S608. Then, when the pulse wave similarity is calculated for all the pulse waves (S607: Yes), the control unit 202 shifts the process to S1604.

For example, the similarity calculation unit 1402 calculates the pulse wave similarity based on the difference value of the pulse wave features of the same type corresponding to at least two attention pulse waves. The difference value of the pulse wave features is the difference between the features of at least two pulse waves of interest. Here, the similarity calculation unit 1402 may calculate the mean square error of at least two feature amounts of the pulse wave of interest as the difference value of the pulse wave feature amounts. Then, the similarity calculation unit 1402 outputs the calculated pulse wave similarity to the pulse wave synthesis unit 1403.

Further, when the feature amount extraction unit 1401 extracts the pulse wave feature amounts of a plurality of attention pulse waves of the same type with respect to at least two attention pulse waves, the similarity calculation unit 1402 may perform a plurality of attention pulse waves of one attention pulse wave. The difference value between the pulse wave feature amount and the plurality of pulse wave feature amounts of the attention pulse wave different from the one attention pulse wave is calculated. That is, the similarity calculation unit 1402 calculates a plurality of difference values for the pulse wave features of the same type. In that case, the similarity calculation unit 1402 outputs the calculated statistical values (for example, average value or median value) of the plurality of difference values to the pulse wave synthesis unit 1403 as the pulse wave similarity. Alternatively, when the similarity calculation unit 1402 calculates a plurality of difference values for the same type of pulse wave feature amount, the similarity calculation unit 1402 may output the plurality of difference values as the pulse wave similarity to the pulse wave synthesis unit 1403.

For example, it is assumed that the feature amount extraction unit 1401 extracts a plurality (for example, 10 pieces) of waveform feature amounts of wave heights such as h1, h2, ... For each of the two pulse waves. In that case, the similarity calculation unit 1402 determines the difference value between the wave height h1 corresponding to one pulse wave and the wave height h1 corresponding to the other pulse wave, the wave height h2 corresponding to one pulse wave, and the other pulse wave. The difference value from the wave height h2 corresponding to, ... Is calculated. Here, the wave height h1 corresponding to one pulse wave and the wave height h1 corresponding to the other pulse wave may be feature quantities related to the pulse wave at the same time. Then, the similarity calculation unit 1402 outputs the difference values of the extracted wave heights h1, h2, ... As the pulse wave similarity to the pulse wave synthesis unit 1403. Here, the average value of the extracted difference values of the wave heights h1, h2, ... May be output to the pulse wave synthesis unit 1403.

Further, when the feature amount extraction unit 1401 extracts the pulse wave feature amounts of a plurality of types of attention pulse waves with respect to at least two types of attention pulse waves, the similarity calculation unit 1402 extracts the pulse waves of one of the plurality of types of attention pulse waves. The difference value between the wave feature amount and the pulse wave feature amount of the attention pulse wave different from the one attention pulse wave of a plurality of types is calculated. That is, the similarity calculation unit 1402 calculates the difference value of each of the plurality of types of pulse wave feature amounts.

For example, it is assumed that the feature amount extraction unit 1401 extracts the pulse wave feature amount including the wave height h and the rising angle θ for each of the two pulse waves. In that case, the similarity calculation unit 1402 calculates the difference value between the wave height h corresponding to one pulse wave and the wave height h corresponding to the other pulse wave. Similarly, the similarity calculation unit 1402 calculates the difference value between the rising angle θ corresponding to one pulse wave and the rising angle θ corresponding to the other pulse wave. Then, the similarity calculation unit 1402 outputs the extracted difference value of the wave height h and the extracted difference value of the rising angle θ to the pulse wave synthesis unit 1403 as the pulse wave similarity.

In S1604, the pulse wave synthesis unit 1403 adds a pulse wave whose pulse wave similarity stored in the storage unit 201 in S606 satisfies a predetermined condition to the synthesis candidate pulse wave list. Then, the control unit 202 shifts the processing to S610.

For example, it is assumed that the similarity calculation unit 1402 calculates the difference value of the pulse wave features of the same type as the pulse wave similarity. In that case, the smaller the difference value of the pulse wave feature amount, the more the two attention pulse waves corresponding to the difference value are signals having similar features. Therefore, when the difference value of the pulse wave features corresponding to at least two attention pulse waves is within a predetermined range, the pulse wave synthesis unit 1403 adds the at least two attention pulse waves to the synthesis candidate pulse wave list. do.

Alternatively, the similarity calculation unit 1402 calculates a plurality of difference values of the pulse wave features of the same type for each of the plurality of types of pulse wave features, and uses the calculated difference values as the pulse wave similarity. It is assumed that it is calculated. When the similarity calculation unit 1402 has a difference value of one pulse wave feature amount within a predetermined range among a plurality of types of pulse wave feature amounts, the pulse wave synthesis unit 1403 has at least corresponding to the difference value. Two notable pulse waves are added to the synthetic candidate pulse wave list.

Alternatively, when the similarity calculation unit 1402 has a difference value of the pulse wave feature amount exceeding a predetermined number within a predetermined range among the plurality of types of pulse wave feature amounts, the pulse wave synthesis unit 1403 has the difference. At least two notable pulse waves corresponding to the values are added to the synthetic candidate pulse wave list.

Alternatively, the pulse wave synthesis unit 1403 determines the priority of the pulse wave of interest for each type of pulse wave feature amount in ascending order of the difference value of the pulse wave feature amount among the plurality of types of pulse wave feature amount. May be good. Alternatively, the pulse wave synthesis unit 1403 may determine the priority of the pulse wave of interest with respect to the plurality of types of pulse wave features in ascending order of the difference value of the pulse wave features. In that case, the pulse wave synthesis unit 1403 determines the pulse wave of interest to be added to the synthesis candidate pulse wave list based on the determined priority.

From the above, the information processing apparatus 100 of the present embodiment compares the pulse wave features and calculates the pulse wave similarity, so that the pulse wave similarity is calculated by comparing the entire pulse wave. The influence of noise contained in the pulse wave can be reduced. Further, the information processing apparatus 100 of the present embodiment can reduce the amount of calculation when calculating the pulse wave similarity as compared with the case of comparing the entire pulse wave by comparing the pulse wave feature amounts.

(Sixth Embodiment)
The sixth embodiment will be described. FIG. 17 is a block diagram showing an example of the functional configuration of the control unit 202 of the present embodiment. The difference between the control unit 202 illustrated in FIG. 17 and the control unit 202 illustrated in FIG. 3 is that a site detection unit 1701 is further provided. Further, the control unit 202 of the present embodiment replaces the pulse wave calculation unit 303, the similarity calculation unit 304, and the pulse wave synthesis unit 305 with the pulse wave calculation unit 1702, the similarity calculation unit 1703, and the pulse wave synthesis. A unit 1704 is provided.

The site detection unit 1701 detects a first site of the living body A and a second site different from the first site of the living body A from the moving image acquired by the image acquisition unit 301. For example, the first part is the neck and the second part is the face.

The pulse wave calculation unit 1702 calculates from the moving image a first pulse wave corresponding to the first part of the living body and at least two second pulse waves corresponding to the second part different from the first part.

The similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave and at least two second pulse waves.

The pulse wave synthesizing unit 1704 synthesizes at least two second pulse waves when the pulse wave similarity between the first pulse wave and at least two second pulse waves exceeds a predetermined threshold value.

Next, an example of the processing flow executed by the information processing apparatus 100 of the present embodiment will be described. FIG. 18 is a flowchart showing an example of a processing flow executed by the information processing apparatus 100 of the present embodiment. In this example, the first part is the neck of the living body A, and the second part is the face of the living body A. Further, in this example, when the number of frames of the moving image stored in the storage unit 201 exceeds a predetermined number of frames after the imaging unit 101 starts photographing the neck and face of the living body A, the control unit 202 , The process of S1801 shown in FIG. 18 is started.

In S1801, the image acquisition unit 301 acquires a moving image of the body surface from the storage unit 201. Each frame image 401 of the moving image includes pixel values of the neck region and the face region of the living body A.

In S1802, the site detection unit 1701 identifies the position of the neck and the position of the face from the moving image acquired in S1801.

For example, the site detection unit 1701 detects facial feature points (for example, both ends of the eyes, nose, and both ends of the mouth) from each frame image 401, and positions the face (second site) included in each frame image 401. To identify. Here, the position of the face is an area including a plurality of pixels. The position of the face of the living body A may be specified from the moving image by using pattern matching, SVM (Support Vector Machine), a neural network, or the like, and the details of the method for specifying the position of the face are not limited.

For example, the site detection unit 1701 identifies the position of the neck (first site) from the area of the body surface different from the specified face position in each frame image 401. Here, the position of the neck is an area including a plurality of pixels.

In S1803, the pulse wave calculation unit 1702 calculates the first pulse wave from the position of the neck specified in S1802. Here, the pulse wave calculation unit 1702 may calculate two first pulse waves corresponding to the left and right carotid arteries.

For example, the pulse wave calculation unit 1702 calculates a plurality of pulse waves (hereinafter, referred to as “first pulse wave candidates”) based on the temporal change of the pixel value of each pixel included in the neck position. Then, the pulse wave calculation unit 1702 determines the first pulse wave from the plurality of calculated first pulse wave candidates. For example, the pulse wave calculation unit 1702 determines the pulse wave having the largest statistical value (for example, average value) of the pulse wave amplitude as the first pulse wave from the plurality of calculated first pulse wave candidates.

Alternatively, the pulse wave calculation unit 1702 determines the pulse wave having the maximum signal noise ratio among the calculated first pulse wave candidates as the first pulse wave. For example, the pulse wave calculation unit 1702 calculates the power spectrum by performing a fast Fourier transform on a predetermined time interval among the plurality of calculated first pulse wave candidates. Then, the pulse wave having the maximum statistical value (for example, maximum value) of the power spectrum in a predetermined frequency band (for example, 0.5 Hz to 4.0 Hz) may be determined as the first pulse wave. Alternatively, for example, the ratio of the integrated value (signal) of the power spectrum in the predetermined frequency band (for example, 0.5 Hz to 4.0 Hz) and the integrated value (noise) of the power spectrum other than the predetermined frequency band is the maximum. The pulse wave may be determined as the first pulse wave.

In S1804, the region of interest extraction unit 302 extracts the region of interest 402 from the position of the face specified in S1802. When the pulse wave calculation unit 1702 calculates two first pulse waves corresponding to the left and right carotid arteries, the region of interest extraction unit 302 performs the two regions of interest corresponding to the left and right cheeks from the specified face position. Extract 402.

In S1805, the pulse wave calculation unit 1702 calculates a plurality of second pulse waves from the region of interest 402. When the region of interest extraction unit 302 extracts two regions of interest 402 corresponding to the left and right cheeks, the pulse wave calculation unit 1805 calculates a second pulse wave from the region of interest 402, respectively.

In S1806, the similarity calculation unit 1703 selects at least two attention pulse waves from the plurality of second pulse waves.

In S1807, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave and the pulse wave of interest selected in S1806. That is, the similarity calculation unit 1703 calculates the pulse wave similarity between the pulse wave at the neck position and the pulse wave at the face position. Specifically, when the pulse wave calculation unit 1702 calculates two first pulse waves corresponding to the left and right carotid arteries, the similarity calculation unit 1703 calculates the first pulse wave corresponding to the left carotid artery. And, the pulse wave similarity with a plurality of second pulse waves corresponding to the left cheek is calculated.

Similarly, when the pulse wave calculation unit 1702 calculates the two first pulse waves corresponding to the left and right carotid arteries, the similarity calculation unit 1703 determines the first pulse wave corresponding to the right carotid artery. The pulse wave similarity with a plurality of second pulse waves corresponding to the right cheek is calculated.

Here, since the carotid artery is thicker than the blood vessel in the facial part, the change in the volume of the carotid artery accompanying the beating of the heart is larger than the change in the volume of the blood vessel in the facial part. Therefore, the change in the color of the body surface of the carotid artery is more remarkable than the change in the color of the body surface of the facial part. In other words, the pulse wave corresponding to the pixel of the carotid artery portion calculated by the pulse wave calculation unit 1702 has a higher signal noise ratio than the pulse wave corresponding to the pixel of the face portion. Therefore, the similarity calculation unit 1703 calculates the pulse wave similarity between the pulse wave of the carotid artery and the pulse wave of the face portion, thereby calculating the pulse wave similarity between the pulse waves of the face portion. The pulse wave similarity of pulse waves connecting to the same blood vessel can be calculated more accurately than in the case.

In S1808, the similarity calculation unit 1703 stores the pulse wave similarity calculated in S1807 and the pulse wave of interest corresponding to the pulse wave similarity in the storage unit 201 in association with each other.

In S1809, the similarity calculation unit 1703 determines whether or not the pulse wave similarity has been calculated for all the second pulse waves. When the pulse wave similarity has not been calculated for all the second pulse waves (S1809: No), the similarity calculation unit 1703 selects at least two new notable pulse waves from the plurality of second pulse waves (S1810). ). Then, the control unit 202 continues the process by returning the process to S1807. On the other hand, when the pulse wave similarity is calculated for all the second pulse waves (S1809: Yes), the control unit 202 shifts the process to S1811.

In S1811, the pulse wave synthesis unit 1704 adds the second pulse wave whose pulse wave similarity satisfies a predetermined condition to the synthesis candidate pulse wave list. Thereby, the pulse wave synthesizing unit 1704 can add the pulse wave corresponding to the blood vessel connected to the same carotid artery to the synthesis candidate pulse wave list among the pulse waves corresponding to the blood vessel of the face portion. Since the details of the processing of S1811 are the same as those of 609 above, detailed description thereof will be omitted.

In S1812, the pulse wave synthesizing unit 1704 synthesizes the second pulse wave included in the synthesis candidate pulse wave list.

In S1813, the pulse wave synthesis unit 1704 outputs the synthesized second pulse wave.

FIG. 19 shows an example of the relationship between the pixels corresponding to the first pulse wave and the second pulse wave and the blood vessel. FIG. 19 shows a frame image 1901 including a region between the face and neck of the living body A. Blood vessels 1903 are facial blood vessels that connect to the carotid artery 1902. Blood vessels 1905 are facial blood vessels that connect to the blood vessels of the carotid artery 1904. Pixel 1906 is a pixel at a position corresponding to the carotid artery 1902. Further, the pixel 1907 is a pixel at a position corresponding to the carotid artery 1904.

The pulse wave calculation unit 1702 calculates the first pulse wave corresponding to the pixel 1907. Further, when the region of interest extraction unit 302 extracts the region of interest 1909, the pulse wave calculation unit 1702 calculates the second pulse wave corresponding to each pixel in the region of interest 1909. For example, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to the pixel 1907 and the second pulse wave corresponding to the pixel 1914. Further, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to the pixel 1907 and the second pulse wave corresponding to the pixel 1915.

Here, since the first pulse wave corresponding to the pixel 1907 is the pulse wave of the carotid artery 1904, the signal noise ratio is higher than that of the second pulse wave corresponding to the blood vessel 1905 of the face. Therefore, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to the pixel 1907 and the second pulse wave corresponding to the pixel 1915, and thereby the second pulse wave corresponding to the pixel 1914. And, the pulse wave similarity can be calculated more accurately than the case where the pulse wave similarity with the second pulse wave corresponding to the pixel 1915 is calculated.

The pulse wave synthesizing unit 1704 synthesizes a pulse wave having a pulse wave similarity with a first pulse wave corresponding to the pixel 1907 satisfying a predetermined condition from at least two pulse waves corresponding to the pixels in the region of interest 1909. Determined as a candidate pulse wave. As a result, the pulse wave synthesizing unit 1704 determines the synthesis candidate pulse wave based on the pulse wave similarity of the pulse wave corresponding to at least two pixels in the region of interest 1909, and calculates the synthetic pulse wave. The signal-to-noise ratio of the synthetic pulse wave becomes high.

Similarly, when the region of interest extraction unit 302 extracts the region of interest 1908, the pulse wave calculation unit 1702 calculates the second pulse wave corresponding to each pixel in the region of interest 1908. Here, since the first pulse wave corresponding to the pixel 1906 is the pulse wave of the carotid artery 1902, the signal noise ratio is higher than the pulse wave corresponding to the pixel 1912 at the position corresponding to the blood vessel 1903 of the face. Therefore, similarly to the above, the similarity calculation unit 1703 corresponds to the pixel 1912 by calculating the pulse wave similarity between the first pulse wave corresponding to the pixel 1906 and the second pulse wave corresponding to the pixel 1913. The pulse wave similarity can be calculated more accurately than in the case of calculating the pulse wave similarity between the second pulse wave and the second pulse wave corresponding to the pixel 1913.

The output control unit 306 calculates synthetic pulse waves corresponding to the left and right sides of the face. If there are no abnormalities in the blood vessels from the heart to the face, the left and right pulse waves of the face are similar to each other. However, when there is an abnormality in the blood vessels from the heart to the face, the shape of one of the left and right pulse waves of the face may change from the normal state. As a result, if there is an abnormality in the blood vessels from the heart to the face, the left and right pulse waves of the face will be different. Therefore, the difference between the left and right pulse waves of the face can be an index for determining whether or not there is an abnormality in the blood vessels from the heart to the face. Therefore, the output control unit 306 may output the difference between the left and right pulse waves of the face. As a result, the information processing device 100 can output information indicating that there is an abnormality in the blood vessels from the heart to the face.

The main body of the information processing device or the pulse wave calculation method in the present disclosure includes a control unit 202 as a computer. When this computer executes a program stored in the storage unit 201, the function of the main body of the information processing device or the pulse wave calculation method in the present disclosure is realized. The computer includes a processor that operates according to the above program as a main hardware configuration. The type of processor does not matter as long as the function can be realized by executing the above program. The processor is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or an LSI (Large Scale Integration). The plurality of electronic circuits may be integrated on one chip or may be provided on a plurality of chips. A plurality of chips may be integrated in one device, or may be provided in a plurality of devices. The program is recorded on a non-temporary recording medium such as a computer-readable ROM, optical disk, or hard disk drive. The above program may be stored in a recording medium in advance, or may be supplied to the recording medium via a wide area communication network including the Internet or the like.

One aspect of the present disclosure is not limited to the above-described embodiments and modifications, and is deformable, and the above configurations are substantially the same configuration, configurations exhibiting the same effects, or the same purpose. Can be replaced with a configuration that can achieve.

100 information processing device, 101 imaging unit, 201 storage unit, 202 control unit, 203 output unit, 301 image acquisition unit, 302 area of interest extraction unit, 303 pulse wave calculation unit, 304 similarity calculation unit, 305 pulse wave synthesis unit, 306 Output control unit, 401 frame image, 402 area of interest, 403a pixel, 403b pixel, 501 pixel, 701 area of interest, 701a pixel, 701b pixel, 702 area of interest, 702a pixel, 702b pixel, 703 area of interest, 703a pixel, 703b Pixel, 704 region of interest, 704a pixel, 704b pixel, 711 blood vessel, 712 blood vessel, 901 biometric information extraction unit, 1001 region determination unit, 1002 pulse wave calculation unit, 1101 region, 1102 region, 1103 region, 1104 region, 1201 pulse wave Calculation unit, 1301 pixels, 1302 peripheral pixels, 1303 regions, 1401 feature amount extraction unit, 1402 similarity calculation unit, 1403 pulse wave synthesis unit, 1701 site detection unit, 1702 pulse wave calculation unit, 1703 similarity calculation unit, 1704 pulse Wave synthesizer, 1901 frame image, 1902 carotid artery, 1903 blood vessel, 1904 carotid artery, 1905 blood vessel, 1906 pixels, 1907 pixels, 1908 region of interest, 1909 region of interest, 1912 pixels, 1913 pixels, 1914 pixels, 1915 pixels

Claims (12)

A pulse wave calculation unit that calculates a plurality of pulse waves corresponding to different positions on the body surface from a moving image of the body surface,
A similarity calculation unit that calculates the similarity of at least two pulse waves among the plurality of pulse waves,
A pulse wave synthesizer that synthesizes pulse waves based on the similarity,
Information processing device equipped with. The information processing apparatus according to claim 1, wherein the pulse wave synthesis unit synthesizes from at least two pulse waves calculated by the pulse wave calculation unit. The information processing apparatus according to claim 1, wherein the pulse wave synthesizing unit synthesizes a pulse wave from the moving image. The information processing apparatus according to any one of claims 1 to 3, further comprising an imaging unit that photographs the body surface and acquires the moving image. The information processing apparatus according to any one of claims 1 to 4, wherein the similarity calculation unit calculates the similarity based on at least one of a correlation value and a difference value of the at least two pulse waves. .. A feature amount extraction unit for extracting a feature amount indicating the feature of the shape of the pulse wave is further provided.
The information processing device according to any one of claims 1 to 4, wherein the similarity calculation unit calculates the similarity based on a comparison result of at least two features. A region determination unit for determining a plurality of regions including a plurality of pixels from the moving image is further provided.
Any one of claims 1 to 6, wherein the pulse wave calculation unit calculates each pulse wave of the plurality of pulse waves based on the statistical value of the pixel value of each region of the plurality of regions. The information processing device described in the section. The pulse wave calculation unit is based on a statistical value of a pixel value of one pixel included in the moving image and a pixel value of at least one pixel adjacent to the one pixel, and among the plurality of pulse waves, the pulse wave calculation unit said. The information processing device according to any one of claims 1 to 6, which calculates one pulse wave corresponding to one pixel. The pulse wave calculation unit calculates from the moving image a first pulse wave corresponding to the first part of the living body and at least two second pulse waves corresponding to the second part different from the first part.
The pulse wave synthesizing unit synthesizes at least two second pulse waves when the similarity between the first pulse wave and the at least two second pulse waves exceeds a predetermined threshold value. The information processing apparatus according to any one of 8. The information processing apparatus according to claim 9, wherein the first part is a neck and the second part is a face. The information processing apparatus according to any one of claims 1 to 10, further comprising a biological information extraction unit that extracts biological information based on a pulse wave synthesized by the pulse wave synthesis unit. A step of calculating a plurality of pulse waves corresponding to different positions on the body surface from a moving image of the body surface, and
The step of calculating the similarity of at least two pulse waves among the plurality of pulse waves, and
The step of synthesizing pulse waves based on the similarity,
Pulse wave calculation method including.

Images