JP7405527B2 - Change timing detection device, change timing detection method, and change timing detection program - Google Patents

Description

The present invention relates to a change timing detection device, a change timing detection method, and a change timing detection program.

BACKGROUND ART Conventionally, a method has been proposed for analyzing an image of a person captured by a camera and detecting whether or not an abnormality has occurred in the person's breathing.

For example, Patent Document 1 discloses that a predetermined illumination pattern is projected onto a person, the amount of movement between frames of the projected illumination pattern is calculated from an image of the person, and the amount of movement between frames is arranged in chronological order. A method of generating volume waveform data and detecting the breathing of a sleeping person based on the movement volume waveform data is disclosed.

Additionally, Patent Document 2 discloses a method for measuring the respiration rate of a symmetric person by specifying the position of the person's surface and measuring changes in the person's shoulders and chest based on a captured image that includes depth data. has been done.

Further, Patent Document 3 describes a method in which a chest region of a person is set in advance in an image of the person, and exhalation and inhalation are measured from temporal changes in image density and changes in edge position of the set region. is disclosed.

Japanese Patent Application Publication No. 2002-175582 JP2017-038673A Japanese Patent Application Publication No. 2005-218507

However, all of the methods described in Patent Documents 1 to 3 detect a person's breathing based on changes in the person's movement on a single time axis obtained from an image of the person. However, even if attention is paid to changes in movement on the one-time axis in this way, abnormal values indicating abnormal breathing or the like may not be accurately detected. In order to accurately detect the timing at which exhalation and inhalation switch, it is necessary to measure the timing at which the movements of a breathing person switch, based on the timing at which the movements measured on individual time axes at multiple observation points change. It is necessary to detect typical timing, but it has been difficult to accurately detect this representative timing from changes in movement on the one-time axis.

In this way, in the past, the timing of change, which is a typical timing when the state of the movement (direction of movement) etc. of the detection target changes, such as the timing when the movements of various parts of a person change simultaneously due to breathing, could be detected with high precision. There was room for improvement in terms of detection.

In view of the above-mentioned problems, an object of the present invention is to provide a change timing detection device, a change timing detection method, and a change timing detection program that are capable of highly accurately detecting typical timings at which the state of a detection target changes. do.

In order to solve the above-mentioned problems and achieve the objects, the present invention provides a change timing detection device for determining change timing of a detection target based on multidimensional time series data of a space including the detection target, a measurement unit that measures individual change timing, which is the timing at which the state of the detection target changes, at each of a plurality of observation points set for the detection target in the multidimensional time series data; and the multidimensional time series data. extracting two or more individual change timings at each observation point from each frame toward the past, and detecting change timing of the detection target based on the two or more individual change timings at each of the extracted observation points. It is characterized by comprising a detection section.

Further, in the above invention, the detection unit detects the change timing by voting measurement times of the two or more individual change timings at each of the extracted observation points in a two-dimensional or more voting space. It is characterized by

Further, the present invention is characterized in that, in the above-mentioned invention, the detection unit votes by changing the weight according to the reliability of the individual change timing at the time of voting.

Furthermore, in the above invention, the change timing detection device may track the change in the value of the time series of the change timing closest to each frame in the past, and The apparatus is characterized by further comprising a determination unit that determines that the switching of the state of the detection target is normal when the value of changes.

Further, the present invention is characterized in that, in the above-mentioned invention, the multidimensional time series data is a video.

Furthermore, the present invention is characterized in that, in the above-mentioned invention, the state is movement.

Further, in the above invention, the measurement unit detects individual change timing candidates at each of the observation points, and detects the detection target at the observation point between temporally consecutive individual change timing candidates. The present invention is characterized in that the average direction of movement is measured, and the measured average direction is classified into two groups according to the direction.

Further, the present invention is characterized in that, in the above-mentioned invention, the detection target is a living body, and the change timing is a timing at which exhalation and inhalation of the living body switch.

Further, the present invention is characterized in that, in the above invention, the state is luminance.

Further, in the above invention, the present invention provides that the detection target is a living body, the plurality of observation points include a skin area of the living body in the image, and the measurement unit is configured to detect a skin region of the living body in the image. The method is characterized in that the individual change timing is measured based on a change in brightness value.

The present invention also provides a change timing detection method for determining change timing of the detection target based on multidimensional time series data of a space including the detection target, the method comprising: a measurement step of measuring the individual change timing, which is the timing at which the state of the detection target changes, at each of the plurality of observation points set in the above-mentioned observation points; The method is characterized by comprising a detection step of extracting two or more individual change timings at each of the observation points, and detecting a change timing of the detection target based on the two or more extracted individual change timings at each of the observation points.

The present invention also provides a change timing detection program for determining change timing of the detection target based on multidimensional time series data of a space including the detection target, the program comprising: A measurement process that measures individual change timing, which is the timing at which the state of the detection target changes, at each of a plurality of observation points set as a A detection process of extracting two or more individual change timings at each of the observation points and detecting a change timing of the detection target based on the two or more individual change timings at each of the extracted observation points is caused to be executed by a computer. shall be.

According to the change timing detection device, change timing detection method, and change timing detection program of the present invention, typical timings at which the state of a detection target changes can be detected with high accuracy.

3 is an example of a graph showing short-term fluctuations in time-series values of change timing detected in the first embodiment. 3 is an example of a graph showing long-term fluctuations in time-series values of change timing detected in the first embodiment. 1 is a block diagram illustrating the overall configuration of a system including a change timing detection device according to a first embodiment; FIG. 1 is a block diagram illustrating the configuration of a change timing detection device according to a first embodiment. FIG. FIG. 2 is a schematic diagram for explaining an image captured in the first embodiment and a processing area set within the image; FIG. It is an example of a graph showing the length of a vector at an observation point synthesized by a measurement unit. 7 is a graph showing the result of Gaussian differentiation of the combined vector length shown in FIG. 6; It is an example of the graph which shows the direction of the vector in the observation point produced|generated by the measurement part. 5 is a flowchart illustrating an example of a procedure of change timing detection processing performed by the change timing detection device according to the first embodiment.

In this specification, "multidimensional time series data" refers to data obtained by sequentially detecting a predetermined physical quantity (for example, brightness) over time at multiple points in a region set in real space. For example, it includes video (video data). Also, similar to the case of video, a series of data blocks detected at multiple points in multidimensional time-series data is called a frame.

Furthermore, in this specification, "each frame of multidimensional time series data" or "each frame" may be each frame itself that constitutes multidimensional time series data acquired by a camera, radar, etc. , frames constituting multidimensional time series data acquired by a camera, radar, etc. may be appropriately thinned out. In addition, the description "closest in the past to a frame" means that the frame (in the case of a video, a still image) is temporally closest in the past from the time when it was acquired. Further, the expression "from a frame to the past" means going back to the past from the time when a frame (in the case of a video, a still image) was acquired.

Hereinafter, preferred embodiments of a change timing detection device, a change timing detection method, and a change timing detection program according to the present invention will be described with reference to the drawings. In the present invention, the detection target whose change timing is to be detected is not particularly limited as long as the state of the detection target (for example, movement or brightness) changes within multidimensional time series data. The detection target is a person, and the image of the person is acquired as multidimensional time-series data of the detection target, and the change timing, which is a typical timing when the state of the detection target changes, is determined based on the movement (direction of movement) of the person, especially breathing. A case will be described in which the timing at which the movement (direction of movement) associated with changes is detected will be described.

<Outline of processing in this embodiment>
First, an overview of the processing in this embodiment will be explained using FIGS. 1 and 2. The flow of processing in this embodiment is as shown in (i) to (v) below.

(i) At each of a plurality of observation points in a video in which a person is photographed, the individual change timing, which is the timing at which the movement of the photographed person changes in time-series data, is measured. The individual change timing is determined based on the timing at which the optical flow reverses at each observation point.

(ii) Using the measurement time of the individual change timing as a coordinate value, vote in the voting space with a weight of one observation point and one vote. The voting space may have any dimension as long as it is two or more dimensions, for example, it may be three or four dimensions or more. However, the coordinate axes of the voting space are the measurement times of two or more individual change timings in order from the closest in the past to each frame of the video toward the past. For example, if the voting space is three-dimensional, the first axis is the measurement time of the most recent (closest in the past) individual change timing of the frame of interest, and the second axis is the measurement time of the previous individual change timing. The third axis is the measurement time of the two previous individual change timings.

(iii) In the voting space, the coordinate value (measured time of individual change timing) voted from the largest number of observation points is detected as the observed time of change timing in that frame. As a result, for each frame, the same number of change timings as the degree of the voting space are detected. For example, when the voting space is three-dimensional, as shown in FIG. 1, three observation times are detected for each frame: the most recent, one previous, and two previous change timings.

(iv) Grouping change timings for a predetermined frame into groups with close observation times. For example, when the voting space is three-dimensional, as shown by the broken line in FIG. 1, frames whose observation times of the most recent change timing, one previous change timing, and two previous change timings are close to each other are classified into the same group.

(v) Determine whether the change timing of each group satisfies a predetermined condition, and if it does, it is determined that the breathing of the photographed person is normal; if it does not, the breathing of the photographed person is abnormal. It is determined that Specifically, for example, if the voting space is three-dimensional, if the change timing of the group of interest satisfies the following two conditions (I) and (II), it is determined that the photographed person's breathing is normal.
(I) A and B of the focus group match B and C of the future group, respectively. (II) B and C of the focus group match A and B of the past group, respectively. A indicates the latest observation time, B indicates the previous observation time, and C indicates the observation time of the previous change timing. Note that "matching" herein includes not only completely matching but also substantially matching. The specific range for when they substantially match can be set as appropriate depending on the detection target and the target for which the change timing is detected, but in this embodiment, which detects the change timing that changes depending on the movement associated with breathing, It is assumed that the observation times of the plurality of change timings match if they fall within a time period of 5 frames (0.5 seconds for 10 fps), for example.

In the above process, multiple observation points (voting with many votes), multiple observation times (multidimensional voting space), and voting result verification rules are used, so only signals with high reliability can be extracted in the end. . Therefore, even if the signal is small and weak, it is possible to vote anyway, and failure to detect change timing can be prevented.

In addition, by verifying the voting results according to predetermined rules, it is possible to more accurately detect non-periodic change timing (such as periods of respiratory arrest or turning over in bed), as shown by the area between the broken lines in Figure 2. can.

<Overall system configuration including change timing detection device>
Next, the overall configuration of a system including the change timing detection device according to the first embodiment will be described using FIG. 3. As shown in FIG. 3, this system is constructed for a sleeping person indoors, and includes a camera 1 that captures two-dimensional images and a change timing detection device 2. . The camera 1 is communicably connected to the change timing detection device 2.

The camera 1 is usually installed at a desired location indoors (for example, at a high place). A person sleeping indoors (for example, a child at a nursery school) is photographed by the camera 1 at a predetermined frame rate (for example, 10 fps). At this time, at least a part of the person that moves as the person breathes (hereinafter referred to as a "movable part") is photographed. Preferred specific examples of the movable parts include, for example, the chest, back, abdomen, shoulders, and the like. The camera 1 may photograph an object that changes at the same timing as the movable part moves. For example, a futon or blanket placed over a person may be photographed. The photographed video (moving image) is output to the change timing detection device 2. The camera 1 may acquire RGB color images or monochrome images. Further, the camera 1 may be a camera sensitive to visible light or a camera sensitive to infrared rays (infrared camera).

The change timing detection device 2 is connected to a monitor (display device, not shown) that displays captured images, etc., and an input device (for example, a mouse, not shown) that allows the operator to perform various input operations. There is. Note that the monitor and the input device may include a display device with an input function, such as a touch panel display.

The change timing detection device 2 is configured so that an operator can view the video captured by the camera 1 in real time on a monitor.

<Configuration of change timing detection device>
Next, the configuration of the change timing detection device 2 will be described using FIG. 4. The change timing detection device 2 is composed of an information processing device having functions equivalent to a general personal computer, and includes a control section 10 and a storage section 20, as shown in FIG.

The control section 10 has the functions of a video input section 11 , a measurement section 12 , a detection section 13 , and a determination section 14 . The control unit 10 includes, for example, a software program for implementing various processes, a CPU (Central Processing Unit) that executes the software program, and various hardware controlled by the CPU. Software programs and data necessary for the operation of the control unit 10 are stored in the storage unit 20.

Note that each part of the control unit 10 shown in FIG. 4 is realized by causing the CPU of the control unit 10 to execute a change timing detection program. The change timing detection program may be installed in the change timing detection device 2 in advance, or may be operated as an application program that can run on a general-purpose OS, recorded on a computer-readable recording medium, or via a network. may be provided to the person.

The storage unit 20 is composed of a storage device such as a hard disk device or a nonvolatile memory.

The video input unit 11 performs processing to acquire video from the camera 1 and outputs the video to the measurement unit 12.

The measurement unit 12 measures individual change timing, which is the timing at which the movement of the person changes, at each of a plurality of observation points set for the person in the video.

As shown in FIG. 5, a processing area 31 is set within the video 30, which is an area where change timing is detected. Note that FIG. 5 shows a processing area 31 set within the video 30 and an enlarged version of the processing area 31. As described above, the processing area 31 is set to include at least a movable part (an object that changes at the same timing as the movable part moves). Further, the processing area 31 is divided into a plurality of segments 32 by a mesh having a predetermined pitch. The segments 32 are rectangular areas of the same size, and the size of each segment 32 is, for example, 10 pixels in the vertical and horizontal directions (10×10 pixels). Then, in each segment 32, the measurement unit 12 performs a process of measuring individual change timing. That is, each of these segments 32 corresponds to an observation point.

The flow of processing by the measurement unit 12 is as shown in (A) to (D) below.

(A) Optical flows (vectors) are calculated at regular intervals for each observation point in the processing area 31 (see FIG. 5).

(B) Convert past vectors at the observation point of interest into unit vectors and synthesize (average) them. The past corresponds to, for example, 10 frames including the frame of interest.

(C) Gaussian differentiation is performed on the length of the combined vector (see FIG. 6), and the point where it changes from positive to negative is set as a candidate for the individual change timing (see the timing indicated by the arrow in FIG. 7).

(D) From the direction of the combined vector, calculate the average direction between two adjacent individual change timing candidates (see Figure 8), and if the calculated average direction is reversed before and after the individual change timing candidate, , the candidate is determined at the individual change timing. In the graph of FIG. 8, the vertical axis, that is, the direction of the combined vector, is expressed as 36=360°.

Details of the process (D) above are as shown in (a) to (c) below.

(a) From the direction of the combined vector, calculate the average direction among temporally consecutive individual change timing candidates for all individual change timing candidates in the past 10 seconds. Note that the term "between temporally consecutive individual change timing candidates" means the time interval between the measurement time of a certain individual change timing candidate and the measurement time of the next individual change timing candidate.

(b) Divide the obtained average direction into two classes. Specifically, it is classified into two classes: a class closer to the maximum value of the direction of the combined vector, and a class closer to the minimum value of the direction of the combined vector.

(c) If the average direction classes are different before and after an individual change timing candidate, that candidate is determined as the individual change timing.

Information regarding the individual change timing at each observation point measured by the measurement unit 12 is stored in the storage unit 20 by the measurement unit 12 as individual change timing information 21. The individual change timing information 21 includes time information of all individual change timings measured at each observation point. Here, the measurement time of each individual change timing is stored in the storage unit 20 in association with the observation point at which the individual change timing was measured.

The detection unit 13 first extracts two or more individual change timings at each of the observation points in order from the one closest in the past to each frame of the video toward the past.

More specifically, the detection unit 13 detects the observation time of the individual change timing measured at each observation point for each frame of the video based on the individual change timing information 21 stored in the storage unit 20. Two or more images are extracted in order from the time when the image was taken, going back in time and starting from the closest one. As a result, for example, when extracting three individual change timings, for each observation point, the observation time of the most recent (closest in the past) individual change timing of the frame of interest and the observation of the previous individual change timing. The time and the observation time of the individual change timing two years before the time are read out from the storage unit 20 in correspondence with each other.

Then, the detection unit 13 detects the change timing of the person who is the detection target based on two or more individual change timings at each of the extracted observation points.

More specifically, the detection unit 13 calculates the observation times of two or more individual change timings at each extracted observation point into a voting space whose coordinate axis is the observation time of the individual change timing, with a weight of one observation point and one vote. , vote by point or area. Here, the number of dimensions of the voting space corresponds to the number of individual change timings extracted at each observation point. For example, if three individual change timings are extracted, the voting space becomes three-dimensional and the focus A coordinate axis showing the observation time of the most recent (closest in the past) individual change timing in the frame to be used, a coordinate axis showing the observation time of the previous individual change timing, and a coordinate axis showing the observation time of the individual change timing two years before that. It consists of coordinate axes.

Then, the detection unit 13 detects the timing of change in the photographed person based on the distribution of the final number of votes. Here, each coordinate value of the coordinate that received the most votes, that is, the observation time of two or more consecutive individual change timings going back to the past from the most recent (closest in the past) to the frame of interest, is determined for each person. This is the timing of change. As a result, two or more change timings are detected in order from the past closest to the frame of interest toward the past.

Furthermore, the detection unit 13 performs the above-mentioned voting process for each frame. As a result, for each frame, two or more change timings are detected in order from the closest one in the past to each frame toward the past.

Information regarding the change timing detected by the detection unit 13 is stored in the storage unit 20 by the detection unit 13 as change timing information 22. The change timing information 22 includes all time information of two or more change timings detected for each frame. Here, information for identifying a frame (for example, a frame number) and observation times of two or more change timings detected for the frame are stored in the storage unit 20 in association with each other.

The determining unit 14 determines whether the photographed person is breathing normally or not based on the change timing information 22 stored in the storage unit 20. At this time, the determination unit 14 tracks the change in the time series value of the change timing closest to each frame in the past (for example, refer to the latest data shown in FIG. 1), and determines the second closest change timing in the past to each frame. When the time-series value of the change timing (for example, refer to the previous data shown in FIG. 1) changes, it is determined that the photographed person's exhalation and inhalation switching, that is, the breathing is normal. On the other hand, if the time-series value of the second closest change timing in the past does not fluctuate following the change in the time-series value of the closest change timing in the past, the photographed person may be suffering from apnea or respiratory arrest. , it is determined that the patient is breathing unstable due to turning over in bed, etc. In that case, the control unit 10 performs a notification process to notify the operator that the photographed person is breathing unstable.

More specifically, as shown in Figure 2, in a graph where the horizontal axis is the frame number corresponding to the time when the frame was captured and the vertical axis is the time when the change timing was detected, the most recent If the observation time of the previous change timing (closest in the past) changes as the frame number increases, then the photographed person The patient's breathing is determined to be normal.

In particular, in this embodiment, as described above, the determining unit 14 determines the time series values of the change timing from the closest to the third in the past for each frame (for example, the latest, the previous and the previous data), it may be determined whether the person photographed is breathing normally. Thereby, it is possible to more accurately determine whether the photographed person is breathing normally. In addition, if the change timing of the focused group satisfies the above two conditions (I) and (II), the breathing of the photographed person is determined to be normal, and at least one of the above two conditions (I) and (II) is satisfied. If the conditions are not satisfied, it may be determined that the breathing of the photographed person is abnormal.

As mentioned above, following the fluctuation of the time series value at a certain change timing (for example, the most recent data shown in Figures 1 and 2), the time series value at another change timing (for example, the most recent data shown in Figures 1 and 2) is followed The change in the previous data (shown in 2) may mean that the corresponding values in both time series, that is, the observation times of the corresponding change timings in both time series match. Note that, as mentioned above, "matching" here includes not only completely matching but also substantially matching, and in the case of substantially matching, the observation time of the corresponding change timing is, for example, It may be within the time of 5 frames (0.5 seconds at 10 fps).

As shown in Figure 1, if the time-series values of the change timing from the closest in the past to at least the second in each frame fluctuate in a step-like manner, the breathing of the photographed person is normal. It can be determined that there is.

<Procedure for change timing detection process>
Next, the procedure of the change timing detection process performed by the change timing detection device 2 will be explained using FIG.

As shown in FIG. 9, first, a video image of a person to be detected is input from the camera 1 to the video input unit 11 (S11). That is, each frame of the video is input sequentially.

Next, when a predetermined frame of video is input from the video input unit 11 to the measurement unit 12, the measurement unit 12 determines the timing at which the movement of the photographed person changes at each observation point, as described above. The change timing is measured (S12).

Next, the detection unit 13 detects the change timing of the photographed person based on the individual change timing measured by the measurement unit 12 as described above (S13).

Then, when change timings are detected for a predetermined number of frames, the determination unit 14 determines whether the change timing is based on the time series values of change timings from the closest to each frame to at least the second one in the past, as described above. Then, it is determined whether the photographed person's breathing is normal (S14), and if it is normal (YES in S14), the above-described process is repeated until an instruction to end the process is received from the operator. If it is not normal (NO in S14), a notification process is performed to notify the operator that there is an abnormality in the breathing of the photographed person (S15), and then until an instruction to end the process is received from the operator. Repeat the above process.

As explained above, in this embodiment, the individual change timing at which the movement (state) of the photographed person changes is measured at each of the plurality of observation points set for the person who is the detection target in the video. , two or more individual change timings at each observation point are extracted in order from the closest one in the past to each frame of the video, and the photographed person is determined based on the two or more individual change timings at each extracted observation point. By detecting the change timing of the person photographed at each observation point, the movement (state) of the person photographed changes from the individual change timing at which the movement (state) of the photographed person changes, specifically the intake and exhaust of breathing. It is possible to detect a typical timing as a change timing with high precision.

Furthermore, in this embodiment, the change timing is detected by voting the measurement times of two or more individual change timings at each extracted observation point in a two-dimensional or more voting space, so the change timing can be determined with higher precision. Can be detected.

Furthermore, in this embodiment, when the time-series value of the change timing that is the second closest in the past to each frame changes, following the change in the time-series value of the change timing that is closest in the past to each frame, the image is captured. Since it is determined that the change in the movement (state) of the photographed person is normal, it is possible to determine that the change in the movement (state) of the photographed person, that is, the change in breath between intake and exhalation, is normal.

Furthermore, in this embodiment, since the video is acquired as multidimensional time series data, it is possible to calculate the moving direction of the person who is the detection target using optical flow. Further, as a means for acquiring multidimensional time-series data (video), it is possible to use a commonly used low-cost photographing device such as a camera.

Furthermore, in this embodiment, since movement is detected as the state of the photographed person, it is possible to detect with high precision a typical timing at which the movement of the photographed person changes.

In addition, in this embodiment, since the timing at which the photographed person (living entity) switches between exhalation and inhalation is detected as the change timing, the change timing accompanying the breathing of the photographed person (living entity) can be detected with high precision. Can be detected.

In addition, in this embodiment, candidates for individual change timings are detected at each observation point, and the average direction of the movement of the photographed person at the observation point is measured between the candidates for temporally consecutive individual change timings, By classifying the measured average direction into two groups depending on the direction, it can be determined whether the movement of the person photographed in any direction has reversed or not, so it is possible to determine the timing of individual changes in the movement of the person photographed. It can be measured with high accuracy.

In addition, in this embodiment, optical flow is used to measure individual change timing, and the reversal timing of optical flow is converted into a unit vector rather than the minimum value of the flow size, extracts only the directional component, and converts the unit vector into a nearby Since the length of the composite vector synthesized in the (sky) region is the minimum value when can.

In addition, in the above embodiment, the case where the individual change timing at each observation point is voted with the weight of 1 observation point 1 vote, that is, the same weight, was explained. You can change your vote. Thereby, the change timing can be detected with higher accuracy. More specifically, as shown in FIG. 6, the average value between the minimum values of the lengths of the combined vectors may be calculated, and the voting may be performed after multiplying the average value by the corresponding individual change timing. .

Further, in the above embodiment, a case has been described in which optical flow is used to measure individual change timing, but the method of measuring individual change timing is not particularly limited to a method that uses optical flow. Specifically, for example, the individual change timing may be measured by tracking the variation (brighter or darker) in the brightness value of the pixel of interest (each observation point).

Further, in the above embodiment, a case has been described in which movement is detected as the state of the photographed person, but brightness may be detected as the state of the detection target. Thereby, typical timings at which the brightness of a detection target (for example, a living entity such as a person) changes can be detected with high precision.

Further, in this case, the plurality of observation points may include the skin area of the living entity (for example, a person) in the video, and the individual change timing may be measured based on the change in the brightness value at each observation point. Thereby, the timing of changes associated with the pulsation of the living body can be detected with high precision. This is because hemoglobin in blood has a strong property of absorbing green wavelengths (near 550 nm), and pulsations can be detected from minute temporal changes in green luminance values. Therefore, from the viewpoint of detecting pulsation with higher precision, it is preferable to acquire RGB color images and measure individual change timings based on changes in the green luminance value at each observation point.

Furthermore, in the above embodiment, the case where the detection target is a person (living object) has been described, but the detection target may be anything that changes its state such as movement or brightness, and is not particularly limited to a living object, but an inanimate object. It may be. Specifically, detection targets that change in movement include, for example, buildings such as steel towers and electric wires, outdoor equipment, etc. If these objects are shaking slightly, the timing of the change in shaking (movement) can be detected. It is possible to detect it from the video.

Further, in the above embodiment, the detection unit 13, the detection step, and the detection process extract two or more individual change timings at each observation point in order from the closest one in the past to each frame of the video, and each extracted The case has been described in which the change timing of a photographed person is detected based on two or more individual change timings at an observation point. You may extract two or more individual change timings at each observation point in order from the one closest to the past toward the past, or every other change timing from the closest or second closest in the past to each frame of the video. Two or more individual change timings at each observation point may be extracted, and the change timing of the detection target may be detected based on these extracted two or more individual change timings. In this way, the detection unit 13, the detection step, and the detection process extract two or more individual change timings at each observation point from each frame of the video toward the past, and extract two or more individual change timings at each of the extracted observation points. The change timing of the detection target may be detected based on the following.

Furthermore, in the above embodiment, a case has been described in which a person who is a detection target is photographed with the camera 1 and the image is acquired as multidimensional time series data, but the means for acquiring multidimensional time series data of a detection target is The photographing means is not particularly limited, and for example, a radar such as a millimeter wave radar may be used. When we transmit millimeter waves toward a person and observe the reflected waves reflected by that person, we find that when the person is breathing in, the phase shift between the transmitted wave and the reflected wave tends to be smaller, and when the person is breathing out, the phase shift between the transmitted wave and the reflected wave tends to be smaller. There is a tendency for the phase shift between the transmitted wave and the reflected wave to become large. Furthermore, the phase shift between the transmitted wave and the reflected wave is also synchronized with the heartbeat. Therefore, in the same way as the brightness value of an image, the individual change timing may be measured by tracking the fluctuation of this phase shift (whether it is increasing or decreasing). Then, by performing the above-mentioned voting using the two or more individual change timings at each observation point obtained in this way, it is possible to detect the change timing of the detection target more accurately in the same way as using video. can.

Although the embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the above embodiments. Furthermore, the configurations of each embodiment may be combined or modified as appropriate without departing from the gist of the present invention.

As described above, the present invention is a useful technique for detecting typical timings at which the state of a detection target changes, based on multidimensional time-series data of a space that includes the detection target.

1: Camera 2: Change timing detection device 10: Control section 11: Video input section 12: Measurement section 13: Detection section 14: Judgment section 20: Storage section 21: Individual change timing information 22: Change timing information 30: Video 31: Processing area 32: Segment (observation point)

Claims (11)

A change timing detection device that determines change timing of a detection target based on multidimensional time series data of a space including the detection target, comprising:
a measurement unit that measures individual change timing, which is the timing at which the movement or brightness of the detection target changes, at each of a plurality of observation points set for the detection target in the multidimensional time series data;
Two or more individual change timings at each observation point are extracted from each frame of the multidimensional time series data toward the past, and based on the two or more individual change timings at each observation point extracted, the detection target is detected. a detection unit that detects the change timing of the
The detection unit detects the measurement times of the two or more individual change timings at each of the extracted observation points in a voting space of two or more dimensions having dimensions corresponding to the number of the extracted individual change timings. Vote for each observation point in a voting space whose coordinate axis is the measurement time of the individual change timing, and then vote according to the number of individual change timings extracted based on the coordinate value that received the most votes in the voting space. Detect two or more change timings
A change timing detection device characterized by: 2. The change timing detection device according to claim 1 , wherein the detection unit votes by changing weights according to the reliability of the individual change timing when voting. If the time-series value of the second closest change timing in the past changes in each frame, following the change in the time-series value of the change timing closest in the past to each frame, the movement or brightness of the detection target changes. 3. The change timing detection device according to claim 1 , further comprising a determination unit that determines that the change timing is normal. The change timing detection device according to any one of claims 1 to 3 , wherein the multidimensional time series data is a video. 5. The change timing detection device according to claim 1, wherein the individual change timing is a timing at which the motion of the detection target changes. The measurement unit detects individual change timing candidates at each of the observation points, and measures the average direction of movement of the detection target at the observation point among the temporally consecutive individual change timing candidates. 6. The change timing detection device according to claim 5 , wherein the average direction is classified into two groups depending on the direction. The detection target is a living organism,
7. The change timing detection device according to claim 5 , wherein the change timing is a timing at which exhalation and inhalation of the living body switch. 5. The change timing detection device according to claim 4 , wherein the individual change timing is a timing at which the brightness of the detection target changes. The detection target is a living organism,
The plurality of observation points include a skin area of the living entity in the image,
9. The change timing detection device according to claim 8 , wherein the measurement unit measures the individual change timing based on a change in brightness value at each observation point. A change timing detection method for determining change timing of a detection target based on multidimensional time series data of a space including the detection target, the method comprising:
a measurement step of measuring individual change timing, which is the timing at which the movement or brightness of the detection target changes, at each of a plurality of observation points set for the detection target in the multidimensional time series data;
Two or more individual change timings at each observation point are extracted from each frame of the multidimensional time series data toward the past, and based on the two or more individual change timings at each observation point extracted, the detection target is detected. a detection step of detecting a change timing of the
The detection step includes measuring the measurement times of the two or more individual change timings at each of the extracted observation points in a two-dimensional or more voting space having dimensions corresponding to the number of the extracted individual change timings. Vote for each observation point in a voting space whose coordinate axis is the measurement time of the individual change timing, and then vote according to the number of individual change timings extracted based on the coordinate value that received the most votes in the voting space. Detect two or more change timings
A change timing detection method characterized by the following. A change timing detection program for determining change timing of a detection target based on multidimensional time series data of a space including the detection target, the program comprising:
a measurement process of measuring individual change timing, which is the timing at which the movement or brightness of the detection target changes, at each of a plurality of observation points set for the detection target in the multidimensional time series data;
Two or more individual change timings at each observation point are extracted from each frame of the multidimensional time series data toward the past, and based on the two or more individual change timings at each observation point extracted, the detection target is detected. A detection process that detects the change timing of
The detection process includes measuring the measurement times of the two or more individual change timings at each of the extracted observation points in a voting space of two or more dimensions having dimensions corresponding to the number of the extracted individual change timings. Vote for each observation point in a voting space whose coordinate axis is the measurement time of the individual change timing, and then vote according to the number of individual change timings extracted based on the coordinate value that received the most votes in the voting space. Detect two or more change timings
A change timing detection program characterized by:

Images