WO2020022362A1

WO2020022362A1 - Motion detection device, feature detection device, fluid detection device, motion detection system, motion detection method, program, and recording medium

Info

Publication number: WO2020022362A1
Application number: PCT/JP2019/028948
Authority: WO
Inventors: 航鈴木; 紀孝一戸; 博臣竹市
Original assignee: 国立研究開発法人国立精神・神経医療研究センター; 国立研究開発法人理化学研究所
Priority date: 2018-07-24
Filing date: 2019-07-24
Publication date: 2020-01-30
Also published as: JPWO2020022362A1

Abstract

Provided is a motion detection device with which motion detection can be carried out without increasing the computation cost. This motion detection device is equipped with an image acquisition unit for acquiring a subject image, a vector derivation unit for deriving a motion-related vector from the subject image acquired by the image acquisition unit, and a motion detection unit for carrying out motion detection by tracking a vector derived by the vector derivation unit.

Description

Motion detecting device, characteristic detecting device, fluid detecting device, motion detecting system, motion detecting method, program, and recording medium

The present invention relates to a motion detection device, a characteristic detection device, a fluid detection device, a motion detection system, a motion detection method, a program, and a recording medium.

Optical flow refers to movements in images such as objects, surfaces, and edges caused by the relative movement of the observer (camera) and the outside world. -It is known that it is useful for recognition of distance and movement, control of self-motion, and the like. It is also useful in the field of engineering related to image processing and navigation, and is used in motion detection, object segmentation, collision time estimation, brightness, coding with motion compensation, parallax measurement, and the like.

In recent years, research on the detection of a moving object using this optical flow has been studied. For example, in Patent Document 1, when detecting a moving object, a moving object detection device that has improved the accuracy of detecting a feature point in a dark part is described. Is disclosed.

Japanese Patent Application Laid-Open Publication No. 2015-076633 (published on April 20, 2015)

However, the above-described moving object detection device has a problem that the calculation cost increases.

The object of the present invention is to provide a motion detection device capable of appropriately performing motion detection while suppressing calculation cost.

In order to solve the above problem, a motion detection device according to an aspect of the present invention is a motion detection device that performs motion detection of a target image, and an image acquisition unit that acquires a target image, and the image acquisition unit includes: A vector derivation unit that derives a vector related to motion from the acquired target image, and a motion detection unit that performs motion detection by tracking the vector derived by the vector derivation unit.

In order to solve the above problem, a motion detection system according to an aspect of the present invention is a motion detection system that performs motion detection of a target image, wherein the image acquisition unit that acquires the target image, the image acquisition unit includes: Refer to a vector deriving unit that derives a vector related to motion from the acquired target image, a motion detecting unit that performs motion detection by tracking the vector derived by the vector deriving unit, and a motion detected by the motion detecting unit. And an image generation unit that generates an image relating to the movement, and a display unit that displays the image generated by the image generation unit.

In order to solve the above-described problem, a motion detection method according to one embodiment of the present invention is a motion detection method that performs motion detection of a target image, and includes: an image obtaining step of obtaining a target image; The method includes a vector deriving step of deriving a vector related to motion from the acquired target image, and a motion detecting step of performing motion detection by tracking the vector derived in the vector deriving step.

In order to solve the above-described problem, a characteristic detection device according to one embodiment of the present invention uses the motion detection device, a detection result of the motion detection unit, and a characteristic of an object included in the target image and the object. And a specifying unit that specifies at least one of the characteristics of the movement.

In order to solve the above-described problem, a characteristic detection device according to an aspect of the present invention includes a motion detection device, and a detection unit that detects a fluid contained in the target image using a detection result of the motion detection unit. And

According to one embodiment of the present invention, motion detection can be performed without increasing calculation cost.

FIG. 1 is a block diagram illustrating a main configuration of a motion detection system according to a first embodiment of the present invention. It is a flowchart which shows the flow of a process of a motion detection system. (A) shows a target image for motion detection, (b) shows a curved surface in a three-dimensional space formed by pixel coordinates and pixel values in the target image, (C) shows the vector derived in the target image, and (d) shows the inter-frame difference of the vector of the target image. It is an example of a target image in a moving image that is a target of motion detection. 5 is an image showing the coordinates of a first vector derived from the target image of FIG. 5 is an image showing a line segment connecting target pixels derived from each frame of FIG. 4. FIG. 9 is a block diagram illustrating a main configuration of a motion detection system according to a second embodiment of the present invention. FIG. 11 is a block diagram illustrating a configuration of a computer that can be used as a server or a terminal. FIG. 11 is a block diagram illustrating a main configuration of a motion detection system according to a fourth embodiment of the present invention. It is an example of a target image in a moving image that is a target of motion detection. 11 is an image showing the coordinates of a first vector derived from the target image of FIG. 11 is an image obtained by superimposing the image of FIG. 11 on the target image of FIG. It is an example of a target image in a moving image that is a target of motion detection. 14 is an image showing the coordinates of a first vector derived from the target image of FIG. 14 is an image obtained by superimposing the image of FIG. 14 on the target image of FIG. 11 is an image exemplifying a motion detection result of the conventional optical flow detection method for the target image in FIG. 10.

[Embodiment 1]
Hereinafter, an embodiment of the present invention will be described in detail.

(Motion detection system)
FIG. 1 is a diagram illustrating a schematic configuration of a motion detection system 1 according to the present embodiment. As shown in FIG. 1, the motion detection system 1 includes a motion detection device 10 and a display device 20.

(Motion detection device)
The motion detection device 10 includes an image acquisition unit 11, a vector derivation unit 12, and a motion detection unit 13.

The image acquisition unit 11 acquires, from a moving image captured by an imaging unit (not shown), a target image that is an individual frame of the moving image. The image acquisition unit 11 outputs the acquired target image to the vector derivation unit 12.

The 導出 vector deriving unit 12 sets one of the frames output by the image acquiring unit 11 as a target image, and derives a motion-related vector (first vector) from the target image. The vector deriving unit 12 outputs the derived vector of the target image to the motion detecting unit 13. The vector deriving unit 12 similarly derives a motion-related vector (first vector) in at least one other frame included in the moving image and outputs the vector to the motion detecting unit 13. Here, the at least one other frame is preferably temporally adjacent to the target image (in other words, a frame immediately before the target image or a frame immediately after the target image). However, this does not limit the present embodiment. For example, the at least one other frame may be a frame temporally separated from the target image by a predetermined number of frames. Unless there is confusion, the at least one other frame may also be referred to as a target image. A method of deriving a vector related to motion in the vector deriving unit 12 will be described later.

The motion detector 13 detects the motion of the moving image by tracking the motion-related vector derived by the vector deriver 12. More specifically, the motion detection unit 13 compares the vector (first vector) derived in the target image with the vector (first frame) temporally adjacent to the target image (also referred to as an adjacent frame). A vector (second vector) related to the motion between the target image and the adjacent frame is derived from the difference from the first vector. The motion detection unit 13 detects the motion of the moving image with reference to the vector related to the derived motion. The motion detection unit 13 can also track the motion of the moving image by tracking the second vector over a plurality of frames.

Also, the motion detection unit 13 may be configured to set a search area according to the characteristics of the vector (second vector) derived by the motion detection unit 13 and track the motion vector in the set search area. Good. The search area is an area in a search space for tracking, and may include the direction of motion. The method for setting the search area will be described later.

The motion detection unit 13 may be configured to select a vector related to a specific object from the vectors derived by the vector derivation unit 12 and perform motion detection by tracking the selected vector. Here, the specific object is not limited to the present embodiment, but includes, for example, an object such as a light in a captured image of a car, whose movement can be easily tracked. Further, the method for identifying these specific objects is not limited to the present embodiment. For example, the objects can be identified by using a technique such as pattern matching or edge detection.

(Display device)
The display device 20 includes an image generation unit 21 and a display unit 22.

The image generation unit 21 acquires the vector derived by the vector derivation unit 12, and generates an image related to the vector. Further, the image generation unit 21 acquires a vector related to the motion derived by the motion detection unit 13 and generates an image related to the vector related to the motion (image related to the motion). The image generated by the image generation unit 21 is output to the display unit 22.

The display unit 22 displays the image output from the image generation unit 21.

(Motion detection method)
FIG. 2 is a flowchart illustrating an example of a processing flow of the motion detection system 1.

<Step S1>
The image acquisition unit 11 acquires a target image, which is an individual frame of a moving image, from a moving image to be subjected to motion detection.

<Step S2>
The vector deriving unit 12 derives a first vector that is a vector related to motion from the target image acquired by the image acquiring unit 11.

<Step S3>
The motion detection unit 13 detects the motion of the moving image, that is, the second vector, with reference to the first vector derived by the vector derivation unit 12. Further, as described above, the motion detection unit 13 can track the motion of the moving image by tracking the second vector over a plurality of frames.

<Step S4>
The image generation unit 21 generates an image related to the motion-related vector with reference to the motion-related vector derived by the motion detection unit 13.

<Step S5>
The display unit 22 displays the image generated by the image generation unit 21.

Note that, in the above-described processing flow, the image generation unit 21 replaces the configuration for generating an image related to the vector related to the motion derived by the motion detection unit 13, and replaces the configuration related to the motion derived by the vector derivation unit 12. The configuration may be such that an image related to a vector related to the motion is generated with reference to the obtained vector.

Hereinafter, a more specific motion detection method will be described with reference to examples.

In step S <b> 1 described above, the image acquisition unit 11 acquires a moving image to be subjected to motion detection and a moving image in which the contrast of the moving image is inverted. The image acquisition unit 11 acquires, from these acquired moving images, target images that are individual frames of these moving images.

In step S2 described above, the vector deriving unit 12 determines the number of dots for deriving the first vector related to motion in the target image. The vector deriving unit 12 sets half of the determined number of dots to be randomly located in the target image (first frame) acquired by the image acquiring unit 11 from the moving image. The other half is set so that the image acquisition unit 11 is randomly located in the target image (first frame) acquired from the moving image in which the contrast of the moving image is inverted. The vector deriving unit 12 derives a first vector related to motion in a pixel where a dot is set in the target image (first frame).

In step S3 described above, the motion detection unit 13 derives a second vector by performing time differentiation (inter-frame difference) on the first vector derived by the vector derivation unit 12.

The motion detecting unit 13 repeats the same operation in a frame adjacent to the target image (first frame) (frame immediately after the target image), and derives a second vector in all frames.

After deriving the second vector for all frames, the motion detection unit 13 performs dot matching for each frame. Specifically, the motion detection unit 13 selects an arbitrary dot in the target image (first frame), and searches for a dot having the same vector direction and size as the arbitrary dot in an adjacent frame. Here, the motion detection unit 13 sets an area on a straight line along a vector of the dot from an arbitrary dot in the target image as a starting point, as a search area in an adjacent frame.

The motion detection unit 13 matches the arbitrary dot with a dot detected in an adjacent frame. When a plurality of dots having the same vector direction and size as the arbitrary dot are detected in the adjacent frame, the motion detection unit 13 determines the most Match with a dot located near.

The motion detection unit 13 tracks the motion of the moving image by performing dot matching in all frames. Note that the motion detection unit 13 deletes all dots that have not been matched.

The image generation unit 21 generates an image related to a motion-related vector by visualizing the dots matched by the motion detection unit 13. The display unit 22 displays the image generated by the image generation unit 21.

In the above specific example, the motion detection system 1 does not derive a motion-related vector for all pixels in the target image, but performs a motion-related vector calculation only on randomly set dots. Since the derivation is performed, motion detection can be performed without increasing calculation cost.

(How to derive the vector)
Next, with reference to FIG. 3, a method for deriving the above-described motion-related vector will be described more specifically. 3A and 3B are images showing a method of deriving a vector in a target image, in which FIG. 3A shows a target image for motion detection, and FIG. FIG. 3C shows a curved surface, FIG. 4C shows a vector derived from the target image, and FIG.

(Derivation process 1)
First, in the present embodiment, the vector deriving unit 12 derives a first vector indicating a gradient of a pixel value for at least one of the pixels included in the target image.

More specifically, first, as shown in FIG. 3B, the vector deriving unit 12 generates a two-dimensional plane (XY plane) indicating the coordinates of the pixel of the target image, and a Z indicating the pixel value of each pixel. In a three-dimensional space spanned by axes, a curved surface indicating each pixel value of a target image is considered. That is, the horizontal position, the vertical position, and the pixel value of each pixel of the target image correspond to the X coordinate, the Y coordinate, and the Z coordinate, respectively, in the three-dimensional space. Here, as the pixel value combined as the Z axis, for example, any one of the colors such as a luminance value, a color difference value, and RGB or a combination thereof can be used.

(Derivation process 2)
Next, the vector deriving unit 12 derives a normal vector of a curved surface in the three-dimensional space for each pixel.

Next, the vector deriving unit 12 normalizes the derived normal vector, and projects the normalized normal vector onto the XY plane, as shown in FIG. Is derived. The X component nx and the Y component ny of the first vector derived by the vector deriving unit 12 are represented by the following equations.

Here, I (x, y, t) is the luminance of the pixel located at the coordinates (x, y) at time t. The time t can be read as a frame number (also referred to as Picture Order Count) assigned to each frame. In addition, the partial derivative with respect to x and y can be read as the difference with respect to x and y, respectively.

(Derivation process 3)
Next, as shown in FIG. 3D, the motion detection unit 13 determines the first vector of the target image obtained by the vector derivation unit 12 and the first vector of the frame temporally adjacent to the target image. From the difference with one vector, a vector (also referred to as a second vector) indicating the difference between the two frames is derived. In other words, the second vector is derived by taking the time differential (time difference) of the X component nx and the Y component ny of the first vector.

Here, the detection result of the motion of the motion detection system 1 according to the present embodiment will be described with reference to FIGS.

FIG. 4 is an image showing a target image in a moving image on which motion detection has been performed. (A) to (c) of FIG. 4 are frames that are temporally continuous in the moving image.

FIG. 5 is an image showing a part of the first vector derived from the target image of FIG. FIGS. 5A to 5C are images showing the coordinates of the first vector derived from the target image in FIGS. 4A to 4C, respectively. In the example shown in FIG. 5, the first vector is derived for some but not all of the pixels included in the target image.

FIG. 6 is an image depicting a line segment connecting the target pixel P1 derived from the first frame and the target pixel P2 derived from the second frame in FIG. FIG. 6A shows a line segment derived from FIGS. 5A and 5B, and FIG. 6B shows a line segment derived from FIGS. 5B and 5C. Indicates minutes.

As is clear from FIG. 6, the motion detection system 1 according to the present embodiment can preferably perform motion detection by tracking a vector related to the motion derived by the motion detection unit 13.

(How to set search area)
An example of a specific method of setting a search area in the motion detection unit 13 is as follows.

The motion detection unit 13 sets a search area in a direction along the second vector V1 derived for the target pixel P1 in the target image N1, and sets a frame N2 adjacent to the target image (for example, immediately after the target image). In the search area in the frame (), a pixel P2 corresponding to the target pixel is specified, and a second vector V2 is calculated for the specified pixel. The motion detection unit 13 performs tracking of the second vector by performing the above operation.

Another example of a specific method of setting the search area in the motion detection unit 13 is as follows.

The motion detector 13 calculates a first vector in each of the target image N1 and the frame N2. The motion detection unit 13 derives a second vector from target pixels in the target image N1 and the frame N2. The motion detection unit 13 sets a search area in a direction along the second vector V1 derived for the target pixel P1 that is the target of vector tracking. The motion detection unit 13 specifies the second vector V2 derived for the target pixel P2 corresponding to the target pixel P1 in the search area in the frame N2, and tracks the vector.

Here, as an example, the search area may be set as a linear area along the second vector V1 starting from the target pixel P1 of the target image, or the linear area may be set as the second area. The vector may be set as a band-shaped area that is thickened along the vertical direction, or may be set as an area having another shape. Further, the search area may be set using, for example, a function value range in which the second vector is defined as a domain. The search area is not limited to a continuous, connected, or linear one, and may be discontinuous, non-connected, or non-linear.

Note that the coordinates of the target pixel P1 in the target image and the coordinates of the target pixel P2 in the adjacent frame may generally be different, but this is not a limitation of the present embodiment. The first vector may be derived between adjacent frames.

対象 In addition, the target pixel in the adjacent frame can be specified based on pixel characteristics (for example, luminance, color difference, difference in pixel value between adjacent pixels, and the like) of the target pixel.

As described above, since the moving image detection system according to the present invention tracks the first vector after setting the search area, the motion detection can be performed without increasing the calculation cost.

(Appendix on machine learning)
The specific configuration of the classification and learning processing for motion detection by the motion detection unit 13 is not limited to the present embodiment. For example, any one of the following machine learning methods or a combination thereof may be used. Can be.

・ Support Vector Machine (SVM)
・ Clustering
・ Inductive Logic Programming (ILP)
・ Genetic Algorithm (GP)
・ Bayesian Network (BN)
・ Neural Network (NN)
When a neural network is used, 3D data may be processed and used in advance for input to the neural network. For such processing, in addition to one-dimensional arrangement or multi-dimensional arrangement of data, for example, a technique such as data augmentation (Data Augmentation) can be used.

In the case of using a neural network, a convolutional neural network (CNN: Convolutional Neural Network) including a convolution process may be used. More specifically, a convolution layer for performing a convolution operation is provided as one or a plurality of layers (layers) included in the neural network, and a filter operation (product-sum operation) is performed on input data input to the layer. It may be configured. When performing the filter operation, processing such as padding may be used in combination, or an appropriately set stride width may be employed.

多層 Also, as the neural network, a multilayer or super multilayer neural network having several tens to thousands of layers may be used.

The specific configuration of the classification and learning processing for motion detection by the motion detection unit 13 is not limited to the present embodiment. For example, the machine learning used in the above processing may be supervised learning. And it may be unsupervised learning.

[Embodiment 2]
Another embodiment of the present invention will be described below. For convenience of description, members having the same functions as those described in the above embodiment are denoted by the same reference numerals, and description thereof will not be repeated.

The image acquisition unit 11, the vector derivation unit 12, the motion detection unit 13, the image generation unit 21, and the display unit 22 are provided in separate devices, and the exchange of information between these devices is performed by wired or wireless communication. It is good also as a structure which performs.

Specifically, as shown in FIG. 7, the motion detection system 1a includes a terminal device (imaging device) 10a, a display device 20, and a server (motion detection device) 30. , The image acquisition unit 11 includes a communication unit 14, the display device 20 includes an image generation unit 21, a display unit 22, and a communication unit 23. The server 30 includes a communication unit 31, a vector derivation unit 32, and a motion detection unit. A configuration including the unit 33 may be adopted. Here, the operations of the vector derivation unit 32 and the motion detection unit 33 are the same as the operations of the vector derivation unit 12 and the motion detection unit 13 described in the first embodiment, respectively.

The server may be configured to manage the motion detection results for a plurality of image sets together with the identification numbers of the image sets. As a more specific example, when the movement of a plurality of subjects is detected by the present system, a configuration may be employed in which an image set (video data) is managed for each subject ID. Further, time information indicating the time when the image was acquired may be further linked.

The server may further include a learning unit that performs machine learning based on the motion detection result. Here, as an example, the learning unit receives the motion information (or the first vector, the second vector, or the like) detected by the motion detection unit 13, the ID of the subject, and time information as inputs. It functions as a learning device that outputs classification information on the movement of the subject's body.

As the learning unit, various methods can be used as described in the first embodiment (additional notes on machine learning).

[Embodiment 3]
The control blocks (the image acquisition unit 11, the vector derivation unit 12, and the motion detection unit 13) of the motion detection device 10 may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, It may be realized by software. In the latter case, each of the motion detection device 10, the display device 20, an output device 20b described later (see Embodiment 4), and the server 30 is configured using a computer (electronic computer) as shown in FIG. Can be. The motion detection device 10, the display device 20, the output device 20b, and the server 30 may be configured as separate devices, respectively, or at least a part of the devices may be configured as an integrated device. May be. For example, the motion detection device 10 and the output device 20b may be configured as an integrated device.

FIG. 8 is a block diagram illustrating the configuration of a computer 910 that can be used as the motion detection device 10, the display device 20, the output device 20b, or the server 30. The computer 910 includes an arithmetic device 912, a main storage device 913, an auxiliary storage device 914, an input / output interface 915, and a communication interface 916 connected to each other via a bus 911. The arithmetic device 912, the main storage device 913, and the auxiliary storage device 914 may be storages such as a CPU, a RAM (random access memory), a hard disk drive, and a flash memory, respectively. The input / output interface 915 is connected to an input device 920 for the user to input various information to the computer 910 and an output device 930 for the computer 910 to output various information to the user. The input device 920 and the output device 930 may be built in the computer 910, or may be connected (externally attached) to the computer 910. For example, the input device 920 may be a keyboard, a mouse, a touch sensor, and the like, and the output device 930 may be a display, a printer, a speaker, and the like. Further, a device having both functions of the input device 920 and the output device 930, such as a touch panel in which a touch sensor and a display are integrated, may be applied. The communication interface 916 is an interface for the computer 910 to communicate with an external device.

Various programs for operating the computer 910 as the motion detection device 10, the display device 20, the output device 20b, or the server 30 are stored in the auxiliary storage device 914. Then, the arithmetic unit 912 expands the program stored in the auxiliary storage device 914 on the main storage device 913 and executes an instruction included in the program, thereby causing the computer 910 to operate the motion detection device 10 and the display device. 20, the output device 20b, or each unit included in the server 30. Note that a recording medium for recording information such as a program provided in the auxiliary storage device 914 may be a computer-readable “temporary tangible medium”, such as a tape, disk, card, semiconductor memory, or programmable logic. It may be a circuit or the like. Further, the main storage device 913 may be omitted as long as the computer can execute the program recorded on the recording medium without expanding the program on the main storage device 913. Note that each of the above devices (the arithmetic device 912, the main storage device 913, the auxiliary storage device 914, the input / output interface 915, the communication interface 916, the input device 920, and the output device 930) may be one each. There may be more than one.

The program may be obtained from outside the computer 910. In this case, the program may be obtained via an arbitrary transmission medium (such as a communication network or a broadcast wave). The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

[Embodiment 4]
Another embodiment of the present invention will be described below. For convenience of description, members having the same functions as those described in the above embodiment are denoted by the same reference numerals, and description thereof will not be repeated.

In each of the above-described embodiments and the present embodiment, the motion of the target image to be detected includes not only the motion of a rigid body such as an automobile, a person, an animal, and a ball, but also a gas (for example, smoke) and a liquid (for example, water). , Oil). As described above, the object to be subjected to motion detection in each of the above embodiments and the present embodiment includes a rigid body and a fluid. In this embodiment, a case will be mainly described in which a target image contains a fluid such as a gas and a liquid, and the detection system detects the movement of the fluid.

FIG. 9 is a block diagram showing a schematic configuration of a motion detection system 1b (an example of a characteristic detection device and a fluid detection device) according to this embodiment. As shown in FIG. 9, the motion detection system 1b includes a motion detection device 10 and an output device 20b. The operations of the image acquisition unit 11, the vector derivation unit 12, and the motion detection unit 13 of the motion detection device 10 are the same as the operations of the image acquisition unit 11, the vector derivation unit 12, and the motion detection unit 13 described in the first embodiment. is there.

The output device 20b includes an image generation unit 21, a display unit 22, a specification unit 24, a detection unit 25, and an output unit 26. The operations of the image generation unit 21 and the display unit 22 of the output device 20b are the same as the operations of the image generation unit 21 and the display unit 22 described in the first embodiment.

The identification unit 24 analyzes the detection result of the motion detection unit 13 and identifies at least one of the characteristics of the object included in the target image and the characteristics of the motion of the object. The object included in the target image may be a rigid body such as a person, an animal, a car, and a ball, or may be a fluid such as a gas or a liquid. The characteristics of an object refer to the characteristic properties or appearance of the object. The property of the object is, for example, the expression of a person, clothes, the shape of the object, or the viscosity of the object. The characteristic of the movement of the object is, for example, a moving direction of the object or a moving speed of the object predicted in the future.

For example, when specifying a human expression as a characteristic, the specifying unit 24 specifies a human expression and / or a change in the facial expression based on the image analysis result of the target image and the detection result of the motion detection unit 13. When specifying clothing, the specifying unit 24 specifies the shape of the clothing and / or the movement of the clothing based on the image analysis result of the target image and the detection result of the motion detection unit 13. In addition, the specifying unit 24 may specify the material of the clothes from the specified shape of the clothes and / or the movement of the clothes.

In addition, when specifying the predicted motion of the object as a characteristic, the specifying unit 24 analyzes the motion detected by the motion detection unit 13 and predicts the motion of the object included in the target image. More specifically, for example, the specifying unit 24 may use the moving direction and the moving speed indicated by the second vector derived by the motion detecting unit 13 as the predicted moving direction and the moving speed of the object. Further, for example, the specifying unit 24 may predict the motion of the object by analyzing a change with time of the moving direction and the moving speed indicated by the second vector derived by the motion detecting unit 13. .

When measuring the viscosity, the specifying unit 24 measures the viscosity of the fluid using the detection result of the motion detecting unit 13. The object whose viscosity is to be measured is, for example, cement or ice cream. For example, the identification unit 24 acquires the second vector derived by the motion detection unit 13, and measures the viscosity of the object based on the moving speed of the object indicated by the second vector.

The detection unit 25 detects the fluid included in the target image using the detection result of the motion detection unit 13. For example, the detection unit 25 may analyze a detection result of the motion detection unit 13 and specify a region where the motion is detected as a region where the object is located.

The output unit 26 outputs information indicating the specification result of the specification unit 24 and the measurement result of the detection unit 25. The output of information by the output unit 26 may be performed, for example, by outputting data to an externally connected device, or may be performed by transmitting data to another device via a communication network. Is also good. The output of the information may be performed, for example, by outputting data representing an image to the display unit, or the information may be output by sound or voice.

FIG. 10 is a diagram illustrating a target image in a moving image on which motion detection has been performed. In the example of FIG. 10, a moving image obtained by capturing a gas such as smoke is used. (A) to (d) of FIG. 10 are frames included in a moving image. Specifically, FIG. 10B is a frame after a fixed time (for example, 20 seconds) has elapsed from the frame of FIG. 10A. (C) of FIG. 10 is a frame after a lapse of a fixed time (for example, 20 seconds) from the frame of (b) of FIG. 10. (D) of FIG. 10 is a frame after a predetermined time (for example, 20 seconds) has elapsed from the frame of (c) of FIG. 10. As shown in FIGS. 10A to 10D, the appearance of a gas such as a captured smoke gradually changes over time.

FIG. 11 is an image showing the coordinates of the first vector derived from the target image of FIG. (A) to (d) of FIG. 11 are images showing the coordinates of the first vector derived in the target images of (a) to (d) of FIG. 10, respectively. In this example, the vector deriving unit 12 derives a first vector for all pixels included in the target image, performs a dot matching process between frames, and determines a first vector that succeeds in matching. Extract. That is, in the example of FIG. 11, the coordinates of the first vector for which matching has succeeded are displayed, and the coordinates of the first vector for which matching has not succeeded are not displayed. Note that the pixels for which the vector derivation unit 12 derives the first vector need not be all the pixels included in the target image. The vector deriving unit 12 may derive the first vector from some pixels included in the target image.

In this embodiment, the image generation unit 21 determines, as a moving image corresponding to the moving image that is the target image of the motion detection, a moving image (a moving image representing the temporal change of the coordinates of the first vector derived by the vector deriving unit 12). Hereinafter, referred to as a “motion detection image”). FIGS. 11A to 11D show examples of frames included in this moving image. This moving image allows the user to grasp the movement of the fluid such as smoke.

FIG. 12 is an image obtained by superimposing the image of FIG. 11 on the target image of FIG. 12A to 12D are obtained by superimposing the images of FIGS. 11A to 11D on the target images of FIGS. 10A to 10D, respectively.

FIG. 16 is an image exemplifying a motion detection result of the target image of FIG. 10 by a conventional optical flow detection method. (A) to (d) of FIG. 16 are images showing detection results of motion from the target images of (a) to (d) of FIG. 10, respectively. As is apparent from a comparison between FIG. 16 and FIG. 11, in the present embodiment, the accuracy of motion detection of a fluid such as smoke is improved.

FIG. 13 is a diagram illustrating a target image in another moving image on which motion detection has been performed. (A) to (d) of FIG. 13 are frames included in a moving image. The relationship between the frames shown by (a) to (d) in each drawing is the same as that in FIG. 10 described above.

FIG. 14 is an image showing the coordinates of the first vector derived from the target image of FIG. FIG. 15 is an image in which an image representing the coordinates of the first vector derived from each target image is superimposed on the target image in FIG.

By the way, when an image obtained by photographing a fluid such as a gas or a liquid is used as a target image, a feature point may not be apparent in the target image. When the feature points do not appear as described above, it has been difficult in some cases to detect the motion from the target image using the conventional optical flow technique. On the other hand, in the present embodiment, even when the feature points do not appear in the target image, the motion detection unit 13 can easily track the motion of the moving image. Thus, according to this embodiment, it is possible to extract and visualize the motion of a fluid such as smoke or water, which is perceived by a person.

The output device 20b or a device (not shown) that has obtained data output from the output device 20b provides various services to the user using the detected movement. The services provided are, for example, fluid viscosity measurement, automatic driving control, monitoring services for children and the elderly, or evacuation support services in the event of a disaster. For example, in the case of measuring the viscosity of a fluid, the output device 20b outputs information indicating the viscosity of the fluid measured by the specifying unit 24.

When the automatic driving control service is provided, the output device 20b analyzes, for example, the movement of an object (a person, a bicycle, a car, a ball, etc.) predicted by the identification unit 24, and a dangerous situation (a collision or the like) is caused. When it is estimated that the warning message is output, a warning sound or a warning message may be output, or the warning message may be displayed on the display unit 22.

In addition, when a watching service is provided, a protected person (child, old man, pet, etc.) to be monitored is photographed, and motion detection is performed from the photographed moving image. The output device 20b monitors the protected person by, for example, analyzing the motion of the person detected by the motion detecting unit 13 or analyzing the expression or the like of the protected person specified by the specifying unit 24. . The output device 20b may output a warning sound or a warning message or display a warning message on the display unit 22 when it is estimated from the analysis result that the protected person is in a dangerous situation. In this case, the captured moving image is the target image for motion detection, but it is not necessary to disclose this moving image. Therefore, the detection system 1b can provide a watching service while protecting the privacy of the protected person.

In the first embodiment, as an example of the dot matching process performed by the motion detection unit 13, an arbitrary dot is selected in the target image (first frame), and the direction of the arbitrary dot and the vector in the adjacent frame is selected. And a process of searching for dots having the same size. The process of dot matching is not limited to that described in the above embodiment. In the dot matching process, for example, the motion detection unit 13 selects an arbitrary dot in the target image, and in a frame adjacent to the frame, a difference between the direction and the size of the arbitrary vector is determined under a predetermined condition (for example, a predetermined value). The search may be performed by searching for a dot that satisfies the following condition.

[Summary]
A motion detection device according to an aspect of the present invention is a motion detection device that performs motion detection of a target image, and includes an image acquisition unit that acquires a target image, A vector deriving unit for deriving a vector, and a motion detecting unit for performing motion detection by tracking the vector derived by the vector deriving unit are provided.

According to the configuration described above, motion detection can be performed without increasing the calculation cost.

In the motion detection device according to one aspect of the present invention, the motion detection unit sets a search area according to the characteristics of the vector detected by the motion detection unit, and sets the search area in the set search area. Perform tracking.

In the motion detection device according to one aspect of the present invention, the motion detection unit sets the search area in a direction along a vector detected by the motion detection unit.

In the motion detection device according to one aspect of the present invention, the vector deriving unit derives a first vector indicating a gradient of a pixel value for at least one of the pixels included in the target image, and Is derived as a vector related to the motion.

According to the above configuration, a suitable vector can be referred to when performing motion detection.

In the motion detection device according to one aspect of the present invention, the vector derivation unit may determine a horizontal position, a vertical position, and a pixel value of each pixel of the target image by an X coordinate, a Y coordinate, and a Z coordinate, respectively. The first vector is derived by projecting the normal vector of the curved surface to the XY plane.

In the motion detection device according to an aspect of the present invention, the vector deriving unit derives the first vector by normalizing the normal vector and projecting the normal vector onto the XY plane.

A motion detection device according to an aspect of the present invention is a motion detection system that performs motion detection of a target image, and includes an image acquisition unit that acquires a target image, and a motion-related image that is acquired by the image acquisition unit. A vector deriving unit that derives a vector, a motion detecting unit that performs motion detection by tracking the vector derived by the vector deriving unit, and a motion detected by the motion detecting unit. The image processing apparatus includes an image generation unit that generates the image, and a display unit that displays the image generated by the image generation unit.

A motion detection method according to one aspect of the present invention is a motion detection method for detecting a motion of a target image, the method including: an image obtaining step of obtaining a target image; and a motion-related method from the target image obtained in the image obtaining step. The method includes a vector deriving step of deriving a vector, and a motion detecting step of performing motion detection by tracking the vector derived in the vector deriving step.

The characteristic detection device according to one aspect of the present invention uses the motion detection device and the detection result of the motion detection unit, and uses at least one of a characteristic of an object included in the target image and a characteristic of movement of the object. And a specifying unit that specifies

According to the configuration described above, it is possible to specify at least one of the characteristics of the object and the characteristics of the motion of the object without increasing the calculation cost.

特性 In the characteristic detecting device according to one aspect of the present invention, the specifying unit predicts a motion of the object by using a detection result of the motion detecting unit.

According to the above configuration, the motion of the object can be predicted without increasing the calculation cost.

In the characteristic detecting device according to one aspect of the present invention, the target image includes a fluid image, and the specifying unit specifies at least one of the characteristic of the fluid and the characteristic of the movement of the fluid.

特性 In the characteristic detecting device according to one aspect of the present invention, the specifying unit specifies the viscosity of the fluid using a detection result of the motion detecting unit.

According to the above configuration, the viscosity of the object can be measured without increasing the calculation cost.

流体 A fluid detection device according to an aspect of the present invention includes the motion detection device, and a detection unit that detects a fluid contained in the target image by using a detection result of the motion detection unit.

According to the configuration described above, the fluid can be detected without increasing the calculation cost.

The motion detecting device, the characteristic detecting device, and the fluid detecting device according to the respective aspects of the present invention may be each realized by a computer, and in this case, the computer may be implemented by the motion detecting device, the characteristic detecting device, or the fluid detecting device. A control program that causes the computer to implement the motion detecting device, the characteristic detecting device, or the fluid detecting device by operating as each unit included in the device, and a computer-readable recording medium that records the control program are also included in the scope of the present invention. enter.

REFERENCE SIGNS LIST 1 motion detection system 10 motion detection device 11 image acquisition unit 12 vector derivation unit 13 motion detection unit 20 display device 21 image generation unit 22 display unit 23 communication unit 24 identification unit 25 detection unit 26 output unit

Claims

A motion detection device that performs motion detection of a target image,
An image acquisition unit that acquires a target image;
A vector deriving unit that derives a vector related to motion from the target image acquired by the image acquiring unit,
A motion detection device comprising: a motion detection unit that performs motion detection by tracking a vector derived by the vector derivation unit.
The motion detection unit includes:
Set a search area according to the characteristics of the vector detected by the motion detection unit,
2. The motion detection device according to claim 1, wherein a vector related to the motion is tracked in a set search area.
The motion detection device according to claim 2, wherein the motion detection unit sets the search area in a direction along a vector detected by the motion detection unit.
The vector derivation unit:
For at least one pixel included in the target image,
Derive a first vector indicating the gradient of the pixel value;
4. The motion detection device according to claim 1, wherein a second vector indicating a difference between frames of the first vector is derived as a vector related to the motion.
The vector derivation unit:
The normal vector of a curved surface having the horizontal position, vertical position, and pixel value of each pixel of the target image as the X coordinate, Y coordinate, and Z coordinate, respectively, is projected on the XY plane, thereby The motion detection device according to claim 4, wherein a vector of 1 is derived.
The vector derivation unit:
The motion detection device according to claim 5, wherein the first vector is derived by normalizing the normal vector and projecting the normal vector onto the XY plane.
A motion detection system that detects motion of a target image,
An image acquisition unit that acquires a target image;
A vector deriving unit that derives a vector related to motion from the target image acquired by the image acquiring unit,
A motion detection unit that performs motion detection by tracking the vector derived by the vector derivation unit,
With reference to the motion detected by the motion detection unit, an image generation unit that generates an image related to the motion,
A motion detection system comprising: a display unit that displays the image generated by the image generation unit.
A motion detection method for detecting motion of a target image,
An image acquisition step of acquiring a target image;
A vector deriving step of deriving a motion-related vector from the target image acquired in the image acquiring step,
A motion detecting step of performing motion detection by tracking the vector derived in the vector deriving step.
A program for causing a computer to function as the motion detection device according to claim 1, wherein the program causes a computer to function as the image acquisition unit, the vector derivation unit, and the motion detection unit.
A computer-readable recording medium on which the program according to claim 9 is recorded.
A motion detection device according to any one of claims 1 to 6,
A characteristic detecting device comprising: a specifying unit that specifies at least one of a characteristic of an object included in the target image and a characteristic of a motion of the object by using a detection result of the motion detecting unit.
The characteristic detecting device according to claim 11, wherein the specifying unit predicts a motion of the object using a detection result of the motion detecting unit.
The target image includes an image of a fluid,
13. The characteristic detecting device according to claim 11, wherein the specifying unit specifies at least one of a characteristic of the fluid and a characteristic of movement of the fluid.
The characteristic detecting device according to claim 13, wherein the specifying unit specifies the viscosity of the fluid using a detection result of the movement detecting unit.
A motion detection device according to any one of claims 1 to 6,
A detection unit configured to detect a fluid contained in the target image using a detection result of the motion detection unit.
A program for causing a computer to function as the characteristic detection device according to claim 11, wherein the program causes a computer to function as the image acquisition unit, the vector derivation unit, the motion detection unit, and the identification unit.
A computer-readable recording medium on which the program according to claim 16 is recorded.
A program for causing a computer to function as the fluid detection device according to claim 15, wherein the program causes the computer to function as the image acquisition unit, the vector derivation unit, the motion detection unit, and the detection unit.
A computer-readable recording medium on which the program according to claim 18 is recorded.