JP2020139934A - Video distance calculation device and video distance calculation program - Google Patents

Abstract

To calculate the distance to a target object by using a video regardless of the moving direction of the video.SOLUTION: A video distance calculation device 100 includes: optical flow extraction means 104a for extracting the optical flow of M number of target objects in an image of the time T in a video taken by a camera 200; optical flow value calculation means 104b for calculating the size of the optical flow as a value qm (m=1,2, (omitted), M); and distance calculation means 104 for calculating the constants a, b as a=ZL exp((μ/(γ-μ))log(ZL/ZN)) and b=(1/(μ-γ))log(ZL/ZN), in which μ denotes the smallest value and γ denotes the largest value of the value qm of the optical flow, ZN is the shortest distance and ZL is the largest distance of the distances from M number of target objects to the camera 200, and calculating a distance Zm (m=1,2,(omitted), M), which is the distance from M number of target objects to the camera 200, by Zm=a exp(bqm).SELECTED DRAWING: Figure 1

Description

The present invention relates to a moving image distance calculation device and a moving image distance calculation program. More specifically, the present invention uses a moving image of an object to calculate the distance from the object reflected in the moving image to the camera. The present invention relates to an image distance calculation device and a moving image distance calculation program.

In recent years, cameras for photographing the outside world are often installed on moving objects such as vehicles and drones. Recently, there is a demand not only to take a picture of the outside world with a camera, but also to acquire information on the surrounding distance that can be used for automatic driving of a vehicle or the like based on the taken moving image.

A method of photographing an object with a camera and calculating the distance from the object to the camera based on the captured moving image has already been proposed (see, for example, Patent Document 1 and Patent Document 2). The method proposed in Patent Document 1 is referred to as an AMP (Accumulated-Motion-Parallax) method, and the method proposed in Patent Document 2 is referred to as an FMP (Frontward-Motion-Parallax) method.

The AMP method is a method of calculating the distance from an object to a camera by using a moving image taken by a camera moving in the lateral direction. The FMP method is a method of calculating the distance from an object to a camera by using a moving image taken by a camera moving forward or backward. By using the AMP method or the FMP method, the distance from the photographed object to the camera can be calculated based on the moving image captured by one camera.

JP-A-2018-40789 Japanese Patent Application No. 2017-235198

However, since the AMP method is characterized in that the distance to the object is calculated using the moving image taken by the camera that moves in the lateral direction, the moving image taken by the camera that does not move in the lateral direction is used. Has a problem that it is difficult to find the distance to the object. Further, when calculating the distance from the object to the camera by the AMP method, the object needs to be stationary. Therefore, when the object reflected in the captured moving image is a moving object, there is a problem that it is difficult to obtain the distance from the object to the camera.

Further, since the FMP method is characterized in that the distance from the object to the camera is calculated by using the moving image taken by the camera moving forward or backward, the camera moving in the lateral direction or the oblique direction There is a problem that it is difficult to obtain the distance from the object to the camera from the moving image taken by the camera moving to.

The present invention has been made in view of the above problems, and the distance from the object to the camera is determined by using a moving image of the object regardless of the moving state or the moving direction of the camera that captures the object. An object of the present invention is to provide a moving image distance calculation device and a moving image distance calculation program that can be calculated.

In order to solve the above problem, the moving image distance calculation device according to the present invention uses a moving image of a camera that has taken M (M ≧ 3) objects and reflects the moving image on the image at time t of the moving image. The magnitudes of the optical flow extraction means for extracting M optical flows corresponding to each pixel from the M pixels of the object and the magnitudes of the M optical flows extracted by the optical flow extraction means. The optical flow value calculating means for calculating the optical flow value q _m (m = 1, 2, ..., M) and the M optical flow values q calculated by the optical flow value calculating means. Of _m , the value with the smallest optical flow value is μ, the value with the largest optical flow value is γ, and the closest distance among the distances from the M objects to the camera. Z _N and the farthest distance Z _L are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
The distance Z _m is the constant a and the constant b, where each distance from the M objects to the camera is Z _m (m = 1, 2, ..., M). And based on the M values of the optical flow q _m .
Z _m = a · exp (bq _m )
It is characterized by having a distance calculation means calculated by.

Further, the moving image distance calculation program according to the present invention uses moving images of a camera that has captured M (M ≧ 3) objects, from the M objects reflected in the moving image to the camera. It is a program for calculating the moving image distance of the moving image distance calculation device that calculates the distance of the moving image, and corresponds to each pixel from the pixels of the M objects displayed in the image at time t of the moving image on the computer. The size of each of the optical flow extraction function for extracting M optical flows and the M optical flows extracted by the optical flow extraction function is set to the optical flow value q _m (m = 1, 2, ... Of the optical flow value calculation function calculated as M) and the M optical flow values q _m calculated by the optical flow value calculation function, the value with the smallest optical flow value is defined as μ. _Let γ be the value having the largest optical flow value, and measure in advance the shortest distance Z _N and the farthest distance Z _L among the respective distances from the M objects to the camera. The constant a and the constant b
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
The distance Z _m from each of the M objects to the camera is Z _m (m = 1, 2, ..., M), and the distance Z _m is the constant a and the constant b. And based on the M values of the optical flow q _m .
Z _m = a · exp (bq _m )
It is characterized by realizing a distance calculation function that is calculated by.

Further, the moving image distance calculation device according to the present invention uses the moving images of a camera that has taken M objects (M ≧ 3) to obtain the optical flow of all the pixels in the image at time t of the moving images. All-pixel optical flow value calculated by calculating the respective sizes of the optical flow of all the pixels extracted by the all-pixel optical flow extraction means and the all-pixel optical flow extraction means as the optical flow value for each pixel. A region dividing means for dividing the image at time t into K (K ≧ M) regions by applying the mean-shift method to the calculation means and the image at time t, and the region dividing means. Of the K regions divided by, M regions including pixels in which the object is reflected in the image at the time t are extracted, and the optical flow of all the pixels in the region is extracted for each region. An optical flow value calculation means for each region that calculates the optical flow value q _m (m = 1, 2, ..., M) corresponding to the M objects by calculating the average of the values of. Of the M optical flow values q _m calculated by the region-specific optical flow value calculating means, the value with the smallest optical flow value is μ, and the value with the largest optical flow value is γ. Then, the shortest distance Z _N and the farthest distance Z _L among the respective distances from the M objects to the camera are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
The distance Z _m is the constant a and the constant b, where each distance from the M objects to the camera is Z _m (m = 1, 2, ..., M). And based on the M values of the optical flow q _m .
Z _m = a · exp (bq _m )
It is characterized by having a distance calculation means calculated by.

Further, the moving image distance calculation program according to the present invention uses moving images of a camera that has captured M (M ≧ 3) objects, from the M objects reflected in the moving image to the camera. It is a moving image distance calculation program of the moving image distance calculation device that calculates the distance of the moving image, and has an all-pixel optical flow extraction function that causes a computer to extract the optical flow of all the pixels in the image at time t of the moving image. The all-pixel optical flow value calculation function that calculates the respective magnitudes of the optical flow of all the pixels extracted by the all-pixel optical flow extraction function as the optical flow value for each pixel, and the image at the time t On the other hand, by applying the mean-shift method, a region division function for dividing the image at time t into K regions (K ≧ M) and a region division function for K regions divided by the region division function. Among them, M regions including the pixels in which the object is reflected are extracted from the image at the time t, and the average of the optical flow values of all the pixels in the region is calculated for each region. A region-specific optical flow value calculation function for calculating each optical flow value q _m (m = 1, 2, ..., M) corresponding to the M objects, and a region-specific optical flow value calculation function. Of the M optical flow values q _m calculated by the above, the value with the smallest optical flow value is μ, the value with the largest optical flow value is γ, and from the M objects. Of the respective distances to the camera, the shortest distance Z _N and the farthest distance Z _L are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
The distance Z _m from each of the M objects to the camera is Z _m (m = 1, 2, ..., M), and the distance Z _m is the constant a and the constant b. And based on the M values of the optical flow q _m .
Z _m = a · exp (bq _m )
It is characterized by realizing a distance calculation function that is calculated by.

The process of extracting an optical flow using a moving image and the process of dividing an area by applying the mean-shift method to an image are called Open CV (Open Source Computer Vision Library), which is open to the public. This is achieved by using the source computer vision library.

Further, the optical flow extracted by the optical flow extraction means or the all-pixel optical flow extraction means is obtained as a vector. Therefore, the optical flow value calculated by the optical flow value calculating means or the all-pixel optical flow value calculating means means the absolute value of the vector of the optical flow. For example, when the vector is (V1, V2), the optical flow value can be calculated by obtaining the square root of the value of V1 ² + V2 ² .

Further, in the moving image distance calculation device described above, the optical flow value calculating means calculates the sum of the sizes of the M optical flows extracted by the optical flow extracting means, and the size of each optical flow. The magnitude of each normalized optical flow obtained by dividing the sum by the sum is taken as the optical flow value q _m (m = 1, 2, ..., M). You may.

Further, in the moving image distance calculation device described above, the all-pixel optical flow value calculating means calculates the sum of the sizes of the optical flow of all the pixels extracted by the all-pixel optical flow extracting means, and each of them. The size of the normalized optical flow for each pixel, which is obtained by dividing the size of the optical flow of the pixels by the sum, may be used as the value of the optical flow for each pixel.

Further, in the above-mentioned moving image distance calculation program, in the optical flow value calculation function, the computer is made to calculate the sum of the sizes of the M optical flows extracted by the optical flow extraction function, respectively. The value of each optical flow obtained by dividing the magnitude of the optical flow by the sum is calculated as the value of the optical flow q _m (m = 1, 2, ..., M). It may be.

Further, in the moving image distance calculation program described above, in the all-pixel optical flow value calculation function, the sum of the magnitudes of the optical flow of all the pixels extracted by the all-pixel optical flow extraction function is sent to the computer. The size of the normalized optical flow for each pixel, which is calculated and obtained by dividing the size of the optical flow for each pixel by the sum, is used as the value of the optical flow for each pixel. There may be.

Further, in the moving image distance calculation device described above, the M elements are the number of pixels of the image at time t in the moving image, and the distance calculating means has the pixels for each pixel of the image at time t. The distance Z _m from the object reflected in the camera to the camera may be calculated.

Further, in the moving image distance calculation program described above, the M elements are the number of pixels of the image at time t in the moving image, and in the distance calculation function, all the pixels of the image at time t are displayed on the computer. The distance Z _m from the object reflected in the pixel to the camera may be calculated for each time.

According to the moving image distance calculation device and the moving image distance calculation program according to the present invention, the moving image obtained by photographing the object is used from the object regardless of the moving state or the moving direction of the camera that captured the object. It becomes possible to calculate the distance to the camera.

Further, according to the moving image distance calculation device and the moving image distance calculation program according to the present invention, the magnitude of each normalized optical flow is set to the optical flow value q _m (m = 1, 2, ... By using it as M), it becomes possible to accurately calculate the distance from the object to the camera.

It is a block diagram which showed the schematic structure of the moving image distance calculation apparatus which concerns on embodiment. It is a flowchart which showed the process which CPU of the moving image distance calculation apparatus which concerns on embodiment calculate the distance to an object. It is a figure which showed the image of the time t of the moving image which photographed the object (person group). It is a figure which showed the state which extracted the optical flow of all the pixels based on the image shown in FIG. It is a figure which showed the state which performed the region division by applying the mean-shift method to the image shown in FIG. The average of the optical flow of each region divided by the mean-shift method is calculated, and the average direction of the optical flow and the average magnitude of the optical flow values are determined by the line extending from the center of each region (white circle P). It is a figure shown by the direction and length of a minute L. It is a figure which showed the geometric model for demonstrating the method of finding the distance from an object to a camera based on dynamic parallax. It is a figure which showed the state of the image shown in FIG. 3 three-dimensionally from a different viewpoint. It is a figure which showed the state of a city three-dimensionally based on the position information acquired from the moving image taken from the sky. It is a figure which acquired the distance information in front of a vehicle by using the moving image which took the front of the traveling vehicle with a camera, and showed the state of the front of a vehicle three-dimensionally. It is a figure which showed the situation in a room three-dimensionally by installing a camera on a robot moving in a room, acquiring distance information using a moving image taken by the camera.

Hereinafter, an example of the moving image distance calculation device according to the present invention will be shown and described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a moving image distance calculation device. The moving image distance calculation device 100 includes a recording unit 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a CPU (Central Processing Unit: a computer, an optical flow extraction means, an optical flow value calculation means, and the like. It has a distance calculation means, an all-pixel optical flow extraction means, an all-pixel optical flow value calculation means, a region division means, and an area-specific optical flow value calculation means) 104.

A camera 200 is connected to the moving image distance calculation device 100. By using the camera 200, it is possible to capture the surrounding state as a moving image. Cameras can be mounted on vehicles, airplanes, drones, etc., for example.

The camera 200 is provided with a solid-state image sensor such as a CCD image sensor or a CMOS image sensor. The moving image taken by the camera 200 is recorded in the recording unit 101. A monitor 210 is connected to the moving image distance calculation device 100.

A moving image taken by the camera 200 is recorded in the recording unit 101. More specifically, the moving image taken by the camera 200 is recorded in the recording unit 101 as digital data in which a plurality of frame images are recorded in time series. For example, consider the case where a moving image for T hours is taken by the camera 200. When the camera 200 has the ability to capture a frame image (frame image) at a rate of one for each ΔT time, the recording unit 101 records T / ΔT frame images in time series.

A frame buffer is provided in the moving image distance calculation device 100 or the camera 200, and the frame image for each unit time taken by the camera 200 is temporarily recorded in the frame buffer, and the frame image recorded in the frame buffer is the time. It may be configured to be sequentially recorded in the recording unit 101. Further, the moving image recorded in the recording unit 101 is not limited to the moving image taken in real time by the camera 200, and may be a moving image (past moving image) taken by the camera 200 in advance.

The moving image used to calculate the distance from the object to the camera 200 is not limited to that recorded as digital data. For example, even a moving image recorded as analog data can be recorded in the recording unit 101 as a time-series frame image by performing a digital conversion process. By using the frame images recorded in time series, it is possible for the moving image distance calculation device 100 to perform the distance calculation process.

The type and configuration of the camera 200 is not particularly limited as long as it is a photographing means capable of capturing a surrounding landscape or the like as a moving image. For example, it may be a general movie camera, or it may be a camera provided in a mobile terminal such as a smartphone.

The recording unit 101 is composed of a general hard disk or the like. The configuration of the recording unit 101 is not limited to the hard disk, and may be a flash memory, an SSD (Solid State Drive / Solid State Disk), or the like. The specific configuration of the recording unit 101 is not particularly limited as long as it is a recording medium capable of recording a moving image as a plurality of frame images in a time series.

The CPU 104 performs a process of calculating the distance from the object reflected in the frame image (moving image) to the camera 200 based on a plurality of frame images (moving images) recorded in the recording unit 101 in time series. .. The CPU 104 performs the distance calculation process based on the program (the program based on the flowchart of FIG. 2), the details of which will be described later.

A program or the like for calculating the distance from the camera 200 to the object reflected in the frame image is recorded in the ROM 102. The RAM 103 is used as a work area used for processing the CPU 104.

In the moving image distance calculation device 100 according to the embodiment, a configuration in which a program (a program based on the flowchart shown in FIG. 2: a moving image distance calculation program) is recorded in the ROM 102 will be described. However, the recording medium on which the program is recorded is not limited to the ROM 102, and may be configured to record the program in the recording unit 101.

On the monitor 210, a moving image taken by the camera 200, an image three-dimensionally converted by a distance calculation process, a moving image, and the like (for example, images shown in FIGS. 8 to 11 described later) are displayed. .. As the monitor 210, for example, a general display device such as a liquid crystal display or a CRT display is used.

Next, a method will be described in which the CPU 104 calculates the distance from the object reflected in the moving image to the camera 200 based on the moving image (frame image recorded in time series) recorded in the recording unit 101. ..

Euclid discussed the visual phenomenon of motion parallax more than 2000 years ago. The visual phenomenon due to dynamic parallax is a phenomenon in which when an object is moving at a constant velocity, a distant object visually moves less than a near object. Visual phenomena due to dynamic parallax are routinely observed. In the AMP method and the FMP method already described, the distance from the object reflected in the moving image to the camera is calculated by using the visual phenomenon due to the dynamic parallax.

The moving image distance calculation device 100 uses a visual phenomenon due to dynamic parallax and calculates the distance from the object to the camera 200 using the moving image taken by the camera. In the AMP method and the FMP method, a pixel at any coordinate in a moving image is set as a target pixel, and the value of dynamic parallax is obtained by finding how the target pixel moves in the moving image.

The moving image distance calculation device 100 uses a technique called optical flow as a method of determining how an object reflected in a moving image has moved. The optical flow is a vector representation of the movement of an object in a moving image (a plurality of frame images that are continuous in time).

Here, the object to which the optical flow is applied needs to be a two-dimensional scalar field at time t. The two-dimensional scalar field at time t is indicated by f (x, y, t). Of f (x, y, t), (x, y) indicates the coordinates of the image, and t indicates the time (time). By indicating the two-dimensional scalar field as f (x, y, t) in this way, it is possible to calculate ∂f / ∂x, ∂f / ∂y, which are partial derivatives of x, y. ..

Since the optical flow is the movement of an object (coordinates) in a moving image, the optical flow can be expressed as (dx / dt, dy / dt). in this case,
−∂f / ∂t = (∂f / ∂x) (dx / dt) + (∂f / ∂y) (dy / dt)
It is possible to obtain the optical flow (dx / dt, dy / dt) from the relational expression of.

Further, when the optical flow is obtained based on this relational expression, the partial differential ∂f / ∂t with respect to the time t is used. Therefore, the object to which the optical flow is applied needs that the images are continuous as a condition that the partial differential can be calculated with respect to the time t. Therefore, it is possible to use a moving image (a plurality of frame images that are continuous in time) taken by the camera 200 as a scalar field having a time t and coordinates (x, y) to which the optical flow is applied. Therefore, it is possible to extract the movement of an object in a moving image as an optical flow in pixel units.

The movement of the object in the moving image includes a case where the object itself actively moves in the moving image and a case where the object is passively moved in the moving image with the movement of the camera. .. Therefore, the optical flow is a vector extracted of the positive movement of the object and the passive movement of the object due to the movement of the camera or the like.

When extracting the optical flow from a moving image, a library for computer vision can be used. Specifically, it is possible to extract the optical flow by using a publicly available library for open source computer vision called Open CV.

FIG. 2 is a flowchart showing a processing content for the CPU 104 of the moving image distance calculation device 100 to extract an optical flow from a moving image and calculate the distance to an object reflected in the moving image. The CPU 104 reads the program recorded in the ROM 102 and executes the process shown in FIG. 2 according to the program. Further, as described above, the moving image taken by the camera 200 is recorded in the recording unit 101 for each frame image. The CPU 104 extracts the optical flow at time t based on the moving image for each frame recorded in the recording unit 101.

FIG. 3 is an image showing a frame image at time t as an example among the moving images taken by the camera 200. The image shown in FIG. 3 shows a state of a scrambled intersection taken from a high-rise part of a building. Color information (hereinafter referred to as RGB information) of three colors of red (Red), green (Green), and blue (Blue) is added to each pixel of the image shown in FIG. The algorithm for extracting optical flow is based on the premise that "the brightness of an object on an image does not change between consecutive frame images" and "adjacent pixels behave similarly". Therefore, the RGB information of each pixel is an important element for extracting the optical flow.

Further, since the optical flow is extracted based on the RGB information of each pixel, when adjacent pixels have similar RGB information (such a group of pixels is referred to as a state without texture), the optical flow is extracted. It becomes difficult to extract the movement of the object by. However, the portion corresponding to the state without texture is often, for example, asphalt (ground) or the like, where there is no moving object. Therefore, even if the movement of the object is not detected by the optical flow, problems and the like are unlikely to occur.

When the state of the scrambled intersection shown in FIG. 3 is photographed by the camera 200, the object of distance calculation is mainly a group of people such as pedestrians moving at the intersection.

The CPU 104 reads out the moving image recorded in the recording unit 101 (S.1 in FIG. 2), and performs an optical flow of the image at time t based on the frame image (moving image) from time t-2 to time t + 2. Extract (S.2 in FIG. 2). Since the CPU 104 performs a process of extracting the optical flow of the image at time t from the moving image (optical flow extraction function, all-pixel optical flow extraction function) based on the program, the “optical flow extraction means” 104a or “all pixels” Corresponds to "optical flow extraction means" 104d (see FIG. 1).

FIG. 4 shows an image in which the optical flow extracted at time t is superimposed on the image shown in FIG.

In the CPU 104 according to the embodiment, a case where the optical flow of the image at time t is extracted based on the moving image from time t-2 to time t + 2 will be described as an example, but the moving image for extracting the optical flow will be described. Is not limited to moving images from time t-2 to time t + 2. Further, the temporal length of the moving image for extracting the optical flow is not limited to the length from the time t-2 to the time t + 2, and may be longer or shorter than this. For example, the data section (start time and end time) and the length of each moving image may be changed according to the characteristics of the movement of the object.

Further, when the optical flow is extracted based on the moving image taken in advance by the camera 200 (moving image taken in the past), the time t is based on the moving image from time t-2 to time t + 2. It is possible to extract the optical flow of the image. However, when the time t taken by the camera 200 is captured as the current time, the frame images (moving images) of the time t + 1 and the time t + 2 have not been taken yet, so that the optical flow at the time t can be extracted. It gets harder. In this case, for example, by extracting the optical flow of the image at time t-2 based on the moving image from time t-4 to time t, shooting with the camera 200 is continued without performing batch processing. However, it is possible to extract the optical flow in chronological order.

As shown in FIG. 4, the optical flow is extracted for each pixel of the moving image. In FIG. 4, the optical flow is shown by a line segment, but the optical flow for each pixel extracted by using the OpenCV library is obtained as a vector. In FIG. 4, the moving direction of the object in each pixel is shown by the direction of the line segment, and the moving distance is shown by the length of the line segment. As shown in FIG. 4, the optical flow extracted for each pixel is oriented in various directions. From the direction of optical flow, it can be determined that the object has moved in various directions.

The conditions under which the optical flow is extracted are not limited to the case where only the object moves. For example, there may be a case where the camera 200 moves in an arbitrary direction, a case where only the object moves while the camera 200 is stationary, or a case where both the camera 200 and the object move. When the camera 200 moves during shooting, the moving objects are recorded (photographed) in a moving image as if the stationary objects moved all at once as the camera 200 moved. Since the optical flow of the stationary object extracted when the camera 200 moves is simultaneously extracted for each stationary object according to the moving direction and the moving distance of the camera, the camera 200 depends on the characteristics of the extracted optical flow. Can be determined whether or not has moved. When the optical flow of a stationary object is extracted by moving the camera 200, the distance from the camera 200 to each stationary object is obtained by using the method described later based on the value of each optical flow. It is possible. On the other hand, when only the object moves while the camera 200 is stationary, the optical flow of the stationary object is not extracted, and the optical flow of the moved object is extracted.

Next, the CPU 104 calculates the value of the optical flow indicating the magnitude of the optical flow based on the extracted optical flow (S.3 in FIG. 2). Since the CPU 104 performs a process of calculating the magnitude of the optical flow as an optical flow value (optical flow value calculation function, all-pixel optical flow value calculation function) based on the program, the “optical flow value calculation means” 104b. Or "All-pixel optical flow value calculation means" 104e (see FIG. 1).

Since optical flow is represented by a vector, the value of optical flow is calculated by the magnitude of the vector (absolute value of the vector). For example, when the vector of Optical flow and (V1, V2), seeking the value ^(V1 2 + V2 ²⁾ of the sum of the square of the value (V1 ²⁾ and the square of the value of V2 of V1 (V2 ²⁾ , The value of optical flow is calculated by calculating the square root of the obtained sum value (V1 ² + V2 ² ).

The CPU 104 of the moving image distance calculation device 100 captures the calculated optical flow value as dynamic parallax and calculates the distance from the object to the camera 200. Therefore, the optical flow value calculated by a stationary object (stationary object) and the optical flow value calculated by a moving object can be judged to be the same by regarding them as dynamic parallax. it can.

Further, when both the camera 200 and the object move, even if the movement of the object is large with respect to the movement of the camera 200, the movement of the camera 200 is large with respect to the movement of the object. Even if there is, each optical flow can be extracted. Therefore, the distance from the object to the camera 200 can be calculated by the distance calculation using the dynamic parallax described later.

However, when the movements of the camera 200 and the object move in the same direction to the same extent, it becomes difficult to extract the optical flow of the object. For this reason, it is difficult to calculate the distance from the object to the camera 200 for an object that has moved to the same extent as the camera 200. On the other hand, when the object moves in the opposite direction to the camera 200, for example, when the own vehicle on which the camera 200 is installed moves forward and the oncoming vehicle moves toward the own vehicle. The value of the optical flow of the oncoming vehicle becomes a large value by adding the speed of the own vehicle and the speed of the oncoming vehicle. In this case, the distance from the oncoming vehicle to the camera 200, which is obtained based on the value of optical flow, is calculated as a distance closer than the actual distance.

In this way, when the distance obtained from the optical flow value of the approaching oncoming vehicle is closer than the actual distance from the oncoming vehicle to the camera 200, this tendency is taken into consideration and the surrounding area is stationary. By comparing the optical flow value calculated by the object with the optical flow value of the approaching object, it is possible to identify the oncoming vehicle or the like.

When the optical flow shown in FIG. 4 is confirmed, the optical flow is extracted not only for moving objects such as a group of people (pedestrians and the like) but also for stationary objects. Therefore, in the moving image from time t-2 to time t + 2, it can be confirmed that both the camera 200 and the object (person group) are moving. However, in the moving image used for extracting the optical flow, the movement of the person group is larger than the movement of the camera 200, so it can be determined that the optical flow is mainly caused by the movement of the person group.

Generally, when extracting an optical flow from a moving image, the extraction process is performed using the moving image for an extremely short time. Therefore, when the movement of the camera 200 is extremely large, that is, when the shooting range of the frame image changes significantly in an extremely short time, the movement of the object becomes smaller than the movement of the camera 200. Further, when the shooting range of the frame image does not change significantly, the movement of the object becomes larger than the movement of the camera 200, and the optical flow extracted from the moving image is the movement of the object (person group). It can be judged that it was caused by.

In the CPU 104 according to the embodiment, the optical flow is extracted based on the moving image from the time t-2 to the time t + 2 at each time t, but as described above, the length of this moving image (start). The interval from the time to the end time) can be set and changed arbitrarily. By adjusting the length of this moving image, it is possible to effectively extract the movement of the object as an optical flow.

Further, when the display state of the road or the white line of the moving image changes with the movement of the camera 200, the optical flow of the road or the white line is extracted with this change. Since roads and the like often correspond to a state without texture (when adjacent pixels have similar RGB information), the calculated optical flow value tends to be relatively small. When the value of optical flow is small, the distance calculated by the distance calculation process from the object to the camera, which will be described later, becomes large (far). On the other hand, the value of optical flow calculated by an object that actively moves such as a group of people tends to be larger than the value of optical flow calculated by a road or the like, and when the value of optical flow is large. Makes the distance from the object to the camera smaller (closer).

Therefore, by setting a predetermined threshold value for the distance from the object to the camera 200 and determining whether the calculated distance is larger or smaller than the threshold value, the extracted optical flow is the movement of the object such as a pedestrian. It is possible to determine whether the camera 200 corresponds to the movement of the road or the like accompanying the movement of the camera 200. However, it is difficult to uniformly extract all objects such as pedestrians by using only the magnitude of the optical flow value and the magnitude with respect to the threshold value. Therefore, it is preferable to flexibly set the threshold value and the like according to the movement of the photographed object, the shooting range of the camera, and the like to improve the detection accuracy of the object.

Next, the CPU 104 of the moving image distance calculation device 100 performs the image region division processing by applying the mean-shift method (intermediate value shift method) to the image at time t (S.4 in FIG. 2). ). Since the CPU 104 performs a process of dividing an image into an area corresponding to an object (area division function) based on a program, it corresponds to the "area division means" 104f (see FIG. 1). FIG. 5 is a diagram showing the result of applying the mean-shift method to the image at time t shown in FIG.

The mean-shift method (median shift method) is known as one of the most promising methods among existing area division methods. The mean-shift method is a well-known area division method, and is realized by using a widely open library for computer vision called Open CV. By applying the mean-shift method to an image (frame image) at time t, the area of the image is divided according to the presence or absence of an object or the like based on the RGB value (color information) for each pixel of the image. It can be interpreted that the distances from the camera are almost the same for the portions of the divided regions that are determined to be the same region.

In the mean-shift method, various parameters can be set, and the size of the divided area can be adjusted by adjusting the set values of the parameters. By appropriately setting the parameter setting values, it is possible to adjust so that there is only one person such as a pedestrian per divided area.

For example, when there are M objects (M ≧ 3) for which the distance from the camera 200 is calculated, the image at time t can be obtained by appropriately setting the parameters and adjusting so that the divided area becomes relatively small. It is possible to divide into K (K ≧ M) regions including regions corresponding to M objects. However, although it is possible to increase or decrease the size of the divided area by setting the parameters, the number K of the divided areas depends on the image as a result. Therefore, although it is possible to adjust the size of the divided area and adjust the increase or decrease in the number of divided areas by setting the parameters, the number of divided areas is set to a predetermined number. It is difficult to set the parameters.

In the image to which the mean-shift method is applied shown in FIG. 5, when the parameters are set appropriately, as a result, a line segment indicating the region boundary is formed so as to correspond to each pedestrian. It is shown. Further, since the pedestrian crossing and the like also have a texture, a line segment indicating the region boundary is formed so as to correspond to the white line of the pedestrian crossing. On the other hand, the asphalt part of the intersection is in a state without texture. For this reason, a line segment indicating a region boundary is not formed so much in the asphalt portion or the like, and is shown as a relatively large region.

Next, the CPU 104 calculates the average of the optical flow values obtained in each area for each area divided by the mean-shift method (S.5 in FIG. 2). Since the CPU 104 performs a process of calculating the average of the optical flow values for each region divided region based on the program (region-specific optical flow value calculation function), the "region-specific optical flow value calculation means" 104 g ( (See Fig. 1).

In the mean-shift method, region division is performed according to the presence or absence of an object or the like based on the RGB value (color information) for each pixel of the image. In particular, by appropriately setting the parameters of the mean-shift method, it is possible to perform division so that one person such as a pedestrian is one person per division area. By calculating the average of the optical flow values for each divided region, it is possible to normalize the optical flow values of pedestrians and the like existing in the divided regions.

FIG. 6 is a diagram in which the average of the optical flow of each region is arranged at the center of each region divided by the mean-shift method. A white circle (○) P is shown at the center position (pixel) of the region, and the average direction of the optical flow and the average magnitude of the optical flow value are the direction and length of the line segment L extending from the white circle P. It is indicated by. However, in the image shown in FIG. 6, the line segment L and the white circle P of the optical flow of the portion corresponding to the ground are not displayed.

As described above, the image shown in FIG. 3 shows a state in which both the camera 200 and the group of people (objects) are moving. Therefore, as shown in FIG. 4, the optical flow is also extracted from the pixels corresponding to the road as the camera 200 moves. However, it is calculated based on the distance from the camera 200 to the road and the optical flow extracted with the movement of both the camera 200 and the person, which is calculated based on the optical flow extracted with the movement of the camera 200. When comparing the distance from the camera to each person, there is a difference in the distance. In the image at time t shown in FIG. 3, since the person is standing on the road, the distance from the camera is shorter for the person than for the road. In other words, the distance differs by the height of the person.

Therefore, it is possible to distinguish between a road and a person by determining an object having a certain height (distance) with respect to the road as a person. By determining in advance a threshold value for determining such a difference between a road and a person by an experiment or the like, it is possible to extract an optical flow of only a group of people excluding the road. In FIG. 6, among the areas divided by the mean-shift method, the average of the optical flow values in the areas determined to indicate a group of people instead of roads is calculated for each area, and a white circle is formed in the center of each area. P is shown, and the average distance and direction of the optical flow values are shown by a line segment L extending from the white circle P. In FIG. 6, the optical flows of the persons moving in various directions are extracted corresponding to the respective positions of the plurality of persons.

Next, the CPU 104 calculates the distance from the object to the camera 200 for each area based on the average of the calculated optical flow values for each area (S.6 in FIG. 2). Since the CPU 104 performs a process (distance calculation function) of calculating the distance from the object to the camera using the optical flow value based on the program, it corresponds to the "distance calculation means" 104c (see FIG. 1). ..

The CPU 104 considers the optical flow value calculated for each area as dynamic parallax, and calculates the distance from the object to the camera 200. A method of calculating the distance from the object to the camera 200 based on the dynamic parallax has already been proposed in the AMP method and the FMP method.

FIG. 7 is a diagram showing a geometric model for explaining a method of obtaining a distance from an object to the camera 200 based on dynamic parallax. The vertical axis of FIG. 7 shows the virtual distance Zv from the object to the camera 200. The positive direction of the virtual distance Zv is the lower direction in the figure. The horizontal axis of FIG. 7 indicates the dynamic parallax q. The dynamic parallax q is an experimental value based on the pixel locus obtained by the optical flow, that is, a value of the optical flow. The positive direction of the dynamic parallax q is the right direction in the figure.

Since the value of the virtual distance Zv is virtual, it corresponds to the value of the parallax q ₀ , which is a coefficient determined after the fact of the dynamic parallax q. As a characteristic of dynamic parallax, the larger the value of dynamic parallax, the shorter the distance from the object to the camera, and the smaller the value of dynamic parallax, the longer the distance from the object to the camera. When the virtual distance Zv is expressed in detail, it is actually expressed by a function called Zv (q ₀ ).

It is assumed that the parallax q ₀ , which is a constant determined after the fact, plus a small amount of Δq (q ₀ + Δq) is the value q of the optical flow determined by one pixel by the optical flow. That is, q = q ₀ + Δq. Further, it is assumed that Zv corresponds to q ₀ and the minute amount ΔZv of the virtual distance Zv corresponds to Δq. At this time, assuming that the relationship between the two is linear, the relationship is as shown in the geometric model shown in FIG. 7, and the following linear proportional relationship is established.
Zv: q ₀ = −ΔZv: Δq

From this proportional relationship, the following linear differential equation is established. Solving this linear differential equation
−Q ₀・ ΔZv = Zv ・ Δq
ΔZv / Zv = −Δq / q ₀
logZv = -q / q ₀ + c (c is a constant)
By transforming the above equation, Zv = a · exp (bq)
Is established. Here, there is a relationship of b = -1 / q ₀ , and if b is determined as a boundary condition, q ₀ will be determined ex post facto.

Further, a and b (a> 0, b <0) are indefinite coefficients. Further, exp (bq) indicates the bq power of the base value of the natural logarithm (Napier's constant). The values of the coefficients a and b can be determined by individual boundary conditions. Once the coefficients a and b are determined, the value of the dynamic parallax q can be calculated based on the moving image taken by the camera 200, and the value of Zv is used as the real distance in the real world instead of the virtual distance. It becomes possible to ask.

The values of the constants a and b are determined based on the fluctuation range of the variable Zv and the variable q. Zv indicates the virtual distance from the object to the camera 200, as described above. The virtual distance is a value that can change depending on the target world (target world, target environment), and is a value different from the real distance in the real world. Therefore, a method such as distance measurement using a laser (hereinafter referred to as laser measurement) or inspection of the fluctuation range of the real distance in the real world corresponding to the virtual distance Zv in the three-dimensional space (target world) of the moving image. Therefore, by measuring in advance (determining in advance), it is possible to obtain the actual distance in the real world in association with the distance in the target world. As described above, the method of calculating the virtual distance Zv using the value of the dynamic parallax q (the value of the optical flow) shows that the relative distance is detected.

If the real distance Z (distance Z from the object to the camera) in the real world can be associated with the virtual distance Zv in the target world,
Z = a · exp (bq) ・ ・ ・ Equation 1
Can be used to obtain the real distance Z in the real world. That is, the distance Z from the object to the camera 200 in the real world can be obtained as a distance function determined from theory.

In the moving image distance calculation device 100 according to the embodiment, the fluctuation range of the real distance in the real world corresponding to the virtual distance Zv in the three-dimensional space (target world) of the moving image is measured in advance by laser measurement as an example. The distance range of the virtual distance Zv measured by the laser measurement is represented by Z _N ≤ Zv ≤ Z _L (Z _N ≤ Z _L ).

More specifically, when a plurality of objects, for example, M objects are shown in a moving image, and the distances from the M objects to the camera 200 (actual distance in the real world) are calculated. , Of the M objects, the distance from the camera 200 to the object located at the nearest location (actual distance) and the distance to the object located at the farthest location (actual distance) are measured by laser measurement. Measure in advance. Of the M objects, the distance from the camera 200 to the object located at the farthest place is Z _L, and the distance to the object located at the closest place is Z _N. Of the M objects, for each of the M-2 objects, excluding the object closest to the camera 200 and the object farthest from the camera 200, the camera from the object based on the optical flow value. The distance to (actual distance) will be calculated. Therefore, in order to calculate the distance from the object to the camera 200, it is desirable that the number of objects is three or more (M-2> 0).

The fluctuation range of the value of the dynamic parallax q is determined by the experimental value individually obtained from the moving image. That is, it is not necessary to measure in advance. The fluctuation range of the dynamic parallax q can be obtained from the fluctuation range of the optical flow values of a plurality of objects. The maximum / minimum range of the dynamic parallax q obtained in this way is set to μ ≦ q ≦ γ. That is, among the optical flow values of a plurality of objects, the smallest value corresponds to μ and the largest value corresponds to γ. That is, μ and γ are experimental values determined by a plurality of optical flow values calculated based on a moving image.

Further, the correspondence between μ, γ and Z _L , Z _N can be obtained based on the nature of dynamic parallax. μ corresponds to Z _L and γ corresponds to Z _N. This is because the farther the virtual distance Zv is, the smaller the amount of movement of the object point (object position) of the moving image is, and the closer the virtual distance Zv is, the larger the amount of movement of the object point (object position) of the moving image is. This is due to the nature of dynamic parallax. In this way, the distance Z _N having the shortest distance in the distance range of the virtual distance Zv corresponds to γ having the largest amount of movement in the fluctuation range of the dynamic parallax q, and the distance in the distance range of the virtual distance Zv is The longest distance Z _L corresponds to μ, which has the smallest amount of movement in the fluctuation range of the dynamic parallax q.

Therefore, by substituting the values of Zv and q of Zv = a · exp (bq) with μ and Z _L , and γ and Z _N , the following simultaneous equations for a and b are established.

Z _L = a · exp (bμ) ・ ・ ・ Equation 2
Z _N = a · exp (bγ) ・ ・ ・ Equation 3
The equations 2 and 3 correspond to the boundary conditions.

By solving this simultaneous equation, the constants a and b can be obtained as follows.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N )) ・ ・ ・ Equation 4
b = (1 / (μ-γ)) log (Z _L / Z _N ) ・ ・ ・ Equation 5
In this way, by obtaining the constants a and b and applying them to the above equation 1, the value of the virtual distance Zv can be calculated as the real distance Z in the real world.

The above-mentioned distance Z is obtained for each divided region. As described above, by appropriately setting the parameters of the mean-shift method, for example, it is possible to divide the area so that there is only one person such as a pedestrian per divided area. is there. That is, by appropriately setting the parameters of the mean-shift method, it is possible to divide the image into K regions, which is more than M, so that each of the M objects has a different division region. is there. Therefore, the distance Z from the camera 200 to each object can be obtained by setting the parameters of the mean-shift method so that the objects reflected in the moving image are in different regions.

After that, the CPU 104 records the value of the distance Z of each region in the image at time t in association with each pixel in the region (S.7 in FIG. 2). That is, the process of pasting the value of the distance Z obtained for each region is performed for each pixel in the image at time t. By pasting (recording) the obtained distance Z in association with each pixel in this way, even if the time t of the moving image changes, for each pixel of the image at each time. , It becomes possible to instantly acquire the distance from the object to the camera 200. Since the distance information is recorded in association with the pixels, the information associated with each pixel in the image at each time becomes (r, g, b, D) consisting of color information and distance information D. This information is recorded in the recording unit 101.

By recording the distance information D of each pixel of the moving image in the recording unit 101, it is possible to three-dimensionally grasp the state of the object reflected in the moving image by using the distance information D. FIG. 8 is an image showing the state of the scrambled intersection shown in FIG. 3 three-dimensionally from different viewpoints. In the image shown in FIG. 8, the position and height of each group of people located in the image are obtained by converting the size of the average optical flow value arranged in the center of each area into a distance. The state of the ground and the group of people at the scrambled intersection is shown by changing the viewpoint.

In the geometric model shown in FIG. 7, the optical flow used as the dynamic parallax q can take any direction, and the direction is not limited. For example, it is possible to take a picture of the city from the sky with a camera 200 installed in a drone, an airplane, or the like, and obtain three-dimensional distance information of the city by using the taken moving image.

FIG. 9 is an image showing the state of the city in three dimensions by acquiring position information for each pixel using a moving image taken from the sky. The moving direction of the camera 200 that captures a moving image is not necessarily horizontal movement with respect to a building or the like in the city to be captured. As a condition for acquiring the distance information by the moving image distance calculation device 100 according to the embodiment, it is not necessary to move the moving direction of the camera 200 for shooting the object to be photographed laterally as in the AMP method. Further, unlike the FMP method, it is not necessary to limit the moving direction of the camera 200 to the front or the rear. Therefore, it is possible to reduce the restrictions on the moving image for calculating the distance to the object, and the distance information to the object is pixelated using the moving image taken by the camera 200 moving in various directions. It becomes possible to obtain it every time.

In FIG. 10, the distance of an object in front of the vehicle is calculated for each pixel based on a moving image of the front of the traveling vehicle taken by the camera 200, and the state in front of the vehicle is three-dimensionally based on the calculated distance information. It is an image shown in. Conventionally, the FMP method has been used to measure the distance from an object to the camera 200 based on a moving image of the front of a traveling vehicle. As shown in FIG. 10, even when the distance from the object to the camera 200 is calculated using optical flow based on a moving image of the front of the vehicle, a three-dimensional object created by using the FMP method is used. It is possible to create a three-dimensional image with the same accuracy as a normal image.

As described above, the moving image distance calculation device 100 is not restricted by the moving direction of the camera unlike the AMP method and the FMP method, and is based on the moving images taken by the camera 200 moving in various directions. Therefore, it is possible to calculate the distance from the object to the camera 200.

Therefore, for example, it is possible to obtain the state of the space around the robot based on a moving image taken by a camera installed in the robot. When a robot enters a space that humans cannot easily enter, such as during a disaster, it is necessary to judge the surrounding situation based on the moving image taken by the robot's camera. The moving image taken by the robot camera is not necessarily limited to the moving image in front of the robot in the traveling direction or the moving image moved in the lateral direction. Cameras are installed on the robot's head, chest, arms, and fingers as needed, and the cameras are moved in any direction according to the movement of the robot to capture moving images. Even if the camera is moved in any direction, the optical flow is extracted according to the movement of the camera or the movement of the captured object, so the target is based on the extracted optical flow. It becomes possible to calculate the distance to an object (including the distance to a wall, floor, etc.).

By controlling the chest, arms, fingers, etc. of the robot based on the calculated distance to the object, etc., the robot can be moved smoothly at the disaster site, and more accurate control can be performed. It will be possible to do. In addition, by acquiring the surrounding distance information three-dimensionally based on the moving image taken by the camera 200, it becomes possible to create a three-dimensional map at a disaster site or the like, and in subsequent relief activities or the like. It becomes possible to increase mobility.

FIG. 11 is a diagram in which a camera is installed in a robot moving in a room, a moving image taken by the robot's camera is used to acquire distance information, and the surrounding situation in the room is shown three-dimensionally. Consider a case where the robot is controlled to move to the valve V shown in FIG. 11 and the valve V is rotated by the arm and the finger of the robot. In this case, since the robot does not always move continuously, there may be a time when the surrounding situation of the moving image taken by the camera does not change at all.

As described above, the optical flow is a vector representation of the movement of an object reflected in a moving image. For this reason, if there is no object that actively moves in the room and the movement of the robot is stopped so that the moving image does not change, the optical flow cannot be extracted. It is not possible to calculate the distance around the room. In this case, the distance information around the room calculated when the camera last moved is maintained in a state where the camera does not move (a state in which the moving image does not change), and then the camera moves. In such a case, the distance around the room can be continuously determined by continuously using the already calculated distance information.

Further, even when the camera moves, the moving speed of the camera is not always constant. In this case, even if the distance from the object to the camera is the same, the optical flow value calculated for each time will be different.

Further, when calculating the distance from the object to the camera 200, two dynamic ranges are required as described above. The dynamic range of the optical flow value (μ, γ) and the dynamic range of the distance to be obtained (Z _N , Z _L ). The dynamic range of the optical flow value can be calculated from the moving image, but the dynamic range of the distance needs to be measured in advance by inspection or laser measurement. However, when the distance from the object to the camera is long (the distance value is large), there is no guarantee that the dynamic range of the distance will be accurately determined.

In addition, the optical flow value calculated based on the moving image is smaller for a long-distance object than for a short-distance object. In addition, the value of optical flow fluctuates not only with the movement of the object but also with the movement of the camera.

In this way, in the CPU 104, the optical flow value is not inaccurate due to the influence caused by whether the distance from the object to the camera is a short distance or a long distance, or the influence caused by the moving speed of the camera. Correction is performed by normalizing the value of optical flow. Specifically, the optical flow values of all pixels are added (total) for each image at each time, and the added value (sum) is used to determine the optical flow of each pixel of the image at the corresponding time. Normalization is performed by dividing the value of.

By performing the normalization in this way, even if the extracted optical flow differs for each time due to the difference in the moving speed of the camera, the distance to the object is short or long. Even if the optical flow may be affected, the distance from the object to the camera can be calculated accurately. This normalization method can be used not only when the moving speed of the camera is not constant, but also in various cases.

When the distance from the object to the camera is long (the distance value is large), the calculated distance Z (q) is multiplied by the coefficient C to obtain CZ (q), which is a long-distance object. Calculate the distance value of the pixel corresponding to the object. This coefficient C can be determined by using some method such as GPS.

Further, if a camera for capturing a moving image and a CPU for calculating a distance to an object using the moving image are provided, it can be regarded as the moving image distance calculation device 100 according to the embodiment. it can.

Mobile terminals such as smartphones these days are generally equipped with a camera, and it is possible to take a moving image. Therefore, a moving image is taken by the camera of the mobile terminal, and the optical flow at each time is extracted by the CPU of the mobile terminal using the taken moving image to calculate the distance from the object to the mobile terminal. Is possible. In addition, it is possible to create a three-dimensional image based on the captured moving image.

In recent years, a method called ToF (Time of Flight) has been proposed as a method for creating a three-dimensional image. In ToF, light is projected onto an object, and by receiving the reflected light of the light, the time from the projection of the light to the reception of the reflected light is measured, and the object is reached based on the measured time. Calculate the distance. In order to create a three-dimensional image using ToF, it is necessary that the object is diffusely reflected. Therefore, there is a problem that the measurement accuracy is lowered for specular reflection objects such as hardware and seto objects. In addition, it was necessary to have an environment in which there was no object such as rain or smoke that obstructed the progress of light. Further, the range in which a three-dimensional image can be actually created using ToF is a range from about 50 cm to about 4 m, and there is a problem that the applicable range is limited. Furthermore, the accuracy of correspondence between the distance to the object to be measured and the pixels of the camera is not sufficient, and the hardware for realizing these functions has been continuously improved for performance improvement.

On the other hand, when the optical flow is extracted based on the moving image of the captured camera and the distance to the object is obtained as in the moving image distance calculating device 100 according to the embodiment, it is general. It suffices to have a camera and a CPU or the like capable of performing optical flow extraction processing. Therefore, even with a general smartphone or the like, it is possible to calculate the distance to the object with high accuracy.

Specifically, when a moving image is taken by a mobile terminal such as a smartphone, an optical flow based on the movement of the mobile terminal can be extracted from the moving image by shaking the mobile terminal slightly. By extracting the optical flow from the frame image of several frames at the moment when the mobile terminal is shaken, it becomes possible to create a three-dimensional image. Further, by taking a moving image while the mobile terminal is stationary, a three-dimensional image can be created based on the optical flow of a moving object. By extracting the optical flow and calculating the distance in this way, the distance from the object to the camera is calculated not only for short-distance objects but also for long-distance objects and moving objects. It is possible to create a three-dimensional image.

The moving image distance calculation device and the moving image distance calculation program according to the present invention have been described in detail by showing the moving image distance calculation device 100 according to the embodiment as an example. The distance calculation device and the moving image distance calculation program are not limited to the examples shown in the embodiments.

For example, in the moving image distance calculation device 100 according to the embodiment, the CPU 104 divides the area by applying the mean-shift method to the image at time t, and the optical flow values of all the pixels in the area are calculated. The case where the distance from the object to the camera 200 is calculated by calculating the average has been described. However, it is not always necessary to apply the mean-shift method for calculating the distance from the object displayed in the image at time t to the camera 200.

For example, even when the mean-shift method is not applied, that is, when the optical flow value is obtained for each pixel and the distance for each pixel is calculated, as described above, all in the image at time t. By finding the sum of the optical flow values of the pixels and dividing the optical flow value of each pixel by the sum of the optical flow values of all the pixels, correction for short and long distances and the movement speed of the camera It becomes possible to obtain the value of optical flow in consideration of the correction for each pixel. Therefore, even if the method does not use the mean-shift method, it is possible to accurately calculate the distance for each pixel.

Even when the mean-shift method is not applied to the image at time t, the value of optical flow calculated in a state without texture such as a road becomes an extremely small value or becomes zero. Similarly, even when the mean-shift method is applied, the average of the optical flow values calculated in the absence of texture becomes extremely small. Therefore, in a region where the average of the optical flow values calculated by applying the mean-shift method is small, there is a risk that a distance farther than the actual distance in that region will be calculated. In such a case, interpolate the distance calculated in the area where the optical flow value is small with the distance calculated in the area around the area where the optical flow value is not small. Make corrections by.

Further, in the moving image distance calculation device 100 according to the embodiment, when the number of objects displayed in the image at time t is, for example, M, the CPU 104 extracts M optical flows corresponding to the objects. The case of calculating the distance to each of the M objects has been described. Here, it is sufficient that the number of M includes the object of the shortest distance Z _N measured in advance by inspection or laser measurement and the object of the farthest distance Z _L , and the object of the distance measurement is It is sufficient that M ≧ 3 or more by inserting another object. Therefore, the number of objects for which the distance from the camera is calculated is not particularly limited as long as it is 3 or more.

Further, since the object may be an object that appears in the image at time t, all the pixels of the image at time t may be the objects. That is, the number M of the objects may be M = the total number of pixels. By calculating the distance from the object to the camera for each pixel, it is possible to acquire the distance information of all the pixels. Further, when all the pixels of the image at time t are objects, it is not necessary to divide the image at time t into regions so as to correspond to M objects by the mean-shift method.

Further, it is possible to set the number M of the objects to be a fraction of the total number of pixels instead of setting the total number of pixels. For example, by setting an area of a total of 4 pixels of 2 vertical pixels and 2 horizontal pixels as one area and setting one pixel for each area as an object, one pixel for each of the four pixels corresponds to the camera. It becomes possible to calculate the distance of the pixel to the object. By calculating the distance at a ratio of one pixel to several pixels instead of calculating the distance with all pixels, it is possible to reduce the processing load of the CPU 104 and speed up the processing.

100 ... Moving image distance calculation device 101 ... Recording unit 102 ... ROM
103 ... RAM
104 ... CPU (computer, optical flow extraction means, optical flow value calculation means, distance calculation means, all pixel optical flow extraction means, all pixel optical flow value calculation means, area division means, area-specific optical flow value calculation means)
200 ... Camera 210 ... Monitor V ... Valve L ... (Indicating the average of the optical flow values in the area) Line segment P ... (Indicating the center of the divided area) White circle

Claims (10)

Using the moving image of the camera that captured M (M ≧ 3) objects, from the pixels of the M objects reflected in the image at time t of the moving image, M corresponding to each pixel. An optical flow extraction method that extracts optical flow,
An optical flow value calculating means for calculating the magnitude of each of the M optical flows extracted by the optical flow extracting means as an optical flow value q _m (m = 1, 2, ..., M). ,
Of the M optical flow values q _m calculated by the optical flow value calculating means, μ is the value with the smallest optical flow value, γ is the value with the largest optical flow value, and M is M. Of the respective distances from the object to the camera, the shortest distance Z _N and the farthest distance Z _L are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
Calculated by
Let the distances from the M objects to the camera be Z _m (m = 1, 2, ..., M), and set the distances Z _m to the constant a, the constant b, and M. Based on the optical flow value q _m and
Z _m = a · exp (bq _m )
A moving image distance calculation device characterized by having a distance calculation means calculated by The optical flow value calculation means is
The sum of the sizes of the M optical flows extracted by the optical flow extraction means is calculated, and the size of each optical flow is divided by the sum to obtain each normalized optical. The moving image distance calculation device according to claim 1, wherein the magnitude of the flow is set to the optical flow value q _m (m = 1, 2, ..., M). The M number is the number of pixels of the image at time t in the moving image.
The first or second aspect of the invention, wherein the distance calculating means calculates a distance Z _m from an object reflected in the pixel to the camera for each pixel of the image at time t. Moving image distance calculation device. An all-pixel optical flow extraction means that extracts optical flow of all pixels in an image at time t of the moving image using a moving image of a camera that has taken M objects (M ≧ 3).
An all-pixel optical flow value calculating means that calculates the respective magnitudes of the optical flow of all the pixels extracted by the all-pixel optical flow extracting means as a value of the optical flow for each pixel.
By applying the mean-shift method to the image at time t, the area dividing means for dividing the image at time t into K (K ≧ M) regions and
From the K regions divided by the region dividing means, M regions including pixels in which the object is reflected in the image at the time t are extracted, and all the regions in the region are extracted for each region. By calculating the average of the optical flow values of the pixels, the optical flow values q _m (m = 1, 2, ..., M) corresponding to each of the M objects are calculated. Value calculation means and
Of the M optical flow values q _m calculated by the region-specific optical flow value calculating means, the value with the smallest optical flow value is μ, and the value with the largest optical flow value is γ. , The shortest distance Z _N and the farthest distance Z _L among the respective distances from the M objects to the camera are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
Calculated by
Let the distances from the M objects to the camera be Z _m (m = 1, 2, ..., M), and set the distances Z _m to the constant a, the constant b, and M. Based on the optical flow value q _m and
Z _m = a · exp (bq _m )
A moving image distance calculation device characterized by having a distance calculation means calculated by The all-pixel optical flow value calculation means is
Normalization obtained by calculating the sum of the optical flow sizes of all the pixels extracted by the all-pixel optical flow extraction means and dividing the optical flow size of each pixel by the sum. The moving image distance calculation device according to claim 4, wherein the magnitude of the optical flow for each pixel is set as the value of the optical flow for each pixel. The moving image distance of the moving image distance calculation device that calculates the distance from the M objects reflected in the moving image to the camera by using the moving image of the camera that has taken M (M ≧ 3) objects. It is a calculation program
On the computer
An optical flow extraction function that extracts M optical flows corresponding to each pixel from M pixels of the object displayed in the image at time t of the moving image, and
An optical flow value calculation function that calculates the respective magnitudes of the M optical flows extracted by the optical flow extraction function as optical flow values q _m (m = 1, 2, ..., M). ,
Of the M optical flow values q _m calculated by the optical flow value calculation function, μ is the value with the smallest optical flow value, γ is the value with the largest optical flow value, and M is M. Of the respective distances from the object to the camera, the shortest distance Z _N and the farthest distance Z _L are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
Calculated by
Let the distances from the M objects to the camera be Z _m (m = 1, 2, ..., M), and set the distances Z _m to the constant a, the constant b, and M. Based on the optical flow value q _m and
Z _m = a · exp (bq _m )
A moving image distance calculation program characterized by realizing a distance calculation function calculated by In the optical flow value calculation function,
On the computer
Each normalized optical was obtained by calculating the sum of the sizes of the M optical flows extracted by the optical flow extraction function and dividing the size of each optical flow by the sum. The moving image distance calculation program according to claim 6, wherein the magnitude of the flow is set to the optical flow value q _m (m = 1, 2, ..., M). The M number is the number of pixels of the image at time t in the moving image.
In the distance calculation function, the computer
The moving image distance calculation program according to claim 6 or 7, wherein the distance Z _m from the object reflected in the pixel to the camera is calculated for each pixel of the image at time t. .. The moving image distance of the moving image distance calculation device that calculates the distance from the M objects reflected in the moving image to the camera by using the moving image of the camera that has taken M (M ≧ 3) objects. It is a calculation program
On the computer
An all-pixel optical flow extraction function that extracts the optical flow of all pixels in the image at time t of the moving image, and
An all-pixel optical flow value calculation function that calculates the respective magnitudes of the optical flow of all the pixels extracted by the all-pixel optical flow extraction function as the optical flow value for each pixel, and
By applying the mean-shift method to the image at time t, the area division function for dividing the image at time t into K (K ≧ M) regions and
Of the K regions divided by the region division function, M regions including pixels in which the object is reflected in the image at time t are extracted, and all the regions in the region are extracted for each region. By having the average of the optical flow values of the pixels calculated, the optical flow values q _m (m = 1, 2, ..., M) corresponding to the M objects are calculated for each region. Flow value calculation function and
Of the M optical flow values q _m calculated by the region-specific optical flow value calculation function, the value with the smallest optical flow value is μ, and the value with the largest optical flow value is γ. , The closest distance Z _N and the farthest distance Z _L of the respective distances from the M objects to the camera are measured in advance, and the constant a and the constant b are set.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (μ-γ)) log (Z _L / Z _N )
Calculated by
Let the distances from the M objects to the camera be Z _m (m = 1, 2, ..., M), and set the distances Z _m to the constant a, the constant b, and M. Based on the optical flow value q _m and
Z _m = a · exp (bq _m )
A moving image distance calculation program characterized by realizing a distance calculation function calculated by In the all-pixel optical flow value calculation function,
On the computer
Normalization obtained by calculating the sum of the optical flow sizes of all the pixels extracted by the all-pixel optical flow extraction function and dividing the optical flow size of each pixel by the sum. The moving image distance calculation program according to claim 9, wherein the magnitude of the optical flow for each pixel is set as the value of the optical flow for each pixel.