JP2001155153A - Device and method for generating depth map - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a depth map generation device which can simultaneously output an original image and a depth map which temporally coincide with each other and accordingly perform highly accurate image processing by using an application utilizing the original image and the depth map. SOLUTION: The original images fetched from cameras 25, 25 are once stored in a memory 33, a generated depth map is made synchronous with the original images in the memory 33, at the moment when the depth map is generated from the original images, and the depth map and the original images are written in memory parts 37 and 39 for output, respectively.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention calculates a distance (depth, depth) to each object in a two-dimensional image and generates a distance image expressing the distance on a two-dimensional screen. Output device.

[0002]

2. Description of the Related Art As an apparatus for generating a distance image from a two-dimensional image, a stereo image processing apparatus using stereo vision is widely known. This device captures the same spatial region with a plurality of (eg, two) cameras arranged at different positions, and receives a two-dimensional image (original image) received from one of the cameras.
For each pixel, 2 received from the other camera
The parallax between the two-dimensional image is obtained, the distance to the object is determined from the parallax, and the value of this distance is used as the value of the pixel.
Generate a two-dimensional image (distance image).

Usually, the generated distance image is passed to an image processing application program of a computer together with an original image, and is used for various processes. For example, by detecting a pixel group having a distance value within a predetermined range from the distance image and selecting the detected pixel group from the original image, only an image of an object existing within a predetermined range of distance from the original image is extracted. Are performed by the image processing application.

[0004]

The amount of calculation processing required to generate a distance image from an original image is extremely large. Therefore, a device that captures the original image from the camera and generates a range image at high speed is connected to the system bus of the computer (for example, in the form of an extended function board such as a video capture board), and the image processing application executes the distance image In many cases, an original image and a distance image are read from a generation device.

However, when a conventional range image generating apparatus captures an original image from a camera, it immediately outputs the original image, generates a range image from the original image in parallel with this, and outputs the range image. I do. Therefore, the output timing of the distance image is delayed from that of the original image by the processing time for generating the distance image. Therefore, the image processing application receives the original image and the distance image whose shooting time is shifted by the delay time, which lowers the accuracy of the image processing.

In an image processing application, it is theoretically possible to accurately grasp the delay time generated in the range image generation device and compensate for the delay time during image processing. However, in order to do so, the distance image generation device notifies the image processing application of the delay time, or such that the user knows the delay time in advance and sets it in the image processing application.
Extra laborious operations are required. Further, in the image processing application, since the work of performing the delay time compensation processing increases, the performance of the image processing, which is the original work, may be reduced. In the first place, since the image processing application and the distance image generation device are different systems and are usually designed and manufactured by different manufacturers, the compensation of the delay time generated in the distance image generation device is performed by the image processing. Expecting the application itself is unreasonable.

Accordingly, an object of the present invention is to simultaneously output an original image and a distance image that are temporally coincident with each other, so that an application using the original image and the distance image can be used to perform highly accurate image processing. An object of the present invention is to provide a range image generating device that enables the distance image generating device.

[0008]

A distance image generating apparatus according to the present invention comprises: an original image input means for inputting an original image; a distance image generating means for generating a distance image from the input original image; , An original image output unit that outputs the input original image, and a time when the original image output unit can output the original image substantially coincides with a time when the distance image generation unit can output the distance image. Control means for performing such control.

This distance image generating apparatus compensates for a delay time until a distance image can be output in the same apparatus, and can output an original image and a distance image at the same photographing time substantially simultaneously. Therefore, on the side that receives and processes the original image and the distance image, there is no need to consider any shift in the shooting time between the original image and the distance image. This is very convenient for the image processing application, especially when the distance image and the original image are passed from the distance image generation device to the image processing application of the computer.

In a preferred embodiment, the above control for matching the output timings of the distance image and the original image is performed for each pixel data. That is, the distance image and the original image can be accurately synchronized in pixel units. However, when the synchronization accuracy up to that point is not required, the output timing may be controlled in units of rasters, not in units of pixels, or may be controlled in units of frames.

[0011]

FIG. 1 shows a schematic system configuration of a main part of an image processing computer system incorporating a range image generating apparatus according to an embodiment of the present invention.

In this computer system 10,
CPU 13 and main memory (for example, S
A RAM 15, a display 17, an external storage device (for example, a hard disk drive) 19, and a distance image generation device (for example, a video capture board) 23 according to the present invention are connected. The image processing application 21 is installed in the external storage device 19 and the CP
U13 loads the image processing application 21 into the main memory 15 and executes it. The distance image generating device 23 is connected to a plurality of video cameras 25 (hereafter, two cameras are provided for simplification of description, but three or more cameras may be arranged) at different positions, The frame images at each point in time are fetched from the cameras 25, 25, and a distance image is generated by stereo vision. CPU
Reference numeral 13 takes in the distance image and the original image captured by any one of the cameras 25 from the distance image generation device 23 and performs desired processing when necessary when the image processing application 21 is executed.

The range image generating device 23 is configured to be able to output a synchronized range image and an original image (that is, to simultaneously output both images whose shooting times coincide with each other). Therefore, the image processing application 21 can automatically obtain the synchronized distance image and the original image simply by simply acquiring the distance image and the original image from the distance image generating device 23. Processing, for example, processing for extracting only an image of an object within a predetermined distance range from the original image based on the distance image can be performed with high accuracy.

Hereinafter, the distance image generating device 23 will be described in detail. FIG. 2 shows the configuration of the distance image generation device 23.

The distance image generating device 23 includes an original image input unit 3
1. Original image input memory unit (buffer memory unit) 33,
It includes a distance image generation unit 35, a distance image output memory unit 37, an original image output memory unit 39, and an original image output memory write control unit 41.

The original image input unit 31 includes two cameras 25,
An original image is taken in from 25 and temporarily stored in an original image input memory unit (buffer memory unit) 33. The distance image generation unit 35 reads the two frames of the original image captured from the two cameras 25 at the same time from the original image input memory unit (buffer memory unit) 33, and based on the two frames of the original image. A one-frame distance image is generated by stereo vision.

Each time the distance image generation unit 35 generates each pixel data of the distance image, the distance image generation unit 35 supplies a write signal to the distance image output memory unit 37, and stores each pixel data of the generated distance image in the distance image output memory. Write in part 37. Further, the distance image generation unit 35 supplies a control signal indicating the timing of writing each pixel data of the distance image to the distance image output memory unit 37 to the original image output memory write control unit 41.

Original image output memory write controller 41
Are the original image input memory unit (buffer memory unit) 3 at the timing according to the control signal from the distance image generation unit 35.
3 to read out the pixel data of the original image from the original image input memory unit (buffer memory unit) 33, and to give a write signal to the original image output memory unit 39, The data is written in the original image output memory unit 39. By controlling the writing timing of the original image by such a control signal from the distance image generating unit 35, the timing at which each pixel data of the original image is written to the original image output memory unit 39 and the generation based on the original image The timing at which each pixel data of the obtained distance image is written to the distance image output memory unit 37 coincides.

The CPU 13 of the computer executing the image processing application adjusts the timing when each pixel data of the distance image and the original image is written in the distance image output memory unit 37 and the original image output memory unit 39, respectively. At timing, each pixel data of the distance image and the original image is read from the distance image output memory unit 37 and the original image output memory unit 39, respectively. That is, the synchronized distance image and the original image are output from the distance image generation device 23.

Next, the operation of the distance image generator 35 will be described more specifically.

FIG. 3 shows a search space 100 to be subjected to a search operation performed when the distance image generation unit 35 generates a distance image.
Is shown.

The search space 100 includes cameras 25, 25
Is a three-dimensional space that is being photographed, and has spatial coordinate components in the X and Y directions corresponding to the vertical and horizontal directions of the original image, and the Z direction corresponding to the distance from the camera. Each (X, Y)
The coordinates correspond to each pixel of the original image, and the Z coordinates correspond to the distance that the distance image generation unit 35 is seeking.

FIG. 4 shows a flow of processing when the distance image generation unit 35 determines the distance Z of each (X, Y) coordinate from the parallax of two original images.

The distance image generation unit 35 includes one camera 25
The original image from the other camera is used as a reference image, the original image from the other camera is used as a reference image, and the upper left pixel (X0, Y0) of the reference image is used as a starting point and the lower right pixel (XMAX, YMA
With (X) as the end point, the pixels (X, Y) in the reference image are scanned in the raster scan order. First, in steps S1 and S2 in FIG. 4, the pixel of interest (X, Y) is set to the starting point (X0, Y
0), and a distance candidate Z for the first target pixel (X0, Y0) is initially set to a minimum distance Z0 in step S3. That is, the search space 100 of FIG.
Attention is paid to the first coordinates (X0, Y0, Z0).

When the distance candidate Z is set to Z0, the parallax d0 between the reference image and the reference image corresponding to the distance candidate Z0 is determined. Subsequently, the distance image generation unit 35 performs Step S4
To the image of the neighboring area centered on the target pixel (X0, Y0) of the reference image, and a pixel (X) on the reference image located at coordinates displaced from the target pixel (X0, Y0) by the parallax d0.
(0, Y0 + d0) is subjected to pattern matching with the image in the vicinity thereof, and the similarity between the two images is calculated.

Next, the distance image generation unit 35 executes step S
Proceeding to 5, the distance candidate Z is incremented to Z1.
That is, focusing on the coordinates (X0, Y0, Z1) of the next distance candidate Z1 in the search space 100, the parallax d1 corresponding to the distance candidate Z1 is determined. Then, step S4 is performed again, and the parallax d1 is calculated from the image of the neighboring area around the target pixel (X0, Y0) of the reference image and the target pixel (X0, Y0).
Pixel on the reference image (X0,
Y0 + d1) between the image of the vicinity and its center,
Pattern matching is performed, and the similarity between the two images is calculated. Hereinafter, similarly, the process of incrementing the distance candidate Z by one step (S5) and performing pattern matching between the reference image and the reference image to obtain the degree of similarity (S4) is performed by the maximum value of the distance candidate Z. Repeat until ZMAX.

When the similarity between the reference image and the reference image has been calculated for all the distance candidates Z from the minimum distance Z0 to the maximum distance ZMAX for the pixel of interest (X0, Y0), the next step is Proceed to S6 and Z0 to ZMAX
Is determined, the distance candidate with the highest similarity is obtained, and the distance candidate with the highest similarity is determined as the target pixel (X0, Y
The distance image data is written in the distance image output memory unit 37 as the distance image data 0).

When performing this step S6, the distance image generation unit 35 sends a control signal to the original image output memory write control unit 41, and the original image output memory write control unit 41
Performs the step S7 in response to the control signal, and outputs the data (that is, the original image data) of the target pixel (X0, Y0) of the reference image (or both the reference image and the reference image) for outputting the original image. The data is written in the memory unit 39. Therefore, step S
The writing of the distance image data of No. 6 and the writing of the original image data of step S7 are performed at substantially the same timing.

Thereafter, the distance image generation unit 35 moves the target pixel (X, Y) one by one in the raster scan order (steps S8 and S9), and proceeds to step S3.
Through the processes of S6 to S6, the distance image data of each target pixel (X, Y) is obtained and written to the distance image output memory 37. In synchronization with this, the original image output memory write control unit 41 writes the data of each pixel of interest (X, Y) of the reference image into the original image output memory unit 39 (S7).

The series of processes described above is performed for the last pixel (XMA).
(X, YMAX), the processing for one frame is completed. Then, the same processing is repeated for the next frame.

By the above operation, the synchronized distance image and the original image are output from the distance image generating device 23.

FIG. 5 shows another example of the configuration of the distance image generating device 23.

In this configuration example, when the original image input unit 23 captures the original image from the cameras 25, 25, the original image (both the reference image and the reference image) is passed to the distance image generation unit 53, and the original image is received. The image (only the reference image or both the reference image and the reference image) is written in the first original image output memory unit 59.

The distance image generation unit 53 sequentially generates pixel data of the distance image in the same procedure as described with reference to FIG. 4, and outputs the first distance image output memory unit (buffer memory unit) 55.
Write to. At the same time, the distance image generation unit 53 outputs a periodic control signal indicating the output timing of each pixel data to the image output memory write control unit 63.

The image output memory write control unit 63
Each pixel data of the distance image in the first distance image output memory unit (buffer memory unit) 55 is read out from the first distance image output memory unit (buffer memory unit) 55 in synchronization with each cycle of the control signal. And writes the pixel data of the original image in the first original image output memory unit (buffer memory unit) 59 into the first distance image output memory unit 57.
The data is read from the original image output memory unit (buffer memory unit) 59 and written into the second original image output memory unit 61. Therefore, the synchronized distance image and the original image are stored in the second distance image output memory unit 57 and the second original image output memory unit 6.
1 can be output through each.

The embodiment of the present invention has been described above.
The above embodiments are merely examples for describing the present invention, and are not intended to limit the present invention to these embodiments only. Therefore, the present invention can be implemented in various modes other than the above-described embodiment without departing from the gist thereof. For example, in the above-described embodiment, the distance image and the original image are written in the output memory in synchronization with each other on a pixel basis. However, the distance image and the original image are not necessarily in pixel units, and synchronization is performed on each raster unit in the frame. (For example, a control signal is issued each time each raster data of the distance image is generated or written to the output memory, and the data of each raster of the original image is written to the output memory in synchronization with the control signal.) Alternatively, synchronization may be performed on a frame basis (for example, a control signal may be issued every time each frame of the range image is generated or written in the output memory,
The data of each frame of the original image may be written to the output memory in synchronization with the timing of the control signal. Alternatively, if the cycle of generating each frame of the distance image is predetermined, a control signal is issued at a timing at which the first frame of the distance image can be output, and the control signal is used as a trigger, and thereafter, the above-described control is performed. Each frame of the original image and the distance image may be automatically written into the output memory at intervals.

In the above embodiment, the distance image generation device is incorporated in the computer in the form of a function expansion board, and outputs the distance image and the original image to an image processing application which is software of the computer. However, this need not be the case, and the range image generation device may be provided as software, or may be configured as a circuit connected to or incorporated in an image processing device separate from the computer.

[Brief description of the drawings]

FIG. 1 is a schematic system configuration diagram of a main part of an image processing computer system incorporating a range image generation device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a distance image generation device 23.

FIG. 3 is a perspective view of a search space 100 that is a target of a search operation performed when a distance image generation unit 35 generates a distance image.

FIG. 4 is a flowchart of a process when a distance image generation unit 35 obtains distance data.

FIG. 5 is a block diagram showing another configuration example of the distance image generation device 23.

[Explanation of symbols]

Reference Signs List 10 image processing computer system 13 CPU 21 image processing application 23 distance image generation device 25 video camera 31, 51 original image input unit 33 original image input memory unit (buffer memory unit) 35, 53 distance image generation unit 37 for distance image output Memory unit 39 Original image output memory unit 41 Original image output memory write control unit 55 First distance image output memory unit (buffer memory unit) 57 Second distance image output memory unit 59 First original image output memory unit (Buffer memory unit) 61 Second original image output memory unit 63 Image output memory write control unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Akihiro Takiguchi 1200 Manda, Hiratsuka-shi, Kanagawa Prefecture F-term (reference) 2F065 AA04 AA06 AA07 FF05 FF26 JJ03 JJ05 JJ26 QQ24 QQ29 QQ38 RR03 UU05 5B057 BA19 CB13 CH11 DA07 DA16 DB02 DB03 DB08 DC03 DC08 5C082 AA27 BA20 BA41 BB15 BB26 BB29 CB01 DA87 MM05

Claims (3)

[Claims] 1. An original image input means for inputting an original image, a distance image generating means for generating a distance image from the input original image, a distance image output means for outputting the generated distance image, and an input original image An original image output means for outputting a time when the original image output means can output the original image,
Control means for controlling the distance image output means to substantially coincide with a time when the distance image can be output. 2. The distance image generating apparatus according to claim 1, wherein the distance image generating apparatus is coupled to a computer and configured to output the distance image and the original image to an image processing application of the computer. 3. An original image inputting step of inputting an original image, a distance image generating step of generating a distance image from the input original image, a distance image outputting step of outputting the generated distance image, An output step of outputting the original image, and a control step of controlling the time at which the original image can be output at the original image output step to substantially coincide with the time at which the distance image can be output at the distance image output step. Provided with a distance image generation method.