JP2002077941A - Apparatus and method for generating depth image as well as computer readable recording medium recording program to execute the method in computer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an apparatus and a method for generating a depth mage for generating the depth image corresponding to a two-dimensional still picture fetched from a video image not specifying photographing conditions. SOLUTION: The method for generating the depth image comprises the steps of storing a dynamic image in a dynamic image memory 12, fetching one two-dimensional still picture for constituting the dynamic image from the memory 12 by a still picture acquiring unit 14, fetching, for example, two-dimensional still picture before and after the two-dimensional still picture in timing from the memory 12 by a still, picture selector 16, and relating the distance between an area on an object and a camera to each pixel of the two-dimensional still picture by a depth image calculator 18 to thereby generate a depth image. Thus, a three-dimensional shape information of the subject and the depth image are obtained.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional restoration technique for automatically restoring a three-dimensional shape of a subject from an image captured by a camera, and more particularly to a technique for restoring the three-dimensional shape of a subject indicated by each pixel in the image. The present invention relates to a technique for outputting a three-dimensional shape as depth information for recording a distance between a region and a camera.

[0002]

2. Description of the Related Art Research on a three-dimensional reconstruction technique for automatically restoring a three-dimensional shape of a subject from an image captured by a camera began in the 1970's. Theories have already been established in the 1980s. The best known method is to use a pair of still images taken with a stereo camera. At present, a new research topic is how to photograph a subject or how to apply three-dimensional reconstruction technology in order to perform stable three-dimensional reconstruction.

[0003] Representative examples of these three-dimensional reconstruction techniques include, for example, a "three-dimensional shape recognizing apparatus" disclosed in Japanese Patent Application Laid-Open No. 5-149727 and a "3-dimensional shape reconstructing method" disclosed in Japanese Patent Application Laid-Open No. 5-314243. And JP-A-6-10202.
7, "3D shape restoration method"
An "object detection method and system" disclosed in JP-A-93954 and a "viewpoint / object point position measurement method and apparatus" disclosed in JP-A-10-239054 are known.

A "three-dimensional shape recognition apparatus" disclosed in Japanese Patent Application Laid-Open No. 5-149727 is a device for irradiating an object with slit light and restoring a three-dimensional shape from its reflection pattern. When the reflection pattern cannot be properly captured, the camera is moved to obtain a three-dimensional shape more reliably than before.

[0005] In addition, a "three-dimensional shape restoration method" disclosed in Japanese Patent Laid-Open No. 5-314243 discloses a method for obtaining two images on which a three-dimensional restoration process is performed. This is a method in which an image of an eye is taken, and then the camera is moved, and the direction of the camera is redirected toward the object to take a second image. In particular, this method realizes more accurate three-dimensional reconstruction than the conventional imaging method that uses only the movement of the camera and does not rotate by using the numerical values of the movement distance and the rotation angle of the camera in the three-dimensional reconstruction processing. Things.

The "three-dimensional shape restoration method" disclosed in Japanese Patent Application Laid-Open No. 6-102027 is disclosed in
When a camera is moved to obtain an image to be used for three-dimensional reconstruction as in the method disclosed in Japanese Patent No. 49727 and Japanese Patent Laid-Open No. 5-314243, luminance information of the image is obtained by moving the camera. By detecting the direction in which the change in the feature amount is greatest and moving the camera in that direction, the three-dimensional shape is obtained more accurately than in the past.

[0007] Further, the "object detection method and system" disclosed in Japanese Patent Application Laid-open No. Hei 10-93954 acquires two images taken at different times by a camera moving on a rail, and acquires the two acquired images. It measures the distance from the camera to the object in the image using the image (similar to estimation of a three-dimensional shape), and detects objects other than the permanent object.

[0008] Further, a "viewpoint / object point position measuring method and apparatus" disclosed in Japanese Patent Application Laid-Open No. 10-239054 discloses a set of stereo images of a subject (a pair of stereo images taken from two cameras whose relative positions are known). The three-dimensional shape of the subject is measured by using a plurality of images).

[0009] In addition to the technology disclosed in the above-mentioned patent publication, a video taken by a video camera is input,
Various software capable of outputting a three-dimensional shape of a subject are provided. For example, OGIS-RI
The provided product "VirtualGAIA" can restore a three-dimensional shape from images taken with a home video camera or frames taken with a digital camera or the like. In particular, this “Virtual GAIA” digitally inputs a video image of a subject while moving a video camera so as to turn around the subject, and converts three-dimensional data into VRML (Virtual Reality).
(Modeling Language) format.

[0010]

However, the prior art disclosed in the above-mentioned patent publication is intended for shooting under specific conditions where the position and moving speed of the camera are known, and the shooting situation is There has been a problem that it is not possible to use an indefinite ordinary video image. In particular, in recent years, inexpensive and easy-to-operate video cameras that can be used even at home have been sold, and video images, that is, a plurality of continuously photographed still images that have been continuously shot can be easily obtained by anyone. In consideration of the above, it is difficult to satisfy the measurement of the three-dimensional shape of the subject with the above-described conventional technology.

Further, the above-mentioned software product “Vir
Tual GAIA "can use video images shot at home, but requires that the camera be moved smoothly during shooting. There was a restriction that only panning or tilting should not be performed, and there should be no part where the camera is moved greatly.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and is intended for video images having portions where the camera is stationary, only panning or tilting is performed, or the camera is largely moved. Also, a depth image generating apparatus capable of calculating a three-dimensional shape of a subject and using the three-dimensional shape to generate a depth image corresponding to a two-dimensional still image extracted from a video image, and An object is to obtain a depth image generation method.

[0013]

Means for Solving the Problems The above-mentioned problems are solved,
In order to achieve the object, in the depth image generation device according to the present invention, a moving image storage means for storing a moving image photographed by a camera, and a first moving image stored in the moving image storage means. A still image acquisition unit for extracting a two-dimensional still image; and a first two-dimensional still image from a moving image stored in the moving image storage unit based on the two-dimensional still image extracted by the still image acquisition unit. Corresponds to a still image selecting means for selecting and extracting a different second two-dimensional still image, and, based on the first and second two-dimensional still images, corresponding to a distance between the subject and the camera to a pixel forming the subject And a depth image calculation means for calculating a depth image.

According to the present invention, the moving image is stored in the moving image storing means, the still image obtaining means takes out the first two-dimensional still image constituting the moving image from the moving image storing means, and the still image selecting means stores the first two-dimensional still image. For example, a second two-dimensional still image that is temporally before or after the first two-dimensional still image is extracted from the moving image storage unit, and the depth image calculation unit outputs the first and second two-dimensional still images. Since the depth image is generated by associating the distance between the area on the subject and the camera with each pixel of the above, three-dimensional shape information and the depth image of the subject can be obtained.

[0015] In the depth image generating apparatus according to the next invention, in the above-mentioned invention, the still image selection means may include:
As the second two-dimensional still image, a two-dimensional still image having a predetermined parallax with the first two-dimensional still image is selected.

According to the present invention, the still image selecting means selects the second two-dimensional still image in consideration of the parallax for the depth image calculating means to calculate a sufficiently accurate depth image. The depth image calculation means can generate an accurate depth image.

In the depth image generating apparatus according to the next invention, in the above-mentioned invention, the depth image calculation means may include a position of the camera when each of the first and second two-dimensional still images is photographed. Alternatively, the orientation is calculated, and the still image selection unit performs the second image processing based on the position or orientation of the camera calculated by the depth image calculation unit.
Is re-selected.

According to the present invention, the still image selecting means sets the second two-dimensional still image based on the position or orientation of the camera obtained by the depth image calculating means with respect to the temporarily selected second two-dimensional still image. Since the two-dimensional still image is reselected, the depth image calculation unit can generate a depth image based on the reselected second two-dimensional still image.

In the depth image generating apparatus according to the next invention, in the above-mentioned invention, the depth image calculation means may include a camera position when each of the first and second two-dimensional still images is photographed. And the orientation, and the still image selection unit calculates the second image based on the camera position and orientation calculated by the depth image calculation unit.
Is re-selected.

According to the present invention, the still image selecting means sets the second two-dimensional still image based on the position and orientation of the camera obtained by the depth image calculating means with respect to the temporarily selected second two-dimensional still image. Since the two-dimensional still image is reselected, the depth image calculation unit can generate a depth image based on the reselected second two-dimensional still image.

In the depth image generating apparatus according to the next invention, in the above invention, the depth image storage means for storing the depth image generated by the depth image calculation means and the depth image storage means. Depth moving image generating means for generating a depth moving image by connecting the depth images.

According to the invention, furthermore, a depth image storage means for storing the depth image generated by the depth image calculation means, and a depth moving image for generating a moving image by connecting the depth images stored in the depth image storage means. Image generating means, it is possible to obtain depth moving image data.

In the depth image generating method according to the next invention, a moving image storing step of storing a moving image photographed by a camera, and a first two-dimensional image based on the moving image stored in the moving image storing step. The first two-dimensional still image is different from the moving image stored in the moving image storing step based on the still image obtaining step of extracting a still image and the two-dimensional still image extracted in the still image obtaining step A still image selecting step of selecting and extracting a second two-dimensional still image, and associating a pixel forming the subject with a distance between the subject and the camera based on the first and second two-dimensional still images. And a depth image calculation step of calculating the configured depth image.

According to the present invention, a moving image is stored, the first two-dimensional still image constituting the stored moving image is taken out, and the first two-dimensional still image is, for example, temporally back and forth. A second two-dimensional still image to be extracted is extracted from the moving image, and a depth image is generated by associating each pixel of the first and second two-dimensional still images with the distance between the region on the subject and the camera. Therefore, three-dimensional shape information and a depth image of the subject can be obtained.

In a depth image generating method according to the next invention, in the above-mentioned invention, the still image selecting step includes:
As the second two-dimensional still image, a two-dimensional still image having a predetermined parallax with the first two-dimensional still image is selected.

According to the present invention, the second two-dimensional still image is selected in consideration of the parallax for calculating a sufficiently accurate depth image. Therefore, in the depth image calculation step, an accurate depth image is calculated. Can be generated.

[0027] In the depth image generating method according to the next invention, in the above invention, the depth image calculation step further includes the step of calculating a depth of the camera when each of the first and second two-dimensional still images is photographed. Calculate the position or orientation, the still image selection step, based on the position or orientation of the camera calculated by the depth image calculation step,
The second two-dimensional still image is reselected.

According to the present invention, the second two-dimensional still image is reselected based on the position or orientation of the camera obtained in the depth image process for the temporarily selected second two-dimensional still image. Therefore, a depth image can be generated based on the reselected second two-dimensional still image.

[0029] In the depth image generating method according to the next invention, in the above invention, the depth image calculation step further includes the step of calculating a depth of the camera when each of the first and second two-dimensional still images is photographed. Calculate the position and orientation, the still image selection step, based on the position and orientation of the camera calculated by the depth image calculation step,
The second two-dimensional still image is reselected.

According to the present invention, the second two-dimensional still image is reselected based on the position and orientation of the camera obtained by the depth image process for the temporarily selected second two-dimensional still image. Therefore, a depth image can be generated based on the reselected second two-dimensional still image.

In the depth image generating method according to the next invention, in the above invention, a depth image storing step of storing the depth image generated by the depth image calculating step and a depth image storing step of storing the depth image are stored by the depth image storing step. A depth moving image generating step of generating a depth moving image by connecting the depth images.

According to the present invention, since the depth image generated in the depth image calculation step is stored and the stored depth images are connected to generate a moving image, it is possible to obtain depth moving image data. Become.

In a computer-readable recording medium according to the next invention, a program for causing a computer to execute the above-described depth image generating method is recorded, so that the program becomes computer-readable. The operation of the generation method can be realized by a computer.

[0034]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of a depth image generating apparatus and a depth image generating method according to the present invention will be described below in detail with reference to the drawings. The present invention is not limited by the embodiment.

Embodiment 1 First, a depth image generation device and a depth image generation method according to the first embodiment will be described. FIG. 1 is a block diagram illustrating a schematic configuration of the depth image generation device according to the first embodiment. In FIG. 1, a depth image generating apparatus 10 includes a moving image storage unit 12 that inputs and stores moving image data generated by video camera shooting, and a moving image data of the moving image data stored in the moving image storage unit 12. A still image acquisition unit 14 that focuses on one frame (image frame) constituting image data and acquires it as a still image, and stores another frame based on the frame acquired by the still image acquisition unit 14 as a moving image A still image selecting unit 16 that selects from the unit 12 and obtains a still image, and a depth image calculating unit 18 that calculates and generates a depth image from the still images obtained by the still image obtaining unit 14 and the still image selecting unit 16, respectively. And is provided.

The depth image generating device 10 can be replaced by a computer. That is,
The moving image storage unit 12 described above is stored in a RAM on a computer.
(Random Access Memory) or a magnetic disk storage device.
The processes executed by the still image selection unit 16 and the depth image calculation unit 18 can also be realized by computer programs.

Next, the operation of the depth image generating apparatus according to the first embodiment will be described. FIG. 2 is an explanatory diagram illustrating a shooting state when the video camera 202 shoots an object 201 having a rectangular parallelepiped shape. As illustrated in FIG. 2, the video camera 202 moves as indicated by a dotted line while capturing an image with the camera directed substantially vertically to the front of the subject 201.

FIG. 3 is an explanatory diagram when the photographing state shown in FIG. 2 is viewed from above. As shown in FIG. 3, it is assumed that the video camera 202 points the camera in the direction of the arrow shown on the points p1 to p12 and moves on a dotted line connecting the points p1 to p12. In particular, here, it is assumed that the video camera 202 has taken a picture with each of the points p1 to p12 as a photographing position. That is, a total of 12 frames are photographed. In FIG. 3,
The contents of the frames shot at the points p1, p2, p4, p6 and p12 are the frames 301, 302 and 3 respectively.
03, 304 and 305. In the following description, for convenience, a frame shot at the point pi is referred to as a frame pi.

The moving image composed of the frames p1 to p12 thus photographed is stored in the moving image storage unit 12. Then, the still image acquisition unit 14 transmits the moving image storage unit 1
For example, frame p4 is obtained from frame 2 and a still image corresponding to the frame is output, and still image selection unit 16 sends second frame before and after frame p4 from moving image storage unit 12, ie, frame p2.
And a frame p6 are selected to output still images corresponding to these frames.

Still picture acquisition section 14 and still picture selection section 16
Corresponding to the frames p4, p2 and p6 output from
The still images 303, 302, and 304 are input to the depth image calculation unit 18. The depth image calculation unit 18
A depth image is generated based on the still images 303, 302, and 304 and output as depth image data. FIG. 4 is a diagram illustrating an example of a depth image generated by the depth image calculation unit 18. In FIG. 4, the depth image is composed of multi-level values for each pixel of the image such that the distance from the video camera at the time of shooting is whiter and the closer the distance to the video camera is black. That is, in the example of FIG.
It is shown that the distance is small for the surface facing the video camera No. 01, and the distance is gradually increased for the other surfaces.

Next, the processing in the depth image calculation unit 18 will be described. In particular, here, a process from inputting three still images to outputting depth image data will be described. FIG. 5 is a flowchart showing the processing in the depth image calculation unit 18. First, the depth image calculation unit 18 extracts about 200 to 300 pixels called feature points for each of the three still images at different photographing positions (step S101). The feature point extraction refers to a process of paying attention to color information of pixels constituting a still image and automatically extracting pixels having color information greatly different from surrounding pixels. Each pixel corresponds to one point on the surface of the subject.

FIG. 6 is an explanatory diagram for explaining extraction of a feature point. As shown in FIG. 6, for example, the feature points are extracted as pixels located at the apex angles of the subject 201 in the frames 302, 303, and 304, that is, the rectangular parallelepiped.

Next, the depth image calculation unit 18 performs a process called feature point matching for associating feature points extracted for each of the three still images (step S102). This association is performed between feature points indicating the same one point on the surface of the subject. FIG.
(A) is an explanatory view for explaining feature point matching. In the feature point matching, as shown in FIG. 7A, for example, between the object 201 in the frames 302, 303, and 304, that is, between the rectangular parallelepipeds, the feature points, that is, the pixels at the apex angles at the same position are associated with each other. Can be

In this feature point matching,
At the same time, the position and orientation of the camera when the still image was captured are calculated (step S102). FIG. 7B is an explanatory diagram for explaining calculation of the position and orientation of the camera.
As shown in FIG. 7B, the position of the camera is specified by, for example, expanding the shooting position viewed from above shown in FIG. 3 on the XY coordinate axes, and the direction of the camera is a vector represented using the coordinate axes. Specified as

Subsequently, the depth image calculation unit 18 calculates the distance from the camera to each feature point from the camera position and orientation obtained in step S102 based on the principle of triangulation (step S103). FIG. 8 is an explanatory diagram for explaining calculation of the distance from the camera to each feature point. As shown in FIG. 8, the camera distance is specified using the information on the feature points shown in FIG. 6 and the information on the camera position on the XY coordinate axes shown in FIG. 7B.

Then, the depth image calculation unit 18 restores the three-dimensional structure of the subject 201 by appropriately connecting the feature points to form a surface, and obtains depth information about a still image (step S104). FIG. 9 is an explanatory diagram for explaining generation of a depth image. For example, as shown in FIG. 9, a depth image is generated by forming a surface by connecting feature points to a still image corresponding to a frame 303 to obtain a three-dimensional shape of a subject and to obtain the above-described feature. Finally, a depth image as described in FIG. 4 is calculated from the distance between the point and the camera.

The generation of the depth image by the depth image calculation unit 18 is only one of the constituent means in the present invention, and the same processing method is widely known, so that the details of the processing will be omitted. Known literature describing the generation of depth images, for example, "PA Beardsley, A. Zisserman,
DW Murray: "Sequential Updating of Projective
and Affine Structure from Motion ", International J
ournal of Computer Vision, 23 (3), pp235-259, 199
There is 7 ".

What is required in the generation of the depth image is that three still images having an appropriate parallax are input. The parallax refers to a difference between images caused by different photographing positions of a camera when photographing a three-dimensional structural object with a camera. For example, FIG.
In the example shown in (1), since the position of the camera is different in every frame, parallax occurs between all the frames at the points p1 to p12.

However, in order to obtain an accurate depth image in the above-described known depth image generation processing, the parallax may not be too large or too small. This is because, if the parallax is too large, the difference between the images becomes large, so that the difference between the extracted feature points also becomes large, and the association of the feature points is often unsuccessful. On the other hand, if the parallax is too small, the calculation error when calculating the distance from the camera to one point on the subject corresponding to the associated feature point by the principle of triangulation increases.

For this reason, in the depth image generating apparatus and the depth image generating method according to the first embodiment, in FIG. 3, for example, the frame to be selected for calculating the depth image for the frame at the point p4 has too small parallax. Point p3
Or not p5, and the point p1, where the parallax is too large,
Instead of p7 to p12, points p2 and p6 are set.

As described above, according to the depth image generating apparatus and the depth image generating method according to the first embodiment, the moving image data is stored in the moving image storage unit 12, and the still image acquisition unit 14 stores the moving image data. One still image (frame) constituting the moving image data is extracted from the unit 12, and the still image selecting unit 16 further extracts a still image temporally before or after the still image from the moving image storage unit 12, and Since the image calculation unit 18 generates a depth image by associating each pixel of the two-dimensional still image with the distance between the region on the subject and the camera, the moving speed of the camera with respect to the source moving image data It is possible to accurately obtain three-dimensional shape information and a depth image of a subject without requiring shooting conditions such as a moving mode and a moving mode.

Further, the still image selecting unit 16 extracts the still image from the moving image storage unit 12 in consideration of the parallax for the depth image calculating unit 18 to calculate a sufficiently accurate depth image. The accuracy of the obtained depth image can be improved.

In the first embodiment described above, the number of still images input to the depth image calculation unit 18 is three. However, even when the number of still images is two, the same operation and effect as described above can be enjoyed. be able to.

Embodiment 2 Next, a depth image generation device and a depth image generation method according to the second embodiment will be described. The depth image generation device and the depth image generation method according to the second embodiment are characterized in that the still image selection unit 16 reselects a still image based on the calculation result by the depth image calculation unit 18. Other configurations and operations are the same as those of the first embodiment.

FIG. 10 is a block diagram showing a schematic configuration of the depth image generating apparatus according to the second embodiment. In addition,
In FIG. 10, portions common to FIG. 1 are denoted by the same reference numerals and description thereof is omitted. The depth image generation device 20 shown in FIG. 10 differs from FIG. 1 in that data is transmitted and received between the still image selection unit 16 and the depth image calculation unit 18.

FIG. 11 is an explanatory diagram for explaining a photographing state of the video camera 202 when photographing an object 201 having a rectangular parallelepiped shape. In the example shown in FIG. 3, the video camera 202 moves from left to right at substantially the same distance with the orientation fixed, but in the example shown in FIG. An example where the direction / orientation is not constant is shown. In general, a shooting state when a user shoots a subject using a video camera is considered to be a state as shown in FIG.

In the depth image generation device 20 according to the second embodiment, first, as in the first embodiment, the still image selection unit 16 determines the frame of the moving image acquired by the still image acquisition unit 14, The second frame before and after the still image,
It is obtained from the moving image storage unit 12 and output as a still image.
For example, in FIG.
Is obtained, the still image selection unit 16 selects the frames p1 and p5 and outputs them as a still image.

Similarly, even when the still image obtaining unit 14 selects the frame p4 or the frame p5, the still image selecting unit 16
A still image having an appropriate parallax suitable for the processing of the depth image calculation unit 18 can be output.

However, in FIG. 11, the photographing time interval is different between the frame p5 two frames before the frame p7 and the frame p9 two frames after the frame p7.
4, even if the still image selection unit 16 selects a frame by the method described in the first embodiment, a still image having an appropriate parallax cannot be output.

That is, according to the still image selecting method in the still image selecting section 16 based on the first embodiment, the frame p7
Are selected, the frames p5 and p9 are selected. At this time, the video camera 202 in the section from p7 to p11
Since the moving speed of the frame p is low, the frame p9 is not suitable because the parallax is too small with respect to the frame p7. In the example shown in FIG. 11, for the frame p7, it is more appropriate to select the frame p11 instead of the frame p9.

Further, the frame p1 is obtained by the still image acquisition unit 14.
When 5 is acquired, the frames p13 and p17 are not appropriate because the parallax is too large, and the frames p12 and p16 must be selected instead. Similarly, p
When frame 21 is obtained, frames p18 and p25 are obtained. When frame p22 is obtained, frames p19 and p26 are obtained.
Each must be selected by the still image selection unit 16.

Therefore, in the second embodiment, as a first condition for obtaining a still image having an appropriate parallax, as a first condition, the video camera 202 at the time when the still image acquired by the still image acquiring section 14 is photographed is taken. Consider the distance between the position and the position of the video camera 202 when the still image selected by the still image selection unit 16 is captured. FIG. 12 is a flowchart illustrating a process of selecting a still image based on the first condition, that is, a camera position, in the depth image generation device according to the second embodiment.

First, the still image selection unit 16 sets the initial value of the variable i to 1 (step S201), and
Tentatively selects the i-th previous frame of the acquired still image (frame), that is, the immediately preceding still image (frame) in the initial state (step S202). Note that this provisional selection corresponds to a still image selection process by the still image selection unit 16 of the first embodiment.

The temporarily selected frame and the still image (frame) acquired by the still image acquiring unit 14 are input to the depth image calculating unit 18. Thereby, the depth image calculation unit 1
8, when the depth image is calculated, the position (inter-camera distance) of the video camera 202 when each still image is captured is calculated (step S20).
3). The still image selection unit 16 receives the inter-camera distance from the depth image calculation unit 18 and checks whether the distance is equal to or greater than the minimum distance α and equal to or less than the maximum distance β (step S204). Since this distance may not be too small or too large, the minimum distance α and the maximum distance β are determined as empirical values.

If the inter-camera distance is equal to or greater than the minimum distance α and equal to or less than the maximum distance β in step S204, the i-th previous frame from the still image (frame) acquired by the still image acquiring unit 14, that is, the temporary frame in step S202. The selected still image (frame) is input to the depth image calculation unit 18 again (step S205). If the inter-camera distance is not equal to or greater than the minimum distance α and equal to or less than the maximum distance β in step S204, the still image selecting unit 16 checks whether the i-th previous frame is the first frame of the moving image (step S204).
206).

In step S206, if the i-th previous frame is not the first frame of the moving image, i is incremented by 1 and the processing from step S202 is repeated (step S207). On the other hand, if the i-th previous frame is the first frame of the moving image, it is determined that the selection of the frame has failed and the processing ends. If the values of α and β are not fixed values and a frame satisfying the condition is not found, α may be decreased and β may be increased, and the processing from step S202 may be repeated.

The frame behind the frame obtained by the still image obtaining unit 14 is also selected by the same method as described above.
It is input to the depth image calculator 18. As described above, the still image selection unit 16 selects a still image so as to satisfy the first condition in consideration of the camera positions and inputs the still image to the depth image calculation unit 18, so that, for example, the frames p3 to p3 illustrated in FIG. A correct frame can be selected for p21.

Next, as a second condition for obtaining a still image having an appropriate parallax, the orientation of the video camera 202 at the time when the still image acquired by the still image acquiring section 14 is photographed, and the selection of the still image Consider a difference from the orientation of the video camera 202 when the still image selected by the unit 16 is captured. This is because if the orientation of the video camera that shot the still image is different, the position of the subject in the still image and the range of the subject in the still image differ greatly, and there are many feature points where matching processing cannot be performed. This is for avoiding the situation that occurs. Here, it is assumed that the directions of the two video cameras are regarded as two vectors extended from the origin, and an effective maximum angle γ formed by the two vectors is determined.

FIG. 13 is a flowchart showing a process of selecting a still image based on the second condition, that is, the direction of the camera, in the depth image generating apparatus according to the second embodiment. First, the still image selection unit 16 sets the initial value of the variable i to 1 (step S301), and i-th frame before the still image (frame) acquired by the still image selection unit 16, that is, one frame before in the initial state. Is temporarily selected (step S302). Note that this provisional selection corresponds to a still image selection process by the still image selection unit 16 of the first embodiment.

The temporarily selected frame and the still image (frame) obtained by the still image obtaining unit 14 are input to the depth image calculating unit 18. Thereby, the depth image calculation unit 1
When the calculation of the depth image is performed in step 8, the orientation of the video camera 202 when each still image is captured is calculated (step S303). The still image selection unit 16 receives the information about the above-mentioned camera orientation from the depth image calculation unit 18 and
The direction of 2 is regarded as two vectors extended from the origin, and it is checked whether the angle formed by the two is less than or equal to the maximum angle γ (step S304).

If the camera orientation is equal to or smaller than the maximum angle γ in step S 304, the i-th previous frame from the still image (frame) obtained by the still image obtaining unit 14, that is, the still image temporarily selected in step S 302 (Frame),
The data is input to the depth image calculation unit 18 again (step S305). On the other hand, if the camera orientation is larger than the maximum angle γ in step S204, the still image selection unit 1
6 checks whether the i-th previous frame is the first frame of the moving image (step S306).

If the i-th previous frame is not the first frame of the moving image in step S306, i is incremented by 1 and the processing from step S302 is repeated (step S307). On the other hand, if the i-th previous frame is the first frame of the moving image, it is determined that the selection of the frame has failed and the processing ends. If the value of γ is not fixed and a frame satisfying the condition is not found, γ may be increased and the processing from step S302 may be repeated.

The frame behind the frame obtained by the still image obtaining unit 14 is also selected by the same method as described above.
It is input to the depth image calculator 18. As described above, the still image selection unit 16 selects a still image so as to satisfy the second condition in consideration of the direction of the camera and inputs the still image to the depth image calculation unit 18, so that, for example, the frames p17 to p17 shown in FIG. p22
, The correct frame can be selected.

Note that the still image selection section 16 performs the first
The still image may be selected so as to satisfy only one of the first condition and the second condition, or the first condition and the second condition may be selected.
May be selected so as to satisfy both of the conditions.

As described above, according to the depth image generating apparatus and the depth image generating method according to the second embodiment, the moving image data is stored in the moving image storage unit 12, and the still image obtaining unit 14 stores the moving image. A still image (frame) constituting moving image data is extracted from the unit 12, a still image selecting unit 16 extracts a still image from the moving image storage unit 12 based on the still image, and converts the extracted still image into a depth image. A more optimal still image is selected according to the position and orientation of the camera obtained by inputting to the arithmetic unit 18, and the depth image arithmetic unit 18 assigns an area on the subject to each pixel of these two-dimensional still images. Since the depth image is generated by associating the distance with the camera, the three-dimensional image of the subject can be obtained without requesting the shooting conditions such as the moving speed and moving form of the camera for the source moving image data. Original shape information and the depth image can be obtained more accurately.

The still image selection unit 16 reselects a still image in accordance with the position and orientation of the camera obtained by the depth image calculation unit 18 in consideration of the parallax for calculating an accurate depth image. Therefore, the accuracy of the finally obtained depth image can be improved.

In the second embodiment described above, the number of still images input to the depth image calculation unit 18 is three. However, even when the number of still images is two, the same operation and effect as described above can be enjoyed. be able to.

Also, in the second embodiment, in order to obtain a still image having an appropriate parallax, the video camera 202 when the obtained still image is shot and the video camera 202 when the selected still image is shot As a method of using the position of the video camera 202, the distance between the video cameras was evaluated.
Other methods may be used. For example, a maximum value and a minimum value may be defined for an included angle of a line segment extending from two video cameras to a specific position (such as the center of gravity) of a subject.

Further, in the second embodiment, the frame is selected by repeating the process until a frame satisfying the condition is found. Instead, for example, the position of the video camera and the position of the video camera are determined for all 10 frames before and after. It is also possible to calculate the orientation and select a frame shot under conditions close to the most ideal position and orientation.

Embodiment 3 Next, a depth image generation device and a depth image generation method according to the third embodiment will be described. The depth image generation device and the depth image generation method according to the third embodiment are characterized in that depth image data generated by a depth image calculation unit is stored, and depth moving image data is generated based on the stored depth image data. And Other configurations and operations are the same as those of the second embodiment.

FIG. 14 is a block diagram showing a schematic configuration of the depth image generating apparatus according to the third embodiment. In addition,
In FIG. 14, portions common to FIG. 10 are denoted by the same reference numerals, and description thereof will be omitted. 14 is different from FIG. 10 in that a depth image storage unit 32 storing depth image data generated by the depth image calculation unit 18 and a depth image storage unit 32
And a depth moving image generation unit 34 that generates depth moving image data based on the depth image data stored in the storage unit.

Next, the operation of the depth image generation device according to the third embodiment will be described. Here, it is assumed that the moving image has been captured according to the moving path of the video camera shown in FIG. The depth image calculation unit 18
The operation up to the generation of the depth image data is as described in the second embodiment, and a description thereof will be omitted here.

FIG. 15 is a flowchart showing a process of generating a depth moving image in the depth image generation device according to the third embodiment. First, the depth image generation device 30 sets the initial value of the variable i to 1 (step S4).
01). Then, the still image acquisition unit 14 stores the moving image storage unit 1
The frame pi of the i-th frame from the second frame is acquired as a still image (step S402), and the still image selection unit 16 selects two frames for the frame pi as still images as described in the second embodiment (step S402). Step S403). Then, the depth image calculation unit 18 calculates the above-described depth image based on the three still images (step S404).

The depth image data generated by the depth image calculation unit 18 is output to the outside and stored in the depth image storage unit 32 (step S40).
5). Subsequently, the depth moving image generation unit 34
Is determined as the last frame of the moving image (step S406).

In step S406, if the frame pi is not the last frame of the moving image, i is incremented by 1 and the processing from step S402 is repeated (step S408). On the other hand, if the frame pi is the last frame of the moving image, the depth moving image generating unit 34 connects the depth images stored in the depth image storage unit 32 to generate a depth moving image (step S407). .

By the way, when the still image acquisition unit 14 selects the frame p1, the still image selection unit 16 fails to select a frame and ends according to the flowchart shown in FIG.
Therefore, the depth image is not calculated for the frame p1, and the depth image corresponding to the frame p1 is not stored in the depth image storage unit 32. Similarly, no corresponding depth image is stored for the frame p2 and the frames p24 to p26.

Therefore, it is assumed that the following exceptional processing is performed in the depth moving image generation unit 34. When the still image selection unit 16 fails to select a frame and does not generate a depth image for a certain frame pj, among the frames for which the depth image was successfully generated, the camera position at the time of shooting the frame pj is selected. The depth image corresponding to the frame shot from the closest position is substituted. As described above, it is possible to obtain a depth moving image generated by connecting the depth images created corresponding to each frame of the moving image stored in the moving image storage unit 12.

As described above, according to the depth image generating apparatus and the depth image generating method according to the third embodiment, in addition to the configuration shown in the second embodiment, the depth generated by the depth image calculating section 18 Since a depth image storage unit 32 for storing images and a depth moving image generation unit 34 for generating a moving image by connecting depth images stored in the depth image storage unit 32 are provided, depth moving image data is obtained. It becomes possible.

In the third embodiment, a certain frame p
For j, when the still image selection unit 16 fails to select a frame and a depth image is not generated, among the frames for which the depth image was successfully generated, the camera position closest to the camera position when the frame pj was photographed. Although the depth image corresponding to the frame shot from the position is substituted, a depth image selected by another method may be substituted. For example, it is also possible to find the depth images for the preceding and succeeding frames in the order from the closest, and substitute the depth image found first.

Further, the depth image generating method according to the first to third embodiments described above uses a hard disk or a CD-ROM.
The program can be provided by being recorded as a computer program on a storage medium such as a ROM. In this case, the computer program is stored in the first to third embodiments.
Is realized.

[0091]

As described above, according to the present invention, according to the present invention, a moving image is stored in the moving image storage means, and the still image obtaining means forms the first two-dimensional still image forming the moving image from the moving image storage means. From the moving image storage unit, and the depth image calculation unit extracts the second two-dimensional still image that is temporally before or after the first two-dimensional still image, for example. And a depth image is generated by associating the distance between the area on the subject and the camera with each pixel of the second two-dimensional still image. Thus, it is possible to obtain the three-dimensional shape information and the depth image of the subject accurately without requesting the photographing conditions such as.

According to the next invention, the still picture selecting means is
Since the second two-dimensional still image is selected in consideration of the parallax for calculating a sufficiently accurate depth image in the depth image calculation means, an accurate depth image can be generated in the depth image calculation means. This has the effect.

According to the next invention, the still picture selecting means is
Since the second two-dimensional still image is reselected based on the position or orientation of the camera obtained by the depth image calculating means for the temporarily selected second two-dimensional still image, the depth image calculating means: It is possible to generate a depth image based on the reselected second two-dimensional still image, and it is possible to improve the accuracy of the finally obtained depth image.

[0094] According to the next invention, the still image selecting means includes:
Since the second two-dimensional still image is reselected based on the position and orientation of the camera obtained by the depth image calculating means for the temporarily selected second two-dimensional still image, the depth image calculating means: It is possible to generate a depth image based on the reselected second two-dimensional still image, and it is possible to improve the accuracy of the finally obtained depth image.

According to the next invention, furthermore, a depth image storage means for storing the depth image generated by the depth image calculation means, and a depth image for generating a moving image by connecting the depth images stored in the depth image storage means. And moving image generating means, so that it is possible to obtain depth moving image data.

According to the next invention, a moving image is stored, a first two-dimensional still image constituting the stored moving image is taken out, and the first two-dimensional still image is temporally A depth image is extracted by extracting a preceding and following second two-dimensional still image from the moving image, and associating each pixel of the first and second two-dimensional still images with the distance between the region on the subject and the camera. Since the image data is generated, the three-dimensional shape information and the depth image of the subject can be accurately obtained without requiring shooting conditions such as a moving speed and a moving form of the camera for the source moving image data. .

According to the next invention, the second two-dimensional still image is selected in consideration of the parallax for calculating a sufficiently accurate depth image.
This produces an effect that an accurate depth image can be generated.

According to the next invention, the second two-dimensional still image is reselected based on the position or orientation of the camera obtained in the depth image process for the temporarily selected second two-dimensional still image. Therefore, it is possible to generate a depth image based on the reselected second two-dimensional still image, and it is possible to improve the accuracy of the finally obtained depth image.

According to the next invention, the second two-dimensional still image is reselected based on the position and orientation of the camera obtained by the depth image process for the temporarily selected second two-dimensional still image. Therefore, it is possible to generate a depth image based on the reselected second two-dimensional still image, and it is possible to improve the accuracy of the finally obtained depth image.

According to the next invention, the depth image generated in the depth image calculation step is stored, and the stored depth images are connected to generate a moving image, so that it is possible to obtain depth moving image data. This has the effect of becoming

According to the next invention, by recording a program for causing a computer to execute the above-described method, the program can be read by a computer, and thereby, the recording medium that can realize the method by a computer Is obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of a depth image generation device according to a first embodiment;

FIG. 2 is an explanatory diagram illustrating a shooting state when a rectangular parallelepiped subject is shot by a video camera in the depth image generation device according to the first embodiment;

FIG. 3 is an explanatory diagram of a shooting state when a rectangular parallelepiped subject is shot by a video camera in the depth image generating apparatus according to the first embodiment when viewed from above;

FIG. 4 is a diagram illustrating an example of a depth image generated by a depth image calculation unit of the depth image generation device according to the first embodiment;

FIG. 5 is a flowchart illustrating a process in a depth image calculation unit of the depth image generation device according to the first exemplary embodiment;

FIG. 6 is an explanatory diagram for explaining extraction of a feature point in the depth image generation device according to the first embodiment;

FIG. 7 is an explanatory diagram for explaining feature point matching and calculation of the position and orientation of the camera in the depth image generation device according to the first embodiment;

FIG. 8 is an explanatory diagram for explaining calculation of a distance from a camera to each feature point in the depth image generation device according to the first embodiment;

FIG. 9 is an explanatory diagram for explaining generation of a depth image in the depth image generation device according to the first embodiment;

FIG. 10 is a block diagram illustrating a schematic configuration of a depth image generation device according to a second embodiment;

FIG. 11 is an explanatory diagram for explaining a shooting state of the video camera 202 when shooting a rectangular parallelepiped subject in the depth image generation device according to the second embodiment.

FIG. 12 is a flowchart illustrating a process of selecting a still image based on a camera position in the depth image generation device according to the second exemplary embodiment;

FIG. 13 is a flowchart illustrating a process of selecting a still image based on a direction of a camera in the depth image generation device according to the second exemplary embodiment.

FIG. 14 is a block diagram illustrating a schematic configuration of a depth image generation device according to a third embodiment;

FIG. 15 is a flowchart illustrating a process of generating a depth moving image in the depth image generation device according to the third exemplary embodiment;

[Explanation of symbols]

10, 20, 30 image generation device, 12 moving image storage unit, 14 still image acquisition unit 16 still image selection unit, 18 depth image calculation unit, 32
Depth image storage unit, 34 depth moving image generation unit, 201
Subject, 202 video camera.

Claims (11)

[Claims] A moving image storage unit that stores a moving image captured by a camera; a still image obtaining unit that extracts a first two-dimensional still image from a moving image stored in the moving image storage unit; A second two-dimensional still image different from the first two-dimensional still image is selected and extracted from the moving image stored in the moving image storage unit based on the two-dimensional still image extracted by the image obtaining unit. A still image selecting unit; and a depth image calculating unit configured to calculate a depth image formed by associating pixels forming a subject with a distance between the subject and a camera based on the first and second two-dimensional still images. A depth image generation device, comprising: 2. The method according to claim 1, wherein the still image selection unit selects a two-dimensional still image having a predetermined parallax from the first two-dimensional still image as the second two-dimensional still image. The depth image generation device according to claim 1. 3. The depth image calculating means calculates a position or an orientation of a camera when each of the first and second two-dimensional still images is photographed, and the still image selecting means calculates the depth image The depth image generation device according to claim 1, wherein the second two-dimensional still image is reselected based on the position or orientation of the camera calculated by the calculation unit. 4. The depth image calculation means calculates a position and an orientation of a camera when each of the first and second two-dimensional still images is photographed, and the still image selection means calculates the depth image The depth image generation device according to claim 1, wherein the second two-dimensional still image is reselected based on the position and orientation of the camera calculated by the calculation unit. 5. A depth moving image generating means for storing a depth image generated by the depth image calculating means, and a depth moving image generating a depth moving image by connecting the depth images stored in the depth image storing means. The depth image generation device according to any one of claims 1 to 4, further comprising: means. 6. A moving image storing step of storing a moving image photographed by a camera, and a first moving image stored in the moving image storing step.
A first two-dimensional still image from the moving image stored in the moving image storage step based on the two-dimensional still image extracted in the still image obtaining step. A still image selecting step of selecting and extracting a second two-dimensional still image that is different from the first and second two-dimensional still images; and, based on the first and second two-dimensional still images, A depth image calculation step of calculating a depth image configured in association with the depth image generation method. 7. The still image selecting step, wherein a two-dimensional still image having a predetermined parallax with the first two-dimensional still image is selected as the second two-dimensional still image. The depth image generation method according to claim 6. 8. The depth image calculating step further calculates a position or an orientation of a camera when each of the first and second two-dimensional still images is photographed, and the still image selecting step includes: The depth image generation method according to claim 6, wherein the second two-dimensional still image is reselected based on the position or orientation of the camera calculated in the image calculation step. 9. The depth image calculation step further calculates a position and an orientation of a camera when each of the first and second two-dimensional still images is photographed, and the still image selection step includes: The method according to claim 6, wherein the second two-dimensional still image is reselected based on the position and orientation of the camera calculated in the image calculation step. 10. A depth moving image generating step of storing a depth image generated by the depth image calculating step, and a depth moving image generating a depth moving image by connecting the depth images stored by the depth image storing step. The method according to any one of claims 6 to 9, further comprising the steps of: 11. A computer-readable recording medium on which a program for causing a computer to execute the method according to claim 6 is recorded.