JP2004229093A - Method, device, and program for generating stereoscopic image, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology capable of easily generating a stereoscopic image of a moving picture. <P>SOLUTION: This stereoscopic image generating method comprises a step for inputting a moving image data in each one frame time, a step for inputting data (hereafter called range image data) for generating a stereoscopic image from the moving image data in an intermittent frame interval, and a step for interpolating and generating the range image data at a frame time when the range image data are not present on the basis of the moving image data and the inputted range image data. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a three-dimensional moving image data generating method, a three-dimensional moving image data generating device, a three-dimensional moving image data generating program, and a recording medium, and is particularly effective when applied to the generation of three-dimensional moving image data to be displayed on a portable device or the like. It is about technology.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, in the field of image communication, attention has been paid to moving image communication using stereoscopic images that can provide a higher sense of realism, and presentations at academic conferences and demonstration productions of companies are also becoming active. Here, it is assumed that the three-dimensional image is a combination of a two-dimensional image in which colors and luminances are recorded, and an image called a distance image or a depth image in which a distance from a measurement point to an imaging target is recorded. Further, the distance image is an image in which the distance to an object to be photographed is recorded in each pixel. In addition, the moving image includes, for example, an image of about 30 frames or more per second.
[0003]
When generating a moving image based on the stereoscopic image, for example, moving image data including a two-dimensional image in which the color and luminance are recorded is input by a TV camera or the like, and the distance image data is input by a distance measuring device or the like.
[0004]
In addition, as a method of generating a moving image using the stereoscopic image, for example, there is a method of generating distance image data using two two-dimensional images having parallax.
[0005]
Further, as a method of generating a stereoscopic image based on a parallax image of a two-dimensional image, a technique is disclosed in which a parallax region is determined from a distance image to determine a parallax region, and a parallax amount is generated using the preceding and following images and motion vectors. (See, for example, Patent Document 1).
[0006]
[Patent Document 1]
JP 2000-253422 A
[0007]
[Problems to be solved by the invention]
However, the conventional technique has a problem that it is difficult to perform moving image communication using the stereoscopic image.
[0008]
In order to perform the moving image communication using the three-dimensional image, for example, distance image data of 30 frames or more per second must be input. However, conventionally, a distance measuring device capable of acquiring distance image data at such a high speed is very expensive. Therefore, it is difficult to use it by incorporating it into a portable device.
[0009]
Further, in the related art, the distance image data is acquired and input every one frame time similarly to the two-dimensional image, so that the power consumption increases. For this reason, there has been a problem that it is difficult to incorporate the battery pack into a portable device used with a rechargeable battery or the like.
[0010]
Further, as a technique that can be incorporated into a portable device, for example, there is a method of inputting distance image data having a low resolution enough to distinguish between near and far and displaying a three-dimensional image in a split type using the input. There is also a method of generating pseudo range image data from the range image having a low resolution and stereoscopically displaying the same. Here, it is assumed that the pseudo distance image data is data generated artificially, not by measuring the distance. That is, the pseudo distance image data does not always match the actual distance, and the degree of approximation depends on the generation algorithm.
[0011]
However, in each of the above methods, the distance image data is acquired every frame time, so that the power consumption is high. Therefore, there is a problem that it is difficult to input a three-dimensional image into a moving image using a portable device.
[0012]
Further, even in the technique described in Patent Document 1, since the distance image data is acquired every frame time, there is a problem that the power consumption is high and it is difficult to use the distance image data incorporated in a portable device.
[0013]
An object of the present invention is to provide a technique capable of easily generating a three-dimensional image of a moving image.
[0014]
Another object of the present invention is to provide a technique capable of reducing the power consumption of a device that generates a stereoscopic image of a moving image.
[0015]
Another object of the present invention is to provide a program and a recording medium capable of facilitating input of a moving image using a three-dimensional image and reducing the amount of power consumption when generating a three-dimensional image.
[0016]
The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.
[0017]
[Means for Solving the Problems]
The outline of the invention disclosed in the present application is as follows.
(1) A step of inputting moving image data for each frame time, and a step of inputting data for generating a stereoscopic image from the moving image data (hereinafter referred to as distance image data) at intermittent frame intervals. And a step of interpolating and generating distance image data at a frame time when the distance image data does not exist based on the moving image data and the input distance image data.
[0018]
(2) In the means of the above (1), the step of generating the distance image data by interpolation includes a step of obtaining a distance boundary area from the input distance image data, and a step of obtaining the same frame as the input distance image data. A step of obtaining a contour region at the same position as the boundary region or a boundary region near the boundary region from moving image data at a time; and corresponding to the contour region from the moving image data at a frame time when the distance image data does not exist. Determining the region to be processed, and the step of generating the distance image data by moving the boundary region and its internal region based on the amount of movement of the outline region and the region corresponding to the outline region. Have.
[0019]
(3) In the means of (1), the step of interpolation generating the distance image data includes the step of selecting moving image data at the same frame time as the input distance image data; From, a step of obtaining a motion vector of an image up to the moving image data at the frame time when the distance image data does not exist, and dividing the input distance image data into a plurality of regions, based on the motion vector Moving each area to generate distance image data.
[0020]
(4) In each of the means (1) to (3), the step of inputting the distance image data includes the step of inputting auxiliary image data capable of generating the distance image data at intermittent frame intervals; Generating distance image data based on the auxiliary image data or moving image data at the same frame time as the auxiliary image data and the auxiliary image data.
[0021]
According to the means (1), even when a means for acquiring the distance image data such as a distance measuring device cannot be operated at a high speed, a moving image of a three-dimensional image can be easily obtained.
[0022]
At this time, in the step of generating the distance image data by interpolation, for example, the processing in each step of the means of (2) or the means of (3) is performed.
[0023]
In the step of inputting the distance image data, the distance image data may be directly input using a distance measuring device or the like, or indirectly from the auxiliary image data as in the means of (4). The generated distance image data may be input.
[0024]
(5) Moving image data input means for inputting moving image data for each frame time, and data for generating a three-dimensional image from the moving image data (hereinafter referred to as distance image data) are transmitted at intermittent frame intervals. Based on the moving image data input from the moving image data inputting means and the distance image data input from the distance image data inputting means, based on the distance image data inputting means input at And a distance image data interpolating unit for interpolating and generating the distance image data.
[0025]
(6) In the means of (5), the distance image data interpolating means includes means for obtaining a distance boundary area from the distance image data input from the distance image data input means, and the same frame as the distance image data. Means for obtaining a contour area in the same position as the boundary area from the moving image data at the time, or a contour area near the boundary area, and corresponding to the contour area from the moving image data at the frame time in which the distance image data does not exist. Means for determining an area to be performed, and means for generating the distance image data by moving the boundary area and an internal area thereof based on the amount of movement of the area corresponding to the outline area and the area corresponding to the outline area. Have.
[0026]
(7) In the means of the above (5), the distance image data interpolating means selects a moving image data at the same frame time as the input distance image data, and the distance image data interpolating means selects the moving image data from the selected moving image data. Means for obtaining a motion vector of an image up to moving image data at a frame time when no distance image data exists, and dividing the input distance image data into a plurality of regions, and dividing each of the regions based on the motion vector. Means for moving to generate distance image data.
[0027]
(8) In the means of (5) to (7), the distance image data input means includes auxiliary image data input means for inputting auxiliary image data capable of generating the distance image data at intermittent frame intervals. Means for generating distance image data based on the auxiliary image data or moving image data at the same frame time as the auxiliary image data and the auxiliary image data.
[0028]
According to the means (5), even when a means for acquiring distance image data such as a distance measuring device cannot be operated at a high speed, a moving image of a three-dimensional image can be easily obtained.
[0029]
Further, even if the means for acquiring the distance image data is a means capable of acquiring the distance image data every frame time, the intermittent operation can reduce the power consumption.
[0030]
Further, the distance image data interpolation means includes, for example, each means of the means (6) or the means (7).
[0031]
The distance image data input means may be a means for directly inputting the distance image data from a distance measuring device or the like, or a means for inputting the auxiliary image data as in the means of (8). Means for generating distance image data from the auxiliary image data may be provided to indirectly input distance image data.
[0032]
As described above, by using the means (5) to (8), it is possible to easily incorporate the stereoscopic image generation device into a portable device.
[0033]
(9) A three-dimensional image generation program for causing a computer to execute each step of the three-dimensional image generation method according to any one of (1) to (4).
[0034]
According to the means (9), a stereoscopic image can be easily generated using a general-purpose device (apparatus) such as a computer.
[0035]
(10) A recording medium on which the stereoscopic image generation program of the means (9) is recorded so as to be readable by a computer.
[0036]
According to the means (10), the recording medium can be provided by copying or through a network. Therefore, any device that can execute the stereoscopic image generation program can generate a stereoscopic image.
[0037]
Hereinafter, the present invention will be described in detail with embodiments (examples) with reference to the drawings.
In all the drawings for describing the embodiments, components having the same function are denoted by the same reference numerals, and the repeated description thereof will be omitted.
[0038]
BEST MODE FOR CARRYING OUT THE INVENTION
(Example 1)
FIG. 1 is a schematic diagram illustrating a schematic configuration of a stereoscopic image generation device according to a first embodiment of the present invention.
[0039]
As shown in FIG. 1, the three-dimensional image generation device according to the first embodiment includes a moving image data input unit 1B for inputting moving image data from a moving image data obtaining unit 1A, and distance image data from a distance image data obtaining unit 2A. Distance image data input means 2B to be input, control means 3 for controlling each data input to the moving image data input means 1B and the distance image data input means 2B, and input from the moving image data input means 1. Moving image data recording means 4 for recording moving image data, distance image data recording means 5 for recording distance image data input from the distance image data input means 2, distance image data interpolation for generating distance image data Means 6.
[0040]
Further, the three-dimensional image generation device is, for example, a device that generates a three-dimensional image including a set of moving image data and distance image data of 30 frames or more per second, and the generated three-dimensional image is, as illustrated in FIG. The three-dimensional moving image can be obtained by the calculation by the calculation means 7 and the display by the display means 8. At this time, the moving image data input means 1B, the distance image data input means 2B, the control means 3, the moving image data recording means 4, the distance image data recording means 5, the distance image data interpolating means 6, the calculation As each means of the means 7, a CPU or a memory of a computer can be used.
[0041]
FIGS. 2 and 3 are schematic diagrams illustrating a stereoscopic image generation method using the stereoscopic image generation device according to the first embodiment, and are diagrams illustrating an overall processing procedure of the stereoscopic image generation method.
[0042]
In order to generate the three-dimensional image using the three-dimensional image generation device of the first embodiment, first, as shown in FIG. 2, moving image data CM (T) and distance image data RM (T) are input, and The moving image data is recorded in the moving image data recording means 4 and the distance image data recording means 5. At this time, the moving image data CM (T) is input every one frame time, but the distance image data RM (T) is intermittent at N frame intervals, for example, every three frames as shown in FIG. At a typical frame interval. At this time, the distance image data RM (T) is, for example, 1-bit resolution distance image data. The black region 11 is a near-view region, for example, a region corresponding to the shooting target 10, and the white region is a distant view. Let it be an area.
[0043]
However, in this state, since the moving image data RM (T + 1) and RM (T + 2) at the frame time when the distance image data RM does not exist, the stereoscopic image is not generated as it is. Therefore, distance image data of a nonexistent frame time is generated by interpolation using the moving image data and the distance image data.
[0044]
In the three-dimensional image generation method according to the first embodiment, as shown in FIGS. 2 and 3, after initialization (step 1401), a boundary area 12 between a near view and a distant view is obtained from the distance image data RM (T). This image is defined as Q (T) (step 1402).
[0045]
Next, from the moving image data CM (T) at the same frame time T as the distance image data RM (T), the contour region 13 in the same region as the boundary region 12 of the image Q (T) or in the vicinity thereof Is obtained, and this image is set to R (T) (step 1403).
[0046]
Next, the same region as the contour region 13 of the image R (T) is obtained from the moving image data CM (T + 1) at the frame time T + 1 where no distance image data exists, and this image is set as R (T + 1) (step). 1404). In order to obtain the image area, for example, an existing pattern matching method (Hideyuki Tamura, "Introduction to Computer Image Processing", Soken Shuppan, pp. 148-153) is used. There are various variations of the pattern matching method, which can be appropriately selected.
[0047]
Next, the distance image data RM (T (T) at the frame time T is set in accordance with the movement amount of the contour region 13 in the image R (T + 1) obtained in the step 1404 with respect to the contour region 13 of the image R (T). The distance image data RM (T + 1) at the frame time T + 1 is generated by moving the foreground area 11 in (1) (step 1405).
[0048]
Thereafter, it is checked whether or not the distance image data RM (T + 2) at the next frame time T + 2 exists (steps 1406, 1407, 1408, 1409). In the first embodiment, since the distance image data RM (T + 2) at the frame time T + 2 does not exist, the distance image data RM (T + 1) is generated in the same procedure as the distance image data RM (T + 1) at the frame time T + 1. At this time, the position of the video area of the moving image data whose frame time is T + 2 may be obtained by, for example, pattern matching directly with the contour area of the moving image data whose frame time is T as shown in FIG. Alternatively, the frame time may be obtained by pattern matching with the contour area of the moving image data at T + 1.
[0049]
By repeating the above procedure, the distance image data at the frame time when the distance image data does not exist can be sequentially generated by interpolation. Therefore, in the three-dimensional image generation device according to the first embodiment, it is not necessary to acquire distance image data every frame time. Therefore, it is possible to use the distance image data acquisition unit 2A that can operate only at intermittent frame intervals. Further, even when the distance image data input means 2A can acquire distance image data for each one frame time, the power consumption can be reduced by operating the distance image data intermittently. In the first embodiment, the distance image data is input every three frame times. However, the present invention is not limited to this. For example, it is possible to input the distance image data every five or ten frame times. By increasing the time interval, power consumption can be significantly reduced.
[0050]
FIGS. 4 to 18 are diagrams illustrating a specific example of the stereoscopic image generation method according to the first embodiment.
[0051]
In the stereoscopic image generation method of the first embodiment, first, as shown in FIG. 4, moving image data CM for each frame and distance image data RM for every N frames are input, and the moving image data And the distance image data recording means 5. At this time, assuming that a frame time at which both the image and the distance image having the low resolution exist is T, RM (T (T), CM (T + 1),..., CM (T + N−1) ) Exists. From these images, moving image stereoscopic image data is generated by generating distance image data of RM (T + 1),..., RM (T + N−1). Here, it is assumed that both the CM and the RM have the image size of X and Y pixels in the vertical and horizontal directions. Further, the data value of the coordinates (x, y) of CM (T) is _{x, y} And the data value of the coordinates (x, y) of RM (T) is expressed as RM (T) _{x, y} Is described. CM (T) _{x, y} Is RGB data, RM (T) _{x, y} Is binary data of 0 or 1. Here, 1 represents a near view, and 0 represents a distant view.
[0052]
When the moving image data CM and the distance image data RM are input, as shown in FIG. 5, a TV camera (moving image data acquisition unit) 1A for moving image shooting and a TV camera ( The control unit 3 controls the distance image data acquisition unit 2B and the light source 15 that emits pulse light such as a flash.
[0053]
At this time, the photographing optical axis of the distance image data acquiring means 2A is aligned with the photographing optical axis of the moving image data acquiring means 1A by the total reflection mirror 16 and the half mirror 17, so that photographing can be performed at the same photographing angle of view and the same photographing viewpoint. Set as follows.
[0054]
When the moving image data CM is input using the moving image data obtaining unit 1A, the control unit 3 previously sets the moving image data CM as shown in FIG. 6A. A control signal is output every frame time, the moving image data is photographed, and input to the moving image data input means 1B. At this time, the frame time interval ΔT is set to, for example, 1/30 seconds.
[0055]
On the other hand, when the distance image data RM is input, as shown in FIG. 6B, the control unit 3 sends the distance image data acquisition unit 2A a time interval N times the preset frame interval. For example, a control signal is output at a time interval three times as long as the frame interval of the moving image data, intermittent distance image data is photographed, and is input to the distance image data input means 2B. At this time, the control unit 3 outputs a control signal to the light source 15 at the same timing to illuminate the imaging target 10.
[0056]
Note that the photographing start times of the moving image data acquiring unit 1A and the distance image data acquiring unit 2A are shifted at least by the exposure time of the moving image data acquiring unit 1A, or to the illumination light emitted from the light source 15. By using an infrared light and using an infrared camera for the distance image data acquisition unit 2A, the illumination light is prevented from affecting the A image capturing of the moving image data acquisition unit 1.
[0057]
Further, by using a high-speed camera, it is also possible to use the moving image data acquiring means 1A and the distance image acquiring means 2A as a single camera. In this case, the total reflection mirror 16 and the half mirror 17 are unnecessary.
[0058]
Further, the distance image data RMP (T) photographed by the distance image data acquisition means 2A is converted into, for example, low-resolution distance image data RM (T) in which only distinction of distance is recorded. Therefore, as shown in FIG. 7, first, distance image data RMP (T) whose shooting time is closest to the moving image data CM (T) shot by the moving image data obtaining means 1A at time T is selected (step 1801). These two pieces of image data can be regarded as image data of the same object to be photographed at the same photographing time with only the presence or absence of illumination different.
[0059]
Next, in the moving image data CM (T) and the distance image data RMP (T), CM (T) at the same coordinates (x, y). _{x, y} / RMP (T) _{x, y} (Step 1802), and if the ratio is equal to or greater than a certain set value k, RM (T) _{x, y} = 1 (step 1803), otherwise RM (T) _{x, y} RM (T) by setting = 0 (step 1804) _{x, y} Generate Thereafter, the coordinates (x, y) are updated (step 1805), and RM (T) _{x, y} Is generated (step 1806), and is set as the distance image data RM (T).
[0060]
When the illuminated captured image is compared with a non-illuminated captured image, the brightness of the distant view image does not change much while the photographic object of the close view is brighter. The low-resolution range image data RM (T) represented by 0 is generated. Note that the distance image data RM (T) having a low resolution may be generated using a difference instead of a ratio.
[0061]
In the first embodiment, the imaging method using the light source 15 such as a flash as the input means of the low-resolution range image data RM (T) is described. However, the low-resolution range image data RM (T) is acquired or input. If possible, the method is not limited to the method shown in the first embodiment, and the method according to the magnitude of the spatial frequency between the image focused on the near view and the image focused on the distant view (or pan-focus image) For example, the distance may be calculated by stereo imaging, or the difference may be simply obtained by taking a stereo image, and a low-resolution range image data RM (T) may be generated using a region having a difference value lower than a certain set value as a distant view region. Is possible.
[0062]
Next, stereoscopic image data of a moving image is generated according to the procedure shown in FIGS.
[0063]
In step 1402 of obtaining the image Q (T) of the boundary region 12 from the distance image data RM (T), first, as shown in FIG. 8, the pixel RM of the coordinates (x, y) of the distance image data RM (T) (T) _{x, y} And the values of adjacent pixels are compared (steps 1402a and 1402b). At this time, the RM (T) _{x, y} As shown in FIG. 9 (a), pixels adjacent to RM (T) _{x-1, y-1} , RM (T) _{x-1, y} , RM (T) _{x-1, y + 1} , RM (T) _{x, y-1} , RM (T) _{x, y + 1} , RM (T) _{x + 1, y-1} , RM (T) _{x + 1, y} , RM (T) _{x + 1, y + 1} And if all have the same data value, Q (T) _{x, y} = 0 (step 1402c), otherwise Q (T) _{x, y} = 1 (step 1402d). Thereafter, the coordinates (x, y) are updated (step 1402e), and all the pixels RM (T) are updated. _{x, y} About Q (T) _{x, y} Is obtained, and an image Q (T) of the boundary area 12 as shown in FIG. 9B is generated.
[0064]
In step 1402a, the pixel RM (T) _{x, y} And RM (T) adjacent in four connections _{x-1, y} , RM (T) _{x, y-1} , RM (T) _{x, y + 1} , RM (T) _{x + 1, y} Q (T) may be generated in comparison with.
[0065]
Next, as shown in FIG. 10, the step 1403 of obtaining the contour region 13 located at the same position or in the vicinity of the boundary region 12 from the moving image data CM (T) first includes the image R ( Block R (T) of T) _{i, j} Q (T) corresponding to _{i, j} Is extracted (step 1403a). At this time, the R (T) _{i, j} Is a block image having X / W pixels in the horizontal direction and Y / W pixels in the vertical direction, as shown in FIG. 11A, and i and j represent block coordinates. At this time, it is assumed that X, Y, and W are set in advance so that X / W and Y / W are integers. Each pixel of R is a binary data storage area Rk _{i, j} And movement vector storage area RV _{i, j} It is assumed to be composed of
[0066]
In addition, the Q (T) _{i, j} Is divided and extracted for each vertical W pixel and horizontal W pixel as shown in FIG. At this time, the Q (T) _{i, j} Is checked (step 1403b), and if all are 0, Rk (T) _{i, j} = 0 (step 1403c), and if a pixel of 1 is included, Rk (T) _{i, j} = 1 (step 1403d). Thereafter, i or j is updated (step 1403e), and all R (T) are updated. _{i, j} Rk (T) for _{i, j} (Step 1403f).
[0067]
Rk (T) _{i, j} Is data indicating only a contour area located at the same position as or near the boundary area near and far from the CM (T). That is, Rk _{i, j} If = 1, the block area having the coordinates ((i + 1) × W−1, (j + 1) × W−1) from the coordinates (i × W, j × W) to the diagonal apex is the same as the boundary area or The contour area is located nearby. Also, Rk _{i, j} If = 0, the block area having the coordinates ((i + 1) × W−1, (j + 1) × W−1) as the diagonal vertices from the coordinates (i × W, j × W) is the same as the boundary area or It is not a contour area in the vicinity. Therefore, Rk (T) _{i, j} Is obtained, an image R (T) of the contour area 13 as shown in FIG. 12 is obtained.
[0068]
In the first embodiment, the block R (T) of the image R (T) of the contour area 13 is used. _{i, j} Is divided into rectangular blocks, but the present invention is not limited to this, and may be divided into various shapes and extracted.
[0069]
Next, in step 1404 for obtaining a contour area corresponding to the contour area 13 obtained in step 1403 from the moving image data CM (T + n) at frame time T + n, first, as shown in FIG. In T), Rk (T) _{i, j} = 1 is extracted (step 1404a).
[0070]
Next, Rk (T) _{i, j} = 1, the coordinates ((i + 1) × W−1, (j + 1) × W−1) from the coordinates (i × W, j × W) of the CM (T) as shown in FIG. Of the reference image B (T) _{i, j} (Steps 1404a and 1404b).
[0071]
Next, as shown in FIG. 14B, the most B (T) in CM (T + n) _{i, j} Is determined, and the upper left coordinate value (i × W + dx, j × W + dy) of the block is determined (step 1404c). At this time, dx and dy are B (T + n) in the image advanced by the number of frames n. _{i, j} Represents the distance moved in the x-direction and the y-direction of the contour area included in. dx and dy are RV (T + n) _{i, j} (Step 1404d).
[0072]
As for blocks having similar image contents, a correlation amount is obtained for each two blocks, and a block having the highest correlation amount in the image is determined as a block having similar image contents. There are various existing expressions for the amount of correlation, from exact expressions used in statistical theory to simple expressions such as the sum of squares of differences for each pixel, and block search methods (for example, KR Rao et al., “Information compression technology for digital broadcasting and the Internet”, Kyoritsu Shuppan, pp. 69-71) also has various existing methods, but the present invention does not limit these to specific formulas or methods. In the present embodiment, an example using the block correlation method has been described. However, the present invention is not limited to the block correlation method as long as the distance of the contour area moved in the x direction and the y direction can be obtained.
[0073]
Next, i or j is updated (step 1404e), and the above processing is performed for all Rk (T). _{i, j} = 1 (step 1404f), and the amount of movement dx, dy is calculated for each block to obtain RV (T + n). _{i, j} To be stored. And further, each B (T) in the image R (T) _{i, j} Is RV (T + n) _{i, j} 15, the image R (T + n) of the contour area 13 corresponding to the moving image data CM (T + n) at the frame time T + n is obtained, as shown in FIG.
[0074]
Next, in step 1405 for generating the distance image data RM (T + n) at the frame time T + n, first, as shown in FIG. 16, the block B (T) is obtained from the distance image data RM (T) at the frame time T. _{i, j} The block corresponding to RV (T + n) _{i, j} Are moved based on dx and dy (step 1405a). At this time, as shown in FIGS. 17A and 17B, the block to be moved is Rk (T) _{i, j} First, only the block of = 1, that is, only the boundary area 12 is moved first.
[0075]
By the way, the boundary area existing in the block before the movement is continuous between the blocks, but may be interrupted between the blocks after the movement depending on the movement direction and the movement amount. Regarding the boundary area that has become discontinuous, the continuity is restored by connecting the broken points at each block boundary with a straight line (step 1405b). In the present embodiment, an example of connection using linear interpolation is shown, but the present invention is not limited to connection using linear interpolation.
[0076]
Next, by replacing the entire inner area surrounded by the boundary area 12 with data of a neighboring area inside the boundary area, the area image data RM (T + n) at the frame time T + n is obtained as shown in FIG. Can be
[0077]
Thereafter, as shown in FIG. 3, by updating the frame time and repeating the processing of the above steps, the distance image data RM (T + n) having a low resolution at the frame time at which the distance image data has not been acquired is obtained. All are generated, and stereoscopic image data of a moving image is generated.
[0078]
FIG. 19 is a schematic diagram for explaining features of a stereoscopic image generated by the stereoscopic image generation method according to the first embodiment.
[0079]
The distance image data generated by the method of the first embodiment is generated by moving the distance image in accordance with the movement in the x and y directions extracted from the moving image data. Therefore, the movement of the imaging target in the z direction cannot be accurately represented. For example, as shown in FIG. 19A, when the photographing target approaches the photographing point, as shown in FIG. 19B, the distance z in the depth (z direction) actually increases as the photographing point approaches. Becomes larger. However, in the distance image generated according to the first embodiment, the depth distance z hardly changes as shown in FIG. Therefore, it is not suitable for generating moving image stereoscopic image data used for industrial measurement. However, when the purpose is human viewing, such as stereoscopic image communication used in the video and telecommunications industries, it is presented in combination with images, and the human visual recognition mechanism makes a reasonable interpretation due to a reasonable interpretation. Nature is not noticeable. In the case of a range image having a low resolution or a pseudo range image, this point does not originally cause a problem.
[0080]
As described above, according to the three-dimensional image generation method using the three-dimensional image generation device of the first embodiment, even if the distance image data acquisition unit 2A that cannot acquire the distance image data for each frame is used, A three-dimensional image of a moving image can be easily generated.
[0081]
In addition, even if the distance image data acquisition unit 2A can acquire the distance image data for each frame, the distance image data acquisition unit 2A can reduce the power consumption by intermittently acquiring the distance image data. it can. Therefore, it is easy to incorporate the stereoscopic image generation device into a portable device.
[0082]
Further, if the stereoscopic image generation method is programmed, it is possible to cause a computer to execute each of the steps. Therefore, a stereoscopic image of a moving image can be easily generated without using a dedicated device. At this time, the program for generating the stereoscopic image can be provided by a recording medium such as a semiconductor memory, a hard disk, or a CD-ROM, or can be provided through a network.
[0083]
In the first embodiment, an imaging method using a flash as a means for acquiring a low-resolution range image is described. However, as long as a low-resolution range image can be acquired, the method is not limited to the method described in the embodiment. Instead, the distance between the image focused on the near view and the image focused on the distant view (or pan-focus image) may be calculated according to the magnitude of the spatial frequency, or the difference may be calculated simply by performing stereo shooting. For example, it is possible to generate a low-resolution range image with a region having a difference value smaller than a certain set value as a distant view region.
[0084]
(Example 2)
20 and 21 are schematic diagrams for explaining the overall processing procedure of the stereoscopic image generation method according to the second embodiment of the present invention.
[0085]
Since the stereoscopic image generation method according to the second embodiment is performed using, for example, the stereoscopic image generation apparatus described in the first embodiment, detailed description of the apparatus will be omitted.
[0086]
In the stereoscopic image generating method according to the second embodiment, first, as shown in FIG. 20, moving image data CM (T) and distance image data RM (T) are input, and the moving image data recording unit 4 and the distance image The data is recorded in the data recording means 5. At this time, the moving image data CM (T) is input every one frame time, but the distance image data RM (T) is intermittent at N frame intervals, for example, as shown in FIG. At a typical frame interval.
[0087]
However, in this state, there is the moving image data CM (T + 1) and CM (T + 2) at the frame time when the distance image data RM does not exist. Therefore, the stereoscopic image is not generated as it is. Therefore, distance image data of a nonexistent frame time is generated by interpolation using the moving image data and the distance image data.
[0088]
In the stereoscopic image generation method according to the second embodiment, first, as shown in FIGS. 20 and 21, distance image data RM at a certain frame time and moving image data at the same frame time are selected (step 1902). Here, a case where the moving image data CM (T) and the distance image data RM (T) at the frame time T are selected as shown in FIG. 20 will be described. In the distance image data RM (T), RM (T + 3) shown in FIG. 20, a black area represents an area 11 (close view area) that is close to the shooting point, and becomes white as the distance increases. .
[0089]
Next, a motion vector of an image between the moving image data CM (T) selected in step 1902 and the moving image data CM (T + 1) at the frame time T + 1 having no distance image data is obtained (step 1903). The obtained motion vector is shown as M (T + 1). The arrow symbol MV is the detected motion vector in the motion vector image M (T + 1). Although the motion vectors are shown by arrows for easy understanding, the actual image data M (T + 1) is an image in which the motion vector information MV is recorded in each pixel, and is not an image drawn by such arrows. .
[0090]
Next, the distance image data RM (T + 1) at the frame time T + 1 is generated by dividing the distance image data RM (T) at the frame time T into a plurality of regions and moving each region separately according to the motion vector MV. (Step 1904). Here, the area of the distance image data RM (T) matches the area of the motion vector MV. For example, if a motion vector MV is given for each pixel, the distance image data RM (T) is moved in pixel units. If the motion vector MV is a motion vector of a certain block area, each area is a block area unit. Although this region depends on the method for obtaining the motion vector, the present invention is not limited to a specific method for generating a motion vector.
[0091]
Next, the frame time is updated (step 1905), and it is determined whether the moving image data CM (T + 2) and the distance image data RM (T + 2) exist (steps 1906 and 1907), and the moving image data CM (T + 2) exists. If there is no distance image data RM (T + 2), the process returns to step 1903 to generate the distance image data RM (T + 2) by interpolation. At this time, for example, as shown in FIG. 20, the distance image data RM (T + 2) is obtained from the moving image data CM (T) at the frame time T and the moving image data CM (T + 2) at the frame time T + 2. Is calculated, and each area of the distance image data RM (T) at the frame time T is moved according to the motion vector to generate the distance image data RM (T + 2) at the frame time T + 2.
[0092]
By repeating the above procedure, the distance image data at the frame time when the distance image data does not exist can be sequentially generated by interpolation. Therefore, even in the stereoscopic image generation method according to the second embodiment, there is no need to input distance image data every frame time. Therefore, it is possible to use the distance image data acquisition unit 1A that can operate only at intermittent frame intervals. In addition, even when the distance image data acquiring unit 1A can acquire the distance image data for each one frame time, the power consumption can be reduced by operating the distance image data intermittently. In the second embodiment, the distance image data is input every three frame times. However, the present invention is not limited to this. For example, the distance image data can be input every five frame times or every ten frame times. By increasing the time interval, power consumption can be significantly reduced.
[0093]
FIG. 22 to FIG. 26 are schematic diagrams for explaining a specific example of the stereoscopic image generation method according to the second embodiment.
[0094]
In the three-dimensional image generation method according to the second embodiment, the moving image data CM for every one frame time and the distance image data RM for every N frame times are stored in the moving image data recording unit 4 and the distance image data recording unit 5 such as semiconductor memories. Is recorded in advance in At this time, assuming that a frame time at which both the moving image data and the distance image data exist is T, the moving image data of CM (T), CM (T + 1),..., CM (T + N−1) There is one distance image data RM (T). From these images, moving image stereoscopic image data is generated by interpolating and generating RM (T + 1),..., RM (T + N−1) distance image data. Here, it is assumed that both the CM and the RM have an image size of X pixels in the horizontal direction and Y pixels in the vertical direction. Further, the data value of the coordinates (x, y) of CM (T) is _{x, y} And the data value of the coordinates (x, y) of RM (T) is expressed as RM (T) _{x, y} Is described. CM (T) _{x, y} Is RGB data, RM (T) _{x, y} Is a scalar value that records the distance.
[0095]
As the moving image data acquiring means 1A for acquiring the moving image data CM (T), the TV camera for photographing moving images described in the first embodiment is used. The distance image data acquiring means 2A for acquiring the distance image data RM (T) includes, for example, a light section method (for example, Toru Yoshizawa, “Three-dimensional optical measurement”, New Technology Communications, pp. 28-37). ), TOF (Optical Time-of-Flight Measurement Method) (for example, Seiji Iguchi, "Latest Trends in 3D Measurement", Measurement and Control, Vol. 34, No. 6, p. 430, Masahiro Kawakita, "Hi-Vision 3D" Camera: Axi-vision Camera ", published by Giken in 2002, Proceedings of the Research and Presentation Conference, p.58-63, various pattern projection methods (for example, Toru Yoshizawa," Three-dimensional optical measurement (second edition) ") , New Technology Communications, pp. 77-99), a distance measuring device based on various stereo methods (eg, the Institute of Image Electronics Engineers of Japan, "3D Image Glossary", New Technology Communications, pp. 51) To use.
[0096]
When the distance image data is generated by interpolation using the moving image data CM (T) and the intermittent distance image data RM (T), first, the distance image data RM (T) at a certain frame time T and the same frame time Is selected (step 1902).
[0097]
Next, step 1903 for obtaining a motion vector of an image between the moving image data CM (T) at the frame time T and the moving image data CM (T + n) at the frame time T + n having no distance image data is performed.
[0098]
In the step 1903, as shown in FIG. 22, the moving image data CM (T) is divided into blocks of vertical W pixels and horizontal W pixels in a grid pattern (step 1903a). At this time, as shown in FIGS. 23A and 23B, each block size is W × W, the number of blocks in the horizontal direction is X / W, and the number of blocks in the vertical direction is Y / W. is there. It is assumed that X and Y are set in advance so that they are divisible by W. At this time, the block at the block coordinates (i, j) is represented by B (T) _{i, j} Is described.
[0099]
Next, a motion vector image M (T + n) is generated. At this time, the motion vector image M (T + n) corresponds to the block B (T) _{i, j} , Ie, block-shaped two-dimensional data MV (T + n) of horizontal X / W and vertical Y / W _{i, j} , And each element stores two-dimensional vector data.
[0100]
At this time, first, B (T) corresponding to each block coordinate (i, j) _{i, j} And B (T) is selected as shown in FIG. _{i, j} Is determined from the CM (T + n) (step 1903b).
[0101]
Next, as shown in FIG. 24B, the upper left coordinate value (i × W + dx, j × W + dy) of this block determined from the CM (T + n) is obtained. At this time, dx and dy are B (T) in the image advanced by n frames. _{i, j} Motion vector MV (T + n) _{i, j} It is. dx and dy are MV (T + n) _{i, j} (Step 1903c).
[0102]
The motion vector MV (T + n) _{i, j} Is called a block matching method, and is a known method used for motion prediction when generating an MPEG2 image, for example. Generally, a correlation amount is obtained for every two blocks, and the block showing the highest correlation amount in the image is determined as a block having a similar image content. There are various existing equations up to a simple equation such as a sum of squares of differences, and various existing methods also exist, such as a block search method and a post-movement inter-block distortion removal method. The present invention does not limit these to any particular formula or approach. In the second embodiment, an example using the block matching method is described. However, the present invention is not limited to the block matching method as long as a motion vector in an image can be obtained. As a result, the generated MV (T + n) _{i, j} Represents the motion in the image when the frame time changes from T to T + n.
[0103]
Thereafter, i or j is updated (step 1903d), and all blocks B (T) of CM (T) are updated. _{i, j} And the same process is repeated (step 1903e).
[0104]
Next, in step 1904 for moving each area of the distance image data RM (T) based on the motion vector image M (T + n), first, as shown in FIG. , The block B (T) _{i, j} Block RM (T) of the same size as _{i, j} (Step 1904a). Thereafter, as shown in FIG. 26A, the block RM (T) _{i, j} Is the motion vector MV (T + n) _{i, j} (Step 1904b). Thereafter, i or j is updated (step 1904c), and all blocks RM (T) are updated. _{i, j} Is moved, distance image data RM (T + n) at frame time T + n is obtained as shown in FIG.
[0105]
After that, the frame time is updated with n = n + 1. If the distance image data RM (T + n) does not exist, the above procedure is repeated to generate the distance image data RM (T + n).
[0106]
At this time, the distance image data RM (T + n) at the frame time T + n is, for example, as shown in FIG. It is generated using T + n moving image data CM (T + n).
[0107]
Hereinafter, by repeating the above-described processing until the frame time of the moving image vector CM in which the distance image data RM exists, the distance image data RM between the frame time T and the frame time T + N can be interpolated.
[0108]
As described above, according to the three-dimensional image generation method of the second embodiment, it is possible to easily generate a three-dimensional image of a moving image even using an acquisition unit that cannot acquire distance image data for each frame. it can.
[0109]
Further, even if the acquisition unit is capable of acquiring the distance image data for each frame, by intermittently acquiring the distance image data, the power consumption of the distance image data acquisition unit 2A can be reduced. Therefore, the device for generating the stereoscopic image can be easily incorporated into a portable device.
[0110]
Further, in the three-dimensional image generation method according to the second embodiment, as described in the above procedure, instead of the distance image data input from the distance measuring device, the distance can be distinguished as described in the first embodiment. It is also possible to use distance image data having a low resolution. In such a case, since only the movement of the contour region needs to be obtained, the amount of calculation is reduced, and the power consumption can be further reduced.
[0111]
FIG. 27 is a schematic diagram for explaining a modification of the second embodiment.
[0112]
In the second embodiment, when interpolation generation of the distance image data at the frame time T + n between the frame time T and the frame time T + N is performed, the moving image data CM (T) at the frame time T is generated as shown in FIG. Although the example in which the distance image data RM (T + n) and the distance image data RM (T + n) are interpolated and generated using the moving image data CM (T + n) at the frame time T + n is shown instead, for example, one frame before The distance image data RM (T + n) may be generated using the moving image data CM (T + n-1), the distance image data RM (T + n-1) generated by interpolation, and the moving image data CM (T + n).
[0113]
The moving image data usually includes a contour portion region always obtained at the first frame time T, as in the second embodiment, since the contour shape of the shooting target gradually changes with the update of the image frame. In the method of performing pattern matching between frames, correct pattern matching cannot be performed in temporally separated frames in which the deformation amount is a certain amount or more. On the other hand, in the case of the method shown in FIG. 27, the motion vector is obtained by performing pattern matching with the contour region obtained in the immediately preceding frame. Irrespective of this, the deformation amount of the contour shape of the photographing target is always within one frame, and the amount is minute. Therefore, in the case of the method shown in FIG. 27, a stereoscopic image of the moving image can be accurately generated even if the number of frames in which no distance image data exists is large.
[0114]
(Example 3)
FIG. 28 is a schematic diagram illustrating a schematic configuration of the stereoscopic image generation device according to the third embodiment of the present invention.
[0115]
As shown in FIG. 28, the three-dimensional image generating apparatus according to the third embodiment includes a moving image data input unit 1B for inputting moving image data from the moving image data obtaining unit 1A, and auxiliary image data from an auxiliary image data obtaining unit 20A. Auxiliary image data input means 20B to be input, control means 3 for controlling each data to be input to the moving image data input means 1B and the auxiliary image data input means 20B, and moving image input from the moving image data input means 1 Moving image data recording means 4 for recording image data; distance image data recording means 5 for recording auxiliary image data and distance image data input from the auxiliary image data input means 20; Image data interpolation generating means 21.
[0116]
The three-dimensional image generation device is a device that generates a three-dimensional image composed of a set of moving image data and distance image data of 30 frames or more per second, for example, and the generated three-dimensional image is, as shown in FIG. The three-dimensional moving image can be obtained by the calculation by the calculation means 7 and the display by the display means 8.
[0117]
FIGS. 29 to 36 are schematic diagrams illustrating a stereoscopic image generation method using the stereoscopic image generation device of the third embodiment.
[0118]
In the three-dimensional image generation device of the third embodiment, unlike the first and second embodiments, as shown in FIG. 29, auxiliary image data capable of generating the distance image data is input, and the auxiliary image data Alternatively, distance image data is generated using the auxiliary image data and the moving image data.
[0119]
Here, the auxiliary image data capable of generating the distance image data means that the data itself does not directly represent the distance information, but the auxiliary image data alone or a combination of the auxiliary image data and the image data is used. The image data is defined as image data capable of generating distance image data, low-resolution distance image data, or pseudo-range image data by calculation.
[0120]
As an example, when the auxiliary image data is a normal image, the auxiliary image data alone does not include the distance information, but the distance image can be generated by using the stereo method by considering it as a stereo image in combination with the other image data. . Therefore, a normal image becomes auxiliary image data.
[0121]
In the third embodiment, only an example in which such a stereo image is used as the auxiliary image data will be described in detail. However, by performing the calculation using the auxiliary image data alone or the combination of the auxiliary image data and the image data, The present invention does not limit the type of auxiliary image data as long as the image data can generate range image data, range image data with low resolution, or pseudo range image data.
[0122]
When a three-dimensional image of a moving image is generated using the three-dimensional image generation apparatus of the third embodiment, an auxiliary device capable of generating moving image data CM for each frame and a distance image for each N frames is stored in a recording unit such as a semiconductor memory. It is assumed that the image data HM has been recorded in advance. Assuming that the frame time at which both the moving image data and the auxiliary image data exist is T, HM (T) is 1 for the moving image of CM (T), CM (T + 1),..., CM (T + N−1). Exists. From these images, moving image stereoscopic image data can be generated by interpolating and generating distance image data of RM (T), RM (T + 1),..., RM (T + N−1).
[0123]
It is assumed that the auxiliary image data HM (T) used in the third embodiment is image data photographed at a position different in the horizontal direction from the photographing position of the moving image data CM (T).
[0124]
When generating the distance image data RM (T) by using the stereoscopic image generation method of the third embodiment, first, as shown in FIG. 30, the auxiliary image data HM at the same frame time as the moving image data CM (T). (T) is selected (step 2201), and the distance image data RM (T) at the frame time T is generated (step 2202).
[0125]
In step 2202 for generating the distance image data RM (T) at the frame time T, as shown in FIG. 31, first, the moving image data CM (T) shown in FIG. The absolute value of the difference between the indicated auxiliary image data HM (T) is taken to generate the image DM (T) (step 2202a). At this time, since the moving image data CM (T) and the auxiliary image data HM (T) are stereo images, the near-view area has a different position in the image between the moving image data CM (T) and the auxiliary image data HM (T). The value will not be 0. In particular, a remarkable difference value occurs in a region where the change in the luminance value is large. On the other hand, in the distant view area, the difference value is very small because the position in the image is hardly changed between the moving image data CM (T) and the auxiliary image data HM (T).
[0126]
Next, each point DM (T) of the difference value image DM (T) _{x, y} Is set to 1 if the difference value is equal to or greater than a predetermined difference value k, and set to 0 if less than k (steps 2202b, 2202c, 2202d). Thereafter, x or y is updated (step 2202e), and all DM (T) are updated. _{x, y} Is binarized (2202f) to generate a binarized image DM (T) as shown in FIG. As a result, the area where the object is not shown (distant view area) is 0. On the other hand, the region where the object is captured (foreground region) has a high probability of being 1, but it may be 0 in a region where the change in luminance is poor, resulting in a noisy image in which 0 and 1 are mixed.
[0127]
Therefore, next, noise is removed from the binarized image DM (T) (step 2202g). The noise removal is performed, for example, as shown in FIGS. 34, 35 (a), and 35 (b) with respect to all the pixels DM (T) of the binarized image DM (T). _{x, y} The value of each of the previously set p pixels × p pixels centered on is checked (steps 2202h and 2202i). If there is a pixel having a value of 1, RM (T) _{x, y} = 1 (step 2202k), otherwise 0 (step 2202j). As a result, noise is removed, and as shown in FIG. 36, a low-resolution range image RM (T) in which the near view is represented by 1 and the distant view is represented by 0 is generated.
[0128]
In addition, as a method of removing noise, there is also a method of using a median filter or applying a smoothing filter and then binarizing again. The present invention does not limit the method of removing noise.
[0129]
Various other images are also conceivable as auxiliary images. For example, there is a method of using an input image with a short depth of focus that is in focus at a short distance as an auxiliary image. In the auxiliary image, the distant view area is photographed out of focus, so that the high-frequency component is extremely small. Therefore, it is possible to calculate the approximate distance at that position from the ratio by obtaining the spatial frequency near the same position between the auxiliary image and the image and obtaining the ratio of the high frequency components. The present invention does not limit what is used for an auxiliary image as long as a distance image can be generated from an auxiliary image or a combination of an auxiliary image and an image.
[0130]
After that, the above procedure is repeated to generate the distance image data RM (T) from the moving image data CM (T) and the auxiliary image data HM (T) intermittently input. Generate a three-dimensional image of a moving image according to the procedure.
[0131]
In the third embodiment, a range image having a low resolution is generated. Therefore, the range image is combined with the generation method described in the first embodiment. However, when the range image generated has a high resolution, the range image is described in the second embodiment. It may be combined with the generated generation method.
[0132]
As described above, according to the three-dimensional image generating method of the third embodiment, the auxiliary image data capable of generating the distance image data is intermittently obtained, and the distance image data is generated from the auxiliary image data. In the same manner as the generation method described in the first embodiment or the second embodiment, a moving image stereoscopic image can be easily generated.
[0133]
Further, by intermittently obtaining the auxiliary image data, the power consumption of the auxiliary image data obtaining unit 20A that obtains the auxiliary image data can be reduced. Therefore, it is easy to incorporate the stereoscopic image generation device into a portable device.
[0134]
As described above, the present invention has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and can be variously modified without departing from the gist of course. is there.
[0135]
【The invention's effect】
The effects obtained by typical aspects of the invention disclosed in the present application will be briefly described as follows.
(1) A three-dimensional image of a moving image can be easily generated.
(2) The power consumption of a device that generates a stereoscopic image of a moving image can be reduced.
[Brief description of the drawings]
FIG. 1 is a schematic diagram illustrating a schematic configuration of a stereoscopic image generation device according to a first embodiment of the present invention.
FIG. 2 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device according to the first embodiment, and is a diagram illustrating an overall processing procedure of the three-dimensional image generation method.
FIG. 3 is a schematic diagram for explaining a stereoscopic image generation method using the stereoscopic image generation device according to the first embodiment, and is a diagram illustrating an overall processing procedure of the stereoscopic image generation method.
FIG. 4 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 5 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 6 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 7 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 8 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 9 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 10 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 11 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 12 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 13 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 14 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 15 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 16 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 17 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 18 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the first embodiment.
FIG. 19 is a schematic diagram for explaining characteristics of a three-dimensional image generated by the three-dimensional image generation method according to the first embodiment.
FIG. 20 is a schematic diagram for explaining the overall processing procedure of the stereoscopic image generation method according to the second embodiment of the present invention.
FIG. 21 is a schematic diagram for explaining the overall processing procedure of the stereoscopic image generation method according to the second embodiment of the present invention.
FIG. 22 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the second embodiment.
FIG. 23 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the second embodiment.
FIG. 24 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the second embodiment.
FIG. 25 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the second embodiment.
FIG. 26 is a schematic diagram for explaining a specific example of the stereoscopic image generation method according to the second embodiment.
FIG. 27 is a schematic diagram for explaining a modification of the second embodiment.
FIG. 28 is a schematic diagram illustrating a schematic configuration of a stereoscopic image generation device according to a third embodiment of the present invention.
FIG. 29 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 30 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 31 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 32 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 33 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 34 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 35 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device of the third embodiment.
FIG. 36 is a schematic diagram for explaining a three-dimensional image generation method using the three-dimensional image generation device according to the third embodiment.
[Explanation of symbols]
1A: Moving image data acquisition means, 1B: Moving image data input means, 2A: Distance image data acquisition means, 2B: Distance image data input means, 3 ... Control means, 4 ... Moving image data recording means, 5: Distance image data Recording means, 6 ... Distance image data interpolation means, 7 ... Calculation means, 8 ... Display means, 10 ... Photographing object, 11 ... Near view area, 12 ... Boundary area, 13 ... Outline area, 15 ... Light source, 16 ... Total reflection Mirror, 17 half mirror, 20A auxiliary image data acquisition means, 20B auxiliary image data input means, 21 distance image data interpolation generation means.

Claims (10)

Inputting moving image data for each frame time;
Inputting data (hereinafter, referred to as distance image data) for generating a stereoscopic image from the moving image data at intermittent frame intervals;
Interpolating and generating distance image data at a frame time when the distance image data does not exist based on the moving image data and the input distance image data. Interpolating and generating the distance image data,
From the input distance image data, obtaining a near and far boundary area,
From the moving image data at the same frame time as the input distance image data, obtaining the same position as the boundary area, or a contour area near the boundary area,
A step of obtaining a region corresponding to the outline portion region from moving image data at a frame time when the distance image data does not exist;
A step of generating the distance image data by moving the boundary region and an inner region thereof based on a movement amount of the outline region and a region corresponding to the outline region. Item 3. The stereoscopic image generation method according to Item 1. Interpolating and generating the distance image data,
Selecting moving image data at the same frame time as the input distance image data,
Obtaining a motion vector of an image from the selected moving image data to moving image data at a frame time when the distance image data does not exist;
Dividing the input distance image data into a plurality of regions and moving each of the regions based on the motion vector to generate distance image data. 3D image generation method. The step of inputting the distance image data,
Inputting auxiliary image data capable of generating the distance image data at intermittent frame intervals,
4. The method according to claim 1, further comprising: generating distance image data based on the auxiliary image data or moving image data at the same frame time as the auxiliary image data and the auxiliary image data. 3. The method for generating a three-dimensional image according to claim 1. Moving image data input means for inputting moving image data for each frame time;
Distance image data input means for inputting data for generating a stereoscopic image from the moving image data (hereinafter referred to as distance image data) at intermittent frame intervals;
A distance image that interpolates and generates distance image data at a frame time when the distance image data does not exist based on the moving image data input from the moving image data input means and the distance image data input from the distance image data input means. A three-dimensional image generation device comprising: a data interpolation unit. The distance image data interpolation means,
Means for obtaining a far and near boundary area from the distance image data input from the distance image data input means,
From the moving image data at the same frame time as the distance image data, means for determining the same position as the boundary region, or a contour region near the boundary region
Means for obtaining an area corresponding to the outline area from moving image data at a frame time when the distance image data does not exist;
The apparatus according to claim 1, further comprising: a unit configured to generate the distance image data by moving the boundary region and an inner region based on a movement amount of the outline region and a region corresponding to the outline region. Item 3. The stereoscopic image generation device according to Item 5. The distance image data interpolation means,
Means for selecting moving image data at the same frame time as the input distance image data,
Means for obtaining a motion vector of an image from the selected moving image data to moving image data at a frame time when the distance image data does not exist;
The apparatus according to claim 5, further comprising: a unit configured to divide the input distance image data into a plurality of regions, move each of the regions based on the motion vector, and generate distance image data. Stereoscopic image generation device. The distance image data input means,
Auxiliary image data input means for inputting auxiliary image data capable of generating the distance image data at intermittent frame intervals,
8. A device according to claim 5, further comprising: means for generating distance image data based on the auxiliary image data or the moving image data at the same frame time as the auxiliary image data and the auxiliary image data. The stereoscopic image generation device according to claim 1. A stereoscopic image generation program for causing a computer to execute each step of the stereoscopic image generation method according to any one of claims 1 to 4. A recording medium in which the stereoscopic image generation program according to claim 9 is recorded so as to be readable by a computer.

Images