JP2001256482A - Device and method for generating parallax image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a parallax image generating device by which an image from a plurality of viewpoints is easily constituted based on a luminance image from one viewpoint and one distance image. SOLUTION: The luminance image from one viewpoint and one distance image are photographed and a shift amount for every pixel is decided based on the focus distance f of a camera, the pixel pitch p of CCD, the parameter of the movement distance H of each viewpoint and distance data for every pixel, which is obtained from the distance image. The images from the various viewpoints are obtained by changing the movement distance H of each viewpoint and calculating the shift amount. A distance to a subject is made into a segment at every prescribed distance, a shift amount corresponding to each distance is previously made into a table and, then, a pixel shift processing is performed at high speed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an apparatus and a method for generating a stereoscopic display image. In particular, the present invention relates to a parallax image generating apparatus and a parallax image generating method capable of generating a parallax image viewed from an arbitrary viewpoint using one distance image and one or more plane images (luminance images) acquired from one viewpoint direction. Things.

[0002]

2. Description of the Related Art In recent years, with the spread of virtual reality / mix reality / electronic commerce and the like, a demand for stereoscopic display of a real image has rapidly increased. Currently used stereoscopic display devices include a glasses system, a lenticular system, a parallax barrier system, and the like. Both are stereo systems using binocular parallax, and only two types of images are displayed, a parallax image for the right eye and a parallax image for the left eye. Stereo method using binocular parallax,
In this method, left and right parallax images are captured by photographing a target object with two cameras installed at a certain interval, and the two parallax images are displayed on a stereoscopic display device.

However, recently, there is a strong demand for a stereoscopic display device having a more stereoscopic effect, and the viewpoint is not a stereoscopic system based on binocular parallax in which only the same stereoscopic image can be displayed regardless of where the eye is placed. A multi-view stereoscopic display device in which an image changes has been proposed. For example, "3D Television without 8-Eye Glasses" Journal of the Institute of Television Engineers of Japan: Vol. 48, No. 10, pp. 1267 (1994) proposes an eight-lens lenticular rear projection type stereoscopic display. The example shown in the magazine is a display example of an animation produced by computer graphics (CG). FIG. 1 shows a configuration example of a photographing system for displaying a video of a real photograph on a display device.

As shown in FIG. 1A, eight cameras 101 to 108 are installed at a predetermined interval toward a subject 100. That is, the subject 10 is viewed from eight different viewpoints.
Capture the 0 image. The images captured by the cameras 101 to 108 in FIG. 1 are real parallax images 121 to 128 as shown in FIG. Real shot parallax images 121 to 128
As can be understood from FIG. 5, in the captured image, the relative positions of the front sphere and the rear cone forming the subject 100 differ depending on each camera position. This is subject 1
This is because the viewpoint at which the camera is installed differs from 00.

[0005] Eight cameras 101 to 1 as shown in FIG.
08, a photographed parallax image 12 from eight viewpoints taken by
The subject 100 is stereoscopically displayed by inputting 1 to 128 into the stereoscopic display means 180. In this case, since the image data from eight viewpoints is retained, the conventional 2
It is possible to display various images from more viewpoints compared to an image of only one viewpoint. However,
In order to acquire these many viewpoint images, the number of cameras required is equal to the number of viewpoints.

In addition, a super multi-view stereoscopic display system, which is an extension of the multi-view system, has been recently proposed. For example,
`` Hologram-Like Video Images by 45-View Stereoscop
ic Display ”SPIE Proc. Vol. # 3012 "Stereoscopic Dis
play and Applications VIII "pp. 154-166 (1997) describes a stereoscopic display device using 45 parallax images.
ch of 3D Display Using the Anamorphic Optics '' SPIE
Proc. Vol. # 3012 "Stereoscopic Display and Applic
ations VIII "pp. 199-207 (1997) prototyped a stereoscopic display device using 72 parallax images. To display live-action video on these devices, 45 or 72 cameras were used. It is necessary to prepare parallax images taken from 45 or 72 viewpoints.

Since this photographing method lacks realism, for example, “Interpolation and compression of a multi-view three-dimensional image by self-similarity modeling”, Journal of the Institute of Television Engineers of Japan: Vol. 48, No. 10, pp1215-
A parallax image generation device as shown in 1221 (1994) has been proposed. A plurality of parallax images are captured by a camera from a point smaller than a required viewpoint. The parallax image viewed from the viewpoint between the photographing points is to be synthesized by image processing (interpolation processing) from a real parallax image. For example, three cameras A, B, and C for capturing images from different viewpoints are installed at predetermined intervals, and based on three real parallax images captured from the three cameras A, B, and C, This is to combine images that would be taken between cameras by image processing (interpolation processing).

FIG. 2 shows three cameras 201, 202 and 20.
3 shows an image capturing apparatus that captures images of the subject 200 from different viewpoints using 3 and generates a total of nine parallax images based on these three images.

In FIG. 2A, three cameras 20
1, 202 and 203 capture images of the subject 200 from different viewpoints. These photographed images are, as shown in FIG. 2B, a real parallax image 221 of the camera 201 and a camera 20.
2 is obtained as a real parallax image of the camera 203. These three real image parallax images are input to the parallax image interpolation / synthesis device 270, and an image that will be captured between the images is generated by interpolation processing.
Interpolated combined parallax images 231 to 233 and interpolated combined parallax images 234 to 236 shown in FIG. As a result, in addition to the actual parallax images 221, 222, and 223 actually captured by the cameras 201, 2020, and 203, six image-interpolated composite parallax images are generated, and the stereoscopic display unit 280 is generated based on a total of nine parallax images. Can display images from various angles.

[0010] Also, a published patent publication: Japanese Unexamined Patent Publication No. 6-26682.
7 describes another example of generating a parallax image. According to the configuration described in this publication, a continuous stereo image (EPI: Epipolar Plane Image) is produced from a plurality of parallax images, and the depth of the object is obtained based on this. Three-dimensional voxel data is generated from this depth information, and a new parallax image is generated by projecting the voxel data onto a two-dimensional plane in an arbitrary direction.

[0011]

However, in the above-mentioned various configurations, a multi-view stereoscopic display device or a super-multi-view stereoscopic display device requires a camera having the same number of viewpoints, which increases the shooting system. There was a disadvantage.

Further, in a method of capturing a plurality of parallax images and obtaining a parallax image viewed from a new viewpoint from the images by interpolation processing, and a method of generating a new parallax image by projecting voxel data onto a two-dimensional plane, There is a disadvantage that it is difficult to process video images in real time because data processing takes a long time. There is also a disadvantage that a high-performance and large-sized data processing device is required.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned various problems in the prior art, and is intended to realize a parallax image viewed from an arbitrary viewpoint in a short time and at a low cost. It is an object of the present invention to provide a parallax image generation device and a parallax image generation method that can be performed.

[0014]

According to a first aspect of the present invention, there is provided:
In a parallax image generation device that generates a parallax image, a luminance image imaging unit that captures a luminance image of a display object, a depth measurement unit that measures depth distance data of the display object, and an imaging parameter of the luminance image imaging unit , A viewpoint moving distance register for storing distance data between the viewpoint position and the observation viewpoint position of the luminance image imaging unit, and a pixel-by-pixel value acquired from the distance image captured by the depth measurement unit. Based on the distance data, the imaging parameters stored in the imaging parameter register, and the viewpoint movement distance data stored in the viewpoint movement distance register, each pixel constituting the luminance image acquired by the luminance image imaging means, Parallax image generation means for determining a shift amount of the image and generating a parallax image from an arbitrary viewpoint In parallax image generating apparatus according to claim.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means includes: distance data D for each pixel obtained from a distance image picked up by the depth measuring means; As the imaging parameters stored in the parameter register, the focal length: f and the pixel pitch: p of the luminance image imaging means, and the viewpoint moving distance data: H stored in the viewpoint moving distance register are input values. The amount of shift of each pixel constituting the luminance image acquired by the luminance image imaging means is determined on the basis of the above, and a parallax image from an arbitrary viewpoint is generated.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means segments distance data of each pixel measured by the depth measuring means at a fixed distance data interval, and And a shift amount determination table in which a pixel shift amount corresponding to each segment number is set, and the distance data for each pixel obtained from the distance image captured by the depth measurement unit. The segment number of the shift amount determination table is associated with each pixel based on the shift amount, and the shift amount set for the associated segment number is selected based on the shift amount determination table, and the shift amount for each pixel is determined. It is characterized by having a configuration for determining.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means includes a focal length: f, a pixel pitch: p, and the viewpoint moving distance data: H of the luminance image capturing means. A plurality of shift amount determination tables corresponding to the focal length: f, the pixel pitch: p, and the viewpoint moving distance data: H input to the parallax image generating means.
The table selection is executed based on the table and the shift amount for each pixel is determined based on the selected table.

Further, in one embodiment of the parallax image generating device of the present invention, the viewpoint moving distance data stored in the viewpoint moving distance register includes moving direction data indicating a moving direction, and the parallax image generating means includes: The shift direction of each pixel constituting the luminance image acquired by the luminance image imaging means is determined based on the moving direction data in the viewpoint moving distance data stored in the viewpoint moving distance register. .

Further, in one embodiment of the parallax image generating apparatus according to the present invention, the parallax image generating means includes, for a blank pixel generated by the shift processing of the shift amount determined for each pixel, among blank pixels adjacent to the blank pixel. A configuration that generates a corrected parallax image by selecting a pixel having the farthest distance data based on the measurement value of the depth measurement unit and embedding the luminance data of the selected adjacent pixel as the luminance value of the blank pixel. It is characterized by having.

According to a second aspect of the present invention, in a parallax image generating method for generating a parallax image, a luminance image capturing step of capturing a luminance image of a display target, and measuring depth distance data of the display target. The depth measurement step, the distance data for each pixel obtained from the distance image captured in the depth measurement step, the imaging parameter of the luminance image, and the viewpoint moving distance data, based on the luminance image imaging means, A parallax image generating method, comprising: a parallax image generating step of determining a shift amount for determining a shift amount of each pixel forming an acquired luminance image and performing a shift of the determined pixel.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes: distance data: D for each pixel obtained from the distance image picked up in the depth measuring step; As the imaging parameters stored in the parameter register, the focal length: f and the pixel pitch: p of the luminance image imaging means, and the viewpoint moving distance data: H stored in the viewpoint moving distance register are input values. The amount of shift of each pixel constituting the luminance image acquired by the luminance image imaging means is determined based on the above, and a parallax image from an arbitrary viewpoint is generated.

Further, in one embodiment of the parallax image generating method according to the present invention, the parallax image generating step includes segmenting distance data of each pixel measured in the depth measuring step at a fixed distance data interval. , And a step of determining a shift amount for each pixel based on a shift amount determination table in which a shift amount is set in accordance with the segment number.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step is performed according to various values of a focal length: f and a pixel pitch: p, and the viewpoint moving distance data: H. From the plurality of generated shift amount determination tables, one table is selected based on the input focal length: f, pixel pitch: p, and viewpoint movement distance data: H, and the selected table is selected. The shift amount for each pixel is determined based on

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes a step of obtaining the luminance acquired in the luminance image capturing step based on viewpoint moving direction data included in the viewpoint moving distance data. A shift direction of each pixel constituting an image is determined.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step further includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a pixel adjacent to the blank pixel. Selecting the pixel having the farthest distance data based on the measurement value obtained in the depth measurement step, and embedding the luminance data of the selected adjacent pixel as the luminance value of the blank pixel. And

[0026]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of a parallax image generating apparatus and a parallax image generating method according to the present invention.

[0027]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of a parallax image generating apparatus according to the present invention will be described with reference to FIG. FIG. 3 is a block diagram illustrating a configuration of the parallax image generation device according to the present embodiment. FIG. 4 shows a slit image, a distance image, and a luminance image captured using the parallax image generation device shown in FIG.

The configuration shown in FIG. 3 will be described. The depth measurement and luminance image imaging unit 500 is a unit that acquires depth information of the subject 300 using a known light sectioning method. The slit light projector 520 emits the slit light 3 toward the subject 300.
60 are emitted intermittently in order from the left. C installed on an optical axis different from the light projecting axis of the slit light projector 520
The CD camera 530 captures an image of the subject 300 in synchronization with the intermittent emission of the slit light 360 by the slit light projector 520, and calculates the distance at each pixel based on the slit image 410 (see FIG. 4). A distance image 420 (see FIG. 4) is generated.

The CCD camera 530 captures a luminance image 430 (see FIG. 4) of the subject 300 without projecting the slit light 360 at the end of the distance image generating operation.

The parallax image generating means 600 generates the distance image 42
A parallax image viewed from each viewpoint is generated based on 0, the luminance image 430, the display parameters of the stereoscopic display unit, and the viewpoint coordinates, and the stereoscopic display unit 390 displays a stereoscopic image of the subject 300.

Next, the operation of generating a luminance image and a distance image will be described in detail with reference to FIG. FIG. 5 is a block diagram showing a detailed configuration of the depth measuring and luminance image capturing means 500 shown in FIG. Control unit 516 and synchronization control unit 515
, The mirror driving unit 513 controls the rotation of the rotating mirror 511.
With the angle fixed at a certain angle, the slit driving laser 512 emits the slit light laser 512, and the slit image 410 (see FIG. 4) of the subject at that time is read by the CCD camera 53.
Image is taken at 0.

The slit image 410 picked up by the CCD camera 530 is input to the slit image memory 532, and the distance calculation processing means 533 calculates the distance between the pixels exposed to the slit by the trigonometric method.

When the distance data based on one slit image is calculated, the slit light laser 512 is emitted while the rotating mirror 511 is rotated and fixed by a small angle by the mirror driving unit 513, and the slit light at that time is emitted. The image 410 is captured by the CCD camera 530. As in the previous case, the distance of the pixel whose slit has been exposed is measured by the distance calculation processing means 533 by trigonometry. By repeating this, the distance of all the pixels imaged by the CCD camera 530 is calculated. Further, a distance image 420 (see FIG. 4) in which a gray value is set based on the distance data for each pixel is generated and stored in the distance image memory 534. The distance image stored in the distance image memory 534 is configured as data having the distance from the viewpoint of the CCD camera to the subject for each pixel. Finally, the luminance image 430 (see FIG. 4) of the subject is imaged without emitting the slit light laser 512, and stored in the luminance image memory 535.

The switches 531a and 531b are
It opens and closes according to image data captured by the CCD camera 530, and the switch 531b closes and the switch 531a opens only when the captured image is a luminance image. If the captured image is a slit image, the state is controlled to the opposite state.

The luminance image and the distance image thus generated are output to the parallax image generation means 600 via the luminance image transmission cable 340 and the distance image transmission cable 350, respectively.

The parallax image generating means 600 stores the luminance image stored in the luminance image memory 535 and the distance image memory 53
Then, a parallax image of the subject 300 is generated based on the distance image stored in No.4. Distance image 420 and luminance image 43
The process of generating a parallax image using 0 will be described in detail with reference to FIG. FIG. 6 is a block diagram illustrating a configuration of the parallax image generation unit 600.

The camera focal length register 602 of the parallax image generating means 600 stores the focal length f of the camera used for photographing the subject, for example, the CCD camera 530 in FIG. The camera CCD pixel pitch register 603 stores the pixel pitch p of the camera used. The viewpoint moving distance register 604 stores a moving distance H between a camera photographing viewpoint and a parallax point (observation viewpoint) for generating a parallax image.

The camera focal length f stored in the camera focal length register 602, the camera pixel pitch p stored in the camera CCD pixel pitch register 603, and the camera photographing viewpoint and parallax point stored in the viewpoint moving distance register 604. The moving distance H between (observation viewpoints) is input to the pixel shift processing means 601, and the pixel shift amount calculating means 605 in the pixel shift processing means 605 converts the input data and the distance data of each pixel obtained from the distance image into Based on the calculated shift amount for each pixel, the pixel shift processing unit 601 executes the shift based on the calculated shift amount for each pixel.

FIG. 7 is a diagram for explaining the relationship between the distance D between the object 700 and the camera shooting viewpoint, the viewpoint moving distance H, and the pixel shift amount. The distance from the viewpoint to the subject is D,
F is the focal length of the camera, a is the distance between the optical point and the imaging point of the subject 700 on the CCD imaging surface photographed from the viewpoint A (camera viewpoint), and the new viewpoint when the viewpoint is moved by H Assuming that the distance between the image forming point of the subject 700 on the CCD imaging surface assumed to be imaged from A ′ (observation viewpoint) and the optical axis is x, the CCD imaging surface when the viewpoint is moved by H The moving distance of the imaging point of the subject 700: (x−a) is represented by the following equation.

[0040]

(Equation 1)

The pixel pitch on the CCD is p, and the subject 70 when the subject 700 is photographed from the viewpoint A '(observation viewpoint).
Object 7 on the CCD imaging surface between the imaging point 0 and the optical axis
The number M of moving pixels of the imaging point of 00 is obtained by the following equation using the number g of CCD pixels between the imaging point of the subject 700 and the optical axis.

[0042]

(Equation 2)

As shown in the above equation, the pixel shift amount M of the subject image on the CCD imaging surface when the viewpoint is moved is determined by using the focal length f of the camera, the pixel pitch p of the CCD, and the moving distance H of the viewpoint as parameters. It can be expressed as a function of the distance D from the viewpoint to the object and the number g of CCD pixels between the image point of the object 700 and the optical axis.

The pixel shift processing means 601 determines the shift value of each pixel of the luminance image input via the luminance image transmission cable 340 based on the pixel shift amount calculated by the above equation, and determines the shift value. The shift processing of the constituent pixels of the luminance image is executed according to the shift value. Images from various viewpoints based on various values of the moving distance H obtained by the shift processing are stored in the parallax image memories 1 to n and 607 a to 60.
7n and output to the stereoscopic display means 390 via the stereoscopic display means signal cable 380 (see FIG. 3).

The above example is an example in which the shift amount of each pixel is obtained by arithmetic processing. The distance D to the subject 700 is classified for each predetermined distance, and is segmented into, for example, 16 steps.
By storing the shift amount determination table in which the shift amount according to each distance is tabulated in the pixel shift processing unit 601, the arithmetic processing can be simplified and the speed of the shift processing can be increased. That is, a plurality of shift amount determination tables corresponding to the parameters of the focal length f of the plurality of cameras, the pixel pitch p of the CCD, and the moving distance H of the viewpoint are generated in advance and stored in the pixel shift processing means 601 to perform pixel shift. A table corresponding to the focal length f of the camera, the pixel pitch p of the CCD, and the moving distance H of the viewpoint input to the processing means 601 is selected, and the segment number corresponding to the distance data for each pixel obtained from the distance image is obtained. By performing a process of calculating the shift amount for the obtained segment number from the table, the calculation process can be simplified.

In a configuration having such a shift amount determination table, the pixel shift processing means 601 receives the distance image 420 (FIG. 5) stored in the distance image memory 534 shown in FIG. 4), the shift amount of each pixel can be determined with reference to a table selected based on the segment number determined from the distance data. FIG. 8 shows an example of this shift amount determination table.

In the example of FIG. 8, the segment number is
This is a value when the distance D between the subject 700 and the camera shooting viewpoint is segmented into 16 steps, and the segment number is set to be larger as the distance D is smaller. That is, in the example of FIG. 8, when the subject distance is a subject at the maximum distance such as infinity, the segment number 0 is assigned, and when the subject distance photographed at the pixel is the minimum, that is, when the subject is a close subject. Is assigned a segment number 15.

An example of shift processing of a captured image using the pixel shift amount determination table shown in FIG. 8 will be described with reference to FIG. The luminance image 900 shown in FIG. 9 is a small image of the luminance image transferred from the luminance image memory 535 of the depth measurement and luminance image imaging unit 500 shown in FIG. 5 to the pixel shift processing unit 601 of the parallax image generation unit 600 shown in FIG. FIG. 9 illustrates a region image, and similarly, a distance image 920 illustrated in FIG.
The distance data of the small area portion corresponding to the luminance image 900 transferred from the distance image memory 534 of the depth measurement and luminance image imaging means 500 is indicated by a segment number.

The pixel shifting method will be described in detail. FIG.
FIG. 9 is an explanatory diagram showing a method of shifting pixels of a luminance image based on distance segment numbers (0 to 15) of a distance image. Each pixel of the luminance image 900 shown in FIG. 9 is set with the luminance value of each pixel when the luminance image is captured. Each rectangular area A11 to A55 in the figure corresponds to each pixel.

The distance image of the pixels corresponding to A11 to A55 is the distance image 920 shown in the upper part of FIG. 9, and distance segment numbers 0 to 15 are assigned as distance data of each pixel. As described above, the distance segment number is obtained by assigning a set value in 16 steps according to the distance from the subject to the viewpoint. In the example shown in FIG. 9, three types of segment numbers 2, 4, and 8 are assigned to the pixels included in the distance image 920, and the center pixel (segment number = 8) is a subject with a small distance D, that is, It can be seen that a subject having a large distance D, that is, a distant subject is captured in a near subject and surrounding pixels (segment number = 2). The shift amount of each pixel is obtained based on the segment number of each pixel according to the shift amount determination table.

In the shift amount determination table of FIG. 8 described above, the pixel shift amount is “0” when the segment number is “2”, the pixel shift amount is “1” when the segment number is “4”, When the segment number is “8”, the pixel shift amount is set to “2”.

First, a process for generating a parallax image 940 when the viewpoint is moved leftward from the photographing viewpoint will be described. The pixels of the luminance image 900 corresponding to the segment number = 2 in the distance image 920 (the outer 16 pixels) are as follows:
Since the pixel shift amount set to the segment number = 2 in the table shown in FIG. 8 is 0, it does not move. Regarding the pixels of the luminance image 900 corresponding to the segment number 4 in the distance image 920 (the inner eight pixels), the pixel shift amount set to the segment number = 4 in the table shown in FIG. For the pixel of the luminance image shifted to the right and corresponding to the segment number 8 of the distance image (one pixel at the center), the pixel shift amount set to the segment number = 8 in the pixel shift amount determination table shown in FIG. Therefore, it is shifted right by two pixels. That is, the parallax image 930 is obtained by shifting each pixel according to the table based on the luminance image 900.
Is obtained.

In this shift processing, for example, A25
Pixel is overwritten by the luminance value of the pixel A24,
For the pixel 35, the luminance value of the pixel A33 is set by the shift processing.

The parallax image obtained when the viewpoint is moved from the photographing viewpoint to the right is a process of shifting pixels to the left by the same operation. By this processing, an image of the parallax image 950 is obtained. The viewpoint movement distance data stored in the viewpoint movement distance register 604 illustrated in FIG. 6 includes movement direction data indicating the viewpoint movement direction, and the pixel shift processing unit 601 in the parallax image generation unit 600 is stored in the viewpoint movement distance register. The shift direction of each pixel constituting the luminance image is determined based on the moving direction data in the viewpoint moving distance data.

The parallax image 930 and the parallax image 95
0 indicates a loss due to the moved pixel (parallax images 930 and 95
Therefore, the pixel shift processing unit 601 corrects the parallax images 940 and 960 corrected by copying a pixel that is farther from the luminance pixels adjacent to the defective pixel and filling the defective pixel. Generate That is, for example, in the parallax image of FIG. 930, three blank pixels (old A2
2, A23, A24). Of the pixels adjacent to these blank pixels, pixels at a long distance are A21, A
31 and A41, a process of copying these pixels to respective blank pixels is performed. Thus, the corrected shifted parallax images 940 and 960 are generated.

As described above, the parallax image generating apparatus of the present invention can generate and display parallax images from various arbitrary viewpoints based on the distance image and the luminance image from one viewpoint. Therefore, imaging processing by a plurality of cameras is not required, and complicated processing such as generation of an image between cameras by interpolation processing based on a real shot image is not required. Further, by performing the shift processing using the shift amount determination table, the shift amount for each pixel can be immediately obtained by the distance segment number obtained based on the distance image.

FIG. 10 shows a second example of the parallax image generating apparatus according to the present invention.
1 shows the configuration of the embodiment. The difference from the configuration shown in FIG.
This is because the depth measuring unit 1001 and the luminance image capturing unit 1002 are separated, and the operation is the same as the configuration in FIG.

In the above-described embodiment, the distance is measured by using the light section method. However, the present invention is not limited to this, and the distance is measured by other methods such as a spatial coding method, a phase difference detection method, and a stereo method. It is needless to say that the distance value may be determined. Also, in the above description, an example was described in which a single image was used as the luminance image. However, a plurality of luminance images having different viewpoints were captured, the luminance image closest to the observation viewpoint was selected, and images at various viewpoints were selected. It is good also as composition which generates.

Further, in the above-described embodiment, the movement of the viewpoint is limited to the horizontal direction. However, it is also possible to generate the vertical parallax image by moving the viewpoint in the vertical direction. For the vertical movement, the same calculation is performed by setting the vertical movement distance V instead of the horizontal movement distance H in each of the above equations 1 and 2. In the shift processing using the table, the shift direction may be set to the vertical direction. In the case where the viewpoint is moved in the horizontal / vertical directions, the horizontal movement distance H and the vertical movement distance V are set in each of the equations (1) and (2), and the respective shift amounts are obtained. The result is set as the actual shift amount in the horizontal and vertical directions. Further, when performing horizontal / vertical shift processing using a table, a configuration in which both horizontal and vertical shift amounts are set, for example, shift amount = (X,
A table having a configuration in which the shift amounts in both the X-axis and the Y-axis directions are set as in Y) may be provided. With such a configuration, it is possible to generate a full parallax multi-viewpoint parallax image. As described above, according to the parallax image generation device of the present invention, the shift amount of each pixel can be immediately determined based on the table and the distance image.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0061]

As described above, according to the parallax image generating apparatus of the present invention, a multi-viewpoint parallax image can be generated only by data from the distance measuring means and one luminance image capturing means. Since there is no need to use a large number of cameras as in the above, the configuration of the input system is simplified. In addition, since it is not necessary to convert the subject into a 3D model or convert it into three-dimensional voxel data, a parallax image can be generated simply by shifting the pixels of the luminance image according to a certain rule. It is possible to generate images from various viewpoints.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example (part 1) of a conventional parallax image generation device.

FIG. 2 is a diagram illustrating a configuration example (part 2) of a conventional parallax image generation device.

FIG. 3 is a diagram illustrating a configuration of a parallax image generation device of the present invention.

FIG. 4 is a diagram illustrating an image acquired by the parallax image generation device of the present invention.

FIG. 5 is a block diagram illustrating a configuration of a depth measurement and luminance image capturing unit of the parallax image generation device of the present invention.

FIG. 6 is a block diagram illustrating a configuration of a parallax image generation unit of the parallax image generation device of the present invention.

FIG. 7 is a diagram illustrating a processing method of a parallax image generation unit in the parallax image generation device of the present invention.

FIG. 8 is a diagram illustrating an example of a shift amount determination table of a parallax image generation unit in the parallax image generation device of the present invention.

FIG. 9 is a diagram illustrating a shift processing example of a parallax image generation unit in the parallax image generation device of the present invention.

FIG. 10 is a diagram showing the configuration of a second embodiment of the parallax image generation device of the present invention.

[Explanation of symbols]

100, 200 subjects 101 to 108 cameras 121 to 128 real parallax images 180, 280 stereoscopic display means 221 to 223 real parallax images 270 parallax image interpolation / synthesis device 300 subjects 340 luminance image transmission cable 350 distance image transmission cable 360 slit light 380 stereoscopic display Means signal cable 390 Stereoscopic display means 410 Slit image 420 Distance image 430 Luminance image 500 Depth measurement and luminance image imaging means 520 Slit projector 511 Rotating mirror 512 Slit light laser 513 Mirror drive unit 514 Laser drive unit 515 Synchronization control unit 516 Control Unit 530 CCD camera 531a, b switch 532 slit image memory 533 distance calculation processing unit 534 distance image memory 535 luminance image memory 600 parallax image generation unit 601 Pixel shift processing means 602 Camera focal length register 603 Camera CCD pixel pitch register 604 View point movement distance register 605 Pixel shift amount calculation means (or shift amount determination table) 607a to 607n Parallax image memory 700 Subject 900 Luminance image 920 Distance image 930, 950 Shift processed parallax images 940, 960 Corrected shifted processed parallax images 1001 Depth measuring means 1002 Luminance image capturing means

Continued on the front page F term (reference) 2F065 AA04 AA53 DD02 DD06 FF01 FF02 FF04 FF09 GG04 HH05 JJ03 JJ05 JJ26 LL13 QQ24 QQ31 5B050 BA04 BA09 EA27 5B057 BA02 CA12 CA16 CB13 CD14 DB02 DB03 DC30 5C06 AB

Claims (12)

[Claims] 1. A parallax image generating apparatus for generating a parallax image, comprising: a luminance image capturing unit that captures a luminance image of a display target; a depth measuring unit that measures depth distance data of the display target; An imaging parameter register that stores imaging parameters of an imaging unit; a viewpoint movement distance register that stores distance data between a viewpoint position and an observation viewpoint position of the luminance image imaging unit; and a distance image captured by the depth measurement unit. Based on the distance data for each pixel, the imaging parameter stored in the imaging parameter register, and the viewpoint moving distance data stored in the viewpoint moving distance register, the luminance image acquired by the luminance image imaging means. A parallax image generating means for determining a shift amount of each pixel constituting a parallax image and generating a parallax image from an arbitrary viewpoint Parallax image generation apparatus characterized by having a. 2. The image processing apparatus according to claim 1, wherein the parallax image generating unit includes: a distance data D for each pixel acquired from the distance image captured by the depth measuring unit; and a luminance image capturing as a capturing parameter stored in the capturing parameter register. The focal length of the means: f, the pixel pitch: p, and the viewpoint movement distance data: H stored in the viewpoint movement distance register are input values, and the luminance acquired by the luminance image imaging means based on the input values. The parallax image generating apparatus according to claim 1, wherein a parallax image from an arbitrary viewpoint is generated by determining a shift amount of each pixel forming the image. 3. The parallax image generating means segments the distance data of each pixel measured by the depth measuring means at a fixed distance data interval, assigns a unique segment number to each of the segments, and A shift amount determination table that sets a pixel shift amount corresponding to the pixel number, and sets the segment number of the shift amount determination table to each pixel based on distance data for each pixel obtained from the distance image captured by the depth measurement unit. And a shift amount set for the associated segment number is selected based on the shift amount determination table to determine a shift amount for each pixel. A parallax image generation device according to claim 1. 4. The parallax image generating means has a plurality of shift amount determination tables according to a focal length: f and a pixel pitch: p of the luminance image capturing means, and the viewpoint moving distance data: H. The table selection is executed based on the focal length: f, the pixel pitch: p, and the viewpoint moving distance data: H, which are input to the parallax image generation means, and the shift amount for each pixel is determined based on the selected table. The parallax image generation device according to claim 3, having a configuration. 5. The viewpoint movement distance data stored in the viewpoint movement distance register includes movement direction data indicating a movement direction, and the parallax image generating means includes a viewpoint movement distance data stored in the viewpoint movement distance register. 2. The apparatus according to claim 1, wherein the shift direction of each pixel constituting the luminance image acquired by the luminance image imaging means is determined based on the moving direction data of the image.
A parallax image generation device according to claim 1. 6. The parallax image generating means, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a pixel having the farthest distance data among pixels adjacent to the blank pixel, and 6. A configuration for generating a corrected parallax image by selecting based on a measurement value and embedding luminance data of the selected adjacent pixel as a luminance value of the blank pixel. A parallax image generation device according to any one of the first to third aspects. 7. A parallax image generating method for generating a parallax image, comprising: a luminance image capturing step of capturing a luminance image of a display object; a depth measuring step of measuring depth distance data of the display object; Based on the distance data for each pixel obtained from the distance image captured in the step, the imaging parameters of the luminance image, and the viewpoint moving distance data, each of the pixels constituting the luminance image acquired by the luminance image imaging unit A parallax image generating method, comprising: determining a shift amount for determining a pixel shift amount, and performing a shift of the determined pixel. 8. The parallax image generating step includes: a distance data D for each pixel obtained from the distance image captured in the depth measuring step; and a luminance image capturing as a capturing parameter stored in the capturing parameter register. The focal length of the means: f, the pixel pitch: p, and the viewpoint movement distance data: H stored in the viewpoint movement distance register are input values, and the luminance acquired by the luminance image imaging means based on the input values. The parallax image generating method according to claim 7, wherein a parallax image from an arbitrary viewpoint is generated by determining a shift amount of each pixel forming the image. 9. The parallax image generating step includes segmenting distance data of each pixel measured in the depth measuring step at a fixed distance data interval,
A step of assigning a unique segment number to each, and a step of determining a shift amount for each pixel based on a shift amount determination table in which a shift amount is set corresponding to the segment number. Item 8. The parallax image generation method according to Item 7. 10. The parallax image generating step includes: selecting a focal length: f, a pixel pitch: p, and a plurality of shift amount determination tables generated according to various values of the viewpoint moving distance data: H. Input focal length: f,
Pixel pitch: p, viewpoint moving distance data: 1 based on H
The parallax image generation method according to claim 9, wherein a selection process of one of the tables is performed, and a shift amount for each pixel is determined based on the selected table. 11. The parallax image generating step determines a shift direction of each pixel constituting a luminance image acquired in the luminance image capturing step based on viewpoint moving direction data included in the viewpoint moving distance data. The parallax image generating method according to claim 7, wherein: 12. The parallax image generating step further includes, for a blank pixel generated by the shift processing of the shift amount determined for each pixel, measuring a depth of a pixel having the farthest distance data among pixels adjacent to the blank pixel. 8. The method according to claim 7, further comprising the step of selecting based on the measurement value acquired in the step, and embedding the luminance data of the selected adjacent pixel as the luminance value of the blank pixel.
12. The parallax image generating method according to any one of claims 11 to 11.