JP2002159022A - Apparatus and method for generating parallax image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus for generating a parallax image capable of easily constituting an image of a plurality of visual points based on luminance images and one distance image from small amounts of different visual points. SOLUTION: A method for generating the parallax image comprises the steps of photographing the luminance images from the small amounts of the different visual points such as, for example, three visual points and one distance image; deciding a shifting amount of each pixel based on a focal distance (f) of a camera, a pixel pitch (p) of a CCD, a parameter of a moving distance H of the point, and distance data of each pixel obtained from the distance image; and executing an embedding process using a luminance value of an adjacent pixel or a luminance value obtained from the images of the different points, in response to continuity of a distance of a blank pixel position of the blank pixel generated by shifting.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an apparatus and a method for generating a stereoscopic display image. In particular, the present invention relates to a parallax image generating apparatus and a parallax image generating method capable of generating a parallax image viewed from an arbitrary viewpoint using one distance image and one or more plane images (luminance images) acquired from one viewpoint direction. Things.

[0002]

2. Description of the Related Art In recent years, with the spread of virtual reality / mix reality / electronic commerce and the like, a demand for stereoscopic display of a real image has rapidly increased. Currently used stereoscopic display devices include a glasses system, a lenticular system, a parallax barrier system, and the like. Both are stereo systems using binocular parallax, and only two types of images are displayed, a parallax image for the right eye and a parallax image for the left eye. Stereo method using binocular parallax,
In this method, left and right parallax images are captured by photographing a target object with two cameras installed at a certain interval, and the two parallax images are displayed on a stereoscopic display device.

However, recently, there is a strong demand for a stereoscopic display device having a more stereoscopic effect, and the viewpoint is not a stereoscopic system based on binocular parallax in which only the same stereoscopic image can be displayed regardless of where the eye is placed. A multi-view stereoscopic display device in which an image changes has been proposed. For example, "3D Television without 8-Eye Glasses" Journal of the Institute of Television Engineers of Japan: Vol. 48, No. 10, pp. 1267 (1994) proposes an eight-lens lenticular rear projection type stereoscopic display. The example shown in the magazine is a display example of an animation produced by computer graphics (CG). FIG. 1 shows a configuration example of a photographing system for displaying a live-action video on a display device.

As shown in FIG. 1A, eight cameras 101 to 108 are installed at a predetermined interval toward a subject 100. That is, the subject 10 is viewed from eight different viewpoints.
Capture the 0 image. The images captured by the cameras 101 to 108 in FIG. 1 are real parallax images 121 to 128 as shown in FIG. Real shot parallax images 121 to 128
As can be understood from FIG. 5, in the captured image, the relative positions of the front sphere and the rear cone forming the subject 100 differ depending on each camera position. This is subject 1
This is because the viewpoint at which the camera is installed differs from 00.

[0005] Eight cameras 101 to 1 as shown in FIG.
08, a photographed parallax image 12 from eight viewpoints taken by
The subject 100 is stereoscopically displayed by inputting 1 to 128 into the stereoscopic display means 180. In this case, since the image data from eight viewpoints is retained, the conventional 2
It is possible to display various images from more viewpoints compared to an image of only one viewpoint. However,
In order to acquire these many viewpoint images, the number of cameras required is equal to the number of viewpoints.

In addition, a super multi-view stereoscopic display system, which is an extension of the multi-view system, has been recently proposed. For example,
`` Hologram-Like Video Images by 45-View Stereoscop
ic Display ”SPIE Proc. Vol. # 3012 "Stereoscopic Dis
play and Applications VIII "pp. 154-166 (1997) describes a stereoscopic display device using 45 parallax images.
ch of 3D Display Using the Anamorphic Optics '' SPIE
Proc. Vol. # 3012 "Stereoscopic Display and Applica
In Actions VIII "pp. 199-207 (1997), a stereoscopic display device using 72 parallax images has been prototyped. In order to display a real image on these devices, 45 or 72 cameras are actually used. It is necessary to prepare parallax images taken from 45 or 72 viewpoints.

Since this photographing method lacks realism, for example, “Interpolation and compression of a multi-view three-dimensional image by self-similarity modeling”, Journal of the Institute of Television Engineers of Japan: Vol. 48, No. 10, pp1215-
A parallax image generation device as shown in 1221 (1994) has been proposed. A plurality of parallax images are captured by a camera from a point smaller than a required viewpoint. The parallax image viewed from the viewpoint between the photographing points is to be synthesized by image processing (interpolation processing) from a real parallax image. For example, three cameras A, B, and C for capturing images from different viewpoints are installed at predetermined intervals, and based on three real parallax images captured from the three cameras A, B, and C, This is to combine images that would be taken between cameras by image processing (interpolation processing).

FIG. 2 shows three cameras 201, 202 and 20.
3 shows an image capturing apparatus that captures images of the subject 200 from different viewpoints using 3 and generates a total of nine parallax images based on these three images.

In FIG. 2A, three cameras 20
1, 202 and 203 capture images of the subject 200 from different viewpoints. These photographed images are, as shown in FIG. 2B, a real parallax image 221 of the camera 201 and a camera 20.
2 is obtained as a real parallax image of the camera 203. These three real image parallax images are input to the parallax image interpolation / synthesis device 270, and an image that will be captured between the images is generated by interpolation processing.
Interpolated combined parallax images 231 to 233 and interpolated combined parallax images 234 to 236 shown in FIG. As a result, in addition to the actual parallax images 221, 222, and 223 actually captured by the cameras 201, 2020, and 203, six image-interpolated composite parallax images are generated, and the stereoscopic display unit 280 is generated based on a total of nine parallax images. Can display images from various angles.

[0010] Also, a published patent publication: Japanese Unexamined Patent Publication No.
7 describes another example of generating a parallax image. In the configuration described in this publication, a continuous stereo image (EPI: Epipolar Plane Image) is produced from a plurality of parallax images, and the depth of the object is determined based on the stereo image. Three-dimensional voxel data is generated from this depth information, and a new parallax image is generated by projecting the voxel data onto a two-dimensional plane in an arbitrary direction.

[0011]

However, in the above-mentioned various configurations, a multi-view stereoscopic display device or a super-multi-view stereoscopic display device requires a camera having the same number of viewpoints, which increases the shooting system. There was a disadvantage.

Further, in a method of capturing a plurality of parallax images and obtaining a parallax image viewed from a new viewpoint from the images by interpolation processing, and a method of generating a new parallax image by projecting voxel data onto a two-dimensional plane, There is a disadvantage that it is difficult to process video images in real time because data processing takes a long time. There is also a disadvantage that a high-performance and large-sized data processing device is required.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned various problems in the prior art, and is intended to realize a parallax image viewed from an arbitrary viewpoint in a short time and at a low cost. It is an object of the present invention to provide a parallax image generation device and a parallax image generation method that can be performed.

Further, according to the present invention, a parallax image viewed from an arbitrary viewpoint is shifted according to the viewpoint of a pixel of a luminance image obtained by one imaging means, and blank pixels generated by the shift are replaced with surrounding pixels or pixels. An object of the present invention is to provide a parallax image generation device and a parallax image generation method that generate a parallax image with high accuracy by executing an embedding process based on image data acquired by a parallax image capturing unit.

[0015]

According to a first aspect of the present invention, there is provided:
In a parallax image generating apparatus that generates a parallax image, a luminance image capturing unit that captures a luminance image of a display target, and a parallax image capturing that captures a luminance image of the display target at a different viewpoint from the luminance image capturing unit. Means, depth measurement means for measuring depth distance data of the display object, an imaging parameter register for storing imaging parameters of the luminance image imaging means, and a distance between a viewpoint position and an observation viewpoint position of the luminance image imaging means. A viewpoint movement distance register for storing data, distance data for each pixel obtained from the distance image captured by the depth measurement unit, an imaging parameter stored in the imaging parameter register, and stored in the viewpoint movement distance register A luminance image acquired by the luminance image imaging means based on the obtained viewpoint moving distance data. A parallax image generating unit that determines a shift amount of a pixel, executes a shift process, and executes a luminance value embedding process on a blank pixel generated by the shift to generate a parallax image from an arbitrary viewpoint. And a parallax image generating apparatus.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image capturing means is arranged at two positions sandwiching the luminance image capturing means.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means includes: distance data D for each pixel obtained from a distance image picked up by the depth measuring means; As the imaging parameters stored in the parameter register, the focal length: f and the pixel pitch: p of the luminance image imaging means, and the viewpoint moving distance data: H stored in the viewpoint moving distance register are input values. , A shift amount of each pixel constituting the luminance image acquired by the luminance image imaging means is determined, and a parallax image from an arbitrary viewpoint is generated.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means segments distance data of each pixel measured by the depth measuring means at a fixed distance data interval, and And a shift amount determination table in which a pixel shift amount corresponding to each segment number is set, and the distance data for each pixel obtained from the distance image captured by the depth measurement unit. The segment number of the shift amount determination table is associated with each pixel based on the shift amount, and the shift amount set for the associated segment number is selected based on the shift amount determination table, and the shift amount for each pixel is determined. It is characterized by having a configuration for determining.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means includes a focal length: f and a pixel pitch: p of the luminance image capturing means, and the viewpoint moving distance data: H. A plurality of shift amount determination tables corresponding to the focal length: f, the pixel pitch: p, and the viewpoint moving distance data: H input to the parallax image generating means.
The table selection is executed based on the table and the shift amount for each pixel is determined based on the selected table.

Further, in one embodiment of the parallax image generating device of the present invention, the viewpoint moving distance data stored in the viewpoint moving distance register includes moving direction data indicating a moving direction, and the parallax image generating means includes: The shift direction of each pixel constituting the luminance image acquired by the luminance image imaging means is determined based on the moving direction data in the viewpoint moving distance data stored in the viewpoint moving distance register. .

Further, in one embodiment of the parallax image generating apparatus according to the present invention, the parallax image generating means includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, among blank pixels adjacent to the blank pixel. A configuration in which a pixel having the farthest distance data is selected based on the measurement value of the depth measurement unit, and a corrected parallax image is generated by embedding the luminance value of the selected adjacent pixel as the luminance value of the blank pixel. It is characterized by having.

Further, in one embodiment of the parallax image generating apparatus according to the present invention, the parallax image generating means determines, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a corresponding pixel of a corresponding position pixel of the parallax image. It is characterized in that a corrected parallax image is generated by embedding a luminance value as a luminance value of the blank pixel.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the parallax image generating means includes a luminance image acquired by the luminance image imaging means, a luminance image acquired by the parallax image imaging means, , And applying the difference image as the luminance value of the blank pixel.

Further, in one embodiment of the parallax image generating apparatus according to the present invention, the parallax image generating means determines, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a position corresponding to the blank pixel position. The continuity of the distance data of the distance image of the distance image and the distance data of the distance image at the position corresponding to the adjacent pixel parallel to the shift direction is determined, and if it is determined that there is continuity, the pixel of the pixel adjacent to the blank pixel is determined. When the pixel having the farthest distance data is selected based on the measurement value of the depth measurement unit, the luminance value of the selected adjacent pixel is embedded as the luminance value of the blank pixel, and it is determined that there is no continuity. And embedding a luminance value of a corresponding position pixel of the parallax difference image as a luminance value of the blank pixel to generate a corrected parallax image.

Further, in one embodiment of the parallax image generating apparatus of the present invention, the continuity determination of the distance data performed by the parallax image generating means includes determining a continuity of a distance held by the parallax image generating means and a blank pixel position. The distance data of the distance image at the corresponding position and the difference between the distance data of the distance image at the position corresponding to the adjacent pixel parallel to the shift direction are compared to make a determination.

Further, according to a second aspect of the present invention, in a parallax image generating method for generating a parallax image, a luminance image capturing step of capturing a luminance image of a display object, and a viewpoint different from the luminance image capturing means. A parallax image capturing step of capturing a luminance image of the display object, a depth measurement step of measuring depth distance data of the display object, and a pixel-by-pixel image acquired from the distance image captured in the depth measurement step. Based on the distance data, the imaging parameter of the luminance image, and the viewpoint moving distance data, determine the shift amount of each pixel constituting the luminance image acquired by the luminance image imaging means, and execute the shift processing. And generating a parallax image from an arbitrary viewpoint by executing a luminance value embedding process on a blank pixel generated by the shift. In the parallax image producing method characterized by having a difference image generation step.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image capturing step captures images from two positions sandwiching the capturing position of the luminance image capturing step.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes: distance data D for each pixel obtained from the distance image picked up in the depth measuring step; As the imaging parameters stored in the parameter register, the focal length: f and the pixel pitch: p of the luminance image imaging means, and the viewpoint moving distance data: H stored in the viewpoint moving distance register are input values. The amount of shift of each pixel constituting the luminance image acquired by the luminance image imaging means is determined based on the above, and a parallax image from an arbitrary viewpoint is generated.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes segmenting distance data of each pixel measured in the depth measuring step at a fixed distance data interval, and , And a step of determining a shift amount for each pixel based on a shift amount determination table in which a shift amount is set in accordance with the segment number.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step is performed according to various values of the focal length: f and the pixel pitch: p, and the viewpoint moving distance data: H. From the plurality of generated shift amount determination tables, one table is selected based on the input focal length: f, pixel pitch: p, viewpoint movement, and distance data: H, and the selected table is selected. The shift amount for each pixel is determined based on the table.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes the step of obtaining the luminance acquired in the luminance image capturing step based on viewpoint moving direction data included in the viewpoint moving distance data. A shift direction of each pixel constituting an image is determined.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step further includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a pixel adjacent to the blank pixel. Selecting the pixel having the farthest distance data based on the measurement value obtained in the depth measurement step, and embedding the luminance data of the selected adjacent pixel as the luminance value of the blank pixel. And

Further, in one embodiment of the parallax image generating method according to the present invention, the parallax image generating step further includes a step of determining a correspondence between the parallax difference image and a blank pixel generated by a shift process of a shift amount determined for each pixel. The method includes embedding a luminance value of a position pixel as a luminance value of the blank pixel.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes the step of: obtaining the luminance image obtained in the luminance image capturing step, the luminance image obtained in the parallax image capturing step. A difference image generating step of generating the difference image of the above, wherein the difference image generated in the difference image generating step is applied as a luminance value of the blank pixel.

Further, in one embodiment of the parallax image generating method of the present invention, the parallax image generating step includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a position corresponding to the blank pixel position. Distance data of the distance image, including a distance continuity determination processing step of determining the continuity of the distance data of the distance image of the position corresponding to the adjacent pixel parallel to the shift direction, in the distance continuity determination processing step, When it is determined that there is continuity, a pixel having the farthest distance data among the pixels adjacent to the blank pixel is selected based on the measurement value of the depth measurement unit, and the luminance value of the selected adjacent pixel is determined. Embedding as a luminance value of a blank pixel, and in the distance continuity determination processing step, if it is determined that there is no continuity, the correspondence of the parallax difference image By embedding the luminance value of 置画 element as the luminance value of the blank pixel, and generates the corrected parallax images.

Further, in one embodiment of the parallax image generating method of the present invention, the distance continuity determination processing step executed in the parallax image generating step corresponds to a threshold value held in the parallax image generating means and a blank pixel position. And comparing the difference between the distance data of the distance image at the position where the distance image is located and the distance data of the distance image at the position corresponding to the adjacent pixel parallel to the shift direction.

[0037]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of a parallax image generating apparatus and a parallax image generating method according to the present invention.

[0038]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of a parallax image generating apparatus according to the present invention will be described with reference to FIG. FIG. 3 is a block diagram illustrating a configuration of the parallax image generation device according to the present embodiment. FIG. 4 illustrates a slit image, a distance image, a luminance image, a left parallax image, and a right parallax image captured using the parallax image generation device illustrated in FIG. 3.

The configuration shown in FIG. 3 will be described. The depth measurement and luminance image imaging unit 500 is a unit that acquires depth information of the subject 300 using a known light sectioning method. The slit light projector 520 emits the slit light 3 toward the subject 300.
60 are emitted intermittently in order from the left. C installed on an optical axis different from the light projecting axis of the slit light projector 520
The CD camera 530 captures an image of the subject 300 in synchronization with the intermittent emission of the slit light 360 by the slit light projector 520, and calculates the distance at each pixel based on the slit image 410 (see FIG. 4). A distance image 420 (see FIG. 4) is generated.

Further, the CCD camera 530 captures a luminance image 430 (see FIG. 4) of the subject 300 without projecting the slit light 360 at the end of the distance image generating operation. At this time, the left parallax image capturing unit 1003 and the right parallax image capturing unit 1004 do not emit the slit light 360, and the left parallax image 440 (see FIG. 4) and the right parallax image 4
50 (see FIG. 4) are respectively imaged.

The parallax image generating means 600 generates the distance image 42
0, luminance image 430, left parallax image 440, right parallax image 4
A parallax image viewed from each viewpoint is generated based on the display parameters of 50, the display parameters of the stereoscopic display means, and the viewpoint coordinates, and the stereoscopic display means 390 displays a stereoscopic image of the subject 300.

Next, the operation of generating a luminance image and a distance image will be described in detail with reference to FIG. FIG. 5 is a block diagram showing a detailed configuration of the depth measuring and luminance image capturing means 500 shown in FIG. Control unit 516 and synchronization control unit 515
, The mirror driving unit 513 controls the rotation of the rotating mirror 511.
With the angle fixed at a certain angle, the slit driving laser 512 emits the slit light laser 512, and the slit image 410 (see FIG. 4) of the subject at that time is read by the CCD camera 53.
Image is taken at 0.

The slit image 410 picked up by the CCD camera 530 is input to the slit image memory 532, and the distance calculation processing means 533 calculates the distance between the pixels exposed to the slit by trigonometry.

When the distance data based on one slit image is calculated, the slit light laser 512 is emitted with the rotating mirror 511 rotated and fixed by a small angle by the mirror driving unit 513, and the slit light at that time is emitted. The image 410 is captured by the CCD camera 530. As in the previous case, the distance of the pixel whose slit has been exposed is measured by the distance calculation processing means 533 by trigonometry. By repeating this, the distance of all the pixels imaged by the CCD camera 530 is calculated. Further, a distance image 420 (see FIG. 4) in which a gray value is set based on the distance data for each pixel is generated and stored in the distance image memory 534. The distance image stored in the distance image memory 534 is configured as data having a distance from the viewpoint of the CCD camera to the subject for each pixel. Finally, a luminance image 430 (see FIG. 4) of the subject is captured without emitting the slit light laser 512, and stored in the luminance image memory 535.

The switches 531a and 531b are
It opens and closes according to image data captured by the CCD camera 530, and the switch 531b closes and the switch 531a opens only when the captured image is a luminance image. If the captured image is a slit image, the state is controlled to the opposite state.

The left parallax image capturing unit 1003 and the right parallax image capturing unit 1004 perform the left parallax image 440 (see FIG. 4) and the right parallax image 450 at the same time when the luminance image 430 is captured.
(See FIG. 4).

The luminance image, the distance image, the left parallax image, and the right parallax image generated as described above are respectively transmitted to the luminance image transmission cable 340, the distance image transmission cable 350,
The image data is output to the parallax image generation unit 600 via the left parallax image transmission cable 320 and the right parallax image transmission cable 330.

The parallax image generating means 600 stores the luminance image stored in the luminance image memory 535 and the distance image memory 53
4, a parallax image of the subject 300 is generated based on the distance image, the left parallax image, and the right parallax image. Distance image 420, luminance image 430, and left parallax image 44
0, a process of generating a parallax image using the right parallax image 450 will be described in detail with reference to FIG. FIG. 6 is a block diagram illustrating a configuration of the parallax image generation unit 600.

The camera focal length register 602 of the parallax image generating means 600 stores the focal length f of the camera used for photographing the subject, for example, the CCD camera 530 in FIG. The camera CCD pixel pitch register 603 stores the pixel pitch p of the camera used. The viewpoint moving distance register 604 stores a moving distance H between a camera photographing viewpoint and a parallax point (observation viewpoint) for generating a parallax image.

The camera focal length f stored in the camera focal length register 602, the camera pixel pitch p stored in the camera CCD pixel pitch register 603, and the camera viewpoint and parallax point stored in the viewpoint moving distance register 604. The moving distance H between (observation viewpoints) is input to the pixel shift processing means 601, and the pixel shift amount calculating means 605 in the pixel shift processing means 605 converts the input data and the distance data of each pixel obtained from the distance image into Based on the calculated shift amount for each pixel, the pixel shift processing unit 601 executes the shift based on the calculated shift amount for each pixel.

FIG. 7 is a diagram for explaining the relationship between the distance D between the object 700 and the camera shooting viewpoint, the viewpoint moving distance H, and the pixel shift amount. The distance from the viewpoint to the subject is D,
F is the focal length of the camera, a is the distance between the optical point and the imaging point of the subject 700 on the CCD imaging surface photographed from the viewpoint A (camera viewpoint), and the new viewpoint when the viewpoint is moved by H Assuming that the distance between the image forming point of the subject 700 on the CCD imaging surface assumed to be imaged from A ′ (observation viewpoint) and the optical axis is x, the CCD imaging surface when the viewpoint is moved by H The moving distance of the imaging point of the subject 700: (x−a) is represented by the following equation.

[0052]

(Equation 1)

The pixel pitch on the CCD is p, and the subject 70 when the subject 700 is photographed from the viewpoint A ′ (observation viewpoint)
Object 7 on the CCD imaging surface between the imaging point 0 and the optical axis
The number M of moving pixels of the imaging point of 00 is obtained by the following equation using the number g of CCD pixels between the imaging point of the subject 700 and the optical axis.

[0054]

(Equation 2)

As shown in the above equation, the pixel shift amount M of the subject image on the CCD imaging surface when the viewpoint is moved is determined by using the focal length f of the camera, the pixel pitch p of the CCD, and the moving distance H of the viewpoint as parameters. It can be expressed as a function of the distance D from the viewpoint to the object and the number g of CCD pixels between the image point of the object 700 and the optical axis.

The pixel shift processing means 601 determines the shift value of each pixel of the luminance image input via the luminance image transmission cable 340 based on the pixel shift amount calculated by the above equation, and determines the shift value. The shift processing of the constituent pixels of the luminance image is executed according to the shift value. Images from various viewpoints based on various values of the moving distance H obtained by the shift processing are disparity image memories 1 to n, 607 (1) to
607 (n) and the three-dimensional display means signal cable 3
80 (see FIG. 3) and output to the stereoscopic display means 390.

The above example is an example in which the shift amount of each pixel is obtained by arithmetic processing. The distance D to the subject 700 is classified for each predetermined distance, and is segmented into, for example, 16 steps.
By storing the shift amount determination table in which the shift amount according to each distance is tabulated in the pixel shift processing unit 601, the arithmetic processing can be simplified and the speed of the shift processing can be increased. That is, a plurality of shift amount determination tables corresponding to the parameters of the focal length f of the plurality of cameras, the pixel pitch p of the CCD, and the moving distance H of the viewpoint are generated in advance and stored in the pixel shift processing means 601 to perform pixel shift. A table corresponding to the focal length f of the camera, the pixel pitch p of the CCD, and the moving distance H of the viewpoint input to the processing means 601 is selected, and the segment number corresponding to the distance data for each pixel obtained from the distance image is obtained. By performing a process of calculating the shift amount for the obtained segment number from the table, the calculation process can be simplified.

In a configuration having such a shift amount determination table, the pixel shift processing means 601 receives the distance image 420 (see FIG. 5) stored in the distance image memory 534 shown in FIG. 4), the shift amount of each pixel can be determined with reference to a table selected based on the segment number determined from the distance data. FIG. 8 shows an example of this shift amount determination table.

In the example of FIG. 8, the segment number is
This is a value when the distance D between the subject 700 and the camera shooting viewpoint is segmented into 16 steps, and the segment number is set to be larger as the distance D is smaller. That is, in the example of FIG. 8, when the subject distance is a subject at the maximum distance such as infinity, the segment number 0 is assigned, and when the subject distance photographed at the pixel is the minimum, that is, when the subject is a close subject. Is assigned a segment number 15.

An example of shift processing of a captured image using the pixel shift amount determination table shown in FIG. 8 will be described with reference to FIG. The luminance image 900 shown in FIG. 9 is a small image of the luminance image transferred from the luminance image memory 535 of the depth measurement and luminance image imaging unit 500 shown in FIG. 5 to the pixel shift processing unit 601 of the parallax image generation unit 600 shown in FIG. FIG. 9 illustrates a region image, and similarly, a distance image 920 illustrated in FIG.
The distance data of the small area portion corresponding to the luminance image 900 transferred from the distance image memory 534 of the depth measurement and luminance image imaging means 500 is indicated by a segment number.

The pixel shifting method will be described in detail. FIG.
FIG. 4 is an explanatory diagram showing a method of shifting pixels of a luminance image based on distance segment numbers (0 to 15) of a distance image. Each pixel of the luminance image 900 shown in FIG. 9 is set with the luminance value of each pixel when the luminance image is captured. Each rectangular area A11 to A55 in the figure corresponds to each pixel.

The distance image of the pixels corresponding to A11 to A55 is the distance image 920 shown in the upper part of FIG. 9, and distance segment numbers 0 to 15 are assigned as distance data of each pixel. As described above, the distance segment number is obtained by assigning a set value in 16 steps according to the distance from the subject to the viewpoint. In the example shown in FIG. 9, three types of segment numbers 2, 4, and 8 are assigned to the pixels included in the distance image 920, and the center pixel (segment number = 8) is a subject with a small distance D, that is, It can be seen that a subject having a large distance D, that is, a distant subject is captured in a near subject and surrounding pixels (segment number = 2). The shift amount of each pixel is obtained based on the segment number of each pixel according to the shift amount determination table.

In the shift amount determination table of FIG. 8 described above, the pixel shift amount is “0” when the segment number is “2”, the pixel shift amount is “1” when the segment number is “4”, When the segment number is “8”, the pixel shift amount is set to “2”.

First, processing for generating a parallax image 940 when the viewpoint is moved to the left from the photographing viewpoint will be described. The pixels of the luminance image 900 corresponding to the segment number = 2 in the distance image 920 (the outer 16 pixels) are as follows:
Since the pixel shift amount set to the segment number = 2 in the table shown in FIG. 8 is 0, it does not move. Regarding the pixels of the luminance image 900 corresponding to the segment number 4 in the distance image 920 (the inner eight pixels), the pixel shift amount set to the segment number = 4 in the table shown in FIG. For the pixel of the luminance image shifted to the right and corresponding to the segment number 8 of the distance image (one pixel at the center), the pixel shift amount set to the segment number = 8 in the pixel shift amount determination table shown in FIG. Therefore, it is shifted right by two pixels. That is, the parallax image 930 is obtained by shifting each pixel according to the table based on the luminance image 900.
Is obtained.

In this shift processing, for example, A25
Pixel is overwritten by the luminance value of the pixel A24,
For the pixel 35, the luminance value of the pixel A33 is set by the shift processing.

The parallax image obtained when the viewpoint is moved rightward from the photographing viewpoint is a process of shifting pixels to the left by the same operation. By this processing, an image of the parallax image 950 is obtained. The viewpoint movement distance data stored in the viewpoint movement distance register 604 illustrated in FIG. 6 includes movement direction data indicating the viewpoint movement direction, and the pixel shift processing unit 601 in the parallax image generation unit 600 is stored in the viewpoint movement distance register. The shift direction of each pixel constituting the luminance image is determined based on the moving direction data in the viewpoint moving distance data.

The parallax image 930 and the parallax image 95
0 indicates a loss due to the moved pixel (parallax images 930 and 95
0 blank pixels). The missing pixel embedding processing means 608 shown in FIG. 6 executes an embedding process of a so-called blank pixel, which is a defective pixel generated by the shift process.

The missing pixel embedding processing means 608 executes the embedding process of the blank pixel in the case of a blank pixel which occurs when the distance changes smoothly like the conical surface of the distance image 420, and in the boundary where a different object is located, for example, The processing is executed as a different process from the case of a blank pixel generated when the distance changes discontinuously, such as a boundary between a cone and a cube.

In the case of a blank pixel generated when the distance changes smoothly, the defective pixel embedding processing means 608 performs processing as shown in FIG. Pixel shift processing means 601
Generates the parallax images 940 and 960 corrected by copying a pixel at a long distance among luminance pixels adjacent to the defective pixel and filling the defective pixel. That is, for example, in the parallax image of FIG.
Three blank pixels (former A22, A23,
A24) will occur. Among the pixels adjacent to these blank pixels, pixels at a long distance are A21, A31, and A41.
Therefore, a process of copying these pixels to respective blank pixels is performed. Thus, the corrected shifted parallax images 940 and 960 are generated.

On the other hand, in the case of a blank pixel generated when the distance changes discontinuously, such as a boundary between different objects, the missing pixel embedding processing means 608 performs processing as shown in FIG. FIG. 10A shows a CCD shown in FIG.
The left parallax image 44 which is a luminance image 430 captured by the camera 530 and the left side is a luminance image obtained by capturing the subject 300 from a different viewpoint by the left parallax image capturing unit 1003.
0, the right parallax image 450 which is a luminance image obtained by photographing the subject 300 from a different viewpoint by the right parallax image capturing means 1004.

First, the defective pixel embedding processing means 608
Is a difference image between the luminance image 430 and the left parallax image 440,
Further, a difference image between the luminance image 430 and the right parallax image 450 is obtained. The difference image is executed as a process of extracting image data at a position not included in the luminance image from each of the left and right parallax images. The difference image thus obtained is shown in FIG.
(B), a left parallax difference image 460 is generated as a difference image between the luminance image 430 and the left parallax image 440, and a right parallax difference image 470 is generated as a difference image between the luminance image 430 and the right parallax image 450.

Next, the defective pixel embedding processing means 608
Executes blank pixel embedding processing. An arbitrary parallax image is generated by a shift process using the above-described pixel shift amount determination table. For example, as shown in FIG.
An arbitrary parallax image (before correction) / left-1,800 is generated as an image viewed from an arbitrary position on the left side, and an arbitrary parallax image (before correction) / right-1 as an image viewed from an arbitrary position on the right side. , 810 are generated. In these parallax images, blank pixels 880 are generated by the shift processing.

The defective pixel embedding processing means 608 applies parallax difference images 460, 4 to these blank pixels 880.
The luminance value of the corresponding position pixel is obtained from 70 and the luminance value is embedded. Arbitrary parallax image from left (before correction)-left-
For 1,800, the luminance value of the corresponding position pixel is acquired from the left parallax difference image 460, and the luminance value is embedded. Also, an arbitrary parallax image from the right side (before correction), right-1,81
For 0, the luminance value of the corresponding position pixel is acquired from the right parallax difference image 470, and the luminance value is embedded. FIG. 1 shows the result of the correction performed by embedding blank pixels.
Arbitrary disparity image shown in 0 (d) (after correction) -left-1,82
0, and arbitrary parallax image (before correction) · right-1,830.

As described above, by performing the blank pixel embedding process at the boundary where the distance changes discontinuously using the luminance value acquired from the actual parallax image, a more natural parallax image can be generated. Becomes possible.

The defective pixel embedding means 608
Is in the blank pixel part generated by the shift processing.
Regardless of whether the boundary is a boundary where the distance changes continuously or a boundary where the distance changes discontinuously, any one of the processes described with reference to FIG. 9 or FIG. 10 may be executed. In the blank pixel part generated by
It may be configured to determine whether the boundary is a boundary where the distance changes continuously or a boundary where the distance changes discontinuously, and select and execute a different process according to the determination result. The determination of the continuity of the distance data is performed by holding a predetermined threshold value by the defective pixel embedding processing unit 608 and determining the threshold value, the distance data of the distance image at the position corresponding to the blank pixel position, and the adjacent pixel parallel to the shift direction. It is possible to make a determination by comparing with a difference between the distance data and the distance data of the distance image at the position. If the difference is smaller than the threshold value, it is determined that there is continuity of the distance, and the embedding process based on the luminance value acquired from the surrounding pixels described with reference to FIG. 9 is performed. The embedding process based on the luminance value acquired from the parallax image described with reference to FIG. 10 is executed.

As described above, the parallax image generating apparatus of the present invention generates parallax images from various arbitrary viewpoints based on a distance image from one viewpoint, a luminance image, and two parallax images from left and right viewpoints. Can be displayed.
Therefore, imaging processing by a large number of cameras is unnecessary, and complicated processing such as generation of an image between cameras by interpolation processing based on a real shot image is unnecessary. Further, by adopting a configuration in which the shift processing is performed using the shift amount determination table, the shift amount for each pixel can be immediately obtained by the distance segment number obtained based on the distance image.

FIG. 11 shows a second example of the parallax image generating apparatus according to the present invention.
1 shows the configuration of the embodiment. The difference from the configuration shown in FIG.
This is because the depth measuring unit 1001 and the luminance image capturing unit 1002 are separated, and the operation is the same as the configuration in FIG.

In the above-described embodiment, the distance is measured by using the light section method. However, the present invention is not limited to this, and the distance is measured by other methods such as a spatial coding method, a phase difference detection method, and a stereo method. It is needless to say that the distance value may be determined. Also, in the above description, an example was described in which a single image was used as the luminance image. However, a plurality of luminance images having different viewpoints were captured, the luminance image closest to the observation viewpoint was selected, and images at various viewpoints were selected. It is good also as composition which generates.

Further, in the above-described embodiment, the movement of the viewpoint is limited to the horizontal direction, but the viewpoint can be moved in the vertical direction to generate a vertical parallax image. For the vertical movement, the same calculation is performed by setting the vertical movement distance V instead of the horizontal movement distance H in each of the above equations 1 and 2. In the shift processing using the table, the shift direction may be set to the vertical direction. In the case where the viewpoint is moved in the horizontal / vertical directions, the horizontal movement distance H and the vertical movement distance V are set in each of the equations (1) and (2), and the respective shift amounts are obtained. The result is set as the actual shift amount in the horizontal and vertical directions. Further, when performing horizontal / vertical shift processing using a table, a configuration in which both horizontal and vertical shift amounts are set, for example, shift amount = (X,
A table having a configuration in which the shift amounts in both the X-axis and the Y-axis directions are set as in Y) may be provided. With such a configuration, it is possible to generate a full parallax multi-viewpoint parallax image. As described above, according to the parallax image generation device of the present invention, the shift amount of each pixel can be immediately determined based on the table and the distance image.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0081]

As described above, according to the parallax image generating apparatus of the present invention, the distance measuring means, one luminance image capturing means, and a small number of parallax image capturing means different from the distance measuring means, for example, from the left and right viewpoints. It is possible to generate a multi-viewpoint parallax image with higher accuracy using only the data from the two parallax images described above, and it is not necessary to use a large number of cameras as in the related art, so that the configuration of the input system is simplified. In addition, since it is not necessary to convert the subject into a 3D model or convert it into three-dimensional voxel data, a parallax image can be generated simply by shifting the pixels of the luminance image according to a certain rule. It is possible to generate images from various viewpoints.

Further, according to the parallax image generating apparatus of the present invention, the distance measuring means and the one luminance image capturing means
For blank pixels generated according to the shift process according to the viewpoint, the continuity of distance data between adjacent pixels at the position corresponding to the blank pixel position is determined.If there is continuity, the luminance value of the adjacent pixel is determined. When the brightness value of the blank pixel is used, and when there is no continuity, a process of embedding the brightness value of the corresponding position pixel of the actual parallax image as the brightness value of the blank pixel is performed, so that a more accurate parallax image is generated. It is possible to do.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example (part 1) of a conventional parallax image generation device.

FIG. 2 is a diagram illustrating a configuration example (part 2) of a conventional parallax image generation device.

FIG. 3 is a diagram illustrating a configuration of a parallax image generation device of the present invention.

FIG. 4 is a diagram illustrating an image acquired by the parallax image generation device of the present invention.

FIG. 5 is a block diagram illustrating a configuration of a depth measurement and luminance image capturing unit of the parallax image generation device of the present invention.

FIG. 6 is a block diagram illustrating a configuration of a parallax image generation unit of the parallax image generation device of the present invention.

FIG. 7 is a diagram illustrating a processing method of a parallax image generation unit in the parallax image generation device of the present invention.

FIG. 8 is a diagram illustrating an example of a shift amount determination table of a parallax image generation unit in the parallax image generation device of the present invention.

FIG. 9 is a diagram illustrating a shift processing example of a parallax image generation unit in the parallax image generation device of the present invention.

FIG. 10 is a diagram illustrating an example of blank pixel embedding processing at the time of generating a parallax image in the parallax image generating device of the present invention.

FIG. 11 is a diagram showing the configuration of a second embodiment of the parallax image generation device of the present invention.

[Explanation of symbols]

100, 200 subject 101-108 camera 121-128 real parallax image 180,280 stereoscopic display means 221-223 real parallax image 270 parallax image interpolation / synthesis device 300 subject 320 left parallax image transmission cable 330 right parallax image transmission cable 340 luminance image transmission Cable 350 distance image transmission cable 360 slit light 380 stereoscopic display means signal cable 390 stereoscopic display means 410 slit image 420 distance image 430 brightness image 440 left parallax image 450 right parallax image 460 left parallax difference image 470 right parallax difference image 500 depth measurement and Luminance image imaging means 520 Slit projector 511 Rotating mirror 512 Slit light laser 513 Mirror drive 514 Laser drive 515 Synchronization controller 516 Controller 530 CCD camera 531 , B switch 532 slit image memory 533 distance calculation processing means 534 distance image memory 535 luminance image memory 600 parallax image generation means 601 pixel shift processing means 602 camera focal length register 603 camera CCD pixel pitch register 604 viewpoint movement distance register 605 pixel shift amount Calculation means (or shift amount determination table) 607 (1) to 607 (n) Parallax image memory 608 Missing pixel embedding processing means 700 Subject 800 Arbitrary parallax image before correction / left-1 810 Arbitrary parallax image before correction / right- 1 820 Corrected arbitrary parallax image / left-1 830 Corrected arbitrary parallax image / right-1 880 Blank pixel 900 Luminance image 920 Distance image 930, 950 Shift processed parallax image 1001 Depth measuring unit 1002 Luminance image capturing unit 1003 Left Parallax image capturing unit 1004 Right parallax image capturing unit

Claims (22)

[Claims] 1. A parallax image generating apparatus that generates a parallax image, comprising: a luminance image capturing unit that captures a luminance image of a display target; and a luminance image of the display target that is different in viewpoint from the luminance image capturing unit. A parallax image capturing unit for capturing, a depth measuring unit for measuring depth distance data of the display object, an imaging parameter register for storing imaging parameters of the luminance image capturing unit, a viewpoint position and observation of the luminance image capturing unit A viewpoint movement distance register for storing distance data with respect to a viewpoint position, distance data for each pixel obtained from a distance image captured by the depth measurement unit, an imaging parameter stored in the imaging parameter register, and the viewpoint Based on the viewpoint moving distance data stored in the moving distance register, the luminance acquired by the luminance image capturing means. Parallax image generation means for determining a shift amount of each pixel constituting an image, executing a shift process, and executing a brightness value embedding process for a blank pixel generated by the shift to generate a parallax image from an arbitrary viewpoint. A parallax image generation device, comprising: 2. The parallax image generating apparatus according to claim 1, wherein said parallax image capturing means is installed at two positions sandwiching said luminance image capturing means. 3. A parallax image generating means, comprising: a distance data D for each pixel obtained from a distance image picked up by the depth measuring means; and a luminance image pick-up as an image pick-up parameter stored in the image pick-up parameter register. The focal length of the means: f, the pixel pitch: p, and the viewpoint movement distance data: H stored in the viewpoint movement distance register are input values, and the luminance acquired by the luminance image imaging means based on the input values. The parallax image generating apparatus according to claim 1, wherein a parallax image from an arbitrary viewpoint is generated by determining a shift amount of each pixel forming the image. 4. The parallax image generating means segments the distance data of each pixel measured by the depth measuring means at a fixed distance data interval, assigns a unique segment number to each segment, and A shift amount determination table that sets a pixel shift amount corresponding to the pixel number, and sets the segment number of the shift amount determination table to each pixel based on distance data for each pixel obtained from the distance image captured by the depth measurement unit. And a shift amount set for the associated segment number is selected based on the shift amount determination table to determine a shift amount for each pixel. A parallax image generation device according to claim 1. 5. The parallax image generating means has a plurality of shift amount determination tables according to a focal length: f and a pixel pitch: p of the luminance image capturing means, and the viewpoint moving distance data: H. The table selection is executed based on the focal length: f, the pixel pitch: p, and the viewpoint moving distance data: H, which are input to the parallax image generation means, and the shift amount for each pixel is determined based on the selected table. The parallax image generation device according to claim 4, having a configuration. 6. The viewpoint movement distance data stored in the viewpoint movement distance register includes movement direction data indicating a movement direction, and the parallax image generating means includes a viewpoint movement distance data stored in the viewpoint movement distance register. 2. The apparatus according to claim 1, wherein the shift direction of each pixel constituting the luminance image acquired by the luminance image imaging means is determined based on the moving direction data of the image.
A parallax image generation device according to claim 1. 7. The parallax image generating means, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a pixel having the farthest distance data among pixels adjacent to the blank pixel, which is determined by the depth measuring means. By selecting based on the measured value and embedding the brightness value of the selected adjacent pixel as the brightness value of the blank pixel,
The parallax image generation device according to claim 1, further comprising a configuration for generating a corrected parallax image. 8. The parallax image generating means embeds a luminance value of a corresponding position pixel of the parallax image as a luminance value of the blank pixel for a blank pixel generated by a shift process of a shift amount determined for each pixel. The parallax image generation device according to claim 1, further comprising a configuration for generating a corrected parallax image. 9. The parallax image generating means, comprising: a luminance image acquired by the luminance image imaging means;
9. The configuration according to claim 8, wherein a process of generating a difference image from the luminance image acquired by the parallax image capturing unit is performed, and the difference image is applied as a luminance value of the blank pixel. Parallax image generation device. 10. A parallax image generating means, for a blank pixel generated by a shift process of a shift amount determined for each pixel, the distance data of a distance image at a position corresponding to the blank pixel position and a distance parallel to the shift direction. Determine the continuity with the distance data of the distance image at the position corresponding to the adjacent pixel, and when it is determined that there is continuity, the pixel having the farthest distance data among the pixels adjacent to the blank pixel is referred to as the depth measurement unit. Is selected based on the measured value of the selected pixel, and the luminance value of the selected adjacent pixel is embedded as the luminance value of the blank pixel.If it is determined that there is no continuity, the luminance value of the corresponding position pixel of the parallax difference image is calculated. The parallax image generation device according to claim 1, further comprising: generating a corrected parallax image by embedding the parallax image as a luminance value of the blank pixel. 11. The continuity determination of the distance data executed by the parallax image generating means includes a threshold value held by the parallax image generating means, distance data of a distance image at a position corresponding to a blank pixel position, and a shift direction. 2. A configuration in which a determination is made by comparing a difference between the distance data of a distance image at a position corresponding to an adjacent pixel parallel to and the distance data.
0. The parallax image generation device according to 0. 12. A parallax image generating method for generating a parallax image, comprising: a luminance image capturing step of capturing a luminance image of a display target; and a luminance image of the display target different from a viewpoint of the luminance image capturing means. A parallax image capturing step of capturing, a depth measuring step of measuring depth distance data of the display target, a distance data for each pixel obtained from the distance image captured in the depth measuring step, and a capturing of the luminance image Based on the parameters and the viewpoint moving distance data, the shift amount of each pixel constituting the luminance image acquired by the luminance image imaging means is determined, and a shift process is performed. Generating a parallax image from an arbitrary viewpoint by performing a value embedding process; A method for generating a parallax image, comprising: 13. The parallax image capturing method according to claim 12, wherein the parallax image capturing step captures images from two positions sandwiching the capturing position of the luminance image capturing step. 14. The parallax image generating step includes: distance data D for each pixel obtained from the distance image captured in the depth measuring step; and a luminance image capturing as a capturing parameter stored in the capturing parameter register. The focal length of the means: f, the pixel pitch: p, and the viewpoint movement distance data: H stored in the viewpoint movement distance register are input values, and the luminance acquired by the luminance image imaging means based on the input values. The parallax image generation method according to claim 12, wherein a shift amount of each pixel forming the image is determined to generate a parallax image from an arbitrary viewpoint. 15. The parallax image generating step includes segmenting distance data of each pixel measured in the depth measuring step at a fixed distance data interval,
A step of assigning a unique segment number to each, and a step of determining a shift amount for each pixel based on a shift amount determination table in which a shift amount is set corresponding to the segment number. Item 13. The parallax image generation method according to item 12. 16. The parallax image generating step includes the steps of: selecting a focal length: f, a pixel pitch: p, and a plurality of shift amount determination tables generated according to various values of the viewpoint moving distance data: H; Input focal length: f,
16. The method according to claim 15, wherein one table is selected based on the pixel pitch: p, the viewpoint movement, and the distance data: H, and a shift amount for each pixel is determined based on the selected table. The parallax image generation method described in the above. 17. The parallax image generating step determines a shift direction of each pixel constituting a luminance image acquired in the luminance image capturing step based on viewpoint moving direction data included in the viewpoint moving distance data. The parallax image generation method according to claim 12, wherein: 18. The parallax image generating step further includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, measuring a depth of a pixel having the farthest distance data among pixels adjacent to the blank pixel. 2. The method according to claim 1, further comprising the step of selecting based on the measurement value obtained in the step, and embedding the luminance data of the selected adjacent pixel as the luminance value of the blank pixel.
18. The parallax image generation method according to any one of 2 to 17. 19. The parallax image generating step further includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, a brightness value of a corresponding position pixel of the parallax difference image as a brightness value of the blank pixel. 18. The method according to claim 12, further comprising the step of embedding. 20. The parallax image generating step includes a difference image generating step of generating a difference image between the luminance image obtained in the luminance image capturing step and the luminance image obtained in the parallax image capturing step, 20. The parallax image generation method according to claim 19, wherein the difference image generated in the difference image generation step is applied as a luminance value of the blank pixel. 21. The parallax image generating step includes, for a blank pixel generated by a shift process of a shift amount determined for each pixel, the distance data of a distance image at a position corresponding to the blank pixel position, the distance data being parallel to the shift direction. The method includes a distance continuity determination processing step of determining continuity with distance data of a distance image at a position corresponding to an adjacent pixel.In the distance continuity determination processing step, when it is determined that there is continuity, a blank pixel is set. Selecting a pixel having the farthest distance data among the adjacent pixels based on the measurement value of the depth measurement unit, embedding the luminance value of the selected adjacent pixel as the luminance value of the blank pixel, and determining the distance continuity. In the processing step, when it is determined that there is no continuity, the luminance value of the corresponding position pixel of the parallax difference image is embedded as the luminance value of the blank pixel. It makes the parallax image producing method according to any one claims 12 to 17, characterized in that to generate the corrected parallax images. 22. The distance continuity determination processing step executed in the parallax image generation step includes: a threshold held by the parallax image generation means; distance data of a distance image at a position corresponding to a blank pixel position; 22. The parallax image generation method according to claim 21, further comprising a step of comparing the difference with distance data of a distance image at a position corresponding to a parallel adjacent pixel to determine the difference.