JP3524147B2 - 3D image display device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional image display device, and more particularly to a three-dimensional image display device having an image processing system for converting position two-dimensional information into three-dimensional information.

[0002]

Most of the image information is represented as a projection of an object in a three-dimensional space onto a two-dimensional surface. That is, most of the image information is recorded as a two-dimensional image. In this case, the object in the three-dimensional space may actually exist or may be a virtual object such as in the case of animation.

By the way, each point on an object in a three-dimensional space has its position represented by a three-dimensional coordinate value and has three-dimensional information in a color space. The three-dimensional information in the color space is, for example, hue, lightness, and saturation, and in the case of the additive method, an address counter (R),
The intensity of each color component of green (G) and blue (B) may be used.

In this way, corresponding to each point on the object being represented, each pixel of the image projected on the two-dimensional surface is
In addition to the two-dimensional coordinate value, at least 1 in the color space
Holds dimensional information. For example, for a monochrome image, the information in the color space is one-dimensional (only the brightness information), and for a color image, the information in the color space is three-dimensional.

Further, when a plurality of images (frames) are continuous in time series, it is a moving image and holds one-dimensional information in the time axis direction. In other words, the position corresponding to the object in the three-dimensional space is three-dimensional, the time is one-dimensional, and the color space is three.
Of the total 7-dimensional information, the information that is made into the 2-dimensional position information by projection is general image information (2-dimensional image)
And this image information is, for temporal color space information,
It holds at least one-dimensional color space information.

[0006] On the other hand, there is also known an image technology, such as a multiplex hologram or a CT (computer tomographic) image, which records the position information in the three-dimensional form without projecting it in the two-dimensional form, although it is small. In these three-dimensional image technologies, the three-dimensional coordinate values of substantially all points are held, the projection in any direction can be supported, and the projected image in any two-dimensional cross section can be obtained. is there.

Further, by using two two-dimensional images having different viewpoints, it is possible to stereoscopically view an object. There is also a method of utilizing this fact to manage the position information three-dimensionally even though it is incomplete only with two different projected images required for stereoscopic vision. The above-mentioned incompleteness is because it is not possible to obtain a projected image from an arbitrary direction because there is no information about a portion that cannot be seen as a shadow of the object from only the projected images of two different viewpoints. . In other words, if two images for stereoscopic viewing as viewed from the front side of the object are used, it is not possible to obtain three-dimensional information about the back side of the object in shadow.

Here, the complete three-dimensional information about the position of each point on the object, such as obtained by a multiplex hologram or a CT image, will be referred to as "complete position three-dimensional information". In addition, position information obtained from two projection images for stereoscopic viewing will be referred to as “stereoscopic position three-dimensional information”.

By the way, JP-A-5-126546, 5
In each of the gazettes of -126547 and 5-181979,
A technique for obtaining three-dimensional shape data from two-dimensional shadow image information is disclosed.

The above technique utilizes the fact that the surface can be calculated by differentiating a three-dimensional shape, and the shadow information can be obtained by converting from the surface using a reflection map. This conversion is performed in reverse and the object is calculated from the shadow information. It seeks to restore a three-dimensional shape by finding the direction of the surface of an object and integrating the obtained surface.

The display of an image projected in a normal two-dimensional position has made remarkable progress such as a CRT display and a liquid crystal display. On the other hand, in recent years, a head mounted display (HM
D), lenticular display, etc.
A three-dimensional display has been devised, and expectations for three-dimensional images are increasing.

[0012]

By the way, most of the existing image information software such as photographs, paintings, movies, etc. hold only two-dimensional information as position information. In recent years, head mounted display (HM
D), lenticular display, and other three-dimensional displays with stereoscopic vision have been developed.

Although expectations for three-dimensional images are increasing, stereoscopic viewing cannot be performed with existing position information two-dimensional video software. Therefore, it is conceivable to convert existing two-dimensional position information video software into stereoscopic position three-dimensional information, but such an attempt has not been made because the number of dimensions is insufficient.

On the other hand, the above-mentioned JP-A-5-126546,
In the case of the techniques described in the publications of Nos. 5 to 126547 and 5 to 181979, two-dimensional shadow image data in the presence of illumination light from a predetermined direction is targeted. Therefore, these techniques cannot be applied to general cases where there are a plurality of light sources or the light source position is unknown.

Further, when an object having a complicated shape is targeted, there is a problem that the amount of calculation becomes enormous, and
There is a problem that it is completely useless when a distant view and a near view are mixed in one image.

When the position three-dimensional information is captured as video information, not only the "complete position three-dimensional information" but also the "stereoscopic position three-dimensional information" is imaged by the compound eye. There is a problem that the image pickup system becomes large in size because it is necessary to use the system for capturing. The large scale of the imaging system as described above has been an obstacle to the widespread use of video software based on the three-dimensional position information.

In view of the above problems, the present invention is stereoscopic by inputting two-dimensional existing image information (including a moving image) as image information obtained by an image pickup system having a simple structure. The purpose is to enable visual display.

[0018]

A three-dimensional image display device of the present invention is an image input system for inputting an image having two-dimensional position information, and an input for capturing and holding an image input from the image input system. An image holding unit, an image processing unit that restores three-dimensional position information for each pixel of the input image based on the pixel value of the pixel and the pixel values of pixels around the pixel, and the image processing unit restores the three-dimensional position information. An image processing system having a two-dimensional position image generating means for generating a two-dimensional image corresponding to a viewpoint different from the image input from the image input system based on the three-dimensional position information, and output from the image processing system. And a three-dimensional display system for displaying a stereoscopic image based on two images with different viewpoints.

Another feature of the present invention is that
In the above three-dimensional image display device, it is determined which of an input image having two-dimensional position information projected from one viewpoint and an input image having two-dimensional position information projected from two viewpoints is input. The input image type determining means is provided in the image input system, and the input of the image input system is
In the case of an input image having two-dimensional position information projected from two viewpoints, the input image signal is directly sent to the three-dimensional display system without passing through the image processing system. I am trying.

Another feature of the present invention is that the image processing means of the image processing system restores three-dimensional position information by inference using a knowledge base.

Further, another feature of the present invention is that an image input system for inputting an image, an input image holding means for fetching and holding an input image input from the image input system, and each of the input images. Regarding a pixel, a point in a three-dimensional space corresponding to the pixel is projected onto a two-dimensional surface from a viewpoint different from the viewpoint of the input image, based on the pixel value of the pixel and the pixel values of pixels around the pixel. Image processing means for calculating a pixel position corresponding to the viewpoint on the two-dimensional surface in a case by inference using a knowledge base,
And an image processing system having a two-dimensional position image generating means for generating an image projected from the different viewpoints and having two-dimensional position information based on the pixel position calculated by the image processing means, and the image processing. And a three-dimensional display system for displaying a stereoscopic image based on two images with different viewpoints, the image information being output from the system. In addition, other features of the present invention include:
In the above three-dimensional image display device, it is determined which of an input image having two-dimensional position information projected from one viewpoint and an input image having two-dimensional position information projected from two viewpoints is input. The input image type determining means is provided in the image input system, and the input of the image input system is 2
When there are two input images projected from one viewpoint and having two-dimensional position information, the input signal is directly sent to the three-dimensional display system without passing through the image processing system. There is. Another feature of the present invention is that the image processing means of the image processing system, when two or more pixels of the input image correspond to the same pixel position on the two-dimensional surface, It is characterized in that the pixel determined to be on the near side by inference and the pixel position are associated with each other. Another feature of the present invention is that the two-dimensional position image generating means of the image processing system is a pixel position on the two-dimensional surface and does not correspond to a pixel of the input image. In this case, the pixel value corresponding to the pixel position is calculated by extrapolation or interpolation processing from the pixel value corresponding to the pixel position close to the pixel position to generate an image. Another feature of the present invention is that two images with different viewpoints displayed on the three-dimensional display system are selected from the image generated by the two-dimensional position image generating means and the input image. It is characterized by two images. Another feature of the present invention is that the input image is a dominant-eye image, and the image generated by the two-dimensional position image generating means is an anti-dominant image corresponding to the dominant-eye image, The three-dimensional display system is characterized by displaying a stereoscopic image based on the dominant-eye image and the dominant-eye image. Another feature of the present invention is that the input image is a central image, and the two-dimensional position image generating means generates a right eye image and a left eye image corresponding to the central image. The three-dimensional display system is characterized by displaying a stereoscopic image by the right-eye image and the left-eye image. Another feature of the present invention is that each pixel of the input image has at least color space information 1 of time information and color space information as a pixel value.
It is characterized by holding information for dimensions.

[0022]

Since the present invention has the above technical means, the following effects can be obtained. That is, when performing stereoscopic vision using a three-dimensional display such as a head-mounted display or a lenticular display, the two two-dimensional images used for the stereoscopic vision have different viewpoints, but there are large differences between them. Do not mean.

Therefore, no matter how complicated the two-dimensional image is, by locally deforming each area of the two-dimensional image, the two-dimensional image is paired with the two-dimensional image for stereoscopic viewing. It is possible to generate a two-dimensional image that is rendered. The question is how to make this local deformation.

Then, when attention is paid to the two-dimensional image which is the input image, the two-dimensional image itself includes a clue to the three-dimensional position information. Therefore, by using this clue, it becomes possible to restore the three-dimensional position information for each pixel of the input image from the pixel value of that pixel and the pixel values of the surrounding pixels.
Then, if the three-dimensional position information corresponding to each pixel can be restored, an image (converted image) from a viewpoint different from that of the input image is generated based on the restored position information, whereby a pair of stereoscopic images is obtained. Image will be obtained.

By the way, from a single two-dimensional image such as a painting or a photograph, a person unknowingly grasps the three-dimensional structure appearing in the image. Here, it is assumed that the two-dimensional image is based on perspective projection, that is, is represented by the perspective method. It is becoming clearer how humans perceive 3D structures ("Vision: Computational Theory of Vision and Brain Expression", by David Ma, Toshiro Inui,
Translated by Hiroshi Ando, Sangyo Tosho, 1987). For example,
Knowledge of the typical size of an object, by deciding that the smaller the object is, the further away it is, and recognizing what the object appears in the image (whether it is a person or a house). It estimates the distance to the object based on.

Therefore, by accumulating the inference rules that objectively express the criteria that humans unconsciously judge to make a knowledge base, and infer using this knowledge base,
For each pixel of the input image, the position of the point in the three-dimensional space corresponding to that pixel can be estimated. From the viewpoint of the “stereoscopic position three-dimensional information”, it is sufficient to know the corresponding pixel position in the projection image from a viewpoint different from the viewpoint of the input image, and therefore inference using a knowledge base can be easily performed. If this pixel position is known, it is possible to generate a two-dimensional image from a viewpoint position different from the input image.
"Stereoscopic position three-dimensional information" can be obtained.

That is, according to the present invention, with respect to existing image information (including a moving image) that is two-dimensional as position information,
The position information one-dimensional information lost by the projection is partially restored as position information three-dimensional information, and at least one projected two-dimensional position image from another viewpoint direction is based on the three-dimensional position information. By providing an image processing system for producing, the two-dimensional position image information for two viewpoint directions from the original projected two-dimensional position image and the produced two-dimensional position image information is displayed on the three-dimensional display system. 3D position information for stereoscopic vision "can be provided.

Specifically, if the input image is to be stereoscopically viewed, one two-dimensional image having different viewpoints can be obtained. Therefore, if the input image is the dominant-eye image, it is a dominant hand. The eye image may be generated.

It is also possible to select two viewpoints that are symmetrical with respect to the viewpoint of the input image and generate a total of two images projected from each of these two viewpoints. in this case,
The input image is the central image, and the generated images are the right-eye image and the left-eye image. In the present invention, a plurality of images can be generated from one input image as needed. Furthermore, it is possible to use the generated image as a new input image and generate the next image based on this new input image. That is, it is possible to generate successive images from one input image, such as child images and grandchild images.

When a plurality of images are generated in this way, the viewpoints of the plurality of images are different from each other, so that the pixels of the input image and the pixels of the generated image (converted image) are one-to-one.
Does not always correspond. When a plurality of pixels on the input image side correspond to one pixel of the converted image to be generated, the pixel (on the input image side) determined to be closest by inference may be selected. On the other hand, with respect to the pixels of the generated converted image that do not have pixels corresponding to the input image, the pixel value can be determined by interpolation or extrapolation.

In the three-dimensional image display device of the present invention, a two-dimensional position image generating means is provided on a three-dimensional display system (three-dimensional display device) such as a head mounted display or a lenticular display based on one input image. When one image is generated, a stereoscopic image may be displayed by this image and the input image.

Further, when the two-dimensional position image generating means generates a plurality of images, it is possible to display a stereoscopic image by two images having different viewpoints among the generated images and the input image. Good.

[0033]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, an embodiment of the three-dimensional image display device of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a three-dimensional image display device according to the first embodiment of the present invention. As shown in the figure, the three-dimensional image display device of this embodiment includes an image input system 1, an image processing system 2 and a three-dimensional display system 3.

In the image input system 1, one input image projected from one viewpoint and two-dimensional as position information, or two input images projected from two viewpoints and two-dimensional as position information. Sheets are entered. Then, the image information is sent to the image processing system 2 or the three-dimensional display system 3 depending on the type of the input image. In this embodiment, the image input system 1 is provided with input image type determining means 1a for determining the type of input image. FIG. 2 shows an example of the flow of the input image type determination means 1a.

In this example, the operation is started and the step P
When an image is input in step 1, it is determined in the next step P2 whether the input image is a plurality of (two) images from a plurality of (two) viewpoints.

This is "stereoscopic position three-dimensional information" when there are a plurality of input images, so it is sent directly to the three-dimensional display system 3 and three-dimensionally displayed (step P3). . Further, in the case of one image from one viewpoint, it is sent to the image processing system 2 to generate an image viewed from another viewpoint (step P4). The configuration of the image processing system 2 will be described in detail with reference to FIG.

As shown in FIG. 3, here, the input image 10 is the original image information which is two-dimensional as the position information.
And an image with a different viewpoint from the input image 10 (converted image)
Is generated, and the input image and the generated converted image are sent to the three-dimensional display system 3 as a stereoscopic image for display.

The input image 10 is, specifically, a photograph, a painting, or the like, in which positional information in a three-dimensional space is actually projected onto a two-dimensional surface. The image information includes two-dimensional position information as a still image, one-dimensional lightness as a color image for a monochrome image, and three-dimensional information in which saturation and hue are further added for a color image. Here, it is assumed that the "dominant eye image 51" as illustrated is input as the input image 10. The dominant eye image is an image from the position and direction of the dominant eye.

In order to perform image processing, generally, an image input unit 11 which reads the input image 10 and digitizes it is provided. Then, the read image information is stored in the frame memory 12 for the input image as a pixel value (information) for each pixel, and is supplied from the frame memory 12 to the image processing unit 14 and the three-dimensional display system 3. It has become. When the input image 10 from the image input system 1 is given as digital information, the input image 10 may be directly input to the frame memory 12.

The image processing unit 14 infers, for each pixel of the input image 10, the one-dimensional position information lost by projection from the information of that pixel and surrounding pixels,
This is for partially restoring as three-dimensional position information. Specifically, the frame memory 12 for the input image is referred to, and for each pixel, the point in the three-dimensional space corresponding to the pixel is determined based on the pixel value of the pixel and the pixel value of the pixel in the vicinity of the pixel. When the image is projected onto a two-dimensional surface (converted image surface) from a viewpoint different from the viewpoint of the input image, the pixel position on the projection surface (converted image surface) is calculated by inference based on a knowledge base.

The image processing unit 1 is provided next to the image processing unit 1.
The projected two-dimensional position image generation unit 15 that receives the image information restored as the position three-dimensional information at 4 and generates a two-dimensional image (converted image) from a viewpoint different from the input image 10
Is provided. At this time, the color space information may be converted using the reflection and diffusion characteristics of the object.

Specifically, the projection two-dimensional position image generation unit 1
5 is based on the pixel position calculated by the image processing unit 14,
The pixel value of each pixel (the position of which is represented by the calculated pixel position) on the converted image side is determined from the pixel value of the input image. For simplicity, the pixel value of the corresponding pixel on the input image side may be used as it is, but the pixel value may be determined by inference from the color space information.

Further, the projection two-dimensional position image generator 15 is
As will be described later, when there is a pixel (missing image) that does not correspond to the input image on the converted image side, processing for determining the pixel value of this pixel by interpolation or extrapolation is also performed. Further, a converted image frame memory 13 for storing the converted image is provided, and the converted image is output from the frame memory 13 to the three-dimensional display system 3. in this case,
Since the input image 10 is the dominant-eye image 51, the viewpoint position and direction of the converted image are generally taken to be those of the dominant hand, and the converted image is the “dominant-eye image” as shown in the figure. Image 52 ".

As the three-dimensional display system 3, a head mounted display (HMD), a system for displaying right and left eye images by using polarized light in an overlapping manner, a system using a lenticular sheet, and the like can be used. As shown in FIG. 4, the three-dimensional display system 3 includes an input image (dominant eye image 51) sent from the input image frame memory 12 and a converted image (reverse image) sent from the converted image frame memory 13. The dominant-eye image 52) is used as two images having a parallax (a right-eye image and a left-eye image), and these two images are projected and stereoscopically displayed as a three-dimensional image.

Next, the inference in the image processing unit 14 will be described. Various types of inference rules (knowledge rules) used for inference performed by the image processing unit 14 are conceivable.
Specifically, the estimation exemplified below may be performed.

(1) The vanishing point is calculated from the contour of the element graphic in the image (especially the use of the trapezoidal part in the straight graphic) to estimate the position of the viewpoint and the depth of each pixel. This is because the trapezoidal portion in the image is often a rectangle in the real space (three-dimensional space) that becomes a trapezoid by projection.

(2) The overlapping of element figures in the image is determined by the stepwise change of density and hue and the crossing condition of contour lines (L type, T type).
Type, X type, etc.) to estimate the depth of each pixel.

(3) Attention is paid to the repetitive figure in the image, and the depth of each pixel is estimated from the regularity of the change in the size and distortion. This is because the repeated figures in the image are often objects of the same size in the real space.

(4) Paying attention to the shadow in the image and the reflection in the mirror surface, the light source position, the front-rear relationship, the distance gradient, etc. are judged from the shape and the density gradient in the figure to estimate the depth of each pixel. To do.

(5) Recognize what an object of a graphic element in an image represents (for example, a house, a person, etc.), and store the absolute size and the corresponding structure of the details of the object stored and the graphic. The depth of the thing is estimated by comparing the size of the thing and its similarity correspondence. For the recognition here, "a pencil is often placed on the desk."
Consistency with surroundings such as can also be used.

(6) For an object judged to be on the back side, the depth information of the hidden part is estimated by extrapolation from the regularity of the visible part.

(7) A general law, for example, in the place where gravity works, an object below the screen is often nearby, and an object above the screen excludes the ceiling and street lamps (lamps). Are often far away. Also, the spatial frequency characteristic and the spectral characteristic are often different between the near view and the distant view, and the poor resolution (blurred portion) is often a distant portion. Furthermore, things whose brightness information changes gently are often composed of curved surfaces, those enclosed by straight lines are polyhedrons, and those enclosed by curves are often solids enclosed by curved surfaces excluding discs. , Etc. are used to estimate the depth of each pixel.

Depth determination is performed by using these estimations singly or in combination, and the three-dimensional position information is restored although it is partial. To improve accuracy, it is desirable to combine these depth determination techniques. Next, the operation of this image processing apparatus will be described in detail with reference to the flowcharts shown in FIGS. 5 and 6 are the first half part and the second half part of the same flowchart, respectively, in which steps P1 to P12 are shown in FIG. 5 and steps P13 to P1 are shown in FIG.
9 are shown respectively.

First, the image input section 11 has color space information [lightness, (hue), (saturation)] as the input image 2
Read the 3D image and decompose it into pixels (Step P
1). The input pixel 10 decomposed into pixels is written in the frame memory 12 for the input image (step P2).

Then, from the frame memory 12, one pixel in the input image to be converted is taken out (step P3). The image processing unit 14 sets the extracted pixel of the input image to the pixel position (address) of the corresponding pixel in the converted image based on the image information (pixel value) of the pixel and the pixel values of the pixels around the pixel. Is calculated (step P4).

The pixel values of the surrounding pixels are the frame memory 12
And the inference using the knowledge base as described above is utilized for the calculation of the pixel position. If the input image 10 is a dominant-eye image, the converted image is, for example, an anti-dominant-eye image. The calculated pixel position is the projected two-dimensional position image generation unit 1.
Sent to 5.

The projected two-dimensional position image generator 15 first
It is determined whether the calculated pixel position is within the frame of the converted image (step P5). When it is determined that the color space information is outside the frame, the process jumps to Step P11. When it is determined that the color space is inside the frame, the frame memory 13 for the converted image is referred to, and the color space information is already stored in the pixel position calculated in the converted image. It is judged whether or not it has been written (step P6).

If the result of determination in step P6 is that it is not written, processing jumps to step P9. Also,
In the case of being written, different pixels on the input image side correspond to the same pixel position on the converted image side. Therefore, in the three-dimensional space corresponding to the pixel (on the input image side) to be handled now. It is determined whether or not the position is in front of the position in the three-dimensional space corresponding to the already written color space information, that is, whether or not it is on the viewpoint side (step P7).

If the result of this determination is that it is not on the front side,
Since that information is the information of the hidden part in the converted image, it is ignored and the process jumps to step P11. If it is on the front side, the process proceeds to step P8, and the already written information should not be seen in the converted image.
The information written in the target pixel position is erased, and the process proceeds to step P9.

In step P9, color space information (pixel value) for the pixel at the calculated pixel position (converted pixel) in the converted image is calculated by inference using the knowledge base. Then, the obtained color space information is written in the calculated pixel position in the frame memory 13 for the converted image (step P10). With the above processing, the conversion for one pixel extracted from the input image 10 is completed, and the process moves to step P11.

In step P11, it is next determined whether or not the pixel to be converted, that is, the pixel which has not been converted remains in the input image. Then, if there are remaining pixels to be converted, the process proceeds to step P12, and the pixels to be converted are set to the frame memory 12 for the input image.
Take out from. Next, the process returns to step P4 in order to execute the above processing on the newly extracted pixel.

On the other hand, if the result of determination in step P11 is that there are no pixels remaining to be converted, that is, if the conversion processing has been executed for all images of the input image, in order to perform processing for the missing pixels. Control goes to step P13.

At step P13, one pixel to be checked is taken out from the frame memory 13 for converted image. When the extraction is completed, the process proceeds to step P14, and it is determined whether the color space information is written in the extracted pixel. If the result of this determination is that the color space information has been written, that pixel is not a missing pixel, and so processing jumps to step P17.

On the other hand, if it is not written, the pixel is a missing pixel, that is, the pixel corresponding to the input image side does not exist, so the color space information to be given to this pixel is supplemented by interpolation or extrapolation. This is when it is necessary. In this case, the process proceeds to step P15, the converted image frame memory 13 is referred to, and the color space information of the missing pixel is calculated based on the pixel values of the pixels around the missing pixel.

Note that the color space information of the missing pixel may be calculated simply by using the average value of the pixel values of neighboring pixels as the color space information of the missing pixel, but here, the knowledge base is used. The color space information is obtained by inference. Then, when the calculation is completed, next, step P1
In step 6, the calculated color space information is written in the corresponding pixel on the frame memory 13. As a result, the process in the case where one pixel extracted from the frame memory 13 for converted images is a missing pixel is completed, so that the process in step P17 is completed.
Move to.

In step P17, it is determined whether or not there is a pixel in the converted image that has not been checked for missing pixels. If there are unchecked pixels, the process proceeds to step P18, the pixels to be checked are extracted, and the process returns to step P14.

If it is determined in step P17 that all the pixels in the converted image have been checked, the process proceeds to step P19, and the converted image is rechecked from the viewpoint of image quality. This recheck is performed by determining whether or not all the pixels of the converted image have a proper relationship with surrounding pixels.

Then, it is collated with the knowledge base, and if the relevance is not appropriate, that is, if the image is unnatural, the image processing unit 14 performs minute correction. Further, in some cases, the user may make a minute correction. The recheck process in step P19 is performed as needed, and can be omitted satisfactorily.

With the above processing, the generation of the converted image from the input image 10 is completed. Then, the frame memory 12
The input image in the frame and the converted image in the frame memory 13 are
Both are sent to the three-dimensional display system 3 and displayed as a stereoscopic image by the three-dimensional display system 3.

Next, a second embodiment of the present invention will be described with reference to FIG. In the second embodiment, the number of converted images generated for one captured input image is not limited to one, and it is possible to generate a plurality of converted images.

FIG. 7 is a block diagram showing the image processing system of the second embodiment of the three-dimensional image display device of the present invention. The image input system 1 and the three-dimensional display system 3 are basically the same as those in the above-described first embodiment, and detailed description thereof will be omitted. In the image processing system according to the second embodiment, two converted images are generated for one input image 20.

A frame memory 22 for the input image is provided on the output side of the image input section 21 for reading the input image 20, and an image processing section 24 is provided on the output side of the frame memory 22. The configuration of the image processing unit 24 is the same as that of the image processing unit 14 of the first embodiment, but in the case of the second embodiment, two converted images are generated, so that the knowledge base is used. It is configured to calculate two pixel positions with respect to one pixel of the input image 20 by the above inference.

In this case, the pixel position from each of the two viewpoints symmetrically arranged with respect to the viewpoint of the input image 20 is calculated. That is, the input image 2
If 0 is the central image 53 as shown, the converted images in this embodiment are the left-eye image 54 and the right-eye image 55. On the output side of the image processing unit 24, a projected two-dimensional position image generation unit 25 to which the calculated pixel position is input is provided.

The projection two-dimensional position image generation unit 25 has the same configuration as the projection two-dimensional position image generation unit 15 of the first embodiment described above, but one converted image (for example, the left eye image) is One converted image is generated from the corresponding pixel position and stored in the frame memory 23L for one converted image, and the other converted image is generated from the pixel position corresponding to the other converted image (for example, the right-eye image) and the other converted image is generated. The converted image is stored in the frame memory 23R.

In this way, each frame memory 23L, 23
Each converted image stored in R is sent to the three-dimensional display system 3 such as a head-mounted display or a lenticular display as two images having parallax and is stereoscopically displayed. In the present embodiment, since the number of converted images generated from the input image is two, it takes a longer processing time than the first embodiment.

Further, in the image processing apparatus of this embodiment, since the portion hidden in the input image but visible on the converted image is estimated by extrapolation or inference, the viewpoint of the input image to that of the converted image is estimated. As the distance to is increased, the degree of error in the image information due to estimation uncertainty increases rapidly.

However, in this embodiment, when it is attempted to obtain the same parallax between two images, the processing time is longer than that in the first embodiment, but the viewpoints of the individual converted images and the input images are different. Since the distance from the viewpoint is half that in the first embodiment, it is possible to reduce errors in the image information as a whole as compared with the first embodiment.

Next, a third embodiment of the present invention will be described with reference to FIG. The image processing apparatus of the present embodiment can be applied not only to a still image but also to a moving image having two-dimensional position information.

FIG. 8 is a block diagram showing the image processing apparatus of the three-dimensional image display apparatus of this embodiment. The image processing apparatus according to the third embodiment is similar to the image processing apparatus shown in the first embodiment in that the image input unit 31, the input image frame memory 32, the converted image frame memory 33, and the image processing unit. It is composed of a unit 34 and a projected two-dimensional position image generation unit 35, and each of these constituent elements corresponds to a moving image.

A movie, a video or the like is used as the input image 30, and the image input section 31 is configured to sequentially read the input image 30 frame by frame and decompose it into pixels. In this case, since the input image 30 is a moving image, the position information is two-dimensional and the color space information is at least one-dimensional, and also has sequential information in the time axis direction. Corresponding to this, in the inference in the image processing unit 34, the following inference rule is used in addition to the inference rule described in the first embodiment.

That is, (8) a number of different frames in the time axis direction are analyzed to extract a moving object that is moving relative to the photographing viewpoint, and the motion parallax of the object is used to extract the object. Estimate the movement in the depth direction of.

(9) By utilizing the fact that the hidden part becomes visible as the object moves, the depth hidden behind the figure is determined from the information of the corresponding object in another frame that is temporally different. Estimate direction information.

Here, it is assumed that the input image 30 is a sequential dominant eye image 56 as shown in the figure. The image processing unit 34 calculates, for each frame of the input image, the pixel position in the converted image corresponding to the pixel included in the frame by inference, and the calculated pixel position is the projected two-dimensional position image. It is sent to the generation unit 35.

The projected two-dimensional value image generation section 35 generates a converted image for each frame based on the pixel position information from the image processing section 34. This converted image becomes a sequential dominant eye image 57 as shown in the figure.

Then, the input image (dominant-eye image 56) and the converted image (anti-dominant-eye image 57) are sent to the three-dimensional display system 3 as needed via the frame memories 32 and 33, respectively, and the parallax is displayed. It is projected as two sets of moving images and is displayed on the three-dimensional display system 3 as a three-dimensional moving image.

Also in the present embodiment, as in the second embodiment, two converted images (moving images) respectively corresponding to two viewpoints different from the input image which is a moving image are generated, and the two generated conversions are generated. It is also possible to display a three-dimensional moving image using an image. For example, when the input image is the central image, the left-eye image and the right-eye image may be generated as converted images. In this case, two frame memories for converted images are provided.

[0087]

As described above, the present invention has a plurality of viewpoints (2
Not only multiple (2) input images from viewpoints, but also 1
For each pixel of the input image that is projected from two viewpoints and has two-dimensional position information, three-dimensional position information is restored based on the pixel value of that pixel and the pixel values of surrounding pixels, and is different from the input image. Since the two-dimensional image is generated from the viewpoint, stereoscopic display can be performed using only one two-dimensional image. Further, it is possible to perform stereoscopic display using images of two viewpoints different from the input viewpoint image. Therefore, it is possible to easily create the three-dimensional pixel information and the video information from the existing two-dimensional image information and the video information, and to enhance the three-dimensional image software.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of a first embodiment of a three-dimensional image display device of the present invention.

FIG. 2 is a flowchart showing a processing procedure of an image input system according to the first embodiment.

FIG. 3 is a block diagram showing a configuration of an image processing system according to a first embodiment.

FIG. 4 is a block diagram showing a configuration of a three-dimensional display system of the first embodiment.

FIG. 5 is a flowchart showing a processing procedure of the image processing system of the first embodiment.

FIG. 6 is a flowchart showing a processing procedure of the image processing system of the first embodiment.

FIG. 7 is a block diagram showing a configuration of an image processing system of a three-dimensional image display device according to a second embodiment.

FIG. 8 is a block diagram showing a configuration of an image processing system of a three-dimensional image display device according to a third embodiment.

[Explanation of symbols]

1 image input system 2 image processing system 3 three-dimensional display system 10, 20, 30 input image 11, 21, 31 image input unit 12, 13, 22, 23L, 23R, 32, 33 frame memory 14, 24, 34 image processing Part 15, 25, 35 Projection two-dimensional position image generation unit 51, 56 Dominant eye image 52, 57 Dominant eye image 53 Central image 54 Left eye image 55 Right eye image

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI G06T 15/00 100 H04N 13/00 H04N 13/00 G01B 11/24 K (58) Fields investigated (Int.Cl. ⁷ , DB) Name) G06T 17/40 G01B 11/24 G06T 1/00 H04N 13/00 G06T 7/00 CSDB (Japan Patent Office)

Claims (11)

(57) [Claims] 1. An image input system for inputting an image having two-dimensional position information, an input image holding means for fetching and holding an image input from the image input system, and each pixel of the input image,
Image processing means for restoring three-dimensional position information based on the pixel value of the pixel and the pixel values of pixels around the pixel,
And an image processing system having a two-dimensional position image generating means for generating a two-dimensional image corresponding to a viewpoint different from the image input from the image input system based on the three-dimensional position information restored by the image processing means. A three-dimensional image display device comprising: a three-dimensional display system that receives video information output from the image processing system and displays a stereoscopic image based on two images with different viewpoints. 2. In the three-dimensional image display device, which one of an input image having two-dimensional position information projected from one viewpoint and an input image having two-dimensional position information projected from two viewpoints is displayed. If the input image type determining means for determining whether or not the image is input is provided in the image input system, and the input of the image input system is an input image having two-dimensional position information projected from two viewpoints, The image signal input thereto is directly sent to the three-dimensional display system without passing through the image processing system.
Dimensional image display device. 3. The three-dimensional image display device according to claim 1, wherein the image processing means of the image processing system restores the three-dimensional position information by inference using a knowledge base. 4. An image input system for inputting an image, an input image holding means for taking in and holding an input image input from the image input system, and for each pixel of the input image, a pixel value of the pixel and the pixel. On the two-dimensional surface when a point in the three-dimensional space corresponding to the pixel is projected on a two-dimensional surface from a viewpoint different from the viewpoint of the input image based on the pixel values of pixels around the Image processing means for calculating the pixel position corresponding to the viewpoint by inference using a knowledge base, and projection from the different viewpoint based on the pixel position calculated by the image processing means, and the position information is two-dimensional. An image processing system having a two-dimensional position image generating means for generating an image, and receiving a video information output from the image processing system, displaying a stereoscopic image based on two images having different viewpoints. 3-dimensional image display apparatus characterized by comprising a three-dimensional display system to be. 5. In the three-dimensional image display device, one of an input image having two-dimensional position information projected from one viewpoint and an input image having two-dimensional position information projected from two viewpoints. In the case where the input image type determining means for determining whether or not the image is input is provided in the image input system, the inputs of the image input system are projected from two viewpoints, and the input information having two-dimensional position information is two. The three-dimensional image display device according to claim 4, wherein the input signal is directly sent to the three-dimensional display system without passing through the image processing system. 6. The image processing means of the image processing system, if two or more pixels of the input image correspond to the same pixel position on the two-dimensional surface, it is inferred to be on the front side. The three-dimensional image display device according to claim 4 or 5, wherein the pixel to be displayed and the pixel position are associated with each other. 7. A two-dimensional position image generating means of the image processing system, if there is a pixel position on the two-dimensional surface that does not correspond to a pixel of the input image, the two-dimensional position image generating means sets the pixel position to the pixel position. 7. An image is generated by calculating a pixel value corresponding to the pixel position by extrapolation or interpolation processing from pixel values corresponding to adjacent pixel positions. The three-dimensional image display device according to item 1. 8. The two images with different viewpoints displayed on the three-dimensional display system are the images generated by the two-dimensional position image generating means and the two images selected from the input images. The three-dimensional image display device according to any one of claims 4 to 7. 9. The input image is a dominant eye image, the image generated by the two-dimensional position image generating means is an antidominant image corresponding to the dominant eye image, and the three-dimensional display system is the dominant image. The three-dimensional image display device according to any one of claims 1, 4 to 7, which displays a stereoscopic image based on an eye image and the anti-dominant eye image. 10. The input image is a central image, the two-dimensional position image generating means generates a right-eye image and a left-eye image corresponding to the central image, and the three-dimensional display system is the right-eye image. The three-dimensional image display device according to claim 1, wherein a stereoscopic image based on the left-eye image is displayed. 11. The pixel according to claim 1, wherein each pixel of the input image holds at least one-dimensional information of color space information among time information and color space information as a pixel value. The three-dimensional image display device according to item 1.