JPH1032840A - Multi-viewpoint image transmission and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To decide a distance of observation with high accuracy to render the same angle of visibility as that of a photographing mode by transmitting a photographing surface size of a camera, te distance between the photographing surface and a center of a lens and the information on the lens focal length respectively after adding them together. SOLUTION: A parallax estimation means 2 calculates a parallax for every pixel from a right and a left image which are decoded by an image decoder means 1. That is, both upper and lower limit numbers for pixels of parallax are decided, based on a the position relationship among a depth value of the nearest point, the depth value of the farthest point and the width value of the congestion point in an image. An average parallax arithmetic means 3 calculates an average of parallax calculated by the means 2. The image shift means 4a and 4b shift the images, so as to display the depth point having the average parallax obtained by the means 3 in the same depth as a display screen. Thus, it is possible to display without producing eye fatigue caused on the display side, by adding together the information on both nearest and farthest points in an image when a multi-viewpoint image is transmitted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method for transmitting a multi-viewpoint image and a method for displaying a multi-viewpoint image. The present invention also relates to a method for generating an intermediate viewpoint image of a multi-view image, a method for estimating disparity, and an apparatus thereof.

[0002]

2. Description of the Related Art Conventionally, various types of stereoscopic video systems have been proposed. A multi-view stereoscopic video using a multi-viewpoint image has been proposed as a system that allows a plurality of persons to observe a stereoscopic video without wearing special glasses. The scheme is promising. In the multi-view stereoscopic video system, the more the number of cameras and the number of display devices used, the more the observer can feel natural motion parallax and the easier the observation by a large number of people. However, the number of cameras that can be used practically is limited due to restrictions such as the size of the imaging system and the setting of the optical axis of the camera. In the transmission and storage processes, it is desired to reduce the amount of information that increases in proportion to the number of cameras.

Therefore, if a multi-view stereoscopic image can be displayed by generating an intermediate viewpoint image from a binocular stereo image on the display side, the load on the imaging system is reduced, and the amount of information during transmission and storage is reduced. You can do it. In order to generate an intermediate viewpoint image to be viewed at an arbitrary viewpoint between different viewpoints from a plurality of images having different viewpoints, it is necessary to estimate the depth by obtaining a correspondence between pixels between the images.

[0004] MPEG-1 and MPEG-2 have been proposed as image compression systems for digitally transmitting moving images. Further, attempts have been made to extend multi-view images by extending MPEG-2 (ISO / IEC13818-2 / PDA).
M3). FIG. 28 is a schematic diagram of the MPEG-2 syntax. MPEG-2 transmission includes Sequence, GOP
This is performed by encoding and decoding image data having a hierarchical structure of (Group Of Picture) and Picture. ISO / IE
According to C13818-2 / PDAM3, the transmission of a multi-view image by extension of MPEG-2 seems to be realized by extending the GOP layer (although it is not clear because it is not specified).

[0005] Fig. 29 shows the relationship in the spatio-temporal direction of a transmitted multi-viewpoint image. An attempt is made to increase coding efficiency by using disparity compensation in addition to motion compensation used in conventional MPEG-2. When transmitting a multi-viewpoint image, it is necessary to add information on each camera (camera parameters such as the position of the camera and the direction of the optical axis of the camera) and transmit the information. ISO / IEC13818-2 / PDAM3 includes:
It is described that the camera parameters are transmitted by being included in Pic.Extension (extension of the Picture layer) in FIG. 28, but the description of specific camera parameters is not described.

For description of camera parameters, see CG
In OpenGL, which is a language, the position of the camera, the direction of the optical axis of the camera, and the distance between the position of the camera and the image plane are defined as camera parameters (see “OpenGL Programming Guide”).
ide, The Official Guide to Learning OpenGL, Release
1, Addison-Wesley Publishing Company, 1993)).

FIG. 30 is an explanatory diagram showing the definition of camera parameters by OpenGL. In FIG. 30, A
Is the center of the lens, B is the center of the image plane (that is, the imaging plane),
C indicates the intersection of the perpendicular drawn from B to the upper end of the image and the upper end of the image. The coordinate values of A, B, and C are (optical_cent
er_X, optical_center_Y, optical_center_Z), (image_p
lane_center_X, image_plane_center_Y, image_plane_cen
ter_Z), (image_plane_vertical_X, image_plane_verti
cal_Y, image_plane_vertical_Z).

[0008] It is easy to add a camera parameter information defined by the above OpenGL to Pic.Extension to transmit a multi-viewpoint image.

[0009]

However, in the conventional method as described above, since there is no information on the closest point and the farthest point in the image (that is, the closest point and the farthest point of the subject),
When performing display (for example, parallax control) in which the eyes are hardly tired at the time of display, there is a problem that it is difficult to know which depth range to perform.

When displaying a multi-viewpoint image, it is necessary to appropriately determine the observation distance based on conditions such as the viewing angle at the time of photographing, the size of the imaging surface, and the distance between the lens center and the imaging surface. (In order to prevent a display with a sense of incongruity due to too much parallax or a display with a poor stereoscopic effect). However, in OpenGL, the size of the imaging surface (the physical size of the CCD) is not defined, and the distance between the center of the lens and the image forming surface is always equal to the focal length of the lens. For this reason, the size of the viewing angle at the time of image capture is not known on the display side, and an appropriate viewing angle at the time of display, that is, an appropriate observation distance cannot be determined. I was

In view of the above, the present invention adds information about the nearest point and the farthest point in an image (that is, the nearest point and the farthest point of the subject) and information about the size of the imaging surface to the image information. It is an object of the present invention to provide a method of transmitting and displaying a multi-viewpoint image in which it is easy to perform stereoscopic display with less discomfort on the display side by performing transmission.

A fundamental problem in associating images for generating an intermediate viewpoint image is that occlusion occurs in an object contour whose depth varies discontinuously. Is difficult. However, the estimated value of parallax near this object contour is
Since the contour position of the object in the generated intermediate viewpoint image is determined, it is very important when synthesizing the intermediate viewpoint image. In other words, if a parallax estimation error occurs near the object contour during parallax estimation, pixels in the foreground area will stick to the background side, and pixels in the background area will stick to the foreground, and the contour of the object will be disturbed. False contours will occur in the background area near the line.

In view of the foregoing, it is an object of the present invention to provide a parallax estimation method and apparatus for accurately estimating a rapid change (discontinuous change) in parallax near an object contour.

[0014]

According to the present invention, a header of a multi-viewpoint image includes information on the nearest point and the farthest point (that is, the nearest point and the farthest point of the subject) in the image and the size of the imaging surface. With the image transmission method including the information, the transmission of the multi-viewpoint image to which the information on the nearest point and the farthest point in the image (that is, the nearest point and the farthest point of the subject) and the information on the size of the imaging surface are added. By doing so, it is possible to perform display in consideration of the image size and observation distance on the display side.

Further, the present invention calculates an initial parallax and a reliability evaluation value of the initial parallax, and extracts an object contour line in which the parallax changes discontinuously using the reliability evaluation value and an edge detection result. Then, extract the low-reliability region of the initial parallax from the reliability evaluation value, and smoothly connect the extracted parallax in the low-reliability region of the initial parallax to the surrounding parallax, and A disparity estimation method and apparatus for performing disparity estimation that changes discontinuously in an object contour by determining the disparity to change.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings showing an embodiment. (First Embodiment) FIG. 4 is a diagram showing parameters defined by an image transmission method according to a first embodiment of the present invention. In FIG. 4, A1 and A2 indicate the positions of the centers of the lenses of the camera, and B1 and B2 indicate the centers of the imaging planes. ing).

In OpenGL, the distance between A1B1 and A2B2 in FIG. 4 is defined as the focal length of the camera lens. However, in the present invention, the distance between the center of the camera lens and the imaging surface is independent of the focal length of the lens. Defined in With this definition, the distance between the lens center and the imaging surface at the time of focusing can be calculated according to the distance to the subject, and the accurate viewing angle can be calculated. The viewing angle can be calculated from the size of the imaging surface and the distance between the lens center and the imaging surface.

Hereinafter, a description will be given of how the distance between the lens center and the imaging surface at the time of focusing changes with the distance between the subject and the lens center with reference to FIG. FIG. 2 shows the position of the subject,
FIG. 4 is a diagram illustrating a relationship between a position of an imaging surface and a focal length during focusing. In FIG. 2, A is the position of the subject, B is the point where light from A forms an image, O is the center of the lens, F is the point where parallel light forms an image with the lens, and a is the distance between the subject and the center O of the lens. , B indicate the distance between the point B where the light from A forms an image and the center O of the lens, and f indicates the focal length of the lens. It is known that the relationship of (Equation 1) holds between a, b, and f.

[0019]

(Equation 1)

From equation (1), if the object is far from the lens (a >> f) so that the focal length can be ignored, 1 / a
→ 0, which can be approximated as b = f. However, when the subject is relatively close to the lens, the term 1 / a cannot be ignored and b ≠ f. Therefore, in order to correctly calculate the viewing angle even when the subject is relatively close to the lens, it is necessary to define the distance between the lens center and the imaging plane independently of the focal length. Then, assuming that the width of the imaging surface is win and the height is hin, the viewing angle at the time of imaging is represented by (Equation 2).

[0021]

(Equation 2)

Therefore, if the width of the image at the time of display is wout and the height is hout, the observation distance for reproducing the viewing angle at the time of imaging is:

[0023]

(Equation 3)

## EQU1 ##

Next, the improvement of the visibility on the display side based on the nearest point and the farthest point in the image will be described. FIG.
FIG. 4 is a diagram for explaining a positional relationship between a convergence distance, a nearest point, and a farthest point when convergence projection is performed using two projectors. In FIG. 3, C is the congestion point, A is the nearest point, B
Indicates the farthest point.

In a projection with convergence, the parallax becomes 0 when the observer views the convergence point C. (In FIG. 3, since both eyes look at the center of the image, the left and right eyes see in the image that the left and right eyes see. Differences in the relative position of are eliminated). When the closest point A is viewed, a so-called cross-eyed state occurs, and a parallax of Da occurs in the direction of the cross-hair on the image. In FIG. 3, the observer sees a point shifted by Da / 2 toward the inside of both eyes as compared with the case of viewing the convergence point C. On the other hand, when the farthest point B is viewed, a so-called distant eye state occurs, and a parallax of Db occurs in the direction of the distant eye on the image.

FIG. 1 is a diagram showing the positional relationship between the nearest point, the farthest point, and the point where the convergence of the observer and the adjustment coincide in the case of parallel projection. In FIG. 1, A indicates the closest point of the displayed image, B indicates the farthest point, and C indicates the point at which the convergence and adjustment of the observer coincide. In the case of the parallel projection shown in FIG. 1, when an image having a parallax of Dc is displayed, it is displayed at the same point on the screen, and the observer's convergence and accommodation coincide.

The above-described parallax in the images shown in FIGS. 3 and 1 is perceived by the observer as a three-dimensional effect of being closer to or farther from the screen surface (the surface including C). It disappears (in a state of double appearance) and gives the observer a feeling of strangeness and discomfort.

Improvement of the viewability of the observer is based on the closest point, the farthest point, and the point of convergence at the time of image pickup. (Horizontal direction), it is possible to change the positional relationship between the convergence point, the farthest distance, and the closest distance. As for the way of shifting the image, for example, the entire image can be uniformly viewed by shifting the average value of the parallax between the images to cancel each other.

FIG. 5 is a block diagram of such a process. FIG. 5 shows an example of binocular (two viewpoints) data for simplicity. In FIG. 5, 1 is an image decoding unit, 2 is a parallax estimating unit, 3 is an average parallax calculating unit,
Reference numerals a and 4b denote image shift means, and reference numerals 5a and 5b denote image display means. The operation of each means will be described below.

The image decoding means 1 receives multi-view image data encoded on the transmission side and decodes the data. The left and right images decoded by the image decoding unit 1 are sent to the parallax estimation unit 2. The disparity estimation unit 2 calculates a disparity (disparity map) at each pixel from the left and right images decoded by the image decoding unit 1. For example, a case where the parallax is calculated by block matching based on the left image will be described below with reference to FIG. First, a window area is set in the left image. Next, the residual sum of squares (SSD) shown in (Equation 4) is calculated.

[0032]

(Equation 4)

The calculation of (Equation 4) is performed at one pixel intervals for d in the range from dmin to dmax. And dmin to dm
The value of d that minimizes the SSD in the range of ax is set as the parallax in the set window area. The parallax at each pixel of the image is obtained by sequentially shifting the window regions and performing the above calculation.

The ranges dmin and dmax for calculating the SSD can be calculated from information on the nearest point and the farthest point. With reference to FIGS. 7 and 8, a method of obtaining dmin and dmax in the case of parallel shooting and the case of convergence shooting will be described below.

FIG. 7 is a diagram showing the case of parallel photographing.
In the coordinate system shown in FIG. 7, the coordinate values of the left and right lens centers are (−D / 2, 0) and (D / 2, 0), the distance between the imaging surface and the lens center is b, and the object in the three-dimensional space The horizontal coordinate value of the position is X0, the coordinate value in the depth direction is Z0, and the horizontal position where light from the object at the position (X0, Z0) is imaged on the left and right imaging planes is xl0 and xr0, respectively (xl0, xr0 is Horizontal coordinates in a plane coordinate system with the origin at the intersection of the camera's optical axis and the imaging plane)
And from the graphical relationship,

[0036]

(Equation 5)

Is as follows. Therefore, each of the parallaxes based on the left and right images is represented by the equation shown in (Equation 6).

[0038]

(Equation 6)

Here, the depth value of the nearest point in the image is Zmi
Assuming that n and the depth value of the farthest point are Zmax, the upper limit dmax and the lower limit dmin of the SSD calculation range are represented by (Equation 7).

[0040]

(Equation 7)

FIG. 8 is a diagram showing a case of convergence photographing. In the coordinate system shown in FIG. 8, the coordinates of the convergence point (the intersection of the optical axes of the left and right cameras) are (0, C), the coordinates of the center of the left and right lenses are (−D / 2, 0), and (D / 2,0), the distance between the imaging surface and the lens center is b, the horizontal coordinate value of the object position in the three-dimensional space is X0, the coordinate value in the depth direction is Z0, and the position (X0, Z0) is on the left and right imaging surfaces. The horizontal position where light from the object is imaged is assumed to be x10 and xr0, respectively (xl0 and xr0 are the horizontal coordinates of a plane coordinate system whose origin is the intersection of the optical axis of the camera and the imaging surface) and

[0042]

(Equation 8)

Is as follows. Therefore, the parallax based on the left and right images is represented by the equation shown in (Equation 9).

[0044]

(Equation 9)

Since X0 remains in the equation (Equation 9), in the convergence imaging, even if the depth is the same, the parallax differs depending on the horizontal position (that is, the reproduced stereoscopic image is distorted). I understand. Now, for the sake of simplicity, considering the parallax at a point on X0 = 0 (that is, the Z axis),
= 0 to obtain (Equation 10).

[0046]

(Equation 10)

From equation (10), the depth value Zmin of the closest point in the image, the depth value Zmax of the farthest point, and the depth value C of the convergence point are obtained.
, The number of horizontal pixels nx, and the width w of the imaging surface (CCD)
The upper limit pixel number dmax and the lower limit pixel number dmin of parallax can be determined from in.

When the parallax at a point other than on the Z axis is considered, the upper limit dmax and the lower limit dmin of the parallax can be determined by calculating the maximum value and the minimum value of (Equation 9).

As described above, given the depth value of the nearest point in the image, the depth value of the farthest point, the position of the camera, and the direction of the optical axis of the camera, the range of values for which parallax should be obtained can be calculated. , The range in which the SSD is calculated during the parallax calculation can be determined. The average parallax calculating means 3 calculates an average of the parallax maps calculated by the parallax estimating means 2. The average of the disparity map is obtained by calculating (Equation 11).

[0050]

[Equation 11]

The image shifting means 4a and 4b display the depth point having the average parallax obtained by the average parallax calculating means 3 at the same depth as the display surface (that is, with no parallax on the display surface). Shift the image as follows.

In FIG. 1 showing display by parallel projection,
A indicates the depth of the nearest point in the displayed image, B indicates the depth of the farthest point, and C indicates the depth of the average parallax. From FIG. 1, it can be seen that in the parallel projection, when there is a parallax of Dc shown by (Equation 12) between the left and right images, the parallax disappears on the screen, and a natural display in which the convergence and the adjustment are matched is obtained.

[0053]

(Equation 12)

The image shift means 4a shifts the left image by the shift amount (the rightward shift is positive) shown in (Expression 13).

[0055]

(Equation 13)

Then, the image shift means 4b shifts the right image by the same amount in the reverse direction. As a result of the shift by the image shifting means 4a and 4b, the point having the average parallax is displayed at the same depth as the screen.

In FIG. 3 showing display by convergence projection, A indicates the depth of the nearest point in the displayed image, B indicates the depth of the farthest point, and C indicates the depth of the average parallax. In the convergence projection, when the parallax is 0 at the center of the image, the image is displayed at the same depth as the screen. Therefore, in the case of convergence projection, the image shift means 4a and 4b reduce the average parallax by-
The left and right images are shifted by half the value.

As described above, according to the present embodiment, when transmitting a multi-viewpoint image, by adding the information of the nearest point and the farthest point in the image, the display side can be displayed without eyestrain. Parallax control).

Also, by adding and transmitting information on the size of the imaging plane (CCD) of the camera, the distance between the imaging plane and the center of the lens, and the focal length of the lens, a display corresponding to the viewing angle at the time of photographing can be obtained. When trying to do so, the viewing angle at the time of shooting can be accurately calculated on the display side even for a video shot close to the subject.

When transmitting without adding information on the nearest point and the farthest point in the multi-viewpoint image, information on the nearest point and the furthest point is replaced with information on the nearest point and the furthest point. Estimate the parallax at the nearest point and the farthest point in the image by transmitting the image with a dedicated code indicating that it is not added and transmitting it, and calculating the parallax within the preset range on the display side. Can be included in the present invention.

Further, by setting the information on the nearest point and the farthest point in the multi-viewpoint image to a specific depth value on the transmission side, the disparity in the specific depth range thus set falls within the fusion range. Parallax control can be performed, and is included in the present invention.

In the present invention, an example in which the calculation of the disparity is performed on the display side has been described. However, the disparity included in the encoded image may be used and is included in the present invention. FIG.
Such an example will be described using 0.

In FIG. 10, the operation of the components other than the image decoding means 6 is the same as that of the parallax control method shown in FIG. 5, and therefore the description thereof is omitted, and the operation of the image decoding means 6 will be described below. The image decoding means 6 decodes the encoded image data and outputs parallax based on the left and right images and the left image. When transmitting a two-lens image using a multi-view image transmission method based on MPEG-2, the compression rate is increased by parallax compensation based on the left image. By extracting the parallax from the encoded image data, it is not necessary to calculate the parallax on the display side, and the amount of calculation on the display side can be reduced.

The calculation of the average of the parallax by the average parallax calculating means 3 may use the weighted average value by (Equation 14) with emphasis on the center of the screen. If you do this,
At the center of the image, parallax control that is easier to fuse can be performed and is included in the present invention.

[0065]

[Equation 14]

FIGS. 9 (a), 9 (b) and 9 (c) show examples of the distribution of weights used for calculating the weighted average by (Equation 14). Although shown one-dimensionally for simplicity, it is actually a two-dimensional distribution where the value is larger at the center of the image than at the periphery. The weight values are all values equal to or greater than 0 (non-negative values). (Second Embodiment) FIG. 11 is a block diagram of a parallax control system according to a second embodiment of the present invention. FIG.
Since the components other than the frequency calculation means 7 and the shift amount calculation means 8 perform the same operations as those in the first embodiment, the same reference numerals as those used in the first embodiment will be used. And the description is omitted. The operation of the frequency calculating means 7 and the shift amount calculating means 8 will be described below.

The frequency calculation means 7 calculates the frequency of the parallax based on the left image decoded by the image decoding means 6. The parallax frequency is the number of pixels calculated for each value of parallax in a certain area of the image (for example, the entire image or a specific area determined based on the criteria). The shift amount calculating means 8 calculates the sum of the frequency of the parallax in the fusion range from the frequency of the parallax (between images) calculated by the frequency calculation means 7 and the fusion range of the human eye according to the viewing angle of the image. Is calculated, and the image shift means 4a, 4b
Output to

FIG. 12 shows an example of the configuration of the shift operation means 8. In FIG. 12, 9 is an MPU, and 10 is a fusion range table. The MPU 9 calculates the horizontal viewing angle shown in (Equation 15) from the width of the image display surface and the observation distance, and reads the fusion range at the viewing angle from the fusion range table 10.

[0069]

(Equation 15)

FIG. 13 shows an example of the characteristics of the fusion range table. In FIG. 13, the horizontal axis is the horizontal viewing angle of the image display surface, and the vertical axis is the parallax fusion range (angle converted by (Equation 16)).

[0071]

(Equation 16)

The sign on the vertical axis in FIG. 13 indicates the parallax perceived in front of the display surface on the negative side, and the parallax perceived behind the display surface on the positive side. FIG.
It is a figure which shows the graphic meaning of 6). FIG. 14 shows that the angle-converted parallax θ is obtained by converting the parallax Δ on the image display surface into a viewing angle.

On the other hand, in the parallel projection and the convergence projection shown in FIGS. 1 and 3, the positions of the images (for example, the positions of the pixels on the liquid crystal in the case of a liquid crystal projector) xl1, xr1 and the positions Xl, Xr on the display surface are determined. The positional relationship is (Equation 17) (Equation 19), and the parallax on the display surface is (Equation 18) (Equation 2).
0).

[0074]

[Equation 17]

[0075]

(Equation 18)

[0076]

[Equation 19]

[0077]

(Equation 20)

Then, the coordinate value (x
l0, yl0), (xr0, yr0) and the position (x
The relationship with (l1, yl0), (xr1, yr1) (for example, the position of the pixel on the liquid crystal in the case of a liquid crystal projector) is expressed by (Equation 21).

[0079]

(Equation 21)

Here, the width win of the imaging surface is obtained from the camera parameters, and the image width wout at the time of display is a value specific to the display system.

According to the conditions at the time of imaging (parallel imaging / convergence imaging), xl0 and xr0 are calculated using (Equation 5) or (Equation 8), and are converted into xl1 and xr1 by (Equation 21). Furthermore,
In accordance with the projection conditions (parallel projection / convergence projection), (Equation 1)
8) or (Equation 20), the parallax on the display screen can be calculated in consideration of both the imaging condition and the projection condition.

The MPU 9 converts the fusion range read from the fusion range table 10 into parallax (distance) on the display surface,
The parallax fusion range on the image display surface is determined. And
The MPU 9 calculates a shift amount with respect to the image data such that the sum of the frequencies of the parallaxes within the fusion range is maximized using the above-described parallax in the image data and the relationship between the parallaxes on the image display surface (parallax). The shift of the image by the control means moving the frequency distribution of the parallax in the horizontal direction in FIG. 15).

The image is shifted in the reverse direction by the output shift amount by the image shift means 4a and 4b.
By displaying by a and 5b, it is possible to perform a display in which the sum of the frequencies of the parallaxes within the fusion range is maximum (that is, the area of the pixels to be fused in the image is maximum).

As described above, according to the present embodiment, by performing parallax control according to the fusion range of human eyes, the parallax can be included in the fusion range in a larger part of the image when displayed. Can be

In the present embodiment, the parallax control in which the sum of the parallax frequencies within the fusion range is maximized has been described. However, even if the parallax control is performed so that the average value of the parallax is at the center of the fusion range, it is almost impossible. Equivalent effects can be obtained and are included in the present invention.

Further, the nearest point and the farthest point on the transmission side are set to values different from the nearest point and the farthest point in the actual image, and the nearest point and the farthest point of the set values are set on the display side. By controlling the parallax so that the average parallax of each corresponding parallax is in the center of the fusion range, an image at the depth intended by the image creator can be preferentially presented to the observer. include. (Third Embodiment) A third embodiment of the present invention is described as follows.
Input a pair of image pairs, calculate the initial disparity and the reliability of the initial disparity, detect the object outline from the reference image and the reliability of the initial disparity, and detect the initial disparity and the reliability of the initial disparity From the object contour, a parallax in an area where the initial parallax near the object contour is unreliable is determined. At this time, a parallax estimating method and a parallax estimating method are used in which the parallax changes in the object outline and is determined to be smoothly connected to the surrounding parallax.

In the present embodiment, with the above-described configuration, the initial parallax and the reliability of the initial parallax are calculated from a pair of image pairs of the reference image and the reference image, and the object is calculated from the reference image and the reliability of the initial parallax. Detecting the contour, from the initial parallax and the reliability of the initial parallax and the detected object contour, the parallax in a region of low reliability of the initial parallax near the object contour changes in the object contour, and Is determined so as to be smoothly connected to the surrounding parallax.

FIG. 16 is a block diagram of a parallax estimating apparatus according to the third embodiment of the present invention.

In FIG. 16, reference numeral 201 denotes an initial disparity estimating unit for calculating an initial disparity by block matching;
Is a reliability evaluation unit at the time of initial parallax estimation, 203 is a contour detection unit, and 204 is a parallax estimation unit near the object contour.

The operation of the above configuration will be described below.

The initial disparity estimating unit 201 calculates the sum of squared differences (Sum of Squared diff) shown in (Equation 22).
calculations (hereinafter SSD). (Equation 2
The value of the SSD according to 2) is small when the distribution of pixel values in the window region set in the reference image and the window region set in the reference image is similar, and conversely, the pixel value in both window regions is small. The value becomes large where the value distribution is different. The initial parallax estimating unit 201 sets the amount of displacement d between images that minimizes the SSD value within a predetermined search range as the parallax at the point of interest (x, y), and estimates the parallax value near the object contour. The minimum value of the SSD within the search range is output to the reliability evaluation unit 202 at the time of initial parallax estimation.

[0092]

(Equation 22)

FIG. 17 is a diagram for explaining the initial parallax estimation (block matching) performed by the initial parallax estimating unit 201. In FIG. 17, the window region set around the point of interest (x, y) indicates the integration region W of (Equation 22). The initial parallax of the entire image can be obtained by sequentially shifting the window regions and calculating the above-mentioned SSD.

The reliability evaluation unit 202 at the time of initial parallax estimation
SSD obtained by parallax calculation by the initial parallax estimating unit 201
From the minimum value in the search range, the number of pixels in the window area (block), the variance of noise between images, and the average value of the square of the luminance gradient in the horizontal and vertical directions of the reference image in the window area. 2
The reliability evaluation value of the association shown in 3) is calculated.

[0095]

(Equation 23)

The smaller the value of (Expression 23), the higher the reliability of the parallax estimation, and the larger the value, the lower the reliability.

FIG. 18 is a block diagram showing an example of the configuration of the contour detection unit 203. In FIG. 18, reference numeral 205 denotes a YC separation circuit for separating a reference image into a luminance component and a color component;
06A, 206B, and 206C are edge detection circuits that detect edges from the separated luminance component Y and color components RY and BY, respectively, and 207 is an edge detection unit that outputs only the edge intensity of the edge detection result. Reference numeral 208 denotes a weight generation circuit that outputs a weight of 1 in an area where the initial disparity estimation value has low reliability, and outputs a weight of 0 in an area where the initial disparity estimation value has high reliability.

The operation of the above configuration will be described below.

The YC separation circuit 205 separates the reference image into a luminance component Y and color components RY and BY and outputs the same.

Edge detection circuits 206A, 206B, 20
6C detects an edge component from the Y, RY, and BY components, respectively. FIG. 19 is a block diagram illustrating an example of a configuration of the edge detection circuit 206. In FIG.
Reference numerals 209A, 209B, and 209C denote direction-dependent filter groups for detecting edge components in a low spatial frequency range, a medium spatial frequency range, and a high spatial frequency range, respectively. 210, 21
Reference numerals 1, 212, and 213 denote direction-specific filters forming respective direction-specific filter groups. FIG. 20 is an example of the spatial weight of the direction-specific filter. FIG. 20 (a),
(B) and (c) show edges that are continuous in the vertical direction,
(D), (e), and (f) detect edges in oblique directions.

(A) and (d) are high spatial frequency ranges,
(B) and (e) show examples of the distribution of weights for the middle spatial frequency range, and (c) and (f) show examples of the distribution of weights for the low spatial frequency range. Edge detection in the horizontal direction and the other diagonal direction may be performed by rotating the arrangement of the counting in FIG. 20 by 90 degrees. Also, the direction of the edge need not be limited to 45 degrees, but may be 30 degrees.

The spatial weights of the direction-specific filters need not be limited to those shown in FIG. 20, but needless to say, a differential type weight distribution for each direction is sufficient. The method of calculating the edge strength for each direction is expressed by (Equation 24).

[0103]

(Equation 24)

The integration unit 214 includes direction-specific filters 210 and 2
11, 212, and 213 are integrated. Integration unit 214
An example of the integration by the following equation is represented by (Equation 25).

[0105]

(Equation 25)

The integration by the integration unit 214 is (Equation 25)
It is needless to say that the present invention is not limited to the form of the sum of squares represented by

For the luminance component Y and the color components RY and BY, the edge intensities integrated by the integration units 214A, 214B and 214C in the high spatial frequency range, middle spatial frequency range and low spatial frequency range are multiplied. Is output. Then, the edge intensities for the Y, RY, and BY components are added and transferred to the ridge line detection unit 7.

Note that the separation of the reference image into the luminance component and the color component in the contour detection unit 203 need not be limited to Y, RY, and BY, but may be performed by separating the reference image into other components such as R, G, and B. Of course it is good. Also, the edge strengths for Y, RY, and BY need not be limited to those transferred to the ridge line detection unit 207 after addition, and may be transferred to the ridge line detection unit 207 after multiplication.

Returning to FIG. 18, the ridge line detecting section 207 outputs only the value of the edge strength added to the ridge line for Y, RY and BY. FIG. 21 is an example of the configuration of the ridge line detection unit 207. In FIG. 21, the horizontal ridge line detection circuit 215 outputs 1 when the edge strength at the target pixel is larger than both the edge strengths at the pixels above and below the target point, and outputs 0 otherwise.

Similarly, the vertical ridge line detection circuit 216 outputs 1 when the edge strength at the target pixel is larger than both the edge strengths at the left and right pixels of the target point, and outputs 0 when it is not. . The outputs of the horizontal ridge line detection circuit 215 and the vertical ridge line detection circuit 216 are subjected to an OR operation, further multiplied by an input signal, and output. That is, the ridge line detection unit 207
Outputs only the edge intensity of a pixel having an edge intensity higher than the edge intensity of a pixel adjacent in the horizontal or vertical direction (that is, a pixel having a ridge line), and outputs 0 for other pixels.

Returning to FIG. 18 again, the weight generation circuit 208
Outputs 1 when the reliability evaluation value of the estimated initial disparity value is equal to or greater than the threshold value, and outputs 0 when the reliability evaluation value is less than the threshold value. By multiplying the output of the weight generation circuit 208 with the output of the edge detection unit 207, it is possible to extract an edge where the reliability of the initial estimated disparity value is low, that is, an object contour line in which the disparity changes discontinuously. The output of the weight generation circuit 208 is
It is stored in the calculation area memory of the parallax estimating unit 204 near the object contour described later. Expression of the object contour is expressed by (Expression 26).

[0112]

(Equation 26)

The edge detection results 206A, 206B,
It is not necessary to add the outputs of the Cs 206 </ b> C and input them to the edge detection unit 7. The outputs may be multiplied and input to the edge detection unit 207. Also, the method of weight generation by the weight generation circuit 208 multiplied by the output of the ridge line detection unit 207 does not need to be limited to binary values of 0 and 1, and continuous values may be calculated according to the reliability at the time of initial parallax estimation. Of course, it may be output.

The disparity estimation unit 204 near the object contour recalculates the disparity in the area where the initial disparity estimation value near the object contour is low in reliability, from the contour strength and the initial disparity. The disparity estimation unit 204 near the object contour calculates a disparity distribution that minimizes the energy of the disparity distribution defined by (Equation 27).

[0115]

[Equation 27]

The weighting function w (x, y) is defined as (Equation 28) by the parameter of smoothness and the contour strength.

[0117]

[Equation 28]

The condition of the parallax distribution that minimizes (Equation 27) is (Equation 29).

[0119]

(Equation 29)

The differential equation (Equation 29) can be numerically solved by a known technique such as the finite element method (FEM).

FIG. 22 shows a parallax estimating unit 2 near the object contour.
FIG. 4 is a block diagram illustrating an example of a configuration of the information management unit 04. In FIG. 22, reference numeral 217 denotes a weighting circuit for disparity distribution energy for generating weights for disparity distribution energy, 218 denotes a calculation area memory, 219 denotes a disparity memory, 220 denotes a weight memory, and 2 denotes a weight memory.
21 is an FEM operation circuit.

Parity distribution energy weight generation circuit 217
Is obtained from the parameters λ of the contour strength and smoothness (Equation 28)
Is calculated and written to the weight memory 220. The FEM operation circuit 221 solves (Equation 29) by the finite element method and calculates the parallax distribution.

As described above, according to the present embodiment, an object contour is detected in an area where the reliability of a parallax estimation value obtained by block matching is low, and the parallax changes discontinuously at the detected object contour. Parallax estimation can be performed as follows.

Further, according to the present embodiment, it is possible to perform parallax estimation such that the parallax changes discontinuously at an object contour having an arbitrary shape.

It should be noted that the parallax estimation in the vicinity of the object contour may be such that the parallax changes at the object outline and is smoothly connected to the surrounding parallax. It is not necessary to limit to the method of calculating as. Such an example will be described below. (Fourth Embodiment) FIG. 23 is a block diagram showing a configuration of a parallax estimating apparatus according to a fourth embodiment of the present invention. In FIG. 23, reference numeral 201 denotes an initial disparity estimating unit that calculates initial disparity by block matching, 202 denotes a reliability evaluation unit at the time of initial disparity estimation, 222 denotes a contour detection unit, and 223.
Is a parallax estimating unit near the object contour.

In the above configuration, the operation of the configuration other than the contour detection unit 222 and the parallax estimating unit 223 near the object contour is the same as that of the third embodiment of the present invention, and the description is omitted. The operation of the parallax estimating unit 223 near the object contour will be described.

First, the contour detecting section 222 is the third embodiment of the present invention.
The contour detection is performed in the same manner as the contour detection unit in the embodiment, and the detection result is binarized (for example, 0 and 1) and output. The disparity estimation unit 223 near the object contour calculates the disparity in an area where the initial disparity estimation value near the object contour is unreliable.
It is calculated from the initial parallax and the object contour detected by the contour detection unit 222.

FIG. 24 shows a parallax estimating unit 2 near the object contour.
FIG. 14 is a diagram showing a state of parallax estimation by 23. In FIG. 24, reference numeral 291 denotes an area where the initial disparity estimation value is unreliable;
Reference numeral 92 denotes an object contour detected by the contour detection unit 222, reference numeral 293 denotes a highly reliable area of the estimated initial disparity value,
Reference numeral 4 denotes a point of interest at which parallax is to be calculated, and 295 denotes a window area set to include the point of interest.

The disparity at the point of interest 294 (x, y) is
A region 29 in which the initial disparity estimation value has low reliability within the setting window region
Using the disparity in the surrounding area (in this case, the area 293a in which the initial estimated disparity estimated value is highly reliable) in contact with 1, the point of interest 2
It is determined that the parallax at 94 is affected by the value of the parallax in the surrounding area according to the distance between the surrounding area and the point of interest 294. At this time, the parallax in the surrounding area changes beyond the object contour line 292 and does not affect the point of interest 294 beyond the object contour line 292, and is smoothly connected to the surrounding parallax. Parallax can be determined as if to do. The disparity estimation near the object contour by the disparity estimating unit 223 is represented by an equation as an example as follows.

[0130]

[Equation 30]

However, the parallax estimating unit 22 near the object contour
The parallax estimation by 3 need not be limited to (Equation 30), but it is only necessary that the parallax changes at the object contour line and is smoothly connected to the surrounding parallax.

As described above, according to the present embodiment, an object contour is detected in a region where the reliability of a parallax estimation value obtained by block matching is low, and the parallax changes discontinuously at the detected object contour. Parallax estimation can be performed as follows.

Further, according to the present embodiment, it is possible to perform parallax estimation so that the parallax changes discontinuously at an object contour of an arbitrary shape.

Further, according to the present embodiment, in an area where the initial estimated disparity value is unreliable, the disparity is calculated by referring to a relatively small number of disparities in the vicinity of the point of interest, thereby reducing the memory capacity. The calculation of the parallax can be performed with the calculation amount.

Further, by shifting and integrating the left and right images using the results of the parallax estimation described in the third and fourth embodiments, a predetermined distance between the viewpoints corresponding to the left and right images can be obtained. Can be generated at an intermediate viewpoint. Here, parallax estimation and intermediate viewpoint image generation may be performed in different places. Hereinafter, transmission and reception methods when performing parallax estimation and intermediate viewpoint image generation in different places will be described. (Fifth Embodiment) FIG. 25 is a diagram illustrating a fifth embodiment of the present invention, in which parallax estimation (or motion estimation) is performed on the transmission side.
1 is an example of a transmission block of a system that performs the following.

In FIG. 25, 170 is a parallax estimating means for estimating the parallax VL based on the left image, 171 is a parallax estimating means for estimating the parallax VR based on the right image, and 172.
a to d are encoders, 173a and b are decoders, 174 is prediction means for predicting the right image R from the left image L and the disparity VL based on the left image, and 175 is the disparity V based on the left image.
Prediction means for predicting parallax VR based on the right image from L,
176a and 176b are hole filling means for determining parallax in an area where parallax is not correctly estimated. The operation of the above configuration will be described below.

First, the left image L is encoded by the encoder 172a. The parallax estimating means 170 and 171 estimate parallaxes VL and VR based on the left and right images, respectively. For regions where parallax is not correctly estimated due to occlusion or the like, hole filling means 176a using the parallax estimating method described in the third or fourth embodiment is used.
The parallax is determined by 176b.

Next, the parallax after the filling with the left image as a reference is encoded by the encoder 172b. The parallax after filling in with the encoded left image as a reference is calculated by the decoder 173.
a, which is used for prediction of the right image R by the predictor 174 and prediction of disparity by the predictor 175 based on the right image after filling. The prediction of the parallax VR based on the right image by the predictor 175 is calculated as (Equation 31) using the parallax based on the left image.

[0139]

(Equation 31)

The right image R takes the residual from the predicted image obtained by the predictor 174, and is coded by the coder 172d. The disparity VR after padding with respect to the right image is obtained by taking the residual from the predicted disparity by the predictor 175, and
c.

FIG. 26 shows an example of a receiving block of a system for performing parallax estimation on the receiving side. In FIG. 26, 18
1a to 1d are decoders, 174 is a predictor for the right image R, 17
5 is a parallax predictor based on the right image. The encoded left image L, left image-based parallax VL, right image-based parallax VR prediction error, and right image R prediction error are decoded by decoders 181a to 181d, respectively. Right image R
Is restored by adding the prediction result of the predictor 174 and the prediction error of the decoded right image. Parallax V based on right image
R is restored by adding the prediction result by the predictor 175 and the decoded prediction error.

The left image L, the right image R, and the parallax V based on the left image
L, when the parallax VR based on the right image is restored, an image at the intermediate viewpoint of the left and right images can be generated by the intermediate viewpoint image generation method disclosed in Japanese Patent Application No. 7-109821, for example. It can be displayed as a multi-viewpoint image together with the right image.

As described above, with the above configuration,
By performing the parallax estimation and the filling process on the transmitting side, the amount of calculation on the receiving side can be reduced, and the device scale on the receiving side can be reduced.

Further, when transmitting a multi-viewpoint image, by generating an intermediate viewpoint image on the transmitting side, image transmission with a reduced transmission amount can be performed. Such an example will be described below. (Sixth Embodiment) FIG. 27 is a configuration diagram on the transmitting side of a multi-viewpoint image compression transmission system according to a sixth embodiment of the present invention. 27, reference numerals 101a to 101d denote cameras for capturing images at respective viewpoint positions, and 102 denotes a camera 1.
103a is a decoded image decompression unit that decodes and decompresses the image data compressed and encoded by the image compression and encoding unit 102;
An intermediate viewpoint image generation unit 104a predicts and generates images at the viewpoints of the camera 2 and the camera 3 from the image of the camera 1 and the image of the camera 4 decoded and expanded by the decoded image expansion unit 103a. This is a residual compression encoding unit that compresses and encodes the residual between the image 2 and the image generated by the intermediate viewpoint image generating unit 104a for the image of the camera 3. The operation of the above configuration will be described below.

The image compression / encoding unit 102 converts a plurality of images in the multi-view image (images at both ends of the four-view image in the present embodiment) into an existing technique using block correlation between the images. Is compressed and encoded. FIG. 31 shows an example of the configuration of the image compression encoding unit 102. In FIG. 31,
Reference numerals 107a and 107b denote DCT means for performing DCT calculation for each 8 × 8 pixel or 16 × 16 pixels to calculate DCT coefficients; 108a and 108b quantizing means for quantizing DCT coefficients; 109a, inverse quantizing means; Inverse DCT
Inverse DCT means for performing calculation, 111 is disparity detection means,
112a is the parallax compensating means, 113a is the quantized DC
This is an encoding unit that encodes the T coefficient and the disparity. The operation of the above configuration will be described below.

The DCT means 107a processes the image of the camera 1 for each block, and calculates a DCT coefficient for each block. The quantization means 108a quantizes the DCT coefficient. The inverse quantization means 109a inversely quantizes the quantized DCT coefficient. The inverse DCT means 110a
The inversely quantized DCT coefficients are inversely transformed to restore the image of the camera 1 obtained on the receiving side. Parallax detecting means 111
Performs block matching between the restored image of the camera 1 and the image of the camera 4, and calculates the parallax based on the image of the camera 1 for each block. The parallax compensation means 112a
The image of the camera 4 is predicted using the restored image of the camera 1 and the parallax of each block (that is, a process corresponding to motion compensation of a moving image is performed). DCT means 107b
Calculates the DCT coefficient by processing the residual between the image of the camera 4 and the predicted image for each block. The quantization means 108b quantizes the residual DCT coefficient. Encoding means 113
a encodes the quantized DCT coefficient of the image of the camera 1, the parallax of each block, and the quantized DCT coefficient of the residual of the parallax compensation.

The decoded image decompression unit 103a decodes and decompresses the image data compressed and encoded by the image compression encoding unit 102. FIG. 32 shows the decoded image decompression unit 1.
It is a figure showing an example of composition of 03a. In FIG. 32,
114a is a decoding unit, 109b and 109c are inverse quantization units, 110b and 110c are inverse DCT units, and 112b is a disparity compensation unit. The operation of the above configuration will be described below.

The decoding means 114a decodes the compression-encoded data, and converts the quantized DC
The quantized DCT coefficients of the T coefficient, the disparity of each block, and the residual of the disparity compensation are expanded. The quantized DCT coefficients of the image of the camera 1 are inversely quantized by the inverse quantization means 109b and decompressed as an image by the inverse DCT means 110b. The motion compensation unit 112b generates a predicted image of the camera 4 from the decompressed image of the camera 1 and the decoded parallax. Then, the inverse quantization means 109c, the inverse D
The image of the camera 4 is expanded by adding the residual expanded by the CT unit 110c to the predicted image.

The intermediate viewpoint image generation unit 104a calculates the parallax of each pixel from the images of the camera 1 and the camera 4 by the method shown in any of the third and fourth embodiments of the present invention, and The image of the camera 3 is predicted and generated.

The residual compression encoding section 105 compresses and encodes the residual of the images of the cameras 2 and 3 and the predicted image. Since the intermediate viewpoint image generation unit 104a calculates the disparity for each pixel, the intermediate viewpoint image generation unit 104a can accurately estimate the disparity as compared with the disparity calculation for each block by block matching. As a result, the prediction error (that is, residual error) of the intermediate viewpoint image can be reduced, the compression efficiency can be increased, more effective bit allocation can be performed, and compression can be performed while maintaining image quality. FIG. 33 shows an example of the configuration of the residual compression encoding unit. In FIG. 33, 107c, 10
7d is DCT means, 108c and 108d are quantization means,
113b is an encoding means. The residuals of the images of camera 2 and camera 3 are converted into DCT coefficients by DCT means 107c and 107d, respectively, and quantized by 108c and 10c.
8d, and is encoded by the encoding means 113b.

FIG. 34 is a block diagram of the receiving side of the multi-viewpoint image compression transmission system according to the sixth embodiment of the present invention. In FIG. 34, reference numeral 103b denotes a decoded image decompression unit for decoding and decompressing the image data of the cameras 1 and 4 compressed and encoded by the image compression / encoding unit 102 on the transmission side, and reference numeral 104b denotes a decoded image decompression unit 103b. An intermediate viewpoint image generation unit 106 for predicting and generating images at the viewpoints of the cameras 2 and 3 from the expanded images of the cameras 1 and 4, and a prediction error (remaining error) of the prediction images at the viewpoints of the cameras 2 and 3. difference)
Is a decoding residual expansion unit that decodes and expands. The operations of the decoded image decompression unit 103b and the intermediate viewpoint image generation unit 104b are the same as the operations of the decoded image decompression unit 103a and the intermediate viewpoint image generation unit 104a on the transmitting side, and therefore the description is omitted. The operation of the residual extension unit will be described.

The decoding residual decompression unit 106 is the camera 2 that has been compression-encoded by the residual compression encoding unit 105 on the transmission side.
And the prediction error (residual error) of the prediction image from the viewpoint of the camera 3 is decoded and decompressed. FIG. 35 shows an example of the configuration of the decoding residual decompression unit 106. In FIG. 35, reference numeral 114b denotes a decoding unit, 109d and 109e denote inverse quantization units, and 110d,
110e is an inverse DCT means. The compression-encoded residual data of the images of the camera 2 and the camera 3 is supplied to a decoding unit 11.
4b, respectively, and the inverse quantization means 10
9d and 109e, inversely quantized, and inverse DCT means 11
0d and 110e. The images of the viewpoints of the cameras 2 and 3 are restored by superimposing the residuals of the decoded and expanded images of the cameras 2 and 3 on the images generated by the intermediate viewpoint image generating unit 104b.

As described above, according to the present embodiment, the transmitting side generates an intermediate viewpoint image from two non-adjacent images in a multi-viewpoint image, and generates the generated intermediate viewpoint image and the intermediate viewpoint image. Of the two images and the residual of the intermediate viewpoint image are compressed and transmitted. The receiving side decodes and expands the two transmitted images and the residual of the intermediate viewpoint image, generates an intermediate viewpoint image from the two images, and superimposes the residual of the decoded and expanded intermediate viewpoint image. To restore the image corresponding to the actual image at the intermediate viewpoint. By doing so, it is possible to efficiently transmit the multi-viewpoint image while maintaining the image quality.

The generation of the intermediate viewpoint image need not be limited to a configuration in which the image at the intermediate viewpoint is generated from the images at the two viewpoints (viewpoints of camera 1 and camera 4) at both ends of the multi-viewpoint image. An image from the viewpoints of the cameras 1 and 3 may be generated from the images of the cameras 2 and 4, and an image from the viewpoints of the cameras 2 and 4 may be generated from the images of the cameras 1 and 3. Further, images from the viewpoints of the cameras 1 and 4 may be generated from the images of the cameras 2 and 3, which are included in the present invention.

It is apparent that the number of viewpoints of the multi-viewpoint image need not be limited to four, and an intermediate viewpoint image between the viewpoints may be generated from images at two or more viewpoints. , Included in the present invention.

In the third and fourth embodiments of the present invention, the reliability evaluation value of the estimated initial disparity value is as follows.
It is not necessary to limit to the one shown in (Equation 23). Even if only the numerator of (Equation 23) is used as the reliability evaluation value, the effect of the luminance gradient of the reference image can be obtained, but almost the same effect can be obtained. included.

When the noise level of the image is low, the same effect is naturally obtained even if a value ignoring the noise term is calculated as the reliability evaluation value, which is included in the present invention.

[0158] To further simplify, as a reliability evaluation value,
The minimum value of the residual sum of squares per pixel or the minimum value of the residual sum of squares may be used, and the calculation can be performed with a simpler circuit, which is included in the present invention.

As the reliability evaluation value of the estimated initial disparity value, the difference of the disparity estimated bidirectionally as shown in (Equation 32) may be used and is included in the present invention.

[0160]

(Equation 32)

Further, as a reliability evaluation value of the initial parallax estimation, a more stable reliability evaluation can be performed by using a combination of two or more of the above, and is included in the present invention.

In the third and fourth embodiments of the present invention, the correlation calculation between images for initial parallax estimation need not be limited to the sum of squares of residuals (SSD), but the sum of absolute values of residuals (SSD). The same effect can be obtained by using (SAD), and such an embodiment is of course included in the present invention.

In the sixth embodiment of the present invention, the method of compressing and encoding an image at two non-adjacent viewpoints need not be limited to a method utilizing correlation between images (between viewpoints). Alternatively, one utilizing correlation in the time direction may be used and is included in the present invention.

[0164]

As described above, according to the present invention, the size of the imaging surface (CCD) of the camera, the distance between the imaging surface and the center of the lens, and the information on the focal length of the lens are added and transmitted. By this, when trying to display according to the viewing angle at the time of shooting, even for an image shot close to the subject, the viewing angle at the time of shooting can be accurately calculated on the display side, and the same as at the time of shooting The observation distance for reproducing the viewing angle can be accurately determined.

Further, by adding information of the nearest point and the farthest point in the image when transmitting a multi-viewpoint image, it is possible to perform display (parallax control) without eyestrain at the time of display.

Further, by performing parallax control in accordance with the fusion range of human eyes, it is possible to cause the parallax to fall within the fusion range in more parts of the image during display.

On the transmission side, the nearest point to be added,
As the information of the farthest point, the nearest point in the actual image, a value different from the farthest point is set, and the parallax corresponding to the nearest point of the set value and the average of the parallax corresponding to the farthest point on the display side. By controlling the parallax so that the parallax of the image is in the center of the fusion range, an image at the depth intended by the image creator can be preferentially presented to the observer.

Further, according to the present invention, an object contour is detected in a region where the reliability of a parallax estimation value obtained by block matching is low, and the parallax is changed discontinuously at the detected object contour. An estimate can be made.

Further, it is possible to perform parallax estimation so that the parallax changes discontinuously at an object contour line having an arbitrary shape. .

By performing a parallax filling process (a parallax estimating process according to the present invention in which the parallax changes at the object contour line and smoothly connects to the surrounding parallax) on the transmitting side, Can be reduced, and the scale of the device on the receiving side can be reduced.

By generating an intermediate viewpoint image on both the transmitting side and the receiving side of the multi-viewpoint image transmission system,
The transmission amount of the intermediate viewpoint image (the transmission amount of the residual) can be reduced, and as a result, the multi-view image can be compressed and transmitted efficiently while maintaining the image quality.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a positional relationship between a nearest point, a farthest point, and a point where a vergence of an observer coincides with adjustment in parallel projection according to the first embodiment of the present invention.

FIG. 2 is a diagram showing the relationship between the position of the subject, the position of the imaging surface at the time of focusing, and the focal length.

FIG. 3 is a diagram showing a convergence distance and a positional relationship between a nearest point and a farthest point when convergence projection is performed using the two projectors.

FIG. 4 is a diagram showing parameters defined in the image transmission method according to the first embodiment of the present invention.

FIG. 5 is a block diagram of processing for shifting an average value of parallax between images so as to cancel each other.

FIG. 6 is a diagram showing a case where parallax is calculated by block matching based on a left image.

FIG. 7 is a diagram showing a case of parallel shooting;

FIG. 8 is a diagram showing a case of convergence imaging;

FIGS. 9A to 9C are diagrams illustrating examples of weight distributions used for calculating a weighted average according to (Equation 14).

FIG. 10 is a diagram showing the operation of an image decoding unit.

FIG. 11 is a block diagram of a parallax control method according to a second embodiment of the present invention.

FIG. 12 is a diagram illustrating an example of the configuration of a shift operation unit.

FIG. 13 is a characteristic diagram of a fusion range table.

FIG. 14 is a diagram showing the graphical meaning of (Equation 16);

FIG. 15 is a parallax frequency distribution diagram.

FIG. 16 is a configuration diagram of a disparity estimation device according to a third embodiment of the present invention.

FIG. 17 is a diagram showing the block matching.

FIG. 18 is a configuration diagram of the contour detection unit.

FIG. 19 is a configuration diagram showing an example of the configuration of the edge detection unit.

FIGS. 20A to 20F are diagrams illustrating examples of weighting factors of filters in the same direction;

FIG. 21 is a configuration diagram of the ridge line detection unit.

FIG. 22 is a configuration diagram of a parallax estimating unit near the object contour;

FIG. 23 is a configuration diagram of a disparity estimation device according to a fourth embodiment of the present invention.

FIG. 24 is a view showing parallax estimation in the vicinity of the contour of the object;

FIG. 25 is a configuration diagram of a transmission unit of a system that performs parallax estimation on the transmission side according to the fifth embodiment of the present invention.

FIG. 26 is a configuration diagram of a receiving unit of a system that performs parallax estimation on the transmitting side according to the fifth embodiment of the present invention.

FIG. 27 is a configuration diagram of a transmission unit of a multi-view image transmission system according to a sixth embodiment of the present invention.

FIG. 28 is a schematic diagram of MPEG-2 syntax.

FIG. 29 is a diagram showing the relationship between a spatio-temporal direction of a transmitted multi-viewpoint image

FIG. 30 is a diagram showing definition of camera parameters by OpenGL.

FIG. 31 is a diagram illustrating an example of a configuration of an image compression encoding unit of a multi-view image transmission system according to a sixth embodiment of the present invention.

FIG. 32 is a diagram illustrating an example of a configuration of a decoded image decompression unit of the multi-view image transmission system according to the sixth embodiment of the present invention.

FIG. 33 is a diagram illustrating an example of a configuration of a residual compression encoding unit of the multi-view image transmission system according to the sixth embodiment of the present invention.

FIG. 34 is a configuration diagram of a receiving unit of the multi-view image transmission system according to the sixth embodiment of the present invention.

FIG. 35 is a diagram illustrating an example of a configuration of a decoding residual decompression unit of the multi-view image transmission system according to the sixth embodiment of the present invention.

[Explanation of symbols]

 A Nearest point of displayed image B Farthest point C Point at which convergence and adjustment of observer coincide A1 and A2 Camera lens centers B1 and B2 Center of image plane C1 Convergence point 201 Initial disparity estimation unit 202 Initial disparity estimation Reliability evaluation section 203 contour detection section 204 parallax estimation section near object contour

Claims (19)

[Claims] 1. In transmission of an image of two or more viewpoints, a distance or a parallax range in an image is added and transmitted so as to be within the fusion range, and the transmitted distance or parallax range and the fusion range are transmitted. And displaying the image in consideration of the following. 2. In transmitting images from two or more viewpoints, information on the image pickup size of the camera and information on the distance between the lens center and the image pickup surface are added and transmitted to obtain information on the viewing angle at the time of image pickup on the display side. And a multi-view image transmission method. 3. The method according to claim 1, wherein a frequency for each parallax between the images is obtained, and parallax control is performed so that the frequency for each parallax is maximized in the fusion range to display images of the two or more viewpoints. The multi-view image transmission and display method according to claim 1. 4. A multi-viewpoint, wherein an average of parallax between the images is obtained, and parallax control is performed so that the average of the parallax coincides with the center of the fusion range to display images of the two or more viewpoints. Image transmission and display method. 5. The nearest point and the farthest point in an image are set on the transmission side, and the average of the parallax corresponding to the nearest point and the parallax corresponding to the farthest point on the display side coincides with the center of the fusion range. A multi-viewpoint image transmission and display method, wherein an image at a depth intended by an image creator is displayed in an easy-to-view manner by displaying parallax-controlled images at two or more viewpoints. 6. An object contour line in which an initial parallax of two images and a reliability evaluation value of the initial parallax are calculated, and the parallax changes discontinuously using the reliability evaluation value and an edge detection result of the images. Is extracted based on the reliability evaluation value, and an area with low reliability of the initial parallax is extracted.The extracted parallax in the area with low reliability of the initial parallax is smoothly connected to surrounding parallax. And determining that the object contour changes so as to perform a disparity estimation that changes discontinuously in the object contour. 7. The disparity in an area where the initial disparity is unreliable is determined so as to minimize the energy of a disparity distribution defined using the disparity and the object contour. 6. The disparity estimation method according to 6. 8. The disparity estimation method according to claim 6, wherein the edge detection is performed by integrating outputs of filters for different directions having different frequency characteristics. 9. The disparity estimation method according to claim 8, wherein in the edge detection, a ridge line is further extracted from an integration result of each output of the direction-dependent filters having different frequency characteristics. 10. The disparity estimation method according to claim 6, wherein the reliability evaluation value of the initial disparity uses a residual sum of squares at the time of initial disparity calculation. 11. The parallax estimating method according to claim 6, wherein the reliability evaluation value of the initial parallax uses a residual sum of squares per pixel at the time of initial parallax calculation. 12. The reliability evaluation value of the initial disparity is calculated using a residual sum of squares at the time of initial disparity calculation, a noise level of an image, and a luminance gradient.
The parallax estimation method according to any one of the above. 13. The method according to claim 6, wherein the reliability evaluation value of the initial parallax is calculated using a difference between correspondences of the initial parallax calculated based on both left and right images. Parallax estimation method. 14. The parallax estimating method according to claim 10, wherein the reliability evaluation value of the initial parallax is set to 2 in the parallax estimating method according to claim 10.
A parallax estimating method characterized by calculating by combining at least two. 15. The parallax in an area where the initial parallax and the reliability evaluation value of the initial parallax are calculated based on the left and right images from the binocular image, and the initial parallax cannot be correctly calculated. Recalculate by the method described in
A prediction error is calculated by predicting one image of the binocular image and the other image of the binocular image from the recalculated disparity based on the one image, and calculating the prediction error based on the one image. The parallax after the recalculation based on the other image is predicted from the parallax and a prediction error is calculated, and one of the two-lens images and the one after the recalculation based on the one image are calculated. A transmission unit that encodes and transmits the disparity, the prediction error of the other image of the binocular image, and the prediction error of the disparity after recalculation based on the other image, and receives the transmitted encoded signal. And decoding the binocular image and the recalculated disparity from the received signal, and generating an intermediate viewpoint image using the decoded binocular image and the decoded recalculated disparity,
An image transmission device comprising: a receiving unit that displays the intermediate viewpoint image and the binocular image together as a multiview image. 16. Using two images among multi-view images,
From the two images, a parallax is calculated based on one image, a prediction error in the other image is calculated using the parallax, and the obtained image, parallax, and prediction error are encoded and transmitted. Intermediate means for generating an image at a viewpoint corresponding to the remaining image of the multiview image other than the two images, using data decoded from one of the encoded images, parallax, and prediction error An image generation unit, and a transmission unit including a residual compression encoding unit that encodes and transmits a residual between the image generated by the intermediate viewpoint image generation unit and the remaining image of the multiview image, One of the encoded images, the disparity, the decoded image is decoded and the two images are received and decoded by using the decoded image decompression means for outputting the two images. Many An intermediate viewpoint image generation unit that generates an image at a viewpoint corresponding to the remaining image of the eye-type image, a decoding residual extension unit that receives and decodes the encoded residual, and the intermediate viewpoint image generation unit. An image transmission device, comprising: a receiving unit having means for restoring a remaining image of the multiview image using a generated image and the decoded residual. 17. The method according to claim 6, wherein the parallax is obtained.
17. The image transmission apparatus according to claim 16, wherein the method uses any one of the parallax estimating methods according to claim 14. 18. The transmitting unit selects a representative image from the multi-view images, and for the unselected image, the two representative images having the closest viewpoint direction across the image are used as reference images. A prediction is performed using the parallax calculated by any of the methods described in 6 to 14 and the two representative images, a residual between the unselected image and the predicted image is calculated, and the representative image and the selection are calculated. Encoding and transmitting the prediction error of the image that has not been transmitted, the receiving unit receives the transmitted coded signal, decodes the representative image, and transmits the image not selected as the representative image by the transmitting unit. A multi-view image transmission method, wherein prediction is performed by the same method as described above, a prediction error of the non-selected image is decoded, and the prediction error is superimposed on the predicted image to restore a multi-view image. 19. The encoding of the representative image includes predicting another representative image from one representative image and parallax calculated for another representative image based on the representative image, and encoding the one representative image and the representative image. 19. The multi-view image transmission method according to claim 18, wherein the method is performed by encoding a prediction error of another representative image.