JP2003333621A - Multi-viewpoint image transmission method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multi-viewpoint image transmission system which enables a display side to precisely calculate a visual field angle for photographing. <P>SOLUTION: In the multi-viewpoint image transmission method, information of an image pickup size of a camera for picking up an image and a distance between the center of a lens and an image pickup face is transmitted at the time of transmitting an image with two or more viewpoints, so that information of a visual field angle for image pickup can be obtained on the display side. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION [0001] BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to transmission of multi-view images.
About the method. [0002] 2. Description of the Related Art Conventionally, various types of stereoscopic video systems have been proposed.
3D video without special glasses
Multi-view images can be used by multiple people to observe images.
A multi-view stereoscopic video system is promising. Multi-view stereoscopic video
In the formula, the number of cameras and display devices used is
The more, the more observer feels natural motion parallax
And observation by a large number of people becomes easy. Only
However, restrictions such as the size of the imaging system and the setting of the optical axis of the camera
Limit the number of cameras that can be used practically
There is a degree. In the transmission and storage process,
It is desirable to reduce the amount of information that increases in proportion to the number
You. Therefore, on the display side, a binocular stereo
Multi-view stereo by generating intermediate viewpoint images from images
If images can be displayed, the burden on the imaging system can be reduced, and transmission and storage can be performed.
The amount of information at the time of integration can be reduced. point of view
Arbitrary views between different viewpoints from different images
To generate an intermediate viewpoint image that should be visible at a point, the image
It is necessary to estimate the depth by finding the correspondence of pixels between them. [0004] An image for digitally transmitting a moving image is also provided.
MPEG-1 and MPEG-2 are proposed as image compression methods
Have been. In addition, MPEG-2 has been expanded to
Attempts have been made to transmit images (ISO / IEC13818-2 / PDA
M3). FIG. 28 is a schematic diagram of MPEG-2 syntax.
is there. MPEG-2 transmission includes Sequence, GOP
(Group Of Picture), has a hierarchical structure called Picture
This is performed by encoding and decoding image data. ISO / IE
According to C13818-2 / PDAM3, many
The transmission of the viewpoint image is not clear (because it is not specified.
It does not seem to be implemented by extending the GOP layer. FIG. 29 shows a spatiotemporal image of a transmitted multi-viewpoint image.
It shows the relationship between directions. For conventional MPEG-2
Use parallax compensation in addition to the conventional motion compensation
To improve the coding efficiency. Multi-view images
When transmitting the information, information about each camera (camera
Camera parameters such as the position and orientation of the optical axis of the camera)
Must be transmitted. ISO / IEC13818-2 / PDAM3 includes:
The camera parameters are Pic.Extension (Picture layer) in FIG.
Is included in the transmission), but the
The description of the physical camera parameters is described.
Absent. For description of camera parameters, see CG
In the language OpenGL, the camera position,
The direction of the optical axis of the camera and the distance between the camera
Parameters ("Open
L Programming Guide "(OpenGL Programming Gu
ide, The Official Guide to Learning OpenGL, Release
1, Addison-Wesley Publishing Company, 1993)). FIG. 30 shows a camera parameter based on OpenGL.
It is explanatory drawing which shows the definition of a meter. In FIG. 30, A
Is the center of the lens, B is the center of the image plane (that is, the imaging plane),
C is the intersection of the perpendicular drawn from B to the top of the image and the top of the image
Show. The coordinate values of A, B, and C are respectively (optical cent
er X, optical center Y, optical center Z), (image pl
ane center X, image plane center Y, image plane cent
er Z), (image plane vertical X, image plane vertic
al Y, image plane vertical Z)
You. [0008] The camera path defined by the above OpenGL
Multi-view image by adding parameter information to Pic.Extension
Can easily be transmitted. [0009] However, as described above,
In conventional methods, such as the closest point and the farthest point (that is,
(The closest point, the farthest point of the subject)
Perform a display that does not cause eye fatigue (for example, parallax control).
Which depth range to use when trying to
I had the problem of not knowing. In displaying a multi-viewpoint image, a photograph
The viewing angle, the size of the imaging surface, and the distance between the lens center and the imaging surface.
Observation distance must be determined appropriately based on conditions such as distance
(There is a display that makes the display uncomfortable due to too much parallax,
So that the display does not have a poor sensation). Only
In OpenGL, the size of the imaging surface (the physical
Size) is not defined, and the lens center and
The distance from the image plane is always equal to the focal length of the lens.
I treat it. Therefore, the size of the viewing angle during imaging is
The display side does not know the
It is not possible to determine a sharp observation distance, resulting in a strange 3D display.
There was a problem that there is a possibility. The present invention has been made in view of the above, and has been described in view of the followings.
Of multi-view images that can calculate the viewing angle of
It is intended to provide a transmission method. [0012] According to the present invention, there are provided two or more viewpoints.
In image transmission, the imaging size of the camera that captures the image
Information about the distance between the lens and the center of the lens and the imaging surface.
With the above, information on the viewing angle at the time of imaging is obtained on the display side.
Multi-view image transmission method characterized in that
You. [0013] BEST MODE FOR CARRYING OUT THE INVENTION The present invention will now be described with reference to the embodiments.
Will be described based on the drawings showing (First Embodiment) FIG. 4 shows a first embodiment of the present invention.
Showing the parameters defined by the image transmission method in the state
It is. In FIG. 4, A1 and A2 are the center of the lens of the camera.
Indicates the position, and B1 and B2 indicate the center of the imaging surface.
The image pickup surface on the subject side with respect to the lens center.
I'm thinking back.) In OpenGL, the distance between A1B1 and A2B2 in FIG.
Is defined as the focal length of the camera lens.
In the invention, the distance between the lens center of the camera and the imaging surface is
Defined independently of the focal length of the lens. By this definition
Distance between the center of the lens and the imaging surface during focusing.
It can be calculated according to the distance, and the accurate viewing angle can be calculated. Field of view
The angle is the size of the imaging surface and the distance between the lens center and the imaging surface
Can be calculated from The center of the lens at the time of focusing will be described with reference to FIG.
The distance between the camera and the imaging surface depends on the distance between the subject and the center of the lens.
Will be described. FIG. 2 shows the position of the subject,
FIG. 4 is a diagram illustrating a relationship between a position of an imaging surface and a focal length during focusing.
You. In FIG. 2, A is the position of the subject, and B is the light from A.
Is the point where the image is formed, O is the center of the lens, F is the parallel light
A is the distance between the subject and the center O of the lens.
The distance b between the point B where the light from A forms an image and the lens center O
The distance, f, indicates the focal length of the lens. between a, b, f
It is known that the relationship of (Equation 1) holds. [0016] (Equation 1) According to (Equation 1), the subject can ignore the focal length.
When it is far from the lens (a >> f), 1 / a
 → 0, which can be approximated as b = f. However, if the subject
When relatively close to the lens, the 1 / a term can be ignored.
Instead, b ≠ f. Therefore, the subject is relatively close to the lens.
In order to calculate the viewing angle correctly even if the
The distance between the mind and the imaging plane must be defined independently of the focal length
There is. Then, the width of the imaging surface is defined as win and the height is defined as hin.
And the viewing angle at the time of imaging is represented by (Equation 2). [0018] (Equation 2) Therefore, the width of the image at the time of display is represented by wout and the height
Let hout be the observation distance that reproduces the viewing angle at the time of imaging
Is [0020] (Equation 3) ## EQU1 ## Next, a table based on the nearest point and the farthest point in the image
The improvement of the visibility on the presentation side will be described. FIG.
Is the case when performing convergence projection using two projectors.
To explain the positional relationship between the convergence distance, the nearest point, and the farthest point
FIG. In FIG. 3, C is the congestion point, A is the nearest point, B
Indicates the farthest point. In a projection with congestion, the observer is
When viewing the point C, the parallax becomes 0 (in FIG.
Both will see the center of the image, so the left and right eyes will see
There is no difference in relative position in the image). And the most
When looking at the near point A, it becomes a so-called cross-eyed state,
A parallax of Da occurs in the direction of the crossed eye on the image. Figure 3
The observer is more likely to see both eyes
Look at the point shifted by Da / 2 to the side. Conversely, look at the farthest point B
If you see the image
Db parallax occurs in the direction of the eyes. FIG. 1 shows the closest point and the most recent point in the case of parallel projection.
Indicates the positional relationship between the far point and the point where the vergence of the observer matches the accommodation.
FIG. In FIG. 1, A represents the most recent image to be displayed.
Point, B is the farthest point, C is the point where the observer's convergence and accommodation match
Is shown. In the case of the parallel projection shown in FIG. 1, there is a parallax of Dc.
When you display an image, it appears at the same point on the screen.
And the observer's convergence and accommodation match. The parallax in the images of FIGS. 3 and 1 described above.
Is closer to the screen than the screen (including C)
Is perceived as a three-dimensional effect
When it becomes impossible to fuse (it looks double), observation
Give the person discomfort and discomfort. The improvement of the observer's visibility is the latest point, the farthest point.
Based on the point and the point of convergence at the time of imaging, the image is displayed in the direction shown in FIG.
(Image 1 and Image 2 are horizontal in the vertical plane of each projection axis.
To the congestion point, the farthest distance, the closest distance
This is made possible by changing the positional relationship with. Without image
For example, cancel the average value of the disparity between images.
So that the entire image is viewed evenly
I can do it. FIG. 5 is a block diagram of such a process.
You. In FIG. 5, for the sake of simplicity, data of a two-lens system (two viewpoints) is shown.
Is shown. In FIG. 5, 1 is an image restoration
Signal means 2, parallax estimating means 3, average parallax calculating means 4,
a and 4b are image shift means, 5a and 5b are image display means
It is. The operation of each means will be described below. [0028] The image decoding means 1
The multi-view image data is received and decoded. Image decoding
The left and right images decoded by the means 1 are sent to the parallax estimating means 2.
Sent. The parallax estimating means 2 is
From the decoded left and right images, the parallax (parallax
Figure). For example, block based on the left image
FIG. 6 shows the case where parallax is calculated by matching.
This will be described below. First, set the window area in the left image
I do. Next, the residual sum of squares (SSD) shown in (Equation 4) is calculated.
I do. [0029] (Equation 4) Equation (4) is calculated in the range of dmin to dmax.
d is calculated at one pixel intervals. And dmin to dm
Set the value of d that minimizes SSD in the range of ax to the set window area.
Parallax in the area. The parallax at each pixel of the image is
By shifting the range sequentially and performing the above calculation,
Obtained. The SSD calculation ranges dmin and dmax are
It can be calculated from the information of the point and the farthest point. Using FIGS. 7 and 8
Dmin and dmax for parallel shooting and convergence shooting
Will be described below. FIG. 7 is a diagram showing the case of parallel photographing.
In the coordinate system shown in FIG. 7, the coordinate values of the left and right lens centers
To (−D / 2,0), (D / 2,0), imaging surface and lens
The distance from the center is b, the horizontal coordinate of the object position in the three-dimensional space
The value is X0, the coordinate value in the depth direction is Z0, and
Horizontal position where light from the object at position (X0, Z0) is imaged
Are defined as xl0 and xr0, respectively (xl0 and xr0 are the camera light
Horizontal coordinate in plane coordinate system with origin at the intersection of axis and imaging plane)
And from the graphical relationship, [0033] (Equation 5) ## EQU1 ## Therefore, based on the left and right images
Each of the parallaxes is represented by an equation shown in (Equation 6). [0035] (Equation 6) Here, the depth value of the nearest point in the image is Zmi
n, calculate the SSD assuming that the depth value of the farthest point is Zmax
The upper limit dmax and the lower limit dmin of the range are expressed by (Equation 7). [0037] (Equation 7) FIG. 8 is a diagram showing a case of convergence photographing.
You. In the coordinate system shown in FIG.
(0, C) coordinate value of the intersection of the optical axes of
Coordinate values of the heart are (-D / 2, 0), (D / 2, 0), and imaging
The distance between the surface and the center of the lens is b, the object position in the three-dimensional space
The horizontal coordinate value of X0, the coordinate value in the depth direction is Z0,
Light from an object at a position (X0, Z0) is imaged on the imaging surface
The horizontal positions are assumed to be x10 and xr0, respectively (xl0 and xr0 are
Water in a planar coordinate system with the origin at the intersection of the camera's optical axis and the imaging surface
(Coordinates) and the graphical relationship, [0039] (Equation 8) ## EQU4 ## Therefore, the left and right images are
The parallax at the time of doing each is expressed by the equation shown in (Equation 9).
You. [0041] (Equation 9) Whether X0 remains in the equation (Equation 9)
Therefore, in convergence imaging, even if the depth is the same,
The parallax differs depending on the position (that is, the reproduced stereoscopic image is
(Distorted). Now, for simplicity, X0 = 0 (ie,
Considering parallax at a point on the (Z-axis),
= 0 to obtain (Equation 10). [0043] (Equation 10)From equation (10), the depth of the nearest point in the image is obtained.
Value Zmin, depth value Zmax at the farthest point, depth value C at the point of convergence
, The number of horizontal pixels nx, and the width w of the imaging surface (CCD)
The upper limit pixel number dmax of parallax and the lower limit pixel number dmin can be determined from in.
Wear. A case where parallax at points other than on the Z axis is considered
In this case, the maximum and minimum values of (Equation 9) are calculated.
Thus, the upper limit dmax and the lower limit dmin of the parallax can be determined. As described above, the depth of the nearest point in the image
Outgoing value, farthest point depth value, camera position, camera light
Given the axis orientation, measure the range of values that parallax should take.
Can calculate the SSD calculation range during parallax calculation.
You. The average parallax calculating means 3 is calculated by the parallax estimating means 2.
The average of the calculated disparity map is calculated. The average disparity map is
It is obtained by calculating (Equation 11). [0047] (Equation 11) The image shift means 4a and 4b are used for the average parallax performance.
Point having average parallax obtained by the calculating means 3
Is the same depth as the display surface (that is, parallax 0 and
Shift the image so that it is displayed as In FIG. 1 showing the display by parallel projection,
A is the depth of the nearest point in the displayed image, and B is the depth of the farthest point.
Go, C indicates the depth of the average parallax. From FIG.
In the shadow, there is a parallax of Dc shown by (Equation 12) between the left and right images.
When there is no parallax on the screen,
It can be seen that the display becomes a natural display that matches. [0050] (Equation 12) The image shift means 4a is represented by (Expression 13)
Left image by shift amount (shift to right is positive)
Shift the image. [0052] (Equation 13) The image shift means 4b operates in the reverse direction.
Shift the right image by the same amount. Image shift means 4a
As a result of the shifts by 4b and 4b, points having average parallax
It will be displayed at the same depth as clean. FIG. 3 shows a display by convergence projection.
A is the depth of the closest point in the displayed image, B is the farthest
Point depth, C, indicates the depth of average parallax. With convergence projection
Is the same as the screen when parallax is 0 at the center of the image
Will be displayed at the depth of. Therefore, the congestion throw
In the case of a shadow, the image shift means 4a and 4b reduce the average parallax by-
The left and right images are shifted by half the value. As described above, according to the present embodiment, multi-view
When transmitting a point image, information on the nearest point and the farthest point in the image
By adding, the display that does not cause eye fatigue on the display side (visual
Difference control). Also, the size of the imaging surface (CCD) of the camera
Lens, the distance between the imaging surface and the center of the lens, and the focal length of the lens.
By adding information about the separation and transmitting it,
Approaching the subject when trying to display according to the viewing angle of the
Angle of view when shooting on the display side
Can be calculated accurately. Note that the nearest point and the farthest point in the multi-viewpoint image are
When transmitting without adding information to the
Instead of information on points, information on the nearest point and the farthest point
With a special code indicating that no information has been added.
Transmit and display the parallax within a preset range on the display side.
By performing calculations, it is possible to view the closest point and the farthest point in the image.
The difference can be estimated and is included in the present invention. Further, on the transmission side,
Set information about the nearest point and farthest point to a specific depth value
This allows the camera to be viewed in the specified specific depth range.
Parallax control can be performed so that the difference falls within the fusion range.
Included in the invention. In the present invention, the calculation of the parallax is displayed.
Although the example performed on the side has been described, in the encoded image
The included parallax may be used and is included in the present invention. FIG.
Such an example will be described using 0. In FIG. 10, components other than the image decoding means 6 are shown.
The operation is the same as the parallax control method shown in FIG.
The description is omitted, and the operation of the image decoding unit 6 will be described below.
I do. The image decoding means 6 decodes the encoded image data.
Output the parallax based on the left and right images and the left image.
You. Binocular image by multi-view image transmission method by MPEG-2
When transmitting the image, the parallax compensation based on the left image is used.
The compression ratio is increased. From the encoded image data
Calculate parallax on the display side by extracting parallax
This eliminates the need and reduces the amount of calculation on the display side. The average parallax calculated by the average parallax calculating means 3 is shown in FIG.
The calculation of the average is based on (Equation 14) with emphasis on the center of the screen.
A weighted average value may be used. If you do this,
At the center of the image, parallax control that is easier to fuse can be performed.
Included in the invention. [0062] [Equation 14] FIGS. 9 (a), 9 (b) and 9 (c) show (Expression 14)
An example of the distribution of weights used to calculate the weighted average by
You. Although shown one-dimensionally for simplicity, the actual
Two-dimensional distribution with a larger value at the center of the image than at the periphery
It is. Also, the weight values are all 0 or more (not negative
Value). (Second Embodiment) FIG. 11 shows a second embodiment of the present invention.
It is a block diagram of the parallax control system in a form. FIG.
, Except for the frequency calculation means 7 and the shift amount calculation means 8
The configuration performs the same operation as in the first embodiment.
This is the same as the explanation in the first embodiment.
The same reference numerals are given and the description is omitted. Below is the frequency calculation means
7. The operation of the shift amount calculating means 8 will be described. The frequency calculation means 7 is
Calculates the frequency of the parallax based on the left image decoded in this way. parallax
Is the frequency of an area of the image (for example,
Good or a specific area determined by the criteria)
Is the number of pixels calculated for each value of the parallax in. shift
The quantity calculating means 8 is calculated by the frequency calculating means 7
The frequency of parallax (between images) and the viewing angle of the image
From the eye fusion range, the sum of the parallax frequencies within the fusion range
A large shift amount is calculated, and the image shift means 4a, 4b
Output to FIG. 12 shows an example of the configuration of the shift operation means 8.
Is shown. In FIG. 12, 9 is an MPU and 10 is a fusion range.
It is a table. MPU9 is the width of the image display surface and the observation distance
The horizontal viewing angle shown in (Equation 15) is calculated from
Reading the fusion range at the field angle from the fusion range table 10
put out. [0066] (Equation 15) FIG. 13 shows an example of the characteristics of the fusion range table.
Show. In FIG. 13, the horizontal axis represents the horizontal direction of the image display surface.
The viewing angle is shown, and the vertical axis is based on the parallax fusion range ((Equation 16)).
Angle conversion). [0068] (Equation 16) The sign on the vertical axis in FIG. 13 indicates the negative side.
Parallax perceived before the surface, positive side is more than the display surface
The parallax perceived in the back is shown. FIG.
It is a figure which shows the graphic meaning of 6). FIG.
The calculated parallax θ converts the parallax Δ on the image display surface into a viewing angle.
Indicates that it was done. On the other hand, the parallel projection and the radiation shown in FIGS.
In convergence projection, the position of an image (for example, a liquid crystal projector
If so, the position of the pixel on the liquid crystal) xl1, xr1 and on the display surface
The positions of the positions Xl and Xr are expressed by (Equation 17) and (Equation 17), respectively.
19), and the parallax on the display surface is (Equation 18) (Equation 2)
0). [0071] [Equation 17] [0072] (Equation 18) [0073] [Equation 19] [0074] (Equation 20)Then, the coordinate value (x
l0, yl0), (xr0, yr0) and the position (x
l1, yl0), (xr1, yr1) (for example, with a liquid crystal projector
(If any, the position of the pixel on the liquid crystal) is given by (Equation 21)
expressed. [0076] (Equation 21) Here, the width win of the imaging surface is determined by the camera parameter.
The image width wout at the time of display is a value specific to the display system.
It is. According to imaging conditions (parallel imaging / convergence imaging)
Xl0 and xr0 are calculated using (Equation 5) or (Equation 8).
And converted to xl1, xr1 by (Equation 21). Furthermore,
In accordance with the projection conditions (parallel projection / convergence projection), (Equation 1)
8) By calculating (Equation 20),
Parallax on the display screen, taking into account both
Can be calculated. The MPU 9 reads from the fusion range table 10
Convert the protruding fusion range into parallax (distance) on the display surface,
The parallax fusion range on the image display surface is determined. And
The MPU 9 calculates the parallax and the image table in the image data described above.
Using the disparity relationship on the display surface, the disparity of the
Shift the image data so that the sum of the frequencies is maximized.
(The image shift due to parallax control is
Moving the frequency distribution of FIG. 15 horizontally in FIG.
Means). The output is output by the image shift means 4a and 4b.
The image is shifted in the reverse direction by the shift amount,
a, 5b, so that the
The sum of the disparity frequencies is the largest (that is, the pixels that fuse in the image
Can be displayed. As described above, according to the present embodiment,
For example, by performing parallax control according to the fusion range of human eyes
The parallax in more parts of the image when displayed
Inside. In the present embodiment, in the fusion range,
Parallax control that maximizes the sum of parallax frequencies was explained
However, the parallax control is performed so that the average value of the parallax is in the center of the fusion range.
Control can produce almost the same effect, and is included in the present invention.
I will. On the transmission side, the nearest point and the farthest point
To a value different from the nearest and farthest points in the actual image.
To the nearest and farthest point of the set value on the display side.
The average disparity of each applicable disparity is in the center of the fusion range.
By controlling the parallax so that the image creator
Images at different depths can be preferentially presented to the observer.
Included in the present invention. (Third Embodiment) A third embodiment of the present invention is described as follows.
Input image pairs, and determine the initial disparity and the reliability of the initial disparity.
Calculates the object contour from the reference image and the reliability of the initial parallax
The initial parallax and the reliability of the initial parallax and the detected object
From the body contour, the reliability of the initial parallax near the object contour
Determine parallax in low areas. At this time, the parallax is
It changes at the contour line and smoothly touches the surrounding parallax.
And a device for estimating the disparity to be determined to be continued.
You. In the present embodiment, the basic
The initial parallax and the initial parallax are calculated from a pair of image pair
Calculate the parallax reliability and the reliability of the reference image and the initial parallax
Object parallax is detected from the
From the detected object contour and the first
The parallax in the area where the period parallax is unreliable is
Change and smoothly connect with the surrounding parallax.
To decide. FIG. 16 shows a third embodiment of the present invention.
FIG. 2 is a block diagram of a parallax estimating device for performing the method. In FIG. 16, reference numeral 201 denotes a block match.
Initial disparity estimating unit for calculating an initial disparity due to weighting, 202
Is the reliability evaluation unit at the time of initial parallax estimation, and 203 is the contour detection
And 204, a parallax estimating unit near the object contour. The operation of the above configuration will be described below. The initial parallax estimating unit 201 is represented by (Equation 22)
Sum of Squared diff
calculations (hereinafter SSD). (Equation 2
The value of the SSD according to 2) is based on the window area set in the reference image.
The distribution of pixel values in the window area set in the reference image is similar.
Where the value is small, and conversely,
The value becomes large where the distribution of pixel values is different. initial
The disparity estimation unit 201 calculates the SSD value within a predetermined search range.
The shift amount d between the images to be minimized is set at the point of interest (x, y).
Estimation of the parallax near the object contour
The minimum value of the SSD within the search range is output to the
It outputs to the reliability evaluation part 202 at the time of period parallax estimation. [0089] (Equation 22) FIG. 17 is a diagram showing the results obtained by the initial parallax estimating unit 201.
FIG. 7 is a diagram illustrating initial parallax estimation (block matching).
is there. In FIG. 17, focusing on the point of interest (x, y),
The set window area indicates the integration area W of (Equation 22). window
The area is sequentially shifted and set, and the above SSD calculation is performed.
Thus, an initial parallax of the entire image can be obtained. The reliability evaluation unit 202 at the time of initial parallax estimation
SSD obtained by parallax calculation by the initial parallax estimating unit 201
The minimum value in the search range of, the pixels in the window area (block)
Number, noise variance between images, water of reference image in window area
From the average value of the square of the luminance gradient in the vertical and vertical directions,
The reliability evaluation value of the association shown in 3) is calculated. [0092] (Equation 23) The smaller the value of (Equation 23) is, the smaller the value of parallax estimation is.
Higher reliability indicates higher reliability
It indicates that. FIG. 18 shows an example of the configuration of the contour detection unit 203.
FIG. In FIG. 18, reference numeral 205 denotes a base.
A YC separation circuit for separating a quasi-image into a luminance component and a color component, 2
06A, 206B, and 206C are the separated luminance components.
Edges are detected from minute Y and color components RY and BY respectively
Edge detection circuit 207 is used to detect the edge line of the edge detection result.
Line detection unit that outputs only the intensity at
Output a weight of 1 in a region where the difference estimation value is unreliable,
Outputs a weight of 0 in regions where the disparity estimation value is highly reliable
It is a weight generation circuit. The operation of the above configuration will be described below. The YC separation circuit 205 converts the reference image into a luminance component.
Then, the image data is separated into minutes Y and color components RY and BY and output. Edge detection circuits 206A, 206B, 20
6C is an edge from the Y, RY, and BY components, respectively.
The di-component is detected. FIG. 19 shows the configuration of the edge detection circuit 206.
It is a block diagram showing an example of a composition. In FIG.
209A, 209B and 209C are low spatial frequencies, respectively.
Components in the low, medium and high spatial frequencies.
This is a group of direction-specific filters for detecting minutes. 210, 21
1, 212, and 213 denote filter groups for respective directions.
It is a direction-specific filter to configure. FIG. 20 shows the above directions.
It is an example of the spatial weight of a filter. FIG. 20 (a),
(B) and (c) show edges that are continuous in the vertical direction,
(D), (e), and (f) detect edges in oblique directions
Things. (A) and (d) are high spatial frequency ranges,
(B) and (e) are medium spatial frequency ranges, and (c) and (f) are low.
6 shows an example of a distribution of weights for a spatial frequency range. Horizontal and
The other edge detection in the oblique direction is based on the arrangement of the counting in FIG.
What is necessary is just to rotate 90 degrees. Edge direction is 45 degrees
It is not necessary to limit the interval to 30 degrees.
Of course. The spatial weights of the direction filters are shown in FIG.
It is not necessary to limit to those shown in FIG.
Of course, it is only necessary that the weight distribution is of a split type. each
The formula for calculating the edge strength for each direction is given by (Equation 24).
Become. [0100] (Equation 24) The integrating unit 214 includes direction-specific filters 210 and 2
11, 212, and 213 are integrated. Integration unit 214
An example of the integration by the following equation is represented by (Equation 25). [0102] (Equation 25) The integration by the integration unit 214 is (Equation 25)
It is not necessary to limit to the form of sum of squares shown by
Of course, it may be in the form of a sum of values. The luminance component Y and the color components RY and BY are described below.
High spatial frequency range, medium spatial frequency range, low spatial frequency range
And the integration units 214A, 214B and 214C respectively.
The integrated edge strength is multiplied and output. Soshi
The edge strength for each of the Y, RY, and BY components
The degrees are added and transferred to the ridge line detection unit 7. Note that the contour detection unit 203
Separation into luminance component and color component is limited to Y, RY, BY
There is no need to separate it into other components such as R, G, B
Is natural. Also, for Y, RY and BY
The edge strength is transferred to the edge detection unit 207 after the addition.
It is not necessary to limit to, and transfer to the edge detection unit 207 after multiplication.
May be. Returning to FIG. 18, the ridge line detecting section 207
Of the edge strengths added for Y, RY and BY
Output only the values at the edges. FIG. 21 shows a ridge line detection unit.
207 is an example of the configuration of FIG. In FIG. 21, the horizontal ridgeline
The detection circuit 215 determines that the edge intensity at the pixel of interest is above the point of interest.
1 if both are greater than the edge strength at the bottom pixel
Output, otherwise output 0. Similarly, the vertical ridge line detection circuit 216 outputs
The edge strength at the element is the edge strength at the left and right pixels of the point of interest
Is output if it is greater than both, otherwise.
Output 0. Horizontal ridge detection circuit 215 and vertical ridge
An output of the detection circuit 216 is subjected to an OR operation, and further, an input signal
Is output. That is, the ridge line detection unit 207
Is the edge of a pixel that is adjacent in the horizontal or vertical direction.
Pixels with an edge strength greater than the
Only the edge intensity at the pixel that is a line) is output
Then, 0 is output for the other pixels. Returning to FIG. 18 again, weight generation circuit 208
Indicates that the reliability evaluation value of the initial parallax
It outputs 1 and outputs 0 when it is less than the threshold value. weight
The output of the generation circuit 208 is multiplied by the output of the edge detection unit 207
Where the reliability of the initial disparity estimate is low
At the edge, that is, the object contour where the parallax changes discontinuously
Lines can be extracted. The output of the weight generation circuit 208 is
The calculation area of the parallax estimating unit 204 near the object contour described later
Stored in memory. The extraction of the object contour is expressed by the following equation.
(Equation 26). [0109] (Equation 26) The edge detection results 206A, 206B,
Add the output of 206C and input it to the edge detection unit 7
It is not necessary to limit to
You may. Also, it is multiplied by the output of the edge detection unit 207.
The method of weight generation by the weight generation circuit 208 is as follows.
It does not need to be limited to binary values, depending on the reliability at the time of initial parallax estimation.
It is natural that a continuous value may be output. The parallax estimating section 204 near the object contour
Initial disparity estimation near body contour
The disparity is recalculated from the contour strength and the initial disparity. Object contour
The nearby disparity estimation unit 204 is defined by (Equation 27).
Disparity distribution that minimizes energy for disparity distribution
Is calculated. [0112] [Equation 27] The weight function w (x, y) is a parameter of smoothness.
(Equation 28) based on the data and the contour strength. [0114] [Equation 28] The condition of the parallax distribution that minimizes (Equation 27) is
(Equation 29). [0116] (Equation 29) The differential equation (Equation 29) is obtained by the finite element method.
It can be solved numerically by a known technique such as (FEM).
it can. FIG. 22 shows a parallax estimating unit 2 near the object contour.
FIG. 4 is a block diagram illustrating an example of a configuration of the information management unit 04. FIG.
217 generates weights for the disparity distribution energy
A weight generation circuit for parallax distribution energy, 218 is a calculation area
Memory, 219 is a parallax memory, 220 is a weight memory, 2
21 is an FEM operation circuit. Parity distribution energy weight generation circuit 217
Is obtained from the parameters λ of the contour strength and smoothness (Equation 28)
Calculates the value of the weight function and writes it to the weight memory 220
No. The FEM operation circuit 221 calculates (Equation 29) by the finite element method.
To calculate the parallax distribution. As described above, according to the present embodiment,
In areas where the reliability of parallax estimation values by
The object contour, and
It is possible to perform disparity estimation so that disparity changes discontinuously in
it can. According to the present embodiment, any shape
Disparity so that it changes discontinuously at the object contour line
An estimate can be made. Note that the parallax estimation near the object contour is based on the parallax
It changes at the contour of the object, and smoothly with the surrounding parallax.
Just connect them and minimize the energy shown in (Equation 27)
It is not necessary to limit to a method of calculating as parallax to be performed. That's it
Such an example will be described below. (Fourth Embodiment) FIG. 23 shows a fourth embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of a parallax estimating device according to an embodiment.
You. In FIG. 23, reference numeral 201 denotes block matching.
The initial parallax estimating unit 202 that calculates the initial parallax
A reliability evaluation unit at the time of difference estimation, 222 is a contour detection unit, 223
Is a parallax estimating unit near the object contour. In the above configuration, the contour detecting section 222, the object
The operation of the configuration other than the parallax estimation unit 223 near the body contour is
The description is omitted because it is the same as the third embodiment of the invention.
In the following, the contour detection unit 222 performs parallax estimation near the object contour.
The operation of the setting unit 223 will be described. First, the contour detecting section 222 is the third embodiment of the present invention.
Performs the same contour detection as the contour detection unit in the embodiment.
No, binarize the detection result (for example, 0 and 1) and output
You. The parallax estimating unit 223 near the object contour calculates the object contour line
The disparity in the region where the initial disparity estimation value in the vicinity is unreliable is
Object wheel detected by initial parallax and contour detection unit 222
Calculate from the contour line. FIG. 24 shows a parallax estimating unit 2 near the object contour.
FIG. 14 is a diagram showing a state of parallax estimation by 23. FIG.
291 is an area where the initial disparity estimated value is unreliable, 2
Reference numeral 92 denotes an object contour detected by the contour detection unit 222
The line 293 is a reliable region of the initial disparity estimation value, 29
4 is the point of interest to calculate parallax, 295 is the point of interest
This is a window area set to be included. The disparity at the point of interest 294 (x, y) is
A region 29 in which the initial disparity estimation value has low reliability within the setting window region
1 surrounding area (in this case, the initial parallax estimation value
Using the parallax in the highly reliable region 293a),
The disparity at 94 is the distance between the surrounding area and the point of interest 294.
Depending on the parallax value in the surrounding area,
decide. At this time, the parallax in the surrounding area is
Be careful not to affect point of interest 294 beyond section line 292
Changes at the object contour line 292,
The parallax so that it is smoothly connected to the surrounding parallax.
Can be determined. Viewing by the disparity estimation unit 223 near the object contour
Expression of the difference estimation as an example is (Equation 30). [0127] [Equation 30] However, the parallax estimating unit 22 near the object contour
The parallax estimation by 3 need not be limited to (Equation 30).
The difference changes at the object contour line, and smoothly changes with the surrounding parallax.
Of course, any connection can be used. As described above, according to the present embodiment,
In areas where the reliability of parallax estimation values by
The object contour, and
It is possible to perform disparity estimation so that disparity changes discontinuously in
it can. According to the present embodiment, any shape
Disparity so that it changes discontinuously at the object contour line
An estimate can be made. Furthermore, according to the present embodiment, the initial parallax
Compare in the vicinity of the point of interest in areas where the estimated value is unreliable
By calculating the disparity by referring to the disparity around the target
Calculation of parallax with a small amount of memory and computation
Can be. Further, the third and fourth embodiments have been described.
Shift and integrate the left and right images using the result of parallax estimation
By doing so, each viewpoint corresponding to the left and right images
Can be generated at a predetermined intermediate viewpoint. here
Perform parallax estimation and intermediate viewpoint image generation in different places
You may. The difference between parallax estimation and intermediate viewpoint image generation is described below.
A transmission and reception method when performing in a certain place will be described. (Fifth Embodiment) FIG. 25 shows a fifth embodiment of the present invention.
In the embodiment, the parallax estimation (or motion estimation) is performed on the transmission side.
1 is an example of a transmission block of a system that performs the following. In FIG. 25, reference numeral 170 denotes a left image as a reference.
Disparity estimating means for estimating the disparity VL obtained, 171 is a right image
Estimating means 172 for estimating disparity VR based on
a to d are encoders, 173a and b are decoders, and 174 is
The right image R is obtained from the left image L and the parallax VL based on the left image.
The prediction unit 175 for predicting the parallax V based on the left image
Prediction means for predicting parallax VR based on the right image from L,
176a and 176b are disparities in an area where disparity is not correctly estimated.
Is a means for filling in the gap. The operation of the above configuration is described below.
explain about. First, the left image L is encoded by the encoder 172a.
Is encoded. Also, the parallax estimating means 170 and 171
Therefore, the parallaxes VL and VR based on the left and right images respectively are
Presumed. Parallax is correctly estimated by occlusion etc.
The third or fourth embodiment is applied to the area not
Filling means 176a using the parallax estimation method described in
The parallax is determined by 176b. Next, the parallax after filling in the holes based on the left image.
Is encoded by the encoder 172b. Encoded
The parallax after filling in with the left image as a reference is calculated by the decoder 173.
a, the prediction of the right image R by the predictor 174.
Measurement and the right image after filling in by the predictor 175
It is used to predict the disparity. Right image by predictor 175
The prediction of the parallax VR based on the left image is based on the parallax VR based on the left image.
Using the difference, calculation is made as (Equation 31). [0136] (Equation 31) The right image R is the same as the image predicted by the predictor 174.
, And encoded by the encoder 172d.
You. The parallax VR after filling in the holes based on the right image is a predictor
The residual with the predicted disparity by 175 is calculated, and the
c. FIG. 26 shows a system for performing parallax estimation on the receiving side.
It is an example of a reception block of a system. In FIG. 26, 18
1a to 1d are decoders, 174 is a predictor for the right image R, 17
5 is a parallax predictor based on the right image. Encoded
Left image L, left image based parallax VL, right image based viewing
The prediction error of the difference VR and the prediction error of the right image R are respectively decoded.
Decoders 181a to 181d decode the data. Right image R
Is the prediction result of the predictor 174 and the decoded right image.
It is restored by adding the prediction error. Parallax V based on right image
R is the prediction result of the predictor 175 and the decoded prediction
The error is restored by adding the error. A left image L, a right image R, and a parallax V based on the left image.
L, when the parallax VR based on the right image is restored, for example,
No. 7-109821
It is possible to generate an image with an intermediate viewpoint between the left and right images.
Display as a multi-viewpoint image along with the left and right images
be able to. As described above, with the above configuration,
Performing parallax estimation and fill-in processing on the transmitting side enables reception
The amount of calculation on the receiving side can be reduced, and the device scale on the receiving side
Can be reduced. When transmitting a multi-viewpoint image, the transmission side
Reduces the amount of transmission by generating intermediate viewpoint images
Image transmission can be performed. For such an example,
This is described below. (Sixth Embodiment) FIG. 27 shows a sixth embodiment of the present invention.
Side structure of multi-view image compression transmission system
FIG. In FIG. 27, 101a to 101d are
A camera for capturing an image at the viewpoint position;
Image compression code for compressing and encoding the image of
The coding unit 103a is a compression code by the image compression coding unit 102.
A decoded image decompression unit for decoding and decompressing the decoded image data,
104a indicates that the decoded image decompression unit 103a has decoded and decompressed
From camera 1 image and camera 4 image, camera 2 viewpoint
Viewpoint image that predicts and generates an image from the viewpoint of camera and camera 3
The image generation unit 105 converts the image of the camera 2 and the image of the camera 3
And the image generated by the intermediate viewpoint image generation unit 104a.
This is a residual compression encoding unit that compresses and encodes the residual. Less than
Next, the operation of the above configuration will be described. The image compression / encoding unit 102 converts a multi-view image
A plurality of images (in the present embodiment, four
Viewpoint images) are already stored using block correlation between images.
Compress and encode using existing techniques. FIG. 31 shows image compression.
1 shows an example of the configuration of an encoding section 102. In FIG. 31,
107a and 107b are 8 × 8 pixels or 16 × 16 pixels
DCT method that performs DCT calculation for each element and calculates DCT coefficients
The stages 108a and 108b are quantum machines for quantizing DCT coefficients.
Means 109a, inverse quantization means, 110a inverse DCT
Inverse DCT means for performing calculation, 111 is disparity detection means,
112a is the parallax compensating means, 113a is the quantized DC
This is an encoding unit that encodes the T coefficient and the disparity. Below on
The operation of the above configuration will be described. The DCT means 107a converts the image of the camera 1
Process block by block and calculate DCT coefficient for each block
calculate. The quantizing means 108a quantifies the DCT coefficient.
Become a child. The inverse quantization means 109a outputs the quantized
The DCT coefficients are inversely quantized. The inverse DCT means 110a
The inversely quantized DCT coefficients are inversely transformed and obtained on the receiving side.
The image of the camera 1 is restored. Parallax detecting means 111
Is the block between the restored camera 1 image and camera 4 image.
To perform a visual matching based on the image of camera 1
The difference is calculated for each block. The parallax compensation means 112a
Using the restored image of the camera 1 and the parallax of each block
To predict the image of the camera 4 (that is,
And performs a process corresponding to compensation). DCT means 107b
Represents the residual of the image of the camera 4 and the above-described predicted image for each block.
To calculate the DCT coefficient. The quantization means 108b
The residual DCT coefficient is quantized. Encoding means 113
a is a quantized DCT coefficient of an image of the camera 1;
Quantized DCT coefficient of parallax for each check, residual of parallax compensation
Encode the number. Further, the decoded image decompression unit 103a
Image data compressed and encoded by the compression encoding unit 102
Decrypts and decompresses the data. FIG. 32 shows the decoded image decompression unit 1.
It is a figure showing an example of composition of 03a. In FIG. 32,
114a is decoding means, 109b and 109c are inverse quantization
Means, 110b and 110c are inverse DCT means, and 112b is
This is parallax compensation means. The operation of the above configuration is explained below.
I will tell. The decoding means 114a outputs the compressed and coded
The data is decoded, and the quantized DC
T coefficient, parallax for each block, quantized residual of parallax compensation
The obtained DCT coefficient is extended. Quantized image of camera 1
The inverse DCT coefficient is inversely quantized by the inverse quantization means 109b.
And is decompressed as an image by the inverse DCT means 110b.
Lengthened. The motion compensating means 112b
Prediction of the camera 4 from the image of the camera 1 and the decoded parallax
Generate an image. Then, the inverse quantization means 109c, the inverse D
The residual expanded by the CT means 110c is
The image of camera 4 is decompressed by adding to the image. [0146] The intermediate viewpoint image generation unit 104a is the same as that of the present invention.
The method shown in either the third or fourth embodiment
Therefore, the parallax of each pixel is obtained from the images of the camera 1 and the camera 4.
Calculate and predict and generate the images of camera 2 and camera 3. [0147] The residual compression encoding unit 105 is
Compresses and encodes the residual between the image of camera 3 and the predicted image
You. The intermediate viewpoint image generation unit 104a calculates the parallax for each pixel.
For each block by block matching.
The parallax can be estimated more accurately than the difference calculation. The result
As a result, the prediction error (that is, residual error) of the intermediate viewpoint image is reduced.
And increase compression efficiency
More efficient bit allocation,
Can be maintained. FIG. 33 shows a residual compression encoding unit.
An example of the configuration will be shown. In FIG. 33, 107c, 10
7d is DCT means, 108c and 108d are quantization means,
113b is an encoding means. Image of camera 2 and camera 3
The residuals of the image are calculated by the DCT units 107c and 107d, respectively.
Are converted into DCT coefficients by the quantization means 108c and 10c.
8d and quantized by the encoding means 113b.
Encoded. FIG. 34 shows a sixth embodiment of the present invention.
FIG. 1 is a configuration diagram of a receiving side of a multi-viewpoint image compression transmission system according to the present invention.
You. In FIG. 34, reference numeral 103b denotes an image compression code on the transmission side.
Of the camera 1 and the camera 4 compressed and encoded by the encoding unit 102
A decoded image decompression unit 104b that decodes and decompresses data,
With the camera 1 that the decoded image decompression unit 103b has decoded and decompressed,
From the image of camera 4, the image from the viewpoint of camera 2 and camera 3
An intermediate viewpoint image generation unit for predicting and generating an image,
Prediction error (residual error) of the prediction image from the viewpoints of camera 2 and camera 3
Is a decoding residual expansion unit that decodes and expands. Decrypted image
The image expansion unit 103b and the intermediate viewpoint image generation unit 104b
Regarding the operation, the decoded image decompression unit 103a and the
And the operation is the same as that of the intermediate viewpoint image generation unit 104a.
In the following, the operation of the decoding residual decompression unit will be described.
Will be explained. The decoding residual decompression unit 106 outputs the residual
Camera 2 compressed and encoded by the compression encoding unit 105
And the prediction error (residual) of the predicted image from the viewpoint of the camera 3
And expand it. FIG. 35 is a block diagram of the decoding residual decompression unit 106.
1 shows an example of the configuration. In FIG. 35, reference numeral 114b denotes decryption
Means, 109d and 109e are inverse quantization means, 110d,
110e is an inverse DCT means. Compression-encoded turtle
The residual data of the image of camera 2 and camera 3 is
4b, respectively, and the inverse quantization means 10
9d and 109e, inversely quantized, and inverse DCT means 11
0d and 110e. Decompressed and expanded
The residual of the images of the camera 2 and the camera 3 is converted into an intermediate viewpoint image generation unit.
104b is superimposed on the image generated by
Restores the images of the viewpoints of camera 2 and camera 3
You. As described above, according to the present embodiment, transmission
On the transmitting side, the two non-adjacent images in the multi-viewpoint image
Generate an intermediate viewpoint image, and use the generated intermediate viewpoint image
The residual from the actual image of the intermediate viewpoint is obtained, and the above two
Image and the residual of the intermediate viewpoint image are compressed and transmitted.
You. On the receiving side, the two transmitted images and the intermediate viewpoint image
Decode and expand the residual of the image, and extract the intermediate viewpoint from the two images.
An image is generated, and the residual of the decoded and expanded
To restore the image corresponding to the actual image at the intermediate viewpoint
You. In this way, multi-view images can be efficiently
In addition, compression transmission can be performed while maintaining image quality. It should be noted that the generation of the intermediate viewpoint image is based on the multi-view image.
At two viewpoints (viewpoints of camera 1 and camera 4) at both ends of
It is not necessary to limit to a configuration that generates an image with an intermediate viewpoint from
For example, from the images of camera 2 and camera 4,
An image from the viewpoint of the camera 3 may be generated.
Image from camera 2 and camera 4 from camera 3 image
May be generated. Furthermore, the images of camera 2 and camera 3
From the viewpoints of camera 1 and camera 4
And each is included in the present invention. The number of viewpoints of a multi-view image is limited to four.
It is not necessary, and it can be
It is clear that an intermediate viewpoint image between each viewpoint may be generated.
And included in the present invention. Further, the third and fourth embodiments of the present invention
In the state, as the reliability evaluation value of the initial parallax estimation value,
It is not necessary to limit to the one shown in (Equation 23).
Even if only the molecule is used as the reliability evaluation value, the brightness
Affected by the distribution, but can achieve almost the same effect
Included in the invention. When the noise level of the image is low,
Calculates a value that ignores the noise term as the reliability evaluation value.
It is natural that the same effect can be obtained even if it is included in the present invention.
It is. Further simplified, as a reliability evaluation value,
Minimum sum of residual squares per pixel, or residual square
The minimum value of the sum may be used, and calculation can be performed with a simpler circuit.
And is included in the present invention. Further, the reliability evaluation value of the initial parallax estimation value is used.
In other words, the difference between the estimated parallaxes in
It may be used and is included in the present invention. [0157] (Equation 32)As the reliability evaluation value of the initial parallax estimation,
By combining two or more of the above
It is possible to perform more stable reliability evaluation
include. Further, the third and fourth embodiments of the present invention
In the state, the correlation calculation between the images for the initial disparity estimation is
It is not necessary to limit to the residual sum of squares (SSD).
The same effect can be obtained by using (SAD).
Such an embodiment is of course included in the present invention. In the sixth embodiment of the present invention,
Compression encoding of images from two non-adjacent viewpoints
Limited to those using correlation between images (between viewpoints)
It is not necessary to use
May also be included in the present invention. [0161] As described above, according to the present invention, the camera
The size of the imaging surface (CCD) and the distance between the imaging surface and the center of the lens
Distance and information on the focal length of the lens.
The display according to the viewing angle at the time of shooting.
When trying to shoot an image taken close to the subject
Can also calculate the viewing angle at the time of shooting with high accuracy on the display side.
Accuracy of the observation distance that reproduces the same viewing angle as when shooting
Can be determined well. When transmitting a multi-viewpoint image,
By adding the information of the nearest point and the farthest point,
A display (parallax control) without eyestrain can be performed. Also, parallax control according to the fusion range of human eyes
By doing so, when displaying more parts of the image
Parallax can be within the fusion range. On the transmitting side, the nearest point to be added,
As the information of the farthest point, the nearest point and the farthest point in the actual image
Sets a different value, and displays the nearest point of the set value.
And the average disparity of the disparity corresponding to the farthest point
Is controlled by parallax control so that it is in the center of the fusion range.
The image at the depth intended by the image creator.
Can be presented to the observer. According to the present invention, a block match
In areas where the reliability of the parallax estimate by
Contour is detected and parallax is discontinued at the detected object contour
The disparity estimation can be performed so as to continuously change. Also, the parallax at the object contour line of an arbitrary shape
Disparity estimation can be performed so that
You. . Also, the parallax filling process on the transmitting side (the present invention)
, The parallax changes at the object contour and the surrounding
Parallax estimating process that smoothly connects to parallax).
The amount of calculation on the receiving side can be reduced,
The device scale can be reduced. Further, the transmission side of the multi-viewpoint image transmission system
By generating the intermediate viewpoint image on both the receiving side,
Reduce the transmission amount of the intermediate viewpoint image (transmission amount of the residual)
As a result, multi-view images can be efficiently
Can be transmitted with compression.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram showing a positional relationship between a nearest point, a farthest point, and a point where the convergence and adjustment of an observer coincide with each other in parallel projection according to the first embodiment of the present invention. FIG. 2 is a diagram showing the relationship between the position of the subject, the position of the imaging surface at the time of focusing, and the focal length. FIG. 3 is the convergence distance, nearest point, and farthest point when performing convergence projection using the two projectors. FIG. 4 is a diagram showing parameters defined in the image transmission method according to the first embodiment of the present invention. FIG. 5 is a block diagram of a process for shifting so as to cancel the average value of disparity between images. FIG. 6 shows a case where parallax is calculated by block matching with reference to the left image. FIG. 7 shows a case of parallel shooting. FIG. 8 shows a case of convergence shooting. FIG. 9 (a) To (c) are distributions of weights used for calculating the weighted average by (Equation 14) FIG. 10 is a diagram showing an operation of an image decoding unit. FIG. 11 is a block diagram of a parallax control system according to a second embodiment of the present invention. FIG. 12 is an example of a configuration of a shift operation unit. FIG. 13 is a characteristic diagram of the fusion range table. FIG. 14 is a diagram showing the graphical meaning of (Equation 16). FIG. 15 is a parallax frequency distribution diagram. FIG. 16 is a diagram showing a third embodiment of the present invention. FIG. 17 is a diagram showing the same block matching. FIG. 18 is a diagram showing the configuration of the same contour detection unit. FIG. 19 is a diagram showing an example of the configuration of the edge detection unit. FIGS. 21A to 21F are diagrams illustrating examples of weight coefficients of filters in the same direction. FIG. 21 is a configuration diagram of the same ridge line detection unit. FIG. 22 is a configuration diagram of a parallax estimation unit near the same object contour. FIG. 24 is a configuration diagram of a parallax estimating device according to a fourth embodiment of the present invention. FIG. 25 is a diagram illustrating a disparity estimation of the transmission unit. FIG. 25 is a configuration diagram of a transmission unit of a system that performs disparity estimation on the transmission side according to the fifth embodiment of the present invention. FIG. 26 is a transmission side according to the fifth embodiment of the present invention. FIG. 27 is a configuration diagram of a transmission unit of a multi-viewpoint image transmission system according to a sixth embodiment of the present invention. FIG. 28 is a schematic diagram of MPEG-2 syntax. FIG. 29 is a diagram showing the relationship between the spatio-temporal directions of the transmitted multi-viewpoint images. FIG. 30 is a diagram showing the definition of camera parameters by OpenGL. FIG. 31 is an image of the multi-viewpoint image transmission system according to the sixth embodiment of the present invention. FIG. 32 is a diagram illustrating an example of a configuration of a compression encoding unit. FIG. 32 is a diagram illustrating an example of a configuration of a decoded image decompression unit of a multi-view image transmission system according to a sixth embodiment of the present invention. Multiple viewpoints in the sixth embodiment FIG. 34 is a diagram illustrating an example of a configuration of a residual compression encoding unit of the image transmission system. FIG. 34 is a configuration diagram of a reception unit of the multi-view image transmission system according to the sixth embodiment of the present invention. The figure which shows an example of a structure of the decoding residual decompression part of the multi-view image transmission system in 6th Embodiment [Description of code] A The nearest point of the displayed image B The farthest point C The congestion and adjustment of the observer Matching points A1, A2 Camera lens centers B1, B2 Image plane center C1 Convergence point 201 Initial parallax estimating unit 202 Reliability evaluation unit 203 for initial parallax estimation 203 Contour detecting unit 204 Parallax estimating unit near object contour

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Jun Morimura             Matsushita Electric, 1006 Kadoma, Kazuma, Osaka             Sangyo Co., Ltd. F term (reference) 5C061 AA21 AB04 AB08 AB12

Claims (1)

Claims: 1. In transmitting an image from two or more viewpoints, information on an image pickup size of a camera for picking up an image and a distance between a lens center and an image pickup surface are transmitted, so that an image is picked up on a display side. A multi-view image transmission method, wherein information on a viewing angle can be obtained.