JPH07220113A - Image recording/reproducing device - Google Patents

Abstract

PURPOSE:To shorten the operation time by preventing the extraction of parallax at a place near a shielding contour and by extracting the parallax for each block and in consideration of the human stereoscopic sense characteristic. CONSTITUTION:The images photographed by two cameras 1L and 1R provided on a stage 2 are recorded as the image data on the memories 3L and 3R respectively. When these image data are supplied to an image processing part 4 from both memories 3L and 3R, the parallax discontinuous contours of the image data are extracted. Then the image data are divided into plural blocks of each prescribed size so as to avoid the crossing to the parallax discontinuous contours. The parallax of each image is extracted for each divided block. Based on the extracted parallax, a both-eye visual area is discriminated from a single- eye visual area. The parallax of the both-eye visual area is converted into the parallax that is accordant with the placement of a reproduction optical system. Thus the stereoscopic images are generated and displayed at an image display part 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image recording / reproducing apparatus for forming various stereo images from images picked up by a plurality of cameras.

[0002]

2. Description of the Related Art Conventionally, three-dimensional image information (image) is obtained by photographing an object with a plurality of cameras, and the obtained three-dimensional image information is displayed in accordance with the visual characteristics of a human. As a result, a binocular parallax method is known as a method for forming a stereoscopic image.

In the above-mentioned binocular parallax method, when photographing an object, when the human being's binocular vision is taken into consideration to some extent, the baseline length and vergence angle of the camera system are set, and the photographed image is reproduced. In addition, a stereo image is formed by giving an appropriate parallax according to the object distance and the three-dimensional shape to the left and right eyes of a human and displaying the image. (Broadcasting technology, 199
1 year, November issue, 119-125, Kumada).

In the above case, when reproducing a photographed image, it is necessary to display a correct image in which the difference in the arrangement of the optical system (base line length, vergence angle, etc.) at the time of photographing and reproduction is reflected. .

Therefore, in the conventional image recording / reproducing apparatus using the binocular parallax method, the distance information to the camera system of each point on the object and the three-dimensional shape information of the object are previously recorded at the time of photographing.
Measured by non-contact measuring means using laser light, or extracting points corresponding to the same object (hereinafter, referred to as corresponding points) in each image captured by two cameras, and calculating the parallax obtained from the points. It is obtained by using the principle of triangulation from the value and the position of the camera, and the parallax and brightness level of each point of the image to be reproduced are obtained based on the obtained distance information and stereoscopic shape information. There is.

For example, an image obtained by photographing an object is stored as image data, and the vertices of the three-dimensional shape of the object (points at which the object photographed is reproduced by tying) are extracted from the image data. It has been proposed to obtain distance information and shape information for the camera system at the apex and perform arithmetic processing on the distance information and shape information according to given viewpoint position information to generate an image observed from the viewpoint position. (See Japanese Patent Laid-Open No. 5-12413).

Other than the above example, Japanese Patent Publication No. 1-4
An image obtained by creating a shape model from image data photographed based on a basic geometric shape model which is a reference for image formation as disclosed in Japanese Patent No. 2427, and from the created shape model and arbitrary viewpoint position information. There is a method in which image data captured is deformed according to the amount of deformation and a stereo image observed from a viewpoint position different from that at the time of image capturing is generated.

As described above, in the conventional image recording / reproducing apparatus, the distance information at each point of the object is measured using either the non-contact measuring means using the laser beam or the measuring means utilizing the principle of triangulation. Then, the stereoscopic shape information is obtained, the parallax and the brightness level at each point are calculated based on the obtained result, and a stereo image with a camera arrangement different from the camera arrangement at the time of shooting is formed.

In the above-mentioned conventional image recording / reproducing apparatus, a stereo image is formed based on images taken by a plurality of cameras. In addition to this, a binocular parallax image is created by using computer graphics. In some cases, the parallax at each corresponding point of the image is calculated and the image is displayed on the head-mounted display or the like.

[0010]

However, in the case of the above-mentioned conventional image recording / reproducing apparatus, the extraction of the corresponding points in each image captured by the two cameras detects the boundary (hereinafter referred to as the boundary where disparity changes discontinuously). It is performed without detecting the occluding contour). Therefore, when the corresponding points are extracted in the vicinity of the occluding contour, the corresponding points are not extracted, and the distance and the three-dimensional shape of the object cannot be obtained accurately. As a result, there is a problem that a good stereo image cannot be obtained from the reproduced image. In particular, the problem of extraction of the corresponding points is conspicuous when the object is at a close range.

Further, in the conventional image recording / reproducing apparatus, the extraction of the parallax (lateral displacement) of each image is required for each corresponding point of the object. In such parallax extraction, the depth resolution (longitudinal resolution) is the viewing angle when there are two objects in which the stereoscopic perception characteristics of human beings overlap in the shooting direction with respect to the depth change of the object. It is on the order of seconds in terms of conversion, and it is not taken into consideration that the depth resolution when two objects are arranged parallel to the shooting direction is about 10 minutes, and as a result parallax is more than necessary. Was highly accurate. Therefore, there is a problem that the calculation time becomes long.

Similarly, when a stereo image is created from a binocular parallax image by computer graphics, the above-mentioned human stereoscopic perception characteristics are not taken into consideration, so that many calculations are unnecessarily performed.

An object of the present invention is to provide an image recording / reproducing apparatus which does not extract corresponding points in the vicinity of an occluding contour, and which can perform parallax extraction and image display in consideration of human stereoscopic perception characteristics. To provide.

[0014]

The image recording / reproducing apparatus of the present invention is an image recording / reproducing apparatus for forming various stereo images from images photographed by using a plurality of cameras, and image display means for displaying the stereo images. A recording means for recording the images captured by the plurality of cameras as image data; a contour extracting means for extracting a occluding contour and a parallax discontinuity boundary line of the image data; and the occluding contour and the parallax extracted. So that it does not intersect the discontinuity boundary line,
An image dividing unit that divides the image data into a plurality of regions having a predetermined size, a parallax extracting unit that extracts the parallax of the image data for each of the plurality of regions, and a parallax extracted for each of the plurality of regions. Based on the discrimination means for discriminating between the binocular vision area and the monocular vision area in the image data, and the arrangement of the camera at the time of shooting and the arrangement of the reproduction optical system at the time of reproduction, the parallax in the binocular vision area is reproduced. And an image generation unit for converting the parallax according to the arrangement of the optical system to generate a stereo image and outputting the stereo image to the image display unit.

In the above-mentioned image recording / reproducing apparatus, the image dividing means is characterized in that the image data is divided so as to be equal to or smaller than the minimum viewing angle with which the human can recognize the depth change of the image.

Further, the image dividing means has a viewing angle of 10 for the image data with respect to the baseline length direction (horizontal direction) between the cameras.
It is characterized in that it is divided into a plurality of areas each having a size equal to or smaller than a minute.

Further, in the above image recording / reproducing apparatus, a first measuring means for measuring the arrangement of the camera at the time of photographing and a second measuring means for measuring the arrangement of the reproducing optical system at the time of reproducing are provided, and an image is generated. The means is characterized by generating a stereo image based on the results measured by the first measuring means and the second measuring means.

[0018]

With the above construction, the images taken by the two cameras are divided into a plurality of regions based on the disparity discontinuity contours, and the parallax of each image is obtained for each of the divided regions. The binocular viewing area and the monocular viewing area in each image are determined based on the obtained parallax.

The parallax obtained in the binocular vision area is converted into a parallax suitable for the arrangement of the reproducing optical system from the camera arrangement at the time of photographing and the arrangement of the reproducing optical system at the time of reproducing to generate a stereo image. It

That is, in the image recording / reproducing apparatus of the present invention,
Unlike the conventional image recording / reproducing apparatus, the parallax at each point of the object is not obtained one by one, but the parallax is obtained for each area having a predetermined size.

In the above-mentioned image division, the divided area is divided based on the disparity discontinuous contour, and the area is divided so as not to intersect the disparity discontinuity contour. As a result, the correspondence between the two images captured by the two cameras is performed for each of the divided areas,
Unlike in the past, the disparity discontinuity contour does not correspond to each image.

When the image is divided as described above, the human stereoscopic perception characteristics are taken into consideration. Considering human stereoscopic characteristics, the minimum viewing angle at which parallax between two parallel objects can be recognized is 10 minutes. In the present invention, since image division is performed in a plurality of regions having a viewing angle of 10 minutes or less with respect to the base line length direction (horizontal direction), parallax is not unnecessarily finely obtained.

Further, in the above-mentioned image recording / reproducing apparatus,
Since the arrangement of the camera at the time of shooting and the arrangement of the reproducing optical system at the time of reproduction are measured by the first and second measuring means each time, even if the arrangement of the camera and the arrangement of the reproducing optical system are changed, the reproduction optical system is changed. A stereo image suitable for the system is generated. When the camera arrangement and the reproduction optical system arrangement are given as information in advance, it is not necessary to measure each arrangement.

[0024]

Embodiments of the present invention will now be described with reference to the drawings.

FIG. 1 is a schematic configuration diagram of an image recording / reproducing apparatus of a first embodiment of the present invention.

The image recording / reproducing apparatus of this embodiment comprises two cameras 1L and 1R provided on a stage 2 and an image processing section 4 for extracting a parallax distribution of each image photographed by each of the cameras 1L and 1R. And an image display unit 5 for displaying the result processed by the image processing unit 4.

The image display unit 5 is a head-mounted display (HMD) using a liquid crystal panel, time-division shutter glasses or the like, or a projection type image display device using a lenticular lens screen and a liquid crystal panel. Used.

Further, each of the cameras 1L and 1R is provided with memories 3L and 3R, respectively, and images photographed by the respective cameras are recorded as image data in the memories 3L and 3R, respectively. The image data recorded here is output from the memories 3L and 3R to the image processing unit 4.

The image processing unit 4 converts a stereoscopic image captured by the cameras 1L and 1R into a stereoscopic image having a disposition different from the disposition of the cameras 1L and 1R (here, the reproduction optical system of the image display unit 5 is arranged). A contour extracting unit 4a, an image dividing unit 4b, a parallax extracting unit 4c, a discriminating unit 4d, and an image generating unit 4e. Each component has the following features.

The contour extracting section 4a is provided with the cameras 1L and 1L.
A portion in which an occluding contour or disparity changes discontinuously (hereinafter referred to as disparity discontinuous boundary line) in an image captured by R is extracted. The image dividing unit 4b divides each image captured by each of the cameras 1L and 1R into a plurality of regions in the shape of a rectangle or the like based on the occluding contour or the disparity discontinuous boundary line extracted by the contour extracting unit 4a. The parallax extraction unit 4c includes the image division unit 4c.
The parallax is extracted for each area divided in b. The determination unit 4d is a monocular area of an image captured by each of the cameras 1L and 1R (an area captured by only one camera).
And the binocular viewing area (area captured by both cameras). The image generation unit 4e generates a stereo image to be incident on the left and right eyes from the parallax distribution in the images captured by the cameras 1L and 1R and the arrangement of the optical system at the time of capturing and reproducing.

Further, in the image recording / reproducing apparatus, the stage 2 is provided with the camera placement measurement storage means 6, and the placement of the cameras at the time of shooting is measured by the camera placement measurement storage means 6, and the measurement result is stored in the camera. It is output from the arrangement measurement storage means 6 to the image generation unit 4e.

Further, the image display section 5 is provided with a reproduction optical system arrangement measuring means 7, and the arrangement of the optical system at the time of reproduction (display optical arrangement, macroscopic optical arrangement, etc.) is the reproduction optical system arrangement measuring means 7. And the measurement result is output from the reproduction optical system arrangement measuring unit 7 to the image generating unit 4e.

In the present embodiment, the occluded contour is a binocular vision region in which one region separated by a portion in which disparity changes discontinuously in an image captured by two cameras,
The case where the other area is a monocular area. In addition, a binocular viewing area in both areas is called a disparity discontinuity contour.

In the above image recording / reproducing apparatus, the left and right images photographed by the cameras 1L and 1R are each divided into a plurality of areas of a predetermined size, and each of the divided areas has a masking contour or no parallax. Since it is provided so as not to intersect the continuous portion contour, when forming a stereo image, the correspondence between the left and right images is not performed by the shield contour or the disparity discontinuous portion contour.

Next, the operation of the above-mentioned image recording / reproducing apparatus will be described in detail.

The images photographed by the left and right cameras 1L and 1R are input to the memories 3L and 3R, respectively, and temporarily recorded as image data. At the same time, the camera 1L at the time of shooting,
The 1R layout is measured and stored by the camera layout measurement storage means. When the image data is recorded in the memories 3L and 3R, the recorded image data is output to the image processing unit 4.

Here, the image processing in the image processing section 4 will be described in detail.

When the image data recorded in the memories 3L and 3R are input to the image processing unit 4, respectively, the image processing unit 4
Then, first, the input image data is passed to the contour extraction unit 4a, and the parallax discontinuous contours of the input left and right images are extracted.

FIG. 2 shows parallax discontinuity contours of images photographed by the cameras 1L and 1R. FIG. 2A shows original images photographed by the left and right cameras (rectangular solid), FIG. 2B. FIG. 2 shows the disparity discontinuity contour extracted from FIG.

The disparity discontinuity contour here is shown in FIG.
In the case where a rectangular parallelepiped is photographed as the original image as shown in (a), the contour of the disparity discontinuity which is the disparity discontinuity in the plane photographed by both the left and right cameras, and only one camera. It is a general term for a shielding contour that is a disparity discontinuity in a flat plane.

As a method of extracting the disparity discontinuity contours, isoluminance lines are detected from a grayscale image, and epipolar conditions (a point on one image corresponds to a point on the other image is a half value of the same scanning line). Existing on the line) to detect the distribution of isoluminance points on the same epipolar line or the inclination of the isoluminance line (Japanese Patent Laid-Open No. 5-341).
17).

When the parallax discontinuity contour shown in FIG. 2B is extracted from the original image shown in FIG. 2A as described above,
The image is divided by the image dividing unit 4b into a plurality of areas having a rectangular shape or the like. The image is divided at this time by dividing the inside of the area separated by the disparity discontinuous contour as shown in FIG. 2B into a plurality of areas of a predetermined size. It is done so that the region does not intersect the disparity discontinuity contour.

Also, the size of the divided area takes into consideration the stereoscopic perception characteristics of human beings, and for example, the cameras 1L, 1
The base line direction (or horizontal direction) of R is converted into a viewing angle of 1
The order is 0 minutes or less, and its vertical direction (direction orthogonal to the above-mentioned baseline direction in the screen on which the image is displayed) is not particularly limited as long as it does not protrude the disparity discontinuous contour.

That is, a human has a horizontal viewing angle of 10
Since the difference in distance between the parallel objects can be recognized when there is more than a minute, the size of the divided area in the horizontal direction is set to a maximum of 10 minutes in terms of the viewing angle. If the area is set larger than this 10 minutes,
The parallax (lateral displacement) described below is unnaturally extracted for each large area, and the resulting stereoscopic image is unnatural.

Further, since the respective cameras at the time of photographing are synchronized in the horizontal direction in the setting, there is almost no shift between the left and right images in the photographed image with respect to the vertical direction. The size does not have to be particularly limited.

When the image is divided as described above, the parallax extraction unit 4c determines the correspondence between the respective regions of the divided images between the left and right images, and if there is a corresponding region, the parallax (lateral displacement) is obtained. To be In this embodiment, for convenience, the parallax is the value of the right image with respect to the left image.

When obtaining the parallax, the intensity discontinuity portions (boundary lines) in the left and right images are respectively detected, and the correlation between the areas separated by the detected boundary lines and having a certain size or less is calculated between the left and right images. How to do (Toh and Forres
t, Third International Conference on computer Visio
n, 1990, PP.126-132) is used. In this method, it is utilized that the regions in the binocular vision region have a high correlation and the regions in the monocular vision region have a low correlation.

As a method for obtaining the parallax, a method using a region-based method such as the above-mentioned correlation method or least square method, and a method using a feature-based method that extracts edges in each area and performs matching. There is (Dhond & Agga
rwal, IEEE Trans on Systems, Man, and Cybernetics, Vo
l.19,1489-1510). When the former method is used, the above-mentioned divided area or an area obtained by further dividing the area is used as the area for calculating the correlation. Also,
When the latter method is used, the above matching is not performed when an edge or a feature overlaps the disparity discontinuous contour described above.

Further, the parallax extraction unit 4c calculates the average parallax of the area divided by the parallax discontinuity contours if necessary.
That is, in parallax extraction, in the area-based method, when divided into areas smaller than the area for calculating the average parallax, or in the feature-based method, after calculating the parallax for each edge, the area where no edge exists When the parallax is obtained by interpolating, the average parallax is calculated when there are a plurality of edges in one disparity discontinuous contour.

The parallax value for each disparity discontinuous contour area obtained as described above corresponds to the average of the distances from the camera to the object corresponding to that area.

When the parallax extraction unit 4c does not find the parallax as a result of the processing, for example, when the corresponding point or the corresponding region does not exist, the corresponding region is the monocular viewing region, and therefore the determination unit 4c determines the region. Each pixel inside is labeled to that effect. In the labeling, for example, a label array L (i, j) (i, j is an index representing the position of a pixel) is defined, and the values are respectively set in the monocular viewing area of the left camera and the monocular viewing area of the right camera. Is set to +1, −1. In this case, the value is set to L (i, j) = 0 for the binocular vision area. The above-mentioned correlation can be used to distinguish between the binocular viewing area and the monocular viewing area.

As described above, when the left and right images are divided into a plurality of regions, the correspondence and parallax of the divided regions are obtained, and the binocular vision region and the monocular vision region are distinguished, The image generation unit 4e converts the parallax obtained in the binocular vision area into the parallax according to the reproduction optical system of the image display unit 5 to generate an image.

That is, in the image generation unit 4e, the parallax of the binocular vision area detected as described above is stored in the camera placement measurement storage unit 6 and the camera placement at the time of photographing and the reproduction optical system placement measurement unit 7 are stored. The calculation processing is performed based on the arrangement of the optical system of the image display unit 5 measured in 1., and the parallax is converted into the parallax according to the reproduction optical system of the image display unit 5.

When the image generation unit 4e converts the parallax of the binocular vision region into the parallax corresponding to the reproduction optical system, the image of the binocular vision region and the image of the monocular vision region described above are imaged from the image processing unit 4. It is output to the display unit 5. Then, the image display unit 5 displays a stereo image that matches the arrangement of the reproduction optical system.

As described above, the images photographed by the left and right cameras are subjected to image processing by the respective constituent parts of the image processing section 4, and are arranged in a camera different from the stereoscopic image obtained by the camera arrangement at the time of photographing. The stereoscopic image is output and displayed on the image display unit 5.

Next, the parallax conversion processing in the binocular vision area by the image generation unit 4e described above is performed when the stereoscopic image displayed on the image display unit 5 is made incident on the human eye to form a stereo image. Description will be made based on the principle of parallax information conversion in the human eye.

FIG. 3 is a conceptual diagram showing the principle of parallax information conversion.

In the conceptual diagram of FIG. 3, the base line length L is on the straight line b.
The planes intersecting with two points having a distance of (the baseline length of the camera system at the time of shooting) intersect the camera sensor surfaces S _L and S _R at the time of shooting
Further, a plane intersecting with two points having a base line length L ′ (base line length of the optical system at the time of reproduction) on both sides of the plane intersects the display surface (or retina of the eyeball) S _L ′ at the time of reproduction, respectively. , S _R '. Further, a point where the perpendicular lines l _L and l _{R at} the intersection of the planes S _L and S _R and the straight line b intersect with each other is Q, and similarly, the perpendicular lines l _L 'and S _R ' of the respective planes S _L 'and S _R '
_Let Q ′ be the point where l _R ′ intersect with each other.

Further, a point P on an object in the real space is a point
It is provided on the extension of the line drawn from Q'to the point Q, and the point P
Each plane S of_L, S_R, S_L’, S_R’
Each P _L, P_R, P_L’, P_R’As each plane S_L, S_R, S
_L’, S_R'On the top. Each image forming point P provided_L, P_R，
P_L’, P_R′ And the point P, and the above-mentioned perpendicular lines l_L，
l_R, L_L’, L_R‘_L, Q_R，
Q_L’、 Q_R’ Where point Q_L, Q_RIs for shooting
Represents the center of the point, point Q_L’、 Q_R’In the optical system during playback
Indicates the center of the lens.

In the following description, in order to simplify the formula,
P is the point Q_L, Q_REquidistant from point Q _L, Q_RIs the plane S_L，
S_RAssume that they are equidistant from u.

Here, assuming that the angle of convergence of the camera at the time of shooting is 2φ in FIG. 3, the distance χ from the point Q to the point P is represented by the following equation in the triangle ΔQ _L PQ.

[0062] χ / Sin (θ) = q / Sin (π-φ-θ) (1) where q is a represents the distance from the point Q to the point _{Q L, q = (L /} 2 ) × (1 / Sin (φ) −u) (2) Therefore, from the above equations (1) and (2),

[0063]

[Equation 1] Is obtained. Further, tan (theta) is _{tan (θ) = | P L} | / u = | P R | distance R from it / u in expressed, from the plane containing the base line b to point Q, R = L / 2 × Cot (φ) Where | P _L |, | P _L '|, | P _R |,
| P _R '| is a straight line b with each plane S _L , S _L ', S _R , S _R '
Represents the distance from each intersection point with and each point P _L , P _L ′, P _R , P _R ′.

Similarly, the convergence angle of the optical system during reproduction is set to 2
If φ ′, the distance χ ′ from the point Q ′ to the point P can be expressed by the following equation.

[0065]

[Equation 2] The road separation R ′ from the plane including the base line b to the point Q ′ is R ′ = L ′ / 2 × Cot (φ ′). Here, since R−χ = R′−χ ′, when solving for P _L ′,

[0066]

[Equation 3] Next to

[0067]

[Equation 4] Is obtained. Where A is the function f (L, L ', φ,
φ ') represents the parameter given by.

As can be seen from the above, u and φ
Is known and P _L is measurable, P when the baseline length changes from L to L ′ and the convergence angle changes from 2φ to 2φ ′
_L ′ can be calculated as a function f (L, L ′, φ, φ ′).

That is, in the image recording / reproducing apparatus of the present embodiment, when the image is reproduced, L and φ are calculated from the information of the camera arrangement at the time of photographing stored in the camera arrangement measurement storage means 6 by the image generating unit 4e. Is read out, and L ′ and φ ′ are detected as the optical arrangement information at the time of reproduction from the reproduction optical system arrangement measuring means 7, and P _L is P _L ′ according to the above equations (1) and (2). Is converted into an image and image generation is performed.

In the above case, the parallax is generally given by (P _L -P _R ), so that the parallax also changes with the change in the base line length (P _L '
-P _R '). Further, the point P as shown in FIG. 3, an optical system with respect to a plane including the Q is when the symmetry parallax changes to 2P _L 'from 2P _L.

Even when the point P is not equidistant from the points O _L and O _R , that is, when | P _L | ≠ | P _R |, it is possible to calculate from P _R using the equations (1) and (2). P _R 'can be calculated.

In this embodiment, the arrangement of the camera and the like at the time of shooting and the arrangement of the reproduction optical system at the time of reproduction are measured by the camera arrangement measurement storage means 6 and the reproduction optical system arrangement measurement storage means 7, respectively. It is not necessary to measure when the optical arrangements at the time of shooting and at the time of reproduction are known in advance.

Further, the shape of the area when dividing the binocular viewing area delimited by the boundary line of the occluding contour or the disparity discontinuity is not limited to the rectangle, and the shape of the occluding contour or the disparity discontinuity is not limited. Any shape may be used as long as it can be divided along the boundary line. However, the size in the horizontal direction shall not exceed the order of 10 minutes as described above, and if it exceeds the order of 10 minutes, the divided area is subdivided so that it does not exceed the order of 10 minutes. To do.

FIG. 4 is a schematic block diagram of an image recording / reproducing apparatus of the second embodiment of the present invention.

[0075] Image recording and reproducing apparatus of this embodiment, the image recording and reproducing apparatus shown in FIG. 1, the camera 1 _L, 1 respectively in _R zoom provided mechanisms 8L, 8R, the area sensor 9
L, 9R and aperture plates 11L, 11R, and arithmetic processing units 10L, 10R for calculating the blur amount of the image captured on the imaging surface based on the detection results of the area sensors 9L, 9R are provided. This is a device having the same configuration as the image recording / reproducing device of the first embodiment. In the image recording / reproducing device,
The part of the image captured on the imaging surface with a large amount of blur is a position that is greatly deviated from the focal length of the camera,
It is determined that the parallax changes discontinuously.

[0076] In the image recording apparatus, the area sensor 9 L, 9R are provided on the camera 1 _L, 1 _R imaging surface of. The image of the object taken by the cameras 1 _L and 1 _R is passed through the zoom mechanisms 8L and 8R to the area sensor 9
Images are taken on L and 9R. The arithmetic processing unit 10 calculates the blur amounts of the images captured on the area sensors 9L and 9R, respectively. The aperture plates 11L and 11R determine the size of the image captured on the imaging surface, and have holes of a predetermined size provided in the front part of the imaging system, for example, the front surface of the lens system.

Regarding the same constituent parts as those of the image recording / reproducing apparatus of the first embodiment, the operations of the constituent parts also operate in the same manner.

Next, the operation of the image recording / reproducing apparatus will be described.

In the above image recording / reproducing apparatus, the camera 1 _L ,
The image of the object captured by 1 _R is the zoom mechanism 8
Images are taken on the area sensors 9L and 9R via the L and 8R. The images captured at this time are adjusted by the zoom mechanisms 8L and 8R so that the size of the images in the horizontal direction (here, referred to as the base line length direction) is, for example, in the order of tens of minutes when converted into a viewing angle. It As a result, the size of the image picked up on the area sensors 9L and 9R is limited to be smaller than the size of 10 areas in which the horizontal size is about 10 minutes.

Here, the openings of the aperture plates 11L and 11R
The size of (Aperture) depends on the size of the divided area.
To set the aperture properly and use each aperture when shooting an object.
Mela 1 _L, 1_RScan with a predetermined step width over the entire angle of view of
To do. Then, the horizontal direction is
Images segmented on the order of 10 minutes are captured.
As described above, the image in the horizontal direction of 10 minutes is that.
When each is imaged, the arithmetic processing units 10L and 10R show areas.
The blur amount of the image can be adjusted through the sensors 11L and 11R.
Calculate each.

The blur amount C calculated by the arithmetic processing unit 10 is obtained from the following equation.

[0082]

[Equation 5] Here, A is the aperture diameter of the aperture plate 11, f is the focal length, Z is the distance from the cameras 1 _L and 1 _R to the object, Z ′ is the distance to the focused object, and C 0 (A, f ) Is a function represented by A and f, and represents a luminous flux diameter at the time of focusing.

As can be seen from the above equation (3), the blur amount C is an approximate standard for the distance to one point of the subject or a minute area, assuming that other parameters are fixed.

When the blur amount is calculated for each step, the change amount of the blur amount (for example, the absolute value of the difference between the calculated front and rear blur amounts or the ratio of the calculated front and rear blur amounts) is obtained. Then, a portion whose value is larger than a predetermined reference value is detected as a candidate for the disparity discontinuous contour portion.

The images picked up by the cameras 1L and 1R are
After the parallax discontinuous contour portion candidates in each image are detected as described above, the information is stored in the memory 3 as image data together with the information of the extracted parallax discontinuous contour portion candidates.

When the image data stored in the memory 3 is input to the image processing section 4, the contour extracting section 4a first
An intensity discontinuity portion and a discontinuity portion such as color and intensity dispersion are detected based on the region detected as a candidate for the disparity discontinuity contour portion.

When the contour extracting unit 4a detects a discontinuous portion, the input image is subdivided by the image dividing unit 4b into a plurality of regions having a predetermined size in consideration of the stereoscopic perception characteristic. The subdivision of the image at this time is also performed in the same manner as in the first embodiment so that each area does not intersect the discontinuous portion. In addition, when a plurality of discontinuities exist in the subdivided area, the area is subdivided into small areas so as not to intersect the discontinuities.

When the image is subdivided into regions, the first
Similar to the case of the embodiment, the parallax extraction unit 4c obtains the parallax for each of the divided regions.

As a result of obtaining the parallax, if the parallax difference between two small areas adjacent to each other with the above-mentioned discontinuity as a boundary is less than a predetermined reference value, the discontinuity is considered as a disparity discontinuity contour candidate. And the area is subdivided. Thus, the parallax discontinuity contour can be accurately specified by repeatedly extracting the parallax discontinuity contour from the candidate of the parallax discontinuity contour and performing image division. As a result, the average parallax of the area delimited by the specified parallax discontinuity contour can be accurately obtained.

When the disparity discontinuity contour is extracted as described above, the image processing section 4 performs the same processing as in the first embodiment to generate a desired stereo image.
The stereo image processed by the image processing unit 4 is the image display unit 5.
Is output to and displayed.

FIG. 5 is a schematic block diagram of an image recording / reproducing apparatus of the third embodiment of the present invention.

The image recording / reproducing apparatus of the present embodiment is for separately incorporating an image taken by each camera into an image generated by computer graphics or an actual image for stereoscopic display, and has the following configuration. ing.

The image recording / reproducing apparatus has two image generating sections for generating different images and a synthesizing section for synthesizing the two generated images.

One of the image generators is input from the two cameras 201L and 201R provided on the stage 202, a memory 203 for storing images taken by the cameras 201L and 201R as image data, and a memory 203. Image processing unit 204 for generating a stereoscopic image from image data
It consists of and.

The other image generator is the camera 201.
A compound eye imaging unit 212 provided separately from L and 201R,
CPU2 for generating desired computer graphics
13 and a memory 21 for storing these generated images
It is composed of 4 and.

Further, the synthesizing section is composed of the image synthesizing section 215, and the images generated by the two image generating sections are respectively synthesized by the image synthesizing section 215. The image combined here is output to the image display unit 205 and displayed as an image.

The image processing unit 204 is provided with a camera arrangement measurement storage unit 206 for measuring and storing the camera arrangement of the cameras 201L and 201R at the time of photographing, and an arrangement of the reproduction optical system during reproduction of the image display unit 205. Reproduction optical system arrangement measurement storage means 207 for measuring and storing is connected, and as in the case of the first and second embodiments, the cameras 201L,
An image having a camera arrangement different from the camera arrangement at the time of photographing is generated based on the image photographed at 201R.

Next, regarding the operation of the above image recording / reproducing apparatus, an example is taken in which a camera 1L, 1R is used to pick up a human image in a place where the background is evenly photographed, and the human image is combined with another image. To explain.

Images taken by the cameras 1L and 1R, here, images of a person imaged in a place where the background is evenly recorded, are recorded in the memory 203 as image data. At the same time, the camera layout at the time of shooting is stored in the camera layout measurement storage unit 206.

When the image data recorded from the memory 203 is output to the image processing unit 204, the image processing unit 204 obtains the average parallax of the image area of the person by the method described in the first embodiment. From the obtained average parallax and the camera placement obtained by the camera placement measurement storage unit 206, an approximate value of the distance from the camera to the human image is obtained using the principle of triangulation.

When the approximate value of the distance from the camera to the human image is obtained as described above, the image processing unit 204 determines a camera arrangement different from that at the time of image pickup based on the obtained distance and the image size of the human image. An image is generated by giving an appropriate parallax and an image size to the image. The generated image is output to the image combining unit 215.

On the other hand, in addition to the person image, the memory 214 stores an image captured by the compound-eye image capturing section 212 or computer graphics created by the CPU 213, and these images are stored as necessary. It is output to the combining unit 215.

When the images are output from the image processing unit 204 and the memory 214 to the image synthesizing unit 215, the image synthesizing unit 215 projects or synthesizes the respective input images. As a result, the two images are fused and displayed on the image display unit 205.

In this embodiment, the projection means image processing such as replacing the pixels in the area where the target image is projected with the brightness level of the background image with the brightness level of the target image. The brightness level of the target image may be corrected and added to the background image. Further, the images of all binocular vision regions divided by the occluding contour or the disparity discontinuous contour including the background image are average parallaxes obtained by the image processing unit 204, or about 10 minutes in each region in the horizontal direction, or An average parallax for each area when divided into areas of a size smaller than that may be given and displayed.

Furthermore, the camera arrangement measurement storage means 206 and the reproduction optical system arrangement measurement storage means 207 are used to measure the arrangement of each optical system at the time of photographing and reproduction, and a stereoscopic image reflecting the difference between the two arrangements is measured. In the case of generating, the average parallax of the binocular vision area divided by the occlusion contour or the disparity discontinuity contour is obtained and displayed for each area by the image processing unit 204 as in the case of the first embodiment. Is also good.

Even if each component of the image recording / reproducing apparatus of the present invention is a device integrally configured on the stage, the memory unit, the image processing unit and the image display unit are provided separately from the camera. The device may have a different configuration.

Further, in the image recording / reproducing apparatus of the present invention, in each of the above-described embodiments, the two cameras are installed horizontally and their horizontal synchronization is preliminarily obtained.
The viewing angle is set to 10 minutes or less in consideration of the stereoscopic perception of human being with respect to the base line length direction, and there is no limitation as long as it does not intersect the parallax discontinuity contour with respect to the direction perpendicular to the base line length direction. However, when the two cameras are not installed horizontally, the direction perpendicular to the baseline length direction may be limited to a predetermined size.

[0108]

Since the present invention is constructed as described above, it has the following effects.

According to the first aspect of the present invention, the extraction of the corresponding points in the images captured by the left and right cameras is performed for each area delimited by the occlusion contour or the boundary line of the disparity discontinuity. It is possible to prevent extraction of corresponding points in each image in the discontinuous area, and it is possible to form a good image.

Further, the area delimited by the boundary of the occlusion contour or the disparity discontinuity is divided into a plurality of areas having a predetermined size in consideration of human stereoscopic perception characteristics, and each of the divided areas is divided. Since the parallax is extracted, the calculation time in the parallax extraction processing and the image generation processing can be shortened without impairing the stereoscopic effect.

According to the second aspect of the present invention, since the division of the image data has a size that takes into consideration the stereoscopic perception characteristics of human beings, in addition to the above respective effects, parallax extraction is made appropriate, and arithmetic processing is performed. There is an effect that it can be performed at high speed.

According to the third aspect, the parallax which is close to the stereoscopic perception of human beings is extracted, and the above effect can be further improved.

According to the fourth aspect of the present invention, there is an effect that an image recording / reproducing apparatus that achieves each of the above effects can be realized.

[Brief description of drawings]

FIG. 1 is a schematic configuration diagram of an image recording / reproducing apparatus according to a first embodiment of the present invention.

2A and 2B show parallax discontinuity contours of images captured by the cameras 1L and 1R, where FIG. 2A is an original image captured by the left and right cameras, and FIG. 2B is a parallax extracted from FIG. Discontinuous contours.

FIG. 3 is a conceptual diagram showing the principle of parallax information conversion.

FIG. 4 is a schematic configuration diagram of an image recording / reproducing apparatus according to a second embodiment of the present invention.

FIG. 5 is a schematic configuration diagram of an image recording / reproducing apparatus according to a third embodiment of the present invention.

[Explanation of symbols]

 1L, 1R, 201L, 201R camera 2, 202 stage 3, 3L, 3R, 203, 214 memory 4, 204 image processing unit 4a contour extraction unit 4b image division unit 4c parallax extraction unit 4d discrimination unit 4e image generation unit 5, 205 Image display unit 6,206 Camera placement measurement storage unit 7,207 Reproduction optical system placement measurement storage unit 8L, 8R Zoom mechanism 9L, 9R Area sensor 10L, 10R Arithmetic processing unit 11L, 11R Aperture plate 212 Compound eye imaging unit 213 CPU 215 image Synthesis Department

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical display location G06T 11/00 H04N 13/00 9365-5L G06F 15/72

Claims (4)

[Claims] 1. An image recording / reproducing apparatus for forming various stereo images from images taken by using a plurality of cameras, and image display means for displaying the stereo images, and images taken by the plurality of cameras. Recording means for recording as data, contour extracting means for extracting the occlusion contour and parallax discontinuity boundary line of the image data, the image data so as not to intersect the extracted occlusion contour and parallax discontinuity boundary line An image dividing unit that divides the image into a plurality of regions having a predetermined size, a parallax extracting unit that extracts the parallax of the image data for each of the plurality of regions, and based on the parallax extracted for each of the plurality of regions, From the discrimination means for discriminating between the binocular vision area and the monocular vision area in the image data, the arrangement of the camera at the time of photographing and the arrangement of the reproduction optical system at the time of reproduction, Converts the definitive parallax disparity corresponding to the arrangement of the reproducing optical system to generate a stereo image, image recording and reproducing apparatus characterized by having an image generating means for outputting the stereo image on the image display means. 2. The image recording / reproducing apparatus according to claim 1, wherein the image dividing unit divides the image data so as to be equal to or less than the minimum viewing angle by which a human can recognize a depth change of an image. An image recording / reproducing apparatus characterized by. 3. The image recording / reproducing apparatus according to claim 2, wherein the image dividing means divides the image data into a plurality of areas having a viewing angle of 10 minutes or less with respect to a baseline length direction between the cameras. An image recording / reproducing apparatus characterized by division. 4. The image recording / reproducing apparatus according to claim 1, wherein the first measuring means for measuring the arrangement of the camera at the time of photographing and the arrangement of the reproducing optical system at the time of reproducing are measured. The image recording / reproducing apparatus is characterized in that a second measuring means is provided, and the image generating means generates a stereo image based on the results measured by the first measuring means and the second measuring means.