JP2000102040A - Electronic stereo camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To separate a background from an object regardless of a simple configuration by referencing both images photographed by a 1st image pickup means and a 2nd image pickup means each having an image forming optical system and whose photographing directions are nearly the same where the number of valid pixels of a solid-state image pickup element of the 2nd image pickup means is less than that of the 1st image pickup means so as to detect image depth information of the 1st image pickup means. SOLUTION: A stereo digital camera 1 consists of main and sub lenses 4, 5 respectively having optical axes 6, 7 directing nearly the same photographing direction and an image pickup element 2 and an image pickup element 3 whose pixel number is less than that of the image pickup element 2 and is controlled by a controller 11. The image pickup element 3 detects an area where a difference of an image from that of a desired area in pixels photographed by the image pickup element 2 is least from an image photographed by the image pickup element 3 and depth information of the desired area is detected based on a deviation in positions from both the areas. The image pickup element 3 separates each area into an area corresponding to a background in the image and an area corresponding to a foreground in the image based on the depth information. Thus the object is separated from the background and the foreground with high quality at a low cost and a three- dimensional image can be obtained.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an electronic stereo camera, and more particularly to an electronic stereo camera using two or more lenses and a solid-state image sensor, and capable of separating an object and a background from a captured image. The present invention relates to a digital still camera, a digital video camera capable of separating an object and a background from a captured image, and an electronic stereo camera applicable to image processing.

[0002]

2. Description of the Related Art Household personal computers (PCs)
As digital cameras become more popular and PCs become more sophisticated, digital cameras for personal use are sold in addition to conventional silver halide cameras, and high-definition digital cameras equipped with a charge-coupled device (CCD) exceeding 1 million pixels are sold. Have been.

[0003] This high-definition digital camera is most suitable for enjoyment by displaying an image on a high-definition display device of VGA or higher or a large-format printer of A4 or higher. But,
There is a limit to the increase in the definition of the device as well as the capacity of the memory and the display device.

On the other hand, since users of digital cameras are overwhelmingly PC owners, there are applications in which photographed images are processed and enjoyed. At present, conversion processing for performing color tone, enlargement, and reduction, morphing processing for adjusting the color tone and the like of pixels according to a certain model, and retouching software for performing change and processing in units of pixels have become mainstream.

However, these processes do not use the object information from the image, but use only the color and density information of the image, so that it is necessary to perform the processing in units of pixels, and it takes a lot of time.

If information in the depth direction can be acquired at the time of photographing with a camera, it is possible to automatically separate an object such as a person from the background, and perform complicated image processing and manual work. Object separation (segmentation) can be easily performed without the need.

[0007] If the segmentation can be performed, it is needless to say that the background is exchanged, only the object is processed, or the object is placed in a three-dimensional space, and various kinds of enjoyment are increased. It is also possible to create a three-dimensional wire frame from three-dimensional information.

For this purpose, there are many technical proposals for a camera for acquiring the depth direction. For these, a method of taking a depth direction using moiré or light cutting method using laser light has been proposed.

However, in this case, there are problems that it takes a long time to perform three-dimensional measurement and cannot be taken together with (at the same time as) color information. In addition, there is a problem that the apparatus is expensive and large because a laser beam is used.

The simplest and compatible with a digital camera is a binocular or multi-view stereo camera. In this case, the cost is slightly higher because two solid-state imaging devices such as CCDs are used. However, if the cost of the imaging device is reduced, there is an advantage that a small-sized and instantaneous image can be taken.

Although this stereo camera has been proposed for a long time, recently, a configuration of a stereo camera capable of electrically correcting an optical axis shift amount has been proposed in JP-A-6-273172. I have.

Japanese Patent Application Laid-Open No. 9-73554 discloses that
As a method for restoring a three-dimensional scene from a stereo image, an input image is converted into a distance image by means called a generalized distance image, and a corresponding point is determined by a correspondence search using both the pixel value of the input image and the pixel value of the distance image. Means for doing so have been proposed.

In Japanese Patent Application Laid-Open No. 9-27969,
There has been proposed a method of estimating parallax in the vicinity of an object contour and generating an intermediate image corresponding to an arbitrary viewpoint between two images from one set of stereo images.

[0014]

As shown in the above-mentioned conventional example, a stereo camera shoots an image by using two image pickup devices and shows two left and right display devices, or a stereo of extracted feature points. The depth direction is obtained by the trigonometric method using the matching method.

However, in general, these stereo cameras are made for a stereo viewer or a distance measuring device. In addition, since triangulation measurement by stereo matching has been a conventionally known technique, in the conventional proposal, it is one of the algorithms for acquiring depth information, or when it is viewed from another angle. It has been proposed as a calculation method for estimating images.

However, these conventionally proposed methods are not suitable for obtaining a digital image or processing the image. Actually, it has been found that there are many problems when an image is acquired with a simple twin-lens camera, a normal stereo matching process is implemented, and a field test is performed.

As a result, the requirements required for a stereo camera for obtaining a next-generation digital image can be summarized as follows. -Obtain minimum depth information for each object required to separate the object from the background. The image sensor for obtaining color tone information has a high definition of 800,000 pixels or more and has excellent lens performance.・ Reduce the cost to 1.5 times that of a normal digital camera. -To suppress the increase in power consumption to 20% or less of that of a normal digital camera.・ The memory for taking images is 1.5 times that of a normal digital camera.
Double or less. -As a technique for obtaining the depth direction, do not impose a complicated burden on the operator.

Unless the above problems are solved, it is judged that practical use of a device for separating an object and a background by a stereo camera is difficult. The present invention has been made in view of the above circumstances, and in particular, an electronic stereo camera using two or more lenses and a solid-state imaging device, and an object from an imaged image using a simple structure and a simple method as much as possible. It is an object to provide a digital still camera capable of separating an object and a background, a digital moving image camera capable of separating an object and a background from a captured image, and an electronic stereo camera applicable to image processing. .

[0019]

According to the present invention, in order to solve the above-mentioned problems, (1) a first imaging optical system and a first
A first imaging means having a solid-state imaging device, a second imaging optical system, and a second solid-state imaging device having a smaller number of effective pixels than the first solid-state imaging device. By referring to a second imaging unit that is substantially the same as the imaging direction of the first imaging unit, an image captured by the first imaging unit, and an image captured by the second imaging unit, An electronic stereo camera comprising: a depth detecting unit configured to detect depth information of an image of the first imaging unit.

Further, according to the present invention, in order to solve the above-mentioned problems, (2) the depth detecting means includes an area having a smallest image difference from a desired area in the image picked up by the first image pickup means. (1) is detected from the image captured by the second image capturing means, and the depth information of the desired area is detected from the displacement of the two areas. .

According to the present invention, in order to solve the above-mentioned problems, (3) the depth detecting means further divides an image taken by the first imaging means into regions, and obtains depth information for each region. The electronic stereo camera according to (1) or (2), wherein the electronic stereo camera according to (1) or (2) is characterized in that each area is detected and each area is divided from the depth information into an area corresponding to the background in the image and an area corresponding to the foreground in the image. Is done.

[0022]

Embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) The principle of the present invention will be described with reference to FIGS. 1 and 2 showing a first embodiment.

That is, as shown in FIG. 1, the stereo digital camera 1 of the present invention has two lenses and an image sensor, and the image sensor 2 and the main lens 4 mounted on the main camera have the optical axis 6. The other components include an image sensor 3 mounted on a sub camera and a sub lens 5 having an optical axis 7.

In a normal stereo camera, the completely equivalent 2
Although two lenses and an image sensor are used, when an image of a digital camera becomes high quality, mounting two or more expensive image sensors or lenses having more than one million pixels (two or more) requires a shape, a weight, and a price. It is hard to say that any of them is practical.

Therefore, in the configuration of the present invention, the lens system for mainly taking an image and the lens system for taking a depth direction are changed. The imaging devices 2 and 3 are integrated photoelectric conversion devices such as a CCD, a CMOS sensor, and a CMD, and the lens systems 4 and 5 have one or more lenses, a lens frame, and an actuator.

In order to capture an image, it is desirable that the image pickup device 2 has at least 810,000 pixels, and preferably 2 million pixels or more. In order to use the image sensor 3 for measurement in the depth direction,
It may be smaller than the image sensor 2.

Further, as a basic configuration, it has a storage device 10, a control device 11, an operation device 12, and a flash system 13.
Here, the storage device 10 may be detachable from the camera body.

The lens system 4 may have an automatic focus, an automatic exposure, a date and time printing function, and may have a zoom function. Further, the control device 11 may have a function of communicating with an external computer.

FIG. 2 is a diagram showing the principle of basic twin-lens stereo vision. Here, for simplicity of explanation, it is assumed that two cameras are arranged in parallel on the left and right.

A straight line passing through the lens centers of the left and right cameras is defined as an x-axis, an optical axis direction is defined as a z-axis, and a direction orthogonal to the x-axis and the z-axis is defined as a y-axis. Also, let the base line length of each camera be d
And the focal lengths are f1 and f2, respectively.

At this time, the Object, which is a three-dimensional scene,
Projected images 20, 2 of one point (X, Y, Z) of t on the left and right images
1 PL (XL, YL) 22, PR (XR, YR) 23
The condition of the epipolar of XL> XR YL = YR is satisfied, and XL = (f1 / Z) * (X−d / 2) XR = (f2 / Z) * (X + d / 2) YL = (f1 / Z ) * Y YR = (f2 / Z) * Y Here, if the optical system has the same focal length (f = f1 = f2), Z = fd / (XL−XR) (1)

Here, if the focal lengths f1 and f2 are different, the X component enters Z, and Z = {(f1-f2) * X + (f1 + f2) d / 2} / (XL-XR). 2)

Therefore, it is desirable that the focal lengths f1 and f2 are the same. Here, 0.
Using a 597 cm 2/3 inch CCD, f = 2.2
cm optical system, the minimum measurement distance is 11c
m, the maximum measurement distance is 6600 cm, the maximum measurement accuracy at 11 cm is 0.018 cm, and triangulation with extremely high accuracy is possible.

However, when the focal lengths f1 and f2 are not equal, the following approximate calculation method is used. Equation (2)
After calculating XL and XR, Z1 is calculated by Z1 = {(f1 + f2) d / 2} / (XL-XR).

After obtaining X from the angle of view determined by Z1, f, and XL, these are substituted into equation (2) to obtain Z. Then, the true Z is obtained by repeating the calculation for obtaining X from the angle of view determined by Z, f, and XL.

FIG. 3 shows an actual stereo matching method. In the present invention, since the two pixels have different numbers,
The number of pixels in the feature point extraction region divided for stereo matching is different.

FIGS. 3A and 3B illustrate this state. Here, reference numerals 25 and 26 in FIG. 3 represent each pixel of the image, and reference numerals 23 and 24 are two feature point extraction regions.

The number of pixels entering the feature point extraction area is different from each other, and a method of pattern matching with this different number of pixels will be described below. First, the feature point extraction regions of FIGS. 3A and 3B are overlapped only in the X direction only in a range that satisfies the epi-roller condition, and the difference (actually, the double sum of the differences of each pixel) is obtained. Find the minimum position (coordinates).

Next, the depth information (Z coordinate) is searched for using the equation (1) based on the coordinate shift amount between the two images.
Here, specifically, the screen obtained from the image sensor 2 is divided into, for example, 6 × 6 divided areas, and G1, G2,.
... Gm.

This Gi is converted to a corresponding image P from the lens system 4.
1, P2 ... Pi ... Pm and stereo matching,
That is, by performing template matching,
Find a place where the difference is minimal.

As a result, the corresponding points are assumed to be XL and XR, and Z is obtained by using Expression (1) or Expression (2), and is assigned to G1. FIG. 4 shows a method of separating the background and the object from the depth information.

After measuring the depth information of the feature point extraction area by the method of FIG. 3, the value of Z is assigned to each area unit.
Normally, this value is stored in the storage device together with the color tone information of the RBG.

Here, a threshold value Zth for the object and the background is defined, and a region smaller than the Zth is set as the object, and a region larger than Zth is set as the background. Thereafter, the object and the background are stored in separate storage areas, or the background and the object are labeled and stored in the storage area, so that background separation, synthesis with a model, and three-dimensional display are extremely easy. Becomes

(Second Embodiment) FIG. 5 shows a configuration diagram of an optical system according to a second embodiment. In this embodiment,
The areas of the imaging elements 2 and 3 are the same, and only the number of pixels is changed.

As an example, the number of pixels of the main lens system 4 of the image sensor 2 is at least twice the number of pixels of the sub-lens system 5 of the image sensor 3. In this case, the f-number (focal length) can be matched (f = f1 = f2) using the same lens.

Here, the relationship between the f value and the angle of view β is as follows: βx = ArcTan (Xmax / f) βy = ArcTan (Ymax / f)

Here, x and y are considered separately. The maximum value of the angle of view is a diagonal line, and βMax = Sqt (βx * βx + βy * βy).

For example, in the case of a 0.5 inch image sensor,
The length of the diagonal image sensor is 12.7 mm and f = 20.
mm, the maximum value of the angle of view is 32.4 degrees. In addition, when the image is projected on the image sensor, the area of the image sensor exactly matches the angle of view, so that there is a feature that the accuracy of stereo matching is maximized.

For example, as the image pickup device 2, a 0.5 inch CCD of 14.1 million pixels is used, and as the image pickup device 3,
A 0.5 inch 350,000 pixel CCD or a 0.5 inch 350,000 pixel CMOS sensor is used.

In this case, even if the same image sensor is used, 35
A million pixels is much cheaper and a 0.5 inch 3
The 50,000 pixel CMOS sensor has a feature that it can be further inexpensive because an amplifier, an analog / digital converter, and the like can be integrated.

However, since the optical system has the same f-number (focal length) and aperture, the size and cost are the same. The method of actual stereo matching and the method of background separation in this embodiment are the same as in the first embodiment.

(Third Embodiment) FIG. 6 shows a configuration diagram of an optical system according to a third embodiment. This embodiment is a case where the areas of the image sensor 2 and the image sensor 3 are changed.

However, the unit pixel area (pixel pitch) of the image sensor 2 and the image sensor 3 may be the same or different. Further, the f-numbers (focal lengths) of the lenses may be the same or different.

F value (focal length) is the same (f = f1 = f
If 2), stereo matching is easy to process,
The sub-lens system 5 has a smaller angle of view, and can only estimate the depth at the center.

For example, when the image sensor 2 is 0.5 inches and the lens system 4 is 0.25 inches, the f-value (f1 = f
When 2) is 20 mm, the diagonal line of the image sensor 2 is 1
Since the diagonal of the lens system 4 is 2.5 mm and the diagonal line is 6.35 mm, the angle of view of the image sensor 2 is 32.4 degrees, and the angle of view of the lens system 4 is 16.2 degrees.

In order to make these angles of view the same, the fourth
As shown in the embodiment, each f-number (focal length) must be changed. In this case, f1 =
It is necessary to set 20 mm and f2 = 10 mm.

For example, as the image pickup device 2, a 0.5 inch CCD of 14.1 million pixels is used.
A 5 inch 350,000 pixel CCD or a 0.25 inch 350,000 pixel CMOS sensor is used.

Even with the same image sensor, 350,000 pixels are much less expensive, and a 0.25 inch 350,000 pixel CMOS sensor can integrate amplifiers, analog / digital converters, and the like. It has the feature of being cheaper.

However, since the f-number (focal length) and the aperture of the optical system are the same, both the size and the cost are the same. The method of actual stereo matching and the method of background separation in this embodiment are the same as in the first embodiment.

(Fourth Embodiment) FIG. 7 shows a configuration diagram of an optical system according to a fourth embodiment. In this embodiment, one or both of the aperture and the f-number (focal length) of the main lens system 4 and the sub-lens system 5 are changed.

However, the unit pixel area (pixel pitch) of the main lens system 4 and the sub lens system 5 may be the same or different. The area of the image sensor 3 is
Smaller than that of.

Since the lens system 5 is a sub lens system, the aperture and f-number (focal length) are smaller than those of the lens system 4 of the main lens system. If the f-values (focal lengths) are different, stereo matching is performed by the method described in the first embodiment using equation (2).

In this case, even if the image pickup device 2 and the image pickup device 3 have different numbers of pixels and different light receiving areas, their apertures and f-numbers (focal lengths) are changed to make the angle of view the same. Can be.

For example, as the image pickup device 2, a 0.5-inch CCD having a capacity of 1,400,000 pixels is used.
A 5 inch 350,000 pixel CCD or a 0.25 inch 350,000 pixel CMOS sensor is used.

Even with the same image pickup device, 350,000 pixels are much cheaper, and a 0.25 inch CMOS sensor with 350,000 pixels can integrate an amplifier and an analog / digital converter. It has the feature of being inexpensive.

However, since the optical system has the same f-number (focal length) and aperture, the size and cost are the same.
The actual stereo matching method and the background separation method in this embodiment are the same as those in the first embodiment.

(Fifth Embodiment) FIG. 8 shows a fifth embodiment. The basic configuration of the camera in this embodiment is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two imaging elements are changed. However, even if they are the same, this method is effective, and even if it is an ordinary stereo camera, Good.

FIG. 8A shows an image photographed by a stereo camera, in which a person, a tree, and a mountain of a background are photographed. For example, it is assumed that this is photographed by a stereo camera having the configuration shown in FIG. 1 and the stereo matching shown in FIG. 3 is performed to obtain depth (z value) information for each region shown in FIG.

Here, although the threshold value Zth of the depth is used in FIG. 4, it is not realistic to set Zth each time. Therefore, in this embodiment, as shown in FIG. 8B, the Z information of each region is arranged by the value of the depth (z), and a frequency distribution (histogram) is created.

Here, specifically, the screen obtained from the image sensor 2 is divided into, for example, 6 × 6 divided areas, and G1, G
2 ... Gi ... Gm. This Gi is used for the lens system 4
... Pm with the corresponding images P1, P2,... Pi, Pm, the location where the difference is minimized is found.

As a result, the corresponding points are assumed to be XL and XR, Z is obtained by using equation (1) or (2), and these are assigned to Gi. In order to obtain the frequency, Zmax is determined (or measured) from Zmin, and the interval is divided to obtain ΔZ, and Gi in the range of Zi−ΔZ / 2>Z> Zi + ΔZ / 2 around Zi is counted.

The image processing may be performed inside the computer, or may be displayed on a screen of a display attached to the computer. In this example, as shown in FIG. 8B, three peaks A,
B and C, of which peak A is the closest and corresponds to the person in FIG.
Let it be tA.

The next closest peak is B, which corresponds to the tree in FIG. 8 (a). This is referred to as Object B. The farthest peak is peak C, which corresponds to the background mountain in FIG. 8 (a). This background is such that the parallax of XL and XR is almost zero, or if there is no parallax, it is beyond the measurement limit, or it is impossible to measure because there is no feature point.

In this case, since each object and the background can be clearly separated, Zthl and Zth2 can be set between the blocks A, B, and C. The setting of Zth may be performed automatically, or may be performed while the operator looks at the screen.

FIG. 8C shows a finally completed separation result. These can be handled as a group of images and stored in a storage device.

FIG. 9 shows a modification of this embodiment.
Here, if there are two deep objects, A and B
Often do not clearly appear.

FIG. 9A shows a case where the peaks of A and B overlap each other. In this case, a frequency threshold Dth is provided.
This is because if there is a repeating pattern or the texture is unclear, it may cause noise,
Therefore, A and B do not separate.

In the example of FIG. 9B, A and B are separated in the region between A and B because Dth does not reach Dth. (Sixth Embodiment) FIG. 10 shows a sixth embodiment.

The basic configuration of the camera according to this embodiment is the same as that of the first embodiment. In addition,
In the first embodiment, the area and the number of pixels of the two imaging elements are changed. However, even if they are the same, this method is effective, and an ordinary stereo camera may be used.

The method of separating the object from the background is the same as that of the fifth embodiment. This embodiment is an example in which the depth axis is logarithmically drawn when the depth histogram is drawn in the second embodiment.
In this case, the logarithm may be a natural logarithm or a logarithm with a base of 10.

The depth is measured by stereo matching.
Because of triangulation, the distance is proportional to the reciprocal of the distance between the two images. For this reason, the accuracy of measurement is high for short distances and low for long distances. In this case, if the logarithm of the distance is taken, the accuracy and error of the measurement become logarithmically uniform.

More specifically, Zmin and Zmax are assigned to Log (Zmax) and Log (Zmin) on the screen, and δZ = {Log (Zmax) −Log (Zmin)} /
The frequency is calculated and displayed as the number of divisions.

Here, the depth (z value) information of each area unit is obtained for the image (here, a person, a tree, and a mountain of a background are photographed) photographed by the stereo camera shown in FIG. FIG. 10 shows a logarithmic representation of the distance.

This logarithmic processing may be performed inside the computer, or may be displayed on the screen of a display attached to the computer. In particular, when the depth is expressed by a logarithm, it is intuitive and easy to understand, which is advantageous when displaying the depth on the screen of a display attached to the computer.

In this example, there are three peaks A, B, and C according to the depth. Of these peaks, the peak A is the closest and corresponds to the person in FIG.
ectA.

The next closest peak is B, which corresponds to the tree in FIG. 8A, which is referred to as Object B. The farthest peak is peak C, which corresponds to the background in FIG. 8A. They are,
It can be handled as a grouped image and stored in a storage device.

(Seventh Embodiment) FIG. 11 shows a seventh embodiment. The basic configuration of the camera according to this embodiment is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two image pickup elements are changed. However, even if they are the same, this method is effective, and an ordinary stereo camera may be used.

The method of separating the object from the background is the same as that of the fifth embodiment. FIG. 11 shows a case in which the stereo camera 1 has two lens systems 4 and 5 and respective image sensors (not shown) corresponding thereto and a memory 10 as a detachable storage device as the configuration of the stereo camera 1. I have.

Of course, the memory 10 may be a memory built in the main body. On the back of the stereo camera 1, a small display device 14 is provided. In this embodiment, the stereo matching process in the first embodiment, the sorting in the depth direction, the work of separating the object from the background, and the like are performed in each process when drawing a histogram in the camera depth direction. It uses a central processing unit as a control device mounted on a camera.

Of course, the obtained digital video may be stored in a storage device inside or outside the camera. Using the display device 14 mounted on the camera, it is also possible to check the intermediate results and the histograms in FIGS.

FIG. 12 shows a modification of the seventh embodiment. The basic configuration of the camera according to this modification is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two image pickup elements are changed. However, even if they are the same, this method is effective, and an ordinary stereo camera may be used.

The method of separating the object from the background is the same as that of the fifth embodiment. In FIG. 12, the configuration of the stereo camera 1 includes a case where the stereo camera 1 includes two lens systems 4 and 5 and respective image sensors (not shown) corresponding thereto and a memory 10 as a removable storage device. Is shown.

Of course, the memory 10 may be a memory built in the main body. Although a small display device 14 may be provided on the back of the stereo camera 1 as shown in FIG. 11, the stereo camera 1 according to this modification is connected to a computer 15 using a connector and a cable 16 (not shown). ing.

This computer 15 has a display device 14. In this modified example, the stereo matching process in the first embodiment, the sorting in the depth direction, the work of separating the object from the background, and the like are performed by external processing in each process when drawing a histogram in the camera depth direction. A central processing unit as the computer 15 is used.

Of course, the obtained digital video may be stored in a storage device inside or outside the camera. By using the display device 14 of the computer 15, it is also possible to check the intermediate results and the histograms of FIGS.

(Eighth Embodiment) FIG. 13 shows an eighth embodiment. The basic configuration of the camera according to this embodiment is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two image pickup devices are changed. However, even if they are the same, this method is effective. Good.

The method of separating the object from the background is the same as that of the fifth embodiment. FIG.
Shows a specific method of processing in a divided area of stereo matching when using the imaging elements of different pixels shown in FIG.

13A and 13C are regions to be compared, and in this example, the density of the mesh of FIG. 13A is twice as high as that of FIG. 13C. That is, in this embodiment, first, the mesh density of FIG. 13A is converted to a mesh density of the same size as that of FIG. 13C, and in this case, Is 4
Although the density and color information of the two elements are averaged, it goes without saying that information of adjacent pixels may be mixed at a certain ratio.

Next, in this embodiment, stereo matching with the image (c) is performed using the image (b) having the coarse mesh density as described above. In this method, high-speed stereo matching is performed. There is the advantage that it is possible.

FIG. 14 shows a method of performing stereo matching with higher accuracy and obtaining finer final depth information in this embodiment. FIG. 14A shows an area to be subjected to pattern matching. Here, 6 × 6 pixels are shown as an example.

In this embodiment, FIG.
As shown in FIG. 4 (b), the central part of FIG.
This is a method in which depth (Z) information is given only to × 3 pixels.
First, for example, by using the area shown in FIG.
After performing the stereo matching by the method described above, Z information is added only to the area shown in FIG.

FIG. 14 (c) shows the state of FIG.
(A) of FIG. In FIG. 14C, assuming that Z1 and Z2 are locations where the Z information is to be measured, a stereo matching process is performed by taking an area of 6 × 6 pixels around Z1 of 3 × 3 pixels.

For this reason, the area where the Z component is actually measured is performed in units of 3 × 3 pixels, and the area to be used is 6 × 6 pixels. Therefore, an overlapping area as shown in FIG. 14C is necessary.
Here, the central pixel (3 × 3) may be a central single pixel instead of a 3 × 3 pixel.

This method is applicable and effective even when the number of pixels of the two image sensors is the same, but as described in the first to fourth embodiments, the two image sensors have the same effect. Is more effective when the number of pixels is different.

That is, in order to obtain the image shown in FIG. 14A, it is possible to obtain the maximum image information from a small amount of image information by using a high-resolution image sensor as shown in FIG. Becomes possible.

This method is also compatible with the fifth and sixth embodiments described above.
In the case where background separation is performed using the method described in the embodiment, the background separation is performed using finer pixels (in this case, 3 ×
3) can be performed.

(Ninth Embodiment) FIG. 15 shows a ninth embodiment. The basic configuration of the camera according to this embodiment is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two imaging elements are changed. However, even if they are the same, this method is effective. Good.

The method of separating the object from the background is the same as that of the fifth embodiment. FIG.
Shows another method for performing processing in a divided region of stereo matching as shown in FIG. 4, obtaining a depth for each divided unit, and performing object separation.

In the method shown in FIG. 4, since the depth is obtained in coarsely divided units, an object can be separated only in, for example, 6 × 6 coarse units. On the other hand, even if the method according to the seventh embodiment is used, it is 3 × 3, and it takes a lot of time to perform it in one pixel unit at the center.

First, information on the depth method in each region as shown in FIG. 15A is given by the method shown in FIG. 4 (or the method shown in FIG. 14), and the fifth and the fifth methods described above are applied. The object is separated from the background by using the method described in the sixth embodiment.

On the other hand, FIG. 15B shows a screen shot by an image sensor having a larger number of pixels. FIG. 15C shows an image obtained by binarizing the screen shown in FIG. 15B by using the grayscale information and the color information.

In this case, the threshold value for binarization is set individually for a certain pixel area. FIG. 15D shows the area of FIG. 15A, that is, in this case, 6 × 6, and if there is the binary information of FIG. FIG. 3 shows an image in which the object and the background are further refined.

In this manner, as shown in FIG. 15D, the object and the background can be finely separated into pixel units, including the depth information. This method is applicable and effective even when the number of pixels of the two imaging elements is the same. However, as described in the first to fourth embodiments, the number of pixels of the two imaging elements is small. If they are different, it is more effective.

That is, in order to obtain the image shown in FIG. 15B, by using a high-resolution image sensor, it is possible to obtain image information obtained by separating an object with the maximum detail from a small amount of image information. Becomes

(Tenth Embodiment) FIG. 16 shows a tenth embodiment.
An embodiment will be described. The basic configuration of the camera according to this embodiment is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two image pickup devices are changed. However, even if they are the same, this method is effective. Good.

The method of separating the object from the background is the same as that of the fifth embodiment. FIG. 16A shows the result of FIG. 4 in the first embodiment, the result after the object separation according to the fifth and sixth embodiments, and the result of FIG. 15D in the eighth embodiment. Is shown.

In this example, like the edge portion of a person,
Although stereo matching is easy and there is no problem when the z value is determined, the problem is that the skin of a person or a fine pattern has no features, so there is no parallax even if stereo matching is performed. This is the case when it is measured.

Therefore, such an area is determined to be infinite or becomes noise. Therefore, in this embodiment, as shown in FIG. 16B, the areas where the z information is determined are arranged two-dimensionally, and if a closed area is created or a closed area including a frame is created, Among them, a method of determining the same is adopted.

(Eleventh Embodiment) FIG. 17 shows an eleventh embodiment.
An embodiment will be described. The basic configuration of the camera according to this embodiment is the same as that of the first embodiment.

In the first embodiment, the area and the number of pixels of the two image pickup elements are changed. However, even if they are the same, this method is effective, and even if it is an ordinary stereo camera. Good.

The method of separating the object from the background is the same as that of the fifth embodiment. In this embodiment, a method of storing a separated object and a background in a storage device is described.

As an example, FIGS. 17A and 17B are shown. Here, the separated information includes Ob
It is assumed that the object is represented by object A and object B, and the background is represented by C.

The embodiment shown in FIG. 17A is an example in which (1) type, (2) image data, and (3) depth information are stored in the recording area in Object units. here,
(2) Image data is a two-dimensional position (angle of view) and RBG information, and (3) is average depth information or maximum and minimum values of the depth of an object.

The embodiment shown in FIG. 17B is an example in which image data including (1) type and (2) depth information in the unit of object is stored in the recording area. Here, (2)
In the image data, in addition to the two-dimensional position (angle of view) and RBG information, depth information is added in pixel units or region units.

The storage location of this result may be a storage area in the camera, a removable storage device, or a storage device of an external computer connected to the camera body.

The present invention described in the above-described embodiment includes claims 1 to 3 described in claims.
In addition to the above, the invention as shown in the following supplementary notes is included.

Supplementary Note 1 The first imaging optical system and the second
2. The electronic stereo camera according to claim 1, wherein the focal lengths of the imaging optical systems are equal. Supplementary Note 2 In a stereo camera device that captures an image including depth information using two solid-state imaging devices and two imaging lenses, the number of first solid-state imaging devices and the number of second solid-state imaging devices are different. A stereo camera device having a configuration characterized in that: (Corresponding Embodiment of the Invention) A first embodiment will be described. (Operation / Effect) A stereo camera device is configured by using the first and second solid-state imaging devices having different numbers of pixels, and the first and second solid-state imaging devices have a large number of pixels. By configuring a stereo camera device capable of taking depth information including color tone, performing stereo matching using an image taken with a solid-state imaging device having a small number of pixels, and estimating depth information, the cost is low. A digital stereo camera device capable of separating a high-quality object and acquiring a three-dimensional image can be provided.

Appendix 3 The number of elements of the first solid-state imaging device is
The number of elements of the second solid-state imaging device is twice or more, and the video output from the first solid-state imaging device is used as a main video,
3. The stereo camera device according to claim 2, wherein an image output from the second solid-state imaging device is used as an image for estimating depth information. (Corresponding Embodiment of the Invention) A second embodiment will be described. (Operation / Effect) A stereo camera device is configured by using first and second solid-state imaging devices whose numbers of pixels are different from each other by more than twice, and among the first and second solid-state imaging devices, the number of pixels is large. To configure a stereo camera device capable of capturing density information including a color tone with a solid-state imaging device, performing stereo matching using an image captured by the solid-state imaging device having a small number of pixels, and estimating depth information. Accordingly, it is possible to provide a digital stereo camera device capable of separating a high-quality object and acquiring a three-dimensional image at low cost.

(Supplementary note 4) The stereo camera device according to Supplementary note 2 or 3, wherein an effective photosensitive area of the first solid-state imaging device is different from an effective photosensitive area of the second solid-state imaging device. (Corresponding Embodiment of the Invention) A third embodiment will be described. (Operation / Effect) In order to capture a clear and bright image with a large number of pixels, a solid-state image pickup device having a large effective area and a stereo camera device using a solid-state image pickup device having a small number of pixels and an effective area have been constructed. A solid-state imaging device with a large area that captures light and shade information including color tones, performs stereo matching using images taken with a small-area solid-state imaging device, and can estimate depth information. With this configuration, it is possible to provide a digital stereo camera device capable of separating a high-quality object and acquiring a three-dimensional image at low cost.

Supplementary Note 5 A feature is that one of the focal length, the aperture, and the f-number (depth of focus) of the lens system that forms an image on the first solid-state image sensor differs from the lens system that forms an image on the second solid-state image sensor. 4. The stereo camera device according to claim 2 or 3, wherein (Corresponding Embodiment of the Invention) A fourth embodiment will be described. (Operation / Effect) In order to capture a clear and bright image with a large number of pixels, a solid-state image pickup device provided with a lens having a large aperture and / or a large f-number (depth of focus) value; A solid-state imaging device using a solid-state imaging device provided with a lens having a small f-number (depth of focus) value or both is used, and the solid-state imaging device having a large effective area captures density information including a color tone. By constructing a stereo camera device capable of estimating depth information by performing stereo matching using an image captured by an image sensor having a small effective area, low-cost, high-quality object separation and three-dimensional images And a digital stereo camera device capable of acquiring the digital stereo camera.

Supplementary Note 6 The divided area unit of the stereo matching processing for estimating depth information is divided into the first solid-state image sensor and the second solid-state image sensor.
4. The stereo camera device according to claim 2, wherein the solid-state imaging device has the same area in the image plane, and the number of pixels included therein is changed between the first image and the second image. (Corresponding Embodiment of the Invention) A second embodiment will be described. (Operation / Effect) In a stereo camera having a plurality of elements having different numbers of pixels and effective pixel areas, complicated processing is required by performing stereo matching in each element in the same area in the solid-state imaging device. By configuring a stereo camera device capable of estimating depth information without providing a digital stereo camera device capable of low-cost, high-quality object separation and acquisition of a three-dimensional image, it is possible to provide a digital stereo camera device. it can.

Supplementary Note 7 The unit of the divided area in the stereo matching processing for estimating depth information is the
4. The stereo camera device according to claim 2, wherein the angle of view is the same in the image plane of the solid-state imaging device, and the number of pixels included therein is changed between the first image and the second image. (Corresponding Embodiment of the Invention) A fourth embodiment will be described. (Operation / Effect) In a stereo camera having a plurality of elements having different numbers of pixels and effective pixel areas, the stereo matching of each element is performed at the same angle of view in the image plane of the solid-state imaging device, thereby complicating the operation. A digital stereo camera capable of low-cost, high-quality object separation and three-dimensional image acquisition by configuring a stereo camera device capable of estimating depth information without requiring complicated processing An apparatus can be provided.

Supplementary Note 8 In a stereo camera device that captures an image including depth information by using two solid-state imaging devices and two imaging lenses, the image obtained by the first solid-state imaging device is divided into regions. Estimation of depth information by division unit is performed by stereo matching, and a measurement value of the depth information is assigned to the division unit. Then, a histogram of the division region in the depth direction is created, and a lump divided region including the peak is formed. Is recognized as an object existing at a certain depth, and image information obtained by separating one or more objects and a background, division information, and depth information are stored in a storage device. Stereo camera device. (Corresponding Embodiment of the Invention) A fifth embodiment will be described. (Operation / Effect) In a stereo camera, complicated processing is performed by automatically or semi-automatically finding a lump classified by distance by creating a histogram for each area and depth information in the area division after stereo matching. By providing a stereo camera device capable of estimating depth information without the need, a digital stereo camera device capable of low-cost, high-quality object separation and acquisition of a three-dimensional image is provided. can do.

Supplementary Note 9 A histogram is created by using a logarithmic scale for the division in the depth direction, and a set of divided regions including the peak is recognized as an object existing at a certain depth. 9. The stereo camera device according to claim 8, wherein (Corresponding Embodiment of the Invention) A sixth embodiment will be described. (Operation / Effect) When using a stereo camera to automatically or semi-automatically find the lump classified by distance by creating a histogram for the depth information in the area division after stereo matching, By converting to and processing, it is possible to construct a stereo camera device capable of estimating depth information with a natural feeling relative to distance and without requiring complicated processing. It is possible to provide a digital stereo camera device capable of separating a high-quality object and acquiring a three-dimensional image at low cost.

Supplementary Note 10 Images obtained from the two image pickup devices are stored in a storage device built in the main body of the stereo camera or in a removable storage device, and subjected to stereo matching processing, separation of an object from a background, and separation. 10. The stereo camera device according to claim 8 or 9, wherein the storage of the image information, the division information, and the depth information in the storage device is performed using another computer system. (Corresponding Embodiment of the Invention) A seventh embodiment will be described. (Operation / Effect) The video information obtained from the two image sensors is stored in a storage device built in the main body of the stereo camera or in a removable storage device, and is subjected to stereo matching processing and separation and separation of the object and the background. Image information, split information,
By storing the depth information in the storage device by using another computer system, a high computer processing ability can be used. A stereo camera device can be provided.

Supplementary Note 11 The processing of the stereo matching, the separation of the object and the background, and the storage of the information in the storage device are performed by the central processing unit mounted on the stereo camera.
Appendix 8 or 9 wherein image information obtained by separating one or a plurality of objects from the background is stored in a storage device built in the main body of the stereo camera, or depth information is stored in a removable storage device. The stereo camera device according to claim 1. (Corresponding Embodiment of the Invention) A seventh embodiment will be described. (Operation / Effect) By performing stereo matching processing, separating an object from a background, and storing separated image information, division information, and depth information in a storage device by a central processing unit mounted on a stereo camera. Since no external computer is used, it is possible to provide a digital stereo camera device which can perform independent processing, can separate a high-quality object at low cost, and can acquire a three-dimensional image.

Supplementary Note 12 The image obtained by the first solid-state imaging device is divided into regions, and depth information is estimated by stereo matching in the unit of division, and the divided region is divided into regions smaller than the central divided region. 9. The stereo camera device according to claim 8, wherein after assigning the measured depth value, a histogram of the divided area in the depth direction is created to separate the object from the background. (Corresponding Embodiment of the Invention) An eighth embodiment will be described. (Operation / Effect) The image obtained by the first solid-state imaging device is divided into regions, depth information is estimated by stereo matching in units of the division, and the region is divided into regions smaller than the central divided region. By assigning depth measurement values, it is possible to improve the accuracy of stereo matching and obtain more detailed depth information of objects, and to obtain high-quality object separation and three-dimensional images at low cost. A digital stereo camera device can be provided.

Supplementary Note 13 Segmentation by setting a threshold based on shading and color tone from an image, and depth information
A feature in which one or more objects are separated from the background using a combination of a certain depth and a depth extraction area of an object existing at a certain position, and using a segmentation separation in a large depth extraction area. 12. The stereo camera device according to item 11. (Corresponding Embodiment of the Invention) A ninth embodiment is shown. (Action / Effect) A large depth extraction area is created by combining a segmentation by setting a threshold based on shading and color tone from an image and a depth extraction area of a certain depth and an object existing at a certain position from depth information. By separating one or more objects from the background using segmentation separation, it is possible to acquire depth information as objects near edges in high-definition images. A digital stereo camera device capable of separating an object and acquiring a three-dimensional image can be provided.

Supplementary Note 14 A histogram of a divided region in the depth direction is prepared, and a two-dimensional contact state of the divided region on the screen space is included in a cluster of divided regions including the peak. 13. The stereo camera device according to claim 11, wherein the stereo camera device recognizes an object existing at a certain depth and a certain position and separates one or more objects from a background. (Corresponding Embodiment of the Invention) A tenth embodiment will be described. (Operation / Effect) A histogram of a divided region in the depth direction is created, and a set of divided regions including a peak includes a two-dimensional contact state of the divided region on the screen space. By recognizing that the object exists at a certain depth and at a certain position, and separating one or more objects from the background, the object information with few feature points inside the edge is also
It is possible to provide a digital stereo camera device that can acquire depth information as an object and can separate a high-quality object and acquire a three-dimensional image at low cost.

Supplementary Note 15 A supplementary note 8 or 9 characterized in that a serial number is assigned to the extracted object and background, or that the extracted object and background are stored in a storage device in units of type, image data, and depth information in units of object and background. The stereo camera device according to claim 1. (Corresponding Embodiment of the Invention) An eleventh embodiment will be described. (Operation / Effect) By assigning a serial number to the extracted object and the background, it is possible to operate on an object basis, and a digital stereo camera capable of low-cost, high-quality object separation and acquisition of a three-dimensional image. An apparatus can be provided.

[0147]

As described above, according to the present invention, in particular, an electronic stereo camera using two or more lenses and a solid-state image sensor with a structure as simple as possible and with a simple method. And a digital still camera capable of separating an object and a background from a captured image, a digital video camera capable of separating an object and a background from a captured image, and an electronic stereo camera applicable to image processing. be able to.

[Brief description of the drawings]

FIG. 1 is a schematic diagram showing a configuration of a stereo digital camera to which an electronic stereo camera according to a first embodiment of the present invention is applied;

FIG. 2 is a schematic diagram showing the principle of a stereo digital camera to which the electronic stereo camera according to the first embodiment of the present invention is applied.

FIG. 3 is a diagram showing an actual stereo matching method.

FIG. 4 is a diagram illustrating a technique of separating a background and an object from depth information.

FIG. 5 is a diagram illustrating a configuration of an optical system according to a second embodiment;

FIG. 6 is a diagram illustrating a configuration of an optical system according to a third embodiment.

FIG. 7 is a diagram illustrating a configuration of an optical system according to a fourth embodiment.

FIG. 8 is a diagram illustrating a fifth embodiment;

FIG. 9 is a diagram illustrating a modification of the fifth embodiment;

FIG. 10 is a diagram illustrating a sixth embodiment;

FIG. 11 is a diagram illustrating a seventh embodiment;

FIG. 12 is a diagram illustrating a modification of the seventh embodiment;

FIG. 13 is a diagram illustrating an eighth embodiment;

FIG. 14 is a diagram illustrating a method of performing stereo matching with higher accuracy and obtaining finer final depth information in the eighth embodiment.

FIG. 15 is a diagram illustrating a ninth embodiment;

FIG. 16 is a diagram illustrating a tenth embodiment;

FIG. 17 is a diagram showing an eleventh embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Stereo digital camera, 2 ... Image sensor, 3 ... Image sensor, 4 ... Main lens, 5 ... Sub lens 6, 7 ... Optical axis 10 ... Storage device, 11 ... Control device, 12 ... Operation device, 13 ... Flash system .

Claims (3)

[Claims] A first imaging unit having a first imaging optical system and a first solid-state imaging device; a second imaging optical system; and an effective pixel number larger than the first solid-state imaging device. A second imaging unit having a small number of second solid-state imaging devices, and having an imaging direction substantially the same as the imaging direction of the first imaging unit; an image captured by the first imaging unit; By referring to the image captured by the second imaging means,
An electronic stereo camera comprising: a depth detecting unit that detects depth information of an image of the first imaging unit. 2. The image processing apparatus according to claim 1, wherein the depth detecting unit detects an area having the smallest image difference from a desired area in the image captured by the first imaging unit from the image captured by the second imaging unit, 2. The electronic stereo camera according to claim 1, wherein the depth information of the desired area is detected from a displacement between the two areas. 3. The depth detecting means further divides an image captured by the first imaging means into regions, detects depth information for each region, and maps each region to a background in the image from the depth information. 3. An electronic stereo camera according to claim 1, wherein the electronic stereo camera is divided into an area corresponding to a foreground and an area corresponding to a foreground in the image.