JP4117041B2 - Digital camera system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a digital camera system.
[0002]
[Prior art]
The human eye does not feel out of focus even with deep objects. In addition, when wearing glasses, the image projected on the retina is greatly distorted, but humans immediately correct it and perceive it. Furthermore, human perception recognizes a clear image over a wide range as a perceptual visual field, although a clear image is obtained only near the center of the visual field.
[0003]
The main differences between such human perceptual images and images taken by conventional cameras can be summarized as the following three points.
(1) The perceptual image includes image depth (three-dimensional) information, but one photographic image does not have this.
(2) The field of view perceived as one (one) image is dramatically wider than that of a photographic image.
(3) The perceived image is pan focus (not out of focus).
[0004]
In the field of digital camera systems, the realization of the functions of human vision as described above is unexplored. With regard to pan focus, the method of selecting the sharpest image pixel from multiple images shot with the focus position slightly shifted and combining the pan focus images is “Kodama et al;” from multiple different focus images. All-focus image reconstruction ", 1995 Television Society Annual Conference, P.149".
[0005]
[Problems to be solved by the invention]
The objective of this invention is providing the digital camera system for acquiring the image close | similar to the above-mentioned human perceptual image.
In particular, the present invention is to provide a digital camera system for acquiring an image to which depth information of a subject is added.
[0006]
[Means for Solving the Problems]
The digital camera according to the present invention includes means for storing a plurality of images of the same subject that are photographed while moving in a direction perpendicular to the direction of the subject, a predetermined image among the plurality of stored images, and Sequentially comparing images other than the above, and using the position where the area where the difference between the two compared images is zero as the reference is the reference position, the position indicating the maximum similarity to the reference position in the predetermined image and the other images And a means for extracting the distance as depth information of the subject.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0010]
According to the first embodiment of the present invention, depth information of a subject can be added to a captured image. This is the subject of the present invention. First, a method for acquiring depth information will be described.
[0011]
In order to find the depth of the subject only from the photographed image, a plurality of images are required. Humans use binocular parallax for depth recognition. If the parallax alone does not provide sufficient accuracy, the person moves to the left and right to intentionally create a larger parallax. To do the same thing with the camera, the photographer moves and shoots the same subject at different locations. By doing so, parallax occurs between the two images.
[0012]
The above will be described with reference to FIG. As shown in FIG. 1, when the distance between the lens 1 of the camera and the imaging plate 2 is Fo and the distance between the lens 1 and the subject Oi is Ro (i), the photographer holds the camera in the direction of the subject. When the lens 1 and the imaging plate 2 move from the positions 1 (1) and 2 (1) to the positions 1 (2) and 2 (2), respectively, when they move by a distance L in a direction perpendicular to the subject, the subject Oi. Is moved by Fo × L / Ro (i) on the imaging plate 1. If Fo is constant and Ro >> Fo, the moving amount of the image on the imaging plate 2 is almost inversely proportional to the distance Ro (i). Therefore, by storing how much each image element (pixel) has moved while a person moves with a camera, the depth information of the subject can be restored.
[0013]
FIG. 2 is a schematic configuration diagram of a digital camera system according to the first embodiment of the present invention based on such a principle. FIG. 3 is a flowchart showing a procedure for adding depth information of a subject to a captured image using the digital camera system of the present embodiment.
[0014]
In FIG. 2, reference numeral 100 denotes an imaging unit including a lens, an imaging plate (CCD, etc.), a focus adjustment, an aperture adjustment, a shutter, and the like. Reference numeral 102 denotes a storage unit for storing data of an image taken by the imaging system 100, 104 denotes a depth information extraction unit for extracting depth information, and 106 denotes an operation such as a shutter operation or switching of a shooting mode by a photographer. An operation unit 108 for performing data transmission, an external interface unit 108 for exchanging data with an external computer or the like, and a control unit 110 for controlling each unit.
[0015]
When adding depth information of a subject to a photographed image, first, one subject is photographed in the still image photographing mode (FIG. 3, step S1). The captured still image frame is stored in the storage unit 102.
[0016]
Next, the mode is switched to the moving image shooting mode, and the same subject is continuously shot while moving in the direction orthogonal to the optical axis direction when the still image is shot (step S2). A series of captured moving image frames is stored in the storage unit 102. However, the moving image frame storage area and the still image frame storage area may be separate physical memories. Since the moving image frame is used for extracting depth information, it is deleted when the depth information has been extracted.
[0017]
When the necessary number of moving image frames have been shot, the shooting operation is terminated, and the depth information extraction unit 104 executes depth information extraction processing (step S3). First, the depth information extraction unit 104 sequentially compares the first moving image frame and each subsequent moving image frame, and searches for a position (reference position) where the area where the difference between the compared images is zero is maximized. Specifically, the difference between the pixel unit or the small region unit is calculated while the two frames to be compared are moved in parallel by a relatively small amount, and the relative position where the sum per frame of the difference value is minimized is calculated. look for. Next, the first moving image frame is divided into m × n rectangular blocks, and the first moving image frame and each subsequent moving image frame are compared in block units. The initial position for comparison is the reference position. By this comparison, a moving image area having the maximum similarity in the vicinity is searched for each block (a search range is determined in advance). Then, a relative distance (I, J) to the moving image area indicating the maximum similarity of each block is obtained, and this is stored in the storage unit 102 as depth information.
[0018]
FIG. 4 is an explanatory diagram of the depth information extraction. 120 (1) is the first moving image frame, and 120 (2), 120 (3),..., 120 (n) are the subsequent moving image frames. Comparing the moving image frame 120 (1) with the next moving image frame 120 (2), it can be seen that the triangular image portion is relatively moved in the horizontal direction. This moved portion is shown as a halftone dot area in the difference frame 121. The relative movement amount (I, J) of the image in the horizontal and vertical directions is extracted in units of blocks.
[0019]
An example of the extracted depth information is shown in FIG. In FIG. 5, (0, 0) means an area where no movement is observed, and (0, -1) and (0, -3) mean an area moved only in one direction. A region where the amount of movement is uniform represents a subject that is substantially equidistant from the camera. The difference in the amount of movement between the regions is the apparent distance between them. In this example, the (0, 0) region and the (0, -3) region are apparently farthest away. The coordinate value of the pixel in the above processing is a value that should be accurately expressed by an angle from the lens center, but it is also possible to use the pixel address of the imaging plate approximately.
[0020]
In this way, the depth information of the subject is extracted using the continuous moving image frames, and is stored in a form added to the still image frame. By importing the depth information into an external computer or the like together with the still image frame, editing such as three-dimensional imaging of the captured image becomes possible.
[0021]
According to the second embodiment of the present invention, the field of view can be dramatically expanded in the digital camera system. As mentioned above, the field of view perceived by humans is far wider than that of photographs. In other words, the size of a single image perceived by a human is much larger than a captured image of a normal camera. Considering this further, humans often recognize images observed at one point as a group of images. The landscape in memory is usually a panoramic image. That is, all images viewed from one point are recognized as wide-angle images or as individual narrow-angle images. Ideally, individual narrow-angle images cover all orientations in succession without interruption. In order to realize this with a camera image, a series of images taken from the same point must be stored in the memory, but the individual images need to be freed from distortion and a series of images. It is necessary to add a tag so that it is a part of an image and its optical axis direction.
[0022]
FIG. 6 is a schematic configuration diagram of a digital camera system according to a second embodiment of the present invention that realizes a field of view expansion based on such a principle. In FIG. 6, reference numeral 200 denotes an image pickup unit including a lens, an image pickup plate (CCD, etc.), mechanisms such as focus adjustment, aperture adjustment, and shutter. Reference numeral 202 denotes a storage unit for storing image data taken by the imaging system 200, 204 denotes a distortion correction unit that removes distortion of the image, and 205 denotes a coordinate conversion unit that converts between plane coordinates and spherical coordinates. , 206 is an image composition unit that executes a process of composing one display image larger than a plurality of images. Reference numeral 208 denotes an operation unit for the photographer to operate the camera such as a shutter operation, 210 denotes an external interface unit for exchanging data with an external computer or the like, and 212 denotes a control unit for controlling each unit.
[0023]
In order to obtain a wide-angle image, first, it is necessary to continuously perform shooting operations while changing the direction little by little from the same point, and the captured image file needs to be stored in the storage unit 202. As a storage method of this image file, either a method of storing as an image of planar lattice points (method 1) or a method of storing in correspondence with the azimuth angle of the pixel (method 2) may be employed. However, whichever storage method is adopted, the distortion aberration correction unit 204 removes the distortion aberration of the image in advance. In order to remove this distortion, for example, the method described in the specification attached to the patent application of Japanese Patent Application No. 8-273294 or Japanese Patent Application No. 8-296025 by the present applicant may be used. Then, explanation is omitted.
[0024]
In the storage method 1, the image captured on the imaging plate is stored as it is. Each pixel corresponds to one incident light to the camera, but in order to enable smooth synthesis of the image, the parameters for enabling the azimuth angle of the incident light to each pixel to be reproduced are retained. It is necessary to keep.
[0025]
In FIG. 7, two different orientations of the camera are indicated by optical axes L1 and L3. 11 (1) and 11 (2) indicate the position of the lens 11 at each orientation, and 12 (1) and 12 (2) indicate the position of the imaging plate 12 (CCD etc.) at each orientation. ing. The arbitrary light beam L2 is shot when the imaging plate 12 is at the position 12 (1) and at the position 12 (2), but the images shot at both positions are connected at the position of the light beam L2. Even if they are combined, one image cannot be synthesized. This is because the two image planes are different planes, and the image does not undergo a smooth shape conversion at the joined portion. In order to be able to synthesize images smoothly, it is necessary to be able to reproduce the azimuth angle of the original incident rays of individual images, and the important parameters that need to be kept are as follows: .
* Image file name * Tag indicating the optical axis direction of the image * Distance between the lens and the imaging plate * Pixel pitch of the imaging plate (horizontal method and vertical direction)
* The relative azimuth angle of each image with respect to a reference image (usually the first image taken in a series of images). The addition, storage, and storage of such parameters are managed by the control unit 212.
[0026]
In the storage method 2, each pixel of the image is stored in correspondence with the azimuth angle of the incident light. Unlike the method 1, in order to facilitate image synthesis, the pixels converted on the flat imaging plate are pre-converted by the coordinate conversion unit 205 so as to be equidistant on the spherical coordinate (azimuth angle). And save it.
[0027]
FIG. 8 shows a state of this coordinate conversion, 21 is a lens, and 22 is an imaging plate. The incident lights L2 and L2 ′ other than the optical axis L1 are converted to the coordinates of the positions q1 and q2 on the spherical surface 23 corresponding to the coordinates of the pixels p1 and p2 having the same pitch formed on the imaging plate 22. The result of such conversion for all pixels is stored. However, as can be seen from FIG. 8, when simple conversion is performed, the pixel pitch decreases on the spherical surface 23 as the distance from the optical axis decreases, which is inconvenient for memory management and is also inconvenient for the composition processing described later. Then, it is converted into pixels on the spherical surface at equal intervals by interpolation. The important parameters that need to be retained for Method 2 are as follows:
* Image file name * Tag indicating the azimuth of the optical axis of the image * Inter-pixel angle pitch (horizontal and vertical directions: In a normal imaging board, the azimuth angle depends on the pixel position, but as described above, it is a constant angle. Pre-converted to be pitch)
* The relative azimuth angle of each image with respect to a reference image (usually the first image taken in a series of images). The addition, storage, and storage of such parameters are managed by the control unit 212.
[0028]
When parameters related to a series of images are prepared in the storage unit 202 as described above, an image with an enlarged field of view can be synthesized in the image synthesis unit 206. FIG. 9 is a flowchart showing the flow of the image composition process. 10 and 11 are diagrams for explaining this image composition.
[0029]
FIG. 10 shows the relationship between images 31 and 32 obtained by photographing different azimuths (optical axis directions) P1 and P2, and it is assumed that the lens center O of the camera is stationary and the subject is at infinity. The coordinate system of the image 31 is represented by (x1, y1, z1), and the coordinate system of the image 32 is represented by (x2, y2, z2). ξ represents the common axis of the (x, y) planes of both coordinate systems, α is the angle formed by the x1 axis and the common axis ξ, β is the angle formed by the x2 axis and the common axis ξ, and γ is the z1 axis and the z2 axis. It is an angle to make. If the image 31 is a reference image, the illustrated α, β, and γ can be used as angle parameters representing the relative angle of the image 32 with respect to the reference image.
[0030]
The contents of the image composition process are as follows. In the first step S11, the orientation (Φ, Ψ) to be viewed and the image size (W) are designated. This is set as (Φ, Ψ, W, α, β, γ).
[0031]
In the next step S 12, an image (Φ ′, Ψ ′, W ′, α ′, β ′, γ ′) having the optical axis direction closest to the previously specified direction (Φ, Ψ) is stored from the storage unit 202. Take out.
[0032]
In the next step S13, a spherical surface in the direction of azimuth (Φ ′, Ψ ′) is defined, and an image (Φ ′, Ψ ′, W ′, α ′, β ′, γ ′) is projected onto this spherical surface. When the image is stored by the method 1, each pixel of the image corresponds to one point on the plane perpendicular to the azimuth (Φ ′, Ψ ′). It is necessary to output the pixel value. This procedure corresponds to the conversion from p1, p2 to q1, q2 in FIG. 8, and is executed using the coordinate conversion unit 205. If the storage method 2 is used, this conversion procedure is not necessary.
[0033]
In the next step S14, the pixel defined as the spherical surface in the previous step is projected onto the target plane P0 perpendicular to the azimuth (Φ, Ψ) (see FIG. 11). The pixel position (i, j) on the target plane P0 is (F0 tan (θ)) ^ 2 = (i px) ^ 2 + (j py) ^ 2
Can be expressed as Here, px and py are pixel pitches in the horizontal and vertical directions on the plane P0, F0 is the distance between the lens and the imaging plate (imaging plane) (the radius of the sphere in FIG. 11), and θ is the optical axis and pixel (i , J) is the angle formed by the incident light.
[0034]
In the next step S15, an unused image (Φ1, Ψ1, W1, α1, β1, γ1) closest to (Φ ′, Ψ ′) is extracted from the storage unit 202, and the processing from step S13 is performed on the image. Continue. The process ends when there is no unused image or when it is determined in step S16 or S17 that the target plane P0 is covered.
[0035]
In this way, a wide-angle composite image using a plurality of images is created on the target plane P 0 and stored in the storage unit 202. The composite image data can be transferred to an external computer or the like via the external interface unit 210 and displayed on the screen as it is.
[0036]
A plurality of images taken from the same point, tags and parameters related to the images are stored in the storage unit 202, the data is transferred to an external computer or the like, and necessary processing is executed by the external computer or the like. Is also possible.
[0037]
As described above, one of the features of an image perceived by humans is that it is pan focus. It is thought that humans are observing an object while continuously moving the focal position of the eye, and reconstructing images of these different focus positions in the brain. With the current technology, a pan-focus image based on only one image has not been obtained, but as the alternative technology, the pixel of the sharpest image is selected from a plurality of images with the focus position gradually shifted. As described above, a method for synthesizing a pan-focus image is known.
[0038]
According to the third embodiment of the present invention, a pan-focus image can be obtained in a digital camera system. FIG. 12 is a schematic configuration diagram of such a digital camera system, and FIG. 13 is an explanatory diagram thereof.
[0039]
In FIG. 12, reference numeral 300 denotes an imaging unit. The imaging unit 300 includes a mechanism that captures a large number of images while shifting the lens position and outputs a series of captured images. As this mechanism, for example, the lens is photographed while sequentially protruding from a position close to the imaging plate (or conversely, moving from a distant position to a position close to the imaging plate), and the sharpest image (with the highest spatial frequency). It is possible to use a known autofocus mechanism that detects the lens position from which the obtained image) is obtained as the in-focus position. However, for the purpose of autofocusing, only one image at the focal point is stored and the other images are discarded, but in the digital camera system of the present invention, all the captured images are stored in the storage unit 301, Used for composition of pan focus images.
[0040]
Reference numeral 302 denotes an arithmetic processing unit that executes pan focus image synthesis processing, 303 denotes an operation unit, 304 denotes an external interface unit for external transfer of data in the storage unit 301, and 305 denotes a control unit that controls each unit. is there. Since the processing for synthesizing the pan focus image requires a large amount of calculation, when the capability of the internal arithmetic processing unit 302 is insufficient, the image and its image are sent to the high-performance arithmetic processing unit 306 such as an external general-purpose computer. It is also possible to transfer related data and execute necessary calculations.
[0041]
Now, all the series of images photographed by the imaging unit 300 are stored in the storage unit 301 together with information on the lens position (distance between the lens and the imaging plate) at the time of each photographing. This series of images is given the same file name, and a serial number is assigned to each image in the order of photographing. For example, landscape 1-1, landscape 1-2,... Attached to each of a series of images 310 shown in FIG. . . “Scenery 1” is a file name, and the number after the hyphen is a serial number. Also, F0, F1, F2,. . . Represents the lens position, F0 is the focal length of the lens (corresponding to an image at infinity), and F1, F2,. . . Is a lens position that is sequentially farther from F0.
[0042]
According to the present invention, a pan-focus image in which defocusing is corrected from a plurality of images weighted using a variable having a high correlation with a density gradient, such as a differential value or a high-order differential value, for each pixel of a series of images. create. Hereinafter, a specific example of the pan-focus image synthesis process will be described with reference to FIG.
[0043]
First, fk (i, j) pixels and surrounding (four neighboring) pixels of the “landscape 1-k” image in the series of images 310 are read, and the next Laplacian is calculated.
Δfk (I (k), J (k))
= | Fk (i (k), j (k)) − fk (i (k) -1, j (k)) |
+ | Fk (i (k), j (k))-fk (i (k), j (k) -1) |
+ | Fk (i (k), j (k)) − fj (i (k), j (k) +1) |
＋ | fk (i (k), j (k)) − fk (i (k) + 2, J (k)) |
However, I (k) = i × (F0 / Fk), J (k) = j × (Fo / Fk)
It is defined as
[0044]
This Δfk (I, J) to the power W, {Δfk (I, J)} ^ W, is stored in the address (I, J) of the first frame memory (weight table) 312. Here, W is a certain constant.
[0045]
The weight addition value Σk {Δfk (I, J)} ^ W is calculated and stored in the address (I, J) of the second frame memory (weight sum table) 314 (actually written before) Read the value, add the new value and store again).
[0046]
Calculation of images at different lens positions Sf (I, J) = Σk {fk (I (k), J (k)) ^ W × Δfk (I, J)}
The calculation result is stored in the address (I, J) of the third frame memory 316. This calculation means that the pixel luminance (density) is integrated with a weight corresponding to the secondary differential value.
[0047]
Here, I (k) and J (k) are generally not integers. However, when they are non-integer numbers, fk ([I (k)], [J (k)]), fk ([I (k)] + 1, [J (k)]), fk ([I (k)], [J (k)] + 1), fk ([I Interpolate from (k)] + 1, [J (k)] + 1).
[0048]
A value obtained by dividing Sf (I, J) captured corresponding to the lens position by the value of the first frame memory 312, that is, Sf (I, J) / Σk {Δfk (I, J)} ^ W
Is required. This is a pan-focus image with defocus corrected. When the image data is generated by the internal arithmetic processing unit 302, the image data is stored in the storage unit 301 and transferred to an external computer or the like via the external interface unit 304 as necessary. If a pan-focus image is obtained, the original series of images is no longer necessary and may be discarded.
[0049]
【The invention's effect】
As described above in detail, according to the present invention, an image to which subject depth information is added can be obtained in a digital camera system, so that editing such as three-dimensional imaging of a captured image can be easily performed. become able to.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining the relationship between the amount of movement of an image on an imaging plate and the depth of a subject when a camera is moved.
FIG. 2 is a schematic configuration diagram of a digital camera system according to a first embodiment of the present invention.
FIG. 3 is a flowchart showing a procedure for extracting depth information.
FIG. 4 is an explanatory diagram of extraction of depth information.
FIG. 5 is a diagram illustrating an example of extracted depth information.
FIG. 6 is a schematic configuration diagram of a digital camera system according to a second embodiment of the present invention.
FIG. 7 is an explanatory diagram relating to stitching images of different orientations.
FIG. 8 is an explanatory diagram relating to coordinate conversion from a plane to a spherical surface;
FIG. 9 is a flowchart showing a flow of wide-angle image composition processing;
FIG. 10 is a diagram for explaining synthesis of a wide-angle image.
FIG. 11 is a diagram for explaining synthesis of a wide-angle image.
FIG. 12 is a schematic configuration diagram of a digital camera system according to a third embodiment of the present invention.
FIG. 13 is a diagram for explaining pan focus image synthesis processing;
[Explanation of symbols]
1,11,21 Lens 2,12,22 Imaging plate 100 Imaging unit 102 Storage unit 104 Depth information extraction unit 106 Operation unit 108 External interface unit 110 Control unit 200 Imaging unit 202 Storage unit 204 Distortion aberration correction unit 205 Coordinate conversion unit 206 Image composition unit 208 Operation unit 210 External interface unit 212 Control unit 300 Imaging unit 301 Storage unit 302 Calculation processing unit 303 Operation unit 304 External interface unit 305 Control unit 306 External calculation processing device

Claims (1)

It means for storing a plurality of images against captured while moving in a direction perpendicular to the direction of the object, the same object,
The predetermined image of the plurality of stored images and the other images are sequentially compared, and the position where the difference between the two compared images is zero is the reference position, and the predetermined image and the others A distance between the reference position and the position indicating the maximum similarity in the block, and extracting the distance as subject depth information;
A digital camera system comprising:

Images