JPH11196318A - Image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce image distortion due to composite processing for the photographed images of the whole object consisting of composition of plural partial images and having high resolution. SOLUTION: After photographing the whole object, a digital camera changes the optical axis direction of an image pickup part 2 and partially photographs the object. An image processing part 23 enlarges the photographed image of the whole object at a prescribed magnification value based on the number of divisions of the object and the superposed positions of photographed images of respective parts of the object in the enlarged image is calculated. The photographed images of respective parts are composited so that their boundary parts are stuck to each other in a state aligned to their superposed positions to generate the image of the whole object. The image of the whole object is entered and used as an aligning reference for the sticking composition of the photographed images of respective parts to reduce image distortion generated on the sticking position due to composition processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus which divides an object into a plurality of blocks, partially captures the image, and combines the partial images by image processing to generate a photographed image of the entire object. .

[0002]

2. Description of the Related Art Conventionally, in a digital camera using a solid-state image pickup device such as a CCD, a subject is divided into a plurality of blocks, a partial image is taken, and partial images are combined so as to be combined to form the entire subject. A method of increasing the resolution by generating a captured image has been proposed.

For example, Japanese Patent Application Laid-Open No. 6-141246 discloses that a subject light image is divided into a plurality of portions so that boundaries are overlapped with each other, and each partial light image is picked up by a plurality of image pickup devices. An image pickup apparatus is shown in which an image of an entire subject is obtained by synthesizing images that have been image-captured so as to be bonded together at an overlapping boundary portion. In this imaging device, when an image shift occurs at a boundary portion, the image shift amount at the boundary portion is represented by a parallel movement amount and a rotational movement amount, and both images are changed based on these movement amounts. After the displacement of the image at the boundary portion is corrected, the combining process is performed.

Japanese Patent Laid-Open Publication No. Hei 3-210873 discloses an image of a document larger than an image capturing area is partially captured by overlapping a boundary portion with each other, and these partial images are connected at an overlapping portion. 1 shows an image forming apparatus that forms an image of an entire document by recording on a recording sheet while recording.

[0005]

SUMMARY OF THE INVENTION The above-mentioned Japanese Patent Laid-Open No. 6-141 is disclosed.
In the method of synthesizing partial images described in Japanese Patent Publication No. 246, since the synthesizing process is performed using only information of local image shift in the overlap area, image shift occurs between partial images to be synthesized. This causes a problem that, for example, a graphic such as a straight line extending from the overlap region to the outside region is distorted or bent near a joint between the two regions. In addition, when the overlapping area does not include characteristic image information for performing alignment of the two partial images,
There is also a problem that the combining process cannot be performed accurately, and image distortion occurs in a combined portion of the combined image.

[0006] In the above-mentioned Japanese Patent Application Laid-Open No. 6-141246, image alignment is performed by parallel movement or the like using only the image shift information in the overlap area. If there is a rotation shift or a shift due to a difference in size between images, it is difficult to perform registration between partial images, and it is not possible to perform a synthesizing process accurately. Therefore, even if the method of synthesizing partial images described in the publication is applied to an imaging device such as a digital camera, the problem of image distortion of a synthesized image caused by the above-described synthesis processing cannot be solved.

SUMMARY OF THE INVENTION The present invention has been made in consideration of the above problems, and has been made in view of the above circumstances. To provide an image pickup device that can obtain a photographed image of the same.

[0008]

According to the present invention, there is provided a first image pickup means for picking up an image of an entire object, a second image pickup means for dividing the entire object into a plurality of portions and sequentially picking up the images, and Combining position calculating means for calculating a combining position in combining the partial images of the subject imaged by the second image capturing means with reference to an image of the entire subject imaged by the image capturing means; and each partial image of the subject image And an image synthesizing means for synthesizing each partial image in a state where the image is aligned with the synthesizing position calculated by the synthesizing position calculating means.

According to the above arrangement, the entire image of the subject is photographed by the first image pickup means, and the entire image of the object is divided into a plurality of parts by the second image pickup means, and each part is photographed sequentially.
The combining position in the combining process is calculated based on the captured image of the entire subject captured by the first imaging unit with respect to the captured image of each part of the subject. Then, the respective partial images of the subject are combined in a state where they are aligned with the combining position calculated by the combining position calculating means, and a captured image of the entire subject is generated.

Further, according to the present invention, there is provided a first image pickup means for picking up an image of an entire object, wherein the entire object is divided into a plurality of portions so that a boundary portion is overlapped, and each portion of the object is divided into an image size of the entire object. Expanding to approximately the same size as
A second imaging unit that captures an image, a storage unit that stores images captured by the first and second imaging units, and the second imaging unit based on a captured image of the entire subject captured by the first imaging unit. A combining position calculating unit that calculates a combining position in the combining of the partial images of the subject imaged by the second imaging unit; and aligning the partial images of the subject with the combining position calculated by the combining position calculating unit. Image combining means for combining the partial images so that the partial images are attached to each other at the boundary portions in the state in which they have been performed (claim 2).

According to the above arrangement, the subject is partially photographed, and a plurality of partially photographed images (hereinafter, referred to as "partial images") are synthesized so as to be pasted together at a boundary portion. , An entire image.) Is generated. In this case, the entire image of the subject is photographed by configuring the screen by the first imaging means so that the entire subject enters the photographing screen.
This photographed image (hereinafter, referred to as a pre-photographed image) is stored in the storage unit.

The second imaging means divides the entire subject into a plurality of portions so that the boundary portions overlap, and enlarges each portion of the subject to a size substantially the same as the imaging size of the entire subject. The image is captured, and the captured image (partial image) is stored in the storage unit. For example, when photographing a subject by dividing the subject into two parts on the left and right, the subject is divided so as to overlap near the vertical center line of the subject. Is performed, shooting is performed with the screen configured so that the right half of the subject enters the shooting screen, and both shot images are stored in the storage means.

[0013] Then, a combined position in the combining of the partial images of the subject is calculated based on the pre-captured image, and the partial images are combined so as to be bonded to each other at a boundary portion while being aligned with the corresponding combined position. Is done. As a result, an overall image substantially the same as the pre-captured image is obtained.

For example, when an object is photographed by dividing it into two parts, the positions corresponding to the left and right ends of the enlarged pre-photographed image are calculated for the left and right partial images, respectively.
The two images are synthesized so as to paste a boundary portion with each other in this positional relationship.

Further, according to the present invention, in the above image pickup apparatus,
The first and second imaging means include an imaging unit in which a lens capable of changing a shooting magnification and an imaging element that photoelectrically converts a subject light image into an electric image and captures the electrical image are integrally configured to be changeable in an optical axis direction; An optical axis changing unit that changes an optical axis direction of the imaging unit;
While setting the optical axis direction of the imaging unit to the front direction,
A first imaging control unit that sets the photographic magnification of the lens so that the entire subject is included in the photographic screen and captures the entire subject; Is enlarged so as to enter a shooting screen with an overlapping size, and the optical axis direction of the imaging unit is sequentially changed to a predetermined direction corresponding to each part of the subject to partially capture the subject. (Claim 3).

According to the above arrangement, when photographing is instructed,
First, the optical axis direction of the imaging unit is set to the front direction, and the photographing is performed by setting the photographing magnification of the lens so that the entire subject is included in the photographing screen. The photographed image (pre-photographed image) is stored in the storage unit. Is done.

Subsequently, the photographing magnification of the image pickup unit is enlarged so that each part of the subject enters the photographing screen at a size where the boundary portions overlap each other, and the optical axis direction of the image pickup unit is set to a predetermined value corresponding to each part of the subject. Are sequentially changed, and each part of the subject is photographed. For example, when photographing a subject by dividing the subject into two parts on the left and right, the subject is divided so as to overlap near the vertical center line of the subject, and the photographing magnification is enlarged so that, for example, the left half of the subject enters the shooting screen. Thereafter, the optical axis direction of the photographing means is shifted to the left by a predetermined angle from the front direction to photograph the right portion of the subject, and then the optical axis direction of the photographing means is shifted to the right by a predetermined angle from the front direction to move the subject. The left portion is photographed.

[0018]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An imaging apparatus according to the present invention will be described.
A digital camera will be described as an example. FIG. 1 is a perspective view showing an appearance of a digital camera according to an embodiment, and FIG. 2 is a rear view of the digital camera. FIG. 3 is a diagram showing a schematic configuration of an imaging system of the digital camera.

In the digital camera 1, an image pickup unit 2 is provided substantially at the center of the front surface so that the optical axis direction L can be changed, and a light meter 3 for measuring the brightness of the subject is provided above the image pick-up unit 2. A distance measuring unit 4 for measuring a subject distance is provided on the left side, and a flash 5 is provided on the right side of the light measuring unit 3.

As shown in FIG.
1 includes a photographing lens 202 formed of a zoom lens protruding from the front surface of the camera 1 and an image pickup device 203 formed of a CCD area sensor disposed at a predetermined position on the optical axis L in the housing 201. On the side surface of the housing 201, a pair of rotating shafts 201a is protruded in a direction orthogonal to the optical axis L.
1a is rotatably supported at both ends of a U-shaped support frame 201b. A rotation shaft 201c projects outward from the center of the U-shaped support frame 201b.
The tip of c is connected to the rotor of the electric motor 15. Further, the tip of one rotating shaft 201 a is connected to the rotor of the electric motor 16.

The electric motor 15 is a drive source for rotating the housing 201 in the left-right direction (H direction in the figure), and the electric motor 16 rotates the housing 201 in the vertical direction (V direction in the figure). It is a driving source to be moved. When the electric motor 15 is driven,
The U-shaped support frame 201b connected to the rotor rotates in a horizontal plane, whereby the housing 201 rotates so that the optical axis L changes in the horizontal plane. In addition, when the electric motor 16 is driven, the rotating shaft 201a connected to the rotor rotates, whereby the housing 201 rotates so that the optical axis L changes in a vertical plane.

An electric motor 17 is provided on the lens barrel of the taking lens 202, and the electric motor 17 drives the taking lens 202 to change the zoom ratio.

In the present embodiment, the taking lens 20
2 and the image sensor 203 form a pair and have a structure in which the zoom ratio can be continuously changed, but the zoom ratio is switched stepwise by switching the combination of a plurality of lenses and the image sensor with a prism, a reflector or the like. It may be structured.

The photometric unit 3 has a light receiving element such as an SPC,
The luminance of the subject is detected by receiving the reflected light from the subject. The distance measuring unit 4 detects a distance to a subject by, for example, an active distance measuring method. The distance measuring unit 4 has a light emitting element for irradiating the subject with infrared light and a light receiving element for receiving the reflected light of the infrared light from the subject, and based on the reflection angle of the infrared light on the subject. It detects the distance from the camera to the subject. In the present embodiment, the active distance measurement method is adopted as the distance measurement method, but a passive distance measurement method may be used.

On the upper surface of the digital camera 1, a shutter button 6 is provided at the left end. Shutter button 6
Is an operation button which turns on an S1 switch for instructing shooting preparation such as focus adjustment and exposure control value setting when pressed halfway, and turns on an S2 switch for instructing release when fully pressed. In addition, a power switch 7 and a card insertion slot 8 into which a hard disk card 10 (hereinafter, referred to as an HD card 10) is attached and detached are provided at the lower side of the digital camera 1. HD card 1
A card ejection button 9 for ejecting 0 is provided.

When the photographing result is to be printed on recording paper, the card ejection button 9 is pressed to remove the HD card 10 from the digital camera 1. Data can be read directly from the HD card 10 and printed on recording paper.

The digital camera 1 is provided with a SCSI cable interface, and the digital camera 1 is connected to the printer with a SCSI cable to connect the digital camera 1 to the printer.
May be transferred to a printer and the photographed image may be printed on recording paper.

In this embodiment, a PCMCIA compliant HD card is used as a recording medium for image data. However, any other medium such as a memory card or a mini disk may be used as long as it can record captured image data. Recording medium.

As shown in FIG. 2, on the back of the digital camera 1, a display unit 11 (hereinafter, referred to as an LCD display unit 11) comprising an LCD (Liquid Crystal Display) is provided substantially at the center.
Is provided. The LCD display unit 11 corresponds to an optical finder.

When the digital camera 1 is activated, the imaging unit 2
Video shooting is performed on the LCD display unit 1.
By displaying the information in 1, the photographer can monitor the subject in the photographing screen. When the photographer presses the shutter button 6 to perform the photographing operation, the photographing operation of the still image is performed, while the photographed image immediately after the release is displayed on the LCD display unit 11 as a still image. At 11, a photographed image (still image) can be monitored.

Further, the digital camera 1 divides a subject into a plurality of portions in a photographing mode, and synthesizes such that partially photographed images (hereinafter, referred to as partial images) are pasted together by image processing to obtain a resolution. A shooting mode (hereinafter, referred to as a whole image) that generates an image of the entire high subject (hereinafter, referred to as a whole image).
This shooting mode is called a high resolution mode. In this high image quality mode, the photographer first captures the entire subject and displays the captured image (still image) on the LCD display unit 11 so that the photographer can quickly display the contents of the combined image. Can be monitored.

On the left side of the LCD display section 11, a still image display release button 12 is provided. The still image display release button 12 returns the display mode of the LCD display unit 11 that is automatically switched from the moving image display to the still image display every time the shooting process is performed, from the still image display to the moving image display for the next shooting. Operation buttons for The reason why the operation of returning the display mode is performed by the manual operation of the operator is to improve the convenience of monitoring the captured image without fixing the monitoring time. After the still image display is performed for a predetermined period of time, the still image display release button 12 may be omitted by automatically returning from the still image display to the moving image display.

At the lower rear end of the digital camera 1, a high resolution mode setting switch 13 is provided at the left corner, and a composite image number setting switch 14 is provided at the right corner.

The high resolution mode setting switch 13 is a switch for setting the high resolution mode described above. When the high-resolution mode setting switch 13 is set to “ON”, the high-resolution mode is set. When the high-resolution mode setting switch 13 is set to “OFF”, the normal shooting mode is set.

In the high resolution mode, as shown in FIG.
When the shutter button 6 is fully depressed, first, the optical axis L of the imaging unit 2 is set to the front direction, and the entire subject is photographed (see the whole photograph in FIG. 4). That is, as shown in FIG. 5, since the optical axis L of the imaging unit 2 is initially set in the front direction (o direction), the screen is set so that the entire subject Q (scenery in the figure) enters the shooting screen P1. When the configuration is adjusted and the shutter button 6 is pressed, shooting of the entire subject Q is performed.

The photographed image (hereinafter, the whole photographed image is referred to as a pre-photographed image) is displayed as a still image on the LCD display section 11 (see the whole image display in FIG. 4). An image G 'enlarged to four times the size by the interpolation processing is generated (refer to the whole image enlargement in FIG. 4). The enlarged image G 'will continue image distortion correction of each partial image G _A ~G _D when combined synthetic bonded four partial images G _A ~G _D obtained by partially photographing the object and image synthesis It is a standard for

Therefore, in the high resolution mode, the entire image finally synthesized immediately after the release is displayed on the LCD display section 11, so that the entire image after the synthesis can be quickly monitored.

Subsequently, the optical axis L of the imaging unit 2 is changed to a predetermined direction other than the front direction, and the photographing magnification is increased to a predetermined magnification (2 times) to divide the entire subject into four parts. The photographing of G _{A to} G _D (capturing of partial images) is sequentially performed (see FIG. 4 divided photographing). That is, as shown in FIG. 5, the direction of the optical axis L of the imaging unit 2 is changed to a predetermined direction (a direction) in which an upper left part is viewed toward the subject Q, and at the same time, about Ｑ of the subject Q , The photographing magnification of the photographing lens 202 is increased to about twice so that the upper left portion of the subject Q is photographed. After this,
Without changing the photographing magnification of the photographing lens 202, the direction of the optical axis L of the imaging unit 2 is changed to the upper right, lower right,
The upper right part, the lower right part, and the lower left part of the subject Q are sequentially photographed in a predetermined direction (c, d, b directions) in which the lower left part is viewed. At this time, the angle of view in capturing the partial image of the subject Q is set so that the boundary portions of adjacent partial images overlap with each other.

Thereafter, the partial images G _{A to} G _D are converted into enlarged images G ′.
, The combined position of the enlarged image G ′ and the amount of conversion of the image data for correcting image distortion are calculated (see FIG. 4 alignment), and each partial image G is calculated based on this amount of conversion.
_{After A} ~G _D is converted geometrically full image the partial image G _A ~G _D after the image conversion based on the combined position as bonded at the boundary is generated. Then, the entire image is recorded on the HD card 10.

In this embodiment, the pre-photographed image G
Is used as the reference image of the synthesis position, the reduced image of the partial images G _{A to} G _D is compared with the pre-image G to obtain the synthesis position. May be calculated.

In FIG. 5, since the photographing screens P1 and P2 are relatively drawn around the subject Q, the photographing screen P2
Is smaller than the shooting screen P1, but the size of the shooting screen P1 and the shooting screen P2 is the same on the imaging surface of the imaging unit 2, so that the screen configuration of the shooting screen P1 is smaller than that of the shooting screen P2. Also, the subject Q is projected on the imaging surface in a small size. Accordingly, since the pre-captured image has a lower resolution than the partial image due to the relationship between the projection light image and the pixel density, the resolution of the entire image obtained by combining the partial images is higher than the pre-captured image. ing.

On the other hand, in the normal photographing mode, when the shutter button 6 is pressed, a photographing operation is performed only once, and the photographed image is subjected to predetermined image processing (appropriate γ correction, WB adjustment, contour After processing such as correction and color unevenness correction), the image data is recorded on the HD card 10. This photographing processing is substantially equivalent to performing predetermined image processing on the pre-photographed image in the high resolution mode and recording the image on the HD card 10. Therefore, the captured image in the normal shooting mode has a lower resolution than the captured image (synthesized image) in the high resolution mode. In the example shown in FIG. 5, since the imaging magnification of the partial image is approximately twice, the image (composite image) captured in the high-resolution mode has approximately four times the resolution of the image captured in the normal imaging mode. doing.

Returning to FIG. 2, the composite image number setting switch 14
Is a switch for setting the number of partial images in the high resolution mode (that is, the number of divisions of the subject). The number-of-synthesized-images setting switch 14 is capable of setting the number of divisions n in the vertical direction and the number of divisions m in the horizontal direction when the subject is divided into an n × m matrix up to a maximum of four. The number-of-synthesized-images setting switch 14 includes a pair of four-contact slide switches 14a and 14b. The number of divisions n in the vertical direction is set by the upper slide switch 14a, and the number of divisions m in the horizontal direction is set by the lower slide switch 14b. Is set.

In this embodiment, the photographer can set the number of divisions. However, the predetermined number of divisions is fixedly set in advance, and the number-of-combined-images setting switch 14 is omitted. Is also good.

FIG. 6 is a block diagram of a digital camera according to the present invention. In the figure, the thick arrows indicate the flow of image data, and the thin arrows indicate the flow of control data. The same members as those shown in FIGS. 1 to 3 are denoted by the same reference numerals.

The control section 20 centrally controls the photographing operation of the digital camera 1, and is composed of a microcomputer. The control unit 20 controls driving of various members described later.

The imaging control section 21 controls the driving of the imaging section 2. The imaging control unit 21 includes a zoom control circuit 2
11, a scanning control circuit 212 and an imaging control circuit 213. The zoom control circuit 211 includes an imaging lens 202
The zoom ratio (photographing magnification) is controlled. The shooting magnification is changed in capturing the partial image in the high resolution mode, and is set in advance according to the number of partial images (that is, the number of divisions of the subject). Information on the number of divisions of the subject set by the composite image number setting switch 14 is input to the imaging control unit 21 via the control unit 20, and the zoom control circuit 211 responds based on the input information on the number of divisions of the subject. The photographing magnification is set, and the electric motor 17 is driven based on the photographing magnification to set the imaging lens 202 to a predetermined photographing magnification.

The scanning control circuit 212 controls the order of capturing the partial images in the high resolution mode, that is, the operation of changing the direction of the optical axis L of the image capturing section 2. The direction of each optical axis L (ie, the amount of movement in the horizontal / vertical direction with respect to the front direction of the housing 201) in capturing the partial image is also set in advance according to the number of partial images (ie, the number of divisions of the subject). . The order of capturing the partial images is also set in advance according to the number of partial images (that is, the number of divisions of the subject). In the example of FIG. 5, the direction of the optical axis L is scanned in the U-shape in the order of a-c-d-b, but the scanning direction is not limited to this, and a-b-d-c or a-c- Any scanning method such as -b-d can be adopted. Scan control circuit 212
Sets the direction of the optical axis L for each partial image and the scanning direction based on the input information on the number of divisions of the subject, drives the electric motors 15 and 16 based on the direction of the optical axis L, and Is set in a predetermined direction.

The image pickup control section 213 controls the image pickup operation (charge accumulation and readout of accumulated charges) of the image pickup element 203 (hereinafter referred to as CCD 203). Imaging control unit 2
In a photographing standby state, the CCD 203 is driven by video to capture an image for a finder (the image pickup operation is repeated every 1/30 second). Further, in the normal shooting mode, the imaging operation of the CCD 203 is performed only once based on the control signal of the shooting timing input from the control unit 20, and in the high resolution mode, the shooting timing of the shooting timing input from the control unit 20 is controlled. CCD 20 based on the control signal
3 is continuously performed a predetermined number of times ((N + 1) times with respect to the number of divisions N of the subject input from the control unit 2).

The A / D converter 22 converts an image signal (analog signal) output from the imaging unit 2 into a digital signal of, for example, an 8-bit configuration (hereinafter, this image signal is referred to as image data). is there.

The image processing section 23 performs predetermined image processing such as γ correction, WB adjustment and contour correction on image data in a normal photographing mode. Further, in the high resolution mode, while performing the above-described image processing, the image data of a plurality of partial images are combined so as to be combined to generate an entire image. The image processing in the high resolution mode will be described later.

The LCD drive control unit 24 includes the LCD display unit 1
1 is controlled to display the captured image on a monitor. The LCD drive control unit 24 controls the light emission at each pixel position of the LCD display unit 11 based on the image data input from the image processing unit 23, and causes the LCD display unit 11 to display an image. The card drive control unit 25 controls the drive of the HD card 10 when recording image data.

The AE / AWB calculating section 26 calculates an exposure value and a WB adjustment value when capturing a partial image in the high resolution mode. The AE / AWB operation unit 26 extracts the image data of the partially captured area from the image data of the pre-captured image captured in the high resolution mode, and uses the image data to capture the corresponding partial image. And the WB adjustment value are calculated.

A RAM (Random Access Memory) 27
This is a memory for the control unit 2 to perform arithmetic processing relating to photographing. A ROM (Read Only Memory) 28 stores a processing program related to shooting control, data necessary for drive control of the imaging unit 2, various data required for image processing in a high-resolution mode described later, a processing program, and the like. Memory.

FIG. 7 is a diagram showing a first embodiment of a block configuration relating to image processing in the high brightness mode in the image processing section.

In the figure, thick arrows indicate the flow of image data. The image memory 231 is a memory that stores image data input from the A / D converter 22. The image memory 231 includes a RAM, and has a pre-captured image storage area 231a and a partial image storage area 231b.

The pre-photographed image storage area 231a is an area for storing image data of the pre-photographed image. In the normal shooting mode, the image data of the shot image is also stored in the pre-shot image storage area 231a. The photographed image in the normal photographing mode and the preliminary photographed image in the high-resolution mode stored in the preliminary photographed image storage area 231a are read out by the LCD drive control unit 24 and displayed on the LCD display unit 11 (moving image display or still image display). ) Is done. Also,
The partial image storage area 231b is an area where the image data of the partial image is stored.

The image enlarging circuit 232 generates an enlarged image by enlarging the pre-photographed image at substantially the same magnification as the photographing magnification at the time of capturing the partial image. That is, the image enlargement circuit 23
2, for example, in the example of FIG.
An enlarged image G ′ is generated by magnifying the image twice and input to the matching operation circuit 234.

The preprocessing circuit 233 is a circuit that performs necessary processing on each partial image before the partial image combining process. The pre-processing includes processes such as luminance correction, noise removal, smoothing, and edge enhancement. The pre-processing for each partial image is intended to correct a luminance difference and a contour shift between the enlarged image and the partial image and to easily determine a combination position. The pre-processing circuit 233 includes an image memory 231
After sequentially reading out each partial image from the partial image storage area 231b, and performing predetermined preprocessing such as the above-described luminance correction,
It is input to the matching operation circuit 234.

The matching calculation circuit 234 converts image data for each partial image when geometrically deforming each partial image so as to substantially match the image (partial image) at the position corresponding to the enlarged image. An amount (hereinafter, referred to as a matching conversion amount) is calculated.

In this embodiment, since the direction of the optical axis L of the image pickup unit 2 is changed to pick up an image of a subject, the exposure control value and the WB adjustment unit in the image pickup of each partial image are different. This may cause a luminance difference between the partial images. Further, the optical axis L with respect to the imaging portion of the subject
Becomes an oblique direction, and perspective image distortion occurs in each of the captured partial images. That is, as shown in FIG. 8, for example, when four divided and photographed Wall Q is subject to the partial image G _A ~G _D, as shown in FIG. 9, the partial image G _A ~G _D
Causes perspective image distortion.

It is also possible to correct only the luminance difference between the partial images and combine the partial images so as to be combined at the boundary portion. However, in the case of simply combining the combined images, as shown in FIG. Perspective image distortion of the partial image occurs in the whole image after the composition, and furthermore, the design may be bent or shifted at the boundary portion, so that the image quality is significantly reduced. To reduce this problem, a method of correcting the perspective image distortion of the partial image based on the shift of the optical axis L and the subject distance can be considered. In this method, the image distortion of each partial image is corrected. New circuits and processes are needed,
The device becomes complicated, large, and expensive.

In this embodiment, by combining each partial image geometrically with reference to the pre-captured image to determine the combining position in the combining and combining, and correcting perspective image distortion, It is intended to reduce the deterioration of the image quality of the overall image after the synthesis without incurring an increase in the complexity, size and cost of the apparatus. Therefore, the matching operation circuit 234
In each of the partial images, a conversion amount of image data for matching each partial image with an image at a corresponding position of the enlarged image is calculated, and the calculation result is sent to the geometric image conversion circuit 235 and the image synthesis circuit 236. Is entered.

The matching operation circuit 234 extracts a plurality of feature points included in the enlarged image, for example, and performs a geometric transformation such as a parallel movement, a rotational movement, and an enlargement / reduction on the partial image as shown in FIG. Is compared with the enlarged image, and the amount of geometric conversion that maximizes the degree of overlap of the feature points is calculated. Note that the position where the partial image image has the highest matching degree on the enlarged image (hereinafter, this position is referred to as a matching position) is subjected to the combining process so that the boundary portions are pasted together with the respective partial images arranged at that position. Therefore, the geometric conversion amount is also information on the combination position in the combination processing of the partial images.

A feature point is an image of an area having characteristic image information such as a specific character or character string, a specific line, a specific geometric figure (for example, a triangle, a circle, an ellipse, etc.), a specific edge portion, or the like. Data. Characters and character strings as feature points are extracted by known character recognition methods, geometric figures as feature points are extracted by known texture analysis, and edges as feature points are extracted by known edge detection methods. Is extracted.

In FIG. 11, g1, g2, g
Reference numerals 3 and g4 indicate matching positions of the upper left, upper right, lower right, and lower left partial images G1, G2, G3, and G4 in the enlarged image G0. The upper left partial image G1 is
The case where the matching position g1 is calculated by the parallel movement method using the character string C1 of "ABC" or the rectangle C2 as a feature point is shown, and the upper right partial image G2 is matched by the enlargement / reduction method using the rectangle C3 as a feature point. The case where the position g2 is calculated is shown. The lower right partial image G3
Shows a case where the matching position g3 is calculated by the rotational movement method using the thick line C4 or the edge of the thick line C4 as a feature point, and the lower left partial image G4 is the rectangle C2 or the thick line C4.
Is used as a feature point, and the matching position g4
Is calculated.

The parameters of the geometric transformation amount differ depending on the matching method, and are the translation amount in the parallel movement method and the rotation center and the rotation angle in the rotation movement method. In the enlargement / reduction method, in the case of enlargement / reduction around a point, the center position and the magnification are used. As shown in FIGS.
In the case of enlargement / reduction in a linear direction or an arc direction around the axis, the position, the enlargement / reduction direction, and the magnification of the axis are used.

Further, the degree of overlap of the feature points is determined by the correlation value between the image data constituting the feature points of the pre-captured image and the image data at the pixel positions of the geometrically transformed partial image corresponding to the feature points, The determination is made using the sum of the absolute values of the differences between the data or the sum of the squares of the differences between the two image data.

When a geometrical transformation of enlargement or reduction is performed on the partial image, a shift occurs between the pixel position after the transformation and the pixel position of the enlarged image. The image data of the partial image after the geometric conversion corresponding to the pixel position to be configured may be corrected by an interpolation process such as a known Nearest Neibougher method or Cubic Convolution method. Further, in the present embodiment, the degree of overlap is determined using the image data near the feature point. However, the degree of overlap may be determined for the entire partial image to improve the image quality. In this case, it is preferable to reduce the number of pixels by thinning out the image data for each block or every several pixels, so as to suppress the reduction in processing speed.

The geometric image conversion circuit 235 performs a geometric conversion of each partial image based on the geometric conversion amount calculated by the matching operation circuit 234. Specifically, the addresses of the image data constituting each partial image are converted by a predetermined conversion formula using the amount of geometric conversion.

The image synthesizing circuit 236 synthesizes a plurality of partial images that have been subjected to the geometric transformation so as to be pasted together at a boundary portion overlapping each other. Image composition circuit 23
6 generates an image of a boundary portion for synthesis using image data of an overlapping boundary portion between adjacent partial images, and generates an image of the boundary portion for synthesis and an image obtained by removing the boundary portion between the two partial images. Are combined to combine the two partial images.

By the way, if the boundary portions of the adjacent partial images subjected to the geometric transformation are exactly the same, if the two partial images are directly connected without generating the image of the boundary portion for synthesis, Images can be easily synthesized. However, for example, when a displacement of less than one pixel occurs at the boundary between adjacent partial images due to rotational movement or the like, or when the luminance distribution differs due to a difference in imaging conditions, the two images are simply connected to each other. Cannot be matched (density distribution matching or figure matching). Therefore, using the image data of the boundary part between the two partial images, the image is used for synthesis such that the density change at the boundary part and the figure deviation do not become unnatural. Images need to be generated.

FIGS. 13A and 13B are diagrams showing an example of a synthesizing method in a case where a difference in luminance level and a pattern shift occur in an image of a boundary portion, where FIG. 13A is a diagram of a left partial image, and FIG. FIG. 4C is a diagram of a partial image on the right side, and FIG. 4C is a diagram in which left and right partial images are combined so as to overlap at a boundary portion.

The boundary between the left and right partial images A and B includes band-shaped figures Ra and Rb, respectively.
a is slightly inclined to the left side with respect to the band-shaped figure Rb, and its density is slightly higher than that of the band-shaped figure Rb. When combining the left partial image A and the right partial image B such that the boundary portion is superimposed, the combined band-shaped figure Rc is represented by
As shown in (c), it is preferable that the image is generated in the middle between the band graphic Ra and the band graphic Rb.

FIG. 14 shows an ideal band-like figure Rc after synthesis.
13C shows the waveform of a signal composed of an image data sequence, which is a signal waveform on a straight line M in FIG.
In the figure, a waveform Sa indicated by a solid line is a waveform of a signal composed of an image data sequence constituting the band-shaped graphic Ra, and a waveform Sb indicated by a broken line is a waveform of a signal composed of an image data sequence constituting the band-shaped graphic Rb. is there. Also, a waveform Sc indicated by a thick line
Is a waveform of a signal composed of an image data sequence constituting an ideal synthesized band-shaped figure Rc.

As shown in FIG. 14, the ideal signal Sc
Is located at an intermediate position between the signal Sa and the signal Sb, and the level and width of the peaks of the signals Sa and Sb substantially coincide with the levels and widths of the signals Sa and Sb.

When the image data of the boundary portion for synthesis is generated using the image data of the boundary portion of the left partial image A and the image data of the boundary portion of the right partial image B, the left partial image A is generated on the left side of the boundary portion. Is linearly increased, and on the right side of the boundary portion, the weighted average of the image data of both partial images A and B is calculated using a weighting coefficient that linearly increases the ratio of image data of the right partial image B. There is known a method of generating image data of a boundary portion for composition by using the method.

In this linear weighted averaging process, as shown in FIG. 16, the total number of pixels in the overlapping boundary portion is set to N, and the addresses of the pixels are set to 1, 2,... From the left end to the right end of the boundary portion.
Assuming that N, the image data g3 (k) at the pixel position k of the image at the boundary portion for synthesis is determined by, for example, the following arithmetic expression.

[0079]

(Equation 1)

In the above equation, the image data g1 (k)
Is set to the distance k from the left end of the right partial image A, and the weight coefficient of the image data g2 (k) is set to the distance (N−k) from the right end of the left partial image B.

FIG. 15 is a diagram showing a waveform of a signal composed of an image data sequence constituting a band-shaped figure Rc generated by performing a linear weighted averaging process on the signal waveforms Sa and Sb in FIG. 10C. is there. A waveform Sa indicated by a solid line is a waveform of a signal composed of an image data sequence constituting the band graphic Ra, and a waveform Sb represented by a broken line is a waveform of a signal composed of an image data sequence constituting the band graphic Rb. A waveform Sc indicated by a bold line is a waveform of a signal composed of an image data sequence constituting a band-shaped figure Rc generated by the linear weighted averaging process.

In the weighted average processing, when the left partial image A and the right partial image B are pasted together at the boundary, the image data at the boundary for synthesis is reduced so that the density difference between the two images is reduced at the boundary. Because it is generated, the left partial image A
When the luminance difference between the image B and the right side image B is large, as shown in FIG. 15, the signal Sc composed of the image data sequence constituting the band-shaped figure Rc included in the boundary portion for synthesis becomes Under the influence, the signal level is reduced as compared with the ideal signal waveform (see FIG. 14), and the shape of the waveform is largely distorted. In particular, if the overlapping size of the boundary portion between the left partial image A and the right partial image B is sufficiently large with respect to the pixel pitch, the image data of the central portion of the image of the boundary portion after the combination becomes the image data of the partial image A Since the component ratio between the data and the image data of the partial image B becomes substantially equal (that is, a simple average value of the two image data), the peak of the signal Sc composed of the image data sequence constituting the band image Rc is further lower. This causes a problem that the density of the belt-shaped image Rc becomes lower.

Therefore, in the present embodiment, when generating the image of the boundary portion for synthesis, the edge reduction processing and the smoothing processing are used together to suppress the density reduction and the shape change of the synthesized band-shaped image Rc. ing. Therefore, the image synthesizing circuit 236 includes a level adjusting circuit 236a, a weighted averaging circuit 236b, an edge emphasizing circuit 236c, a smoothing circuit 236d, and an image pasting circuit 236e. Then, an image of the boundary portion for synthesis is generated.

FIG. 17 is a waveform diagram for explaining a process of generating an image of a boundary portion for synthesis using both the edge enhancement process and the smoothing process. FIG. 17 shows the state shown in FIG.
5A shows waveforms of various signals in the process of generating an image of a boundary portion for synthesis in the case of FIG. FIG. 4B is a waveform diagram of a signal Sc obtained by adjusting the level of the signal Sb composed of columns, FIG. 4B is a waveform diagram of a signal Sc obtained by weighted averaging the signal Sa and the signal Sb after the level adjustment, and FIG.
FIG. 4D is a waveform diagram of a signal Sc ′ in which the edge of c is emphasized, and FIG. 4D is a waveform diagram of a signal Sc ″ in which a protruding edge of the signal Sc ′ is smoothed.

In generating an image of the boundary portion for synthesis, the image synthesis circuit 236 firstly generates the level adjustment circuit 2
At 36a, the levels of the signal Sa composed of the image data sequence constituting the band graphic Ra and the signal Sb composed of the image data sequence constituting the band graphic Rb are adjusted (see the state of FIG. 17A). That is, the level average value Va of the image data group forming the boundary portion of the left partial image A and the level average value Vb of the image data group forming the boundary portion of the right partial image B are calculated, and both level average values Va are calculated. , Vb average value Vab
(= (Va + Vb) / 2) is calculated. Then, the correction value ΔVa (= | Vab−Va |) is subtracted from the level value of each image data constituting the boundary portion of the left partial image A, and the signal S is obtained.
The level of the signal Sb is reduced overall, and the correction value ΔVb (= | Vab−Vb |) is added from the level value of each image data constituting the boundary portion of the right partial image B to reduce the level of the signal Sb as a whole. To adjust the levels of both signals Sa and Sb.

Subsequently, the weighted averaging circuit 236b performs the above-described weighted average calculation using the image data forming the boundary portion of the left partial image A and the image data forming the boundary portion of the right partial image B after the level adjustment. Then, an image of a boundary portion for synthesis (hereinafter, this image is referred to as a boundary image for synthesis) is generated. By this weighted averaging process, a band-shaped figure Rc having the waveform Sc of FIG. 17B is generated in the synthesis boundary image.

Subsequently, assuming that a function representing the boundary image for synthesis in the edge enhancement circuit 236c is f (i, j), f (i, j) −∇ ²
The edge processing is performed by performing the arithmetic processing of f (i, j). That is, specifically, edge emphasis filtering processing is performed on the image data constituting the synthesis boundary image using, for example, a Laplacian filter shown in FIG. In this filtering process, assuming that the synthesis boundary image is composed of nxm image data, the pixel position (i,
j) (i = 1, 2,... n, j = 1, 2,... m) is represented by g (i, j) ′ = ｛4 g (i, j) −g (i -1, j) -g
(i, j-1) -g (i, j + 1) -g (i + 1, j)} / 4. As a result of this filtering process, the waveform of the signal Sc composed of the image data sequence constituting the band-shaped figure Rc included in the boundary image for synthesis changes as a signal Sc 'shown in FIG.

In the present embodiment, a Laplacian filter having a size of 3 × 3 is used, but the Laplacian filter may be of another size. Further, the present invention is not limited to the Laplacian filter, and a wide-range emphasis filter having a coefficient similar to Laplacian capable of edge emphasis may be used.

Subsequently, the smoothing circuit 236d performs a smoothing process on the boundary image for synthesis after the edge enhancement. Specifically, for example, two edges generated at both ends of the peak of the signal Sc 'by the edge enhancement using the smoothing filter shown in FIG.
A filtering process for smoothing two peaks and two valleys generated at the foot of the signal Sc 'is performed. In this filtering process, the image data g (i, j) of the synthesis boundary image is replaced with image data g (i, j) ″ calculated by the following equation.

[0090]

(Equation 2)

The waveform of the signal Sc 'composed of the image data sequence constituting the band-shaped figure Rc included in the boundary image for synthesis included in the boundary image for synthesis changes as a signal Sc "in FIG. 17D.

In this embodiment, a 3 × 3 size smoothing filter is used, but the size of the smoothing filter may be other. Further, the smoothing filter is not limited to the one shown in FIG. 19, and may be, for example, a filter that simply averages image data of adjacent pixels.

As described above, when the above-described edge emphasis processing and smoothing processing are performed on the signal Sc composed of the image data constituting the band-shaped figure Rc generated by the linear weighted average calculation, FIGS. ), The signal Sc
Is changed, and a signal Sc ″ close to the signal Sc having the ideal signal waveform shown in FIG. 14 can be obtained.

However, the edge emphasizing process and the smoothing process shape the waveform of the signal Sc composed of the image data sequence constituting the band-shaped figure Rc generated by the linear weighted average calculation into an ideal waveform. The waveform of the signal Sc ″ after shaping is greatly affected by the waveform of the original signal Sc. The waveform of the signal Sc is obtained by converting the signal Sa and the band graphic Rb, which are composed of the image data sequence, that constitute the band graphic Ra. Since the waveform greatly changes depending on the degree of deviation of the waveform from the signal Sb comprising the image data sequence to be composed, the waveform of the signal Sc ″ ultimately changes greatly depending on the amount of deviation between the signal Sa and the signal Sb.

In particular, when the peak of the signal Sa and the peak of the signal Sb are separated (that is, the left partial image A and the right partial image B
20 (a) and 20 (b), the band-shaped figure Rc generated by the linear weighted averaging operation, as shown in FIGS.
, The peak level of the signal Sc composed of the image data string constituting the image data sequence decreases, and the waveform thereof also becomes a gentle mountain shape. on the other hand,
If the peak of the signal Sa and the peak of the signal Sb are close to each other (that is, if the deviation between the left partial image A and the right partial image B is small), as shown in FIGS. 21 (a) and 21 (b), The peak level of the signal Sc is substantially the same as the signal Sa (or the signal Sb), and its waveform is also a rectangle close to the signal Sa.

Therefore, in order to reduce the influence of the shift amount between the left partial image A and the right partial image B in the waveform shaping by the edge enhancement processing and the smoothing processing, for example, the pixel position (i, j) is set for each pixel position. weighted average calculation after the image of the image data g3 of (i, j) 2-order differential value ∇ ² g3 to be added to (i,
j) is converted into image data g1 (i, j) of the left partial image A and image data g2 (i, j) of the right partial image B at the pixel position (i, j).
Should be changed according to the level difference Δg (= | g1 (i, j) −g2 (i, j) |). Specifically, when the level difference Δg is 0 or small, the second derivative ∇ ² g3 (i,
j) is set to 0 or the smile amount, and the second derivative ∇ ² g3 (i, j) is increased as the level difference Δg increases.

In this manner, the peak of the signal Sa and the signal Sb
20 (ie, when the deviation between the left partial image A and the right partial image B is large)
As shown in (c), the signal Sc 'is obtained by the edge enhancement processing.
When the amount of protrusion of the edges at both ends of the peak and the amount of dip in the valley at the bottom of the peak are large and the peak of the signal Sa and the peak of the signal Sb are close to each other (that is, the left partial image A and the right partial image B When the deviation is small), as shown in FIG.
As a result of the edge emphasis processing, the amount of protrusion of the edge at both ends of the peak of the signal Sc 'and the amount of dip in the valley at the bottom of the peak are reduced.
As shown in (d) and FIG. 21 (d), the waveform and peak level of the signal Sc ″ can be set to a substantially ideal state (the state of the signal Sc shown in FIG. 17).

The control for changing the ^second derivative ∇ ² g3 (i, j) may be performed for each pixel position as described above, or may be performed for each block using a predetermined number of pixels. In the above description, the case where the left and right partial images A and B are combined has been described. However, when the upper and lower partial images are combined, the combining process can be performed in a similar manner.

Referring back to FIG. 7, the post-processing circuit 237 performs correction processing such as edge enhancement, smoothing, and color unevenness correction on the entire image synthesized by the image synthesis circuit 236. Further, the encoding circuit 238 encodes image data constituting the combined whole image. The encoding circuit 238 is, for example, DCT (discrete cosine transform)
The encoding process (compression process) of the image data is performed by the JPEG compression method which combines the Huffman encoding with the image data. Then, the image data output from the encoding circuit 238 is stored in the HD card 10.

Next, the photographing operation of the digital camera 1 will be described with reference to the flowcharts of FIGS.

FIG. 22 and FIG. 23 are flow charts showing the photographing operation procedure. When the power switch 7 is turned on and the digital camera 1 is started, setting or changing of a shooting mode, a shooting condition, and the like are performed according to the setting states of the high resolution mode setting button 13 and the number of synthesized images setting switch 14 (# 2). ). Further, the imaging unit 2 is driven in the video mode,
The captured image is displayed on a monitor (finder display) on the LCD display unit 11, whereby the image can be captured (#).
4). In this state, the shutter button 6 is fully pressed,
The process is continued until the release is instructed (# 2 to # 6 loop). If the photographer operates the high resolution mode setting button 13 or the composite image number setting switch 14 during this time, the photographing mode and photographing conditions are changed according to the operation (# 2).

When the shutter button 6 is fully depressed and a release is instructed (YES in # 6), it is determined whether or not the photographing mode is set to the high resolution mode (# 8). Is not set (#
No in step 8), the process proceeds to step # 10, and a shooting process in a normal shooting mode is performed.

That is, first, the photographing direction (the direction of the optical axis L) of the image pickup unit 2 is set to the front direction (# 10), and the photometric unit 3 detects the subject brightness (# 12).
The subject distance is detected by the distance measuring unit 4 (# 14), and the focus is adjusted based on the subject distance (# 16).
Further, an exposure control value (aperture value and integration time of CCD 203) is set using the detected subject luminance (# 1).
8).

Subsequently, the subject is imaged based on the set exposure control value (# 20). An image signal constituting a captured image output from the imaging unit 2 is output from the A / D conversion unit 2
2, the pre-photographed image storage area 23 of the image memory 231 in the image processing unit 23 is converted to digital image data.
1a, and output to the LCD drive control unit 24 via the image memory 231.
1 is displayed as a still image (# 22). With this still image display, the photographer can monitor the photographed image.

The image data stored in the image memory 231 is subjected to correction processing such as edge enhancement, smoothing, color correction, etc.
8, the encoding process is performed (# 24), recorded on the HD card 10 (# 26), and the photographing process ends.

When the photographing process is completed, it is determined whether or not the photographer has operated the still image display release button 12 to instruct release of the still image display (# 28), and an instruction to release the still image display is issued. If YES (YES in # 28), the process returns to step # 2 to perform the next photographing process. If the instruction to cancel the still image display has not been issued (NO in # 28), the instruction to cancel the still image display is issued. Until then, the still image display of the captured image is continued (loop of # 28).

If the high-resolution mode has been set in step # 8 (YES in # 28), the flow shifts to step # 30, and the photographing process in the high-resolution mode is performed. In the description of the photographing process in the high resolution mode, a case where a subject shown in FIGS. 4 and 5 is divided into four and partial images are taken will be described as an example.

First, the photographing direction (direction of the optical axis L) of the image pickup unit 2 is set to the front direction (# 30), and the zoom lens 202 of the image pickup unit 2 is set to a predetermined wide position (# 3).
2). Subsequently, the luminance of the subject is detected by the photometry unit 3 (# 34), the subject distance is detected by the distance measurement unit 4 (# 36), and the focus is adjusted based on the subject distance (# 38). Further, an exposure control value (aperture value and integration time of CCD 203) is set using the detected subject brightness (# 40).

Subsequently, the subject is imaged based on the set exposure control value (# 42). This captured image is
The entire subject is photographed, and has a screen configuration substantially the same as the photographed image after the synthesis. The image signal constituting the captured image output from the imaging unit 2 is converted into digital image data by the A / D conversion unit 22 and then stored in the pre-captured image storage area 231 a of the image memory 231 in the image processing unit 23. The image memory 231 is stored (# 44).
Is output to the LCD drive control unit 24 via the CPU and displayed on the LCD display unit 11 as a still image (# 46). By displaying the still image, the photographer can monitor the photographed image (that is, an image that is substantially the same as the entire image after the combination).

Subsequently, using the image data stored in the image memory 231, the AE / AWB calculation unit 26 calculates an exposure control value and a WB adjustment value in each shooting operation when a partial image is captured (# 48). ). The exposure control value is calculated by calculating an average level of all image data or a part of image data of the entire image of the subject initially photographed, and using the average level. The WB adjustment value is calculated for each of the R, G, and B color components by calculating the average level of all or part of the image data of the entire subject and calculating the R, G, and B values.
Is calculated as a correction coefficient for equalizing the average level of each component. The exposure control value and the WB adjustment value may be set manually.

Subsequently, the photographing direction of the image pickup unit 2 is divided into the number of combined images set by the number-of-combined-images setting switch 14, and the first photographing direction when partially photographing the subject (the direction a in FIG. 5). ) (# 50), and the telephoto position at which the zoom lens 202 of the imaging unit 2 has a predetermined photographing magnification (in the example of FIG. 5, the number of divisions is four, and the photographing magnification is approximately twice). Position) (# 52). Further, when the exposure control value is changed to the exposure control value for the first photographing direction calculated in step # 48 (in the example of FIG. 5, the exposure control value for the upper left subject portion toward the subject), The WB adjustment value is set to the WB adjustment value for the partial image that is captured first (# 54).

Subsequently, the upper left portion of the subject is imaged based on the set exposure control value (# 56). Then, the image signal constituting the captured image (partial image) output from the imaging unit 2 is converted into digital image data by the A / D conversion unit 22 and then converted to a portion of the image memory 231 in the image processing unit 23. It is stored in the image storage area 231b (# 5
8).

Subsequently, it is determined whether or not the capture of all the partial images has been completed (# 60). If the capture has not been completed (NO in # 60), the flow returns to step # 50 to capture the next partial image. . In this case, since the capture of the first partial image has only been completed, the flow returns to step # 50, and the imaging direction of the imaging unit 2 is changed to the second imaging direction (direction c in FIG. 5).
(# 50), the zoom lens 202 of the imaging unit 2 is set to a telephoto position (a position almost doubled) at a predetermined photographing magnification (# 52), and an image of the upper right portion of the subject is taken. (# 54 to # 58). Hereinafter, in the same procedure, the lower right and lower left portions of the subject are imaged, and when the capturing of all the partial images is completed (YES in # 60), according to the flowchart of the "combining and combining process" shown in FIG. By combining the four partial images so as to be bonded together, a captured image of the entire subject is generated (# 62).

That is, first, an interpolation process is performed on the pre-photographed image by the image enlargement circuit 232 to generate an enlarged image enlarged to four times the size, and the pre-processing circuit 233 generates image data constituting each partial image. Are subjected to pre-processing such as brightness correction, noise removal, smoothing, and edge enhancement, and these images are input to the matching calculation circuit 234 (# 70).

Subsequently, the matching operation circuit 234 compares the enlarged image with each of the partial images to calculate the amount of matching conversion (the amount of geometric conversion to a matching position on the enlarged image) (# 72). The geometric image conversion circuit 235 performs a geometric conversion process (conversion of each partial image to a position corresponding to the enlarged image. The processing shown in FIG. ) Is performed (# 74).

Subsequently, with the level adjustment circuit color 236a of the image synthesizing circuit 236, the brightness adjustment (the processing of FIG. 17 (a)) of the boundary portion (overlapping portion) where the pasting process of each partial image is performed is performed. Thereafter (# 76), the weighted averaging circuit 236b performs weighted averaging of the image data constituting the boundary portion between the two partial images with a predetermined weighting coefficient to generate image data of the boundary portion for synthesis (# 78, FIG. 17). (Process of (b)). Further, the edge enhancement circuit 236c and the smoothing circuit 236d perform edge enhancement processing (addition processing of the second derivative) and smoothing processing on the image data of the boundary portion for synthesis (# 80, # 82).

Then, the image combining circuit 236e combines the image data of the parts other than the boundary part of each partial image and the image data of the boundary part for combination so as to simply connect them to generate a composite image of the entire subject. After the post-processing circuit 237 performs various correction processes such as edge enhancement, smoothing, and color unevenness correction on the composite image (# 84), the process returns.

After the encoding (compression) processing is performed on the image data of the synthesized whole image by the encoding circuit 238 (# 64), the image data is recorded on the HD card 7 (# 66). ), The photographing process ends. When the photographing process is completed, it is determined whether or not the photographer has operated the still image display release button 12 to instruct the release of the still image display (# 68). (#
(YES at 68), step # 2 for performing the next photographing process
And if the instruction to cancel the still image display is not given (#
(NO at 68), the still image display of the captured image is continued until the cancellation of the still image display is instructed (loop of # 68).

FIG. 25 is a diagram showing a second embodiment of the block configuration relating to image processing in the high brightness mode in the image processing section.

In the first embodiment, each partial image is deformed by calculating a matching conversion amount that substantially matches the enlarged image for each partial image. In the second embodiment, the partial image is transformed. Only the combination position in the enlarged image is calculated for each image, and the partial images are combined at the boundary portion while being arranged at the combination position in the enlarged image, thereby simplifying the image combination process. That is, in the second embodiment, the synthesis processing of each partial image is performed without correcting perspective image distortion, and the processing is simplified to speed up the synthesis processing. When the subject distance is long to some extent, the swing angle of the optical axis L of the imaging unit 2 from the front direction when capturing the partial image is small, and perspective perspective image distortion occurring in each partial image is reduced. In addition, a high-resolution composite image can be effectively obtained in the second embodiment under photographing conditions.

Accordingly, the block diagram shown in FIG.
In FIG. 7, the geometric image conversion circuit 235 is replaced with a level correction circuit 239.

In FIG. 25, level correction circuit 239
Is to correct the level of the partial image based on the luminance difference between the partial image at the composite position of the enlarged image and the partial image corresponding to the composite position. Since the image combining circuit 236 combines the partial images and finally generates an image substantially the same as the enlarged image, the luminance of each partial image is combined with the enlarged image, and the luminance difference between the partial images is conspicuous after the combination. This is to prevent it. Note that the matching calculation circuit 234 in FIG. 25 only calculates a region where each partial image matches on the enlarged image, as described later.
Each partial image is not deformed to calculate the best matching area on the enlarged image.

The second embodiment differs from the first embodiment mainly in the operation of the matching operation circuit 234 and the level correction circuit 239, and therefore, in the following description, both circuits will be mainly described. The operation will be described.

The matching calculation circuit 234 compares the enlarged image with each partial image, moves each partial image up, down, left, and right at a predetermined pixel pitch on the enlarged image, and determines a position where the two images are most coincident with each other at the combined position. Is calculated as In this case, the degree of coincidence between the enlarged image and the partial image is determined by the absolute value | Δ of the luminance difference between the pixel data constituting the enlarged image and the pixel data constituting the partial image corresponding to the pixel position of the pixel data.
Is determined by the sum Σ | ΔDi | of Di | (i = 1, 2,..., N), and the position at which the sum Σ | ΔDi | is minimum is calculated as the combined position. Note that the movement range of each partial image in the enlarged image is roughly known in advance in the enlarged image (the range of the upper right corner, the lower left range, and the like). Is determined in consideration of the relative displacement of the partial images due to

When the combined position of each partial image is determined, the sum of the luminance differences at that position Σ | ΔDi | is divided by the total number n of pixels constituting the partial image, and the luminance correction value per pixel Σ | ΔDi | / n is calculated and the level correction circuit 2
At 39, the luminance correction value Σ | ΔDi | / n is added to the pixel data constituting the partial image to correct the luminance difference between the enlarged image and each partial image. By this luminance correction, the pasted portion (boundary portion) in the pasting and combining process of each partial image
, The luminance step is reduced.

FIG. 26 is a flowchart showing the calculation processing of the combined position according to the second embodiment. The flowchart of FIG. 26 is a process corresponding to the matching calculation process in step # 72 of FIG. Therefore, if the processing of step # 72 is replaced with the calculation processing of the combining position shown in FIG. 26, the simplified combining processing can be performed by the photographing operation in the high resolution mode according to the flowcharts of FIGS.

[0127] In the arithmetic processing of the synthesis positions shown in FIG. 26, first, the pixels constituting the partial image G _A from the partial image storage area 231b of the image memory 231 data ga (i) are sequentially with read (# 90), Matching operation circuit 23
4, corresponding pixel data g (i) in a predetermined comparison range of the enlarged image G stored in a memory (not shown) is sequentially read (# 92), and both pixel data g a (i) and g (i) are read. )), The sum of the luminance differences | ΔDi | = | ga (i) −g (i) |
Di | (i = 1, 2,..., N is the total number of pixel data constituting the partial image) is calculated (# 94).

Subsequently, it is determined whether the calculated sum 総 | ΔDi | is smaller than the minimum value minΣ of the sum calculated up to the previous time (# 96), and if minΣ> Σ | ΔDi | (# 96), the total sum 今 回 | ΔDi |
Is updated and stored as a minimum value of the sum in a memory (not shown) of minΣ provided in the matching operation circuit 234 (# 98), and the position of the comparison range of the partial image G _A in the enlarged image G is determined as the synthesis position. Matching operation circuit 2
The data is stored in a memory (not shown) in the storage 34 (# 100). On the other hand, if minΣ ≦ Σ | ΔDi | (NO in # 96), mi
The process proceeds to step # 102 without changing the contents of the memory at nΣ and the combining position.

In step # 102, it is determined whether or not the calculation of the sum of the absolute values of the luminance differences Σ | ΔDi | has been completed for all the comparison ranges.
O), the comparison range is moved by a predetermined moving amount (for example, one pixel to several pixels) in a predetermined moving direction (for example, up and down direction or left and right direction) (# 104), and the process returns to step # 92.
Then, the corresponding pixel data g (i) is successively in the comparison range after movement of the enlarged image G, read, the absolute value of the luminance difference between the image data of the partial image G _A | [Delta] Di | sum sigma | delta
Di | is calculated, the sum {| ΔDi | is compared with min}, and the above-described predetermined processing is performed according to the comparison result (# 92 to # 100). Hereinafter, similarly, the comparison range is sequentially changed,
The position of the comparison range (that is, the synthesis position) where the total sum of the luminance differences Σ | ΔDi | in the enlarged image G from the partial image G _A is changed is calculated (the loop of # 92 to # 104).

When the combined position is calculated for partial image G _A (YES in # 102), the sum of luminance differences Σ |
[Delta] Di | minimum value Minshiguma divided by the total number n of the image data constituting the partial image G _A luminance correction value Minshiguma / n is calculated, this calculation result is input to the level correction circuit 239 (# 106).

Subsequently, it is determined whether or not the calculation of the combined position has been completed for all the partial images (# 108). If the calculation has not been completed (NO in # 108), the process returns to step # 90 and the next partial image is obtained. Is calculated (loop of # 90 to # 104). In this case, since the process is for the first partial image G _A , step # 90
Returning to, calculation processing of the combined position for the next partial image G _B is performed. When the calculation process of the combined position and the luminance correction value minΣ / n for all partial images G _A ~G _D is completed (YES in # 108), and terminates the calculation processing of the combined position.

The information on the combination position calculated by the matching calculation circuit 234 is input to the image combination circuit 236, and the image data G _{A to} G _D constituting each partial image are converted by the level correction circuit 239 into the luminance correction value minΣ / After adding and correcting n,
It is input to the image synthesis circuit 236. And after this,
According to the process of step # 74 in FIG. 24, the partial images G _{A to} G _D are combined so as to be pasted at the boundary, and the entire image is combined.

In the above-described matching calculation processing, all the image data constituting the partial image is used. However, the processing is performed by using the image data which is thinned out every block or every few pixels to reduce the number of pixels. The reduction in speed may be suppressed.

In the above embodiment, the same CCD 2 is used by changing the optical axis direction of the photographing lens 202 of the image pickup section 2.
03, a plurality of partial images are continuously taken in. However, the subject light image is divided into the entire structure of the subject and a plurality of partial light images by a spectral means such as a prism, and the light images are divided into a plurality of CCDs. May be formed on the image pickup surfaces of the respective objects, and an image of the entire subject and an image of a part of the subject may be captured at the same time.

As described above, when the subject is divided into a plurality of parts and photographed, and the partial images are combined to generate a photographed image of the entire subject, an image of the entire subject is captured when the partial image is captured. Since an image obtained by enlarging the photographed image at a predetermined magnification based on the number of divisions of the object so as to match the photographed image of the entire object is generated, and the synthesized position of each partial image is determined based on the enlarged image. In addition, the combining position of each partial image is accurately set, the combining process can be performed accurately, the image displacement and the color displacement in the bonding process at the boundary portion can be prevented, and the image quality deterioration due to the combining process is reduced. be able to.

[0136]

As described above, according to the present invention,
An image capturing apparatus that combines a plurality of partially captured images to generate a captured image of the entire subject, and that captures the entire subject, partially captures the image, and combines captured images of the respective portions of the subject. Since the combination position in the processing is calculated based on the photographed image of the entire subject, and the respective partial images are combined while being aligned with the combination position, the combination position of each partial image is accurately determined, and the combination processing is performed. Can be performed accurately.

Further, according to the present invention, the entire subject is divided into a plurality of portions so that the boundary portions are overlapped, and each portion of the subject is enlarged to substantially the same size as the entire subject and sequentially imaged. The captured image of the entire subject is generated by combining the captured images of the respective portions of the subject so as to be pasted together at the boundary portion in a state where the captured images are aligned with the composite position calculated based on the captured image of the entire subject. Therefore, the accuracy of the combining position of each partial image is improved, and the combining process can be performed more accurately.

The first imaging means for imaging the entire subject and the second imaging means for partially imaging the subject are provided with a lens capable of changing the magnification and a subject light image converted to an electric image by photoelectric conversion. An image pickup device and an image pickup device are integrally configured so as to be capable of changing the optical axis direction. The optical axis direction and the photographing magnification of the image pickup unit are changed so that the entire image of the subject and the partial image of the subject are changed. Because I tried to take in sequentially,
The structure of the imaging system for capturing the whole image and the partial image of the subject becomes compact. In addition, the difference in image quality due to the difference in the photographing optical system between the image of the entire subject and the image of the part of the subject is small, so that the synthesizing process is facilitated and the reduction in the image quality of the synthesized image is suppressed. Can be.

[Brief description of the drawings]

FIG. 1 is a diagram showing an appearance of a digital camera according to an embodiment of the present invention.

FIG. 2 is a rear view of the digital camera according to the present invention.

FIG. 3 is a diagram showing a schematic configuration of an imaging system of the digital camera according to the present invention.

FIG. 4 is a diagram for explaining a shooting method in a high resolution mode.

FIG. 5 is a diagram illustrating a relationship between a subject and a shooting range in shooting in a high resolution mode.

FIG. 6 is a diagram showing an embodiment of a block configuration of a digital camera according to the present invention.

FIG. 7 is a diagram illustrating a first embodiment of a block configuration related to image processing in a high resolution mode in an image processing unit.

FIG. 8 is a diagram illustrating a photographing range at each photographing when a bulletin board is divided into four parts and partly photographed.

FIG. 9 is a diagram illustrating perspective image distortion occurring in a partial image.

FIG. 10 is a diagram showing an image obtained by pasting and combining partial images having perspective image distortion.

FIG. 11 is a diagram illustrating a method of a matching process.

12A and 12B are diagrams illustrating directions of enlargement or reduction with a straight line as an axis, and FIG. 12A is a diagram illustrating enlargement or reduction in a linear direction;
(B) is a figure which shows expansion or contraction in an arc direction.

FIG. 13 is a diagram showing an example of a synthesizing method when a difference in luminance level and a pattern shift occur in an image of a boundary portion;
(A) is a diagram showing a partial image on the left side, (b) is a diagram showing a partial image on the right side, and (c) is a diagram showing a combined image of a boundary portion between both partial images.

FIG. 14 shows a band-shaped figure R on a straight line M in FIG.
An ideal band figure Rc obtained by properly combining the image data forming the a and the image data forming the band figure Rb.
FIG. 5 is a diagram showing a waveform of an image data sequence constituting the image data.

FIG. 15 shows a band-shaped figure R on a straight line M in FIG.
FIG. 9 is a diagram showing a waveform of an image data sequence constituting a band-shaped figure Rc obtained by linearly weighting and averaging the image data constituting a and the image data constituting the band-shaped figure Rb.

FIG. 16 is a waveform chart for explaining a method of synthesizing images by linear weighted averaging processing.

FIGS. 17A and 17B are waveform diagrams for explaining a generation process of an image of a boundary portion for synthesis using both the edge enhancement process and the smoothing process. FIG. 17A is a signal Sa including an image data sequence constituting a band-shaped graphic Ra. A waveform diagram in which the level of a signal Sb composed of an image data sequence constituting a band-shaped figure Rb is adjusted;
(B) is a waveform diagram of a signal Sc obtained by weighted averaging the signal Sa and the signal Sb after level adjustment, (c) is a waveform diagram of a signal Sc 'in which edges of the signal Sc are emphasized, and (d) is a waveform diagram of the signal Sc'. FIG. 14 is a waveform diagram of a signal Sc ″ obtained by smoothing a protruding edge.

FIG. 18 is a diagram illustrating an example of a filter for edge enhancement processing.

FIG. 19 is a diagram illustrating an example of a filter for smoothing processing.

FIG. 20 is a waveform diagram for explaining a process of generating an image of a boundary portion for synthesis using both the edge enhancement process and the smoothing process when the displacement between the left and right partial images is large;
(A) is a waveform diagram in which the level of a signal Sa composed of an image data sequence constituting the band graphic Ra and a signal Sb composed of an image data sequence constituting the band graphic Rb is adjusted, and (b) is a signal Sa after the level adjustment. (C) is a signal Sc 'in which edges of the signal Sc are emphasized.
(D) is a waveform diagram of the signal Sc ″ obtained by smoothing the protruding edge of the signal Sc ′.

FIG. 21 is a waveform diagram for explaining a generation process of an image of a boundary portion for synthesis using both the edge enhancement process and the smoothing process when the shift between the left and right partial images is small;
(A) is a waveform diagram in which the level of a signal Sa composed of an image data sequence constituting the band graphic Ra and a signal Sb composed of an image data sequence constituting the band graphic Rb is adjusted, and (b) is a signal Sa after the level adjustment. (C) is a signal Sc 'in which edges of the signal Sc are emphasized.
(D) is a waveform diagram of the signal Sc ″ obtained by smoothing the protruding edge of the signal Sc ′.

FIG. 22 is a flowchart illustrating shooting control of the digital camera according to the present invention.

FIG. 23 is a flowchart showing photographing control of the digital camera according to the present invention.

FIG. 24 is a flowchart illustrating a processing procedure of a “sticking process”.

FIG. 25 is a diagram illustrating a second embodiment of a block configuration related to image processing in a high resolution mode in the image processing unit.

FIG. 26 is a flowchart illustrating a processing procedure of “composition position calculation”.

[Explanation of symbols]

 Reference Signs List 1 digital camera (imaging device) 2 imaging unit (first and second imaging means) 201 housing 202 shooting lens 203 imaging device (CCD) 3 photometry unit 4 distance measurement unit 5 flash 6 shutter button 7 power switch 8 card insertion Mouth 9 Card ejection button 10 HD card 11 LCD display unit 12 Still image display release button 13 High resolution mode setting switch 14 Number of combined images setting switch 15, 16, 17 Electric motor 20 Control unit 21 Imaging control unit 211 Zoom control circuit 212 Scanning Control circuit (optical axis changing unit) 213 Imaging control circuit (first and second imaging control units) 22 A / D conversion unit 23 Image processing unit 231 Image memory (storage means) 232 Image enlargement circuit 233 Preprocessing circuit 234 Matching Arithmetic circuit (combined position calculating means) 235 Geometric image conversion circuit 23 Image combining circuit (image combining means) 236 Image combining circuit 236a Level adjustment circuit 236b Weighted average circuit 236c Edge enhancement circuit 266d Smoothing circuit 236e Image combining circuit 237 Post-processing circuit 238 Encoding circuit 239 Level correction circuit 24 LCD drive control unit 25 Card drive control unit 26 AE / AWB operation unit 27 RAM 28 ROM

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁶ Identification code FIG06F 15/66 470J

Claims (3)

[Claims] A first imaging unit that images the entire subject; a second imaging unit that divides the entire subject into a plurality of portions and sequentially captures images; and a subject that is imaged by the first imaging unit. Combining position calculating means for calculating a combining position in combining the partial images of the subject imaged by the second image capturing means on the basis of the entire captured image; and calculating each of the partial images of the subject by the combining position calculating means An image synthesizing means for synthesizing each of the partial images in a state of being aligned with the synthesized position. A first imaging unit for imaging the entire subject; dividing the entire subject into a plurality of portions so that a boundary portion is overlapped; each portion of the subject having substantially the same size as the imaging size of the entire subject; The second to take images sequentially after expanding to the size
Image capturing means, storage means for storing images captured by the first and second image capturing means, and second image capturing means based on a captured image of the entire subject captured by the first image capturing means A combining position calculating means for calculating a combining position in the combining of the partial images of the subject imaged at the position and the combining position calculated with the combining position calculated by the combining position calculating means. An image pickup apparatus, comprising: an image synthesizing unit that synthesizes partial images so as to be bonded at a boundary portion. 3. The image pickup apparatus according to claim 1, wherein the first and second image pickup means include a lens capable of changing a photographing magnification and an image pickup element that photoelectrically converts a subject light image into an electric image and takes in the electric image. An imaging unit integrally configured so that the axial direction can be changed, an optical axis changing unit that changes the optical axis direction of the imaging unit, and an optical axis direction of the imaging unit set to the front direction, and a shooting screen is displayed. A first imaging control unit that sets the photographic magnification of the lens so that the entire subject is included to image the entire subject, and a photographic magnification of the imaging unit where each part of the subject has a boundary portion overlapping with each other A second imaging control for enlarging the image so as to fit in the imaging screen and changing the optical axis direction of the imaging unit in a predetermined direction corresponding to each part of the object in order to partially image the object; And a department Imaging device.