JP2002112007A - Image processing system, image pick-up device, image processing method, and recording medium recording image processing method therein - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an image processing time to be shortened as much as possible when two picked-up images which are continuously imaged are synthesized into an image high in both quality and effectiveness. SOLUTION: For instance, in image processing of synthesizing an overexposed picked-up image and an underexposed picked-up image into a properly exposed image as a whole, an R and a B color component both having a smaller effect on resolution due to a human being's visual characteristics than a G color component undergo a simpler image processing than the G color component, so that processing time can be shortened. For instance, alignment processing for an R and a B color component image is carried out taking advantage of a misalignment volume calculated for a G color component (#251 to #255). Data are interpolated at the positions of all picture elements for a G color component, then image processing is carried out (#247, #257), and image processing is carried out without interpolating data for an R and a B color component (#259 to #263, #267 to #271).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of combining a plurality of photographed images continuously photographed for the same subject, performing these image processings, and then synthesizing the images. The present invention relates to an image processing system capable of improving image quality and obtaining an image with an enhanced video effect.

[0002]

2. Description of the Related Art Conventionally, there has been proposed a technique for synthesizing an image having a higher image quality and video effect than an original image by using a plurality of images having the same contents and different image qualities.

[0003] For example, a digital still camera has been commercialized which is capable of obtaining a high-resolution photographed image by performing two consecutive photographing operations by minutely displacing an image pickup device and synthesizing both photographed images (JVC). GC-X1 “PIXSTER”, a digital still camera manufactured by the Company. This digital still camera sets two types of exposure control values in accordance with the bright and dark parts of a subject, performs two consecutive shots with those exposure control values, and combines both shot images. It also has a function that can obtain a photographed image with an appropriate density from dark to bright parts.

[0004]

In recent years, in the technical field of digital still cameras, with the spread of digital still cameras, as seen in the above-mentioned digital still camera "GC-X1" PIXSTER "manufactured by Victor Company,
Adds a digital still camera with a function to improve the image quality or obtain an image with enhanced video effect by combining a plurality of continuously shot images after performing predetermined image processing and combining them New technologies are being developed to increase value.

[0005] However, in recent digital still cameras, the number of pixels has been increased, and when an attempt is made to provide a function of synthesizing a plurality of captured images, the processing time of image processing becomes longer as data increases. As a result, there arises a problem that the photographing process becomes longer as a whole. For example, when a high-gradation image is obtained by combining two color photographed images A and B having different exposure conditions, the color photographed image A and the color photographed image B are aligned so that the photographed subjects overlap each other. It is necessary to perform an image data combining operation for each of the R, G, and B color component images. However, the alignment processing of two images generally searches for a matching position between the two images while performing parallel movement or rotation of the other image at a predetermined pixel pitch with reference to the one image. Since the operation of determining the degree of coincidence with the reference image is repeated by moving the image, this operation takes a long time, and the synthesized image is recorded on a recording medium such as a memory card after the image is captured. There is a problem that the processing time until is long.

[0006] When an image pickup element used in a digital still camera is constituted by a single-plate CCD such as a Bayer type, data is usually stored in pixel positions where data is insufficient for each color component. After performing the interpolation processing, the position alignment processing and the image synthesis processing are performed. Therefore, the processing time is further increased by the increase in the data interpolation processing, and the problem of the processing time is serious.

Therefore, it is conceivable to construct an image processing system in which a digital still camera performs only continuous photographing, and an image processing apparatus constituted by a computer or the like performs a process of synthesizing a plurality of photographed images.
However, even in this case, if the number of data of the original photographed image is enormous, it takes a long time to combine the photographed images in the image processing apparatus, and the processing response is reduced. It is desirable to reduce the time required for image processing.

However, conventionally, when a plurality of color images are combined to obtain a high gradation image, if the color image is composed of, for example, R, G, B color component images, Since the same alignment processing and image synthesis processing are performed, the same processing is repeated three times, and the processing efficiency decreases as the number of data increases.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus which can simplify processing in accordance with a color component. An object of the present invention is to provide a system, a color imaging device, an image processing method, and a recording medium on which a processing program of the method is recorded.

[0010]

According to the present invention, there is provided a color image pickup apparatus for continuously photographing two color images of the same subject so as to obtain an image having a different image quality from a photographed image by image synthesis. Image processing means for performing predetermined image processing for each color component on two color images captured by the imaging device to create one color component image; and image processing means for each color component created by the image processing means. An image processing apparatus comprising: an image synthesizing unit for synthesizing an image to create one color image, wherein the image processing unit includes a human being among the plurality of color component images. (1) The processing of an image of a first color component having a small effect on resolution in terms of visual characteristics is simplified compared to the processing of an image of another second color component.

Specifically, the predetermined image processing for each of the color components is performed by performing a positioning process for performing positioning between photographed images and a predetermined operation relating to image quality between the photographed images having been subjected to positioning. Wherein the image processing means simplifies the alignment processing for the image of the first color component as compared with the alignment processing for the image of the second color component. (Claim 2). For example, the image processing means performs the alignment processing on the image of the first color component using the positional deviation information calculated in the alignment processing on the image of the second color component, thereby obtaining the first color component. This simplifies the registration process for the color components.

Further, in the predetermined image processing for each component,
The image processing means includes an interpolation process for increasing the number of original image data by interpolating data for each captured image before the positioning process, and the image processing means performs an interpolation process on the image of the first color component in a second process. This is more simplified than the interpolation processing for the color component (claim 4). Specifically, the image processing means sets the data interpolation number of the image of the first color component in the interpolation processing to the second By making the number smaller than the number of data interpolations of the color component image, the interpolation process for the first color component is simplified.

If the image processing includes interpolation processing, the image processing means may determine that the number of data of the first color component image is equal to the number of data of the second color component image in the arithmetic processing. The number of data of the image of the first color component may be interpolated so as to match.

The above color components are represented by R (red), G (green),
Assuming that the color component is B (blue), the first color component is R, B
The second color component is preferably a G color component (claim 7), and the color components are Y (luminance), Cr (red color difference), and Cb (blue color difference) color components. Then, it is preferable that the first color component is a Cr component and a Cb component, and the second color component is a Y component.

[0015] The predetermined arithmetic processing relating to the image quality includes a blur adjustment processing for adjusting the degree of defocus of the image, a gradation adjustment processing for adjusting the gradation of the image, or a super-resolution processing for increasing the resolution of the image. (Claims 9 to 1).
1).

According to the present invention, an image pickup apparatus has the same function as that of the image processing system.
19).

Furthermore, the present invention provides a method for processing a plurality of color images obtained by continuously imaging the same subject to obtain an image having a different image quality from the photographed image by image synthesis. And an image combining step of combining the images of the respective color components created in the image processing step to create a single color image. The image processing step may include, among the plurality of color component images, a first image having a small effect on resolution in terms of human visual characteristics.
The processing for the image of the second color component is simplified compared to the processing for the image of the other second color component.
0).

In addition, the present invention provides a method for processing a plurality of color images obtained by continuously imaging the same subject to obtain an image having a different image quality from the photographed image by image synthesis. And a second image processing for creating a single color image by combining the images of the respective color components created in the image processing step. A computer-readable recording medium having recorded thereon a program to be executed by a computer, wherein the first image processing is performed on a plurality of color component images among the plurality of color component images having a small effect on resolution in terms of human visual characteristics. The processing for an image of one color component is simplified compared to the processing for an image of another second color component.

According to the above arrangement, two photographed images obtained by continuously photographing the same subject with the color image pickup apparatus are:
After one color component image is created by performing predetermined image processing such as blur adjustment processing, gradation adjustment processing, or super-resolution processing for each color component in the image processing device, the color component images are processed. An image having a different image quality from the captured image is created by combining.

When the two shot images are, for example, a shot image focused on the foreground and a shot image focused on the background, the two-shot image is synthesized by combining the two shot images to obtain, for example, the near view and the background. This is a process for creating images in a focused state appropriate for both. Further, when the two captured images are, for example, a captured image in which exposure control is performed on a bright portion of a subject and a captured image in which exposure control is performed on a dark portion of the subject, the two captured images are combined, For example, this is a process of creating an image having an appropriate density distribution in both a bright part and a dark part of the subject. Further, in the super-resolution processing, when two photographed images have a slight shift in the photographing position with respect to the subject due to camera shake, by combining the two photographed images, the resolution is higher than the photographed image. This is the process of creating an image.

In the predetermined image processing for each color component, among the plurality of color component images, processing is performed on an image of a first color component that has a small effect on resolution in terms of human visual characteristics (hereinafter referred to as a first color component). ) Is simplified compared to the process for an image of another second color component (hereinafter, referred to as a second color component process), whereby the processing time for the first color component process is reduced to the second color component process. The time required for the color component processing is shorter, and the processing time is shorter than when the same processing is performed on all the color component images.

For example, a predetermined image processing for each color component is performed to perform positioning between photographed images, and a predetermined operation relating to image quality such as blur adjustment is performed between the photographed images that have been subjected to the positioning, thereby obtaining one image. If the color component image is composed of R, G, and B, if the color components are R, G, and B, the R and B colors are calculated using the displacement information calculated in the registration process for the G color component image. By performing the alignment process on the image of the component, the processing time of the alignment process on the image of the R and B color components is shorter than the processing time of the alignment process on the image of the G color component. The processing time is shorter than performing the same processing on the image of FIG.

If the color components are Y, Cr, Cb,
By performing the registration process on the Cr and Cb color component images using the displacement information calculated in the registration process on the Y color component image, the registration process on the Cr and Cb color component images is performed. The processing time is shorter than the processing time of the alignment processing for the image of the Y color component, and the processing time is shorter than performing the same processing for all the color component images.

Further, if the image processing includes an interpolation process of interpolating data for each captured image to increase the number of original image data before the alignment process, if the color components are R, G, B,
By making the number of data interpolations of the image of the R and B color components in the interpolation process smaller than the number of data interpolation of the image of the G color component, the processing time of the interpolation process for the image of the R and B color components is G Is shorter than the processing time of the interpolation processing for the image of the color component, and the processing time is shorter than the same processing for the image of all the color components.

If the color components are Y, Cr, Cb,
By making the number of data interpolation of the image of the color component of Cr and Cb in the interpolation process smaller than the number of data interpolation of the image of the color component of Y, the processing time of the interpolation process for the image of the color component of Cr and Cb becomes G Is shorter than the processing time of the interpolation processing for the image of the color component, and the processing time is shorter than the same processing for the image of all the color components.

When the image processing includes interpolation processing, the number of data of the R and B color component images is set so that the number of data of the R and B color component images matches the number of data of the G image. Is interpolated, the image quality of the synthesized image is improved.

[0027]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a front view of a camera body of a digital still camera (color imaging device) applied to an image processing system according to the present invention. FIG. 2 is a view showing main members incorporated in the digital still camera. Top view showing arrangement, FIG. 3
Is a right side view showing an arrangement of main components built in the digital still camera, and FIG. 4 is a rear view of the digital still camera.

The digital still camera 1 includes a camera body 2
A single-lens reflex camera comprising an interchangeable lens 3 comprising a zoom lens which is detachably mounted substantially at the center of the front of the camera body 2. The camera body 2
Mount part 20 to which interchangeable lens 3 is mounted substantially at the center of the front
1 is provided, and a grip portion 202 is provided at the left end of the front.

The battery compartment 2 is provided inside the grip 202.
03 and a card storage room 204 are provided.
03 stores four AA batteries E1 to E4 (power supply batteries of the camera), and a card storage room 204 detachably stores a memory card MC for recording image data of a captured image. It has become.

A plurality of contacts ST for making electrical connection with the mounted interchangeable lens 3 and a plurality of couplers CP for making mechanical connection are provided below the mount portion 201.
Are provided. The electrical contact ST stores information unique to the lens from the lens ROM 301 (see FIG. 3) incorporated in the interchangeable lens 3 (information such as the open F-number and focal length) in the overall control unit (see FIG. 6) and outputs information on the position of the zoom lens and the position of the focus lens in the interchangeable lens 3 to the overall control unit. The coupler CP includes a driving force of a zoom lens driving motor ZM (see FIG. 3) provided in the camera body 2 and a focus lens driving motor FM.
This is for transmitting the driving force (see FIG. 3) to the interchangeable lens 3 side.

When the interchangeable lens 3 is mounted on the mount section 201, a color image pickup element 205 is disposed at an appropriate position in the camera body 2 on the optical axis L of the lens 3.
Color imaging device 205 (hereinafter, referred to as CCD 205)
Is a CCD (Charge-Coupled Device) as shown in FIG.
e) on the surface of each pixel of the area sensor 205A comprising
R (red), G (green), B (blue) color filters 205
B is a so-called Bayer type single-plate type color area sensor which is pasted in a checkered pattern, and has, for example, 1600 × 1200 = 1.92 million pixels.

Note that, as shown in FIG.
The pixel position of the row j column is (i, j) (i = 1, 2,... N,
j = 1, 2,... m), n = 1600, m = 1200
Then, R, G, and B color filters are: R; (2h + 1, 2k + 1) G; (2h + 2, 2k + 1), (2h + 1, 2k + 2) B; (2h + 2, 2k + 2) where h = 0, 1, 2 , ... n / 2 (= 800), k =
, M / 2 (= 600) pixel positions.

A shutter button 206 is provided on the upper surface of the grip portion 202 of the camera body 2. The half-press operation and the full-press operation of the shutter button 206 are detected by switches S1 and S2 described later. When the switch S1 is turned on (when the shutter button 206 is half-pressed), a preparatory operation for photographing a still image of a subject (a preparatory operation such as setting of an exposure control value and a focus adjustment) is performed, and the switch S2 is turned on. When the shutter button 206 is turned on (when the shutter button 206 is fully pressed), a shooting operation (a series of operations of exposing the CCD 205, performing predetermined image processing on an image signal obtained by the exposure, and recording the image signal on the memory card MC) is performed. Done.

An electronic view finder (EVF: Electronic View Finder) 4 and a pop-up type flash 5 are provided substantially at the center of the upper surface of the camera body 2. The electronic viewfinder 4 displays a monitor image of the subject photographed by the CCD 205 (the CCD 2 in a photographing standby state).
A display unit 401 (hereinafter, referred to as L) composed of a color liquid crystal display device for displaying an image of a subject photographed as a moving image according to
This is called a CD display unit 401. ) And an eyepiece 402 for guiding the monitor image displayed on the color liquid crystal display device to the outside of the finder window 403.

In the photographing standby state, the electronic viewfinder 4
Since the monitor image (moving image) of the subject is displayed on the screen, the photographer can visually recognize the subject from the monitor image by looking through the finder window 403.

A display unit 207 (hereinafter, referred to as an LCD display unit 207) formed of a color liquid crystal display device is provided substantially at the center of the rear surface of the camera body 2. The LCD display unit 207 displays a menu screen for setting a photographing mode, photographing conditions, and the like in the recording mode, and reproduces and displays a photographed image recorded on the memory card MC in the reproduction mode.

The shooting modes include a mode relating to exposure control and a mode relating to image combining processing. The mode relating to the exposure control is a mode relating to a method of determining an exposure control value (aperture value of an aperture and an exposure time) at the time of release. The modes related to exposure control include at least a program mode, a shutter priority mode, and an aperture priority mode.
The exposure control value is set using one of a plurality of preset program charts. In the program mode, the exposure control value is set using a standard program chart, and the exposure control value is set. In the priority mode, the exposure control value is set using a program diagram that gives priority to the shutter speed (exposure time) over the aperture value. In the aperture priority mode, the exposure control value is set using a program diagram that gives priority to the aperture value over the shutter speed. The exposure control value is set.

The mode relating to the image synthesizing process means that the photographing conditions are changed at the time of release, or the photographing is performed twice continuously while keeping the photographing conditions as they are, and the two photographed images obtained by the photographing are respectively stored in the memory card MC. Mode. Two photographed images consecutively photographed in this mode are subjected to predetermined image processing by an image processing device composed of a computer or the like, which will be described later, and combined to form an image having higher image quality and video effect than the original photographed image. To gain.

The modes relating to the image synthesis processing include at least a "blur taste adjustment mode", a "gradation adjustment mode", and a "super-resolution mode". The blur adjustment mode refers to a photographed image A focused on a main subject (for example, a person or the like) in image processing and a photographed image B focused on the background of the main subject.
In order to obtain an image having a desired degree of blur by combining the two, the shooting position is changed by one shutter operation and the shooting operation is performed twice consecutively. Is the mode in which

The gradation adjustment mode is a mode in which a photographed image A whose exposure is adjusted to the main subject in image processing and a photographed image B whose exposure is adjusted to the background of the main subject are combined so that, for example, the entire screen is properly adjusted. In order to intentionally increase the contrast between an image having a high density distribution or the main subject and the background, and obtain a highly creative image, two consecutive shots with the exposure condition changed by one shutter operation Performing the operation, the photographed image A
And a captured image B.

The super-resolution mode is used to combine a photographed image A and a photographed image B whose photographing positions with respect to the main subject have been slightly changed to obtain an image having a higher resolution than the original photographed image. Two consecutive shooting operations without changing the focus and exposure conditions by the two shutter operations, and the main subject in the screen due to a slightly different camera angle between the first shooting and the second shooting Is a mode in which a photographed image A and a photographed image B whose positions are slightly changed are obtained.

The details of the blur adjustment processing, gradation adjustment processing and super-resolution processing in the image processing apparatus will be described later.

On the left side of the LCD display unit 207, a power switch 208 is provided. The power switch 208 also functions as a mode setting switch for switching between a recording mode (a mode for performing a photographing function) and a reproduction mode (a mode for reproducing a recorded image on the LCD display unit 207).
That is, the power switch 208 is formed of a three-point slide switch, and when the contact is set to the central “OFF” position,
When the power is turned off and the contacts are set to the upper “REC” position, the power is turned on and the recording mode is set. When the contacts are set to the lower “PLAY” position, the power is turned on and the playback mode is set. Is set.

At the upper right position of the LCD display unit 207, 4
A continuous switch 209 is provided. Quad switch 20
Reference numeral 9 denotes a circular operation button, and the operation of pressing the operation button in four directions of up, down, left, and right is detected. The quad switch 209 is multifunctional, and functions as, for example, an operation switch for changing an item selected on a menu screen displayed on the LCD display unit 207 or changing a playback target frame selected on the index screen. In addition, the switch in the left-right direction also functions as a switch for changing the zoom ratio of the interchangeable lens 3.

A switch group 210 is provided below the four-way switch 209 for performing operations related to display on the LCD display unit 207 and display contents. The switch group 210 includes a cancel switch 210a and a decision switch 21.
0b, a menu display switch 210c, and a display switch 210d.

The cancel switch 210a is a switch for canceling the contents selected on the menu screen. The confirmation switch 210b is a switch for confirming the content selected on the menu screen. Menu display switch 21
Reference numeral 0c denotes a switch for displaying a menu screen on the LCD display unit 207 or switching the contents of the menu screen. Each time the menu display switch 210c is pressed, the menu screen is switched. Display switch 210d
Is a switch for displaying on the LCD display unit 207 or for turning off the display. Power batteries E1 to E4
In order to save power, the LCD display unit 207
Is not displayed. Each time the display switch 210d is pressed, the display and non-display of the LCD display unit 207 are alternately performed.

On the menu screen, for example, a plurality of items are displayed in an array, and a display (for example, a cursor or an inverted display) indicating a selected state is performed on the currently selected item. For example, when selecting a mode relating to image synthesis processing in the shooting mode, a menu screen as shown in FIG. The item of “normal shooting” on this menu screen is a mode for performing normal shooting similar to the shooting operation of the silver halide camera.

In the menu screen of FIG. 6, when the up switch of the quad switch 209 is pressed, the display position of the black triangle cursor K (ie, the item indicated by the cursor K) cyclically moves upward. , 4 switches 2
When the down switch 09 is pressed, the item display position indicated by the cursor K moves cyclically downward. When the confirmation switch 210b of the switch group 210 is pressed, the item (the gradation adjustment mode in FIG. 6) indicated by the cursor K at that time is set as the shooting mode.

Therefore, the photographer selects a desired photographing mode by operating the four-way switch 209 in the vertical direction on the photographing mode selection menu screen, and
By operating d, the shooting mode can be set. It should be noted that a desired photographing mode can be set in the same manner for the mode relating to the exposure control.

On the index screen, nine thumbnail images of all images recorded on the memory card MC are displayed in an array, and the display selected for the currently selected frame (for example, blinking display or frame display) Etc.) are performed. When one of the up, down, left and right direction switches of the quad switch 209 is pressed, the display indicating the selected state of the menu screen or the index screen moves to the item or frame in that direction. For example, when the upward switch is pressed, the display indicating the selected state of the menu screen or the index screen moves to the upward item or frame.

In the zoom operation of the interchangeable lens 3, when the right switch of the quadruple switch 209 is pressed, the interchangeable lens 3 continuously moves to the wide side, and when the leftward switch of the quadruple switch 209 is pressed, the interchangeable lens 3 is pressed. The zoom lens 3 continuously moves to the telephoto side.

FIG. 7 is a block diagram showing the internal configuration of the digital still camera 1.

The digital still camera 1 mainly includes a lens 1
01, imaging unit 102, signal processing unit 103, light emission control unit 1
04, lens control unit 105, display unit 106, operation unit 10
7 and the overall control unit 108.

The lens 101 corresponds to the interchangeable lens 3. The lens 101 is a focusing lens 101a
And a lens 101b for adjusting the amount of transmitted light.

The image pickup section 102 photoelectrically converts a light image of a subject incident through the lens 101 into an image signal (electric image) and takes in the image signal. The imaging unit 102 includes a CCD 102a corresponding to the CCD 205, a timing generator 102b for controlling an imaging operation of the CCD 102a, and a timing control circuit 102c for controlling driving of the timing generator 102b.

The CCD 102a receives a drive control signal (integration start signal /
Based on the integration end signal), the subject light image is received for a predetermined time (exposure time) and converted into an image signal (charge accumulation signal), and the image signal is read out by a read control signal (horizontal synchronization) input from the timing generator 102b. (A signal, a vertical synchronization signal, a transfer signal, etc.). At this time, the image signal is separated into R, G, and B color components and output to the signal processing unit 103. That is, the image signals of the R color component are output by sequentially reading the image signals received by the pixels at the pixel positions (2h + 1, 2k + 1), and the pixel positions (2h + 2, 2k + 1), (2h + 1,
By sequentially reading out the image signals received by each pixel at 2k + 2), an image signal of the G color component is output. By reading out the image signals received at each pixel at the pixel position (2h + 2, 2k + 2), the B signal is read out. An image signal of a color component is output.

The timing generator 102b generates a drive control signal based on a control signal input from the timing control circuit 102c, generates a read control signal based on a reference clock, and outputs the read control signal to the CCD 102a. The timing control circuit 102c controls the photographing operation of the imaging unit 102. Timing control circuit 102
c generates a shooting control signal based on a control signal input from the overall control unit 108. The photographing control signal includes a photographing control signal for displaying a moving image of a subject (hereinafter, referred to as a live view image) on the electronic viewfinder 4 during monitor standby in the recording mode, and the shutter button 6 is operated to operate the subject. Control signal, reference clock, and CC for capturing a still image (hereinafter, referred to as a recorded image)
D102a outputs the image signal to the signal processing unit 103
Signal (synchronous clock) for signal processing by
And so on. This timing signal is transmitted to the signal processing unit 10.
3 signal processing circuit 103a and A / D conversion circuit 103
b.

The signal processing section 103 performs predetermined analog signal processing and digital signal processing on the image signal output from the CCD 102a. The signal processing of the image signal is performed for each light receiving signal of each pixel constituting the image data. Hereinafter, for convenience of explanation, in order to distinguish a light receiving signal of each pixel from an image signal forming a captured image based on a set of these signals, the light receiving signal of each pixel is converted to a pixel signal (analog signal) or pixel data (digital) as necessary. Signal).

The signal processing section 103 includes an analog signal processing circuit 103a, an A / D conversion circuit 103b, a black level correction circuit 103c, a WB circuit 103d, a γ correction circuit 103e, and an image memory 103f. The black level correction circuit 103c, the WB circuit 103d, and the γ correction circuit 103
"e" constitutes a circuit for performing digital signal processing.

The analog signal processing circuit 103a is mainly composed of a CD.
The CCD 102a includes an S circuit (correlated double sampling) circuit and an AGC (auto gain control) circuit.
To reduce the sampling noise and adjust the signal level of the pixel signal output from. The gain control in the AGC circuit includes the aperture value of the aperture 101 c and the CCD 102.
This also includes a case where insufficient level of a captured image is compensated for when an appropriate exposure cannot be obtained with the exposure time a (for example, when a very low-luminance subject is captured). The A / D conversion circuit 102b converts each pixel signal output from the analog signal processing circuit 103a into digital image data. The A / D conversion circuit 102b converts a pixel signal received by each pixel into, for example, 10-bit pixel data.

The black level correction circuit 103c corrects the black level of each A / D converted pixel data to a reference black level. The WB circuit 103d adjusts the white balance of the captured image. The WB circuit 103 d adjusts the white balance of the captured image by converting the levels of the pixel data of the R, G, and B color components using the level conversion table input from the overall control unit 108. The conversion coefficient of each color component in the level conversion table is set by the overall control unit 108 for each captured image. γ correction circuit 1
03e corrects the γ characteristic of the pixel data. γ
The correction circuit 103e corrects the level of each pixel data using a preset correction table.

The image memory 103f is a memory for temporarily storing image data for which signal processing has been completed. Image memory 1
03f has a capacity capable of storing image data of at least two frames. This is because two consecutive exposures are performed in the imaging in the blur adjustment mode or the like and two frames of image data are captured, so that these can be stored. The storage capacity capable of storing one frame of image data is, for example, the CCD 102
Assuming that the number of pixels of a is 1600 × 1200 = 1.92 million,
This is a capacity capable of storing 1.92 million pixel data.

The light emission control unit 104 controls light emission of the flash 5 based on a light emission control signal input from the overall control unit 108. The light emission control signal includes a light emission preparation instruction, light emission timing, and light emission amount. Upon receiving an instruction to prepare for light emission from the overall control unit 108, the light emission control unit 104 charges the main capacitor to enable light emission, and discharges the accumulated charge in the main capacitor in synchronization with the light emission timing signal to emit the flash 5. Let it. Then, based on the light emission stop signal input from the overall control unit 108, the discharge of the accumulated charge in the main capacitor is stopped. Thus, the flash 5 emits light with a required light emission amount.

The lens control unit 105 controls the driving of the focusing lens 101a, the zoom lens 101b, and the diaphragm 101c in the lens 101.
The lens control unit 105 includes an aperture control circuit 105a for controlling the aperture value of the aperture 101c, a focus control circuit 105b for controlling the drive of the focus motor FM, and a zoom control circuit 105c for controlling the drive of the zoom motor ZM.

The aperture control circuit 105a is a general control unit 108
The aperture 101a is driven on the basis of the aperture value input from, and the aperture amount is set to the aperture value. The focus control circuit 105b controls the AF input from the overall control unit 108.
The drive amount of the focus motor FM is controlled based on a control signal (for example, a control value such as the number of drive pulses), and the focus lens is set at the focal position. Zoom control circuit 105c
Is a zoom control signal (4
The zoom motor ZM is driven based on the operation information of the continuous switch 209), and the zoom lens 101b is moved in the direction specified by the quad switch 209. That is, when the right operation information of the quad switch 209 is input from the overall control unit 108, the zoom control circuit 105c drives the zoom motor ZM in the forward direction to move the zoom lens 101b to the wide side, and When the left operation information of the continuous switch 209 is input, the zoom motor ZM is driven in the reverse direction to move the zoom lens 101b to the tele side.

The display unit 106 performs display on the LCD display unit 207 and display on the electronic viewfinder 4. The display unit 106 includes a display 106a and a VRAM 106b corresponding to the LCD display unit 207, and a display 106c and a VRAM 106d corresponding to the LCD display unit 401 in the electronic viewfinder 4. The display 106a has, for example, 640 × 480 = 307.
The display 106c has, for example, 400 × 300 = 120,000 pixels. Therefore, the VRAM 106b is connected to the display 106a.
370,000 pixel data can be stored corresponding to the number of pixels of VRAM 106d.
Approximately 120,000 pixel data can be stored corresponding to the number of pixels of c.

During the photographing standby, the photographing is performed by the CCD 102a, and after predetermined signal processing, the data of each frame image of the moving image stored in the image memory 103f is sequentially read out to the overall control unit 108, and the data size is displayed on the display 10
The display size is adjusted to 6c (that is, the image data of 1.92 million pixels is adjusted to image data of approximately 120,000 pixels) and transferred to the VRAM 106d. As a result, a preview image of the subject is displayed on the display 106c (display surface of the LCD display unit 401) on the monitor. When display of the LCD display unit 207 is instructed by operating the display switch 210d, the data of each frame image read to the overall display unit 108 is adjusted to the display size of the display 106a (that is, 1.92 million pixels). VRA is adjusted to image data of approximately 307000 pixels)
The preview image of the subject is also displayed on the display 106a (display surface of the LCD display unit 207) on the monitor.

Further, when the menu display is instructed by operating the menu switch 210c, the image data of the menu screen stored in the ROM 108a in the overall control unit 108 is read out to the VRAM 106b, and the display contents of the display 106a are changed to the menus. You can switch to the screen.

In the reproduction mode, thumbnail images of the shot images are read from the image file of each frame recorded on the memory card MC by the overall control unit 108, arranged in a predetermined format, and image data for index display is created. , The image data is read out to the VRAM 106b. As a result, a list of captured images recorded on the memory card MC is displayed on the display 106a. When the frame to be reproduced is designated by operating the quad switch 209, the image data of the photographed image recorded in the image file corresponding to the frame recorded on the memory card MC is read, and the image data of the display 106a is read. The image is adjusted to the display size and transferred to the VRAM 106b.
As a result, the captured image is reproduced and displayed on the display 106a (the display surface of the LCD display unit 207).

The operation unit 107 transmits operation information of operation members related to photographing and reproduction provided in the camera body 2 to the overall control unit 1.
08. Operation information input from the operation unit 107 includes a shutter button 206, a power switch 2
08, four-switch 209, switch group 210, and other operation members.

The overall control unit 108 centrally controls the photographing function and the reproducing function of the digital still camera 1. The overall control unit 108 has a card interface 1
09 is connected to a memory card MC. Also, a personal computer PC is externally connected via the communication interface 110.

The overall control unit 108 is composed of a microcomputer, and includes a processing program for performing various specific processes in the photographing function and the reproduction function, and the above-described imaging unit 102, signal processing unit 103, light emission control unit 104, lens The control unit 105 includes a ROM 108a in which a control program for controlling driving of the display unit 106 and the like is stored, and a RAM 108b for performing various arithmetic operations in accordance with a processing program and a control program.

The specific processing performed by the overall control unit 108 includes an exposure control value (an exposure time Tv [A
PEX value] and the aperture value Av of the aperture 101c [APEX
Value]) (exposure control value calculation process), and in the recording mode, the live view image taken into the image memory 103f from the CCD 102a is converted into an electronic viewfinder 4.
Or the recorded image read out from the memory card MC to the image memory 103f in the reproduction mode by the LCD.
A process for displaying on the display unit 207 (image display process), a process for recording a recording image taken from the CCD 102a into the image memory 103f in the recording mode in the recording mode (recording process), and a reproduction from the memory card MC in the reproduction mode The process includes a process of reading a recording image to be recorded to the image memory 103f (reproduction process), a process of continuously performing an exposure operation in a special shooting mode such as a blur adjustment mode (special exposure control process), and the like.

Exposure value calculator 108c, display controller 108
d, a recording control unit 108e, a reproduction control unit 108f, and a special exposure control unit 108g represent the above-described processes in the overall control unit 108 by functional blocks.

The exposure value determination unit 108c performs an exposure value calculation process, determines the brightness of the subject using image data of the G color component of the live view image, and calculates an exposure control value based on the determination result. I do.

The display control unit 108d performs an image display process. The display operation of the display unit 106, that is,
After reading out the image data temporarily stored in the image memory 103f and adjusting the image size to the image size of the display destination as necessary, the VRAM 106c or the VRAM 10
6d.

The recording control section 108e performs a recording process. In the normal shooting mode, when shooting is instructed by the shutter button 206 in the normal shooting mode, the recording control unit 108e reads out the image data (still image data) temporarily stored in the image memory 103f after the shooting instruction to the RAM 108b. A predetermined compression process by the JPEG method such as DCT conversion and Huffman coding is performed to create image data for recording. Also, pixel data is stored in the image memory 103f every 8 pixels in both the vertical and horizontal directions.
By reading it out to 108b, a thumbnail image having 200 × 150 pixels is created. Further, tag information relating to a captured image to be recorded accompanying these image data for recording is created. This tag information includes the lens name, frame number,
The information includes a focal length at the time of photographing, an F-number at the time of photographing, a focal position, a subject brightness, a white balance adjustment value, a photographing mode, a compression ratio, a photographing date, and whether or not a flash is emitted.

The recording control unit 108e attaches tag information to the compressed image data of the photographed image and the thumbnail image, and adds EXIF (Exchangeable Image File Format)
A format image file is created, and this image file is recorded on the memory card MC.

Also in the blur adjustment mode, the gradation adjustment mode, and the super-resolution mode, an image file is created after the captured image is A / D converted and the memory card M
Recorded in C.

In these photographing modes, two photographed images A and B are generated by one shutter operation, so that each photographed image A and B is uncompressed TIFF (Tag Image File FFF).
(ormat) format image files are created and recorded on the memory card MC. Further, since the photographed images A and B are used to generate an image having excellent blur, gradation, resolution, and the like, the tag information of both image files includes information that can be recognized as a photographed image for image synthesis. Is included. Further, since the photographed images A and B are for improving the image quality by image synthesis, all the pixel data are recorded as they are without performing compression processing which causes a decrease in image quality.

FIG. 8 shows an example of the structure of an image file recorded on the memory card MC.

In the figure, “P000001.TI”
“F”, “P000002.TIF” and the like indicate image file names. In the present embodiment, the image file name is "P
X. Y ”,“ X ”of“ PX.Y ”is a 6-digit number indicating the order in which the image files were created,
“Y” is a symbol indicating the format of the image data. The symbol "TIF" in "Y" indicates that the pixel data output from the CCD 205 is recorded as it is, and the symbol "JPG" indicates that the data is compressed in the JPEG format.

In the blur adjustment mode, the gradation adjustment mode, and the super-resolution mode, two captured images are taken in continuously, and an image file is created for each of these image data. Both image files are combined into one holder and recorded on the memory card MC. Therefore, “1b”, “2b”, “1”
“t”, “2h”, and the like indicate the holder names of the image files captured in the blur adjustment mode, the gradation adjustment mode, and the super-resolution mode, respectively.

In this embodiment, the holder name is “Ni”.
Where “i” of “Ni” is a symbol (b, t, h) indicating a shooting mode, and “N” is a number indicating a shooting order in each shooting mode. For example, “1b” indicates that the image data of the first image captured in the blur adjustment mode is stored in the holder, and “2b” indicates that the image data of the second image captured in the blur adjustment mode is stored. Indicates that it is a stored holder.

Therefore, FIG. 8 shows two holders 1b and 2b photographed and produced in the blur adjustment mode and holders 1t and 1h photographed and produced in the gradation adjustment mode and the super-resolution mode, respectively. This shows that the two image files 1n and 2n that have been photographed and created in the normal photographing mode are collected in the holder P and stored in the memory card MC.

FIG. 9 is a diagram showing a method of recording an image file on the memory card MC.

An index area for storing holder information and information on image files belonging to each holder is provided in the head storage area of the memory card MC, and the respective image files are stored in the subsequent area in the order of the file number X. I have.

The recording area of each image file in the memory card MC is composed of three areas, and the data of the tag information, the data of the photographed image, and the data of the thumbnail image are stored in each area from the top. The data sizes of the tag information data and the thumbnail image data do not change depending on the image file, but the data size of the shot image changes depending on the compression ratio and the shooting mode. Therefore, the number of files that can be stored in the image file storage area of the memory card MC changes depending on the data size of the captured image data of each image file.

The reproduction control unit 108f performs a process of reproducing the captured image recorded on the memory card MC on the LCD display unit 207. When the reproduction mode is set by the power switch 208, the reproduction control unit 108f reads out the thumbnail images from each image file recorded on the memory card MC, and sequentially stores the thumbnail images in the VRAM 106b according to a predetermined index format. For example, nine thumbnail images are stored in the VRAM 106b so as to be arranged in a 3 × 3 secondary array per page. Accordingly, nine two-dimensionally arranged thumbnail images are index-displayed on the LCD display unit 207.

Note that the same thumbnail image is stored in both image files for the related image files in which the shooting mode is the blur adjustment mode, the gradation adjustment mode, and the super-resolution mode. A thumbnail image is read from only the file.

When a thumbnail image of a frame to be reproduced with respect to the index-displayed thumbnail image is designated by the four-way switch 209 and the switch group 210, the reproduction processing unit 108f reads the captured image from the image file corresponding to the frame. Is read out, and when the frame is in the normal shooting mode, since the data is compressed, a predetermined decompression process is performed, and the data is stored in the image memory 103f.
If the frame is not in the normal shooting mode, the data is not compressed and is stored in the image memory 103f as it is. The data read out to the image memory 103f is adjusted in data size by the display control unit 108d and transferred to the VRAM 106b as described above, and is thereby reproduced and displayed on the LCD display unit 207.

If the frame to be reproduced is a frame shot in the bokeh adjustment mode, an image of a focused shot image (an image shot second time as described later in this embodiment). If the captured image data is read from the file and the frame is captured in the gradation adjustment mode or the super-resolution mode, the captured image data is read from one of the corresponding image files. It is.

The special exposure control unit 108g operates the CCD when the shutter button 6 is fully pressed in a state where the blur adjustment mode, the gradation adjustment mode, and the super-resolution mode are set.
The exposure operation 102a is controlled. Basically, when the S2 switch is turned on, the special exposure control unit 108g continuously captures an image for image synthesis.
The exposure operation of 102a is repeated twice.

Next, with respect to the photographing operation in the recording mode of the digital still camera 1, the photographing operation in the blur adjustment mode, the gradation adjustment mode and the super-resolution mode according to the present invention will be described.

FIG. 10 is a flowchart showing the procedure for photographing a still image in the blur adjustment mode.

In the blur adjustment mode, when the shutter button 206 is half-pressed (YES in # 1), preparations for photographing a still image are performed. That is, the diaphragm 101c
Is set to an open aperture value (for example, F = 2.8) (# 3), and the focal point of the lens 101 is adjusted to the ∞ position (# 5). Further, the exposure time of the CCD 102a is set, and white balance adjustment is performed (# 7).

When the shutter button 206 is fully pressed in this state (YES in # 7), the CCD 102a is exposed for the exposure time set in step # 7, and a still image of the subject is captured (# 11). After exposure, the CCD 102
The image signal output from a is subjected to predetermined analog signal processing, A / D conversion, and predetermined digital signal processing in the signal processing unit 103, and is stored in the image memory 103f. In this captured image, the focus of the lens 101 is adjusted to the position ∞, so that the background of the screen is in focus and the main subject such as a person in the screen is out of focus.

Subsequently, the recording control unit 1 of the overall control unit 108
08e creates tag information data,
Image data of a thumbnail image is created using the image data stored in the image memory 103f, an image file is created from these data and the image data stored in the image memory 103f, and stored in the memory card MC ( # 13). The shooting mode of the tag information for the shot image is “blur taste adjustment mode 1/2” (see FIG. 8).
As described above, information indicating that the image has been shot for the first time in the blur adjustment mode is added.

The information relating to the photographing mode is included in the tag information because it is distinguished from the photographed image in the normal photographing mode, and is automatically blurred when this image file is selected by the image processing apparatus as described later. This is in order to start the processing program of the taste adjustment processing and immediately perform the blur adjustment processing. Note that the same applies to the second captured image captured in the bokeh adjustment mode. Similarly, for the image captured in the gradation adjustment mode or the super-resolution mode, the tag information includes information about the capture mode. I try to record it.

Subsequently, the focal point of the lens 101 is, for example, 3
m (# 15), and the CCD 102a
Is set, and the white balance is adjusted (# 17).

Then, the CCD 102a is moved to step # 17.
The still image of the subject is captured by exposing for the exposure time set in (# 19). After the exposure, the image signal output from the CCD 102a is subjected to predetermined analog signal processing, A / D conversion, and predetermined digital signal processing in the signal processing unit 103, and is stored in the image memory 103f. In this captured image, the focus of the lens 101 is adjusted 3 m ahead, so that the main subject in the screen is almost in focus.

Subsequently, the display control unit 1 of the overall control unit 108
08d, the image data is read from the image memory 103f to the RAM 108b, and the data size is adjusted.
(# 21). This display process is to enable monitoring of the captured image immediately after shooting, and after a predetermined time (for example, 2 seconds) elapses, the image is switched to the live view image.

The recording control unit 10 of the overall control unit 108
The tag information data is created by 8e, and the image data of the thumbnail image is created using the image data stored in the image memory 103f. Note that information indicating that the image has been shot for the second time in the blur adjustment mode, such as “blur taste adjustment mode 2/2”, is given to the shooting mode of the tag information for the shot image.

The data and the image memory 10
An image file is created from the image data stored in the memory card 3f and stored in the memory card MC (# 23). Thus, the photographing process ends, and the process returns to the photographing standby state (the display state of the live view image).

In this embodiment, the focus is first set on the distant view side, and then the focus is set on the near view side for the second time. However, the order of changing the focus position may be reversed. Good. In addition, when shooting is performed in the blur adjustment mode after the camera is started, when the camera is started, the focusing lens 10 is used.
Since 1a is initially set to the ∞ position, it is more reasonable to focus on the distant view side in the first shooting without wasteful focus control.

In the second photographing, the in-focus position is fixed at a position 3 m from the camera. However, since the combined image in an arbitrary focused state is created by the blur adjustment processing, the photographing before combining is performed. Considering that the need to accurately determine the focus position of the image is low and that the processing time for focus control is as short as possible so that continuous shooting can be performed quickly, The distance from the camera may be fixed to another distance, or the distance to the main subject may be accurately determined by AF processing.

FIG. 11 is a flowchart showing a procedure for photographing a still image in the gradation adjustment mode.

In the gradation adjustment mode, when shutter button 206 is half-pressed (YES in # 31), preparations for photographing a still image are performed. That is, the lens 101
Is adjusted to the main subject, an exposure control value (Tv, Av) is calculated using the live view image, and a white balance adjustment value is set (# 33).

If the shutter button 206 is fully pressed in this state (YES in # 35), a shutter speed (Tv-2) two steps lower than the shutter speed Tv (APEX value) set in step # 33 is set. (# 3
7), the exposure time corresponding to this shutter speed is C
The still image of the subject is captured by exposing the CD 102a (# 39). After the exposure, the image signal output from the CCD 102a is subjected to predetermined analog signal processing, A / D conversion, and predetermined digital signal processing in the signal processing unit 103, and is stored in the image memory 103f. This shot image is C
Since the exposure time of the CD 102 is shorter than the appropriate value,
The image is dark overall.

Subsequently, the recording control unit 1 of the overall control unit 108
08e creates tag information data,
Image data of a thumbnail image is created using the image data stored in the image memory 103f, an image file is created from these data and the image data stored in the image memory 103f, and stored in the memory card MC ( # 41). It should be noted that information indicating that the image has been shot for the first time in the gradation adjustment mode, such as “gradation adjustment mode 1/2”, is added to the imaging mode of the tag information for the captured image.

Subsequently, a shutter speed (Tv + 2) that is two steps over the shutter speed Tv (EV value) set in step # 33 is set (# 43), and the CCD 102a is exposed for an exposure time corresponding to this shutter speed.
And a still image of the subject is captured (# 4).
5). After the exposure, the image signal output from the CCD 102a is subjected to a predetermined analog signal
D conversion and predetermined digital signal processing are performed and stored in the image memory 103f. This photographed image is CCD10
Since the exposure time of No. 2 is longer than the appropriate value, an overall bright image is obtained.

Subsequently, the recording control unit 1 of the overall control unit 108
08e creates tag information data,
Image data of a thumbnail image is created using the image data stored in the image memory 103f, an image file is created from these data and the image data stored in the image memory 103f, and stored in the memory card MC ( # 47).

[0113] Note that information indicating that the image has been shot for the second time in the gradation adjustment mode, such as "gradation adjustment mode 2/2", is added to the photographing mode of the tag information for the photographed image. You.

Subsequently, the display control unit 1 of the overall control unit 108
08d, the image data of the first shot image A and the image data of the second shot image B are read out from the image memory 103f into the RAM 108b, and an image C having an average density of both the shot images (hereinafter referred to as an average density image C) Is created (# 49). That is, photographed image A
Ga (i, j) (i = 1, 2,... N,
j = 1, 2,... m) and gb (i, j) as pixel data constituting the captured image B, gc (i, j) = {ga (i, j) + gb (i, j)}.
By calculating / 2, image data of the average density image C is created.

After adjusting the data size, the image data is transferred to the VRAM 106b and
7 is displayed (# 51). This display process is to enable monitoring of the captured image immediately after shooting, and after a predetermined time (for example, 2 seconds) elapses, the image is switched to the live view image.

The average density image C is used as the monitor image because the densities of the first and second shot images are shifted from the appropriate densities to the minus side and the plus side, respectively. Thus, an image having an appropriate density balance is immediately displayed without taking much time for the arithmetic processing.

In this embodiment, two images are continuously photographed with an exposure value shifted by ± 2 steps with respect to the appropriate exposure control. It is not limited. Further, the positive / negative shift amount may be different, and the photographer may be able to set the shift amount.

FIG. 12 is a flowchart showing a procedure for photographing a still image in the super-resolution mode.

In the super-resolution mode, the shutter button 2
When 06 is half-pressed (YES in # 61), preparation for still image shooting is performed. That is, the focus of the lens 101 is adjusted to the main subject, the exposure control value (Tv, Av) is calculated using the live view image, and the white balance adjustment value is set (# 63).

If the shutter button 206 is fully depressed in this state (YES in # 65), the exposure time CC corresponding to the shutter speed Tv set in step # 63 is used.
D102a is exposed to capture a still image of the subject (# 67). After the exposure, the image signal output from the CCD 102a is subjected to predetermined analog signal processing, A / D conversion, and predetermined digital signal processing in the signal processing unit 103, and is stored in the image memory 103f.

Subsequently, the recording control unit 1 of the overall control unit 108
08e creates tag information data,
Image data of a thumbnail image is created using the image data stored in the image memory 103f, an image file is created from these data and the image data stored in the image memory 103f, and stored in the memory card MC ( # 69). It should be noted that information indicating that the image has been captured for the first time in the super-resolution mode, such as "super-resolution mode 1/2", is added to the imaging mode of the tag information for the captured image.

Subsequently, the CCD 102 for the exposure time corresponding to the shutter speed Tv set in step # 63.
After exposing a, a still image of the subject is captured again (# 71). After the exposure, the image signal output from the CCD 102a is subjected to predetermined analog signal processing, A / D conversion, and predetermined digital signal processing in the signal processing unit 103, and is stored in the image memory 103f. This photographed image is 1
The photographing conditions are the same as the first photographed image, but the photographing timing is different, so that the camera angle is slightly different from the first photographed image.

Subsequently, the recording control unit 1 of the overall control unit 108
08e creates tag information data,
Image data of a thumbnail image is created using the image data stored in the image memory 103f, an image file is created from these data and the image data stored in the image memory 103f, and stored in the memory card MC ( # 73).

It should be noted that information indicating that the image has been shot for the second time in the super-resolution mode, such as “super-resolution mode 2/2”, is added to the imaging mode of the tag information for the captured image. You.

Subsequently, the display control unit 1 of the overall control unit 108
08d, the first captured image is read from the image memory 103f to the RAM 108b, the data size is adjusted, and then transferred to the VRAM 106b, where the LCD display unit 2
07 (# 75). The second captured image may be used for this display processing.

This display processing is for enabling the photographed image to be monitored immediately after photographing, and is switched to a live view image after a predetermined time (for example, 2 seconds) has elapsed. In addition, processing time is required to create a high-resolution image by combining two images, and the image is not suitable for display on a monitor. Therefore, in step # 75, the captured image before processing is directly displayed. .

Next, an image processing apparatus applied to the imaging system according to the present invention will be described.

FIG. 13 is an external view showing the configuration of the image processing apparatus. The image processing apparatus 6 shown in FIG. 1 includes a personal computer 7 and a printer 8 as its output device.

A personal computer 7 (hereinafter, referred to as a personal computer 7) includes a computer main body 701 having an FD driver 701a, a memory card reader 701b and a CD-ROM driver 701c, and a CRT or LCD.
702, a keyboard 703, and a mouse 704.

[0130] The personal computer 7 combines the two captured images captured in the blur adjustment mode to create a captured image having a desired degree of blur (hereinafter referred to as a blur adjustment processing program). A processing program for synthesizing two photographed images photographed in the gradation adjustment mode to create a photographed image having a desired gradation (hereinafter, referred to as a gradation adjustment processing program) and a super-resolution mode. CD-ROM 9 in which a processing program for image synthesis such as a processing program (hereinafter, referred to as a super-resolution processing program) for synthesizing two shot images and creating a high-resolution shot image is recorded. To the CD-ROM driver 70
1c, and stores the processing program for image synthesis in a hard disk (main storage device) in the computer main body 701.
Function as an image processing apparatus when installed in a computer.

In this embodiment, the image processing apparatus is configured by installing the processing program recorded on the CD-ROM 9 into the computer 7. However, the processing program is stored in the main memory of the apparatus in advance. The stored dedicated image processing device may be used.

The procedure for synthesizing the two photographed images by the image synthesizing processing program and the synthesized image (image with adjusted blur and gradation or high-resolution image) are displayed on the display 702 as described later. Is displayed. Also,
A hard copy of the combined image can be created by the printer 8.

Next, the image synthesizing process in the image processing device 6 will be described.

FIG. 14 is a flowchart showing a processing procedure for starting a processing program for image composition.

For each processing program of the blur adjustment processing program, the gradation adjustment processing program, and the super-resolution processing program, a folder having an image file of an image photographed in the photographing mode corresponding to each processing program is designated. And it is started in conjunction with it.

For example, the keyboard 703 or the mouse 7
By operating the button 04, access to the memory card MC after shooting in each of the above modes is instructed, and a folder storing images shot in each of the above modes is designated, or a cursor is moved to an icon indicating an image file in the folder. When the icon is moved and an instruction to open the image file of the icon is input, the tag information is read from the image file stored in the memory card MC into the personal computer body 7 (# 8).
1), the contents of the photographing mode in the tag information are determined (# 83, # 87, # 91).

If the shooting mode is the blur adjustment mode (YES in # 83), the blur adjustment processing program is started (# 85), and if the shooting mode is the gradation adjustment mode (YES in # 87), The gradation adjustment processing program is started (# 89), and if the shooting mode is the super-resolution mode (# 89).
The super-resolution processing program is started (YES at 91) (#
93), if the shooting mode is the normal shooting mode (# 91)
NO), a predetermined processing program other than the above-described processing program for a normal captured image, for example, a processing program for image correction is started (# 95).

FIG. 15 is a diagram showing an example of a dialog box for blur adjustment displayed on the display 702 when the blur adjusting program is started.

The work screen shown in FIG. 15 is a screen displayed when the blur adjustment processing program is started on a predetermined OS (Operating System) of the personal computer 7.
A dialog box 10 for performing blur adjustment processing is displayed on the top screen.

The dialog box 10 displays a thumbnail image of a photographed image related to the blur adjustment processing.
Display areas 11a to 11c and six button displays 12a to 12f for inputting processing contents and processing conditions are displayed.

The display area 12a and the display area 11b are arranged at the upper part of the screen, and the display area 11a displays the thumbnail image A of the second photographed image (the photographed image focused on the foreground).
Is displayed, and a first captured image (a thumbnail image B of a captured image focused on a distant view) is displayed in a display area 11b. A display area 11c is arranged below the display area 12a, and a thumbnail image is displayed. A thumbnail image C (see FIG. 24) in which A and the thumbnail image B are combined to simulate the blur adjustment is displayed.

The button displays 12a to 12e are displayed in the display area 11.
The button display 12f is displayed in the lower part of the display area 11c in a vertical arrangement in the lower part of the display area b.

Button display 12 labeled “File”
a (hereinafter, referred to as a file button 12a) is a button for opening an image file of another photographed frame. When the cursor K is moved to the file button 12a and the mouse 704 is double-clicked, predetermined processing software for opening an image file of another photographed frame is performed, and image file reading processing is performed according to the flowchart shown in FIG. Done.

That is, a dialog box for opening an image file is displayed on the screen of the display 702, and when a frame to be opened is designated (# 101), the image file corresponding to the frame is read from the memory card MC. The information on the shooting mode included in the tag information is read (# 103). Then, it is determined from the information on the shooting mode whether or not the shot frame was shot in the blur adjustment mode (# 105). If the shot was shot in the blur adjustment mode (# 105). YES), since the image compositing processing for the specified photographed frame matches the currently open image compositing software, the photographed image and the thumbnail image are respectively extracted from the two image files included in the holder corresponding to the photographed frame. Image data is stored in RAM 108
b (# 107), and unless the image was shot in the blur adjustment mode (NO in # 105), the image synthesis processing for the designated shooting frame matches the currently open image synthesis processing software. No, a predetermined error message is displayed (# 109).

Note that, in the flowchart shown in FIG.
If the specified shooting frame is not taken in the bokeh adjustment mode corresponding to the currently open bokeh adjustment process, an error message is displayed indicating that the image combining process for the shooting frame is incompatible, and the operator is notified. In FIG. 17, a frame shot in the blur adjustment mode is designated. However, as shown in the flowchart of FIG. 17, the blur adjustment mode corresponding to the blur adjustment process in which the designated frame is currently open. If the image was not captured by the camera, image synthesis processing software corresponding to the shooting mode of the shooting frame may be activated to immediately perform image synthesis processing on the shooting frame.

The flowchart shown in FIG. 16 gives priority to the currently activated image synthesizing software, and unless the image synthesizing software is terminated, a frame shot in a shooting mode that is not compatible with the image synthesizing software is executed. This prohibits the opening of the image file. This flowchart eliminates the need for complicated processing to resolve the subsequent inconsistent processing due to the opening of the image file of the shooting frame that is not compatible with the currently activated image synthesis processing software, and the processing is simplified. It is advantageous.

On the other hand, the flowchart shown in FIG. 17 gives priority to the photographing mode of the designated photographing frame. If the image composing software for the photographing frame is not compatible with the currently activated image composing software, it is activated. This is to switch image synthesis processing software to be performed. If the shooting mode of the specified shooting frame is not compatible with the currently activated image synthesis processing software, this flowchart restarts the image synthesis processing software that matches the shooting mode of the specified shooting frame each time. The processing is more complicated than the flowchart shown in FIG.

However, as described above, in this embodiment,
The image synthesis processing software started first corresponds to the shooting mode of the shooting frame in which the image file is first opened, and is attached to the shooting mode of the shooting frame specified by the operator. This takes into account the workability of the worker performing a required image synthesis process for a desired photographing frame and then viewing it. The flowchart of FIG. Has the advantage of good work efficiency.

In the flowchart shown in FIG. 17, step # 109 of the flowchart of FIG. 16 is replaced with steps # 109-1 to # 109-4 having the same processing contents as steps # 87 to # 95 of the flowchart of FIG. If the shooting mode is not the blur adjustment mode (NO in # 105), it is determined whether the shooting mode is the gradation adjustment mode, the super-resolution mode, or the normal shooting mode (# 109- 1, # 109-3), if the shooting mode is the gradation adjustment mode (YES in # 109-1), the gradation adjustment processing program is started (# 109-2), and the shooting mode is set to the super-resolution mode. If there is (YE in # 109-3
S), the super-resolution processing program is started (# 109-
4) If the shooting mode is the normal shooting mode (# 109)
(NO at -3), a predetermined processing program other than the above-described processing program for a normal captured image, for example, a processing program for image correction is started (# 109-5).

Returning to FIG. 15, the button display 12b described as “execute” (hereinafter referred to as the “execute button 12b”) is actually displayed by using two images of different focus states photographed in the blur adjustment mode. Is a button for executing the blur adjustment processing.

When the cursor K is moved to the execution button 12b and the mouse 704 is double-clicked, predetermined processing software for creating a blur adjustment image using two shot images having different focus states is activated. The blur adjustment processing is performed according to the flowcharts shown in FIGS.

Here, a brief description will be given of a method for creating a blur adjustment image employed in the present embodiment. This method of creating a blurred image is also applied to a case where a blurred image is simulated using a thumbnail image.

Two images having different focus states (for example, as shown in FIG. 18, the near view is focused, the distant view is focused on the near view focused image Ir and the distant view is focused, and the near view is the blurred image If Various methods are known as a method for creating an image I in an arbitrary focused state by using the method (1). Basically, positioning (registration processing) between two images Ir and If and a predetermined method are performed. (The image synthesis processing) of the blur adjustment image by the above calculation processing.

In the registration processing, the positions of both images to be synthesized are aligned in order to accurately synthesize the same symbol in the screen in the image synthesizing processing. The registration process is performed in two stages: registration of the entire image (coarse registration) and subsequent local registration (fine registration).

In the registration of the entire image, for example, the two images Ir and Ir are enlarged / reduced, translated, rotated, and the like with respect to the distant focused image If with reference to the focused near image Ir.
By comparing If, the enlargement ratio, the translation amount, the rotation angle, and the like of the far-field focused image If in which both images Ir and If match are calculated. Note that the degree of coincidence of the far-field focused image If with the near-field focused image Ir is evaluated by calculating the root mean square value ΔV of the level difference of each corresponding pixel between the two images, and using the sum ΣΔV thereof as an index. Values such as an enlargement ratio, a translation amount, and a rotation angle that minimize the sum ΣΔV are calculated as image alignment information.

It should be noted that the image positioning information such as the enlargement ratio, the amount of parallel movement, and the rotation angle is different from the in-focus state of the near-field focused image Ir and the far-field focused image If, and the two images Ir and If are directly compared. Since there is a possibility that an error occurs, the calculation is performed using a hierarchical matching method.

The local registration process is for correcting a positional shift of each pixel between the near-field focused image Ir and the far-field focused image If obtained by registering the entire image. This correction is based on the pixel position (x, y) of the near-field focused image If and the pixel position (x + m, y + n) of the far-field focused image If obtained by registering the entire image. 2 of the level difference of the corresponding pixel
This is performed by calculating a square mean value and further calculating parameters (m, n) that minimize the total of r × r.
When the parameters (m, n) are calculated for all the pixels, and the far-field in-focus image If is corrected using the parameters (m, n), the position is finally adjusted with respect to the near-field in-focus image Ir. The obtained distant view image If is obtained.

Note that the registration processing is not necessarily required if the near-field focused image Ir and the far-field focused image If are exactly the same. In general, the registration processing is always performed in the present embodiment because the subject is slightly displaced within the screen even if the subject is photographed in a targeted manner.

As the image synthesizing process, various methods such as a segment method, an iterative method and an inverse filter method are known. Since the present embodiment employs an iterative method, the following description describes the iterative method.

As shown in FIG. 18, the function g1 (x, y) representing the near-field focused image Ir is represented by f1 (x, y) representing the focused near-view area, and the focused far-field image is represented by f1 (x, y). The function representing the region is represented by f2 (x,
y), a function representing the degree of blurring of the out-of-focus distant view area is h
Assuming that 2 (x, y), g1 (x, y) = f1 (x, y) + h2 (x, y) · f2 (x, y) (1)

Similarly, a function g2 representing the far-field in-focus image Ir
(x, y) is given by h2 (x, y), where h1 (x, y) is a function representing the degree of blurring of the unfocused foreground area, and g2 (x, y) = f2 (x, y) + h1 (x, y). f1 (x, y)... (2)

On the other hand, the defocus adjustment image I is a close-up in-focus image I
Assuming that an image obtained by blurring the foreground area of r with the blur function ha (x, y) and an image obtained by blurring the distant view area of the far-field focused image If with the blur function hb (x, y) are created, The function f (x, y) representing the blur adjustment image I is f (x, y) = ha (x, y) · f1 (x, y) + hb (x, y) · f2 (x, y) … (3)

[0163] It should be noted that the blurring function h1, h2 is, the function h representing the extent of blurring, h (x, y) = (1 / πR 2) · exp (- (x 2 + y 2) / R 2) ... ( 4) R: The blur amount (blur radius) is assumed to be represented by a two-dimensional Gaussian function, and can be estimated from the near-field image of the far-field in-focus image If or the far-field image of the near-field in-focus image Ir. In the case of a shot image, the amount of blur can be calculated from the focus position of the shooting lens,
In the present embodiment, the setting is made by referring to a table set in advance based on the type of the photographing lens included in the tag information of the image file and the information on the focal length at the time of photographing. The blur functions ha and hb have default values set in advance, but as described later, a button 12c described as "setting" (hereinafter, referred to as a setting button 12c).
The user can set the desired amount of blur from a dialog box opened by operating.

From equations (1) to (3), f1 (x, y) and f2 (x, y)
Is eliminated, a relational expression between the function g1 (x, y) of the near-field focused image Ir, the function g2 (x, y) of the far-field focused image If, and the function f (x, y) of the blur adjusted image I (Ha−hb · h1) · g1 + (hb−ha · h2) · g2 = (δ−h1 · h2) · f (5) In equation (5), δ is a Dirac delta function. In equation (5), the variable (x, y) part of the function notation is omitted for convenience. Variables (x, y) in the following explanation
Each function is described by omitting the part.

Therefore, by solving the equation (5), a function f indicating the blurred image I can be obtained. In the iterative method, in equation (5), the left side is g = (ha−hb · h1) · g1 + (hb
-ha · h2) · g2, and the initial value f ^{(0) of the} function f representing the blur adjustment image (eg, functions g1, g2, (g1 + g2) / 2, etc.)
Is set appropriately, and g + h1 · h2 · f ^(K) = f ^{(K + 1)} (k =
0, 1, 2,...), K → ∞
^{Since (} ∞ ⁾ converges to f that satisfies the expression (5), this method is a method of obtaining the solution of the expression (5) by repeating this repetitive operation. Therefore, the blur adjustment image I having a desired degree of blur is calculated by repeating the repetition of g + h1, h2, f ^(K) .

Returning to FIG. 15, the setting button 12c is used to set parameters for the blur amount R1 of the near-field focused image Ir, the blur amount R2 of the far-field focused image, and the blur adjustment process of the number of repetition operations k. Button. When the cursor K is moved to the setting button 12c and the mouse 704 is double-clicked, a processing program for setting parameters of the blur amounts R1 and R2 and the number of repetition operations k is started, and the screen of the display 702 is displayed on the screen of FIG. A dialog box 13 for setting parameters as shown in FIG.

In the dialog box 13, the amount of blur is displayed.
Slide switch display 14a for setting 1, r2
14b, an area 15 for inputting the number k of repeated operations, and an "OK" button display 16 for confirming the set contents are displayed. The blur amount r1, r2 is indicated by a slide switch display 14.
By moving the cursor K to a and 14b and moving the cursor K left and right with the mouse 704 being one-clicked, the set value is changed. Defocus amount r
The set values of 1 and r2 are indicated by slide switch displays 14a and 14b.
Moves to the left and decreases and moves to the right.

It is to be noted that the operator can only imagine the degree of blur, and cannot recognize the relationship between the specific numerical values of the blur amounts r1 and r2 and the actual degree of blur. The sensational blur of the in-focus image and the far-field in-focus image can be input, for example, in a 10-step slider format.

The number of repetition operations k is set by moving the cursor K to the window 15 and inputting a numerical value directly from the keyboard 703. When the cursor K is moved to the OK button display 16 and the mouse 704 is double-clicked, the degree of blur of the set near-field focused image and the far-field focused image and the number of repetition operations k are determined.

Returning to FIG. 15, the button display 12d labeled "simulation" (hereinafter, simulation button 12d) is a thumbnail image A displayed in the display area 11a and a thumbnail image B displayed in the display area 11b. This button is used to simulate the bokeh adjustment process using.

In the case of combining two images having different focus states using two images having different focus states, it is usually impossible to know how the finished state will be without actually combining the images. Therefore, if the blur adjustment processing was performed using the actual photographed image, the image data of the photographed image was enormous, so it took a long time until the processing result was obtained, and the bokeh was easily confirmed. This is extremely inconvenient in some cases. Therefore, in the present embodiment, the blur adjustment is simulated using a thumbnail image having a small number of pixels, so that the blur adjusted image can be easily and quickly confirmed.

When the cursor K is moved to the simulation button 12d and the mouse 704 is double-clicked,
Predetermined processing software for creating a blur adjustment image using the thumbnail image is activated, and the blur adjustment processing using the thumbnail image is performed according to the flowchart shown in FIG.

The button display 12e described as "end" (hereinafter referred to as an end button 12e) is a button for ending the blur adjustment processing, and the dialog box 10 is provided.
Has the same function as the X mark displayed in the upper left corner of the. When the cursor K is moved to the end button 12e and the mouse 704 is double-clicked, the software of the blur adjustment processing ends.

The button display 12f described as "cancel" (hereinafter referred to as "cancel button 12f") is a button for interrupting the simulation processing of the blur adjustment processing in the middle. When the cursor K is moved to the stop button 12f and the mouse 704 is double-clicked, the simulation of the blur adjustment processing is stopped.

Next, the blur adjustment processing will be described in detail with reference to the flowcharts shown in FIGS.

When the software for the blur adjustment process is started, two image files P (X +
1). TIF, PX. The thumbnail images A and B are read from the TIF and displayed in the display areas 11a and 11b of the dialog box (# 111 and # 11).
3).

Subsequently, the blur amounts R1, R2 and the number of repetition operations k for the near-field focused image and the far-field focused image are set (# 115, # 117). Unless the amount of blur R1, R2 and the number of repetition operations k are set by the operator on the setting screen shown in FIG. 19 as described above, a preset default value (for example, the degree of blur is level 5, the number of repetition operations k is 20), and when newly set by the operator, the set value is set.

Subsequently, it is determined whether or not a simulation of the bokeh adjustment process using the thumbnail image is instructed (# 119). If the simulation is instructed (YES in # 119), it is shown in FIG. After the simulation of the blur adjustment process is performed according to the flowchart (# 121), the process proceeds to step # 123, and if the simulation is not instructed (N in # 119).
O), skip step # 121 and skip to step # 12
Move to 3.

When the software for simulating the blur adjustment process is started, first, a registration process between the thumbnail image A and the thumbnail image B is performed using the G color component image (# 191). The shift amount (δx, δ) of the thumbnail image B with respect to the thumbnail image A
y) is determined (# 193). In the present embodiment, since the registration processing is simply performed only by the parallel movement, the shift amount (δx, δy) is calculated for the vertical direction x and the horizontal direction y.

Subsequently, the images of the R, G, and B color components of the thumbnail image B are moved as a whole by the amount of shift (δx, δy), and the alignment between the thumbnail image A and the thumbnail image B is performed. # 195).

Subsequently, based on the type of the photographing lens and the information of the focal length at the time of photographing included in the tag information, the ROM 1
The blur amount (blur radius) r1 of the photographed image corresponding to the thumbnail image A is referred to by referring to a predetermined table stored in 08a.
(Corresponding to the blur amount R in the above equation (4)) and the photographed image B
Is set as the blur amount (blur radius) r2 (corresponding to the blur amount R in the above equation (4)) of the captured image corresponding to (# 1).
97), the blur amounts r1 and r2 are reduced to 1/4, and the blur amount r1 '(= r1 / 4) for the thumbnail image A and the blur amount r2' (= r2 / 4) for the thumbnail image B are determined. (# 199).

The reason why the blur amounts r1 'and r2' for the thumbnail images A and B are calculated by setting the blur amounts r1 and r2 to 1/4 is that the size of the thumbnail images A and B is determined by Since the blur amounts r1 and r2 calculated in advance for the captured image are not reduced to 1/4, the blur amounts r1 'and r2' for the thumbnail images A and B are set.
Because it does not. If the size of the thumbnail images A and B is 1 / n of the photographed image, the blur amounts r1 'and r2' for the thumbnail images A and B are r1 '= r1 / n and r2' =
It is calculated by r2 / n.

Subsequently, the blur adjustment image C is obtained by an iterative method.
A counter that counts the number of repetitive operations when calculating
The count value N is reset to "0" (# 201). Continued
And the first bokeh for each color component of R, G, B
F representing the adjusted image C⁽¹⁾Is f^{( 1)}= G + h1, h2, f⁽⁰⁾
(# 203), and the calculation result is displayed in the display area.
11c (# 205). Where the function f⁽⁰⁾
Is the initial value of the function f representing the blur adjustment image C.
The function g1 representing the thumbnail image A and the thumbnail image B are shown.
Average value (g1 + g2) / 2 with the function g2. Also,
The blurring functions h1 and h2 are respectively added to the blurring amount R in the above equation (4).
The blur amounts r1 'and r2' set in step # 199 are
It is a blur function obtained by substitution. The function g is
The blur amount preset in each of the blur amounts R in equation (4)
Function obtained by substituting the default values ra and rb of
ha, hb, the blurring functions h1, h2 and the image functions g1, g2
G = (ha-hb · h1) · g1 + (hb-ha · h2) · g2
It is what is done.

Subsequently, by moving the cursor K to the stop button 12f and double-clicking the mouse 704, it is determined whether or not the stop of the operation is instructed, and whether or not the result of the iterative operation has converged during the iterative operation. It is sequentially determined whether or not the number of repetition operations N has reached the set predetermined number k (# 207, # 209, # 211), and the stop of the repetition operation is instructed (YES in # 207). If the calculation result converges (YES in # 209), or if the number of repetition calculations N reaches the set predetermined number k (# 21)
If 1 is YES), the simulation of the blur adjustment processing based on the thumbnail image ends.

On the other hand, it is not instructed to stop the iterative operation (NO in # 207), the result of the iterative operation has not converged (NO in # 209), and the number N of iterative operations has not reached the predetermined number k. If (NO in # 211), the count value N of the counter is incremented by 1 (# 213), and the process returns to step # 203 to perform the next blur adjustment calculation. That is, in the second case, f ⁽²⁾ = g + h 1 · h 2 · f ⁽¹⁾ using f ⁽¹⁾ representing the first bokeh adjustment image C.
Is calculated, and the calculation result f ⁽²⁾ is renewedly displayed in the display area 11c (# 203, # 205, # 207).

FIG. 24 is a diagram showing a simulation state of the blur adjustment processing using the thumbnail images. The figure shows a state where the repetitive operation has been performed five times, and the display area 1
In FIG. 1c, a bokeh adjustment image C as the calculation result is displayed, and a progress bar 17 indicating that the processing is being performed and a repetition calculation count display 18 are displayed at the bottom of the dialog box. Since the operator can sequentially confirm the progress of the repetitive operation processing on the display image in the display area 11c, the operator moves the cursor K to the stop button 12f and double-clicks the mouse 704 when an image of a desired degree of blur is obtained. Thus, the simulation can be stopped.

Thereafter, an iterative operation for calculating f ^(k) representing the blurred image C is repeatedly performed in the same manner (# 2).
During this period, if it is instructed to stop the iterative operation or the result of the iterative operation converges (YES in # 207 or # 209), the simulation of the blur adjustment process using the thumbnail image is terminated at that time. However, if the stop of the iterative operation is not instructed and the result of the iterative operation does not converge (NO in # 207 or # 209), when the number N of iterative operations reaches the predetermined number k (YES in # 211), the thumbnail is displayed. The simulation of the blur adjustment process using the image ends.

Returning to FIG. 20, when the flow shifts to step # 123, it is determined whether or not execution of the blur adjustment process using the photographed image is instructed. If the execution of the blur adjustment process is not instructed ( (NO in # 123), Step # 1
The process returns to step 15 to be in a standby state for an execution instruction.

When execution of the blur adjustment processing is instructed (#
123), the flow proceeds to step # 125, where the image file P (X + 1). TIF, PX. The image data of the R, G, and B color components of the captured images (the distant view focused image ga and the close view focused image gb) are respectively obtained from the TIF.
01 (# 125, # 12)
7). Subsequently, with respect to the image data of the G color component of the two captured images ga and gb, the image data at the insufficient pixel position is interpolated (# 129, # 131).

That is, in this embodiment, a single-plate type collar is used.
Bokeh tone for captured image captured by image sensor 205
Since the adjustment process is performed, the color of G is
The component image data is (2h + 2, 2k + 1) and (2h +
1,2k + 2) (h = 0,1,2, ... n / 2, k = 0,
1, 2,..., M / 2)
As shown in FIG. 2B, the pixel positions (2h + 1, 2k +
1), image data G 'is interpolated in (2h + 2, 2k + 2)
Is done. The interpolation processing of the image data G '
Image data of the pixel position to be
This is performed by calculating the average value of the image data G.
You. For example, the image data G at the pixel position (2, 2) _2,2Is a picture
Elementary positions (2,1), (1,3), (3,2), (2,
3) Image data G_2,1, G_1,3, G_3,2, G_2,3Using
G_2,2= (G_2,1+ G_1,3+ G_3,2+ G_2,3) / 4
Will be issued.

Subsequently, it is determined whether or not the simulation of the blur adjustment process using the thumbnail image has been performed (# 133), and if the simulation of the blur adjustment process has already been performed (YES in # 133). The shift amount (δx, δy) of the thumbnail image B with respect to the thumbnail image A calculated by the simulation is set as the initial value (δx0, δy0) of the shift amount in the registration processing between the shot image ga and the shot image gb. Are set to 4 times (4δx, 4δy) (# 135), and if the simulation of the blur adjustment process is not performed (NO in # 133), an appropriate value is set (# 13).
7).

In step # 135, since the shift amount (δx, δy) is calculated for the thumbnail image, the initial value based on the shift amount (δx, δy) is used to determine the shift in the registration process in the photographed image. The amount (δx ′, δy ′) can be calculated easily and at high speed. The initial value (δx0, δy0) of the shift amount in the registration processing of the shot image is set as the shift amount (4δx, 4δy) because the thumbnail image is created by reducing the size of the shot image to １ ／. Because it is. Therefore, if the thumbnail image is created by reducing the size of the captured image to 1 / n, the initial value (δ
x0, δy0) becomes (n · δx, n · δy).

Subsequently, a registration process is performed between the photographed image ga and the photographed image gb using the image of the G color component (# 139), and the shift amount (δx) of the photographed image gb with respect to the photographed image ga. ', Δy') is determined (# 1
41). Then, the images of the R, G, and B color components of the photographed image gb are moved as a whole by the amount of shift (δx ′, δy ′), and the positions of the photographed image ga and the photographed image gb are aligned (# 143). ), Image file P (X + 1). TIF,
PX. The ROM 108a is determined in advance based on the type of photographing lens included in the tag information of the TIF and information on the focal length during photographing.
The blur amount r1 of the photographed image ga and the blur amount (blur radius) r2 of the photographed image gb are set with reference to a predetermined table stored in (# 147).

Subsequently, with respect to the image data of the R and B color components of the two captured images ga and gb, the image data at the pixel position where no image data exists is thinned out to create image data for iterative calculation processing (#). 147- # 153). For example, in the case of the R color component, (2h + 1,
Since there is image data only at the pixel position of 2k + 1), the image data at other pixel positions are thinned out and the repetitive operation is reduced to an image size of n / 4 × m / 4 as shown in FIG. Image data for processing is created. Similarly, since there is image data only at the pixel position of (2h + 2, 2k + 2) for the color component B, the image data at other pixel positions is thinned out and reduced to an image size of n / 4 × m / 4. Image data for the iterative calculation process is created.

As described above, before performing the repetitive calculation of the blur adjustment, the image data is interpolated with respect to all the pixel positions for the G color component, and the pixel positions having no image data for the R and B color components are thinned out. The reason is that the G color component has a remarkable effect on the resolution, whereas the R and B color components have a small effect on the resolution and a large effect on the color. Is considered.

Generally, in digital image processing, the higher the resolution, the higher the accuracy of the processing result. However, the number of data to be processed becomes enormous, and the processing time becomes longer. Therefore, it is necessary to shorten the processing time by reducing the number of data to be processed as much as possible in relation to the allowable quality for the processing result, and also required from the viewpoint of operability, power saving, and the like. .

In the bokeh adjustment processing by the iterative method,
Since it is required in quality to be able to recognize the degree of blur adjustment as accurately as possible, the image data of the G color component, which has a large effect on the image data, is subjected to interpolation processing to create image data of all pixel positions. For R and B image data that have little effect on the degree of bokeh adjustment, the image data at the pixel position where no image data originally exists is excluded from the target of the arithmetic processing, and the processing time can be increased. I try to keep it as short as possible.

As a result, the R and B color components are subjected to image data interpolation at all pixel positions in the same manner as the G color components without affecting the quality of the blurred state after the processing, and the blur adjustment processing is performed. The bokeh adjustment process can be performed at a higher speed.

Subsequently, in steps # 155 to # 173, the blur adjustment processing is performed by an iterative method for each image of the R, G, and B color components. This process is basically a blur adjustment process in the simulation using the thumbnail image (steps # 203 to # 21 in the flowchart shown in FIG. 23).
3), but the blur amounts are r1 and r2 for the G color component image, whereas the blur amounts are r1 / r for the R and B color component images. 2 and r2 / 2 (# 157, # 161).

The amount of blur for the image of the R and B color components is represented by G
The reason why the blur amount of the color component image is set to １ ／ is that the data size of the G color component image is the full size, while the image of the R and B color components is step # 147.
~ # 153 to デ ー タ the data size to full size
This is because it has been reduced to × 1/2. When the R and B color component data sizes are reduced to 1 / s × 1 / s with respect to the full size, the blur amounts of the R and B color component images with respect to the image are r1 / s and r2 / s. .

When the blur adjustment processing ends (# 167,
Then, an interpolation process is performed on the image data of the R and B color components of the blur adjustment image gc (# 175 and # 177). In this interpolation processing, since the data size of the blur adjustment image gc of the R and B color components has been reduced by the thinning-out processing, the interpolation processing restores full-size image data. For example, the blur adjustment image gc of the R color component corresponds to the pixel position of each pixel data R as shown in FIGS. 27A and 27B (2h + 1,
After returning to the pixel position of 2k + 1), the image data R 'at the pixel position where data is insufficient is interpolated using these image data R as shown in FIG. The interpolation of the image data R 'is performed in the same manner as the above-described interpolation method of the image data of the G color component. Interpolation processing is also performed on the blur adjustment image gc of the B color component in the same manner.

Subsequently, the blur adjustment image gc of the R, G, and B color components is synthesized (# 179), and the synthesis result is displayed on the display 702 as shown in FIG. 19 is displayed, and is displayed in the display area 20 (# 181).

At the bottom of the dialog box 19, a button display 21 (hereinafter referred to as a save button 21) for instructing to save the created bokeh adjustment image gc is displayed. Cursor K
Is moved and the mouse 704 is double-clicked (#
183), and a dialog box 22 for inputting storage conditions is displayed on the display 702 as shown in FIG.
Is displayed, and a compression ratio, a file name, a folder name, and the like are input in accordance with the dialog box 22 (# 18).
5, # 187), when the cursor K is moved to the OK button 16 displayed at the bottom of the dialog box 22 and the mouse 704 is double-clicked, the image data of the blur adjustment image gc is changed according to the input condition. It is stored in a predetermined memory in the computer main body 701 (# 18)
9).

That is, if the compression ratio is set (#
185, YES) and a predetermined compression method (JP
The image data of the blur adjustment image gc is compressed by the EG method or the like (# 187), and if the compression ratio is not set (NO in # 185), the computer performs the compression processing without performing the compression process and uses the set file name. It is stored in a predetermined memory in the main body 7 (# 189), and returns to step # 111 to perform the next blur adjustment processing.

In the dialog box 22 shown in FIG. 29, the input display of the compression ratio is displayed on the slider display 23, the cursor K is moved to the slider display 23, and the mouse 704 is moved left and right with one click. By moving the cursor K, the compression ratio is set. When the compression ratio is set to 0%, no compression processing is performed.

Next, the gradation adjustment processing will be described.

FIG. 30 is a diagram showing an example of a dialog box for tone adjustment displayed on the display 702 when the tone adjustment processing program is started.

The configuration of the dialog box 24 for gradation adjustment is basically the same as the dialog box 10 for blur adjustment shown in FIG. Therefore, the same numbers are given to those that perform the same functions as the dialog box 10.

Display areas 12a, 1 arranged at the top of the screen
2b, a thumbnail image A of a second photographed image (an overexposed photographed image) and a thumbnail image B of a first photographed image (an underexposed photographed image) are displayed, and a thumbnail is displayed in a display area 12c. A thumbnail image C in which the image A and the thumbnail image B are combined and a simulation of gradation adjustment is performed is displayed.

Here, the contents of the gradation adjustment processing will be briefly described.

In the present embodiment, the photographed images used in the gradation adjustment processing are a photographed image G1 underexposed by two steps from the proper exposure and a photographed image G2 overexposed by two steps with respect to the luminance of the subject. The gradation characteristic of an image photographed at an appropriate exposure with respect to the luminance of the subject is such that the luminance level of the subject and the A / D conversion level are substantially equal as shown in the characteristic (c) of FIG. Since the exposure amount of the photographed image G1 to the image pickup device is suppressed, the gradation characteristic is A / A with respect to the luminance level of the subject, as shown by the characteristic (a) in FIG.
The D conversion level is kept low.

On the other hand, since the overexposed photographed image G2 has an excessive amount of exposure to the image sensor, its gradation characteristic is A / A with respect to the luminance level of the subject as shown in characteristic (b) of FIG. The D conversion level is highly emphasized.

In the gradation adjustment processing, the photographed image G1
32 is added to the image data of the photographed image G2 at an appropriate addition ratio to obtain an arbitrary gradation characteristic within the range between the gradation characteristics (a) and (b) in FIG. Is created. The addition ratio is not constant irrespective of the level of the image data, but changes so that the addition ratio of the image data of the overexposed captured image G2 increases as the level of the image data decreases as shown in FIG. Let me. The reason why the addition ratio of the overexposed captured image G2 is increased in this way is to make it easy to see a dark part of the subject.

FIG. 33 shows the addition ratio at each level based on the level of the overexposed photographed image G2. Pixel data g2 (i, j) at the pixel position (i, j) of the photographed image G2 is shown. Is, for example, a level D that is approximately の of the A / D conversion range, the pixel data g2 (i, j) and the pixel data g1 (i, j) at the pixel position (i, j) of the captured image G1. ) Is added at R2: R1 (= 0.4: 0.6) to create pixel data gc (i, j) at the pixel position (i, j) of the gradation adjusted image Gc. That is, if the level of the pixel data g1 (i, j) is D 'as shown in FIG. 34, the pixel data gc (i, j)
Is D1 + D2 + D2 = 0.4D '+ 0.
6D.

As a result, as shown in the figure, the brightness of the subject at the pixel position (i, j) d
1 and G2, the gradation levels are point a and point b, respectively. However, in the gradation adjustment image Gc, the gradation level is point c between the two points.

Therefore, in the gradation adjustment processing, the addition ratio of the photographed images G1 and G2 corresponding to the level of the pixel data g2 (i, j) constituting the photographed image G2 overexposed is set to R1 (g2 (i, j). j)), R2 (g2 (i, j)) (where 0 ≦ R1 (g
2 (i, j)), R2 (g2 (i, j)) ≦ 1, R1 (g2 (i, j)) + R2 (g2
(i, j)) = 1), each pixel position (i, j) (i =
1,2, ... n, j = 1,2, ... m) R1 (g2 (i,
j)) · g1 (i, j) + R2 (g2 (i, j)) · g2 (i, j) to calculate pixel data gc (i, j) of the gradation-adjusted image Gc. A gradation adjustment image Gc having an arbitrary gradation characteristic between the gradation characteristics of the image G1 and the photographed image G2 is created.

Returning to FIG. 30, the simulation of gradation adjustment has a shorter processing time than the simulation of bokeh adjustment. Therefore, when the gradation adjustment software is started, the simulation of gradation adjustment using a thumbnail image is always performed automatically. And is displayed in the display area 11c. Therefore, the simulation button 12c is not provided in the tone adjustment dialog box 24.

The slider display 25 (hereinafter, slider 25) is for setting the composition ratio of the thumbnail images A and B in the simulation of the gradation adjustment using the thumbnail images. The combination ratio here is not an addition ratio for each level v of an overexposed image shown in FIG. 33 but a combination ratio of the entire thumbnail image A and thumbnail image B. In FIG. This is similar to the area ratio between the area (the area to which the captured image G1 is added) and the lower area (the area to which the captured image G2 is added).

Therefore, the cursor K is moved to the slider 25 and the mouse 704 is moved left and right with one click of the mouse 704 so that the cursor K (that is, the slider display 2) is clicked.
By moving 5), the curve of the addition ratio characteristic R in FIG. 33 is changed up and down like the characteristics of R ′ and R ″ indicated by the dotted line, and the composite ratio between the thumbnail images A and B is arbitrarily set. When the slider 25 is at the middle position of the slide range, the addition ratio characteristic is set to the characteristic R in FIG. 33, and when the slider 25 moves rightward from the middle position of the slide range, the characteristic R in FIG. (In other words, the composition ratio of the overexposed photographed image G2 increases), and when moving to the left, the characteristic R "in FIG.
(Ie, the underexposed captured image G1)
Is increased).

[0220] The button display 12g labeled "Variation" (hereinafter referred to as the variation button 12g) has a thumbnail image A with five kinds of preset addition ratio characteristics.
And the thumbnail image B to create five types of gradation-adjusted images C1 to C5.

A simulation of gradation adjustment using a thumbnail image is displayed in the display area 11c. If it is desired to know how the gradation adjustment result changes when the synthesis ratio is roughly changed, It is necessary to move the slider 25 to appropriately change the composition ratio, and the operation is troublesome. The variation button 12g is used to simulate five-stage gradation adjustment with five types of preset addition ratio characteristics so that the gradation adjustment condition roughly changes, and collectively display the simulation results as shown in FIG. Thus, it is easy for the operator to confirm what kind of addition ratio characteristics the desired gradation condition can be obtained.

If there is a desired gradation characteristic in the simulation result of the variation displayed collectively, the cursor K is moved to the image and the mouse 704 is double-clicked to execute the gradation adjustment processing in the main image. Can be set. In FIG. 31, the middle simulation image C3 (image C3 surrounded by a thick frame) is selected, and when the gradation adjustment processing is performed on the main image, the gradation characteristics almost the same as the gradation characteristics of the simulation image C3 Is obtained.

Next, the tone adjustment processing will be described in detail with reference to the flowcharts of FIGS.

When the software for the gradation adjustment process is started, two image files P (X
+1). TIF, PX. The thumbnail images A and B are read from the TIF and displayed in the display areas 11a and 11b of the dialog box 20 (# 221 and # 2).
23).

Subsequently, the addition ratio characteristic is read from the set position of the slider 25 (# 225), and using the addition ratio characteristic, a simulation of gradation adjustment using a thumbnail image is performed according to the flowchart shown in FIG. 37 (# 227). ).

When the software for simulating the gradation adjustment processing is started, first, the registration processing between the thumbnail image A and the thumbnail image B is performed using the G color component image (# 291). The shift amount (δx, δ) of the thumbnail image B with respect to the thumbnail image A
y) is determined (# 293). Subsequently, the images of the R, G, and B color components of the thumbnail image B are moved as a whole by the amount of shift (δx, δy), and the position of the thumbnail image A and the position of the thumbnail image B are adjusted (# 29).
5). This positioning process is the same as the blur adjustment process.

Subsequently, for each of the R, G, and B color components,
The thumbnail image A and the thumbnail image B are combined based on the set addition ratio characteristic (# 29).
7), the processing result is displayed in the display area 11c (#
299). That is, the overexposed thumbnail image A
Ga (i, j) (i = 1, 2,... N /
4, j = 1, 2,..., M / 4), and pixel data constituting the underexposed thumbnail image B is represented by gb (i, j) (i = 1,
2,... N / 4, j = 1, 2,..., M / 4), pixel data g
The addition ratios of the thumbnail images A and B corresponding to a (i, j) are Ra (ga (i, j)) and Rb (ga (i, j)) (where 0 ≦ R
a (ga (i, j)), Rb (ga (i, j)) ≦ 1, Ra (ga (i, j)) + Rb
Assuming that (ga (i, j)) = 1), Ra (ga (i, j)) · ga for each pixel position (i, j) for each of the R, G, and B color components.
By calculating (i, j) + Rb (ga (i, j)) · gb (i, j), pixel data gc (i, j) of the gradation adjusted image C is calculated, and the calculation result is displayed in the display area. 11c is displayed.

Returning to FIG. 35, when the gradation adjustment process using the thumbnail image is completed, it is determined whether or not the addition ratio characteristic of the simulation has been changed by the slider 25 (# 229), and the addition ratio characteristic must be changed. If (NO in # 229), the cursor K is moved to the variation button 12g and the mouse 704 is double-clicked (whether the variation button 12g is operated).
Is determined (# 231), and the variation button 1
If 2g has not been performed (NO in # 231), it is further determined whether or not the cursor K has been moved to the execute button 12b and the mouse 704 has been double-clicked (the execute button 12b has been operated).

If neither the variation button 12g nor the execution button 12b is operated (NO in # 231, # 233), the process returns to step # 229, and enters a standby state in which the simulation result of the gradation adjustment using the thumbnail image is displayed. .

On the other hand, if the addition ratio characteristic is changed by moving the slider 25 (YES in # 229), the process returns to step # 225, and a simulation of gradation adjustment is performed using the changed addition ratio characteristic (#) 225 to # 22
9).

When the variation button 12g is operated (YES in # 231), for example, five kinds of preset addition ratio characteristics are used as shown in FIG. A simulation of the kind of gradation adjustment is performed (# 235), and the simulation result is displayed as shown in FIG. 31 (# 2).
37).

Then, the displayed five gradation-adjusted images C are displayed.
When the cursor K is moved to any one of the gradation adjustment images from 1 to C5 and the mouse 704 is double-clicked (when the gradation adjustment image is selected) (YES in # 239),
The process proceeds to step # 241 to execute a gradation adjustment process on the captured image using the addition ratio characteristic of the selected gradation adjustment image. In the example of FIG. 31, since the gradation adjustment image C3 is selected, the gradation adjustment processing is performed on the captured image using the addition ratio characteristics of FIG. Gradation adjustment image C
If there is no selection of 1 to C5 (NO in # 239), the process returns to step # 229, and enters a standby state with the simulation result displayed.

If the execution button 12b is operated (YES in # 233), the flow shifts to step # 241 to change the addition ratio characteristic set by selecting the slider 25 or the five gradation-adjusted images C1 to C5. The gradation adjustment process is performed on the captured image using the image data.

In the gradation adjustment process for a captured image, first, the image file P (X + 1). TIF, PX. TIF
Then, the image data of the R, G, and B color components of the captured images (the underexposed image G1 and the overexposed image G2) are read out to storage devices in the computer main body 701 (# 241 and # 243). Subsequently, both photographed images G1, G
With respect to the image data of the second G color component, the image data at the insufficient pixel position is interpolated (# 245, # 247).
This interpolation processing is the same as the interpolation processing in the blur adjustment processing.

Subsequently, as the initial value (δx0, δy0) of the shift amount in the registration processing between the photographed image G1 and the photographed image G2, the thumbnail image A calculated by the simulation of the gradation adjustment processing using the thumbnail image is used. Of the thumbnail image B with respect to (δx, δy)
(4δx, 4δy) are set (# 24).
9). In the gradation adjustment process, a simulation of the gradation adjustment process using a thumbnail image is always performed. Therefore, the shift amount obtained from the simulation result of the thumbnail image is used as an initial value of the shift amount in the registration process of the captured image. I have to.

Subsequently, a registration process is performed between the photographed image G1 and the photographed image G2 using the image of the G color component (# 251), and the displacement amount (δx) of the photographed image G2 with respect to the photographed image G1. ', Δy') is determined (#
253). Then, the images of the R, G, and B color components of the photographed image G2 are entirely moved by the amount of shift (δx ′, δy ′), and the positions of the photographed image G1 and the photographed image G2 are adjusted (# 255). First, gradation adjustment processing is performed on each color component of G (# 257). This gradation adjustment processing is the same as the gradation adjustment processing in the simulation using the thumbnail image.

Subsequently, gradation adjustment processing is performed on the R color component (# 259 to # 265), and thereafter, gradation adjustment processing is performed on the B color component (# 265 to # 2).
73). For the R and B color components, image data for gradation adjustment processing is created by thinning out image data at pixel positions where no image data exists, similarly to the blur adjustment processing (# 259, # 261, # 267, # 269), and tone adjustment processing is performed using the image data (# 263, # 269).
Then, the image data is interpolated (# 265, # 373).

Steps # 259, # 261, # 267,
The image data thinning process of # 269 is the same as the image data thinning process in the blur adjustment process (see the processes of steps # 147 to # 153 in the flowchart shown in FIG. 21). Further, the gradation adjustment processing in steps # 263 and # 271 is the same as the gradation adjustment processing in the simulation using the thumbnail image. Step #
The interpolation processing of the image data of 265 and # 273 is the same as the interpolation processing of the image data in the blur adjustment processing (see the processing of steps # 175 to # 177 in the flowchart shown in FIG. 22).

Subsequently, the gradation adjusted image Gc of the R, G, B color components is synthesized (# 275), and the synthesized result is displayed on the display as shown in FIG. A dialog box 19 for displaying the processing result is displayed on the display area 702 and displayed in the display area 20 (# 2).
77).

When the cursor K is moved to the save button 21 and the mouse 704 is double-clicked (# 27
(YES at 9), the image data of the gradation adjusted image Gc is stored in a predetermined memory in the computer main body 701 (# 2)
81).

Next, the super-resolution processing will be described.

FIG. 40 is a diagram showing an example of a dialog box for super-resolution processing displayed on the display 702 when the super-resolution processing program is started.

Dialog box 26 for super-resolution processing
Is basically the same as the dialog box 10 for blur adjustment shown in FIG. 15. In FIG. 15, the display area 11c and the simulation button 12d are removed. In the super-resolution processing, even if the super-resolution processing is performed using the thumbnail images, it is difficult to visually recognize the effect. Therefore, the simulation button 12d for performing the super-resolution processing on the thumbnail images and the result are displayed. The display area 11c is not provided.

The display areas 11a and 11b, the file button 11a, the execution button 11b, and the end button 11e have the same functions as those of the dialog box 10.

Here, the contents of the super-resolution processing will be described with reference to FIG.
1 will be briefly described. For convenience of description, one-dimensional image data will be described.

When the photographer holds the camera and performs a shutter operation on the subject, two photographed images obtained by performing two consecutive exposure operations are usually completely captured by the photographer. Since the camera is not stationary, the photographing positions with respect to the subject are slightly displaced from each other due to a slight difference in the camera angle.

Therefore, in the present embodiment, the first photographed image and the second photographed image are each interpolated, and the two photographed images are aligned and synthesized.
Try to get a high resolution image.

That is, FIG. 41A shows the image data of the first photographed image, FIG. 41B shows the image data of the second photographed image, and FIG. 41B shows the image data of the second photographed image. It is assumed that the shooting position with respect to the subject is shifted to the right by Δx. 41 (a) and 41 (b), a curve P indicates the luminance characteristic of the subject, and a (1), a (2),.
(1), b (2),... Are pixel positions, C (1), C (2),.
(1), D (2),... Indicate the light receiving level of each pixel. C (1) ′, C (2) ′,...
Are levels interpolated between pixel positions a (1), a (2),... Using D (1) ′, D (2) ′,. ), D (2),..., Pixel positions b (1), b (2),.
This is the level interpolated between.

As shown in FIGS. 41A and 41B, the light receiving position of each pixel in the first photographed image with respect to the subject
Since the light receiving position of each pixel with respect to the subject in the second captured image is different, image data of twice the number of pixels of the image sensor can be obtained from the image data of both captured images.

Therefore, as shown in FIG. 27C, the first photographed image and the second photographed image are aligned and the two image data are synthesized, thereby doubling the pixel density of the image sensor. Is obtained.

In the present embodiment, since the shift of the light receiving position of the image pickup device with respect to the subject based on the slight difference in the camera angle between the two exposures is used, the pixel position a
The shift amount Δx between (i) and the pixel position b (i) is not constant, and the pixel position b (i) is not always located in the middle between the pixel position a (i) and the pixel position a (i + 1). Therefore, when the deviation amount Δx is very small, there is a possibility that the positioning of the two images cannot be performed with sufficient accuracy.

Therefore, in order to improve this, in the present embodiment, the image data of the first captured image and the image data of the second captured image are interpolated and then combined. By doing so, as shown in FIG. 41C, the image data C (i) ′ and D (i) ′ are between the pixel position b (i) and the pixel position a (i + 1). Interpolation is performed and the two captured images are aligned with high accuracy, so that a high-resolution image can be obtained.

Returning to FIG. 40, the setting button 12c is a button for setting the above-described interpolation method. In this embodiment, a well-known cubic convolution (Qu
A desired interpolation method can be selected from an ibicconvolution method, a quadratic spline method, and an integrating resampler method.

When the cursor K is moved to the setting button 12c and the mouse 704 is double-clicked, a processing program for selecting the above-mentioned interpolation method is started, and the screen of the display 702 displays the interpolation method as shown in FIG. A setting dialog box 27 is displayed.

A list of the above three interpolation methods is displayed in the dialog box 27. When the cursor K is moved to the display position 16 of the desired interpolation method and the mouse 704 is clicked once, the interpolation method is selected. , The leading 〇 is changed to ◎. Then, in this state, the cursor K is moved to the “OK” button 16 for confirming the setting contents, and
When the mouse 704 is clicked once, the interpolation method is set. In the example of FIG. 42, the cubic convolution method is set.

Next, the super-resolution processing will be described in detail with reference to the flowcharts shown in FIGS.

When the super-resolution processing software is started, two image files P (X +
1). TIF, PX. The thumbnail images A and B are read from the TIF, respectively, displayed in the display areas 11a and 11b of the dialog box 20 (# 29).
1, # 293), the default value of the super-resolution method is read (# 295), and the super-resolution processing enters a standby state (loop of # 297).

When execution of super-resolution processing is instructed by execution button 12c (YES in # 297), first,
The registration process between the thumbnail image A and the thumbnail image B is performed using the G color component image, and the shift amount (δx, δy) of the thumbnail image B with respect to the thumbnail image A is determined (# 299). ~ # 30
5).

Subsequently, the image file P (X + 1). TI
F, PX. The image data of the R, G, and B color components of the captured images (the first captured image g1 and the second captured image g2) are read out from the TIF to the storage device in the computer main body 701, and the two captured images g1 and g2 are read. With respect to the image data of the G color component, the image data at the insufficient pixel position is interpolated by the interpolation method set respectively, and the image data at the pixel position increasing according to the resolution magnification n is interpolated (# 307). , # 309).

That is, as shown in FIG. 25 (a), the image data of the G color component is (2h + 2, 2k + 1) and (2
h + 1, 2k + 2) (h = 0, 1, 2,... n / 2, k =
0, 1, 2,... M / 2)
Because there is no pixel position, as shown in FIG.
The image data G ′ of (1,2k + 1) and (2h + 2,2k + 2) are interpolated, and further, for example, when the resolution magnification n is set to 4 times, as shown in FIG. Pixel data G ″ is interpolated.

Subsequently, as the initial value (δx0, δy0) of the deviation amount in the registration processing between the photographed image g1 and the photographed image g2, the deviation of the thumbnail image B from the thumbnail image A calculated in step # 305 A value (4δx, 4δy) obtained by quadrupling the amount (δx, δy) is set (# 311), and further, the deviation amount (4δx, 4δy)
, A registration process is performed between the photographed image g1 and the photographed image g2 on the image of the G color component (# 313), and the shift amount (δx ′, δy ′) of the photographed image g2 with respect to the photographed image g1. ) Is determined (# 315).

Then, after the image of the G color component of the photographed image g2 is moved as a whole by the amount of shift (δx ′, δy ′), the positions of the photographed image g1 and the photographed image g2 are adjusted (# 317), for each interpolated pixel position of the image data G ″, the average value of the interpolation data G ″ of the photographed image g1 and the interpolation data G ″ of the photographed image g2 is set as data after synthesis (# 319). .

The processing in step # 317 will be described with reference to the example of FIG. 41. As shown in FIGS. 41A and 41B, the image data of the second image is added to the image data of the first image by Δx. The luminance characteristics P of the subject are shifted to match.

In the process of step # 319, since the two photographed images whose pixel densities are n times are aligned with each other in pixel pitch units, the data of the first photographed image and the two photographed images are located at the same pixel position. Since the data of the photographed image of the eye exists, the average value of these data is used as the data of the pixel position.

Returning to FIG. 43, subsequently, at step # 321
In steps # 327 to # 327, the image of the R color component is subjected to image data combining processing in the same manner as in the case of the G color component.
In steps # 329 to # 335, the image of the B color component is subjected to image data synthesis processing in the same manner as in the case of the G color component.

The image data of the R color component is shown in FIG.
Since the image data R exists only at the pixel position (2h + 1, 2k + 1) as shown in (a), other pixel positions (2h + 2, 2k + 1), (2h) as shown in FIG.
After the image data R 'of (+1, 2k + 2) and (2h + 2, 2k + 2) are interpolated, the pixel data R "is interpolated between the pixel positions for each row and column as shown in FIG. The image data B ′ and B ″ are interpolated in the same manner for the component image data.

Then, the image data of each of the R, G, and B color components after the super-resolution processing is synthesized to increase the pixel density by n times (FIG. 45,
A super-resolution image g3 of 4 × in the example of FIG. 46 is created (#
337), the super-resolution image g3 is displayed on the entire screen of the display 702 as shown in FIG. 47 (# 33).
9). In the super-resolution processing, the super-resolution image g3 is displayed on the display 702 in order to easily recognize the effect of the processing result.
To be displayed on the entire screen. If the processing effect can be recognized, the super-resolution image g3 may be displayed on a partial screen of the display 702.

Then, when the cursor K is moved to the save button 21 and the mouse 704 is double-clicked (# 34)
If YES at 1), the image data of the super-resolution image g3 is stored in a predetermined memory in the computer main body 7 (# 34).
3).

As described above, a set of two shot images shot by the digital still camera 1 in the bokeh adjustment mode, the gradation adjustment mode, and the super-resolution mode is processed by the image processing device 6 including a computer to obtain R, After the alignment process of the captured images is performed for each of the G and B color components, a color component image having a desired degree of blur and gradation characteristics and a color component image having a high resolution are created by a predetermined combination process. Further, by combining these color component images, a color image having a desired degree of blur and gradation characteristics and a high-resolution color image are created. For the G color component that has a large effect, alignment processing is performed using image data at all pixel positions, and for the R and B color components that have a small effect on resolution due to human visual characteristics, Since the alignment processing is simplified by using the alignment information calculated for the color components of the color components R, G, and B, the same alignment processing is repeated for each of the R, G, and B color components. Processing time can be shortened.

Also, in predetermined image processing such as bokeh adjustment, gradation adjustment, and super-resolution processing for each color component, image processing of the G color component image is performed using image data at all pixel positions. Since the image processing of the R and B color components is performed without using the image data of all the pixel positions, the image processing of all the color components is performed using the image data of all the pixel positions. The processing time can be reduced as compared with the case where the processing is repeated.

Specifically, in this embodiment, since the image is photographed using the Bayer-type single-chip color CCD, the images of the R, G, and B color components constituting the photographed image are insufficient. Although it is necessary to interpolate the image data at the pixel position, in predetermined image processing such as bokeh adjustment, gradation adjustment, and super-resolution processing for each color component, the G color component image is completely interpolated. Since the image processing is performed by interpolating the image data at the pixel position, and the image processing of the R and B color components is performed without using the interpolation processing, the image processing is performed using the original image data. After interpolating the image data of all pixel positions for the image of each color component, the processing time can be shortened as compared with the case where image processing is performed.

Note that the image of the R and B color components is
Since the predetermined image processing is completed and the image data at the pixel position that is insufficient when the color component images are combined is interpolated, the image quality of the combined image does not deteriorate.

In the above embodiment, the image processing is R,
The case of performing with G and B color components has been described.
The present invention is also applicable to the case where the processing is performed using the Cr and Cb color components, and the image processing for the image of the Cr and Cb color components may be simplified compared to the image processing for the Y color component. That is, in general, when image processing is performed by separating color components having a large influence on resolution and color components having a small effect on resolution but having a large effect on color reproducibility in terms of human visual characteristics, the latter color component The image processing may be simplified compared to the former image processing of the color component image (for example, the interpolation processing for equalizing the number of data to be processed or the alignment processing may be omitted).

In the above-described embodiment, the personal computer 7 is used to combine two photographed images photographed in the blur adjustment mode, the gradation adjustment mode, and the super gradation processing mode. ROM 108 of still camera 1
a, the above-described bokeh adjustment processing program, gradation adjustment processing program, and super-resolution processing program are installed, and the overall control unit 108 performs a synthesis process of two captured images according to these programs, and the processing result is As shown in FIG. 48, the information may be recorded on the memory card MC.

FIG. 48 is a diagram showing an example of the configuration of the image file in the memory card MC shown in FIG. 8, which is obtained by synthesizing each of the photographed images in the blur adjustment mode, the gradation adjustment mode, and the super-resolution mode with the holder. Image files "P000001.JPG", "P000003.JP"
G "," P000005.JPG "," P00000
7. JPG ”.

As can be seen from these file names, the image data after the synthesizing process is compressed by the JPEG system. The reason that not only the image data after the synthesizing process but also the raw data of the photographed image is recorded in the memory card MC is that the personal computer 7 can perform the image processing. In order to increase the capacity efficiency of the memory card MC, only the image data after the synthesizing process may be recorded on the memory card MC.

When the image processing function of the digital still camera 1 is installed, if the synthesis processing is performed by the overall control unit 108 using software, the processing load on the overall control unit 108 increases. The configuration may be performed.

FIG. 49 is a block diagram showing the internal configuration of a digital still camera having a super-resolution mode in which super-resolution processing in this mode is executed by hardware. FIG. 50 is a block diagram showing the internal configuration of the multiple image processing unit.

The figure is different from the signal processor 103 shown in FIG.
And a multi-image processing unit 111 for performing a combining process of two captured images captured in the super gradation processing mode. Members having the same functions as those shown in FIG. 7 are denoted by the same reference numerals.

As shown in FIG. 50, the composite image processing section 111 converts the image data of the R, G, and B color components into luminance (Y),
An RGB / YCrCb converter 111a for converting color difference (Cr = RY) and color difference (Cb = BY) image data, filters 111b and 111c, and interpolation processors 111d and 111.
e, a registration processing unit 111f and an image synthesizing unit 111g.

When the RGB / YCrCb conversion unit 111a receives the image data of the R, G, and B color components of the first captured image and the second captured image from the overall control unit 108, the image memory 103f After that, the luminance data Y is created by adding and synthesizing each photographed image at a ratio of R: G: B = 0.3: 0.59: 0.11, and R, G, The luminance data Y is subtracted from the B image data to create color difference data Cr and Cb. The reason that only the first captured image is used for the color difference data Cr and Cb is that the color difference data Cr and Cb have little effect on the resolution, and therefore the alignment process between the first captured image and the second captured image is performed. Don't
This is to make the processing speed as fast as possible. For the color difference data Cr and Cb, only the second captured image may be used.

When the luminance data Y of both photographed images is created, a noise component is removed from the luminance data Y of each photographed image by the filter circuit 111b, and then the data is interpolated by the interpolation processing circuit 111d. This interpolation processing is performed by the data C (1) ′, C (2) ′,
.., Data D (1) ′, D (2) ′,.
In the flowcharts of FIGS. 43 and 44, step # 30
7, # 308.

Also, the color difference data Cr,
When Cb is created, the color difference data Cr and Cb are subjected to a data interpolation process by an interpolation processing circuit 111e after a noise component is removed by a filter circuit 111c. Note that this interpolation processing is performed by the data C (1) ′, C (2) ′,..., Data D (1) ′, D (2) ′,.
43. In the flowcharts of FIGS. 43 and 44, this corresponds to the interpolation processing of steps # 321 and # 329.

For the luminance data Y, when the interpolation processing is completed, the registration processing section 111f aligns the first photographed image with the second photographed image. This processing corresponds to steps # 313 to # 319 in the flowcharts of FIGS. 43 and 44, and obtains high-resolution image data for the luminance data Y. On the other hand, with respect to the color difference data Cr and Cb after the interpolation, the alignment between the first captured image and the second captured image is not performed.

The luminance data Y subjected to the registration processing and the color difference data Cr,
Cb is combined by the image processing unit 111g (corresponding to step # 337 in the flowchart of FIG. 44), and the combined image data is stored in the image data 103f.

In this embodiment, in order to make the processing speed as fast as possible, no registration processing section is provided for the color difference data Cr and Cb, but the registration processing section is also provided for the color difference data Cr and Cb. A part may be provided.

Further, in the above embodiment, the initial value of the registration processing in the super-resolution processing is calculated by the simulation using the thumbnail image, but the registration processing of the photographed image is performed directly without performing the simulation. You may do so.

In the gradation adjustment processing, a simulation is always performed using a thumbnail image. However, the simulation may be performed when a simulation instruction is issued, similarly to the blur adjustment processing.

[0289]

As described above, according to the present invention,
For two color images taken by a color imaging device,
An image processing system that performs a predetermined image processing for each color component to create one color component image, and combines the images of each color component to create one color image, wherein an image of a plurality of color components is provided. Of these, the image processing for a color component having a small effect on resolution in terms of human visual characteristics is simplified compared to the image processing for other color components, so the same image processing is repeated for all color components As compared with the case, the time of the image processing becomes shorter, and the time of the combining processing can be reduced as much as possible without deteriorating the image quality of the combined image.

Further, the same subject is imaged twice consecutively, and the two color images obtained thereby are subjected to predetermined image processing for each color component to form one color component image. A color imaging apparatus that creates and combines images of respective color components to create a single color image, wherein a color image that has a small effect on resolution in terms of human visual characteristics among captured images of a plurality of color components is provided. Image processing is simplified compared to image processing for other color components, so image processing time is shorter than when the same image processing is repeated for all color components. It is possible to obtain an image having a different image quality from the captured image in the shortest possible time without deteriorating the image quality.

Further, by recording a processing program comprising the above-described image processing method on an information recording medium, a processing program of the image processing method is mounted on the image processing apparatus from a color imaging device or a computer via the information recording medium. Accordingly, it is possible to easily construct a color imaging device or an image processing system having a function of obtaining an image having a different image quality from a captured image by image synthesis.

[Brief description of the drawings]

FIG. 1 is a front view of a camera body of a digital still camera applied to an image processing system according to the present invention.

FIG. 2 is a view showing an arrangement of main members of the digital still camera.

FIG. 3 is a right side view showing an arrangement of main members built in the digital still camera.

FIG. 4 is a rear view of the digital still camera.

FIG. 5 is a diagram illustrating a configuration of an imaging surface of a color imaging device.

FIG. 6 is a diagram for explaining a method of selecting a shooting mode.

FIG. 7 is a block diagram showing an internal configuration of the digital still camera.

FIG. 8 is a diagram showing a configuration example of an image file recorded on a memory card.

FIG. 9 is a diagram showing a method of recording an image file on a memory card.

FIG. 10 is a flowchart illustrating a shooting procedure in the blur adjustment mode.

FIG. 11 is a flowchart showing a shooting procedure in a gradation adjustment mode.

FIG. 12 is a flowchart illustrating a shooting procedure in a super-resolution mode.

FIG. 13 is an external view illustrating a configuration of an image processing apparatus.

FIG. 14 is a flowchart illustrating a processing procedure for activating a processing program for image composition.

FIG. 15 is a diagram illustrating an example of a blur adjustment dialog box displayed on the display when the blur adjustment processing program is activated.

FIG. 16 is a flowchart of an image file reading process.

FIG. 17 is another flowchart of an image file reading process.

FIGS. 18A and 18B show two images having different focus states used in the blur adjustment processing, in which FIG. 18A is a near-field focused image and FIG. 18B is a far-field focused image.

FIG. 19 is a diagram showing an example of a dialog box displayed on a display for setting parameters necessary for the blur adjustment processing.

FIG. 20 is a flowchart showing a processing procedure of a blur adjustment process.

FIG. 21 is a flowchart illustrating a processing procedure of a blur adjustment process.

FIG. 22 is a flowchart illustrating a processing procedure of a blur adjustment process.

FIG. 23 is a flowchart illustrating a processing procedure of a blur adjustment process using a thumbnail image.

FIG. 24 is a diagram showing display contents on a display in a case where a blur adjustment process is simulated using a thumbnail image.

25A and 25B are diagrams for explaining an image data interpolation process for a G color component. FIG. 25A is a diagram illustrating image data before interpolation, and FIG. 25B is a diagram illustrating image data after interpolation.

FIG. 26 is a diagram illustrating a method of creating image data for iterative operation processing by thinning out image data of the R color component;
(A) is a diagram showing image data before thinning, and (b) is a diagram showing image data after thinning.

FIG. 27 is a diagram for explaining a method of creating a blur adjustment image by interpolating image data for the R color component;
(A) is a diagram showing image data before interpolation, (b) is a diagram showing a state where image data is restored to the original pixel position, and (c) is a diagram showing image data after interpolation.

FIG. 28 is a diagram illustrating an example of a dialog box displayed on a display to display a blur adjustment image.

FIG. 29 is a diagram showing an example of a dialog box displayed on a display for inputting a condition for saving a blur adjustment image.

FIG. 30 is a diagram illustrating an example of a dialog box for tone adjustment displayed on the display when the tone adjustment processing program is activated.

FIG. 31 is a diagram illustrating a display example of a simulation result when a gradation adjustment is simulated with five types of addition ratio characteristics preset by a variation button.

FIG. 32 is a diagram illustrating a difference in an A / D conversion level between an overexposed captured image and an underexposed captured image.

FIG. 33 is a diagram illustrating an example of a combination ratio of an overexposed photographed image and an underexposed photographed image in the gradation adjustment processing.

FIG. 34 is a diagram illustrating A / D conversion levels of an overexposed photographed image, an underexposed photographed image, and a gradation adjusted image with respect to the luminance level of the same subject.

FIG. 35 is a flowchart of a tone adjustment process using two shot images having different exposure values shot in the tone adjustment mode.

FIG. 36 is a flowchart illustrating a processing procedure of gradation adjustment processing.

FIG. 37 is a flowchart illustrating a procedure of a tone adjustment process using a thumbnail image.

FIG. 38 is a diagram illustrating an example of an addition ratio characteristic in a variation process.

FIG. 39 is a diagram illustrating an example of a dialog box displayed on a display to display a created gradation adjustment image.

FIG. 40 is a diagram illustrating an example of a dialog box for super-resolution adjustment displayed on the display when the super-resolution processing program is activated.

FIG. 41 is a diagram for describing super-resolution processing.

FIG. 42 is a diagram illustrating an example of a dialog box displayed on a display for setting a correction method in super-resolution processing.

FIG. 43 is a flowchart showing a processing procedure of super-resolution processing.

FIG. 44 is a flowchart showing a processing procedure of super-resolution processing.

FIG. 45 is a diagram for describing a method of interpolating image data of a G color component.

FIG. 46 is a diagram for explaining a method of interpolating image data of an R color component.

FIG. 47 is a diagram illustrating a display example of a processing result of the super-resolution processing.

FIG. 48 is a diagram illustrating a method of recording an image file on a memory card when performing image combining processing with a digital still camera.

FIG. 49 is a block diagram showing an internal configuration of a digital still camera having an image processing function.

FIG. 50 is a block diagram illustrating an internal configuration of a multiple image processing unit.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Digital still camera (color imaging device) 101 lens 102 imaging part 103 signal processing part 104 light emission control part 105 lens control part 106 display part 107 operation part 108 card I / F 108 whole control part 109 card interface 110 communication interface 111 plural Image processing unit (image processing unit, image synthesizing unit) 111a RGB / YCrCb conversion unit 111b, 111c filter 111d, 111e interpolation processing unit 111f registration processing unit 111g image synthesis unit 2 camera body 205 color image sensor 206 shutter button 207 display unit 208 Power switch 209 Quad switch 210 Switch group 3 Interchangeable lens 4 Electronic viewfinder 5 Flash 6 Image processing device 7 Personal computer 701 Compiler Computer body (image processing device) 701b Memory card reader 701c CD-ROM driver 702 Display 703 Keyboard 704 Mouse 9 CD-ROM MC Memory card (recording medium)

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 9/04 H04N 9/64 R 5C079 9/07 9/69 9/64 101: 00 9/69 1 / 40 D // H04N 101: 00 1/46 Z (72) Inventor Noriyuki Okisu 2-13-13 Azuchicho, Chuo-ku, Osaka City Osaka International Building Minolta Co., Ltd. (72) Inventor Shinichi Fujii Chuo-ku, Osaka City 2-13-13 Azuchicho Osaka International Building Minolta Co., Ltd. F-term (reference) 5B057 BA02 BA23 BA26 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CC02 CD05 CE08 CE11 CE16 CE18 CH08 CH11 CH18 DA07 DA16 DA17 5C065 AA03 BB02 BB03 BB03 CC DD02 DD17 EE05 GG13 GG17 GG21 GG26 5C066 AA01 AA11 BA20 CA05 EA14 EC05 EE01 GA01 GA02 GA05 KE02 KE03 KE07 KM02 KM13 5C076 AA19 AA21 AA22 AA26 AA27 AA40 BA01 BA06 BB25 BB32 5C0PP PP20 PP23 PP32 PP34 PP58 PQ08 PQ12 RR19 TT09 5C079 HB01 HB04 JA01 LA02 LA12 LA17 LA28 LA31 LA37 LA40 MA11 NA02 NA04 NA05 NA11 NA27 PA00

Claims (21)

[Claims] 1. A color imaging device for continuously capturing two color images of the same subject in order to obtain an image having a different image quality from a captured image by image synthesis, and two images captured by the color imaging device. Image processing means for performing a predetermined image processing for each color component on the color image to create one color component image, and combining the image of each color component created by the image processing means into one An image processing system comprising: an image synthesizing unit that generates a color image; and wherein the image processing unit has an effect on resolution of human visual characteristics among the plurality of color component images. An image processing system that simplifies a process for an image of a first color component with a smaller value than a process for an image of another second color component. 2. The predetermined image processing for each color component includes a positioning process for performing alignment between photographed images, and a predetermined operation relating to image quality between the aligned photographed images to perform one color component processing. The image processing means simplifies the positioning process for the image of the first color component compared to the positioning process for the image of the second color component. The image processing system according to claim 1, wherein: 3. The image processing device according to claim 1, wherein the image processing unit performs position alignment processing on the image of the first color component using positional deviation information calculated by alignment processing on the image of the second color component. 3. The image processing system according to claim 2, wherein the positioning process for the first color component is simplified. 4. The predetermined image processing for each component includes an interpolation process of interpolating data for each photographed image to increase the number of original image data before the alignment process. 4. The image processing system according to claim 2, wherein the interpolation processing for the image of the first color component is simplified compared to the interpolation processing for the second color component. 5. The image processing means according to claim 2, wherein the number of data interpolations of the image of the first color component in the interpolation processing is the second number.
By making it smaller than the number of data interpolations of the image of the color component of
5. The image processing system according to claim 4, wherein the interpolation processing for the first color component is simplified. 6. The image processing means according to claim 1, wherein in the calculation processing, the first color component image data number is equal to the second color component image data number. The image processing system according to claim 5, wherein the number of data of the component image is interpolated. 7. The color components are R (red), G (green), B
7. A color component (blue), wherein the first color component is an R and B color component and the second color component is a G color component. An image processing system according to claim 1. 8. The color component includes Y (luminance), Cr (red color difference), and Cb (blue color difference) color components, the first color component is a Cr, Cb component, and the second color component is a second color component. 7. The image processing system according to claim 1, wherein the color component is a Y component. 9. The image processing system according to claim 2, wherein the predetermined calculation processing relating to the image quality is a blur adjustment processing for adjusting a degree of blur of focus of the image. 10. The predetermined arithmetic processing relating to the image quality,
3. The image processing system according to claim 2, wherein the image processing is a gradation adjustment process for adjusting a gradation of an image. 11. The predetermined arithmetic processing relating to the image quality,
3. The image processing system according to claim 2, wherein the image processing is super-resolution processing for increasing the resolution of the image. 12. An image pickup means for photoelectrically converting a light image of a subject into an image signal and outputting the image signal.
An image capturing control unit for performing the image capturing operation of the image capturing unit; and performing a predetermined image processing for each color component on the two color images captured by the image capturing unit to generate one color component image. An image processing apparatus comprising: an image processing unit that performs image processing; and an image combining unit that combines images of the respective color components created by the image processing unit to create a single color image. It is characterized in that, among images of a plurality of color components, processing on an image of a first color component having a small effect on resolution due to human visual characteristics is simplified compared to processing on an image of another second color component. Imaging device. 13. The predetermined image processing for each color component includes a positioning process for performing alignment between photographed images, and a predetermined operation relating to image quality between the aligned photographed images to perform one color component processing. The image processing means includes an arithmetic process for creating an image, wherein the image processing means uses the positional displacement information calculated in the alignment process for the image of the second color component to perform an alignment process for the image of the first color component. 13. The method according to claim 12, wherein the step (c) simplifies the positioning process for the image of the first color component.
An imaging device according to claim 1. 14. The predetermined image processing for each color component includes:
The image processing means includes an interpolation process for increasing the number of original image data by interpolating data for each photographed image before the positioning process, wherein the image processing means performs data interpolation of the image of the first color component in the interpolation process. 14. The image processing apparatus according to claim 12, wherein the number is smaller than the data interpolation number of the image of the second color component, thereby simplifying the interpolation processing for the image of the first component. Imaging device. 15. The image processing means according to claim 1, wherein in the calculation processing, the number of data of the image of the first color component is equal to the second data number.
So that the number of data of the image of the color component
The image pickup apparatus according to claim 14, wherein the number of data of the image of the color component is interpolated. 16. The color components are R (red), G (green), B
The color component of (blue), wherein the first color component is an R and B color component and the second color component is a G color component. An imaging device according to any one of the preceding claims. 17. The predetermined arithmetic processing relating to the image quality,
13. The imaging apparatus according to claim 12, wherein the blur adjustment processing is for adjusting the degree of out-of-focus of the image. 18. The predetermined arithmetic processing relating to image quality,
The image pickup apparatus according to claim 12, wherein the image pickup apparatus performs a gradation adjustment process for adjusting a gradation of an image. 19. The predetermined arithmetic processing relating to the image quality,
13. The image pickup apparatus according to claim 12, wherein the image pickup apparatus performs a super-resolution process for increasing an image resolution. 20. Performing predetermined image processing for each color component on a plurality of color images obtained by continuously imaging the same subject so as to obtain an image having a different image quality from the captured image by image synthesis. An image processing method comprising: an image processing step of creating a single component image; and an image combining step of combining images of respective color components created in the image processing step to create one color image. In the image processing step, of the plurality of color component images, processing on an image of a first color component which has little effect on resolution in terms of human visual characteristics is simpler than processing on an image of another second color component. An image processing method characterized by: 21. Performing predetermined image processing for each color component on a plurality of color images obtained by successively capturing the same subject in order to obtain an image having a different image quality from the captured image by image synthesis. A computer executes first image processing for creating one component image and second image processing for creating one color image by combining images of the respective color components created in the image processing step. Computer-readable recording medium having recorded thereon a first color component of the plurality of color component images, the first color component having a small effect on resolution in terms of human visual characteristics. A recording medium for simplifying the processing for the image of the second color component compared with the processing for the image of another second color component.