JP3408146B2 - Image processing apparatus and medium storing image processing program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and a medium for storing an image processing program for improving the image quality of an image obtained by photographing a subject under low illuminance.

[0002]

2. Description of the Related Art In recent years, so-called digital electronic still cameras and the like have been known as image equipment having an image pickup device. The image device obtains an electrical image signal by photoelectrically converting light from a subject into the image sensor,
The image signal is converted into image data and stored. The image pickup device is, for example, a so-called CCD image sensor. The image signal and the image data are analog signals and digital data representing an image composed of a plurality of pixels. The image data includes a plurality of pixel data representing the color of each pixel of the image. Hereinafter, the image signal and the image data may be abbreviated as an image represented by the image signal and the image data. The color is generally defined by three attributes of color: brightness, saturation, and hue.

One of the image processes performed on the image data to improve the image quality of the image is a brightness correction process for improving the brightness level of a pixel. When the brightness correction process is performed on the image data, the brightness component of each pixel data in the image data is amplified by a predetermined amplification factor according to the value of the brightness component before correction. Generally, the lower the brightness component value before correction, that is, the brightness level of the pixel before correction, the higher the amplification factor.

When the subject is photographed using the image device, if the illuminance of the subject is low, the luminance level of each pixel of the obtained image becomes low, and
The pixel data of each pixel is strongly influenced by so-called random noise. The random noise includes, for example, CCD noise generated due to the image sensor. Therefore, if the above-described brightness correction processing is applied to the image obtained at low illuminance as it is, the random noise is emphasized.
The image after being subjected to the brightness correction processing becomes very messy. Therefore, when the brightness correction process is performed on the image, it is necessary to further perform a process for suppressing the random noise on the image data.

FIG. 13 is a first prior art noise elimination circuit 1 relating to the processing for suppressing the random noise.
3 is a block diagram showing the electrical configuration of FIG. Noise elimination circuit 1
Is a circuit that performs processing for removing noise while keeping the appearance of noise in balance according to the signal level of the image signal, and is disclosed in Japanese Patent Application Laid-Open No. 7-111605. The noise removing circuit 1 includes three low-pass filters 3-5.
, Signal level detection circuit 6, determination circuit 7, and selector 8
With.

The image signal to be processed by the noise removing circuit 1 is given to the three low pass filters 3 to 5 and the signal level detecting circuit 6, respectively. Each low pass filter 3
5 to 5 are for removing noise from the image signal, respectively, and have different pass band characteristics. The determination circuit 7 compares the signal level of the image signal detected by the signal level detection circuit 6, that is, the luminance level of the pixel, with each of three types of predetermined threshold values. Selector 8
On the basis of the comparison result of the determination circuit 7, the lower the luminance level of the pixel is, the greater the effect of removing noise in the video signal is. Therefore, the smaller the signal level of the image signal is, the narrower the band is. Select one of the low-pass filters. The image signal output from any one selected low-pass filter is output from the selector 8 to the outside of the noise removal circuit 1.

Since the noise removing circuit 1 uses a low-pass filter to remove the noise, the image represented by the image signal output from the noise removing circuit 1 may be spatially blurred. Therefore, it is difficult to sufficiently improve the image quality of the image. Furthermore, since it is generally difficult to sufficiently remove the random noise from the image signal using only a low-pass filter, the random noise may remain in the image signal output from the noise removing circuit 1. .

The applicant of the present invention has disclosed in Japanese Patent Application Laid-Open No. Hei 5 (1999) -58200 as a second prior art relating to the processing for suppressing the random noise.
Video signal processing device 11 disclosed in Japanese Patent Laid-Open No. 91395
Is proposed. FIG. 14 is a block diagram showing an electrical configuration of the video signal processing device 11. Video signal processing device 1
Reference numeral 1 includes a camera 14, a signal level detection circuit 15, and an image averaging circuit 16.

The camera 14 continuously photographs a subject a plurality of times through the wide-angle lens 13 and sequentially supplies a video signal AS1 showing images of a plurality of frames to a signal level detection circuit 15. Further, the video signal is supplied to the image averaging circuit 16 via the circuit 17 involved in the extraction of the region in the image after being converted into digital data. The signal level detection circuit 15 detects, based on the image of each frame of the video signal, whether or not the illuminance at the time of shooting the space in which the subject is present is low. When it is detected that the illuminance is low, the signal level detection circuit 15 outputs the low illuminance detection signal S0 to activate the image averaging circuit 16. Image averaging circuit 16
Averages the image of the latest frame and the image one frame before in order to reduce random noise in the video signal. Signal level detection circuit 15 and image averaging circuit 16
Means that the above process is repeated for each frame. As a result, the image averaging circuit 16 passes the video signal AS2, in which the random noise is temporally averaged, to the video signal processing circuit 1 via the circuit 18 which is involved in another process of random noise reduction.
1 Output to outside.

[0010]

The video signal processing device 11 of the second prior art described above has the signal level detection circuit 15
And the larger the number of images processed by the image averaging circuit 16, the more the random noise is suppressed. Therefore, a large number of images, for example, four or more images are required to sufficiently remove the random noise of the video signal. Therefore, the video signal processing device 11 can be applied to an image device capable of continuously obtaining a large number of images, such as a video camera, but is applied to an image device other than the image device, such as a digital still camera. Is difficult.

Further, for example, the video signal processing circuit 11
When the camera 14 continuously captures a moving subject, the position and shape of the subject appearing in the image of each frame of the video signal are displaced from each other between the images of each frame. Therefore, the image of each frame of such a video signal is detected by the signal level detection circuit 15 and the image averaging circuit 16.
To the video signal output from the video signal processing circuit 11, a correction error due to the deviation is accumulated. This makes it difficult to improve the image quality of the image of each frame.

The object of the present invention is to obtain the image in which the random noise in the images is sufficiently suppressed and the image of a moving subject is photographed by using as few images as possible. Another object is to provide an image processing device capable of suppressing random noise and a medium storing an image processing program.

[0013]

According to a first aspect of the present invention, a processed image input means for inputting a color processed image composed of a plurality of pixels and a luminance of each pixel of the processed image are predetermined. A brightness correction unit for a processed image that corrects each pixel based on a function, and a saturation of each pixel of the processed image whose brightness component has been corrected is calculated so that the corrected brightness of each pixel is low and The image processing apparatus is characterized in that it includes processing target image saturation suppression means that suppresses the higher the saturation.

According to the present invention, the image processing apparatus further applies the saturation suppression processing to the processed image subjected to the so-called brightness correction processing by using the saturation suppression means for the processed image. The saturation suppression processing strongly suppresses the saturation as the brightness of the pixel is low and the saturation of the pixel is high. Accordingly, when so-called random noise superimposed on the image to be processed is amplified due to the brightness correction process, only the random noise can be selectively suppressed. Therefore, when the processed image is obtained by photographing a subject in a space with relatively low illuminance, the image quality of the processed image can be improved. Further, the processing target image saturation suppression means determines the degree of saturation suppression of each pixel based on the corrected luminance and saturation of each pixel itself. Therefore, the processing for suppressing the saturation can be performed for each pixel of the processed image. Therefore, the image processing device can efficiently improve the image quality of the processed image as compared with the image processing device that uniformly processes the entire processed image.

A second aspect of the present invention is auxiliary image input means for inputting at least one color auxiliary image including the same subject as the image to be processed and including a plurality of pixels.
It further includes a combining means for combining the processed image in which the saturation is suppressed and all the auxiliary images to obtain a combined image, and the color of each pixel of the combined image has the saturation suppressed. The average of the color of each pixel of the processed image and the color of each pixel of all of the auxiliary images corresponding to each pixel in the processed image.

According to the present invention, the image processing apparatus includes auxiliary image input means and combining means in addition to the first image processing apparatus. As a result, the image processing device not only suppresses the saturation of each pixel in the processed image whose brightness has been corrected, but also the at least one auxiliary image and the processed object in which the saturation is suppressed. The image is averaged and combined. The composite image obtained by the average composition is an image in which the same subject as the image to be processed is captured, and the color of the pixel of the composite image is determined as described above. by this,
At the time of synthesis, the random noise remaining in the processed image in which the saturation is suppressed is canceled by the random noise superimposed on the auxiliary image. Therefore, the image quality of the image output from the second image processing device, that is, the image quality of the combined image is improved as compared with the image quality of the processed image in which the saturation is suppressed.

In a third aspect of the present invention, the auxiliary image brightness correction means for correcting the brightness of each pixel of all the auxiliary images based on a predetermined function, and all the auxiliary images whose brightness has been corrected. Further includes auxiliary image saturation suppressing means for suppressing the saturation of each pixel as the corrected luminance of each pixel is lower and the saturation of each pixel is higher, and the combining means is The image to be processed whose saturation has been suppressed and all the auxiliary images whose saturation has been suppressed are combined.

According to the present invention, the image processing device includes, in addition to the second image processing device, auxiliary image brightness correction means and auxiliary image saturation suppression means, and the synthesizing means includes all of the above. Instead of the auxiliary image, all the auxiliary images whose luminance has been corrected and whose saturation has been suppressed are combined with the processed image whose saturation has been suppressed. Thus, the image to be combined by the combining unit, that is, the processed image and the plurality of auxiliary images are subjected to the brightness correction processing by the saturation suppression unit for the processed image and the auxiliary image before being combined by the combining unit. The random noises amplified by are suppressed respectively. As a result, the synthesizing means synthesizes a smaller number of images than the case of synthesizing the images to be synthesized without suppressing the saturation of the pixels,
It is possible to reliably prevent random noise from being superimposed on the composite image. Therefore, the synthesizing process performed by the synthesizing unit is simplified.

According to a fourth aspect of the invention, the color of each pixel of the composite image is the color of each pixel of the processed image in which the saturation is suppressed, and all the colors corresponding to each pixel of the processed image. Is a simple average with the color of each pixel of the auxiliary image.

According to the invention, the synthesizing means of the image processing apparatus, when determining the color of each pixel of the synthetic image,
The so-called simple average is used. As a result, the random noise remaining in the processed image in which the saturation is suppressed is canceled by the random noise superimposed on the auxiliary image by a simple arithmetic process. Therefore, the synthesizing unit can create a synthesized image having a higher image quality than the processed image in which the saturation is suppressed by a simple processing procedure.

In a fifth aspect of the present invention, the correlation between each pixel of each auxiliary image and each pixel of the processed image in which the saturation is suppressed, which corresponds to each pixel of the auxiliary image, is determined by calculating the correlation of each auxiliary image. The color of each pixel of the composite image may be the color of each pixel of the processed image in which the saturation is suppressed and the color of each pixel in the processed image. It is characterized in that it is an average with the color of each pixel in all of the auxiliary images that corresponds and is weighted by the correlation value.

According to the invention, the synthesizing means of the image processing apparatus, when determining the color of each pixel of the synthetic image,
A weighted average using the correlation is used. For example, the weight of the color of each pixel of all the auxiliary images is smaller as the correlation between the pixel and the pixel of the image to be processed in which the saturation is corresponding is smaller. Therefore, any one
The degree of the influence of the colors of the pixels of the one auxiliary image on the colors of the pixels of the composite image decreases as the correlation between the pixels of the auxiliary image and the pixels of the processed image decreases. by this,
For example, when there is a shift caused by so-called camera shake between the arbitrary one auxiliary image and the processed image in which the saturation is suppressed, in addition to the suppression of the random noise, blurring caused by the shift is caused in the combined image. It is possible to prevent the occurrence of an afterimage. Therefore, the fifth image processing device can improve the image quality of the composite image as compared with the processed image in which the saturation is suppressed.

A sixth aspect of the invention is a shift detecting means for detecting a shift between each of the auxiliary images and the image to be processed, and a shift detecting means for detecting the shift between each of the auxiliary images and the image to be processed. The reliability of the detection further includes reliability calculation means for each of the auxiliary images, the synthesizing means, the auxiliary image of the reliability of all the auxiliary images is equal to or more than a predetermined evaluation criteria, It is characterized in that the image to be processed whose saturation is suppressed is combined.

According to the present invention, the image processing apparatus includes the second image processing apparatus, the shift detecting means and the reliability calculating means, and the synthesizing means includes all the auxiliary images. Instead of the above, only the auxiliary images of which the reliability is equal to or higher than a predetermined evaluation standard of all the auxiliary images and the processed image in which the saturation is suppressed are combined. As a result, the detected auxiliary image of which the deviation is low in reliability is removed from the image to be combined. This is because when the synthesizing unit synthesizes the auxiliary image and the image to be processed in consideration of the shift, if the detected shift is wrong, the wrong shift and the shift are detected. This is because the image quality of the combined image may deteriorate if the supplementary image is used for the combining process. Therefore, from all of the auxiliary images, the saturation is suppressed only for the remaining images except for the auxiliary images with low reliability of the detected deviation, that is, the auxiliary images that may have an error in the deviation. The processed image is combined with the processed image. Accordingly, the sixth image processing apparatus can prevent the image quality of the composite image from being deteriorated due to the error in detecting the shift.

In a seventh aspect of the invention, a shift detecting means for detecting a shift between each of the auxiliary images and the processed image and a shift detecting means for canceling the shift between the processed image and each of the auxiliary images are provided. The image forming apparatus may further include a deforming unit that deforms each auxiliary image, and the combining unit may combine all the deformed auxiliary images with the processing image in which the saturation is suppressed.

According to the present invention, the image processing apparatus includes, in addition to the second image processing apparatus, the deviation detecting means and the deforming means for deviation correction processing. The synthesizing unit synthesizes the processed image in which the saturation is suppressed and all the deformed auxiliary images, instead of synthesizing the processed image in which the saturation is suppressed and all the auxiliary images. To do. By deforming all the auxiliary images so as to cancel the deviation, the deviation of each of the auxiliary images is corrected. Therefore, the seventh image processing apparatus, even if, for example, there is a shift due to so-called camera shake between any one auxiliary image and the processed image, the image quality of the processed image in which the saturation is suppressed is higher than that of the processed image. The quality of the composite image can be improved.

An eighth aspect of the invention is that the image to be processed has 3
It further includes a feature point extracting means for extracting feature points one by one from the set regions, and at least three of the regions have a figure whose vertex is a reference point predetermined in each region. Arranged so as to form an equilateral triangle, the deviation detection means individually searches a plurality of corresponding points in each of the auxiliary images respectively corresponding to the plurality of feature points for each of the auxiliary images, Based on the positional relationship between all the feature points in the processed image and the positional relationship between all the corresponding points in each of the auxiliary images, the deviation between the processed image and each of the auxiliary images, respectively. Characterized by seeking.

According to the present invention, the image processing apparatus further comprises the feature point extracting means in addition to the sixth or seventh image processing apparatus, and the shift detecting means specifically puts the feature points into consideration. Using the above, the deviation is detected by the above procedure. Since the feature points are extracted one by one from the plurality of regions arranged as described above in the image to be processed, the positional relationship of all the feature points is linear, and the feature points are It is possible to prevent a part of the image to be processed from being biased. Therefore, the eighth image processing apparatus, even when the image to be processed and the plurality of auxiliary images are obtained by photographing a subject having relatively low illuminance a plurality of times, shifts the image to be processed and each auxiliary image from each other. , Can be reliably detected.

In a ninth aspect of the invention, the processed image brightness correction means obtains the sum of the brightness of each pixel of the processed image and a predetermined reference brightness, and the sum is calculated based on the function. It is characterized by correction.

According to the present invention, the brightness correction means of the image processing apparatus corrects the brightness of each pixel of the image to be processed by the procedure described above. Accordingly, it is possible to prevent the random noise superimposed on the image to be processed from being amplified by the brightness correction process. Therefore, by suppressing the saturation of each pixel of the processing target image whose brightness is corrected by the above-described brightness correction processing by using the saturation suppressing unit to a degree according to the brightness and the saturation of each pixel, The random noise can be sufficiently removed. Therefore, the ninth image processing apparatus can further improve the image quality of the processed image in which the saturation is suppressed.

A tenth aspect of the invention is characterized by further comprising a saturation enhancing means for enhancing the suppressed saturation of each pixel of the image to be processed as the corrected luminance of each pixel is higher. To do.

According to the present invention, in addition to the first image processing device, the image processing device further comprises the saturation device for correcting the saturation component of each pixel lost by the process of the brightness correction means. An emphasis means is further provided. The saturation enhancing means is
For each pixel, the suppressed saturation is emphasized as the corrected luminance is higher. As a result, it is possible to correct only the saturation of each lost pixel without increasing the random noise suppressed by the saturation suppressing unit. Therefore, the image quality of the processed image in which the saturation of each pixel is emphasized is better than the image quality of the processed image in which the saturation of each pixel is suppressed. Further, the saturation enhancing unit determines the degree of saturation enhancement of each pixel based on the corrected luminance of each pixel itself. Therefore, the processing for enhancing the saturation can be performed for each pixel of the processed image. With this, the tenth image processing apparatus can efficiently improve the image quality of the processed image as compared with the image processing apparatus that uniformly processes the entire processed image.

An eleventh aspect of the present invention further includes sharpening means for sharpening the image to be processed in which the saturation of each pixel is suppressed, and the degree of sharpening is the object whose saturation is suppressed. It is characterized in that the higher the corrected pixel in the processed image is, the stronger the pixel is.

According to the invention, the image processing apparatus further comprises a sharpening means in addition to the first image processing apparatus. The sharpening degree of the sharpening means is stronger in a region including the pixel having the corrected high brightness in the image to be processed whose saturation is corrected. As a result, the eleventh image processing apparatus can sharpen the image in which the saturation is suppressed without increasing the random noise suppressed by the saturation suppression unit. Therefore, the image quality of the sharpened image to be processed is better than that of the processed image in which the saturation of each pixel is suppressed.

A twelfth invention is a medium for storing an image processing program for improving the image quality of a color processed image composed of a plurality of pixels, wherein the image processing program is the processed image. The luminance of each pixel is corrected for each pixel based on a predetermined function, and the saturation of each pixel of the processed image whose luminance has been corrected is It is a medium for storing an image processing program characterized in that the higher the saturation of each pixel is, the more the saturation is suppressed.

According to the present invention, the medium stores the above-mentioned image processing program. Therefore, when the image processing program is installed in the computer and the computer is caused to execute the image processing program, the central processing circuit of the computer functions as a brightness correction unit and a saturation suppression unit of the first image processing apparatus. work. As a result, the image quality of the processed image provided to the computer is improved. Therefore, the image processing apparatus according to claim 1 can be easily realized by using the computer.

[0037]

1 is a block diagram showing an electrical configuration of an image creating apparatus 21 including an image processing apparatus 22 according to an embodiment of the present invention. The image creating device 21 includes a photographing device 23 in addition to the image processing device 22. The image processing device 22 includes a communication unit 29, an image quality improvement unit 31, and a storage unit 30.
Including and The imaging device 23 includes an optical processing unit 25, an image sensor 26, an analog / digital conversion circuit 27, and a storage unit 28. The image processing device 22 and the imaging device 23 are electrically connected via a connection cable 33, and data and signals can be transmitted and received mutually. The connection cable 33 is attachable to and detachable from the image processing device 22 and the image capturing device 23, for example, and is attached and detached by the operator of the image creating device 21.

The image processing device 22 roughly determines the image quality of at least one of the plurality of images to be processed, with a plurality of images each showing a single subject as the images to be processed. Image quality improvement processing is performed for improvement. The at least one image with improved image quality,
Hereinafter, it will be referred to as an “output image”. The image capturing device 23 uses a plurality of image data 3 representing the plurality of images to be processed.
In order to obtain 5, roughly, the subject is continuously photographed a plurality of times.

The single image data is digital data representing one color image formed by arranging a plurality of pixels in a matrix, and a plurality of pixel data individually corresponding to a plurality of pixels are obtained. Including. Any one of the pixel data is for representing the color of the corresponding pixel, that is, the luminance, the saturation, and the hue, and includes one or a plurality of components according to the color system of the image. In general, the storage units 28 and 30 often collectively read and write a set of a plurality of image data. Hereinafter, a set of a plurality of image data will be referred to as "image file".

The general operation of the parts 25 to 28 inside the image pickup device 23 in one image pickup operation for obtaining a single image data is as follows. Light from the subject and the space in which the subject is present is imaged on the imaging surface of the image sensor via the optical processing unit 25. Optical processing unit 25
Performs a predetermined optical process on the light while the light passes through the optical processing unit 25. The image pickup device 26 photoelectrically converts the light to generate an image signal which is an electric signal representing an image. The A / D conversion circuit 27 quantizes the image signal to generate the image data. The image data is stored in the storage unit 28 of the photographing device 23. The above operation is the operation of the components 25 to 28 in one shooting operation.

The operator of the image forming device 21 is the photographing device 2
In order to generate the plurality of image data 35 to be processed in 3, the angle of the photographing device 23 is fixed and the operation unit provided in the photographing device 23 is operated a plurality of times. The operation section is for instructing the execution of the above-described shooting operation, and is, for example, a so-called shutter button. Each time the operation unit is operated, the image capturing device 23 captures an image of a subject according to the procedure described above. While the photographing operation is performed a plurality of times, the angle of the photographing device 23, that is, the positional relationship between the photographing device 23 and the subject needs to be substantially maintained at the positional relationship at the time of the first photographing. As a result, a plurality of image data 35 to be processed are obtained. The plurality of image data 35 form a single image file 36.

When the communication unit 29 operates while the photographing device 23 and the image processing device 22 are connected by the connection cable, the image file 36 is stored in the storage unit 2 of the photographing device 23.
8 is provided to the storage unit 30 of the image processing apparatus 22 via the connection cable 33 and the communication unit 29 and stored therein.
After the image file 36 is provided to the storage unit 30 of the image processing device 22, the image file 36 may or may not remain in the storage unit 28 of the photographing device 23.
The storage unit 30 of the image processing device 22 can store a plurality of image files.

The operator selects any one image file from all the image files in the storage unit 30 of the image processing device 22, and transfers the selected one image file to the image processing device 22. And specify it to the image quality improvement section 31,
The image quality improvement processing is performed based on a plurality of image data in any one of the image files. The processing result of the image quality improvement processing, that is, the image data representing at least one output image constitutes a single image file 37. The image file 37 showing the processing result is stored again in the storage unit 30. When the image file 37 showing the processing result is reproduced, the processing result of the image quality improving processing,
That is, the output image is obtained.

The photographing device 23 is realized by, for example, a so-called electronic still camera or a so-called video camera. The image pickup device 26 is, for example, a so-called CCD device,
It is realized by a CCD image sensor. Image processing device 22
The storage unit 28 of is realized by, for example, a hard disk device. The storage unit 30 of the imaging device 23 is realized by, for example, a storage device that uses a magneto-optical disk, for example, a so-called mini disk as a recording medium. The optical processing unit 70 includes, for example, a lens.

The photographing device 23 may further include an image compression unit interposed between the analog / digital conversion circuit 27 and the storage unit 28. The image compression unit,
The image data output from the analog / digital conversion circuit 27 is compressed based on a predetermined compression method, and the compressed image data is stored in the storage unit 28.
The compression method is, for example, JPEG (Joint Photogra
phic coding ExpertGroup). As a result, the data amount of the image data is reduced as compared with the uncompressed image data, so that the number of image data stored in the storage unit 28 can be increased. Furthermore, the photographing device 23 is provided with a continuous photographing control unit for controlling the internal parts 25 to 28 in advance, and in response to the operation unit being operated once, the continuous photographing control unit is set Parts 2
The photographing operation according to the above-described steps 5 to 28 may be repeatedly executed a plurality of times. As a result, a plurality of images required for the image quality improvement processing can be obtained by the operator only operating the operation unit once. Therefore, the operation of the photographing device 23 is simplified, which is preferable.

The image quality improving process performed by the image quality improving unit 31 will be specifically described below. In the image quality improving unit 31, images are always handled in the form of image data. In the following description, the image data and the pixel data will be replaced with the images and pixels respectively represented by the respective data, and the image data may be referred to as an “image” and the pixel data may be referred to as a “pixel”. Further, in the following description, it is assumed that all the image data in any one image file designated by the operator among all the image files 36 in the storage unit 30 are a plurality of images to be processed. I assume. The plurality of images to be processed are obtained by the photographing device 23 photographing the subject a plurality of times while substantially maintaining the positional relationship between the photographing device 23 and the subject.

When the illuminance in the space where the subject is present is relatively low, so-called random noise may be superimposed on each of the plurality of images to be processed. This is because generally, the lower the illuminance, the lower the brightness of all the pixels forming the image obtained by shooting the subject, and the electrical and optical noises are more likely to affect the pixel data. Because. Among the noises, the random noises in which it is difficult to uniquely determine the appearance frequency and the appearance place include, for example, so-called CCD noise generated due to the image pickup device. Hereinafter, when the illuminance in the space is relatively low, an image obtained by shooting a subject in the space is referred to as a “low illuminance image”. The case where the illuminance is relatively low is, for example, a case where only one so-called room light is lit in a pitch-dark room, and, for example, one or two fluorescent lights are lit at night or outdoors. If there is.

Further, since the plurality of images to be processed are obtained by continuously photographing a single scene by the photographing device 23, the same scenes should be reflected in each other. At least one of the position in which the subject is captured and the shape of the captured subject may be displaced from each other between the plurality of images. The above-mentioned shift may occur, for example, during the above-described multiple shootings.
It occurs because the subject moves or the photographing device 23 shakes. The shaking of the photographing device 23 is caused by so-called camera shake when the operator holds the photographing device 23 by hand, for example.

In the image quality improving process, any one of the plurality of images to be processed is a reference image Ir which is a processed image whose image quality is improved by the image quality improving process.
And Further, an auxiliary image Iq (1) for using at least one remaining image other than the reference image Ir among the plurality of images, that is, the remaining N images as a reference for improving the image quality of the reference image Ir. ~ Iq (N). N is an arbitrary natural number of 1 or more. In addition, an arbitrary natural number of 1 or more and N or less is described as n.

Further, a plurality of images Ir, Iq to be processed are provided.
(1) to Iq (N) are stored in the storage unit 30 to the image quality improvement unit 31.
Are given in the so-called RGB color system. Images Ir, Iq (1) to Iq
When (N) is represented by the RGB color system, any one of the pixel data includes color components Rx, Gx, Bx of three colors, that is, so-called R component, G component and B component. Each color component indicates the brightness of red, green, and blue light, respectively. In the following description, each color component Rx, Gx, Bx is assumed to represent a numerical value representing each of the above luminances, that is, a pixel value. The combination of the three colors is not limited to red, green, and blue, and other combinations may be used as long as they are a combination of a plurality of colors that are mixed into white or black.

FIG. 2 is a block diagram showing a specific functional configuration of the image quality improving section 31. The image quality improvement unit 31 includes a reference preprocessing unit 41, a feature point extraction unit 42, and a corresponding point search unit 43.
(1) to 43 (N), deforming portions 44 (1) to 44 (N),
It includes auxiliary pre-processing units 45 (1) to 45 (N), a plural-sheet combining unit 46, and a post-processing unit 47. Reference preprocessing unit 41
Includes a brightness correction unit 51 and a color noise removal unit 52. The auxiliary pre-processing units 43 (1) to 43 (N) include the brightness correction units 53 (1) to 53 (N) and the color noise removal unit 5, respectively.
4 (1) to 54 (N). The post-processing unit 47 includes a saturation enhancement unit 55 and a sharpening unit 56. Corresponding point search unit 43
(1) to 43 (N) and deforming portions 44 (1) to 44 (N)
And the auxiliary pre-processing units 45 (1) to 45 (N) are each one less than the number N + 1 of all images to be processed, that is, the auxiliary images Iq (1) to Iq.
There are as many as (N). The reference image Ir is given to the brightness correction unit 51 of the reference preprocessing unit 41, the feature point extraction unit 42, and all the corresponding point search units 43 (1) to 43 (N). Each of the auxiliary images Iq (1) to Iq (N) corresponds to each of the corresponding point searching units 43 (1) to 43 (N) and each of the transforming units 44 (1) to.
44 (N), respectively.

The arbitrary one corresponding point searching unit 43 (n), the arbitrary one transforming unit 44 (n), and the optional auxiliary preprocessing unit 45 (n) are combined to generate an optional auxiliary image Iq. (N)
It is used to treat the. In the following description, any one of the parts 43 (n), 44 (n), 45 (n) in the image quality improving part 31 having the same value of the reference character n of the reference characters is mutually equal.
It is assumed that the portion including is used for the processing on any one of the auxiliary images Iq (n) having the same reference characters (n). The portion is referred to as "auxiliary processing unit 48 (n)". The operation of each of the auxiliary processing units 48 (1) to 48 (N) and the parts inside thereof is the same for all the auxiliary images Iq (1) to Iq.
Only one of (N) is the processing target,
The others are equal to each other. Therefore, in the following description, the corresponding point searching units 43 (1) to 43 (N) and the deforming units 44 (1) will be described.
~ 44 (N) and each auxiliary pretreatment section 45 (1) -45
The respective operations with (N) are performed by the corresponding point searching unit 43 (n) and the transforming unit 44 in any one auxiliary processing unit 48 (n).
(N) and the reference preprocessing unit 45 (n) will be described as an example.

The reference preprocessing section 41 adds the reference image Ir to
Pre-determined pre-processing is performed. In the pre-processing, the reference image I
A brightness correction process for improving the brightness of the entire r and a color noise removal process for removing color noise from the brightness-corrected reference image Ir * are included. That is, first, the brightness correction unit 51 of the reference preprocessing unit 41 corrects the brightness of each pixel of the reference image Ir for each pixel as a brightness correction process based on a predetermined function. Next, the color noise removing unit 52 of the reference preprocessing unit 41 performs the reference in order to suppress the random noise in the reference image Ir * in which the brightness of the pixel is corrected by the brightness correcting unit 41 as the color noise removing process. The saturation of each pixel in the image Ir * is suppressed. The degree of suppression of the saturation of any one pixel is greater as the brightness of the pixel itself is lower and the saturation of the pixel itself is higher. The random noise is one of the causes of color noise in an image.
As described later, the luminance correction unit 51 performs the luminance correction processing on the reference image Ir, so that the reference image Ir is amplified. As a result, the reference image Ir is subjected to the preprocessing.
The preprocessed reference image Ir $ is provided from the color noise removing unit 52 to the combining unit 46.

The feature point extraction unit 42 extracts a plurality of feature points pa1 to paM from the reference image Ir, and the reference image Ir of the plurality of feature points pa1 to paM in the reference image Ir.
The coordinates within are all corresponding point search units 43 (1) to 43
(N) respectively. The feature points pa1 to paM are used as a reference for detecting the deviation between the reference image Ir and the auxiliary image Iq (n). In this embodiment, it is assumed that the number M of feature points is three. Feature points pa1 to paM
Are preferably points in the remaining regions of the reference image Ir other than the low-luminance region including many low-luminance pixels.

Any one corresponding point search unit 43 (n)
First, from within a single auxiliary image Iq (n), a plurality of corresponding points pb individually corresponding to a plurality of feature points pa1 to paM.
1 to pbM are searched. Next, the corresponding point searching unit 43
(N) is a plurality of feature points pa1 to pa in the reference image Ir.
Based on the positional relationship between M and the positional relationship between a plurality of corresponding points pb1 to pbM in the auxiliary image Iq (n),
A shift between the reference image Ir and the single auxiliary image Iq (n) is obtained. The shift information indicating the shift is given from the corresponding point searching unit 43 to any one of the transforming units 44 (n).

The deforming unit 44 (n) deforms the auxiliary image Iq (n) based on the displacement information in order to cancel the displacement between the reference image Ir and the auxiliary image Iq (n). . The transformed auxiliary image Iq (n) has the brightness correction unit 53 of any one of the auxiliary preprocessing units 45 (n).
Given to (n). The auxiliary pretreatment unit 45 (n)
Performs the same pre-processing as that performed by the reference pre-processing unit 41 on the deformed auxiliary image Iq (n). That is, first, the brightness correction unit 53 (n) includes the reference preprocessing unit 41.
The brightness of each pixel of the deformed auxiliary image Iq (n) is corrected in pixel units by the same method as the brightness correction unit 51.
Color noise removing unit 54 in the auxiliary preprocessing unit 45 (n)
(N) removes color noise from the auxiliary image Iq (n) whose pixel brightness has been corrected by the brightness correction section 53 (n) by the same method as the color noise removal section 52 of the reference preprocessing section 41. To do. As a result, in the transformed auxiliary image Iq (n),
The pretreatment is performed. The pre-processed auxiliary image Iq (n) $ is supplied from the color noise removing unit 54 (n) to the combining unit 46.

The combining unit 46 combines the preprocessed reference image Ir $ and all the preprocessed auxiliary images Iq (1) $ to Iq (N) $ into one sheet. The composite image Is is generated. The post-processing unit 47 performs post-processing for improving the image quality of the composite image Is on the composite image Is.
The post-processing restores the blurring of the contour of the image of the subject in the composite image Is and the saturation enhancement processing for compensating the saturation lost due to the brightness correction processing and the color noise removal processing. And a sharpening process for effecting. That is, first, the saturation emphasis unit 55 performs the reference image I as the saturation emphasis process.
With reference to the luminance and saturation of each pixel of r, the synthesized image Is
The saturation of each pixel is emphasized as the brightness of each pixel increases. Next, the sharpening unit 56 sharpens the combined image Is in which the saturation of each pixel is emphasized as a sharpening process. As a result, the sharpening unit 56 outputs one output image Io which is the processing result of the image quality improvement processing.

FIG. 3 is a block diagram illustrating a specific functional configuration of the brightness correction unit 51 of the reference preprocessing unit 41. The brightness correction unit 51 includes the same number of color components of single pixel data as addition units 61 (1) to 61 (3) and brightness conversion units 62 (1) to 62 (3), and a single table storage. Part 6
Including 3 and. The brightness correction unit 51 performs so-called gamma correction. For this purpose, the characteristic of the function, that is, the pixel value of each color component Rx, Gx, Bx of the pixel data before the luminance conversion, and the Ry, Gy, of each color component of the pixel data after the luminance conversion,
A brightness correction lookup table that represents the correspondence between By and the pixel value X is stored in the table storage unit 63.

FIG. 4 is a graph showing the characteristic curve 64 of the brightness correction lookup table. The horizontal axis of the graph in FIG. 4 represents the pixel value x of each color component Rx, Gx, Bx before the luminance conversion, and the vertical axis the respective color component Ry, Gy, after the luminance conversion.
The pixel value X of By is shown. Since the brightness correction section 64 performs gamma correction, the characteristic curve 64 is a so-called general gamma curve. That is, the smaller the pixel value x before the luminance conversion, the larger the amplification factor of the pixel value x. Therefore, the smaller the pixel value x before the luminance conversion, the more the pixel value x before the luminance conversion.
And the pixel value X (x) after the luminance conversion of the pixel value x is large.

Referring again to FIG. All the pixel data of the reference image Ir are given to the brightness correction unit 51 one by one. The operation of the brightness correction section 51 at the time when any one of all the pixel data is given is as follows. The adders 61 (1) to 61 (3) add the color components Rx, Gx, Bx of the pixel data and a predetermined reference correction value α to obtain the color components Rx, Gx,
Sum of Bx and reference correction value α Rx + α, Gx + α, Bx +
α is given to each of the brightness conversion units 62 (1) to 62 (3). The reference correction value α is a predetermined constant that is sufficiently smaller than the color component, and is about 3 to 5 when the maximum value of the color component is 255, for example. Each of the brightness conversion units 62 (1) to 62 (3) calculates the sum Rx + α, Gx + α, Bx based on the brightness correction lookup table.
+ Α is subjected to luminance conversion to obtain the respective color components Ry, Gy, By after the luminance conversion. The three color components Ry, Gy, By after the brightness conversion are included in the pixel data after the brightness correction. The above is the operation of the brightness correction unit 51 at the time.

The brightness correction section 51 repeats the above-mentioned operation every time one pixel data in the image data representing the reference image Ir is given. As a result, the color components Ry, Gy, By in the pixel data after the luminance correction individually corresponding to all the pixels of the reference image Ir are sequentially output from the luminance correction unit 51. A set of all the pixel data after the brightness correction corresponds to image data representing the brightness-corrected reference image Ir *.

The brightness correction unit 51 performs the brightness conversion after adding the reference correction value α to the color components Rx, Gx, Bx before the brightness conversion for the following reason. In general, an image obtained by the image capturing device 23 is more strongly affected by noise such as random noise as the color components of pixels having lower luminance. Therefore, when the image is the low illuminance image, the S / N ratio of the color components of the pixels of the image is worse than the S / N ratio of the color components of the pixels of the images other than the low illuminance image. Further, in a general gamma correction process, the color components Rx, Gx, and Bx of each pixel of the image to be processed before the brightness conversion are directly referred to the brightness correction lookup table in the table storage unit 63 to perform the brightness conversion. Since the conversion is performed, the amplification factor increases as the color component of the pixel having the lower brightness. from these things,
When the color component of the pixel with low luminance is amplified based on the general gamma correction process, the random noise is also amplified. As a result, annoying noise is generated in the low-luminance portion in the image after the general gamma correction process. The low luminance part is a region including many pixels that are easily affected by the random noise, that is, a region including many pixels whose luminance is lower than a predetermined reference luminance.
As a result, by the general gamma correction process,
The image quality of the entire image is greatly impaired.

As measures against the above-mentioned deterioration of image quality,
The brightness correction unit 51 of the image processing device 22 adds a small value, which is a reference correction value α, to each of the color components Rx, Gx, and Bx before performing general gamma correction processing, and then performs the brightness correction. Perform processing. In general, the random noise in the low-illuminance image appears as very low level, highly saturated pixel values. That is, the pixels in the low-illuminance image are more strongly influenced by the random noise as the brightness of the pixels is lower, and the pixels affected by the random noise are more than the original saturation before being affected. , The saturation becomes higher. Before performing the brightness correction process, the reference correction value α
Is added to each color component Rx, Gx, Bx before the luminance conversion, each color component Rx, Gx, Bx is raised.

As shown in FIG. 4, the ratio between the pixel value x before addition and the pixel value X (x) after the luminance conversion of the pixel value x, the pixel value after addition x + α and the luminance conversion of the pixel value. Subsequent pixel value X
Comparing the ratio with (x + α), the latter pixel value x + α has a smaller amplification factor than the former pixel value x, so the latter ratio is smaller than the former ratio. As a result, the lower the pixel value of the color components Rx, Gx, Bx before the luminance conversion, the stronger the effect of suppressing saturation due to the addition of the reference correction value α. Therefore, by adding the reference correction value α to the pixel values of the color components Rx, Gx, and Bx before performing the brightness correction process, it is possible to suppress random noise that occurs in the low brightness area in the image. is there.

The brightness correction section 51 uses the brightness correction look-up table of the characteristic shown by the characteristic curve 64 in FIG.
A luminance correction look-up table having the characteristic indicated by the characteristic curve 65 in FIG. 5 may be provided. The characteristic curve 65 is the characteristic curve 6
4 corresponds to the parallel displacement of the reference correction value α in the direction in which the pixel value x decreases, that is, in the left direction on the paper surface.
Therefore, the shape of the characteristic curve 65 in FIG. 5 is equal to a so-called general gamma curve, and the pixel value x before the luminance conversion is set.
In the characteristic curve 64 of FIG. 4, the pixel value X (x + after the luminance conversion corresponding to the sum x + α of the pixel value x and the reference correction value α.
α) corresponds.

When the brightness correction look-up table having the characteristics indicated by the characteristic curve 65 in FIG. 5 is stored in the table storage unit 63, the brightness correction unit 51 refers to the brightness correction look-up table and refers to each color before the brightness conversion. The components Rx, Gx,
The brightness of Bx is directly converted. As a result, the pixel value X (x + α) after luminance conversion corresponding to the sum x + α of the pixel value x and the reference correction value α can be obtained. Therefore, when the brightness correction look-up table of FIG. 5 is used for the brightness correction processing, it is possible to save the labor of adding the reference correction value α to each color component Rx, Gx, Bx before the brightness conversion. Further, in this case, all of the addition units 61 (1) to 61 (1) to 61
Since (3) can be excluded, the configuration of the brightness correction unit 51 is simplified. Further, in the above case, the brightness correction unit 51
Is compared to the brightness correction unit that performs general gamma correction,
Since only the brightness correction look-up table needs to be rewritten, realization becomes easy.

As described above, the pixel value Rx of each color component,
It is difficult to completely suppress the random noise only by adding the reference correction value α to Gx and Bx and converting the luminance. Therefore, the color noise removing units 52, 54 (1) to 5
4 (N) and the synthesizing unit 46 are used to further suppress random noise.

Further, in the brightness correction processing, the reference correction value α
Is not limited to the gamma correction after the addition, and a luminance correction process other than the gamma correction may be performed after the reference correction value α is added. The other brightness correction processing is, for example, so-called gradation correction processing according to the gradation characteristics of the display device. Further, in the brightness correction processing, general brightness correction processing such as gamma correction and gradation correction processing may be performed without adding the reference correction value α.

The color noise removing process performed by the color noise removing section 52 will be specifically described below. Color noise removal unit 52
In order to suppress random noise in the luminance-corrected reference image Ir *, the saturation of each pixel in the reference image Ir * is set to be lower as the luminance of each pixel itself is lower, and The higher the saturation of itself, the more suppressed it is. This is for the following reason.

The C which is the main cause of the random noise
CD noise occurs in the low brightness area in the image. The CCD noise tends to bias the color components of the pixels in the low brightness area. That is, in the pixel on which the CCD noise is superimposed, the pixel value of any one or any two color components included in the pixel data of the pixel is sufficiently larger than 0, The pixel values of the remaining color components of the color components of the above tend to be close to zero. Further, since the CCD noise is superimposed only on a part of the color components of all the color components, the configuration of all the color components, that is, the relative ratio of all the color components may be biased. . Further, due to the superposition of CCD noise, the original information indicated by the pixel data composed of all the color components may be destroyed, and in this case also, the configuration of all the color components is biased. That is, the random noise is so-called color noise.

When the configurations of all the color components are biased, the saturation of the pixel indicated by the pixel data including all the color components becomes extremely large as compared with the pixel data in which the configurations are not biased. The fact that the pixels whose saturation has increased due to such reasons are locally present in the image causes the random noise to stand out. Therefore, it is possible to selectively suppress only the random noise by reducing the saturation of the pixel with high saturation among the pixels in the low luminance area where the random noise tends to occur.

As a concrete method of color noise removal processing,
A first method of converting the color system of pixel data from the RGB color system to the HSV color system and then suppressing only the saturation component S of the pixel data, and the color components Ry, Gy, By of the pixel data
There is a second method of directly suppressing First, the first method will be described.

FIG. 6 is a block diagram showing a specific electrical configuration of the color noise removing section 52 of the reference preprocessing section 41 using the first method. The color noise removing unit 52 includes first and second color system converters 66 and 68 and a saturation suppressing unit 67. All the pixel data in the pixel data indicating the reference image Ir * whose brightness has been corrected are provided to the color noise removing unit 52 one by one. Color noise removing unit 5 at the time when any one of all pixel data is given
The operation of No. 2 is as follows.

First, the first color system converter 66 converts the color system of any one of the pixel data from the RGB color system to
Convert to so-called HSV color system. The pixel data is H
When represented by the SV color system, the pixel data includes a hue component H indicating hue, a saturation component S indicating saturation, and a luminance component V indicating luminance. In the following description, it is assumed that each of the above components H, S, and V represents a numerical value that represents the magnitude of hue, saturation, and luminance, that is, a pixel value. As a result, the three color components Ry, Gy, B in the pixel data are
y is converted into a hue component H, a saturation component S, and a luminance component V. Further, at the time of this conversion, the saturation component S and the luminance component V are each normalized within the range of 0 or more and 1 or less. Pixel values indicated by the normalized luminance and saturation components V and S are real numbers within the range of 0 or more and 1 or less. The hue component H and the luminance component V are converted into the second color system converter 6
Given to 8. The saturation component S and the luminance component V are provided to the saturation suppression unit 67.

The saturation suppressing section 67 roughly determines, based on the saturation component S and the luminance component V from the first color system converter 66, that the luminance component V is lower and the saturation component S is lower. Is higher, the saturation component S is suppressed. As a result, the suppressed saturation component S * is obtained.

Specifically, the saturation suppressing section 67 first uses the brightness and saturation components V and S, and based on the brightness and saturation evaluation functions shown in Expressions 1 and 2, the brightness evaluation coefficient k.
i and the saturation evaluation coefficient ks are obtained. In the following equation, “I” is the normalized luminance component V.
“S” is the normalized saturation component S. “Pi” and “Ps” are change coefficients for determining the degree of change in the luminance and saturation evaluation functions. “It” and “St” are thresholds of the luminance and saturation evaluation functions. The change coefficients Pi and Ps are each 0.5, for example.
Is. The threshold It of the brightness evaluation function is, for example, 0.1.
And the threshold value St of the saturation evaluation function is 0.9, for example.
Is.

Ki = 1-1 / (1 + exp (-Pi * (I-It))) (1) ks = 1 / (1 + exp (-Ps * (S-St))) (2) 9 is a graph showing a characteristic curve 69 representing the relationship between the normalized luminance component V and the luminance evaluation coefficient ki. In FIG. 7, it is assumed that the threshold It of the luminance evaluation function is approximately 0.5. The luminance evaluation coefficient ki is a real number in the range of 0 or more and 1 or less, and the larger the luminance component V in the range of 0 or more and 1 or less, the smaller the luminance evaluation coefficient ki. Luminance evaluation coefficient ki
Change rate, that is, the slope of the characteristic curve 69 of the luminance evaluation coefficient ki is always negative. Further, the absolute value of the rate of change is the first luminance range W in which the luminance component V is 0 or more and less than the threshold It.
When the value is within V1, the larger the brightness component V is, the larger the brightness component V is, and the second brightness range W in which the brightness component V is equal to or more than the threshold value It is 1 or less.
When the value is within V2, the larger the luminance component V is, the smaller it is.

FIG. 8 is a graph showing a characteristic curve 70 showing the relationship between the normalized saturation component S and the saturation evaluation coefficient ks. In FIG. 8, it is assumed that the threshold value St of the saturation evaluation function is approximately 0.5. Saturation evaluation coefficient ks is 0 or more 1
It is the following real number, and within the range of 0 or more and 1 or less, the saturation component S
Is larger, the saturation evaluation coefficient ks is larger. The rate of change of the saturation evaluation coefficient ks, that is, the slope of the characteristic curve 70 of the saturation evaluation coefficient ks is always positive. Further, when the saturation component S is a value within the first saturation range WS1 in which the saturation component S is 0 or more and less than the threshold value St, the absolute value of the change rate is larger as the saturation component S is larger, and the saturation component S is the threshold value St. When the value is within the second saturation range WS2 of 1 or more, the smaller the saturation component S, the smaller the value.

Referring back to FIG. Then, the saturation suppression unit 6
7 obtains the product of the luminance evaluation coefficient ki and the saturation evaluation coefficient ks as a pixel evaluation coefficient k for evaluating the degree of low brightness and high saturation of the pixel, as shown in Expression 3. As a result, the pixel evaluation coefficient k increases as the luminance component V increases and the saturation component S decreases. Finally, the saturation suppressing unit 67 uses the pixel evaluation coefficient k to convert the saturation component S given from the color system converter 66 based on Expression 4. In the following equation, “λ” is a predetermined constant that is sufficiently smaller than the pixel evaluation coefficient k, and the pixel evaluation coefficient k is set to Is added. The constant λ is a real number of 0 or more and 1 or less as shown in Expression 5, and is extremely close to 0. The suppressed saturation S * obtained as a result of the conversion is supplied from the saturation suppression unit 67 to the second color system converter 68.

K = ki × ks (3) S * = S ÷ (k + λ) (4) 0 <λ << 1 (5) The pixel evaluation coefficient k is used for conversion of the saturation component S. Therefore, the degree of suppression of the saturation component S reflects the evaluation of the luminance component V and the saturation component S. That is, the lower the luminance component V of the pixel data before saturation suppression and the larger the saturation component S of the pixel data, the larger the pixel evaluation coefficient k, and thus the amplification factor of the saturation component S becomes smaller. Therefore, the degree of suppression of the saturation component S increases. The hue component H, the suppressed saturation component S *, and the luminance component V are included in the pixel data when the saturation suppressed pixel data is represented by the HSV color system. The second color system converter 69 converts the color system of the pixel data after saturation suppression from the HSV color system to the RGB color system. As a result, the hue component H, the suppressed saturation component S *, and the luminance component V are converted into three color components Rz, Gz, and Bz of the pixel data after the saturation suppression. The above is the operation of the color noise removing unit 52 at the time.

The color noise removing section 52 repeats the above operation each time one pixel data in the image data representing the reference image Ir * after the luminance correction is given. As a result, the color noise removing unit 52 sequentially outputs the color components Rz, Gz, and Bz of the pixel data after saturation suppression individually corresponding to all the pixels of the reference image Ir *. A set of all the pixel data after the saturation suppression is the reference image I subjected to the pre-processing.
Corresponds to image data representing r $. This is the end of the description of the color noise removal process using the first method.

Next, the second method of color noise removal processing will be described below. Reference image Ir with corrected brightness
All the pixel data in the image data indicating * are given to the color noise removing unit 52 one by one. In order to suppress random noise, the color noise removing unit 52 that uses the second method roughly includes three color components of pixel data Ry, Gy, and By each time one pixel data is given. Of the three color components Ry, Gy, By are closer to the average value ac of the color components as the luminance component of the pixel data is lower and the saturation component of the pixel data is higher. Replace. The operation of the color noise removing unit 52 at the time when any one pixel data of all the pixel data is given is as follows.

First, the color noise removing section 52 converts the color system of any one of the pixel data from the RGB color system to a so-called HSV color system, and further converts the pixel data of the HSV color system. The saturation and luminance components S and V are normalized within the range of 0 or more and 1 or less. Then, based on Equations 1 and 2,
The luminance and saturation evaluation coefficients ki and ks are obtained, and further, Equation 3
The pixel evaluation coefficient k is calculated based on Next, based on Equation 6, an average value ac1 of the three color components Ry, Gy, By of the one pixel data before conversion of the color system is obtained. Finally, the pixel evaluation coefficient k and the average value ac of the color components
And three color components Rz, Gz, and Bz of the pixel data after saturation suppression are calculated based on Expressions 7 to 9, respectively. As a result, each color component R of the pixel data after saturation suppression
z, Gz, and Bz become the average value ac of the color components as the pixel evaluation coefficient k is larger, that is, the luminance component V of the pixel data before the saturation suppression is lower and the saturation component S of the pixel data is higher. Approach each.

Ki = 1-1 ÷ (1 + exp (-Pi x (I-It))) (1) ks = 1 ÷ (1 + exp (-Ps x (S-St))) (2) k = ki × ks ... (3) ac = (Ry + Gy + By) / 3 (6) Rz = k × (ac1-Ry) + Ry (7) Gz = k × (ac1-Gy) + Gy (8) Bz = k × ( ac1-By) + By (9) Like the second method, the three color components Rz, Sz after saturation suppression,
When Gz and Bz are obtained by directly converting the three color components Ry, Gy and By before the saturation correction using the average value ac of the color components, the calculation amount can be reduced as compared with the first method. it can. This is for the following reason. In general, an image converted into digital data, that is, image data, often uses the RGB color system when being stored in the storage units 28 and 30 of the image pickup device 23 and the image processing device 22. Therefore, the color system of the image data output from the color noise processing unit 52 also needs to be the RGB color system.
Therefore, in the first method, it is necessary to convert the color system after the saturation suppression processing, but in the second method, it is not necessary to convert the color system. Therefore, the second method is more advantageous in calculation amount than the first method. This is the end of the description of the color noise removal process using the second method.

In both the first and second methods, the pixel evaluation coefficient k approaches 1 as the pixel data in which the random noise is more likely to be distributed, that is, the pixel data of the pixel having lower luminance and higher saturation. Is defined as Further, as the pixel evaluation coefficient k approaches 1, the color components Rz, Gz, and Bz after saturation suppression have an average value ac1 of the color components.
Approach. With this, the random noise component in each of the color components Ry, Gy, By is whitened and the saturation is suppressed by using both the first and second methods, so that the random noise is removed. Later reference image Ir $
, The random noise is less noticeable.

In the description of the color noise removing section 52 using the first and second methods, it is assumed that the color system of the pixel data of the image whose brightness is corrected is the RGB color system. May be another color system. For example, if the color system is YU
In the case of the S color system, the pixel data includes a luminance component Y and two types of color difference components Uc and Sc. In this case, the color noise removing unit 52 first calculates the luminance component V as shown in Expression 10.
The luminance component Y is used as it is, and the saturation component S
May be determined as shown in Equation 11. In this case, if the color noise removing unit 52 uses the first method,
The color system converter 68 calculates the suppressed saturation component S * using the equation 11
It is sufficient to convert into two types of chroma components using the inverse function of, and further add the luminance component Y to convert into the YUS color system.

Luminance component V = Y (10) Saturation component S = √ (Uc ² + Sc ² ) ... (11) Further, as described above, the luminance-corrected reference image Ir
Instead of continuously increasing the degree of saturation suppression of each pixel of * as the pixel evaluation coefficient k increases, the luminance component V of the pixel is equal to or more than a predetermined luminance threshold value, and the pixel Is checked for each pixel of the reference image Ir *, and the luminance component V is equal to or more than the luminance threshold and the saturation component S is the color saturation. Only the saturation of pixels that are less than the threshold of the saturation may be suppressed. This is the end of the description of the color noise removing unit 52.

The feature point search processing performed by the feature point extraction unit 42 will be described in detail below with reference to FIG. When the reference image Ir and the auxiliary images Iq (1) to Iq (N) are the low-illuminance images, each of the images Ir, Iq (1) to Iq.
In (N), the pixels having the characteristic point information sufficient for the corresponding point search processing are rarely distributed over the entire surface of each image, and the pixels having the characteristic point information are unevenly distributed in a part of each image. It is believed that The feature point information is information for defining a texture necessary for matching processing for searching corresponding points described later. Therefore, the feature point extraction unit 42 determines the pixel in the reference image Ir that is most suitable for the search process of the corresponding points on each of the auxiliary images Iq (1) to Iq (N) as the feature point p.
It is necessary to extract as a1 to paM.

Therefore, the characteristic point extracting section 42 extracts the characteristic points pa1 to paM in the following procedure. In the following description, the number M of feature points is assumed to be 3. First, the feature point extraction unit 42 sets the same number of feature point search areas 81 to 83 as the number M of feature points in the reference image Ir in the arrangement shown in FIG. FIG. 9 shows the feature point search areas 81 to 8 in the reference image Ir.
It is a schematic diagram which shows the arrangement state of FIG. Each feature point search area 8
1 to 83 are smaller than the reference image Ir. Further, the feature point search areas 81 to 83 are separated from each other by a predetermined appropriate distance, and the virtual figure having the reference point of each feature point search area 81 to 83 as an apex becomes a substantially equilateral triangle. Will be placed. The appropriate distance is, for example, the reference image I.
It is 20% to 30% of the width of r. Next, the feature point extraction unit 42 selects from each of the feature point search areas 81 to 83,
Search feature points one by one. As a result, the three feature points p
a1 to pa3 are obtained.

A method for extracting a single feature point from any one of the feature point search areas will be described below. In the present embodiment, the feature point extraction unit 42 first sets Equation 1 for all pixels in any one of the feature point extraction regions.
The variance value kp of the feature point information shown in 2 is obtained. Specifically, the variance value kp of any one pixel is the brightness value y of each of the one pixel and the neighboring pixels of the pixel.
i and the sum Σ | yi-ay | of absolute values of the difference | yi-ay | It is a value divided by the number N of the neighboring pixels. Next, the feature point extraction unit 42 compares the variance values kp of all the pixels with each other, and sets the pixel having the largest variance value kp as the feature point.

Kp = Σ _i | yi−y | ÷ N (12) The above-mentioned brightness value y is a numerical value indicating the brightness of a pixel, and is three values of the pixel data shown in the RGB color system of the pixel. It is empirically derived from the color components R, G, and B. The brightness value and the brightness component V of the pixel data represented by the HSV color system described above.
Are different in the calculation method based on the three color components. Formula 13
Is a definitional expression of the brightness value y. In the following formula, "R",
"G" and "B" are the red, blue and green color components of the pixel for which the brightness value is to be defined.

Y = 0.3R + 0.6G + 0.1B (13) The feature point search areas 81 to 83 are arranged as shown in FIG. 9 to search the feature points pa1 to pa3 for the following reason. When all the feature points pa1 to pa3 are located very close to each other, or all the feature points pa1 to pa3
When 3 are arranged in almost one straight line, a deforming portion 44 described later is provided.
The correction error of the deformation process for correcting the deviation performed in (n) may increase. Therefore, all feature points pa1
It is preferable that ~ pa3 are scattered in the reference image so that they are separated from each other by a distance close to the appropriate distance and are not aligned in a straight line. Here, when the feature point search areas 81 to 83 are arranged as shown in FIG. 9 and one feature point is extracted from each feature point search area, the feature points pa1 to pa
A virtual figure 3 having a distance close to the appropriate distance from each other and having the feature points pa1 to pa3 as vertices has a shape close to an equilateral triangle. Further, as a result, in the corresponding point searching unit 43 (n) described later, even when the auxiliary image Iq (n) is a low illuminance image with little characteristic information, the corresponding point pb
1 to pb3 can be easily found. For these reasons, the shift correction process can be stably performed.
That is, the accuracy of the deformation process can be improved. Further, this also makes it possible to shorten the time required to search for the feature points and the corresponding points.

The above-mentioned characteristic points are not limited to three points, and four or more points may be extracted. In this case, three points may be further selected from the extracted four points, and only the selected three feature points may be given to all corresponding point searching units 43 (1) to 43 (N). , All the extracted feature points may be used together with the least squares method to perform the subsequent processing. Furthermore, the feature points are
A method other than the method using the variance value kp of the feature point information described above may be used. In addition, the feature point search areas 81 to 83 are
It may always be set to a predetermined position in the reference image Ir, and the feature point search unit 42 finds a portion in the reference image Ir that has many pixels including the feature point information and sets the portion as a feature point search area. Good.

In the following, any one of the corresponding point search units 43
The corresponding point search process performed by (n) will be described. The corresponding point searching unit 43 (n) determines any one of the auxiliary images Iq.
(N) From above, the pixels corresponding to all the feature points pa1 to paM are designated as the corresponding points pb1 of the auxiliary image Iq (n).
Search as pbM. Specifically, each feature point pa
1-paM near so-called texture and corresponding point pb1-
The corresponding points pb1 to pbM are extracted so that the correlation with the texture near pbM is maximized.

Texture near the feature point and the auxiliary image I
Correlation Ct with texture near any point in q (n)
Is, for example, as shown in Expression 14, any one of the feature points pam and the luminance value yi of each of the neighboring pixels of the feature point pam, and the luminance value yj of each of the arbitrary points and the neighboring pixels of the arbitrary point. It is assumed that it is the integrated value Σ | yi-yj | of the absolute value | yi-yj | The brightness values yi and yj are respectively determined by using Expression 13.

Ct = Σ _i , _j | yi−yj | (14) Any one of the corresponding point search units 43 (n) is processed image I
Of all the pixels in q (n), one of the pixels having the smallest correlation value Ct and the correlation value Ct is set as the corresponding point pbm of the search point pam. For this reason, any one of the corresponding point searching units 43 (n) specifically defines the pixel with the maximum correlation value Ct as the pixel with the maximum matching, and uses a so-called template. Perform the pyramid matching method. This processing is performed for each feature point pa.
Repeating every 1 to paM, all the feature points pa1 to paM
Corresponding points pb1 to pbM are obtained. Correlation value Ct between the texture near the feature point and the texture near an arbitrary point
Is not limited to the integrated value Σ | yi-yj |, and may be another value.

In the following, any one of the deformation parts 44 (n)
The transformation process performed by will be described. The deforming portion 44 (n)
First, the validity of the corresponding points pb1 to pbM of any one of the auxiliary images Iq (n) is evaluated. This is for the following reason. If the auxiliary image Iq (n) is the low-illuminance image, the low-illuminance image lacks texture information, and thus the search result of the corresponding point search unit 43 (n) is not always reliable. That is, the auxiliary image Iq (n)
Is the low illuminance image, the part of the reference image Ir having the feature points pa1 to paM and the part having the corresponding points pb1 to pbM in any one of the auxiliary images Iq (n) have the object of the subject. The same part is not always shown. In this case, when the deformation process is performed using the corresponding points pa1 to paM, so-called blurring or afterimage occurs in the output image Io finally output from the image processing device 22, so that the output image I
o becomes very hard to see. Therefore, the deformation portion 44
In (n), first, the reliability of the result of the corresponding point search process of the corresponding point search unit 5, that is, the validity of the corresponding points pb1 to pbM is evaluated, and the highly valid process result, that is, the search process succeeds. Only the corresponding points pb1 to pb3 that are present are used for the transformation process.

A method for evaluating the validity of the corresponding points is shown in FIG.
Will be explained. In the following description, the number M of feature points is 3
Suppose that FIG. 10A is a diagram showing a mutual positional relationship between the feature points pa1 to pa3 in the reference image Ir. FIG. 10B is a diagram showing a mutual positional relationship between the feature points pb1 to pb3 in the auxiliary image Iq (n).

In order to evaluate the validity, any one of the deformation units 44 (n) obtains the validity evaluation value θ based on the following equations 15 to 21. The validity evaluation value θ is calculated from the triangle Tr having the feature points pa1 to pa3 as vertices and the corresponding point p.
The variation of the three sides of the triangle to the triangle Tq (n) having vertices b1 to pb3 is shown. The fluctuations of the three sides of the triangles Tr and Tq (n) are caused in the reference image Ir and the auxiliary image Iq.
It occurs due to the difference in the position and shape of the subject shown in (n).

Lr1 = | Vpa1-Vpa2 | ... (15) Lr2 = | Vpa2-Vpa3 | ... (16) Lr3 = | Vpa3-Vpa1 | ... (17) Lq1 = | Vpb1-Vpb2 | ... (18) Lq2 = | Vpb2-Vpb3 | ... (19) Lq3 = | Vpb3-Vpb1 | ... (20) .theta. = (| Lq1-Lr1 | ÷ Lr1 + | Lq2-Lr2 | ÷ Lr2 + | Lq3-Lr3 | ÷ Lr3) ÷ 3 ... (21) In the above equation, "Vpa1", "Vpa2", "Vpa"
3 ”is a coordinate vector of each of the feature points pa1 to pa3 on the reference image Ir. "Vpb1", "Vpb2", "V
“Pb3” is a coordinate vector of each corresponding point pb1 to pb3 on any one of the auxiliary images Iq (n). Each point pa
The coordinate vectors 1 to pa3, pb1 to pb3 are the origin (0, 0) of the coordinate system set for the images Ir and Iq (n).
To a representative point of the pixel corresponding to the points pa1 to pa3 and pb1 to pb3, and the representative point of the pixel is, for example, the center of the pixel. Image Ir, Iq (n)
The origin of the coordinate system may be anywhere in the images Ir, Iq (n). "Lr1", "Lr2", "Lr3", "L
"q1", "Lq2", and "Lq3" are triangles Tr and Tq.
(N) is the length of each of the three sides.

Then, the transformation section 44 (n) uses the validity evaluation value θ as a threshold θ for evaluating the validity of the corresponding points.
Compare with c. The threshold value θc is 0.9, for example. When the validity evaluation value θ is equal to or less than the threshold value θc, it is determined that the validity of the corresponding points is low and the deformation process for the shift correction is difficult, and the auxiliary image Iq (n) is evaluated for the validity. Not used for subsequent processing. Therefore, the deformation portion 44
(N) ends the process at this point. Validity evaluation value θ
Is greater than the threshold value θc, it is determined that the validity of the corresponding points is high enough to perform the deformation process for the deviation correction, and the auxiliary image Iq (n) is processed after the validity evaluation process. Used for. Therefore, the deforming portion 44 (n)
Performs a transformation process on the auxiliary image Iq (n). The above is the validity evaluation processing.

A concrete method of the deformation process of any one of the auxiliary images Iq (n) will be described below. In the following description, it is assumed that the number M of feature points is 3.

First, prior to the transforming process, the corresponding point searching unit 43 (n) obtains the shift information and gives it to the transforming unit 44 (n). Specifically, the corresponding point searching unit 4
3 (n) first sets two vectors having two ends of any two of the three feature points pa1 to pa3.
In the following description, the two vectors will be referred to as the feature point pa1.
"Pa2-pa1" from the point toward the feature point pa2
And a vector “pa3-pa1” from the feature point pa1 toward the feature point pa3. Next, the corresponding point searching unit 43 (n) linearly decomposes the coordinate vector p of each pixel of the reference image Ir using the two vectors as shown in Expression 22. As a result, the two vectors pa when the coordinate vector p is linearly decomposed
The ratios α and β of 2-pa1 and pa3-pa1 are obtained for all pixels of the reference image Ir. The ratios α and β of all pixels of the reference image Ir are used as the shift information from the corresponding point searching unit 43 (n) to the transforming unit 44.
Given to (n).

P = α × (pa2-pa1) + β × (pa3-pa1) (22) Next, the transforming unit 44 (n) performs the transforming process based on the shift information based on the auxiliary information Iq (n). Of the corrected coordinate vector p * of all the pixels of the The ratios of the two vectors at the time of decomposition are calculated so as to be equal to the ratios α and β of the pixels of the reference image Ir. Specifically, the auxiliary image Iq
Corrected coordinate vector p * of any one pixel in (n)
Is calculated based on Expression 23 using the ratios α and β of the pixels in the reference image Ir corresponding to the pixel.

P * = α × (pb2-pb1) + β × (pb3-pb1) (23) Finally, the transformation unit 44 (n) causes the auxiliary image Iq (n) to be generated.
The coordinate vector p of all the pixels is changed to the corrected coordinate vector p *, and all the pixels are rearranged. The displacement correction image generated as a result is the auxiliary image Iq (n).
However, it has been modified so as to cancel the deviation.
When the pixel data of pixels having coordinates that do not exist in all of the coordinate vectors p * after the correction is required due to the deformation of the auxiliary image Iq (n) when the shift correction image is created, The pixel data of the pixel in the reference image Ir having the same coordinates as the pixel data of the nonexistent coordinates is used. Above,
The description of the transformation process ends.

The above-described misalignment corrected image, that is, the deformed auxiliary image Iq (n) is given to one of the auxiliary preprocessing sections 45 (n) and subjected to the preprocessing.
The specific configuration of the brightness correction unit 53 (n) and the color noise removal unit 54 (n) in any one of the auxiliary preprocessing units 45 (n) is the same as the brightness correction unit in the reference preprocessing unit 41. The specific configurations of the color noise removing unit 51 and the color noise removing unit 52 are the same. As a result, the auxiliary image Iq that has been subjected to the deformation process
Pretreatment is further applied to (n).

The combining process performed by the combining unit 46 will be described below. The synthesizing unit 46 includes a preprocessed reference image Ir.
$ And all the auxiliary images Iq (1) $ to Iq (N) $ that have been subjected to the transformation process and the preprocess are given. Deformation part 4
4 (1) to 44 (N), if there is an auxiliary image that is not subjected to the deformation process due to the evaluation result of the validity evaluation process,
The auxiliary image is not given to the combining unit 46. In the following description, all auxiliary images Iq (1) $ to Iq (N) $ are
It is assumed that this is given to the synthesizing unit 46. Further, in the following description, the reference image Ir $ that has been subjected to the preprocessing is
The auxiliary images Iq (1) $ to Iq (N) $ that have been subjected to the deformation process and the preprocess are referred to as “correction reference image Ir $”,
“Corrected auxiliary image Iq (1) $ to Iq (N)”
"$".

In order to suppress the random noise, the synthesizing unit 46 averages the above-mentioned corrected reference image Ir $ and all the corrected auxiliary images Iq (1) $ to Iq (N) $ to make one sheet. A composite image Is is obtained. This is because the random noise tends to be distributed probabilistically around the so-called true value, and the random noise can be suppressed by averaging a plurality of images. In the process for obtaining the image data representing the composite image Is, roughly, the pixel data of all the pixels in the composite image Is are sequentially arranged from the pixel corresponding to the coordinate (0,0) in the composite image Is. , All images Ir $, Iq (1) above
The pixel data of each pixel of $ to Iq (N) $ is obtained by weighted averaging using the correlation value of each pixel, and sequentially output.

FIG. 11 is a functional block diagram for explaining the method of setting the pixel data of the pixel ps to be processed corresponding to any one of the coordinates (x, y) in the composite image Is. The setting method will be described with reference to FIG. In FIG. 11, the image data of the corrected reference image Ir $, the image data of all the corrected auxiliary images Iq (1) $ to Iq (N), and the image data of the combined image Is are virtually drawn in a quadrangle. . Actually, in the image data of the corrected reference image Ir $ and all the corrected auxiliary images Iq (1) $ to Iq (N), pixel data in the image data is one by one, and each of the pre-processing units 41 and 45. (1) From $ to 45 (N) to the synthesis unit 46
1 and the pixel data in the image data of the composite image Is are given to the post-processing unit one by one, and there is not necessarily an image buffer for storing each image data.

First, a pixel p corresponding to the same coordinate as any one of the coordinates (x, y) in the corrected reference image Ir $.
The pixel data of r are all correlation value calculation units 91 (1) -9
1 (N) and the average calculation unit 93. A pixel p corresponding to the same coordinate as any one of the coordinates (x, y) in each of the correction auxiliary images Iq (1) $ to Iq (N) $.
Pixel data of q (1) to pq (N) corresponds to the correlation value calculation units 91 (1) to 91 (N) and the correction term calculation units 92 (1) to.
92 (N). Correction reference image I
The coordinates (x,
The pixel pr corresponding to the same coordinates as y) is hereinafter referred to as “reference pixel pr”, and each corrected auxiliary image Iq (1) $ to Iq.
The pixels pq (1) to pq (N) corresponding to the same coordinates as the coordinates (x, y) in (N) $ will be referred to as “auxiliary pixel pq” hereinafter.
(1) to pq (N) ".

Next, the correlation value calculation units 91 (1) -91 (91)
(N) obtains the correlation values C1 to CN between the auxiliary pixels pq (1) to pq (N) and the reference pixel pr, respectively. The correlation value is an integer of 0 or more and 1 or less. Correlation value C1-CN
The detailed calculation method of will be described later. Then, the correction term calculation unit 9
2 (1) to 92 (N) are auxiliary pixels pq (1) to pq.
From each of the three color components ri, gi, bi of the pixel data of (N), the three color components r0, g of the reference pixel data
The correlation values C1 to CN are set to half of the difference obtained by subtracting 0 and b0.
And the product p
It is obtained as a correction term for the three color components r *, g *, and b * of s. Finally, the average calculation unit 93 determines that each correction term calculation unit 92
Three color components r *, g obtained in (1) to 92 (N)
An average of the sum of the correction terms of * and b * and the three color components r0, g0 and b0 of the pixel data of the reference pixel pr is obtained. The average of the sums is the color components r *, g *, b * of the pixel data of the pixel ps to be processed in the composite image.

Through the above processing, the calculations shown in equations 24 to 26 are performed for every pixel of the composite image Is. In the following equation, “N” is the sum of the number of all auxiliary images and the number of reference images given to the combining unit 45.
“Σ” means addition for the auxiliary image i. That is, each color component of the reference pixel pr and all the auxiliary pixels pq weighted by the correlation values C1 to CN, respectively.
The average value of each color component of (1) to pq (N) becomes each color component of the pixel ps to be processed in the combined image Is.

R * = Σ _i ((ri−r0) × Ci × 0.5 + r0) ÷ N (24) g * = Σ _i ((gi−g0) × Ci × 0.5 + g0) ÷ N (25) ) B * = Σ _i ((bi−b0) × Ci × 0.5 + b0) ÷ N (26) By the calculation using such a weighted average, the color of the pixel data of the pixel ps to be processed in the composite image Is Component r
When calculating *, g *, and b *, if the correlation values C1 to CN are large, that is, if the correlation values C1 to CN are close to 1, respectively, the pixel data of the pixel ps of the composite image Is are all The average of the pixel data of the auxiliary pixels pq (1) to pq (N) and the pixel data of the reference pixel pr approaches.
In the above case, if the correlation values C1 to CN are small, that is, if the correlation values C1 to CN are close to 0, the pixel data of the pixel ps to be processed in the combined image Is becomes the pixel data of the reference pixel pr. Get closer.

For example, taking the red color component r * of the pixel ps of the composite image Is as an example, when the correlation values C1 to CN are 1, the red color component r * is approximately as shown in Expression 27. , The red color component r of each auxiliary pixel pq (1) to pq (N)
The average (ri + r) of i and the red color component r0 of the reference pixel pr
It is an average of 0) × 0.5. Also, the correlation values C1 to CN
Is 0, the red color component r * is approximately the red color component r0 of the reference pixel pr, as shown in Expression 28.

R * to Σ _i ((ri + r0) × 0.5) / N (27) r * to Σ _i r0 / N (28) As described above, the combining unit 46 processes the composite image Is. For each pixel of, the above average is obtained as pixel data of each pixel ps by using only the auxiliary pixel pq (n) having a large correlation value with the reference pixel pr. A set of pixel data of all pixels of the combined image Is corresponds to image data representing the combined image Is. As a result, it is possible to make conspicuous a synthesis error caused by the deviation that cannot be completely corrected by the deformation units 44 (1) to 44 (N).

In the calculation of the pixel data of the pixel ps to be processed in the combined image Is in the above-described combining processing,
Instead of using the weighted average using the correlation values C1 to CN, a simple average may be used. For example, if the subject is not moving when the auxiliary image and the reference images Iq (1) to Iq (N) and Ir are photographed, the correlation value is calculated from the process of obtaining each pixel data of the synthetic image. The calculation of Ci and the weighting using the correlation value Ci are omitted, and the combining unit 46 converts the pixel data of the combined image into the pixel data of the pixels of all the images Ir and Iq (1) to Iq (N). It may be simply averaged. This is because in the above case, the reference image and the auxiliary images Ir and Iq.
(1) to Iq (N), since the positions and shapes of the subjects are substantially equal to each other, the deviation between the reference image Ir and each of the auxiliary images Iq (1) to Iq (N) is small. This is because the resulting composite error is unlikely to occur.

In the above description, the synthesizing unit 46 uses the auxiliary images Iq (1) to Iq which have been subjected to the deformation processing and the preprocessing as the remaining images other than the corrected reference image Ir $ in the images to be synthesized. Iq (N), that is, the correction auxiliary image Iq
(1) Use $ to Iq (N) $. If the above-described combining process is performed to suppress the random noise, the combining unit 4
6 may use auxiliary images Iq (1) to Iq (N) that have undergone only preprocessing as the residual image, and the original auxiliary image Iq that has not undergone neither deformation processing nor preprocessing. (1) to Iq (N) may be used.

As the residual image, it is preferable to use at least preprocessed auxiliary images Iq (1) to Iq (N). This is for the following reason. If the residual image has been preprocessed in advance,
Random noise is reduced from each of all the images to be combined by using saturation suppression processing. Therefore, when the pre-processed auxiliary images Iq (1) to Iq (N) are used as the residual images, a synthetic image Is sufficiently removed of random noise is obtained by the synthesizing process using the residual images. This is because the minimum number of images to be combined is smaller than the number of images to be combined in the image processing apparatus using the conventional technique that suppresses random noise using the combining process.

In the composite image Is obtained by the above-described various combining processes, the position and the shape of the subject in the image are almost the same as the reference image Ir, and the image quality is improved satisfactorily. In addition, the auxiliary images Iq (1) to Iq (N)
When synthesizing the reference image Ir and the reference image Ir, if a weighted average using the correlation value Cn is used in the synthesizing process, a so-called double-image-free synthesized image can be generated. preferable. Further, instead of using the weighted average using the correlation values C1 to CN, only the pixel data of the auxiliary pixel pq and the reference pixel pr whose correlation value is equal to or more than the threshold value are compared by comparing the correlation value with a predetermined threshold value. The simple average may be used as the pixel data of the pixel ps to be processed. This also makes it possible to obtain the composite image without the double reflection.

Below, all the correlation value calculation units 91 (1) ...
The calculation processing of the correlation value performed by 91 (N) will be described by taking one of the correlation value calculation units 91 (n) as an example. The calculation processing of the correlation values of all the correlation value calculation units 91 (1) to 91 (N) is
Only the correction auxiliary images to be processed are different from each other, and other points are equal to each other. In the calculation processing of the correlation value,
~ There is a third calculation method.

First, the first method of calculating the correlation value will be described. In the first calculation method, the correlation value Cn is the reference pixel pr.
Luminance value y1 and the luminance value y2 of the auxiliary pixel pq (n) of any one corrected auxiliary image Iq (n) $ to be processed.
And are calculated based on the equation 29. In the following equation, “ya” is an average value of the above two brightness values y1 and y2. The brightness values y1 and y2 are defined by using the above-described equation 13.

Cn = 1 / (| y1-ya | + | y2-ya | +1) (29) In the first calculation method, when the image data of one pixel ps to be processed of the composite image Is is determined. , The correlation value Cn using only the reference image pr in the corrected reference image Ir $ having the same coordinates as the pixel ps and the pixel pq (n) in the correction auxiliary image Iq (n) $ having the same coordinates as the pixel ps. Is calculated. As a result, the combining unit 46 can calculate the correlation value for each pixel of the combined image Is.

Therefore, each image Ir $, Iq (n) $
Since it is not necessary to use the pixel data of the pixels in the vicinity of the pixels pr and pq (n) in the calculation of the correlation value, the processing result of each processing is held during the calculation of the deviation correction processing, the preprocessing, and the combining processing. It is not necessary to have a buffer to That is, a so-called intermediate image buffer for holding the original image used for the combining process is not required. Therefore, the image processing device 22 of the present embodiment can save memory, and the number of parts thereof can be reduced as compared with the image processing device of the prior art.

Furthermore, by omitting the intermediate image buffer, the manufacturing cost of the image processing apparatus can be greatly reduced. This is for the following reason.
In general, the capacity of the intermediate image buffer is, as shown in Equation 30, three times the product of the width of the auxiliary image and the height of the auxiliary image, and a number one more than the number of all auxiliary images. It becomes the product byte. For example, the size of the auxiliary image is 640 × 48.
When the number of auxiliary images is 0 and the number of auxiliary images is 10, the capacity of the intermediate image buffer is 10 Mbytes. In this way, the intermediate processing buffer tends to have a relatively large capacity, so that the component cost tends to be high. Therefore, the product cost of the image processing apparatus is likely to increase due to the provision of the intermediate image buffer. Since the image processing device 22 of the present embodiment does not include the above-mentioned intermediate processing buffer, the product cost can be greatly reduced.

Capacity = 3 × (width of auxiliary image × height of auxiliary image) × (number of auxiliary images + 1) (30) Next, a second calculation method of the correlation value will be described. 12
Is a functional diagram for explaining the method of setting the pixel data of the pixel ps to be processed corresponding to any one of the coordinates (x, y) in the composite image Is using the second calculation method of the correlation value. It is a block diagram. The functional block diagram of FIG. 12 is
Similar to the functional block diagram of FIG. 11, each image Ir $, I
The quadrangle representing q (1) $ to Iq (N) $, Is has the same meaning as the quadrangle in FIG. 11. When the second calculation method of the correlation value is used, as shown in FIG. 12, between the auxiliary preprocessing units 45 (1) to 45 (N) and the synthesizing unit 46,
Corrected image buffers 94 (1) to 94 (N) are respectively interposed. Correction image buffers 94 (1) to 94 (N)
Are auxiliary pretreatment units 45 (1) to 45 (N), respectively.
A set of pixel data respectively output from, that is, image data representing corrected auxiliary images Iq (1) $ to Iq (N) $ is stored.

In the second calculation method, first, the image vector Vi of the corrected reference image Ir $ and the image vector Vj of any one of the corrected auxiliary images Iq (n) $ to be processed are obtained. Each image Ir $, Iq (n) $
Image vectors Vi and Vj of
The luminance values of the pixels pr and pq (n) having the same coordinates as the pixel ps to be processed in r $ and Iq (n) $, and the pixel p.
The brightness value of each of a plurality of neighboring pixels of r and pq (n) is used as a component.

For example, it is assumed that the coordinates of the pixels pr and pq (n) are (x, y), and that the plurality of neighboring pixels are so-called 3 × 3 neighboring pixels. In this case, the coordinates of the pixels pr, pq (n) and the plurality of neighboring pixels are a combination as shown in the matrix 31 below. The array of elements of the matrix 31 below is the image Ir $, Iq
It is equal to the array of the processing target pixels pr and pq (n) in (n) $ and the plurality of neighboring pixels.

(X-1, y-1), (x, y-1), (x + 1, y-1) (x-1, y), (x, y), (x + 1, y) (x- 1, y + 1), (x, y + 1), (x + 1, y + 1) (31) Therefore, the image vector V of any one pixel p (x, y) of the image I is the brightness value of the nine pixels described above. Is a vector whose elements are and is represented by Expression 32. In the following formula, “I (i, j)” indicates the luminance value of the pixel corresponding to any one coordinate (i, j) in the image I. The brightness value I (i, j) is defined using Equation 13.
"I" is any one of x-1, x, x + 1, "j" is any one of y-1, y, y + 1, and the image I is the corrected reference image Ir $ or Either one
One corrected auxiliary image Iq (n) $.

V = [I (x-1, y-1), I (x, y-1), I (x + 1, y-1), I (x-1, y), I (x, y) , I (x + 1, y), I (x-1, y + 1), I (x, y + 1), I (x + 1, y + 1)] (32) Next, one of the correlation value calculation units 92 (n) A vector correlation value Cv between the image vector Vi of the corrected reference image Ir $ and the image vector Vi of the corrected auxiliary image Iq (n) $ is obtained as the correlation value Cn. There are first to third calculation methods for the calculation processing of the image vector correlation value, and details of each calculation method will be described later. As described above, when the second calculation method is used, a set of pixel data output from all the auxiliary preprocessing units 45 (1) to 45 (N), that is, all of the transformation processing and the preprocessing are performed. Corrected image buffers 94 (1) to 94 for storing the image data of the auxiliary images Iq (1) to Iq (N) of
(N) is required.

The two image vectors Vi, Vj
Are reference pixels pr and auxiliary pixels pq (n), respectively.
The so-called textures in the vicinity of are respectively represented. The high correlation between the two image vectors Vi and Vj indicates that the texture near the reference pixel pr and the texture near the auxiliary pixel pq (n) are similar to each other. Therefore, when the correlation between the image vectors Vi and Vj is used as the correlation value Cn in the synthesizing process, the synthesizing rate of the synthesizing process can be controlled according to the similarity of the texture. in this way,
When the second calculation method is used, the correction reference image Ir $ and any one correction auxiliary image Iq are included in the combining process.
(N) Information on the texture of $ can be utilized. Therefore, so-called robustness can be improved.

The first method of calculating the vector correlation value will be described below. In the first calculation method, the inner product of the image vectors Vi and Vj shown in Expression 33 is used as the first vector correlation value Cv1 of the image vectors Vi and Vj. This is because the inner product generally approaches 1 when two vectors to be processed are in the same direction, and is less than 1 when the two vectors to be processed are in different directions. It takes advantage of the value.

Cv1 = Vi · Vj (33) The second calculation method of the vector correlation value will be described below. In the second calculation method, the image vectors Vi, Vj
As the second vector correlation value Cv2 of, the sum of the absolute values of the differences between the components of the image vector shown in Expression 34 is used. The second vector correlation value Cv2 does not require a multiplication process to calculate the vector correlation value Cv2 because the calculation formula 34 is configured only by addition and subtraction. Therefore, as compared with the case where the first calculation method is used, when the second calculation method is used, the correlation value calculation units 91 (1) to 91
The calculation amount of (N) is reduced.

Cv2 = Σ _k | Vik-Vjk | (34) The third calculation method of the vector correlation value will be described below. In the third calculation method, the image vectors Vi, Vj
As the third vector correlation value Cv3 of, the reciprocal of the luminance component of the reference pixel pr and the auxiliary pixel pq (n) is used. This is for the following reason. Generally, the random noise occurs in the low-luminance region in the image, and a so-called afterimage caused by an error in misregistration becomes prominent in the high-luminance region in the image that includes many pixels with high luminance. For this reason, weakening the synthesizing process for the high-luminance region in the image,
In addition, if the combining process is positively performed in the low-luminance region in the image, it is possible to suppress the random noise by using the combining process while suppressing the afterimage caused by the error in the shift correction. Specifically, the third vector correlation value Cv3
Is the reciprocal of the value obtained by adding 1 to the average value of the luminance values ya and yb of the reference pixel pr and the auxiliary pixel pq (n) as shown in Expression 35. The brightness values ya and yb are defined by using the above-described Expression 13. In the following equation, 1 is added to the average value to prevent the third vector correlation value Cv3 from becoming peculiar when the average value becomes 0. This is the end of the description of the vector correlation value calculation method.

Cv3 = 1 / ((y1 + y2) / 2 + 1) (35) In the above description of the combining process, the correlation value Cn is calculated by the expression 36.
It may be replaced with the product Cn * of the correlation value Cn and the range designation coefficient ka shown in. This is because, in the synthesizing process, the range in which the correlation value Cn influences the effective range that the brightness value y can take is limited to a limited range of a part of the effective range, or the brightness component of the pixel. This is so that the distribution of the applied strength of the correlation value Cn can be changed.
The range designation coefficient ka is determined by the evaluation function shown in Expression 37. In the following equation, “p” is a coefficient that determines the degree of change in the evaluation function. “Ia” is an average value of the luminance values ya and yb of the reference pixel pr and the auxiliary pixel pq (n) as shown in Expression 38. “It” is a predetermined threshold value of the average value Ia of the brightness values. The threshold It is, for example, 0.5, and the coefficient p is, for example, 10. As described above, by using the product Cn * by substituting the correlation value Cn, it is possible to easily specify that the range of the luminance value to be combined is greatly changed only by changing the coefficient p.

Cn * = Cn × ka (36) ka = 1 / (1 + exp (-p × (Ia-It))) (37) Ia = (ya + yb) / 2 (38) Saturation enhancement unit 55 The saturation emphasis processing performed by will be described below. The saturation enhancement unit 55 includes the brightness correction units 51 and 53 (1) to
This is for correcting the deterioration of the image quality of the composite image Is due to the colors lost from the reference and auxiliary images Ir, Iq (1) to Iq (N) due to the brightness correction processing at 53 (N). Generally, the saturation of each pixel of the combined image Is obtained by the combining unit 46 is emphasized more strongly as the brightness of each pixel is higher. There are first and second methods for the saturation enhancement processing. In both the first and second methods, the pixels of the combined image are sequentially processed one by one, and the processing is repeated until all the pixels of the combined image are processed.

First, the first method of saturation enhancement processing will be described. In the first method, the saturation enhancement unit 55 first converts the color system of the pixel data of one pixel of the processing target of the combined image Is from the RGB color system to the HSV color system. Next, the luminance component V of the pixel data of the pixel is examined, and as the luminance component V is larger, the saturation component S of the pixel is enlarged. Finally, the color system of the pixel data of the pixel is changed from the HSV color system to the RGB color system. This process is repeated for each pixel of the composite image.

Next, the second method of saturation enhancement processing will be described. In the first method, the saturation enhancement unit 55 first obtains the average value ac2 of the three color components r *, g *, and b * of the pixel data of one pixel to be processed in the combined image Is. Next, the saturation enhancement unit 55 causes the color components r *, g *, b *.
Is the difference ac2-r *, a between each color component and the average value ac2.
It is replaced with a value obtained by enlarging the product of c2-g *, ac2-b * and the pixel evaluation coefficient k. That is, the color components r *, g *, b * are replaced with the color components R *, G *, B * shown in the following equations 39 to 41, respectively. The pixel evaluation coefficient k is defined by the equations 1 to 3 in the description of the saturation suppressing section 67 described above. As a result, the color components R *, G *, B
* Is emphasized as the luminance component of the pixel increases. At this time, when the pixel data including the converted color components R *, G *, and B * is converted into the HSV color system, the brightness component V of the pixel data is out of the predetermined effective range acquired by the brightness component V. It may protrude. Therefore, after the conversion of the color component, it is preferable to use so-called clipping to check whether or not the luminance component is within the effective range, and if it is not within the effective range, it is preferable to correct the pixel data.

R * = − (1-k) × (ac2-r *) + r * (39) G * = − (1-k) × (ac2-g *) + g * (40) B * = -(1-k) * (ac2-b *) + b * (41) The pixel data whose saturation has been emphasized by the above-described first or second method is sequentially output from the saturation emphasizing unit 55. . The set of pixel data is a combined image I that has been subjected to saturation enhancement processing.
Corresponds to s *. In the saturation emphasis process, the degree of emphasis of the saturation component of the pixel of the composite image is set as described above.
Instead of continuously increasing as the pixel evaluation coefficient increases, it is checked for each pixel of the composite image Is whether or not the brightness component V of the pixel is greater than or equal to a predetermined brightness threshold, and the brightness component is checked. You may make it emphasize only the saturation of the pixel which is above the said threshold value of brightness. This is the end of the description of the saturation enhancement processing.

The sharpening process performed by the sharpening unit 56 will be described below. The sharpening process of the sharpening unit 10 is performed by the composite image Is.
In order to improve the image quality of the composite image Is, blurring occurring in the composite image Is is suppressed and the sharpness of the composite image is restored. The blur and the deterioration of the sharpness are caused by the reference image Ir and each auxiliary image Iq.
The deviation from (1) to Iq (N) is caused by the deformation portions 44 (1) to 4
This occurs when correction cannot be completely performed by the 4 (N) transformation process.

In general, the sharpening unit 10 performs sharpening processing using a filter for sharpening processing on the composite image Is * with enhanced saturation, and the sharpness of the composite image Is * is increased. Will be recovered. The filter is, for example, unsharp masking using a so-called Laplacian operator. When the sharpening process is applied to the low-luminance region in the composite image Is * during the sharpening process, the suppressed random noise becomes noticeable again. Therefore, the sharpening unit 56 strongly sharpens the high-luminance region in the composite image Is * so that the low-luminance region in the image is not affected by the sharpening process. That is, each of the three color components of the pixel data of each pixel in the composite image Is * is converted as shown in Expression 42.

I * = I−αc × Y × L (I) (42) In the above equation, “I” represents three colors in the pixel data of any one pixel of the composite image Is * to be processed. Any one of the color components, “L (I)” is the so-called Laplacian of the arbitrary pixel, “αc” is a constant indicating the strength of edge enhancement, and “Y” is the above Any 1
The luminance value of one pixel, and “I *” is any one of the color components after the sharpening process. The brightness value is given by
Is determined using. Further, when the sharpening process is performed, the luminance values Y of all the pixels p in the composite image Is *.
Is normalized to a range of 0 or more and 1 or less.

The color component I * converted by the above method is sequentially output from the sharpening unit 56. The set of the three transformed color components I * of all the pixels of the composite image corresponds to the output image Io. As a result, the sharpening process is performed only on the high-luminance region in the composite image Is * while suppressing random noise. As a result, the image quality of the output image Io can be improved as compared with the reference image Ir. Above, description of the sharpening part 56 is complete | finished.

An image processing apparatus according to the second embodiment of the present invention will be described below. The image processing device is realized by a computer including a central processing unit, a storage device, a display device, and a communication unit. Image processing software including programs and data for causing the central processing unit to perform the image quality improvement processing including the various kinds of processing described in the first embodiment is installed in the computer. The image processing program is stored in, for example, a storage medium at the time of distribution, the storage medium is installed in the computer, and the software in the storage medium is installed in the computer.
Examples of the storage medium include a CD-ROM and a floppy disk. The image processing software is, for example, bundled software of a digital still camera. It is assumed that the computer can exchange image files with the photographing device 23 via the communication unit 29.

In this case, the block diagram of FIG. 2 corresponds to a functional block diagram showing the overall flow of the program,
The block diagrams of FIGS. 3, 6 and 10 correspond to the functional block diagrams showing the flow of the subroutine for performing the various processes described in FIGS. The charts in FIGS. 4, 5, 7-9 are
It represents the characteristics of the data in the software. In the functional block diagram, a single block represents a series of processing operations for a certain purpose in the operation program of the central processing unit, that is, a so-called subroutine, and an arrow toward the block indicates the processing operation. The necessary input signals and data are represented, and the arrows coming out of the block represent the output signals and data showing the processing result of the processing operation.

To operate the computer as the image processing apparatus 22, the operator first installs one of the image files 36 in the storage unit 30 while the image processing software is installed in the computer. One of them is designated, and the execution of the installed software is instructed. The central processing unit of the computer executes the program in the installed image processing software in response to the instruction.
As a result, the central processing unit and the memory in the computer sequentially operate as the respective components in the image quality improving unit 31, so that the entire computer operates as the image processing device 22.

As a result, the central processing unit reads the image file 36 designated by the operator from the storage unit 30, executes the above-mentioned image quality improving process using a plurality of images in the image file, and The image file 37 including the output image Io obtained as a result of the improvement process is stored in the storage unit 30 again. After the image quality improving process, the operator further causes the central processing unit of the computer to execute a so-called image file viewer program suitable for displaying the output image Io. As a result, the image file 37 is reproduced and the output image Io is displayed on the display device of the computer. Thereby, the operator can obtain the output image Io whose image quality is improved as compared with the reference image Ir. With such a procedure, the image processing apparatus 22 of the first embodiment can be easily realized by using a general-purpose computer.

The image processing device 22 of the first embodiment described above.
The computer in which the image processing software of the second embodiment is installed is an external device to the photographing device 23. Image quality improvement unit 3 in the image processing device 22
1 and a portion that executes the image processing program in the computer may be incorporated in the image capturing device 23.

The photographing device 23 may be a device other than the digital still camera as long as it is a device capable of photographing a single subject a plurality of times. For example, the photographing device 23 may be a video camera. Imaging device 23
Is a video camera, each frame of the movie is
It is regarded as a plurality of images to be processed by the image quality improvement processing. As a result, the image quality of each frame of the moving image can be improved by the image quality improving process. As a result of this,
When a still image is obtained using the video camera, the image quality of the still image can be improved.

Further, when the photographing device 23 is a video camera, the first frame of the plurality of consecutive frames of the moving image is set as the reference image Ir, and the second to the last of the plurality of consecutive frames are set. The frames may be the auxiliary images Iq (1) to Iq (N). Further, in the above case, any one frame of the plurality of continuous frames of the moving image is set as a reference image Ir, and the remaining frames other than the one frame of the plurality of continuous frames are set as auxiliary images. Iq (1) to Iq
It may be (N). In this case, it is preferable that any one of the plurality of frames has a largest area overlapping with other frames in the plurality of frames. Furthermore, when selecting the plurality of frames from the moving image, if the frame sequence composed of the plurality of frames spans two scenes,
In the combining process within the image improving process, the image quality of the combining process may be impaired. Therefore, the frame sequence is
It is preferable to use a so-called cut detection technique to extract only a plurality of frames in a single scene and set the plurality of frames as a plurality of images to be processed by the image quality improvement processing.

Further, in the image quality improving process, only a part of the image quality improving process may be selectively performed.
Which part of all the processes in the image quality improving process is executed can be switched according to, for example, switching of the execution mode of the image quality improving process. Of the plurality of processes in the image quality improving process, the larger the number of processes to be executed,
It is possible to improve the image quality of the output image Io compared to the reference image Ir. In particular, the brightness correction process,
When performing a combination with at least one of the remaining processes other than the brightness correction process in the image quality improvement process, the influence of random noise in Ir in the reference image amplified by the brightness correction process The image quality of the output image Io can be improved by sufficiently removing it from the output image Io.

For example, in the first method in which only the preprocessing, that is, the luminance correction processing and the color noise removal processing are executed, the image quality improvement processing of the reference image Ir can be performed by repeating the processing for each pixel. . In the first case, the efficiency of the image quality improving process is higher than that of uniformly processing the entire image. Further, as a second method, at least one of the saturation enhancement processing and the sharpening processing in the pre-processing and the post-processing may be executed. In the second approach,
Repeating the process for each pixel to enhance or sharpen the saturation of the portion of the reference image Ir where the random noise is likely to occur while suppressing the protrusion of the saturation component due to the random noise. Therefore, the efficiency of the image quality improvement process is improved.

As a third method, the pre-processing,
The corrected reference image Ir $ and the auxiliary image Iq are added to the image to be combined.
You may perform only the said synthetic | combination process using (1) -Iq (N). In the third method, since the random noise can be removed by using two methods, that is, saturation suppression and image composition, the image quality of the output image Io is further improved. Furthermore, as a fourth method, the above-mentioned pre-processing and the correction reference image Ir $ and the correction auxiliary image Iq are added to the image to be combined.
(1) Only the combining process using $ to Iq (N) $ may be executed. According to the fourth method, the image quality of the output image Io can be improved, and the noise removal accuracy can be increased while synthesizing a smaller number of images than the image processing device of the related art. Further, in the fifth method of executing the preprocessing, the deformation processing, and the synthesis processing of the third method, a plurality of images obtained by photographing the subject while holding the photographing device 23 by hand are processed. Even when targeted,
The image quality of the reference image Ir can be sufficiently improved by performing highly accurate noise removal and preventing the influence of the blurring of the composite image.

As a sixth method, at least one of the two processes in the post-process may be performed on the reference image Ir that has been subjected to only the brightness correction process. In the sixth method, since the post-processing can be performed on a pixel-by-pixel basis, the efficiency of the image quality improvement processing is better than that of the conventional image quality improvement processing. Furthermore, as a seventh method, the reference image Ir and the auxiliary images Iq (1) to Iq that have been subjected to the brightness correction processing are used.
(N) may be combined by the combining process of the second or third method, and at least one of the post-processes may be added to the combined image obtained as a result. As a result, two types of noise removal processing and post-processing can be performed at the same time, so that the image quality of the output image can be further improved over the image quality of the reference image.

Furthermore, as an eighth method, after the deviation correction processing is applied to the reference image Ir and the auxiliary images Iq (1) to Iq (N), the reference image Ir and the auxiliary image Iq (1) after the deviation correction are added. ) To Iq (N) may be combined by the combining process, and at least one of the post-processes may be added to the resulting combined image. As a result, the photographing device 2 is used when photographing the images Ir, Iq (1) to Iq (N).
3 is also held by hand, noise is removed with high accuracy and blurring of the composite image is prevented, and the reference image I
The image quality of r can be sufficiently improved. Further, as a ninth method, only at least one of the two processes of the pre-processing, the second combining processing, and the post-processing may be performed. In the ninth method, since two types of noise removal processing using saturation suppression and image synthesis and post-processing are performed at the same time, the image quality of the reference image Ir can be sufficiently improved.

In the first to ninth methods of the image improving process, the synthesizing process may be the synthesizing process using the simple average, or the synthesizing process using the weighted average using the correlation value. It may be. When the latter synthesizing process is used, it is possible to prevent the output image Io from being affected by the blurring of the synthetic image due to so-called camera shake, in addition to the effects of the plurality of cases. This further improves the image quality of the output image Io.

As in the first and second embodiments described above, as the image improvement processing, the luminance correction processing, the saturation suppression processing, and the deviation correction of the auxiliary images Iq (1) to Iq (N) are performed. When performing the deformation process, the combining process using the weighted average, and the post-process, the correction process of the blur caused by the hand-held blur and the image shift caused by the hand-held process and the removal process of the random noise are performed by the conventional image processing. This can be done using a smaller number of images than the processing device. Therefore, the quality of the output image is best improved,
In addition, the image processing becomes easier than the image processing device of the related art.

The image processing device 22 and the image processing program of this embodiment are examples of the image processing device and the image processing program of the present invention.
It can be implemented in various other forms. In particular, the detailed operation of each device and unit is not limited to this and may be realized by another operation as long as the same processing result is obtained.

[0158]

As described above, according to the present invention, in the image processing apparatus, the lower the brightness of the pixel and the higher the saturation of the pixel are, the more the processed image subjected to the so-called brightness correction process. The saturation suppression processing that strongly suppresses the saturation is performed in pixel units. Accordingly, even if the processed image is obtained by photographing a subject having low illuminance, the image quality of the processed image can be efficiently improved.

Further, according to the present invention, the image processing apparatus not only suppresses the saturation of each pixel in the processed image whose brightness is corrected, but also at least one auxiliary image and the saturation. The average is combined with the processed image in which the noise is suppressed. The image quality of the composite image thus obtained is higher than that of the processed image in which the saturation is suppressed. Furthermore, according to the present invention, the image processing apparatus performs brightness correction processing and saturation suppression processing on the images to be combined in the average combining processing, that is, the processed image and the auxiliary image in advance. As a result, it is possible to reduce the minimum number of images to be combined that are required to remove the influence of random noise from the combined image.

Further, according to the present invention, the image processing device uses so-called simple averaging in the synthesizing process. As a result, the influence of random noise can be removed from the composite image by a simple calculation process. Furthermore, according to the present invention, the image processing device, in the combining process,
A weighted average using the correlation is used. This makes it possible to improve the image quality of the combined image even when the image to be combined has a shift due to so-called camera shake. Further, according to the present invention, the image processing apparatus excludes, from among the images to be combined, an auxiliary image having low reliability in detecting a deviation from the image to be processed. As a result, it is possible to prevent the image quality of the composite image from being deteriorated due to the misdetection of the shift.

Further, according to the present invention, the image processing apparatus deforms the auxiliary image based on the shift between the auxiliary image and the processed image, and then the deformed auxiliary image and the processed image. And are combined. This makes it possible to improve the image quality of the composite image even when there is a shift between the auxiliary image and the image to be processed due to so-called camera shake. Further, according to the present invention, the image processing device selects a plurality of feature points to be used as a reference for obtaining the shift from a plurality of regions arranged in the image to be processed with a predetermined distance from each other. Extract one by one. As a result, it is possible to reliably detect the deviation between the processed image and each auxiliary image.

Further, according to the present invention, the brightness correction means of the image processing device converts the brightness of each pixel of the image to be processed based on a predetermined function of the sum of the brightness and a predetermined reference brightness. Replace with the specified value. Thereby, it is possible to prevent the random noise superimposed on the image to be processed from being amplified due to the brightness correction process.

Further, according to the present invention, the image processing apparatus further sets the saturation of the pixel of the processing target image subjected to the brightness correction processing and the saturation suppression processing as the corrected brightness is higher. Emphasize. This makes it possible to correct only the saturation of each pixel lost by the brightness correction processing without increasing the random noise suppressed by the saturation suppression processing. Furthermore, according to the present invention, the image processing apparatus further sharpens the processed image that has been subjected to the luminance correction processing and the saturation suppression processing. The degree of sharpening is stronger in the higher brightness area in the image to be processed. As a result, without increasing the random noise suppressed by the saturation suppression processing,
The image to be processed can be sharpened.

Further, according to the present invention, the image processing apparatus stores in the medium an image processing program for further subjecting the processed image subjected to the so-called brightness correction processing to the saturation suppression processing in pixel units. To be done. The image processing apparatus can be easily realized only by installing and executing this image processing program in a computer.

[Brief description of drawings]

FIG. 1 is an image processing device 22 according to a first embodiment of the present invention.
FIG. 3 is a block diagram showing an electrical configuration of the image creating device 21 including the above.

FIG. 2 is a block diagram showing a functional configuration of an image quality improving unit 31 in the image processing device 22.

FIG. 3 is a block diagram showing a functional configuration of a brightness correction unit 51 in a reference preprocessing unit 41 in the image quality improvement unit 31.

FIG. 4 is a graph showing characteristics of a brightness correction lookup table provided in the image quality improving unit 31.

5 is a graph showing the characteristics of a brightness correction lookup table provided in the image quality improving unit 31. FIG.

FIG. 6 is a block diagram showing a functional configuration of a color noise removing unit 52 in the reference preprocessing unit 41 in the image quality improving unit 31.

FIG. 7 shows a relationship between a luminance evaluation coefficient ki used for determining a degree of suppression of a saturation component S of a pixel and a luminance component V of a pixel in a saturation suppression unit 67 in the color noise removing unit 5. It is a graph shown.

FIG. 8 illustrates a saturation evaluation coefficient ks used for determining a degree of suppression of a saturation component S of a pixel and a saturation component S of a pixel in a saturation suppression unit 67 in the color noise removing unit 5. It is a graph which shows a relationship.

9 is a schematic diagram of a reference image Ir for explaining the operation of a feature point extraction unit 42 in the image quality improvement unit 31. FIG.

10 is a reference image Ir and an auxiliary image Iq for explaining the operation of the auxiliary preprocessing unit 45 in the image quality improving unit 31. FIG.
It is a schematic diagram of (n).

11 is a functional block diagram for explaining a combining process of a first method in a combining unit in the image quality improving unit 31. FIG.

FIG. 12 is a functional block diagram for explaining the combining process of the second method in the combining unit 46 in the image quality improvement unit 31.

FIG. 13 is a block diagram showing an electrical configuration of a noise removal circuit 1 which is a first conventional technique relating to suppression of random noise.

FIG. 14 is a block diagram showing an electrical configuration of a video signal processing device which is a second conventional technique relating to suppression of random noise.

[Explanation of symbols]

22 Image processing device 23 Imaging device 29 Communications Department 31 Image Quality Improvement Section 42 Feature Point Extraction Unit 43 (1) to 43 (N) corresponding point searching unit 44 (1) to 44 (N) Deformation part 46 Composition Department 51, 53 (1) to 53 (N) brightness correction unit 52, 54 (1) to 54 (N) color noise removing unit 55 Saturation emphasis section 56 Sharpened part 81, 82, 83 Feature point search area

─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Yasuhisa Nakamura 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka Inside Sharp Corporation (56) Reference JP-A-57-44385 (JP, A) JP-A-5- 91395 (JP, A) JP-A-6-189324 (JP, A) JP-A-7-135599 (JP, A) JP-A-7-143509 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 9/64-9/68 H04N 1/46 H04N 1/60

Claims (12)

(57) [Claims] 1. A to-be-processed image input means for inputting a to-be-processed image of a color composed of a plurality of pixels, and a to-be-processed image for correcting the brightness of each pixel of the to-be-processed image based on a predetermined function. Image brightness correction means, and the saturation of each pixel of the processed image whose brightness component has been corrected is suppressed as the corrected brightness of each pixel becomes lower and the saturation of each pixel becomes higher. An image processing apparatus comprising: a saturation suppressing unit for an image to be processed. 2. An auxiliary image inputting means for inputting at least one color auxiliary image including a plurality of pixels, in which the same subject as the processed image is captured, and the processed image whose saturation is suppressed. , Further comprising a synthesizing unit for synthesizing all the auxiliary images to obtain a synthetic image, wherein the color of each pixel of the synthetic image is the color of each pixel of the processed image in which the saturation is suppressed, The image processing apparatus according to claim 1, wherein an average of colors of pixels of all the auxiliary images corresponding to pixels of the image to be processed is obtained. 3. The brightness of each pixel of all the auxiliary images
Auxiliary image brightness correction means for respectively correcting based on a predetermined function, and saturation of each pixel of all the brightness-corrected auxiliary images, the corrected brightness of each pixel is low and As the saturation of the pixel is higher, it further includes auxiliary image saturation suppression means for suppressing the saturation, and the synthesizing means is the processed image in which saturation is suppressed,
The image processing apparatus according to claim 2, wherein the image processing apparatus combines all auxiliary images of which saturation is suppressed. 4. The color of each pixel of the composite image is the color of each pixel of the processed image in which the saturation is suppressed and each of the auxiliary images corresponding to each pixel of the processed image. The image processing device according to claim 2, wherein the image processing device is a simple average of the color of the pixel. 5. Correlation between each pixel of each auxiliary image and each pixel of the target image in which the saturation is suppressed, which corresponds to each pixel of the auxiliary image, is individually calculated for each pixel of each auxiliary image. Further comprising a correlation calculating means for obtaining, the color of each pixel of the composite image, corresponding to the color of each pixel of the processed image in which the saturation is suppressed, and each pixel in the processed image, and 3. The image processing apparatus according to claim 2, wherein the image processing apparatus is an average of colors of respective pixels in all the auxiliary images weighted by the correlation values. 6. A deviation detecting unit for detecting a deviation between each of the auxiliary images and the processed image for each auxiliary image, and reliability of detection of a deviation between each of the auxiliary images and the processed image. And a reliability calculation unit that obtains each auxiliary image, wherein the combining unit is an auxiliary image in which the reliability of all the auxiliary images is equal to or more than a predetermined evaluation criterion, and the saturation is suppressed. The image processing apparatus according to claim 2, wherein the processed image is combined with the processed image. 7. A shift detecting unit for detecting a shift between each of the auxiliary images and the processed image, and each of the auxiliary images so as to cancel the shift between the processed image and each of the auxiliary images. The image processing according to claim 2, further comprising a deforming unit that deforms, wherein the combining unit combines all the deformed auxiliary images and the processed image in which the saturation is suppressed. apparatus. 8. The image processing apparatus further includes a feature point extracting unit that extracts one feature point from each of the three or more regions set in the image to be processed, wherein at least three regions of the region are each regions. Are arranged so that the figure having the reference point predetermined as a vertex becomes an equilateral triangle, and the deviation detecting means defines a plurality of corresponding points in each of the auxiliary images respectively corresponding to the plurality of feature points, Individually searched for each auxiliary image, based on the positional relationship between all the feature points in the image to be processed and the positional relationship between all the corresponding points in each of the auxiliary image, 8. The image processing apparatus according to claim 6, wherein a shift between the processed image and each of the auxiliary images is obtained. 9. The processed image brightness correction means obtains the sum of the brightness of each pixel of the processed image and a predetermined reference brightness, and corrects the sum based on the function. The image processing apparatus according to claim 1. 10. The saturation enhancement means for enhancing the suppressed saturation of each pixel of the processed image as the corrected luminance of each pixel is higher. Image processing device. 11. A sharpening unit for sharpening the processed image in which the saturation of each pixel is suppressed, wherein the degree of sharpening is within the processed image in which the saturation is suppressed. 2. The image processing apparatus according to claim 1, wherein the portion having the corrected high luminance pixel is stronger. 12. A medium for storing an image processing program for improving the image quality of a color processed image composed of a plurality of pixels, wherein the image processing program is the brightness of each pixel of the processed image. Is corrected for each pixel on the basis of a predetermined function, and the saturation of each pixel of the processed image whose brightness is corrected is calculated as follows: A medium for storing an image processing program, which is characterized in that the higher the degree, the more the degree is suppressed.