JP2001197356A - Device and method for restoring picture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately restore the deterioration of a picture. SOLUTION: A prescribed threshold is obtained from a picture and a restoration area to be restored is decided (steps S211 and S212). The picture is restored in the restoration area by using a deterioration function showing the deterioration characteristic of the picture (step S213). It is judged whether the correction of the restoration area is required or not based on the state of the boundary of the restoration area (step S214). When correction is required, the restoration area is extended and restoration is conducted again in the restoration area (steps S216 and S213). Thus, ringing occurred when the whole picture is restored can be prevented and the deterioration of the picture can appropriately be restored.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a technique for restoring image deterioration.

[0002]

2. Description of the Related Art Conventionally, for an image obtained as image data using a light receiving element array represented by a CCD,
Various techniques for restoring image degradation have been proposed. Deterioration of an image means that an actually obtained image has deteriorated with respect to an ideal image to be obtained from an imaging target.For example, an image obtained by using a digital camera is
The image is degraded by aberration depending on the aperture value, the focal length, the focus position, and the like, and further degraded by an optical low-pass filter for preventing spurious resolution. Also,
The image also deteriorates due to camera shake during shooting.

[0003] For such a deteriorated image, restoration has been conventionally performed to model an image obtained by modeling the deterioration of the image so as to approach an ideal image. For example,
Image deterioration is considered to be caused by light beams to be incident on each light receiving element spreading while following a Gaussian distribution, and a restoration function is applied to the entire image, or a filter for enhancing the edges of the image (a so-called aperture correction filter). ) Is applied to the entire image to restore the image.

[0004]

However, image degradation does not always occur in the entire image. For example, when photographing a subject having a single color of a geometric pattern or background, or when scanning a document for character recognition, a region that is not affected by deterioration exists in the image.

[0005] On the other hand, when the restoration process is performed on the entire image, there may be a case where a region that does not require the restoration process is adversely affected. For example, if restoration processing is performed in an area where noise or edges are present, ringing or noise enhancement occurs, and this adversely affects areas that do not require restoration.

The present invention has been made in view of the above problems, and has as its object to appropriately restore an image by performing a restoration process only on a specific area.

[0007]

According to a first aspect of the present invention, there is provided an image restoring apparatus for restoring image deterioration, wherein an image is restored based on a difference in pixel value between each pixel and a pixel in the vicinity of each pixel. And a restoration unit for restoring the image in the restoration region using at least one deterioration function indicating the deterioration characteristic of the image.

According to a second aspect of the present invention, there is provided an image restoring apparatus for restoring an image deterioration, wherein the area determining means determines a restoration area in the image based on at least one deterioration function indicating image deterioration characteristics. Restoring means for restoring the image in the restoration area using the at least one deterioration function.

According to a third aspect of the present invention, there is provided an image restoring apparatus for restoring image deterioration, comprising: an area determining means for determining a restored area in an image based on a pixel value of each pixel; Restoring means for restoring the image in the restoration area using at least one of the degradation functions shown.

According to a fourth aspect of the present invention, there is provided an image restoring apparatus for restoring image deterioration, wherein the area determining means determines a restoration area in the image to be restored, and the restoration means restores the image in the restoration area. Determining means for determining whether the restored image is appropriate, and area correcting means for correcting the restored area so that the image is restored by the restoring means again.

According to a fifth aspect of the present invention, there is provided an image restoration method for restoring image deterioration, wherein an area determination step of determining a restoration area to be restored in an image, and a restoration step of restoring the image in the restoration area. And a determining step of determining whether the restored image is appropriate, and, when it is determined in the determining step that the restored image is inappropriate, correcting the restored area; And a region correcting step.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS <1. First Embodiment><1.1 Configuration of Digital Camera> FIGS. 1 to 3 are views showing the appearance of a digital camera 1 according to a first embodiment of the present invention. FIG. 1 is a front view and FIG. Is a rear view, and FIG. 3 is a left side view. 1 and 2 show a state in which the memory card 91 is mounted, and FIG. 3 does not show the memory card 91.

The main structure of the digital camera 1 is the same as that of a normal digital camera. As shown in FIG. 1, a lens unit 2 for guiding light from a subject to a CCD as shown in FIG. Flash 1 that emits
1 is disposed, and a finder 12 for capturing a subject is disposed above the lens unit 2.

A shutter button 13 to be pressed at the time of a shooting operation is arranged on the upper surface, and a card slot 14 for mounting a memory card 91 is provided on the left side surface as shown in FIG.

As shown in FIG. 2, on the back of the digital camera 1, a liquid crystal display 15 for displaying an image obtained by photography and an operation screen, a changeover switch 161 for switching between a photography mode and a reproduction mode, A four-way key 162 for receiving a selection input is arranged.

FIG. 4 is a longitudinal sectional view showing the internal structure of the digital camera 1 in the vicinity of the lens unit 2. The lens unit 2 includes a lens system 2 including a plurality of lenses.
An optical low-pass filter 31 and a single-plate color CCD 32 having a two-dimensional light receiving element array are sequentially arranged behind the lens unit 2. That is, in the digital camera 1, the lens system 21,
The diaphragm 22 and the optical low-pass filter 31 constitute an optical system for guiding light from a subject to the CCD 32.

FIG. 5 is a block diagram showing a main configuration relating to the operation of the digital camera 1. In FIG. 5, the shutter button 13, the changeover switch 161, and the four-way key 162 are illustrated as the operation unit 16.

A CPU 41, a ROM 42, and a RAM 43 shown in FIG. 5 are configured to control the entire operation of the digital camera 1, and the CPU 41, the ROM 42, and the RAM 4
Various configurations are connected to the bus line as well as 3. Then, the CPU 41 sets the RO 43 while using the RAM 43 as a work area.
By performing the arithmetic processing according to the program 421 in the M42, the operation of each part of the digital camera 1 and the image processing are performed.

The lens unit 2 includes a lens system 21 and a diaphragm 22 and a lens driving unit 211 for driving them.
An aperture driving unit 221 is provided, and the lens system 21 and the aperture 22 are appropriately controlled by the CPU 41 according to the output of the distance measurement sensor and the brightness of the subject.

The CCD 32 is connected to an A / D converter 33, and outputs an image of a subject formed through the lens system 21, the aperture 22, and the optical low-pass filter 31 to the A / D converter 33 as an image signal. I do. Image signal is A / D
After being converted into a digital signal (hereinafter referred to as “image data”) by the converter 33, the digital signal is stored in the image memory 34. That is, the image of the subject is obtained as image data by the optical system, the CCD 32 and the A / D converter 33.

The correction section 44 performs various image processing such as white balance correction, gamma correction, noise removal, color correction, and color enhancement on the image data in the image memory 34. The corrected image data is transferred to a VRAM (video RAM) 151, whereby an image is displayed on the display 15. Further, the image data is transferred to the memory card 91 via the card slot 14 as necessary by the operation of the operator.
Will be recorded.

In the digital camera 1, a process for restoring the deterioration of the acquired image data due to the influence of the optical system is performed.
This is realized by the PU 41 performing arithmetic processing according to the program 421 in the ROM 42. In the digital camera 1, image processing (correction and restoration) is substantially performed by processing image data, but in the following description, “image data” to be processed is simply referred to as “image”. That.

<1.2 Image Deterioration by Optical System>
An image deterioration in the digital camera 1 will be described. The image deterioration means that the CCD 32 of the digital camera 1
An image acquired via the A / D converter 33 or the like does not become an ideal image. Such image deterioration occurs because light rays emitted from one point on the subject do not converge on the CCD 32 but have a spread distribution. In other words, when an ideal image is obtained, a light beam to be incident on one light receiving element (that is, a pixel) of the CCD 32 spreads and is incident on a surrounding light receiving element, so that image deterioration occurs.

In the digital camera 1, the deterioration of the image is restored by the optical system mainly composed of the lens system 21, the stop 22, and the optical low-pass filter 31.

FIG. 6 is a diagram for explaining image deterioration due to the lens unit 2. Reference numeral 71 in FIG. 6 indicates the entire image, and is an ideal image (that is, an image that is not degraded by the influence of the optical system, hereinafter, an “ideal image”).
That. Assuming that the area indicated by reference numeral 701 becomes brighter, in an image actually obtained (hereinafter, referred to as an “acquired image”), the focal length and focus position of the lens system 21 (lens extension amount in a zoom lens) And the area 70 according to the aperture value of the aperture 22.
The area 711 wider than 1 becomes brighter. That is,
Ideally, the luminous flux to be incident on the area on the CCD 32 corresponding to the area 701 spreads and is incident on the area corresponding to the area 711.

Also in the peripheral area of the image 71, assuming that the area indicated by the reference numeral 702 is bright in the case of the ideal image, the area which is substantially elliptical as indicated by the reference numeral 712 is bright in the acquired image. Become.

FIGS. 7 to 9 are schematic diagrams for explaining the deterioration of an image due to the optical influence of the lens unit 2 on the light receiving element level of the CCD 32. FIG. FIG. 7 shows a state in which a light flux of intensity 1 is incident only on the light receiving element at the center of the 3 × 3 light receiving element array in a state where there is no influence of the lens unit 2 (that is, a state where an ideal image is obtained). . 8 and 9 show how the state shown in FIG. 7 changes due to the influence of the lens unit 2.

FIG. 8 shows an example of a state in the vicinity of the center of the CCD 32. Light with an intensity of 1/3 is incident on the central light receiving element, and light with an intensity of 1/6 is incident on adjacent upper, lower, left and right light receiving elements. It shows how to do. That is, the figure shows that the light beam to be incident on the central light receiving element spreads and enters the periphery due to the influence of the lens unit 2. FIG. 9 shows a CCD 32
The figure shows an example of the state of the peripheral portion of FIG. 5, in which light having an intensity of 1/4 is incident on the central light receiving element and light is incident from the upper left to the lower right while spreading.

A function of converting the pixel value of each pixel of the ideal image into a distribution of pixel values exemplified in FIGS. 8 and 9 (that is, a two-dimensional filter based on a point image distribution) is based on such image degradation characteristics. Therefore, it is called a deterioration function (or deterioration filter).

A deterioration function indicating a deterioration characteristic due to the influence of the lens unit 2 is calculated for each position of the light receiving element (that is, for each pixel position) based on the focal length of the lens system 21, the focus position, and the aperture value of the aperture 22. 2) can be obtained in advance. Therefore, in the digital camera 1, as described later, information on the lens arrangement and the aperture value are obtained from the lens unit 2, a deterioration function corresponding to the position of the pixel is obtained, and restoration of the acquired image is realized based on the deterioration function. ing.

The deterioration function of the lens unit 2 is generally a non-linear function having parameters such as a focal length, a focus position, an aperture value, and two-dimensional coordinates on the CCD 32 (that is, a pixel in an image). Function. Also,
Although the colors of the image are not described in FIGS. 7 to 9 for the sake of convenience, in the case of a color image, a deterioration function corresponding to each of the RGB colors or a deterioration function obtained by summing the deterioration functions of the respective colors is obtained. However, in order to simplify the process, the chromatic aberration may be ignored, and the degradation functions corresponding to the respective RGB colors may be the same.

FIG. 10 is a schematic diagram for explaining the deterioration due to the influence of the optical low-pass filter 31 at the light receiving element level of the CCD 32. The optical low-pass filter 31 is for preventing false resolution by performing band limitation using a birefringent optical material.
As exemplified by 0, the light to be incident on the upper left light receiving element is first split up and down as indicated by an arrow 721, and further split right and left as indicated by an arrow 722.

In a single-plate color CCD, four CCDs adjacent to each other
Of the two light receiving elements, green (G) filters are formed on two diagonal light receiving elements, and red (R) and blue (B) filters are formed on the remaining two light receiving elements. Then, the RGB values of each pixel are obtained by interpolation processing with reference to information obtained from surrounding pixels. However, since a single-chip color CCD has G pixels twice as many as R and B pixels, an image in which the resolution of G is higher than the resolution of R or B can be obtained by directly using the data obtained from the CCD. As a result, a high-frequency component of the subject image that cannot be captured by the light receiving element on which the R and B filters are formed appears as a false resolution.

Therefore, an optical low-pass filter 31 having a characteristic as shown in FIG. 10 is provided on the front surface of the CCD 32. Due to the influence of the optical low-pass filter 31, the high-frequency component of the image obtained from the G light receiving element is deteriorated. Would.

FIG. 11 is a diagram exemplifying the distribution of light beams to be incident on the central light receiving element by the optical low-pass filter 31 having the characteristics shown in FIG. 10, that is, the characteristics of the deterioration function corresponding to the optical low-pass filter 31 are schematically shown. FIG. As shown in FIG. 11, the optical low-pass filter 31
Divides a light beam to be incident on the central light receiving element into 2 × 2 light receiving elements. Therefore, in the digital camera 1, as described later, a deterioration function corresponding to the optical low-pass filter 31 is prepared in advance, and restoration of an acquired image based on the deterioration function is realized.

In the restoration using the deterioration function relating to the optical low-pass filter 31, a luminance component is obtained from the RGB values after the interpolation processing, and the luminance component is restored. As another restoration method, the G component may be restored after the G component interpolation processing, and the R and B components may be interpolated using the restored G component.

In the above description, it has been described that a deterioration function is obtained for each pixel. However, a deterioration function for a plurality of pixels or a deterioration function for all pixels (that is, a plurality of deterioration functions) is obtained. (A transformation matrix corresponding to pixel degradation) may be obtained.

<1.3 Image Restoration> Next, three specific examples of restoration of an acquired image using a deterioration function will be described. In the digital camera 1, any image restoration method may be adopted. As will be described later, the digital camera 1 obtains a restoration area in advance and performs restoration processing only on the restoration area. In the following description, first, the case of restoring the entire image will be described. The case where only the restoration area is restored is described.

FIG. 12 is a flowchart showing the flow of processing in the first image restoration method. The first image restoration method is a method of obtaining a restoration function from a deterioration function, and applying the restoration function to the acquired image to perform restoration.

When considering a deteriorated image in which a deterioration function is applied to each pixel of an ideal image, the deterioration function has a function of changing the pixel values of surrounding pixels based on the pixel value of each pixel. Is an image having a size larger than the ideal image. Here, assuming that the size of the ideal image and the size of the acquired image are the same, it is possible to regard the image obtained by deleting pixels around the deteriorated image as the acquired image. Therefore, when an attempt is made to obtain a restoration function that performs a reverse conversion to the deterioration function, the restoration function cannot be appropriately obtained because information outside the processing target region (that is, the outer periphery) is missing.

Therefore, in the first image restoration method, first,
A virtual pixel is provided outside the processing target area, and the pixel value of the virtual pixel is appropriately determined (step S11). For example, the pixel value of the pixel inside the boundary of the acquired image is determined as the pixel value of the pixel outside the boundary as it is. This allows
A vector Y which is an array of pixel values of the corrected acquired image
And a vector X, which is an array of pixel values of an ideal image, can be assumed to satisfy the relationship of Expression 1.

[0042]

(Equation 1)

Here, the matrix H is a deterioration function (hereinafter, referred to as an "image deterioration function") that applies the deterioration function of each pixel to the entire ideal image in which all the pixels are combined.

Thereafter, an inverse matrix H ^-1 of the matrix H, which is an image deterioration function, is obtained as a restoration function for restoring an image (step S12), and a vector X is obtained from Expression 2.

[0045]

(Equation 2)

That is, an image is restored by applying a restoration function to the corrected acquired image (step S13).

When restoring only the restoration area, not the entire image, only the pixel values of the pixels in the restoration area are used as the vectors X and Y, and a restoration function for converting the vector Y into the vector X is obtained. As a result, the processing amount can be reduced as compared with the case where the entire image is restored. Further, since the pixel values of many pixels around the restoration area are known, an appropriate restoration function can be easily obtained.

FIG. 13 is a flowchart showing the flow of processing in the second image restoration method. Since the degradation function generally has a characteristic of attenuating a specific frequency component in an ideal image, the second image restoration method restores an image by restoring a specific frequency component in an acquired image.

First, the acquired image is divided into blocks having a predetermined number of pixels (step S21), and each block is subjected to a two-dimensional Fourier transform (ie, discrete cosine transform (DCT)), and each block is transformed into a frequency space. (Step S22).

Next, restoration of the frequency component attenuated based on the characteristics of the deterioration function is performed (step S23). Specifically, the Fourier-transformed block is divided by the Fourier-transformed degradation function. Thereafter, the block is subjected to an inverse Fourier transform (inverse DCT) (step S24),
A restored image is obtained by combining the restored blocks (step S25).

When restoring only the restoration area instead of the entire image, only the restoration area is divided into blocks and processed. As a result, the processing amount can be reduced as compared with the case where the entire image is restored.

FIG. 14 is a flowchart showing the flow of processing in the third image restoration method. In the third image restoration method, an image before deterioration is assumed by assuming an image before deterioration (hereinafter, the assumed image is referred to as an “assumed image”) and updating the assumed image using an iterative method. It is.

First, the acquired image is used as the assumed image in the initial state (step S31). Next, a deterioration function (more precisely, a matrix H which is an image deterioration function) is applied to the hypothetical image (step S32), and a difference between the obtained image and the obtained image is obtained (step S33). Thus, the hypothetical image is updated (step S35).

More specifically, based on a vector Y as an acquired image and a vector X as a hypothetical image, W is defined as a weight matrix (or may be a unit matrix), and Obtained as a modified hypothetical image.

[0055]

(Equation 3)

Thereafter, the updating of the assumed image is repeated until the difference between the acquired image and the deteriorated assumed image falls within the allowable range (step S34), and the finally obtained assumed image becomes the restored image.

That is, the difference between the vector Y as the acquired image and the vector HX as the image obtained by deteriorating the hypothetical image is calculated as the sum of squares (or the sum of squares of the weights) of the pixel values of the pixels. By solving the linear equation Y = HX by an iterative method, a vector X that minimizes the difference is obtained. As a reference mentioning details of the third image restoration method, for example, "RESTORATION OF A SINGLE SUPER-R"
ESOLUTION IMAGE FROM SEVERAL BLURRED, NOISY AND UN
DER-SAMPLED MEASURED IMAGES "(M. Elad and A. Feuer, I
EEE Trans., On Image Processing, Vol.6 No.12 pp164
6-1658 Dec / 1997). Of course, various other techniques can be used for the details of the iterative method.

Even when restoring only the restoration area, not the entire image, the entire image is processed, but only the restoration area in the hypothetical image is updated. This improves the convergence stability of the solution in the iterative calculation for the restoration area.

By using the third image restoration method,
Although more appropriate image restoration can be performed than the first and second image restoration methods, the image restoration method adopted in the digital camera 1 can be any of the first to third image restoration methods. Often, other methods may be used.

<1.4 Determination of Restoration Area> As described above,
The digital camera 1 restores an image by determining a restoration area in an acquired image in advance and restoring only the restoration area. Next, as one method of determining the restoration area in the digital camera 1, a method of determining the restoration area using contrast will be described. In addition,
The contrast refers to a difference between a pixel value of a target pixel (hereinafter, referred to as a “target pixel”) and a neighboring pixel.

Any method may be used to determine the contrast of each pixel in the acquired image. For example, for example, the pixel values of the target pixel and the neighboring pixels (8 adjacent pixels, 24 adjacent pixels, etc.) may be used. The sum of the differences can be used. It is also possible to use the sum of the squares of the differences between the pixel values of the target pixel and the neighboring pixels or the sum of the ratios of the two pixel values as the contrast.

When the contrast of each pixel is obtained, a predetermined threshold value is compared with the contrast of each pixel, and a region of a pixel having a contrast larger than the threshold value is determined as a restoration region. The threshold is set higher as the exposure level (ie, the brightness of the entire image) is higher, and is set higher as the noise level is higher.

According to such a method, for example, a region indicated by a reference numeral 741 in FIG. 16 with respect to the acquired image shown in FIG. 15 is determined as a restoration region.

<1.5 Operation of Digital Camera> The configuration of the digital camera 1, the degradation function indicating the degradation of the acquired image, the restoration of the image using the degradation function, and the determination of the restoration area have been described above. The operation of the digital camera 1 for restoring an image in the restoration area will be described.

FIG. 17 is a flowchart showing an operation flow of the digital camera 1 at the time of photographing, FIG. 18 is a flowchart showing an operation flow of the digital camera 1 at the time of restoration, and FIG.
FIG. 2 is a block diagram illustrating a functional configuration related to photographing of the digital camera 1. 19, a lens control unit 401, an aperture control unit 402, a deterioration function calculation unit 403, a deterioration function storage unit 404, a restoration unit 405, and a restoration area determination unit 406.
The CPU 41 executes the arithmetic processing based on the program 421 in the ROM 42 so that the CPU 41
3 shows functions realized by the M42, the RAM 43, and the like.

When the shutter button 13 is pressed, the CCD
The optical system of the digital camera 1 is controlled to form an image of a subject on the image 32 (step S101). That is, the lens control unit 401 provides a control signal to the lens driving unit 211, and thereby, the arrangement of a plurality of lenses constituting the lens system 21 is controlled. Further, the aperture control unit 402
A control signal is supplied from the camera to the aperture driving unit 221 and the aperture 2
2 is controlled.

On the other hand, from the lens control unit 401 and the aperture control unit 402, information on the lens arrangement and the aperture value are sent to the degradation function calculation unit 403 as degradation information 431 for obtaining the degradation function (step S102). Thereafter, exposure is performed (step S103), and the CCD 32
The image of the subject acquired by the above is stored in the image memory 34 as image data. Subsequent image processing is performed on the image data stored in the image memory 34.

The deterioration function calculator 403 obtains a deterioration function of each pixel in consideration of the effects of the lens system 21 and the aperture 22, using the deterioration information 431 given from the lens controller 401 and the aperture controller 402 ( Step S1
04). The obtained deterioration function is stored in the deterioration function storage unit 404. Further, a deterioration function relating to the optical low-pass filter 31 is prepared in the deterioration function storage unit 404 in advance.

In step S104, the deterioration function may be once obtained individually for each configuration and characteristic of the lens unit 2, and thereafter, the deterioration function considering the entire optical system may be obtained. For example, a deterioration function for the lens system 21, a deterioration function for the stop 22, and
A deterioration function for the optical low-pass filter 31 may be separately prepared, and a deterioration function for the lens system 21 may be separately obtained as a deterioration function for the focal length and a deterioration function for the focus position.

Further, in order to simplify the calculation processing for obtaining the deterioration function for each pixel, a deterioration function of a representative pixel in the image is obtained, and the deterioration functions of the other pixels are replaced with the deterioration function of the representative pixel. May be obtained by interpolation.

When the deterioration function is obtained, the restoration area is determined by the restoration area determination unit 406, and the restoration processing described above is performed on the restoration area of the acquired image by the restoration unit 405 (step S105). Thus, the deterioration of the acquired image due to the influence of the optical system is restored only in the restoration area. That is, restoration of an image using the deterioration function regarding the optical low-pass filter 31 is performed in the restoration area,
Image restoration using the deterioration function for the lens unit 2 is performed in the restoration area.

At the time of image restoration, as shown in FIG. 18, first, a threshold value calculation unit 407 calculates a threshold value for determining a restoration area from an acquired image (step S201). 406 determines a restoration area by comparing the threshold value with the control of each pixel (step S20).
2). Then, the image is restored in the restoration area by the above-described restoration method using the deterioration function regarding the optical system (step S203).

In the restoration of an image using the deterioration function relating to the optical low-pass filter 31, a luminance component and a color component are obtained from the RGB values of each pixel in the restoration area after the interpolation processing, and the luminance component is restored. The luminance and color components are returned to RGB values.

On the other hand, in the image restoration using the deterioration function relating to the lens unit 2, restoration is performed in consideration of chromatic aberration for each of the RGB values of each pixel in the restoration area.
Of course, restoration of the image deterioration by the optical low-pass filter 31 and the lens unit 2 is performed only for the luminance component,
The processing may be simplified.

Further, the restoration of the deterioration by the optical low-pass filter 31 and the lens unit 2 may be performed simultaneously. That is, the image restoration in the restoration area may be performed after obtaining the deterioration function of the entire optical system.

The restored image is subjected to various image processing such as white balance correction, gamma correction, noise removal, color correction, and color enhancement by the correction unit 44 (step S106).
The image data after the correction is stored in the image memory 34.
Further, the image data in the image memory 34 is appropriately stored in the memory card 91 via the card slot 14 (Step S107).

As described above, in the digital camera 1, since the image deterioration due to the influence of the optical system is restored only in the restoration area by using the degradation function indicating the degradation characteristic of the optical system, ringing or noise is generated. Therefore, it is possible to suppress an influence on an undegraded area such as an increase in the number of images, and to appropriately restore an acquired image.

<2. Second Embodiment> Next, another restoration method of the digital camera 1 in the first embodiment will be described as a second embodiment. Note that the digital camera 1 according to the second embodiment has the same configuration as that shown in FIGS. 1 to 5, and the basic operation is also the same as that shown in FIG.

FIG. 20 shows the operation of the digital camera 1 according to the second embodiment in step S1 shown in FIG.
It is a figure which shows the content of 05. FIG. 21 is a block diagram showing a functional configuration around the restoring unit 405 in the functional configuration of the digital camera 1, and has a configuration in which a restoring area correcting unit 408 is added. The restoration area correction unit 408
19 is a function realized by the CPU 41, the ROM 42, the RAM 43, and the like, and the other functional configurations are the same as those in FIG.

Digital camera 1 according to second embodiment
Then, when restoring the acquired image, similarly to the first embodiment, first, the threshold value calculation unit 407 calculates a threshold value for contrast using the acquired image acquired by the CCD 32 (step S211). Then, the restoration area determination unit 406 determines an area where a pixel having a contrast exceeding the threshold value exists as a restoration area (step S212). Further, the restoration unit 405 restores the image in the restoration area using the deterioration function stored in the deterioration function storage unit 404 (Step S213).

FIG. 22 is a diagram showing an example of an acquired image.
FIG. 23 is a diagram illustrating a result of determining a restoration area using contrast and restoring an image. When the restoration area is determined using the contrast, the contrast is low in the completely collapsed area, and this area is not included in the restoration area. That is, the deterioration function has a characteristic of eliminating or reducing a specific frequency component. For example, the contrast of a striped area in an ideal image may become substantially zero in an acquired image. In FIG. 23, reference numeral 751 indicates a non-restoration area in spite of an area to be restored, and as a result, an area where restoration processing has not been performed.

In such an area to be restored but not subjected to the restoration processing, the restored pixel value in the processing area adjacent thereto generally changes greatly in the direction along the boundary of the area. . Therefore, in the digital camera 1, the state of the pixel values around the non-restored area (that is, the variation of the pixel values) in the restored image is confirmed, and when the pixel values around the non-restored area vary, The determination unit 409 determines that the restoration area needs to be corrected (step S214).

It should be noted that whether or not a non-restored area should be restored is determined by noting that pixel values near the boundary between adjacent restored areas do not converge at the time of restoration using an iterative method. Is also good.

If it is determined that the restoration area needs to be modified (step S215), the restoration area modification unit 408
Is modified to reduce the non-restoration area targeted (that is, expand the restoration area) (step S216). Then, the restoring unit 405 executes the restoring process again in the corrected restoration area (step S213), and returns to the determination step (step S214) again. After that, the correction of the restoration area and the restoration processing are repeated as necessary (steps S213 to S216). FIG. 24 is a diagram illustrating a result of performing image restoration accompanied by correction of a restoration area.
3 also shows a state in which restoration has been appropriately performed in the area 751 shown in FIG.

The restoration processing after the expansion of the restoration area may be performed on the image after the previous restoration processing, or may be performed on the first image (that is, the acquired image).

As described above, by expanding and correcting the restoration area based on the state of the boundary of the non-restoration area, the non-restoration area to be subjected to the restoration processing can be eliminated, and appropriate image restoration can be performed. Is achieved. The restored image is stored in the image memory 34 after being corrected by the correction unit 44 as in the first embodiment.

<3. Third Embodiment> In the first and second embodiments, the method of restoring the image degradation due to the optical system has been described. However, in the third embodiment, the image degradation due to other causes is restored. As a method, a digital camera 1 that restores image degradation due to camera shake during shooting
Will be described. The basic configuration of the digital camera 1 is shown in FIG.
5 to FIG. In the third embodiment, only correction of camera shake will be described, but of course, restoration of image deterioration due to the optical system may be performed together.

FIG. 25 is a diagram for explaining image deterioration due to camera shake. FIG. 25 shows a 5 × 5 light receiving element of the CCD 32. A light flux having an intensity of 1 to be incident on the light receiving element on the left side of the middle stage spreads to the right side due to camera shake, and as a result, the intensity becomes 1/5 to the left and right. FIG. That is, it shows a deterioration function when a point image becomes an image having a distribution due to deterioration.

Digital camera 1 according to third embodiment
In this document, it is possible to obtain a deterioration function relating to camera shake using a displacement sensor, and to appropriately restore an acquired image using the deterioration function.

FIG. 26 is a diagram showing details of step S105 in the overall operation of the digital camera 1 shown in FIG. 17, and FIG. 27 is a block diagram showing a functional configuration around the restoration unit 405. In the digital camera 1 according to the third embodiment, a displacement sensor 24 (a sensor for obtaining displacement by an acceleration sensor) for detecting the direction and amount of camera shake is provided. This is different from the first embodiment (FIG. 19) in that the data is transferred to the determination unit 406. Other functional configurations are the same as in the first embodiment.

In the third embodiment, as shown in FIG. 17, the digital camera 1 first controls the optical system to perform shooting (steps S101 and S10).
3) At this time, the information from the displacement sensor 24 is used as the deterioration information 431 indicating the deterioration of the acquired image.
3 (step S102). Thereafter, the deterioration function calculator 403 calculates a deterioration function having the characteristics illustrated in FIG. 25 based on the deterioration information 431 (step S10).
4) Transfer to the deterioration function storage unit 404.

Next, the restoration area is determined and the acquired image is restored (step S105). As shown in FIG. 26, the restoration area is determined based on the deterioration function regarding the blur transferred from the deterioration function storage unit 404. After the restoration area is determined by the unit 406 (step S221), the restoration of the image in the restoration area is performed using the deterioration function.
5 (step S222).

In the case where the deterioration of the obtained image has the deterioration characteristics shown in FIG. 25, even if there is a change in the pixel value in the vertical direction in the ideal image, if there is no change in the pixel value in the left and right direction, the obtained image is obtained. No degradation occurs in the image. For example, FIG.
In the case of the deterioration function having the deterioration characteristic of 5, even if a straight line extending in the left-right direction is photographed, the image does not deteriorate. Therefore, based on the deterioration function, the restoration area determination unit 406 determines only an area where there is a pixel whose contrast in the left-right direction (that is, the direction perpendicular to the blurring direction) is equal to or larger than a predetermined threshold.

As described above, by determining the restoration area based on the deterioration function, for example, the area 742 shown by the oblique lines in FIG. 28 is determined as the restoration area in the acquired image shown in FIG.

When the determination of the restoration area and the restoration of the acquired image are completed, the correction processing is performed in the same manner as in the first embodiment (step S106), and the corrected image is appropriately stored in the image memory 34 from the memory card. Transferred to 91.

As described above, the determination of the restoration area may be performed based on the deterioration function (for example, by setting a predetermined calculation formula from the deterioration function). In the above description, the deterioration function related to camera shake is used, but another deterioration function may be used. For example, with reference to the frequency characteristics of the deterioration function, a region where a predetermined frequency component is lost or a region where so-called two-line blur occurs may be determined as a restoration region. Further, the restoration area may be corrected as in the second embodiment.

<4. Fourth Embodiment> Next, an example in which a restoration area is determined by another method in the first embodiment will be described as a fourth embodiment. The configuration and the basic operation of the digital camera 1 are the same as those in FIGS. 1 to 5, FIG. 17, and FIG. 19, and will be described using the same reference numerals. Note that the digital camera 1 according to the fourth embodiment can also be used when restoring image deterioration due to various causes other than the optical system.

FIG. 29 is a flowchart showing the flow of the restoration processing (FIG. 17: step S105) in the fourth embodiment. In the fourth embodiment, the restoration area is determined based on the luminance. That is, a predetermined threshold is calculated based on the brightness of the acquired image (step S231),
An area whose luminance is equal to or less than a predetermined threshold is determined as a restoration area (step S232).

Thereafter, in the restoration area, restoration processing is performed in the same manner as in the first embodiment using the deterioration function relating to the optical system (step S233).

As described above, in the fourth embodiment, the restoration area is determined based on the luminance. This allows, for example,
The background can be reliably set as the non-restoration area for the image of the subject photographed with the white background. As a result, it is possible to appropriately prevent ringing around the main subject and emphasizing background noise when the restoration processing is performed on the entire image.

In particular, in the above description, a description has been given using a digital camera. However, when character images are acquired by a scanner and character recognition is performed, appropriate character recognition is realized.

In the above description, an area where the luminance is equal to or lower than the predetermined threshold is determined as the restoration area. However, it is needless to say that an area where the luminance is equal to or higher than the predetermined threshold is determined in accordance with the brightness of the background. It may be determined as a restoration area. Furthermore, when the brightness of the background is known, an area where the luminance is within a predetermined range may be set as the restoration area.

When the background has a predetermined color, such as an identification photograph, an area within the predetermined color range may be determined as a restoration area. That is, by determining the restoration area based on the pixel value (including the luminance) of the pixel, the processing area can be appropriately determined, and an appropriate restoration of the image can be realized.

In the fourth embodiment, similarly to the second embodiment, the restoration area may be modified.

<5. Fifth Preferred Embodiment> FIG. 30 is a view showing a fifth preferred embodiment of the present invention. In the first to fourth embodiments, image restoration is performed by the digital camera 1, but in the fifth embodiment, image restoration is performed by the computer 5. ing. That is, data can be transferred between the digital camera 1 having no image restoration function and the computer 5 using the memory card 91 or the communication cable 92, and the image acquired by the digital camera 1 is restored by the computer 5. It is supposed to.

The restoration processing by the computer 5 is as follows.
Any of the restoration processing in the first to fourth embodiments may be performed. However, in the following description, restoration of image deterioration by the optical system and correction of the restoration area are performed in the same manner as in the second embodiment. Will be described.

The digital camera 1 according to the fifth embodiment has the first configuration except that the image is not restored.
In the following description, the same components as those in the first embodiment are denoted by the same reference numerals. The data output from the digital camera 1 may be output from any output means such as the card slot 14 and the output terminal.
The description will be made on the assumption that data is transferred from the digital camera 1 to the computer 5 via the digital camera 1.

A program for performing a restoration process via a recording medium 8 such as a magnetic disk, an optical disk, or a magneto-optical disk is installed in the computer 5 in advance, and the CPU in the computer 5 follows the program using the RAM as a work area. By performing the processing, the image restoration processing is executed in the computer 5.

FIG. 31 is a schematic diagram showing the structure of data recorded on the memory card 91. In the digital camera 1, an image is acquired as image data by the same method as that of the ordinary digital camera 1, but at the same time, a deterioration function indicating a deterioration characteristic given to the image by the optical system is obtained (or stored in advance). The image data 911
And the deterioration function 912 are combined to form the memory card 91
Is output to.

FIG. 32 is a flow chart showing the flow of operation when taking a picture with the digital camera 1 according to the fifth embodiment.
FIG. 33 is a flowchart showing an operation flow in the computer 5. FIG. 34 is a block diagram showing a functional configuration of the digital camera 1 and the computer 5 related to the restoration processing. FIG. 34 shows only a configuration for recording image data and a deterioration function in the memory card 91 among the functional configurations of the digital camera 1.
With respect to, a card slot 51 for reading data in the memory card 91, a fixed disk 52, a CPU,
Only the restoration unit 505, restoration region determination unit 506, and restoration region correction unit 508, which are functions realized by a RAM or the like, are illustrated. Hereinafter, operations of the digital camera 1 and the computer 5 according to the fifth embodiment will be described with reference to FIGS.

When a photograph is taken by the digital camera 1, similarly to the first embodiment (see FIG. 19), the optical system is controlled by the lens control unit 401 and the aperture control unit 402 (see FIG. 19). 32: Step S111), information on the optical system is obtained as the deterioration information 431 (Step S11).
2). After that, the CCD 32 is exposed (step S113), and a captured image is obtained as image data.

The deterioration function calculator 403 obtains a deterioration function based on the deterioration information 431 relating to the lens unit 2 (step S114), and transfers it to the deterioration function storage 404. Further, similarly to the first embodiment, the deterioration function for the optical low-pass filter 31 is stored in the deterioration function storage unit 404 in advance. On the other hand, the acquired image is subjected to correction processing by the correction unit 44, and the image memory 34
(Accurately, correction processing is performed on the image data in the image memory 34) (step S11).
5).

Thereafter, in the digital camera 1, as shown in FIG. 34, the image data corresponding to the corrected image and the deterioration function are stored in the memory card 91 via the card slot 14.
(Step S116).

When the image data and the deterioration function are stored in the memory card 91, the memory card 91 is inserted into the card slot 51 of the computer 5, and the computer 5 reads the image data and the deterioration function onto the fixed disk 52 and performs a restoration process. Is prepared (FIG. 33: step S121).

Thereafter, the restoration area determination unit 506 determines a restoration area based on the image indicated by the image data, and
5 and the restoration area correction unit 508 repeatedly perform the above-described restoration processing using the deterioration function and the modification of the restoration area (step S122). These operations are the same as the restoration processing in the second embodiment shown in FIG.

When the restoration of the image is completed, the restored image is stored in the fixed disk 52 (step S123).

As described above, in the digital camera 1 according to the fifth embodiment, the deterioration function is output to the outside together with the image data.
Determination of a restoration area and restoration processing using the deterioration function are performed. This eliminates the need for the digital camera 1 to perform a restoration process, so that the digital camera 1 (as compared with the first embodiment,
When an image having a large number of pixels is obtained (when an image having a large number of pixels is acquired), the time from the start of shooting to storage of image data can be reduced.

<6. Modifications> Although an example of restoring an image acquired using the digital camera 1 has been described as an embodiment of the present invention, the present invention is not limited to the above embodiment, and various modifications are possible. Is possible.

For example, in the above-described embodiment, the degradation function relating to the lens unit 2, the degradation function relating to the optical low-pass filter 31, and the degradation function relating to camera shake have been described as degradation functions. May be required (or prepared in advance). Also, in the 3CCD digital camera 1, only the deterioration function relating to the lens unit 2 or the diaphragm 2
As in the case of using only the deterioration function relating to No. 2, only a specific type of deterioration of the image may be restored using only one type of deterioration function.

Further, as described above, the deterioration function does not need to be obtained for all the pixels. Instead, the deterioration function for the representative pixel (that is, the light receiving element) is obtained using the LUT or the like, and the other functions are obtained. The pixel deterioration function may be obtained by interpolation. Further, when the deterioration function is constant for all the pixels as in the case of the deterioration function relating to the optical low-pass filter 31, it is sufficient to prepare one deterioration function in the ROM 42 in advance.

That is, by using at least one type of deterioration function, a specific type of deterioration can be appropriately restored.

In the above embodiment, the calculation of the deterioration function and the restoration of the image have been described as being performed by the CPU, ROM, and RAM in the digital camera 1 and the computer 5, but the lens controller 401 of the digital camera 1 Aperture control unit 402, deterioration function calculation unit 403, restoration unit 405, restoration region determination unit 406, restoration region correction unit 408, and restoration unit 505 and restoration region determination unit 5 of computer 5.
06 and the restoration area correction unit 508 may be constructed by a dedicated electric circuit, or may be constructed only by a part of a dedicated electric circuit.

The program 421 for restoring an image by the digital camera 1 may be installed in the digital camera 1 via a recording medium such as the memory card 91 in advance.

Further, the present invention is not limited to the restoration of an image acquired by the digital camera 1,
It is also used for restoring an image acquired by another imaging device that acquires an image using the light receiving element array, for example, an electron microscope or a film scanner. Of course, the light receiving element array is not limited to the two-dimensional array, but may be a one-dimensional array like a scanner.

Further, the method of determining the restoration area and the method of correcting the restoration area are not limited to the above embodiment, but various methods can be used. For example, the restoration area may be determined based on the distribution or change of the spatial frequency of the acquired image, or a non-restoration area surrounded by the restoration area may be forcibly set as the restoration area.

Further, two types of thresholds for determining the restoration area are obtained, and the two thresholds are used to divide the image into three areas of a restoration area, a half restoration area, and a non-restoration area. In, the average value (or weighted average value) of the pixel values before and after restoration may be updated. Thereby, a clear boundary between the restoration area and the non-restoration area can be eliminated.

[0127]

According to the first to third aspects of the present invention, since an image is restored in a restoration area, an appropriate restoration of an image can be performed.

Further, according to the fourth and fifth aspects of the present invention, since the restoration area is corrected, more appropriate image restoration can be performed.

[Brief description of the drawings]

FIG. 1 is a front view of a digital camera according to a first embodiment.

FIG. 2 is a rear view of the digital camera.

FIG. 3 is a side view of the digital camera.

FIG. 4 is a vertical sectional view near a lens unit.

FIG. 5 is a block diagram illustrating a configuration of a digital camera.

FIG. 6 is a diagram for explaining image deterioration due to a lens unit.

FIG. 7 is a diagram for explaining image deterioration due to a lens unit.

FIG. 8 is a diagram for explaining image deterioration due to a lens unit.

FIG. 9 is a diagram for describing image deterioration due to a lens unit.

FIG. 10 is a diagram for explaining image degradation due to an optical low-pass filter.

FIG. 11 is a diagram for explaining image degradation due to an optical low-pass filter.

FIG. 12 is a flowchart showing the flow of processing in the first image restoration method.

FIG. 13 is a flowchart showing the flow of processing in the second image restoration method.

FIG. 14 is a flowchart showing the flow of processing in a third image restoration method.

FIG. 15 is a diagram illustrating an example of an acquired image.

FIG. 16 is a diagram illustrating an example of a restoration area.

FIG. 17 is a flowchart showing a flow of an operation of the digital camera at the time of shooting.

FIG. 18 is a flowchart showing the flow of a restoration process according to the first embodiment.

FIG. 19 is a block diagram illustrating a functional configuration of a digital camera.

FIG. 20 is a flowchart showing the flow of a restoration process according to the second embodiment.

FIG. 21 is a block diagram illustrating a part of a functional configuration of a digital camera according to a second embodiment.

FIG. 22 is a diagram illustrating an example of an acquired image.

FIG. 23 is a diagram illustrating an example of an image after restoration.

FIG. 24 is a diagram illustrating an example of an image after restoration.

FIG. 25 is a diagram for describing image deterioration due to camera shake.

FIG. 26 is a flowchart showing the flow of a restoration process according to the third embodiment.

FIG. 27 is a block diagram illustrating a part of a functional configuration of a digital camera according to a third embodiment.

FIG. 28 is a diagram illustrating an example of a restoration area.

FIG. 29 is a flowchart showing the flow of a restoration process according to the fourth embodiment.

FIG. 30 is a diagram showing an overall configuration according to a fifth embodiment.

FIG. 31 is a schematic diagram showing a data structure in a memory card.

FIG. 32 is a flowchart showing a flow of operation of the digital camera at the time of shooting.

FIG. 33 is a flowchart showing an operation flow of the computer.

FIG. 34 is a block diagram illustrating a functional configuration of a digital camera and a computer.

[Explanation of symbols]

 Reference Signs List 1 digital camera 5 computer 741,742 restoration area 41 CPU 42 ROM 43 RAM 406,506 restoration area determination unit 912 deterioration function 405,505 restoration unit 408,508 restoration area correction unit 409 determination unit S212 to S216

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Yusuke Nakano 2-3-1-13 Azuchicho, Chuo-ku, Osaka-shi, Osaka F-term in Osaka International Building Minolta Co., Ltd. 5C021 PA17 PA51 PA52 PA58 PA71 PA85 PA92 XA03 XA08 XA66 XA69 5C022 AA13 AB51 AB55 AC42 AC51 AC69 5C076 AA01 AA36 BA07

Claims (5)

[Claims] 1. An image restoration apparatus for restoring image deterioration, comprising: an area determination unit that determines a restoration area in an image based on a difference in pixel value between each pixel and a pixel in the vicinity of each pixel; A restoring unit for restoring the image in the restoration area using at least one deterioration function indicating deterioration characteristics of the image. 2. An image restoration apparatus for restoring image deterioration, comprising: area determination means for determining a restoration area in the image based on at least one deterioration function indicating image deterioration characteristics; A restoring unit for restoring the image in the restoration area using a deterioration function. 3. An image restoration apparatus for restoring image degradation, comprising: area determination means for determining a restoration area in an image based on a pixel value of each pixel; and at least one degradation indicating degradation characteristics of the image. A restoring unit for restoring the image in the restoration area using a function. 4. An image restoration apparatus for restoring image deterioration, comprising: area determination means for determining a restoration area in an image to be restored; restoration means for restoring the image in the restoration area; An image restoration apparatus comprising: a determination unit that determines whether the image is appropriate; and an area correction unit that corrects the restoration area so that the restoration unit restores the image again. 5. An image restoration method for restoring deterioration of an image, comprising: an area determination step of determining a restoration area in an image to be restored; a restoration step of restoring the image in the restoration area; A judging step of judging whether the image is appropriate, and, if it is judged that the image restored in the judging step is inappropriate, correcting the restored area and returning to the restoring step And an image restoring method.