JP2007226520A - Image processing program and computer readable recording medium with the same recorded thereon, and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a means for performing highly accurate sharpening processing at high speed by using a common algorithm irrespective of whether a blur in an image is caused by out of focus or a camera-shake blur. <P>SOLUTION: An image processing program 6 is characterized in that a total quantity of light which is emitted from a subject H and received by a prescribed imaging device (x, y) positioned on the imaging device surface S of a camera is approximated to be equal to a total sum of a quantity of light received by the prescribed imaging device (x, y) out of lights which are emitted from the prescribed part of the subject H and reach the imaging device surface S and a total quantity of light received by other imaging devices except for the prescribed imaging device (x, y) among the imaging devices included in a diffusion range of lights emitted from the prescribed part of the subject H, and thus a computer executes processing for generating a new image on the basis of original image data. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing program and an image processing apparatus for correcting a defocused image or a camera shake image into a sharp image, and in particular, it is possible to correct both a defocused image and a camera shake with high accuracy with a short processing time using the same algorithm. The present invention relates to an image processing program, a computer-readable recording medium on which the image processing program is recorded, and an image processing apparatus.

  When a subject is photographed using a digital camera or the like, the photographic image may be blurred. Examples of the blur include a blur caused by a shift in focal length, a blur caused by camera shake of the photographer, and a subject blur caused by movement of the subject. As a method for correcting such a blurred image into a sharpened image, that is, a sharp image, processing using a so-called unsharp mask or a Laplacian filter has been conventionally used (see, for example, Patent Document 1 and Patent Document 2). The process using the unsharp mask is to correct a blur by generating a blur image in which the original image in which the blur has occurred is further blurred and adding the difference between the blur image and the original image to the original image. On the other hand, the processing by the Laplacian filter is to correct the blur by finding the boundary where the light and darkness or the hue changes rapidly by examining the second derivative value of the original image and adding it to the original image. .

JP 2006-031195 A JP 2006-011619 A

  However, conventional image processing programs and image processing apparatuses that perform sharpening processing using an unsharp mask or a Laplacian filter cannot comprehensively sharpen both out-of-focus and hand-blurring. Therefore, it is necessary to use an unsharp mask, Laplacian filter, etc. appropriately, and the processing becomes complicated, and effective sharpening can be performed for images in which both out-of-focus and blurring occur. There was a problem that was difficult.

  The present invention has been made in view of such a problem, and sharpening at high speed and high accuracy using a common algorithm regardless of whether blurring generated in an image is out of focus or hand blurring. Provide a means for processing.

  The image processing program according to claim 1 for achieving the above object is an image processing program for causing a computer to perform sharpening processing on an original image in which out-of-focus or hand-blurring occurs. The total amount of light received by the predetermined image sensor located above is the amount of light emitted from the predetermined portion of the subject and reaching the image sensor surface, and the amount of light received by the predetermined image sensor and the predetermined amount of the subject. Approximating to be equal to the sum of the total amount of light received by other imaging elements other than the predetermined imaging element among the imaging elements included in the diffusion range of the light emitted from the part, from the predetermined part of the subject Generate a new image based on the original image data by calculating the amount of light emitted and reaching the image sensor surface It is intended to execute that process to the computer.

  A computer-readable recording medium according to a second aspect stores the image processing program according to the first aspect.

  The image processing apparatus according to claim 3 is an image processing apparatus that performs sharpening processing on an original image in which out-of-focus or hand-blurring occurs, and an original image input means for inputting image data of the original image, The total amount of light received by the predetermined image sensor positioned on the image sensor surface of the camera is received by the predetermined image sensor out of the light emitted from the predetermined portion of the subject and reaching the image sensor surface. Approximate to be equal to the sum of the amount of light and the total amount of light received by other imaging elements other than the predetermined imaging element among imaging elements included in the diffusion range of light emitted from the predetermined part of the subject Approximate ideal image calculation means for calculating the amount of light emitted from a predetermined portion of the subject and reaching the image sensor surface, and the calculation result of the approximate ideal image calculation means And new image generation means for generating a new image based on, but with a.

  According to the image processing program of the first aspect of the present invention, an ideal image of a subject in the case where there is no shift in focal length or camera shake at all without approximating the simultaneous equations including enormous variables can be approximated. By calculating the above, sharpening processing can be performed at high speed and with high accuracy. Further, by appropriately adjusting the light diffusivity, an effective sharpening process can be performed even for an image in which both out-of-focus and hand-blurring occur.

  According to the computer-readable recording medium of the second aspect of the present invention, the image processing program according to the first aspect can be carried and used at a desired place.

  According to the image processing apparatus of the third aspect of the present invention, an ideal image of a subject in the case where there is no shift in focal length or camera shake of a photographer without solving simultaneous equations including a large number of variables. By calculating approximately, sharpening processing with high speed and high accuracy is possible. Further, by appropriately adjusting the light diffusivity, an effective sharpening process can be performed even for an image in which both out-of-focus and hand-blurring occur.

  Hereinafter, an image processing apparatus and an image processing program according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 includes a media reader (original image input means) 2 for inputting image data of an original image to be corrected, and a computer 3 that generates a new image sharpened based on the input original image. And a photographic printer 4 for printing out the generated new image.

  The media reader 2 is for reading original image data stored in various storage media such as a memory card and a CD and inputting them to the computer 3. Of course, by using a conventionally known image scanner or the like instead of the media reader 2 of the present embodiment as the original image input means according to the present invention, it is also possible to read image data from a printed photographic image.

  The computer 3 includes an image memory 5 that stores original image data input from the media reader 2, a hard disk 7 that stores an image processing program 6 for sharpening the image, and an image processing program that is read from the hard disk 7. In order to display a RAM (Random Access Memory) 8 that temporarily stores 6, a CPU (Central Processing Unit) 9 that sharpens the original image data according to the image processing program 6, and an original image and an image after the sharpening process The display unit 10 and an operation unit 11 configured with a mouse, a keyboard, and the like are connected via a system bus 12.

  The algorithm of the image processing program 6 according to the present invention will be described below. The image processing program 6 is created using various programming languages, and approximately calculates an ideal image when no focus shift or camera shake occurs based on original image data.

  Prior to the description of the algorithm of the image processing program 6, first, a mechanism of occurrence of out-of-focus or camera shake in a digital camera will be described. Generally, an element that reacts electrically when receiving light (hereinafter referred to as an “imaging element”) is provided inside a digital camera, and the surface of an object to be photographed (hereinafter referred to as an “subject”). The photons that are the emitted light particles reach the image sensor inside the camera while the shutter of the digital camera is open, so that an image corresponding to the subject is formed. This makes it possible to shoot a subject with a digital camera. In the present invention, as shown in FIG. 2, the surface of the subject H is divided into a grid, and a predetermined portion of the subject H positioned in each grid is represented as a subject (x, y), and the subject (x, y) The number of photons that have reached the image pickup device surface S of the digital camera, that is, the amount of light among all the photons emitted from the portion is expressed as output light quantity_subject (x, y). On the other hand, the image pickup device surface S of the digital camera is formed by arranging the same number of image pickup devices as the number of divisions of the subject, and a predetermined portion of the image pickup device is represented as an image pickup device (x, y). The number of photons received by the portion y), that is, the amount of light, is expressed as input light quantity_imaging element (x, y).

Here, when ideal conditions such as the following formula (1) are completely satisfied, that is, all photons emitted from the subject (x, y) and reaching the image pickup device surface S of the digital camera are all unique. When there is no photon received by one image sensor (x, y) and other image sensors, the same image as the subject H is formed on the image sensor surface S. It is possible to take a picture of out-of-focus and hand-blurred zero with no hand-blurring.

  However, in actual photography, the lens performance limit, the photographer moves the camera while the shutter is open (camera shake), the subject H is out of focus (out of focus), etc. Due to circumstances, Equation (1) does not hold completely, and some of the photons emitted from the subject (x, y) and reaching the image sensor surface S of the digital camera are other than the image sensor (x, y). It is received by the image sensor. For example, when camera shake occurs, as shown in FIG. 3, some of the photons that have reached the image sensor are caused by the image sensor located in the direction opposite to the camera shake direction with respect to the image sensor (x, y). As a result, camera shake occurs. In addition, when the focal length does not match the subject H, as shown in FIG. 4, some of the photons that have reached the image sensor are diffused in all directions and are captured by the image sensor around the image sensor (x, y). It is received and this causes blurring.

In this way, in the image processing program 6 according to the present invention, the diffusivity of photons is appropriately set in order to obtain an ideal image without blur from the original image in which out-of-focus / blurred blur occurs. Table 1 is a table showing an example of the photon diffusion rate. In this case, about 40% of the photons that have reached the imaging element surface S from the subject (x, y) are received by the imaging element (x, y), and about 10% are the imaging element (x, y−1), The image pickup element (x, y + 1), the image pickup element (x-1, y), and the image pickup element (x + 1, y) are received by the image pickup element (x-1, y-1) and the image pickup element (x −1, y + 1), the image sensor (x + 1, y−1), and the image sensor (x + 1, y + 1). That is, the range in which photons emitted from the subject (x, y) are diffused is limited to the range of a total of nine image sensors adjacent to the image sensor (x, y) so as to surround it. This means that it does not diffuse into the range of the outer image sensor.

When the photon diffusivity is set as shown in Table 1, as shown in FIG. 5, the total number of photons received by the image sensor (x, y) surrounds the subject (x, y). Thus, the total of photons emitted from a total of nine adjacent object grids is obtained, and the following equation (2) is established.

  Here, since the input light quantity_imaging element (x, y) in Expression (2) is a known value obtained for all x and y by referring to the original image data input from the media reader 2, Based on this known input light amount_output light amount that is the total number of photons that have reached the image sensor surface S of the digital camera among all photons emitted from the subject (x, y) based on the image sensor (x, y) By obtaining the subject (x, y) for all x and y, an accurate image of the photographed subject H can be obtained. That is, an accurate image of the subject H without blur can be known based on the original image with blur.

In this image processing program 6, even when the number of divisions of the subject H and the image sensor surface S is increased, the output light quantity for all x and y is not solved in Equation (2) without solving simultaneous equations including enormous variables. _Output light quantity_Subject (x-1, y) to input light quantity_Image sensor (x-1, y) and Output light quantity_Subject to enable approximate calculation of subject (x, y) (X + 1, y) is input light quantity_imaging element (x + 1, y), output light quantity_subject (x, y-1) is input light quantity_imaging element (x, y-1), output light quantity_subject (x , Y + 1) to input light quantity_imaging element (x, y + 1), output light quantity_subject (x-1, y-1) to input light quantity_imaging element (x-1, y-1), and output light quantity_subject. (X + 1, y-1) to input light quantity_imaging device (x + 1, y-1), output light quantity_subject ( −1, y + 1) is replaced with input light quantity_imaging element (x−1, y + 1), and output light quantity_subject (x + 1, y + 1) is replaced with input light quantity_image sensor (x + 1, y + 1). Formula (3) is used.

Here, formula (3) can be rewritten as the following formula (5) by newly introducing a value of the total amount of light and surroundings defined as the following formula (4).

ここで、前述のように、入力光量＿撮像素子（ｘ，ｙ）と光総量・周囲分とは、共に入力された原画像データから得られる既知の値である。従って、前記ＣＰＵ８は、この式（６）を用いて全てのｘとｙについて出力光量＿被写体（ｘ，ｙ）を算出し、これに応じた画像を新たに生成する。このように、ボケている原画像を修正するのではなく、ボケている原画像データに基づいてボケがない状態での被写体Ｈの理想像を近似的に算出した上で、これに応じた新たな画像を生成することとしたので、膨大な数の変数を含んだ連立方程式を解く必要がない分、処理時間を短縮することができ、且つ、高い精度でボケを修正することができる。 Then, the following equation (6) can be derived by modifying the equation (5).

Here, as described above, the input light amount_imaging device (x, y) and the total light amount / peripheral amount are known values obtained from the input original image data. Therefore, the CPU 8 calculates the output light quantity_subject (x, y) for all x and y using this equation (6), and newly generates an image corresponding to this. In this way, instead of correcting the blurred original image, the ideal image of the subject H in the absence of blur is approximately calculated based on the blurred original image data, and a new image corresponding to this is calculated. Therefore, since it is not necessary to solve simultaneous equations including a large number of variables, the processing time can be shortened and blurring can be corrected with high accuracy.

Note that the diffusion rate of photons is not limited to that shown in Table 1 and can be changed to an arbitrary setting. By changing the setting of the diffusion rate, the degree of blur correction is adjusted to be increased or decreased. be able to. Further, in this embodiment, photons emitted from the subject (x, y) are diffused to a range of a total of nine image sensors adjacent to the image sensor (x, y) so as to surround the image sensor (x, y). Although the diffusivity is set not to diffuse in the range of the image sensor, the present invention is not limited to this. For example, the image sensor adjacent to the image sensor (x, y), that is, the image sensor (x, y) is 5 It is also possible to set the diffusivity by assuming that photons diffuse in a range of x5 = 25 total or 7 × 7 = 49 total image sensors. Tables 2 and 3 below show examples of setting the spreading factor in this case. Table 2 shows a setting example when the degree of out-of-focus correction is increased. Table 3 is corrected from Table 2. The example of a setting when the degree of is weakened is shown.

  It is also possible to set the photon diffusion rate on the assumption that the aspect of photon diffusion follows a two-dimensional normal distribution. In this case, it is possible to freely adjust the degree of correction of defocusing by simply giving a single numerical value called standard deviation as a parameter, so that there is an advantage that the operation becomes extremely simple. Note that the image processing algorithm in this case is the same as described above in that it is performed using Expression (6), and the description thereof is omitted here.

この場合、前記式（５）と同様に下記の式（７）が成立し、これを変形することで式（８）に示すように出力光量＿被写体（ｘ，ｙ）を算出することができる。これにより、手ブレボケした原画像データに基づいて手ブレがない状態での被写体Ｈの理想像を近似的に算出した上で、これに応じた新たな画像を生成することにより、手ブレボケを修正することができる。

On the other hand, the present image processing program 6 can also correct camera shake by setting the photon diffusivity to a setting different from that for correcting blur. Table 4 is a setting example of the photon diffusivity in the case of correcting camera shake blur, and in addition to the image sensor (x, y), the image sensor (x-1, y) adjacent in the direction opposite to the camera shake direction. In addition, the image sensor (x-2, y) adjacent thereto receives the photons.

In this case, the following equation (7) is established as in the equation (5), and the output light quantity_subject (x, y) can be calculated by modifying this as shown in the equation (8). . As a result, the ideal image of the subject H in the absence of camera shake is approximately calculated based on the original image data that has been shaken, and a new image is generated accordingly, thereby correcting camera shake. can do.

  In this way, by making it possible to correct both out-of-focus and hand-blurring using the same algorithm, there is no need to use different correction algorithms depending on whether out-of-focus or out-of-focus blur occurs in the image, simplifying processing. Is done. In addition, there is an advantage that an effective sharpening process can be performed even for an image in which both out-of-focus and hand-blurring occur.

In addition, in the case of an image photographed by a recent digital camera in which the number of pixels is increasing and the photon emitted from the subject H is assumed to diffuse over a wide range of the image sensor, the application of the present invention is applied. In this case, the following applied method can be used. That is, a mesh having a predetermined size including a plurality of pixels is set on the original image to be corrected. The size of the mesh is set so that the number of meshes does not exceed 1000 in the horizontal and vertical directions of the original image. Then, an ideal image of the subject H is approximately calculated by using Equation (9) obtained by changing Equation (6) under the assumption that the diffusivity of photons is set as shown in Table 2. Note that {average in mesh_input light quantity_imaging element (m, n)} in equation (9) is the mesh (m, n) positioned in the mth in the horizontal direction and the nth in the vertical direction in the original image. ), {Input light quantity_imaging element (x, y)} is examined for all imaging elements (x, y) existing therein, and an average value thereof is calculated.

  The present invention can be applied to an image processing program and an image processing apparatus that correct a out-of-focus image or a camera shake image into a sharp image.

1 is a block diagram showing a configuration of an image processing apparatus 1 according to an embodiment of the present invention. FIG. 4 is an explanatory diagram for explaining a state in which photons emitted from a subject H reach an image sensor surface S. Explanatory drawing for demonstrating generation | occurrence | production of camera shake blur. Explanatory drawing for demonstrating out-of-focus. Explanatory drawing for demonstrating the total number of photons which an image pick-up element (x, y) receives.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Media reader 3 Computer 6 Image processing program 9 CPU
H Subject S Image sensor surface

Claims (3)

In an image processing program that causes a computer to perform sharpening processing on an original image in which out-of-focus or blurring occurs,
The total amount of light emitted from the subject and received by the predetermined image sensor located on the image sensor surface of the camera is the predetermined image sensor out of the light emitted from the predetermined portion of the subject and reaching the image sensor surface. And the total amount of light received by other imaging elements other than the predetermined imaging element out of the imaging elements included in the diffusion range of light emitted from the predetermined part of the subject. Approximating and calculating the amount of light emitted from a predetermined portion of the subject and reaching the image sensor surface, thereby causing a computer to execute a process of generating a new image based on original image data Image processing program.   A computer-readable recording medium on which the image processing program according to claim 1 is recorded. In an image processing apparatus that performs sharpening processing on an original image in which defocusing or blurring has occurred,
The total amount of light emitted from the subject and input from the subject and received by the predetermined image sensor located on the image sensor surface of the camera is emitted from a predetermined portion of the subject. The amount of light received by the predetermined image sensor out of the light reaching the surface of the image sensor and the image sensor included in the diffusion range of light emitted from the predetermined portion of the subject are excluded from the predetermined image sensor An approximate ideal image calculating means for calculating an amount of light emitted from a predetermined portion of the subject and reaching the surface of the image sensor, approximating it being equal to a total amount of light received by another image sensor, and the approximation An image processing apparatus comprising: a new image generation unit that generates a new image based on a calculation result of the target ideal image calculation unit.

Images