JP2009244944A - Image-recovering apparatus and photographing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image-recovering apparatus capable of recovering a beautiful image regarding a specified object, and to provide a photographing apparatus which has the image-recovering apparatus. <P>SOLUTION: The image-recovering apparatus has an object recognizing unit 15 for recognizing the specified object 18 from an image, a calculation unit 14 for acquiring a point image distribution function in an area within the image, including at least a part of the specified object 18 recognized by the object recognizing unit 15, and a recovery processing unit 2 for processing recovery of the image based on data acquired at the calculation unit 14. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image recovery device and a photographing device.

  A technique has been proposed in which a point spread function (PSF) is calculated from a captured image and image restoration is performed (see Patent Document 1). In the conventional image restoration technique, image restoration is performed on the premise that there is one PSF data for one photographed image.

  However, in practice, there may be a plurality of PSF data for one photographed image. For example, in the case of a photographed image with subject blurring, the PSF is different between the background and the subject, and when a plurality of subjects are moving in different ways, the PSF of each subject is different.

In such a case, the conventional image restoration apparatus performs image restoration on the premise that there is one PSF data for one photographed image. It was not always possible to recover.
Japanese Patent Application Laid-Open No. 11-134481

  The present invention has been made in view of such a situation, and an object of the present invention is to provide an image restoration apparatus capable of neatly restoring an image of a specific subject and an imaging apparatus having the image restoration apparatus.

In order to achieve the above object, an image restoration apparatus according to the present invention provides:
A subject recognition unit (15) for recognizing a specific subject (18, 20, 20a) from the image;
A calculation unit (14) for obtaining a point spread function in a region in the image including at least a part of the specific subject (18, 20, 20a) recognized by the subject recognition unit (15);
A recovery processing unit (2, S7) for recovering the image based on the data obtained by the calculation unit (14);

  According to the image restoration device of the present invention, the PSF is obtained in an area including at least a part of the specific subject (18, 20, 20a) obtained by the subject recognition unit (15), and an image is obtained based on the data. Recovery process. Therefore, it is possible to restore the image of a specific subject in a clean manner.

  The image restoration apparatus may further include a selection unit (2, S5) that selects at least one of the plurality of specific subjects (20a to 20d) recognized by the subject recognition unit (15). At that time, the selection unit (2, S5) may select the specific subject (20a) having the largest area in the image among the plurality of specific subjects (20a to 20d). This is because it can be estimated that the largest specific subject (20a) is the target specific subject.

  The selection unit (2, S5) may select at least one of the plurality of specific subjects (20a to 20d) based on an external input signal. In that case, the operator can select a specific subject.

The calculation unit (14) obtains a point spread function in a region in the image including at least a part of one specific subject (18, 20, 20a) selected by the selection unit (2, S5). ,
In the recovery processing unit (2, S7), the image may be recovered using the obtained point spread function.

In the calculation unit (14), a point spread function is respectively obtained in a region including at least a part of each of the plurality of specific subjects (18, 20, 20a) obtained by the subject recognition unit (15).
In the recovery processing unit (2, S7), the image may be recovered using an average value of a plurality of obtained point spread functions. In that case, a plurality of specific subjects (20a to 20d) are averagely restored with high quality.

  The subject recognition unit (15) may recognize a human face or a part of the face as the specific subject.

  An imaging device according to the present invention includes an imaging unit (3) that acquires an image of a subject, and the image restoration device described above.

  In the above description, in order to explain the present invention in an easy-to-understand manner, the description has been made in association with the reference numerals of the drawings showing the embodiments, but the present invention is not limited to this. The configuration of the embodiment described later may be improved as appropriate, or at least a part of the configuration may be replaced with another component. Further, the configuration requirements that are not particularly limited with respect to the arrangement are not limited to the arrangement disclosed in the embodiment, and can be arranged at a position where the function can be achieved.

Hereinafter, the present invention will be described based on embodiments shown in the drawings.
FIG. 1 is an overall configuration diagram of a camera having an image restoration device according to an embodiment of the present invention
FIG. 2 is a flowchart showing an example of the control of the image restoration apparatus.
FIG. 3 is a detailed flowchart of the face recognition process shown in FIG.
FIG. 4 is a schematic diagram showing an example of a template for face recognition.
FIG. 5A is a diagram illustrating an example of a captured image of a single subject, and FIG. 5B is a diagram illustrating an example of a captured image of a plurality of subjects.
FIG. 6 is a schematic diagram showing the concept of PSF.
FIG. 7 is a flowchart showing details of the PSF calculation process shown in FIG.
FIG. 8 is a schematic diagram of block division in PSF calculation processing.
FIG. 9 is a schematic diagram showing a continuation process of FIG.
FIG. 10 is a schematic diagram showing a continuation process of FIG.
FIG. 11 is a schematic view showing a step subsequent to FIG.

  As shown in FIG. 1, a single-lens reflex camera 1 according to an embodiment of the present invention includes a camera body 1a and a lens barrel 1b, and a CPU 2 as a control unit built in the camera body 1a. Have The image sensor 3 is connected to the CPU 2 via the signal processing circuit 12. The signal processing circuit 12 performs noise processing, A / D conversion, and the like.

  The image pickup device 3 picks up and acquires an image of the subject 18 incident through the zoom lens group 4, the focus lens group 5, and the shake correction lens group 6 mounted in the lens barrel 1b. Yes. That is, the image sensor 3 is a photoelectric conversion element, receives an input signal from the image sensor drive circuit, and outputs image data to the signal processing circuit 12.

  The zoom lens group 4 is mounted so as to be movable in the direction of the optical axis Ax by a zoom group driving mechanism 7 mounted inside the lens barrel 1b. The focus lens group 5 is mounted so as to be movable in the direction of the optical axis Ax by a focus group driving mechanism 8 mounted in the lens barrel 1b.

  The shake correction lens group 6 is mounted so as to be movable in a plane direction perpendicular to the optical axis Ax by a shake correction group drive mechanism 9 mounted inside the lens barrel 1b. The shake correction lens group 6 moves in a plane direction perpendicular to the optical axis Ax in response to a shake operation, thereby providing a shake correction effect.

  These drive mechanisms 7 to 9 are controlled by the CPU 2. A gyro sensor 16 is connected to the CPU 2. The gyro sensor 16 is a sensor that detects the shake of the camera 1 by detecting the angular velocity of the camera 1.

  The CPU 2 is connected to a face recognition calculation unit 15. In the embodiment shown in FIG. 1, the face recognition calculation unit 15 and the PSF calculation unit 14 are displayed separately from the CPU 2, but actually, the CPU 2 itself may process these calculations. The function of the PSF calculation unit 14 will be described later.

  The CPU 2 is connected to a driving mechanism for the shutter 10 disposed on the front surface of the image sensor 3. The CPU 2 controls the opening and closing of the shutter 10. Further, a display device 11 is connected to the CPU 2. The CPU 2 can display the image of the subject 18 acquired by the image sensor 3 on the screen of the display device 11. The display device 11 can also display a menu or the like.

  Since the recording medium 13 is connected to the CPU 2, the CPU 2 can display image data stored in the recording medium 13 on the display device 11. Although it does not specifically limit as the recording medium 13, SD card, CF card, etc. are illustrated. The control by the CPU 2 can be started by operating an operation member 17 such as a power switch or a relief switch mounted on the camera body 1a.

  The CPU 2 performs normal control such as acquisition of a captured image in the camera, and also performs image restoration control as shown in FIG. In the image restoration control shown in FIG. 2, when the image restoration process is started in step S1, a face recognition process (details will be described later) is performed on the selected image in step S2. Next, in step S3, the number of recognized faces is determined. If no face can be detected, the image restoration processing control ends in step S9.

  If one or more faces are detected in step S3, it is determined in step S4 whether the number of recognized faces is one or more. If there are two or more recognized faces, step S4 is performed. In S5, a PSF calculation area to be described later is selected, and in step S6, PSF calculation is performed. If it is determined in step S4 that the number of recognized faces is one, step S5 is skipped and the process goes to step S6 to perform PSF (Point Spread Function) calculation. Details of the PSF calculation will be described later.

  Next, in step S7, image restoration processing (described later) is performed using the PSF calculated in step S6, the restored image is written in the recording medium 13 in step S8, and the processing ends in step S9. .

  In the present embodiment, the face recognition operation in step S2 shown in FIG. 2 is performed based on the flowchart shown in FIG. That is, when step S2 starts, in step S21, the CPU 2 shown in FIG. 1 reads a subject image. The subject image may be read from the image sensor 3 shown in FIG. 1 or the subject image stored in the recording medium 13 may be read.

  Next, in step S22 and step S23 shown in FIG. 3, template matching processing is performed and face detection determination is performed. In steps S22 and S23, for example, the CPU 2 determines whether an image similar to the front template shown in FIG. 4 stored in advance in the face recognition calculation unit 15 shown in FIG. 1 is included in the read subject image. To do.

  FIG. 4 is a view of a human face as seen from the front, and the dotted line portion in FIG. 4 is used as a template. In the case of a template on the premise that a person is facing the front, the template is as shown by a dotted line portion. That is, a part having physical shape features such as eyes, nose, mouth, and ear is a template.

  If a shape that matches the template is obtained, it can be recognized as a face. If template matching is obtained, information such as the number, position, and size of faces can be obtained. Note that the template may be stored in the face recognition calculation unit 15 illustrated in FIG. 1 or may be stored in the recording medium 13.

  If the face of the person viewed from the front can be recognized in step S23, then the process goes to step S24 to specify the number, position, and size of the face, and to send the information to the face recognition calculation unit 15. Are calculated and stored, and the face recognition process ends (step S25).

  Based on the information such as the number, position, and size of the face obtained in step S24, the CPU 2 shown in FIG. 1 performs the processing after step S3 shown in FIG. In steps S3 and S4, as described above, it is determined whether the number of recognized faces is one or more. The case where the number of recognized faces is one is, for example, the case shown in FIG. 5 (A). In such a case, the process goes to step S6 shown in FIG. 2 and shown in FIG. 5 (A). The PSF calculation shown in FIG. 2 is performed on a part or all of the region 20 including the recognized face.

  The case where the number of recognized faces is plural is, for example, the case shown in FIG. In such a case, the process goes to step S5 shown in FIG. 2, and the CPU 2 determines, for example, the area 20a having the largest area among the areas 20a to 20d each including the recognized face shown in FIG. To do. Thereafter, the process goes to step S6 shown in FIG. 2, and the PSF calculation in step S5 shown in FIG. 2 is performed on part or all of the region 20a including the recognized face shown in FIG.

  Next, details of the PSF calculation in step S5 shown in FIG. 2 will be described. First, an outline of PSF will be described.

  As a known method for restoring a deteriorated image including blurring and blurring, there is generally known a method for estimating the blurring and blurring as a point spread function (PSF) and restoring the deteriorated image. (X, y) as coordinates on the image, degraded image including blurring and blurring is g (x, y), ideal image without blurring and blurring is h (x, y), point image spread by blurring and blurring If the information (PSF) is p (x, y), these three expressions satisfy the following relational expression 1.

  Here, * represents a convolution (convolution integration) operation. FIG. 6 is a diagram schematically showing the relational expression 1. Here, when the relational expression 1 is Fourier-transformed into the spatial frequency (u, v) region 2, the expression 1 becomes the following expression 2.

  Here, in addition to the deteriorated image g (x, y), if the PSF function p (x, y) can be known by some method, each spectrum is calculated and The spectrum H (u, v) of the ideal image can be calculated.

  If H (u, v) is subjected to inverse Fourier transform, an ideal image h (x, y) can be calculated. That is, if the PSF function p (x, y) can be obtained from the captured image by any method, the image can be restored to the ideal image.

  Known methods for calculating PSF include a method of detecting the period and direction in which the amplitude spectrum of the Fourier transform of a degraded image becomes zero to detect the magnitude and direction of blur (Japanese Patent Laid-Open No. 2006-221347), a motion vector, and the like. And a method of calculating by using JP-A-2007-6045. Here, a method for calculating the PSF based on the autocorrelation of the image will be described. FIG. 7 shows a flow of PSF calculation performed by the PSF calculation unit 14 (CPU 2) shown in FIG.

  As shown in FIG. 7, in the PSF calculation in step S5, first, in step S51, the CPU 2 shown in FIG. 1 reads the PSF calculation area selected in step S5 shown in FIG. Into. The calculation area is the area 20 when there is one face as shown in FIG. 5 (A), and is the selected area 20a when there are a plurality of faces as shown in FIG. 5 (B). .

  Next, in step S <b> 52 shown in FIG. 7, the CPU 2 extracts the G component in the color RGB signal for the selected region. Since the G component data is larger than the R component and B component data, and no color component is required for PSF calculation, the G component in the RGB component of the image is extracted.

  Next, in step S53 shown in FIG. 7, the image size is reduced by down-sampling in order to reduce the calculation amount. This image is an image g shown in FIG.

  Next, in step S54 shown in FIG. 7, as shown in FIG. 8, the image g is divided into blocks of, for example, 7 × 7 areas. The number of blocks to be divided is not particularly limited.

  Next, in step S55 shown in FIG. 7, blocks having pixels with saturated pixel values are excluded from the calculation. In step S56, the contour is emphasized by Laplacian processing. Further, in step S57, blocks having no texture (pattern) among all the blocks are excluded because blur and blur cannot be detected.

  Next, in step S58, an autocorrelation value is calculated for the remaining blocks that are not excluded. In the example of FIG. 9, four blocks with thick inner lines of the image g remain, and the calculation is performed for each block. The two-dimensional autocorrelation function value Rff is defined by the following equation 4.

  Here, image B is a blocked image (5 × 5 pixels), a and b are the inter-pixel distances in the X and Y directions, N is the length in the X direction of the region where autocorrelation is calculated, and M is Y Indicates the length of the direction. As shown in FIG. 9, the calculation of the autocorrelation value is a value obtained by adding the product of the pixels in the overlapping region while shifting the image B and dividing the result by the area of the region.

  Next, in step S59 shown in FIG. 7, an autocorrelation image is created based on the calculation result of the autocorrelation value calculated in step S58. The autocorrelation value becomes maximum when a = 0 and b = 0, that is, when the pixels are completely overlapped. Normalization is performed based on Rff (0, 0), and an image is created in correspondence with gray scales 0 to 255. FIG. 10 shows an example of the calculated autocorrelation image. In this example, the autocorrelation in the diagonal direction is high in each of the four blocks, and it is estimated that blurring occurs in the diagonal direction.

  Next, in step S60 shown in FIG. 7, an image obtained by averaging the obtained autocorrelation images (an average of four images in this example) is calculated as a PSF estimation result. FIG. 11 shows an example of PSF calculation results. From the PSF calculation result, the length, direction, and blur width of the linear blur are known.

  If the straight blur length, direction, and blur width are known from the PSF calculation result, based on the data, the CPU 2 shown in FIG. 1 performs the image restoration process in step S7 shown in FIG. Thereafter, in step S8, the CPU 2 shown in FIG. 1 writes the recovered image on the recording medium 13, and the process ends in step S9. For example, in FIG. 5A, the recovered image can be viewed as a clear image without blurring the face of the photographed person. For example, in FIG. 5B, among a plurality of photographed people, a particular person's face can be seen as a clear image without blurring.

  The embodiment described above can be variously modified.

  For example, in the above-described embodiment, as shown in FIG. 5B, the PSF is calculated from the recognition area 20a of the face having the largest area. However, the PSF is calculated from any of the areas 20a to 20d, and an image is obtained. Whether to recover may be selected by a person operating the camera 1. For example, the operator may be able to select which region 20a to 20d to select on the touch panel type. Then, similarly to the above-described embodiment, PSF may be calculated from the selected face recognition area, and image restoration may be performed using the PSF.

  Alternatively, the areas 20a to 20d that can be selected by the operator or the areas 20a to 20d that are automatically selected by the face recognition calculation unit 15 shown in FIG. 1 may be plural (including the total number). In that case, the PSF may be calculated in a plurality of selected areas, and the average value may be used to restore the image.

  In addition, as a method for recognizing a specific subject such as a face, instead of using the template described above, or in combination with a method using a template, image feature quantities such as skin color and eye shape are detected and statistically used. For example, a method using statistical analysis or a method using learning typified by a neural network may be used.

  In the above-described embodiment, the front template is used as the face recognition template. Alternatively, a horizontal template or another template may be used. Further, the template may be only a part of the face, and may be a template of a machine such as an animal, a plant, or an automobile other than a human face.

  Furthermore, the image recovery apparatus of the present invention is not limited to a still camera, and may be mounted on a video camera, and may be mounted on an image processing device other than the camera.

FIG. 1 is an overall configuration diagram of a camera having an image restoration apparatus according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of control of the image restoration apparatus. FIG. 3 is a detailed flowchart of the face recognition process shown in FIG. FIG. 4 is a schematic view showing an example of a template for face recognition. FIG. 5A is a diagram illustrating an example of a captured image of a single subject, and FIG. 5B is a diagram illustrating an example of a captured image of a plurality of subjects. FIG. 6 is a schematic diagram showing the concept of PSF. FIG. 7 is a flowchart showing details of the PSF calculation process shown in FIG. FIG. 8 is a schematic diagram of block division in the PSF calculation process. FIG. 9 is a schematic view showing a step subsequent to FIG. FIG. 10 is a schematic view showing a step subsequent to FIG. FIG. 11 is a schematic view showing a step subsequent to FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Camera 1a ... Camera body 1b ... Lens barrel 2 ... CPU
DESCRIPTION OF SYMBOLS 3 ... Imaging device 13 ... Storage medium 14 ... PSF calculating part 15 ... Face recognition calculating part 18 ... Subject 20, 20a-20d ... Area

Claims (8)

A subject recognition unit for recognizing a specific subject from an image;
A calculation unit for obtaining a point spread function in an area in the image including at least a part of the specific subject recognized by the subject recognition unit;
An image recovery apparatus comprising: a recovery processing unit that recovers the image based on data obtained by the arithmetic unit.   The image restoration apparatus according to claim 1, further comprising a selection unit that selects at least one of the plurality of specific subjects recognized by the subject recognition unit.   The image restoration device according to claim 2, wherein the selection unit selects a specific subject having the largest area in the image among a plurality of the specific subjects.   The image restoration device according to claim 2, wherein the selection unit selects at least one of the plurality of specific subjects based on an input signal from the outside. The calculation unit obtains a point spread function in a region in the image including at least a part of one specific subject selected by the selection unit,
The image recovery apparatus according to claim 2, wherein the recovery processing unit recovers the image using the obtained point spread function. The calculation unit obtains a point spread function in an area including at least a part of each of the plurality of specific subjects recognized by the subject recognition unit,
The image recovery apparatus according to claim 1, wherein the recovery processing unit recovers the image using an average value of a plurality of obtained point spread functions.   The image recovery device according to claim 1, wherein the subject recognition unit recognizes a human face or a part of the face as the specific subject. An imaging unit for acquiring an image of a subject;
An imaging device comprising the image recovery device according to claim 1.

Images