JP4629537B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  Conventionally, a photographed image photographed by a camera using an imaging device such as a CCD (Charge Coupled Devices) in the imaging unit is a camera shake during photographing, various aberrations of the photographing optical system, or a lens constituting the photographing optical system. It is known that when there is a distortion or the like, this is a factor and the captured image deteriorates.

  For such a deteriorated captured image, with regard to camera shake at the time of shooting, a change in the optical axis of the camera's shooting optical system is detected by an angular acceleration sensor or the like, and the blurred state at the time of shooting is represented from the detected angular velocity or the like. A method is known in which a transfer function is acquired, the acquired transfer function is inversely transformed with respect to a captured image, and a restoration process is performed on an image without deterioration (see Patent Document 1).

Japanese Patent Laid-Open No. 11-24122 (see abstract)

  By the way, in shooting, an object that moves in one direction, for example, a horizontal direction (that is, a horizontal direction), a vertical direction that is perpendicular to the camera field, or a direction that intersects these directions is taken. There is a case where so-called panning is performed in which shooting is performed by following the shooting direction of the camera. According to this panning shot, for example, when shooting a car that runs in the left-right direction, if the shooting is performed by following the shooting direction of the camera to the movement of the car, the car is shot as a blur-free image. However, the scenery that is the background of the car is an image that is blurred (flowed) along with the change in the shooting direction of the camera. However, in this panning shot, there is a case in which a camera shake occurs in a direction intersecting with the direction in which the panning shot is performed, thereby taking a picture. In other words, if the camera shakes in the up-down direction when panning a car that moves in the left-right direction as in the above example, the car is shot in the up-down direction.

  When image restoration processing is performed on an image shot in a direction that intersects with the direction of the panning shot, not only the blurring direction but also the panning shot was intentionally blurred. There is also a problem that the image restoration process is performed for the (flowed) direction, and the intended image cannot be obtained.

  Therefore, the present invention provides an image processing apparatus that can restore an intended image by panning without performing image restoration processing in the direction of panning for a shot image that has been shot. Objective.

  In order to solve the above-described problem, the image processing apparatus of the present invention is an image processing apparatus having a processing unit that processes an image. Does the processing unit perform image restoration processing for a panning image on captured image data? The determination process for determining whether or not to perform the image restoration process for the normal captured image, and in this determination process, if it is determined that the image restoration process for the panning image is to be performed, Image restoration processing for a panning image without performing restoration processing is performed.

  According to the present invention, it is possible to perform appropriate image processing for a shot image that has been shot. That is, the processing unit determines whether to perform the image restoration process for the panning image or the image restoration process for the normal shooting image, and performs the image restoration process for the panning image. If it is determined, since the image restoration process for the panning image without performing the image restoration process is performed for the direction of the panning shot, the intended image by the panning shot can be restored.

  Further, in addition to the above-described invention, the image restoration process for a panning shot image may be a panning shot that does not have change factor information about the direction of panning shot from change factor information data that causes image change. Using the panning change factor information data generation process to generate change factor information data and the panning change factor information data, comparison image data is generated from arbitrary image data and compared with the captured image data to be processed The image data is compared, and the obtained difference data is used to generate restored image data. The restored image data is used in place of the arbitrary image data, and the same processing is repeated to obtain the original image. An original image restoration image data generation process for generating original image restoration image data that approximates to is performed. When this configuration is adopted, since the comparison image data is created from the restored image data and the comparison image data and the captured image data are compared, the original image restored image data is generated. Image restoration processing that is resistant to noise and the like can be performed.

According to another aspect of the invention, panning is performed in an image processing apparatus that performs image restoration processing using change factor information data that is a factor of image change detected by a detection unit that detects movement of the image processing apparatus. In such a case, change factor information data about the direction of panning shot should not contribute to the change factor information data, and change factor information data that does not contribute to the direction of panning shot should be used. The image data for comparison is generated from the data of an arbitrary image, the captured image data to be processed is compared with the image data for comparison, and the restored image data is generated using the obtained difference data. By using the restored image data instead of the above-mentioned arbitrary image data and repeating the same process, the original image restored image data that generates the original image restored image data that approximates the original image is generated. And to perform the data generation process. According to the present invention, since the image restoration process is performed using the change factor information data that does not have the change factor information data about the direction of the panning shot, the image restoration process is not performed in the direction of the panning shot, and the intention is based on the panning shot. You can restore the image.

  According to the present invention, it is possible to restore the intended image by the panning without performing the image restoration process in the direction of the panning for the shot image that has been shot.

(First embodiment)
Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to the drawings. The image processing apparatus 1 is a so-called consumer digital camera that uses a CCD as an imaging unit. However, the image processing apparatus 1 is a surveillance camera, a television camera, and an endoscope camera that use an imaging element such as a CCD as the imaging unit. The present invention can also be applied to devices other than cameras, such as cameras for other uses, image diagnostic apparatuses for microscopes, binoculars, and NMR imaging.

  The image processing apparatus 1 includes an imaging unit 2 that captures a subject such as a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. The image processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4, an angular velocity sensor, and the like, and changes factor information that causes changes such as image degradation. It has a detection unit 6 for detecting, and a factor information storage unit 7 for storing known change factor information that causes image degradation and the like.

  The imaging unit 2 includes a photographing optical system having a lens and an imaging element such as a CCD or C-MOS (Complementary Metal Oxide Semiconductor) that converts light passing through the lens into an electrical signal. The control system unit 3 controls each unit in the image processing apparatus 1 such as the imaging unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is composed of an image processing processor, and is composed of hardware such as ASIC (Application Specific Integrated Circuit). The processing unit 4 includes a recording unit (not shown), and the recording unit has an arbitrary image as a base when generating image data having a high frequency component of complex space in processing image data, which will be described later, or comparison image data. Image data is saved. The processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 includes a semiconductor memory. However, a magnetic recording unit such as a hard disk drive, an optical recording unit using a DVD (Digital Versatile Disk), or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the image processing apparatus 1. is there. By the way, camera shake at the time of shooting with a camera may cause movement in each direction of the X direction, Y direction, and Z direction, and rotation about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as aberrations of the photographing optical system. In this embodiment, the factor information storage unit 7 stores information on aberrations of the photographing optical system and lens distortion. However, when correcting image degradation due to camera shake, which will be described later, such information is stored. Is not used.

  Next, a processing method of the processing unit 4 of the image processing apparatus 1 configured as described above will be described with reference to FIG.

  In FIG. 3, “Img ′” is captured image data that is data of an image formed on the imaging surface of the imaging unit 2. This photographed image data “Img ′” is stored in the recording unit 5 when a photographing operation is performed, for example, by turning on a shutter button (not shown) of the image processing apparatus 1 that is a digital camera. This photographed image data “Img ′” is an image that blurs along the direction in which the camera is moved when camera shake occurs or when panning is performed. Note that the image data imaged on the imaging unit 2 when it is assumed that the image is taken without blurring due to camera shake or panning during the shooting operation is referred to as original image data “Img”.

  “G” is change factor information (deterioration) that causes the original image data “Img” to be changed to the photographed image data “Img ′” by the movement of the camera at the time of the photographing operation such as camera shake or panning during the photographing operation. Factor information (point spread function) data, which is detected by the detection unit 6. It can be said that the photographed image data “Img ′” and the original image data “Img” have a relationship of “Img ′” = “Img” × “G”.

  “G1” is panning change factor information data, and does not have change factor information on the direction of panning for the change factor information data “G” due to camera movement during a shooting operation. “Io” is arbitrary image data stored in advance in the recording unit of the processing unit 4. “Io ′” is comparison image data in which the image data “Io” is changed by the change factor information data “G” or the panning change factor information data “G1”. “Δ” is difference data between the captured image data “Img ′” and the comparison image data “Io ′”. “K” is a distribution ratio based on the change factor information data “G” or the panning change factor information data “G1”.

“Io + n” is restored image data generated by distributing the difference data “δ” to the image data “Io” based on the distribution ratio “k”. The difference data “δ” may be a simple difference between corresponding pixels, but in general, the difference data “δ” differs depending on the change factor information data “G” or the panning change factor information data “G1”. It appears in.
δ = f (Img ′, Img, G (or G1))

  A processing routine of the processing unit 4 will be described.

  First, the photographed image data “Img ′” stored in the processing unit 4 is called (step S101). Next, it is determined whether to perform the image restoration process for the panning image or the image restoration process for the normal photographed image with respect to the photographed image data “Img ′” (step S102). As described above, the panning image refers to a photographed image when photographing is performed with the photographing direction of the camera following the movement of the subject. In addition, the normal photographed image is a photographed image taken with the intention of the person who performs the photographing operation intending to perform photographing while keeping the photographing direction of the camera constant. If it is determined in this determination (step S102) that the captured image data “Img ′” is an image obtained by panning, panning change factor information data “G1” is generated (step S103). Specifically, in steps S102 and S103, the following processing is performed.

  First, the determination in step S102 is made based on, for example, the operation position of a selection switch (not shown) (hereinafter referred to as a process selection SW) that selects the content of image processing provided in the image processing apparatus 1. Specifically, the processing operation SW is performed by selecting a position (hereinafter referred to as “H position”) when the captured image data “Img ′” is a panning shot in the left-right (horizontal) direction, and the vertical direction. The position to be selected when the panning is performed (hereinafter referred to as V position) and the position to be selected when the captured image “Img ′” is a normal captured image captured without performing the panning ( 3 positions (hereinafter referred to as “off positions”) can be selected.

  The operator of the image processing apparatus 1 performs the restoration process of the photographed image data “Img ′”. Whether the photographed image data “Img ′” has been subjected to panning in the left-right (horizontal) direction or in the up-down direction. It is determined whether the panning has been performed or the panning has been performed without performing the panning, and the processing selection SW is operated to an appropriate position.

  In step S102, when the process selection SW is operated to the H position or the V position, it is determined that the photographed image data “Img ′” is the photographed image data of the panning shot, and the process of step S103 is performed. Then, image restoration processing for the panning image is performed. On the other hand, if the process selection SW is operated to the off position in step S102, it is determined that the captured image data “Img ′” is captured image data of a normal captured image, and the process of step S203 is performed. Then, the image restoration process for the normal photographed image is performed.

  In the present embodiment, the image restoration process for the panning image and the image restoration process for the normal photographed image are the image restoration process for the normal photographed image and the image restoration process for the image blur in the vertical direction and the horizontal direction. In contrast, the image restoration process for the panning image is the same as the image restoration process for the normal photographed image except that the restoration process for the direction in which the panning is performed is not performed. Therefore, first, a method of image restoration processing for a normal captured image will be described.

  First, arbitrary image data “Io” stored in advance in the recording unit of the processing unit 4 is called (step S203). The initial image data “Io” is arbitrary, and any data can be adopted. For example, using the photographed image data “Img ′”, the processing may be started with “Io” = “Img ′”. Also, image data such as black solid, white solid, gray solid, and checkered pattern may be used.

  In step S204, comparison image data “Io ′” is obtained from the image data “Io” using the change factor information data “G”. It can be said that the comparison image data “Io ′” is image data in which the initial image data “Io” is changed (deteriorated) by the change factor information data “G”.

  Next, the photographed image data “Img ′” is compared with the comparison image data “Io ′”, and difference data “δ” is calculated (step S205).

  In step S206, it is determined whether or not the difference data “δ” is equal to or greater than a predetermined value (for example, 5 or less in absolute value). If the difference data is equal to or greater than the predetermined value, the restored image data “Io + n” is generated. Is performed (step S207). That is, the difference data “δ” is distributed to the initial image data “Io” based on the change factor information data “G” to generate restored image data “Io + n”.

  Then, the restored image data “Io + n” is used as arbitrary initial image data (= “Io”) in step S204, and step S204 is executed. Thereafter, S205, S206, and S207 are repeated, and new restored image data “ Io + n "is generated.

  If the difference data “δ” is smaller than the predetermined value in step S206, the process ends (step S208). Then, the restored image data “Io + n” at the time when the processing is completed is recorded in the recording unit 5 as original image restored image data that approximates the original image data “Img”. Note that the initial image data Io and the change factor information data “G” may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary.

  The concept of the above processing method is summarized as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible as described in Patent Document 1, but it is difficult as a real problem. That is, in Patent Document 1, a change in the optical axis of a camera is detected by an angular acceleration sensor, a transfer function representing a blurring state at the time of shooting is acquired from the detected angular velocity and the like, and the acquired number of transfer times is obtained for a shot image. A method of performing an inverse transformation and restoring an image is employed. However, although it is theoretically possible to perform image restoration by inverse transformation of the acquired transfer function, as a practical problem, image restoration is difficult for the following two reasons.

  First, the transfer function to be acquired is very weak against noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. For this reason, the restored image obtained by the inverse transformation is far from an image taken with no camera shake, and cannot be used in practice. Second, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. In practice, there is a high risk of being unable to solve. On the other hand, if the problem is solved as an optimization problem as in the present embodiment, the above problem does not occur.

Solving as an optimization problem assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The solution is converged by iteratively processing while updating the input so that the outputs are the same.

  In other words, as shown in FIGS. 4A and 4B, if the comparison image data “Io ′ (Io + n ′)” approximate to the captured image data “Img ′” can be generated, The initial image data “Io” or the restored image data “Io + n” serving as the original data is approximate to the original image data “Img” that is the original of the photographed image data “Img ′”.

  In this embodiment, the angular velocity detection sensor detects the angular velocity every 5 μsec. In addition, in this embodiment, the value serving as a determination criterion for the difference data “δ” is “6” when each data is represented by 8 bits (0 to 255). That is, when it is smaller than “6”, that is, when it is equal to or smaller than “5”, the processing is terminated. In addition, the raw shake data detected by the angular velocity detection sensor does not correspond to the actual shake when the sensor itself is insufficiently calibrated. Therefore, in order to cope with actual blurring, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

  Next, details of the processing method shown in FIGS. 3 and 4 will be described based on FIGS. 5, 6, 7, 8, 9, 10, 11, and 12.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how the energy is dispersed during the exposure time, so that it is possible to create an image without blur from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. The pixels are set to n-1, n, n + 1, n + 2, n + 3,... In order from the left, and attention is paid to a certain pixel n. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 6 is the original image data “Img”. Each data is represented by 8 bits (0 to 255).

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the nth pixel, 30% of time is shifted to the n + 1th pixel, and 20% of time is shifted to the n + 2th pixel. . The way of energy dispersion is as shown in the table of FIG. This is the change factor information data “G”.

  Since blurring is uniform for all pixels, if there is no upper blurring (vertical blurring), the blurring situation is as shown in the table of FIG. In FIG. 8, the data shown as “imaging result” is the original image data “Img”, and the data shown as “blurred image” is taken image data “Img ′”. Specifically, for example, “120” of the pixel “n−3” is in accordance with the distribution ratio of “0.5”, “0.3”, and “0.2” of the data G of the change factor information that is the blur information. It is distributed such that “60” is distributed to “n-3” pixels, “36” is distributed to “n-2” pixels, and “24” is distributed to “n-1” pixels. Similarly, “60” that is the pixel data of “n−2” is distributed as “30” in “n−2”, “18” in “n−1”, and “12” in “n”. From this photographed image data “Img ′” and the change factor information data “G” shown in FIG. 7, a photographing result without blur is calculated.

  Any initial image data “Io”, which is data of an arbitrary image shown in step S203, can be used. For this description, the photographed image data “Img ′” is used. That is, the process starts with “Io” = “Img ′”. In the table of FIG. 9, “input” corresponds to the initial image data “Io”. The change factor information data “G” is applied to the initial image data “Io”, that is, “Img ′” in step S204. That is, for example, “60” of the “n-3” pixel of the initial image data “Io” is “30” for the “n-3” pixel, “18” for the “n-2” pixel, “12” is assigned to each pixel of “n−1”. Other pixels are similarly distributed, and comparison image data “Io ′” is generated. Therefore, the difference data “δ” in step S205 is as shown in the bottom column of FIG.

  Thereafter, the size of the difference data “δ” is determined in step S206. Specifically, the process ends when all the difference data “δ” is 5 or less in absolute value, but the difference data “δ” shown in FIG. Proceed to That is, the difference data “δ” is distributed to the initial image data “Io” using the change factor information data “G”, and the restored image data “Io + n” shown as “next input” in FIG. Generate. In this case, since this is the first time, “Io + 1” is shown in FIG.

  The distribution of the difference data “δ” is, for example, “15” obtained by multiplying the data “30” of the pixel “n−3” by 0.5, which is the distribution ratio of the place (= “n−3” pixel). ”Is distributed to the pixel“ n-3 ”, and the data“ 15 ”of the pixel“ n-2 ”is multiplied by 0.3 which is the distribution ratio that should have been to the pixel“ n-2 ”. Furthermore, “4.5” is allocated, and further, 0.2, which is the distribution ratio that should have come to the “n−1” pixel, is assigned to the data “9.2” of the “n−1” pixel. Allotted “1.84” is allocated. The total amount allocated to the “n-3” pixels is “21.34”, and this value is added to the initial image data “Io (here, the captured image data“ Img ′ ”is used)”. Thus, restored image data “Io + 1” is generated.

  As shown in FIG. 11, the restored image data “Io + 1” becomes the input image data (= initial image data “Io”) in step S204, step S204 is executed, the process proceeds to step S205, and the new difference Data “δ” is obtained. The size of the new difference data “δ” is determined in step S206, and if it is larger than the predetermined value, the new difference data “δ” is distributed to the previous restored image data “Io + 1” in step S207, and a new restored image is obtained. Data “Io + 2” is generated (see FIG. 12). Thereafter, new comparison image data “Io + 2 ′” is generated from the restored image data “Io + 2” by performing step S204. As described above, after steps S204 and S205 are executed, the process proceeds to step S206, and the process proceeds to step S207 or the process proceeds to step S208 depending on the determination there. Such a process is repeated.

  In this image processing apparatus 1, in processing, in step S <b> 206, one or both of the number of times of processing and the determination reference value of the difference data “δ” can be set in advance. For example, an arbitrary number such as 20 times or 50 times can be set as the number of times of processing. Also, the value of the difference data “δ” for stopping the process is set to “5” in 8 bits (0 to 255), and when it becomes 5 or less, the process is terminated or set to “0.5”. The process can be terminated when the value is "0.5" or less. This set value can be set arbitrarily. When both the number of processing times and the judgment reference value are input, the processing is stopped when either one is satisfied. When both settings are possible, the determination reference value may be given priority, and if the predetermined number of processes does not fall within the determination reference value, the predetermined number of processes may be repeated.

  In the description of this embodiment, the information stored in the factor information storage unit 7 is not used, but known deterioration factors stored here, such as optical aberrations and lens distortions, are used. It may be used. In this case, for example, in the processing method of the previous example (FIG. 3), it is preferable to perform processing by combining blur information and optical aberration information as one deterioration factor. You may make it correct | amend with the information of an optical aberration after it complete | finishes. Further, the factor information storage unit 7 may not be installed, and the image may be corrected or restored only by dynamic factors at the time of shooting, for example, only blurring.

  The basic operation of the image restoration process of the image processing apparatus 1 according to the present embodiment has been described above with reference to FIGS. 3 to 12. With this basic operation, the image processing apparatus 1 displays an image. In restoring, it is possible to prevent the apparatus from becoming large and to carry out realistic circuit processing. The processing performed by the processing unit 4 is configured by software. However, each processing may be configured by hardware composed of parts that perform part of the processing.

  In addition to the photographed image, the original image to be processed may be a photographed image subjected to processing such as color correction or Fourier transform. Further, as comparison image data, in addition to the data generated using the change factor information data “G”, color correction is applied to the data generated using the change factor information data “G”, or Fourier transform is performed. It is good also as the data. Further, the data of the change factor information includes not only the data of the deterioration factor information but also information that simply changes the image and information that improves the image contrary to the deterioration.

  When the number of processing iterations is set automatically or fixedly on the image processing apparatus 1 side, the set number of times may be changed by the change factor information data “G”. For example, when the image data of a certain pixel is distributed over a large number of pixels due to blurring, the number of repetitions may be increased, and when the dispersion is small, the number of repetitions may be decreased.

  Further, when the difference data “δ” diverges during the iterative process, that is, when the difference data becomes larger, the process may be stopped. For example, it is possible to adopt a method of determining whether or not the light is diverging by observing the average value of the difference data “δ” and determining that the light is diverging when the average value is larger than the previous value. In addition, during an iterative process, if an input is to be changed to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed exceeds 255, the process is stopped. Further, during an iterative process, when an input that is new data is to be changed to an abnormal value, the value may not be used but may be set to a normal value. For example, when a value exceeding 255 within the 8-bit range of 0 to 255 is used as input data, it is processed as a maximum value of 255.

  As described above, when it is determined in step S102 that the photographed image data “Img ′” is not data by panning, the above-described image restoration processing for a normal photographed image in steps S203 to S208 is performed.

  On the other hand, in step S102, it is determined that the captured image data “Img ′” is data by panning, the process proceeds to step S103, and image restoration processing for the panning image described below is performed.

  In the image restoration process for panning, first, in step S103, panning change factor information data “G1” is generated according to the direction of panning. That is, when the process selection SW is operated to the H position, the panning change factor information data “G1” is obtained by compressing the data related to the change factor information data “G” in the left-right direction as the panning direction. Generated as a thing. For example, from the change factor information data “G”, information on the change factor in the left-right direction (the direction of panning) is removed, and only the factor information data in the direction orthogonal to the direction of panning, that is, the vertical direction, The panning change factor information data is “G1”.

  When the process selection SW is operated to the V position, the panning change factor information data “G1” is obtained by compressing the data related to the change factor information data “G” in the vertical direction, which is the direction of panning. Generated as a thing. For example, from the change factor information data “G”, the information about the change factor in the vertical direction (the direction of panning) is removed, and only the change factor information data in the direction orthogonal to the direction of panning, that is, the horizontal direction, is set. It is assumed that the photographing change factor information data “G1”. Then, the image restoration process shown in steps S104 to S109 is performed. The image restoration processing shown in steps S104 to S109 is the same as that described above except that the change factor information data “G” in steps S203 to S208, which is the normal image restoration processing described above, is set to “G1”.

  In other words, the panning change factor information data “G1” does not have image change factor information data for the direction of panning, so in the image restoration process shown in steps S104 to S109, The image restoration processing is not performed, and the blurred (flowed) image remains as a result of panning. On the other hand, in the direction orthogonal to the direction of panning, image restoration processing is performed based on the panning change factor information data “G1” so that an image without blurring is obtained.

  In step S109, the restored image data “Io + n” at the time when the processing is completed is recorded in the recording unit 5 as restored image data of an image taken by panning.

  The determination of whether to perform the image restoration process for the panning image or the image restoration process for the normal photographed image in step S102 of the above-described embodiment is performed by the methods (1) and (2) described below. You can also.

  (1) In conjunction with the shooting operation, the photographer determines whether the shot image data “Img ′” has been shot in the shot image data “Img ′” including the direction of the shot. An identifier such as a flag is assigned. The assignment of the identifier is performed, for example, by pressing a predetermined switch as described below during the photographing operation. For example, when performing panning in the left-right direction, if the panning is performed in a state where the switch for panning in the left-right direction is pressed, an identifier indicating that the panning has been performed in the left-right direction is given. Further, when performing panning in the vertical direction, an identifier indicating that panning has been performed in the vertical direction is given by performing panning while the switch for panning in the vertical direction is pressed. When no switch is pressed, no identifier is given.

  In step S102, it is determined based on the identifier whether the captured image data “Img ′” is a panning image or a normal captured image. If it is a panning image, in step 103, panning change factor information data “G1” is generated according to the contents of the identifier. In other words, in the case of panning in the horizontal direction, the panning change factor information data “G1” is generated by compressing the data related to the change factor information data “G” in the horizontal direction, which is the direction of panning. The That is, the panning change factor information data “G1” is generated as having only change factor information data in the vertical direction.

  Further, in the case of panning in the vertical direction, the panning change factor information data “G1” is generated as the data related to the change factor information data “G” compressed in the vertical direction as the panning direction. Is done. That is, the panning change factor information data “G1” is generated as having only change factor information data in the left-right direction.

  If no identifier is given, it is determined that the captured image data “Img ′” is captured image data of a normal captured image, and the process proceeds to step S203 described above. It should be noted that the identifier can be assigned to the photographed image data “Img ′” even after the shooting is completed, so that a panning shot can be performed without assigning the identifier due to an unexpected photo opportunity or the like. An identifier can also be assigned to the captured image data “Img ′”.

  (2) In step S102, a trajectory of change in the shooting direction of the camera is calculated from the change factor information data “G”, and the presence / absence of panning is determined based on the length and direction of the straight line obtained from the trajectory by the least square method. Determine the direction of panning. That is, if it can be determined that the length of the straight line is larger than the amount of change in the shooting direction of the camera due to normal camera shake, it is determined that the panning has been performed in the direction of the straight line. In step S103, panning change factor information data “G1” is generated by compressing the change factor information data “G” in the direction of the straight line (direction of panning). Thus, from the change factor information data “G”, the locus of change in the shooting direction of the camera is calculated and the direction of panning is determined, and the direction (diagonal direction) that intersects the horizontal direction and the vertical direction is assumed. Even when panning is performed, image restoration processing can be performed in consideration of the direction of panning.

  In the above-described embodiment, the panning change factor information data “G1” is generated from the change factor information data “G”. However, when the panning is performed, the change factor information data regarding the direction of the panning is obtained. The change factor information data “G” may not be included. For example, with respect to the direction of panning, the detection unit 6 does not detect the movement of the image processing apparatus 1 so that the change factor information about the direction of panning does not contribute to the change factor information data “G”. can do. By using the change factor information data “G” that does not include the change factor information data regarding the direction of such a panning shot, the determination process as in step S102 described above and the panning change factor information data as in step S103 are performed. Without generating “G1”, it is possible to perform the restoration process of the panning image only by performing the image restoration process for the normal captured image.

  Each processing method described above may be programmed. The program may be stored in a storage medium such as a CD (Compact Disc), a DVD, or a USB (Universal Serial Bus) memory so that it can be read by a computer. In this case, the image processing apparatus 1 has reading means for reading a program in the storage medium. Further, the program may be stored in an external server of the image processing apparatus 1, downloaded as necessary, and used. In this case, the image processing apparatus 1 has communication means for downloading a program in the storage medium.

It is a block diagram which shows the main structures of the image processing apparatus which concerns on each embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the image processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. FIG. 2 is a processing flowchart for explaining a basic processing method (processing routine) according to the first embodiment performed by a processing unit of the image processing apparatus shown in FIG. 1. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows image data when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which comparison data is generated from an arbitrary image. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example. The situation in which the comparison data is compared with the blurred original image to be processed to generate difference data. It is a figure for demonstrating. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, in which new comparison data is generated from the generated restored data, the data and the blurred original image to be processed, It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and a diagram for explaining a situation in which newly generated difference data is allocated and new restoration data is generated. is there.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 4 Processing part Io Initial image data (data of arbitrary images)
Io ′ comparison image data G change factor information data G1 panning change factor information data Img ′ photographed image data δ difference data k distribution ratio Io + n restored image data Img original image data

Claims (2)

In an image processing apparatus having a processing unit for processing an image,
The processing unit
A determination process for determining whether to perform an image restoration process for a panning image or an image restoration process for a normal shot image on the shot image data;
In the above determination process, when it is determined to perform the image restoration process for the panning image, the image restoration process for the panning image is performed without performing the image restoration process for the direction of the panning image,
The image restoration process for the above panning image is
Panning change factor information data generation processing for generating panning change factor information data that does not have information on the change factor about the direction of panning from the change factor information data that causes image change;
Using the panning change factor information data, comparison image data is generated from arbitrary image data, the captured image data to be processed is compared with the comparison image data, and the obtained difference data is obtained. Use this to generate restored image data, use this restored image data instead of the above-mentioned arbitrary image data, and repeat the same process to generate original image restored image data that approximates the original image Image data generation processing,
An image processing apparatus characterized by In an image processing apparatus that performs image restoration processing using change factor information data that is a factor of an image change detected in a detection unit that detects movement of the image processing apparatus,
When panning is performed, make sure that the change factor information data about the direction of panning does not contribute to the change factor information data .
Using the change factor information data to which the change factor information data about the direction of the panning shot does not contribute, comparison image data is generated from arbitrary image data, and the captured image data to be processed and the comparison image By comparing the image data and using the obtained difference data to generate restored image data, using the restored image data in place of the above-mentioned arbitrary image data and repeating the same processing, the original image Original image restoration image data generation processing for generating original image restoration image data approximating to
An image processing apparatus characterized by

Images