JP2005328117A - Image processing apparatus and method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and method, a recording medium, and a program whereby a user can quickly correct a blurred motion. <P>SOLUTION: A reproduction section 62 reproduces image data recorded by a recording section 61 and displays the image data on a display section 67 via a capture section 63. A motion vector detection section 102 detects a motion vector of a correction region designated by an input section 69 among frames captured by the capture section 63. A parameter conversion section 102 converts the motion vector into a parameter, which is stored in the recording section 61 together with the image data corresponding to the parameter. When the image data of the frame are supplied to the display section 67 and displayed thereon, a prediction section 65 corrects the blurred motion on the basis of a stored blur parameter, and the display section 67 displays the resulting image data. When correction of a blur of the image is instructed, an initial value on the basis of the stored blur parameter is calculated and the image corrected by the initial value is displayed. The image processing apparatus or the like can be applied to digital cameras. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly to an image processing apparatus and method, a recording medium, and a program that can correct a blurred image and efficiently save it.

  Recently, digital cameras (electronic cameras) have become widespread and are used by many users. Image data captured by the digital camera is stored in a memory. Since this image data is recorded electronically and digitally, the user can correct the brightness and color tone of the captured image. Since image data has a larger amount of data than audio data or the like, it is not preferable to store both uncorrected image data and corrected image data in the memory in terms of effective use of the memory. . Further, if the image data before correction is updated with the corrected image data and stored, the image cannot be restored. Thus, it is known that image data before correction (raw data) and image quality setting parameters are associated with each other and stored in a memory (for example, Patent Document 1).

JP 2001-251551 A

  In order to adjust the image quality of the captured image, a relatively troublesome operation is required. For this reason, even if the digital camera itself has a function for image quality adjustment, a general user often does not use the function. In particular, when correcting blur of an image, it is necessary to repeat the process of setting parameters and correcting the image many times, and it takes time to obtain a desired corrected image (an image in which blur is sufficiently corrected).

  The present invention has been made in view of such circumstances, and enables a user to correct image blur relatively easily and quickly.

  The image processing apparatus according to claim 1, a storage unit that stores original data of an image and a parameter for correcting an image based on the original data in association with the original data, and an image based on the stored original data A first generating means for correcting the original data to generate first correction data so as to correct the blur of the image based on the parameter stored correspondingly, and First display control means for controlling the display of the first image based on the first correction data, determination means for determining an instruction for correcting blur of the first image, blur of the first image When correction is instructed, an initial value setting means for setting an initial value of the parameter for correcting blur, and the original data is corrected so as to correct blur based on the initial value, and second correction data is obtained. Second to generate Generating means, second display control means for controlling display of the second image based on the second correction data, acquisition means for acquiring a parameter for correcting the second image, and the acquired parameter A third generation unit that corrects the original data so as to correct blur based on the first correction data to generate third correction data; and a third control unit that controls display of a third image based on the third correction data. Display control means, and update means for updating the parameter stored in correspondence with the original data by the parameter acquired by the acquisition means when storage of the third image is instructed. It is characterized by providing.

  The initial value setting means sets a plurality of initial values, the second generation means generates a plurality of second correction data based on the plurality of initial values, and the second display control means And controlling the display of the plurality of second images based on the second correction data, and the image processing apparatus further includes selection means for selecting one of the plurality of second images, and the acquisition means is selected A parameter for correcting the second image can be acquired.

  The initial value setting means may set a value that minimizes the sum of differences from stored parameters, an absolute value of differences, or a sum of squares of differences as an initial value.

  The initial value setting means may set an initial value based on a date, a day of the week, or a time when an image is taken.

  6. The image processing method according to claim 5, wherein the original data of the image and a parameter for correcting the image based on the original data are stored in association with the original data, and the image based on the stored original data A first generation step of generating the first correction data by correcting the original data so as to correct the blur of the image based on the parameter stored correspondingly, A first display control step for controlling display of the first image based on the first correction data; a determination step for determining an instruction for correcting blur of the first image; and blur of the first image When correction is instructed, an initial value setting step for setting an initial value of the parameter for correcting blur, and the original data is corrected so as to correct blur based on the initial value, and a second correction data is set. A second generation step for generating data, a second display control step for controlling display of the second image based on the second correction data, and an acquisition step for acquiring parameters for correcting the second image A third generation step of generating third correction data by correcting the original data so as to correct blur based on the acquired parameter, and a third image based on the third correction data When the third display control step for controlling the display of the image and the storage of the third image are instructed, the parameter stored in correspondence with the original data by the parameter acquired in the acquisition step And an updating step for updating.

  The recording medium program according to claim 6 is based on a storage step of storing original data of an image and a parameter for correcting an image based on the original data in association with the original data, and on the stored original data A first generation step of generating first correction data by correcting the original data so as to correct blur of the image based on the corresponding stored parameters when displaying an image; A first display control step for controlling display of the first image based on the first correction data; a determination step for determining an instruction to correct blur of the first image; and blur of the first image When the correction is instructed, an initial value setting step for setting an initial value of the parameter for correcting blur, and the original data is corrected so as to correct blur based on the initial value, and the second value is set. A second generation step for generating correction data; a second display control step for controlling display of the second image based on the second correction data; and acquisition for acquiring a parameter for correcting the second image. A third generation step of generating third correction data by correcting the original data so as to correct blur based on the acquired parameter, and a third step based on the third correction data A third display control step for controlling display of an image; and when the storage of the third image is instructed, the parameter acquired in the acquisition step is stored in correspondence with the original data And an update step for updating the parameter.

  The program according to claim 7, wherein a storage step of storing the original data of the image and a parameter for correcting the image based on the original data in association with the original data, and displaying the image based on the stored original data A first generation step of generating the first correction data by correcting the original data so as to correct the blur of the image based on the correspondingly stored parameters; A first display control step for controlling display of the first image based on the correction data, a determination step for determining an instruction to correct blur of the first image, and correction of blur of the first image. When instructed, an initial value setting step for setting an initial value of the parameter for correcting the blur, and correcting the original data so as to correct the blur based on the initial value, and a second correction data A second generation step for generating the second image, a second display control step for controlling the display of the second image based on the second correction data, and an acquisition step for acquiring a parameter for correcting the second image. A third generation step of generating third correction data by correcting the original data so as to correct blur based on the acquired parameter; and a third image based on the third correction data. When the third display control step for controlling display and the storage of the third image are instructed, the parameter stored in correspondence with the original data is determined by the parameter acquired in the acquisition step. An update step for updating is executed by a computer.

  In the present invention, when the first image corrected based on the already stored parameters is displayed, the initial value is set when the blur correction is instructed. Then, the second pixel corrected based on the initial value is displayed, and a parameter for correcting the second pixel is acquired. When saving is instructed, the parameters already saved are updated with the acquired parameters.

  According to the present invention, image blur can be corrected. In particular, according to the present invention, blurring of an image can be corrected easily, easily, and quickly.

  BEST MODE FOR CARRYING OUT THE INVENTION The best mode of the present invention will be described below. The correspondence relationship between the disclosed invention and the embodiments is exemplified as follows. Although there is an embodiment which is described in the specification but is not described here as corresponding to the invention, it means that the embodiment corresponds to the invention. It doesn't mean not. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in the specification. In other words, this description is for the invention described in the specification and not claimed in this application, i.e., for the invention that will be applied for in the future or that will appear as a result of amendment and added. It does not deny existence.

  The image processing apparatus according to claim 1 saves the original data of the image and a parameter for correcting the image based on the original data in association with the original data (for example, executes the process of step S26 in FIG. 6). 3 and when the image based on the stored original data is displayed, the original data is corrected so as to correct the blur of the image based on the corresponding stored parameter. Is generated based on the first correction data (for example, the first generation unit 139 in FIG. 3 that executes the process of step S6 in FIG. 4) and the first correction data. First display control means for controlling the display of the first image (for example, the first display control unit 142 in FIG. 3 that executes the process of step S7 in FIG. 4), and correction of blur in the first image Determining means (for example, the step of FIG. 3 and the initial value setting means for setting the initial value of the parameter for correcting the blur (for example, when determining the blur of the first image is instructed) The initial value setting unit 133 in FIG. 3 that executes the process of step S12 in FIG. 5 and a second correction data that generates second correction data by correcting the original data so as to correct blur based on the initial value. Generating means (for example, the second generation unit 140 in FIG. 3 that executes the process of step S14 in FIG. 5) and second display control for controlling the display of the second image based on the second correction data. Means (for example, the second display control unit 143 in FIG. 3 for executing the processing in step S15 in FIG. 5) and acquisition means (for example, in step S18 in FIG. 6) for acquiring parameters for correcting the second image. The acquisition unit 137) of FIG. 3 that executes the process and the acquired parameter Accordingly, third generation means for correcting the original data so as to correct blur and generating third correction data (for example, the third generation unit 141 in FIG. 3 for executing the processing of step S20 in FIG. 6). ) And third display control means for controlling the display of the third image based on the third correction data (for example, the third display control unit 144 of FIG. 3 for executing the processing of step S21 of FIG. 6). When the storage of the third image is instructed, update means (for example, FIG. 6) updates the parameter stored corresponding to the original data by the parameter acquired by the acquisition means. And an update unit 135) of FIG. 3 for executing the process of step S26.

  The initial value setting means sets a plurality of the initial values, and the second generation means generates a plurality of the second correction data based on the plurality of the initial values, and the second display The control unit controls display of the plurality of second images based on the plurality of second correction data, and the image processing apparatus selects selection unit (for example, one of the plurality of second images). 3 further executes the process of step S16 in FIG. 5, and the acquisition means acquires a parameter for correcting the selected second image.

  The initial value setting means sets, as an initial value, a value that minimizes the sum of differences from the stored parameters, the absolute value of the differences, or the sum of squares of the differences (for example, step S71 in FIG. 13). To S74).

  The initial value setting means sets an initial value based on the date, day of the week, or time when the image was taken (for example, processing in steps S91 to S95 in FIG. 16).

  An image processing method according to claim 5, a program for a recording medium according to claim 6, and a program according to claim 7, wherein the original data of an image and a parameter for correcting an image based on the original data are associated with the original data. When displaying an image based on the saved step (eg, step S26 in FIG. 6 at the time of saving) and the saved original data, the image is blurred based on the corresponding saved parameter. A first generation step (for example, step S6 in FIG. 4) for correcting the original data so as to correct the first data and generating a first image based on the first correction data A first display control step (for example, step S7 in FIG. 4) for controlling the image, a determination step (for example, step S10 in FIG. 4) for determining an instruction for correcting the blur of the first image, and the first When the correction of the blur of the image is instructed, an initial value setting step (for example, step S12 in FIG. 5) for setting the initial value of the parameter for correcting the blur, and the blur is corrected based on the initial value. A second generation step (for example, step S14 in FIG. 5) for correcting the original data to generate second correction data, and a second control for controlling the display of the second image based on the second correction data. 2 display control steps (for example, step S15 in FIG. 5), an acquisition step for acquiring parameters for correcting the second image (for example, step S18 in FIG. 6), and blurring based on the acquired parameters. A third generation step (for example, step S20 in FIG. 6) for correcting the original data to generate the third correction data so as to correct, and the display of the third image based on the third correction data Control When a third display control step (for example, step S21 in FIG. 6) and storage of the third image are instructed, the parameter is acquired corresponding to the original data by the parameter acquired in the acquisition step. And an updating step (for example, step S26 in FIG. 6 at the time of updating) of updating the parameter that has been performed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of an embodiment of an image processing apparatus to which the present invention is applied. The image processing apparatus 1 is configured as a digital camera (electronic camera), for example. In FIG. 1, a CPU (Central Processing Unit) 21 executes various processes according to a program stored in a ROM (Read Only Memory) 22 or a program loaded from a storage unit 29 to a RAM (Random Access Memory) 23. To do. The RAM 23 also appropriately stores data necessary for the CPU 21 to execute various processes.

  The CPU 21, ROM 22, and RAM 23 are connected to each other via a bus 24. An input / output interface 25 is also connected to the bus 24.

  The input / output interface 25 includes an imaging unit 26 including an imaging device such as a complementary metal-oxide semiconductor (CMOS) and a charge coupled device (CCD), an input unit 27 including keys, switches, buttons, and the like, and a CRT (Cathode Ray Tube). Further, a display made up of LCD (Liquid Crystal Display), etc., an output unit 28 made up of speakers, a storage unit 29 made up of a hard disk, etc., and a communication unit 30 made up of a modem are connected. The communication unit 30 performs communication processing via a network including the Internet.

  A drive 31 is connected to the input / output interface 25 as necessary, and a removable medium 32 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from these is loaded. Installed in the storage unit 29 as necessary. In addition to image data and audio data, a parameter table and the like are also stored in the removable medium 32.

  FIG. 2 illustrates a functional configuration example of the image processing apparatus 1. The image processing apparatus 1 also has a sound data processing unit, but is omitted in FIG. 2 for the sake of simplicity. The imaging unit 26 images a subject based on the control from the control unit 70, supplies the obtained image data to the recording unit 61, and records it. If the subject is photographed only once, a still image (image data for one frame) is recorded, and if the subject is photographed continuously, a moving image (image data of a plurality of continuous frames) is recorded.

  The recording unit 61 corresponds to the storage unit 29 or the removable medium 32 and records image data and parameters for correcting the image data in association with each other. The reproduction unit 62 reproduces the image data and parameters recorded in the recording unit 61 as appropriate. The capture unit 63 operates from the image data of the moving image reproduced from the recording unit 61 by the reproduction unit 62, and the image data of the frame for which the user commands the control unit 70 to capture by operating the input unit 69. To capture. When the reproduced image is still image data, the image data of the frame is acquired as it is. The display unit 67 acquires the image data supplied from the reproduction unit 62 via the capture unit 63 and displays a corresponding image.

  The class classification unit 64 converts the parameters supplied from the capture unit 63 and the control unit 70 into a class code, and outputs the class code to the prediction unit 65.

  The prediction unit 65 includes a coefficient storage unit 91 and a product-sum operation unit 92. The coefficient storage unit 91 stores a prediction coefficient for each class code in advance. This prediction coefficient is obtained in advance as a coefficient necessary for converting an image with motion blur into an image without motion blur by learning the relationship between many images with motion blur and images without motion blur. It is what was done. By increasing the number of learning samples, it is possible to obtain a prediction coefficient that can convert various types of motion blur images into motion blur-free images (motion blur corrected). As the principle, for example, those disclosed in Japanese Patent Laid-Open No. 2001-250119 can be used. Since the details are disclosed in the publication, the description thereof is omitted, but in this principle, motion blur is a pixel value of one pixel of a stationary background and a number corresponding to the moving speed of the moving object. A model can be considered that is the result of mixing (adding) pixel values of (M) pixels. Therefore, in the model, a pixel value of an image in which motion blur is corrected is obtained by solving an equation in which the pixel value of one pixel is equal to a value obtained by weighting and adding the pixel values of M pixels. The coefficient storage unit 91 reads the prediction coefficient corresponding to the class code supplied from the class classification unit 64 and supplies the prediction coefficient to the product-sum operation unit 92.

  The product-sum operation unit 92 adapts the prediction tap (prediction pixel) supplied from the tap extraction unit 66 to the linear linear combination formula prepared in advance together with the prediction coefficient supplied to the coefficient storage unit 91. A pixel whose motion blur is corrected is calculated. Based on the class code supplied from the class classification unit 64, the tap extraction unit 66 extracts a predetermined prediction tap from the pixel data supplied from the capture unit 63, and outputs it to the product-sum operation unit 92.

  The display unit 67 controls one of the image data supplied from the capture unit 63, the image data supplied from the product-sum operation unit 92, or the image data supplied from the imaging unit 26 from the control unit 70. The image based on the input image data is displayed.

  The parameter generation unit 68 sets parameter initial values. The parameter generation unit 68 includes a motion vector detection unit 101 and a parameter conversion unit 102.

  The motion vector detection unit 101 detects a motion vector in a correction region (range) specified by the control unit 70 that is a part of the frame supplied from the capture unit 63 and outputs the motion vector to the parameter conversion unit 102. This motion vector is expressed by coordinate values. The parameter conversion unit 102 converts a motion vector expressed by coordinate values into a parameter as a vector. The control unit 70 adds the offset value input by the user from the input unit 69 to the vector-represented parameter input from the parameter conversion unit 102, and classifies the parameter (initial value) obtained by the addition as a class. The data is output to the classification unit 64.

  The user operates the input unit 69 corresponding to the input unit 27 in FIG. 1 to input an offset value (a value for correcting the parameter) as a parameter for correcting the blur of the image, and to control each unit. Command to 70. The control unit 70 controls each unit based on a command from the input unit 69 and supplies necessary parameters, control information, and the like to each unit.

  FIG. 3 illustrates a more detailed functional configuration example of the control unit 70.

  The determination unit 131 performs various determination processes necessary for the image processing apparatus 1 to perform an editing process. The area setting unit 132 performs a process of setting a correction area based on a user instruction. The initial value setting unit 133 performs processing for setting an initial value of the blur parameter. The storage unit 134 associates the image data with the blur parameter for correcting the motion blur corresponding thereto, and stores the image data on the table. The update unit 135 performs an update process on the motion parameter stored by the storage unit 134.

  The selection unit 136 performs a process of selecting one of the displayed correction images. The acquisition unit 137 acquires the offset value input by the user. The correcting unit 138 performs processing for correcting the blur parameter based on the acquired offset value.

  The first generation unit 139 generates corrected image data for displaying a corrected image in which the motion blur is corrected based on the already-stored blur parameter. The second generation unit 140 generates corrected image data for the corrected image in which the motion blur is corrected based on the blur parameter set as the initial value. The third generation unit 141 generates corrected image data for the corrected image in which the motion blur is corrected based on the blur parameter corrected by the offset value input from the user.

  The first display control unit 142 controls display of an image corresponding to the corrected image data generated by the first generation unit 139. The second display control unit 143 controls display of an image corresponding to the corrected image data generated by the first display control unit 142. The third display control unit 144 controls display of the corrected image data generated by the third generation unit 141.

  3 are connected to each other, and necessary data can be supplied from other processing units as necessary.

  Next, the operation of the image processing apparatus 1 will be described. When the user operates the input unit 69 to command the imaging mode, the control unit 70 controls the imaging unit 26 to image the subject. Image data of the subject imaged by the imaging unit 26 is supplied to the display unit 67 and displayed as an image. The user can confirm the subject by viewing this display.

  Further, when the user views the image on the display unit 67 and instructs recording, the image data of the image captured by the imaging unit 26 is recorded in the recording unit 61 as a still image or a moving image.

  Next, image editing processing based on image data recorded in the recording unit 61 will be described with reference to the flowcharts of FIGS. 4 to 6. This process is started when the user operates the input unit 69 to instruct the editing mode and instructs the control unit 70 to start the editing process.

  In step S <b> 1, the reproduction unit 62 reproduces an image designated by the user among images recorded in the recording unit 61. That is, the reproduction unit 62 reproduces image data (for example, moving image data) designated by the user from the image data recorded in the recording unit 61 and outputs the image data to the capture unit 63. The capture unit 63 outputs the image data supplied from the reproduction unit 62 to the display unit 67 for display. As a result, an image designated by the user and based on the image data recorded in the recording unit 61 is displayed as a moving image. For example, when the image processing apparatus 1 constitutes a part of a home security system, the user can confirm an intruder from this image.

  In step S2, the determination unit 131 determines whether capture has been instructed. That is, the user views the moving image displayed on the display unit 67 and instructs the capture by operating the input unit 69 at the timing when the image to be confirmed is displayed. If capture is not instructed, the determination unit 131 determines whether or not to end in step S3. That is, when ending the editing process, the user operates the input unit 69 to instruct termination. If termination is instructed, the editing process is terminated. If not, the process returns to step S1, and the subsequent processes are repeatedly executed.

  If it is determined in step S2 that capture has been commanded, in step S4, the capture unit 63 captures a still image. That is, among the images supplied from the playback unit 62, at the timing when capture is commanded from the control unit 70, the image of the displayed frame is captured and output to the display unit 67. Thereby, the captured still image is displayed on the display unit 67.

  In step S5, the determination unit 131 determines whether the blur parameter is registered. That is, when editing processing for correcting motion blur has been performed even once in the past, the blur parameter may be stored together with corresponding image data in step S26 of FIG. As described above, when the blur parameter is already registered, the capture unit 63 reads out the blur parameter corresponding to the captured frame from the reproduction data supplied from the reproduction unit 62, and classifies it into the class classification. To the unit 64.

  In step S6, the first generation unit 139 executes processing for generating corrected image data in which motion blur is corrected based on the blur parameter. That is, the first generation unit 139 controls the class classification unit 64 and the prediction unit 65 to execute processing for generating corrected image data. Specifically, the class classification unit 64 acquires and supplies the blur parameter recorded in the recording unit 61 corresponding to the frame being captured, so that the supplied blur parameter is supplied. The value of the class code C corresponding to is calculated by the following equation.

  C = (a + A) × Nmax + (n + N) (1)

  In the above formula (1), a represents the x coordinate component of the motion vector in the designated area, and n represents the y coordinate component. A represents the x coordinate component of the offset value input by the user, and N represents the y coordinate component. The blur parameter stored corresponding to the image data is the value of ((a + A), (n + N)) in equation (1). Accordingly, the class classification unit 64 calculates the class code C by applying the value of the blur parameter ((a + A), (n + N)) to the above equation (1).

  In formula (1), Nmax means the total number of classes of y coordinate components.

  The coefficient storage unit 91 reads the prediction coefficient corresponding to the class code C supplied from the class classification unit 64 and outputs the prediction coefficient to the product-sum operation unit 92. The tap extraction unit 66 extracts a prediction tap corresponding to the class code supplied from the class classification unit 64 from the pixel data of the frame supplied from the capture unit 63. The extracted prediction tap is supplied to the product-sum operation unit 92.

  The product-sum operation unit 92 generates pixel data as corrected image data in which blur is corrected by substituting a prediction coefficient and a prediction tap into a linear linear combination formula prepared in advance and performing a product-sum operation. .

  In step S7, the display unit 67 displays an image corresponding to the corrected image data generated in step S6. Therefore, the user can correct the motion blur even once and save the corrected image (parameter) in the past, and can view the corrected image.

  If it is determined in step S5 that no blur parameter is registered, the corrected image data generation process in step S6 is skipped. Therefore, in this case, in step S7, an image corresponding to the image data reproduced from the recording unit 61 (original data not corrected) is displayed as it is.

  In step S8, the determination unit 131 determines whether or not to end the process. If the user gives an instruction to end the process, the editing process ends. On the other hand, when the end is not instructed, in step S9, the determination unit 131 determines whether to reproduce the moving image. In other words, the user captures and confirms the image displayed on the display unit 67, and when there is no need for further correction, the user operates the input unit 69 to instruct the moving image reproduction. At this time, the process returns to step S1, and the control unit 70 controls the reproduction unit 62 to reproduce the image data recorded in the recording unit 61 again. Thereby, the image of the frame following the captured frame is sequentially displayed on the display unit 67 as a moving image.

  When it is determined in step S9 that the moving image reproduction is not instructed, in step S10, the determination unit 131 determines whether correction is instructed. If it is determined that correction is not instructed, the process returns to step S8, and the subsequent processes are repeatedly executed.

  When the user looks at the image displayed on the display unit 67 and wants to correct the motion blur, the user operates the input unit 69 to instruct that effect. Further, the user operates the input unit 69 to instruct which region of the image displayed on the display unit 67 is to be corrected. In step S11, the region setting unit 132 sets a correction region based on an instruction from the user. For example, as shown in FIG. 7, when frames F1 to F3 are sequentially displayed as moving images, if capture is commanded at the timing when the frame F2 is displayed, as shown in FIG. The captured frame image is displayed as a still image. The user views the image and designates an area where motion blur is to be corrected by using the input unit 69 (for example, a mouse constituting the input unit 27). In the example of FIG. 8, since the person is moving in the left direction in the figure, motion blur occurs in the image of the person. Therefore, the user designates an area including the person's image as the correction area 151.

  In step S12, the initial value setting unit 133 performs an initial value setting process. In other words, the user has instructed the correction of the motion blur of the image, but if the user inputs the parameter of the motion blur correction from the unset state, an image in which the motion blur is completely corrected is realized. It will take time. In view of this, a correction candidate for an image to be corrected is presented to the user, and the user can select a desired image from the presented correction candidate images, thereby realizing a quick correction process. The user further corrects the selected candidate image as necessary.

  Therefore, in the initial value setting process in step S12, for example, as shown in the flowchart of FIG. 9, two motion vectors are detected, and corresponding blur parameters are generated.

  That is, in step S51, the motion vector detection unit 101 detects two motion vectors. As the motion vector, when the blur parameter has not yet been saved, the direction and absolute value of each motion vector are compared among the motion vectors for each pixel in the set correction area 151, and both motions are equal. Two motion vectors having the highest frequency are detected using the same motion vector. When the blur parameter is already stored, a value other than the motion vector represented by (a, n) is detected among the stored parameters ((a + A), (n + N)). For example, when the motion vector value (a, n) already stored corresponds to the motion vector having the highest frequency, the motion vectors having the second and third frequencies are detected. That is, even if the motion vector value already stored is the motion vector having the highest frequency, the correction amount of motion blur requested by the user does not correspond to the motion vector. Even if it is the second or the third, it is considered that it is more preferable as a motion vector for correcting the motion blur required by the user. In other words, this makes it possible to complete the motion blur correction process desired by the user more quickly.

  In step S52, the parameter conversion unit 102 converts the first motion vector detected in step S51 into a first blur parameter. In step S53, the parameter conversion unit 102 converts the other second motion vector detected in step S51 into a second blur parameter.

  When two blur parameters are generated in this way, the initial value setting unit 133 supplies the two blur parameters to the class classification unit 64 and the tap extraction unit 66.

  Returning to FIG. 5, in step S <b> 13, the class classification unit 64 performs class classification based on the blur parameter set as the initial value supplied from the initial value setting unit 133 of the control unit 70. That is, two class codes are obtained by this process. In step S14, the second generation unit 140 generates corrected image data in which motion blur is corrected based on the blur parameter. That is, the second generation unit 140 controls the prediction unit 65 to generate corrected image data. Specifically, the coefficient storage unit 91 calculates two sets of prediction coefficients corresponding to the two class codes supplied from the class classification unit 64, and outputs them to the product-sum calculation unit 92. Further, the tap extraction unit 66 extracts two sets of prediction taps corresponding to the input two sets of class codes from the pixel data of the captured frame, and outputs them to the product-sum operation unit 92. The product-sum operation unit 92 generates the first corrected image data by applying the first prediction coefficient and the first prediction tap to the linear linear combination formula, and generates the second prediction coefficient and the second prediction coefficient. The second corrected image data is generated by applying the tap to the linear linear combination formula.

  In step S15, the second display control unit 143 executes processing for displaying the corrected image of the corrected image data generated in step S14. That is, the display unit 67 controls the second display control unit 143 to display an image based on the first correction image and an image based on the second correction image supplied by the product-sum calculation unit 92 on the screen. For example, the left half and the right half are displayed simultaneously. Alternatively, the first corrected image may be displayed on the entire screen, and after the predetermined time has been displayed, the second corrected image may be displayed instead. That is, a plurality of (in this case, two images) are displayed by dividing the space or dividing the time.

  In step S <b> 16, the selection unit 136 executes a process for selecting a correction image. Specifically, the user selects one of the two displayed correction images by operating the input unit 69. As a result, if parameters have already been saved, a different parameter is generated as an initial value based on the saved parameter, and an image with the blur corrected correspondingly is automatically (user's Therefore, the user can obtain a corrected image in which motion blur is corrected more quickly.

  Next, in step S <b> 17, the determination unit 131 determines whether correction is instructed by the user. That is, the user looks at the selected correction image, determines whether or not the motion blur is sufficiently corrected to the state desired by the user, and if it is determined that the correction is sufficiently performed, instructs the correction. Instruct to register without In this case, it is determined in step S17 that correction is not instructed, and in step S22, it is determined that registration is instructed by the determination unit 131. In this case, in step S26, the storage unit 134 stores the blur parameter and the captured still image data in association with each other.

  When the above processing is performed, the user can realize only by the value set by the initial value without substantially specifying the blur parameter himself / herself, thereby enabling quick editing processing. .

  On the other hand, when it is determined that further correction is necessary by looking at the corrected image based on the initial value, the user operates the input unit 69 to instruct correction. Then, the user further operates the input unit 69 to input an offset value for further correcting the corrected image.

Therefore, when it is determined in step S17 that the determination unit 131 has not instructed correction, the acquisition unit 137 acquires the offset value input by the user in step S18. Specifically, as shown in FIG. 10, the movement in the correction area 151 represented by the upper left point P ₁ (X ₁ , Y ₁ ) and the lower right point P ₂ (X ₂ , Y ₂ ). When the vector 152 is represented by a vector (a, n) of a value a as the x component and a value n as the y component, the offset value (offset parameter) is the origin Q (A, N) is specified.

  In step S19, the correction unit 138 executes a process for correcting the blur parameter. Specifically, the blur parameter ((a + A), (n + N)) is obtained by adding the offset value (A, N) acquired in step S18 to the already obtained motion vector (a, n). Generated.

  In step S20, the third generation unit 141 executes a process of generating corrected image data based on the blur parameter corrected in the process of step S19. That is, the third generation unit 141 supplies the corrected blur parameters ((a + A), (n + N)) to the class classification unit 64 and the tap extraction unit 66. The class classification unit 64 outputs a class code corresponding to the input blur parameter ((a + A), (n + N)) to the coefficient storage unit 91 and the tap extraction unit 66. The coefficient storage unit 91 reads a prediction coefficient corresponding to the input class code and outputs the prediction coefficient to the product-sum operation unit 92. The tap extraction unit 66 extracts the class tap corresponding to the input class code from the pixel data of the captured frame supplied from the capture unit 63, and outputs the prediction tap to the product-sum operation unit 92. The product-sum operation unit 92 assigns the input prediction coefficient and prediction tap to the linear linear combination formula, and generates corrected image data of the corrected image in which motion blur is corrected.

  In step S21, the third display control unit 144 displays the corrected image. That is, the third display control unit 144 supplies the corrected image data generated in the process of step S20 from the product-sum operation unit 92 to the display unit 67 and displays it as a corrected image.

  The user looks at this corrected image to determine whether or not the motion blur is sufficiently corrected, and when it is determined that further correction is necessary, the user operates the input unit 69 to instruct further correction. . In this case, in step S22, the determination unit 131 determines that registration is not instructed, and in step S23, determines that correction is instructed. Therefore, the process returns to step S18 again, and an offset value acquisition process is executed. That is, the user operates the input unit 69 again to input a new offset value (A, N). Then, a corrected image generation process based on the new offset value is executed in the same manner as described above, and a corrected image based on the corrected image generation process is output to the display unit 67 and displayed.

  If it is determined in step S23 that correction is not instructed, in step S24, the determination unit 131 determines whether to end the process based on an instruction from the user. That is, when ending the process, the user operates the input unit 69 to instruct that effect. In step S24, when it is determined that the end of the process is not instructed, in step S25, the determination unit 131 determines whether to play a moving image based on a user instruction. That is, when it is determined that the user does not need to register by looking at the corrected image, the user instructs the moving image reproduction by operating the input unit 69 without instructing the registration. When the moving image reproduction is instructed, the process returns to step S1, and the subsequent processes are repeatedly executed.

  When it is determined in step S25 that the moving image reproduction is not instructed, the determination unit 131 determines whether registration is instructed in step S22. If it is determined that registration is not instructed, the process returns to step S23, and the subsequent processes are repeatedly executed.

  When the user sees the corrected image and determines that registration is to be performed, the user instructs registration by operating the input unit 69. At this time, in step S26, the update unit 135 executes a process of storing the blur parameter and the captured still image image in association with each other. That is, when the blur parameter is already stored, the blur parameter is updated to a new blur parameter. The image data itself (original data) is not updated. Therefore, it is possible to prevent image data from increasing more than necessary.

  After the process of step S26, the process returns to step S23, and the subsequent processes are repeatedly executed.

  As described above, in this embodiment, an image that has been corrected based on an already stored blur parameter is displayed, and a parameter different from that is set as an initial value, and a corrected image based on the parameter is displayed. Therefore, the user only needs to make further corrections from the corrected image as necessary, the correction process is simplified, and quick correction is possible.

  Also, the initial value as the blur parameter is different from the blur parameter that has already been saved, but it is set to the value set according to the same standard, so the number of times to enter the offset value can be reduced. Thus, it is possible to obtain an image in which motion blur is sufficiently corrected with a smaller number of corrections.

  In the above, the initial value of the blur parameter is set based on the motion vector corresponding to the motion blur. However, the initial value may be set by another method on the basis of the blur parameter already stored. Is possible.

  For example, as shown in FIG. 11, it is assumed that the recording unit 61 stores a blur parameter and a number for identifying the corresponding image data in association with each other on the table. In this example, the content ID that identifies the content, the frame number that identifies the frame of the content specified by the content ID, the blur parameter corresponding to the frame, and the date and time when the frame (content) was captured are associated with each other. Are recorded. Specifically, for the content ID “001”, the blur parameter “P1” is recorded for the frame number “125”, and the blur parameter “P2” is recorded for the frame number “160”. The content with the content ID is taken on “October 22, 2003”.

  Further, the blur parameter “P21” is recorded for the frame number “295” of the content whose content ID is “002”. The shooting date of this content is “November 15, 2003”. Similarly, the blur parameter “P31” is recorded for the frame having the frame number “330” having the content ID “003”. The shooting date of this content is “December 26, 2003”.

  FIG. 12 illustrates a functional configuration example of the initial value setting unit 133 when initial value setting is performed based on blur parameters registered in advance as illustrated in FIG. 11. As shown in the figure, the initial value setting unit 133 includes a detection unit 201, a weighting unit 202, a minimum value calculation unit 203, and a setting unit 204.

  The detection unit 201 detects a blur parameter that is already stored from data on the table reproduced by the reproduction unit 62 from the recording unit 61. The weighting unit 202 weights the blur parameter detected by the detection unit 201. The minimum value calculation unit 203 calculates a value that minimizes the sum of differences from the weighted parameters. The setting unit 204 performs processing for setting the minimum value calculated by the minimum value calculation unit 203 as an initial value and outputting the initial value to the class classification unit 64 and the tap extraction unit 66.

  Next, processing for setting an initial value based on a stored motion parameter will be described with reference to the flowchart of FIG.

  In step S71, the detection unit 201 detects a blur parameter. Specifically, as shown in FIG. 11, the blur parameters P1, P2, P21, P31,... Recorded on the table are detected. In step S72, the weighting unit 202 executes a process of weighting the blur parameter detected by the detection unit 201 in the process of step S71. The weighting coefficient is arbitrary, but is set such that the sum is 1, for example.

  In step S73, the minimum value calculation unit 203 calculates a value that minimizes the sum of differences from the weighted blur parameter. That is, in step S72, a value that minimizes the sum of differences from the weighted blur parameter is calculated here. In step S74, the setting unit 204 executes processing for setting the calculated value as an initial value. That is, the value of the blur parameter calculated in the process of step S73 is set as an initial value.

  As described above, the initial value set by the setting unit 204 is supplied to the class classification unit 64, converted into a class code as described above, and converted into a corresponding prediction coefficient by the coefficient storage unit 91. The The tap extraction unit 66 outputs the prediction tap extracted based on the initial value to the product-sum operation unit 92. The product-sum calculation unit 92 performs a product-sum calculation process of the prediction tap and the prediction coefficient, and the blur parameter set as the initial value on the display unit 67 (the blur generated by weighting the stored blur parameter). The corrected image based on the parameter) is displayed.

  Then, as described above, an offset value is further added to the corrected image as necessary, and correction is performed based on the offset value.

  In this embodiment, since only one correction image is displayed based on the initial value, the process in step S16 in FIG. 5 is substantially bypassed.

  Of course, also in this case, two or more initial values may be set and a plurality of corrected images may be displayed.

  In the processing of FIG. 13, all blur parameters are used, but it is also possible to set initial values based on only blur parameters with more approximate shooting conditions. In this case, as shown in FIG. 14, in the table, not only the date but also the day of the week and the time are stored as date and time information corresponding to each frame. In the example of FIG. 14, the shooting date of the frame number “125” of the content ID “001” is “October 22, 2003”, and the day of the week is “Wednesday”. The shooting time is “10:05:03”, and the shooting time of the frame “185” is “10:05:05”. The date / time information of the content ID “002” is “November 15, 2003”, “Saturday”, and “15:23:03”. The date and time information of the content ID “003” is “December 26, 2003”, “Friday”, and “19:55:25”.

  In this case, the initial value setting unit 133 includes a detection unit 251, a first search unit 252, a second search unit 253, a third search unit 254, and a setting unit 255, as shown in FIG. The

  The detection unit 251 detects the shooting date and time of the image data recorded in the recording unit 61. The first search unit 252 searches for a frame shot at a time closest to the capture time of the capture frame. The second search unit 253 searches for the frame of the day of the week closest to the capture frame shooting date. The third search unit 254 searches for a frame shot on the date closest to the shooting date of the capture frame. The setting unit 255 performs processing for setting the blur parameter of the searched frame as an initial value and outputting it to the class classification unit 64 and the tap extraction unit 66.

  Next, the operation will be described with reference to the flowchart of FIG. In step S91, the detection unit 251 detects the shooting date, day of the week, and time of the capture frame. These date and time information is also stored in advance on the table (however, since the editing process has not been performed yet, the blur parameter is not registered).

  In step S92, the first search unit 252 searches for a frame that is captured at a time closest to the capture time of the capture frame detected in the process of step S91 and in which the blur parameter is stored. For example, when the capture frame is an image taken at “19:50:03”, a frame taken at a time close to that time and storing a blur parameter is searched. In step S <b> 93, the second search unit 253 searches for a frame that is shot on the same day of the week as the shooting date of the capture frame or the day of the week that is the closest, and in which the blur parameter is stored. For example, when the capture frame is shot on “Saturday”, a frame that is shot on the day of the week closest to “Saturday” and stores the blur parameter is searched.

  In step S94, the third search unit 254 searches for a frame that has been shot on the date that is the same as or closest to the shooting date of the capture frame and in which the blur parameter is stored. For example, when the shooting date of the capture frame is “November 15, 2003”, the shooting date is closest to “November 15, 2003” and the blur parameter is stored. The frame is searched.

  In step S95, the setting unit 255 sets the blur parameter of one searched frame as an initial value. Specifically, one with the least error is selected from the frames searched in the processes of steps S92 to S94, and the blur parameter of that frame is set as an initial value. In this case, for example, when the shooting time is 3 hours or more away, it may be determined that there is no other frame shot at the time closest to the shooting time, and the frame may be searched by day of the week. . Further, for example, when the day of the week is more than 3 days away, it is possible to search by the shooting date, assuming that the frame of the closest day of the week has not been searched.

  As described above, the initial value of the set frame is supplied to the class classification unit 64 and the tap extraction unit 66, and is displayed on the display unit 67 as a corrected image based on the initial value in the same manner as described above. .

  Note that the motion blur described above includes not only motion blur due to movement of the subject but also motion blur due to camera shake. In the case of camera shake, motion blur occurs throughout the frame (screen), so that the correction area is the entire screen. Various parameters can be used in various formats. For example, the parameter corresponding to the motion vector may be represented by the angle and magnitude of the motion vector.

  Although the present invention has been described with respect to correcting motion blur, the present invention can also be applied to correcting not only motion blur but also focus blur. In this case, the coefficient storage unit 91 learns and stores a prediction coefficient necessary for correcting the focus blur. In this case, the parameter generation unit 68 is substantially unnecessary. In the case of correcting the focus blur, there is a model in which the focus blur is a result of mixing (adding) pixel values of a plurality of peripheral pixels to a pixel value of one pixel according to a Gaussian distribution having a σ value. Conceivable. Then, by learning the relationship between many images that are out of focus and images that are not out of focus, a prediction coefficient for obtaining an image that is corrected from the image that is out of focus can be obtained. The user sets the σ value as a parameter when correcting the focus blur. In this case, the class code C is calculated as follows, for example (in this case, image data is supplied to the class classification unit 64).

  C = σ × ADRCmax + ADRC (2)

  ADRC in the above equation is a value representing the result of comparing the pixels in a given block with a threshold value (defined by the dynamic range and minimum value in that block), and ADRCmax is the class code by the ADRC processing. Represents the total number.

  Furthermore, both the blur parameter for correcting the motion blur and the blur parameter for correcting the focus blur can be stored in the coefficient storage unit 91. Then, using the corrected image corrected based on the blur parameter for correcting the motion blur and the blur parameter for correcting the focus blur as the initial values, the corrected image corrected is displayed on the display unit 67 so that the user can select it. It may be. In this way, even if the user cannot accurately recognize whether the image is blurred due to motion blur or focus blur, it is possible to quickly correct the blurred image.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer is installed from a network or a recording medium.

  As shown in FIG. 1, the recording medium is distributed to provide a program to the user separately from the main body of the apparatus, and includes a magnetic disk (including a floppy disk) on which the program is recorded, an optical disk (CD- In addition to ROM (Compact Disk-Read Only Memory), DVD (including Digital Versatile Disk)), magneto-optical disk (including MD (Mini-Disk)), or removable media 32 consisting of semiconductor memory, etc. The program is provided with a ROM 22 in which a program is recorded, a hard disk included in the storage unit 29, and the like provided to the user in a state of being incorporated in the apparatus main body in advance.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The present invention can be applied to a digital camera.

It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. It is a block diagram which shows the functional structural example of the image processing apparatus of FIG. It is a block diagram which shows the functional structural example of the control part of FIG. It is a flowchart explaining an edit process. It is a flowchart explaining an edit process. It is a flowchart explaining an edit process. It is a figure which shows the example of a moving image. It is a figure which shows a captured image. It is a flowchart explaining an initial value setting process. It is a figure explaining the parameter corresponding to motion blur. It is a figure which shows the example of a table. It is a block diagram which shows the functional structural example of an initial value setting part. It is a flowchart explaining an initial value setting process. It is a figure which shows the structural example of a table. It is a block diagram which shows the other functional structural example of an initial value setting part. It is a flowchart which shows an initial value setting process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 61 Recording part, 62 Playback part, 63 Capture part, 64 Class classification part, 65 Prediction part, 66 Tap extraction part, 67 Display part, 68 Parameter generation part, 69 Input part, 70 Control part, 91 Coefficient Storage unit, 92 product-sum operation unit, 101 motion vector detection unit, 102 parameter conversion unit, 131 determination unit, 132 area setting unit, 133 initial value setting unit, 134 storage unit, 135 update unit, 136 selection unit, 137 acquisition unit , 138 correction unit, 139 first generation unit, 140 second generation unit, 141 third generation unit, 142 first display control unit, 143 second display control unit, 144 third display control unit

Claims (7)

Storage means for storing image original data and parameters for correcting an image based on the original data in association with the original data;
When displaying an image based on the stored original data, the first correction data is obtained by correcting the original data so as to correct the blur of the image based on the corresponding stored parameter. First generating means for generating;
First display control means for controlling display of the first image based on the first correction data;
Determining means for determining an instruction to correct blur of the first image;
An initial value setting means for setting an initial value of the parameter for correcting blur when an instruction to correct blur of the first image is given;
Second generation means for correcting the original data so as to correct blur based on the initial value and generating second correction data;
Second display control means for controlling display of a second image based on the second correction data;
Obtaining means for obtaining a parameter for correcting the second image;
Third generation means for correcting the original data so as to correct blur based on the acquired parameter to generate third correction data;
Third display control means for controlling display of a third image based on the third correction data;
Update means for updating the parameter stored in correspondence with the original data by the parameter acquired by the acquisition means when storage of the third image is instructed. Image processing device. The initial value setting means sets a plurality of the initial values,
The second generation means generates a plurality of the second correction data based on the plurality of initial values,
The second display control means controls display of the plurality of second images based on the plurality of second correction data;
The image processing apparatus further includes a selection unit that selects one of the plurality of second images,
The image processing apparatus according to claim 1, wherein the acquisition unit acquires a parameter for correcting the selected second image. The initial value setting means sets, as the initial value, a value that minimizes a sum of differences from the stored parameter, an absolute value of the difference, or a sum of squares of the differences. Item 8. The image processing apparatus according to Item 1. The image processing apparatus according to claim 1, wherein the initial value setting unit sets the initial value based on a date, a day of the week, or a time when the image is captured. A storage step of storing image original data and parameters for correcting an image based on the original data in association with the original data;
When displaying an image based on the stored original data, the first correction data is obtained by correcting the original data so as to correct the blur of the image based on the corresponding stored parameter. A first generating step to generate;
A first display control step for controlling display of the first image based on the first correction data;
A determination step of determining an instruction to correct blur of the first image;
An initial value setting step for setting an initial value of the parameter for correcting blur when an instruction to correct blur of the first image is given;
A second generation step of generating second correction data by correcting the original data so as to correct blur based on the initial value;
A second display control step for controlling the display of the second image based on the second correction data;
Obtaining a parameter for correcting the second image;
A third generation step of generating third correction data by correcting the original data so as to correct blur based on the acquired parameter;
A third display control step for controlling display of a third image based on the third correction data;
An update step of updating the parameter stored in correspondence with the original data with the parameter acquired in the acquisition step when storage of the third image is instructed. Image processing method. A storage step of storing image original data and parameters for correcting an image based on the original data in association with the original data;
When displaying an image based on the stored original data, the first correction data is obtained by correcting the original data so as to correct the blur of the image based on the corresponding stored parameter. A first generating step to generate;
A first display control step for controlling display of the first image based on the first correction data;
A determination step of determining an instruction to correct blur of the first image;
An initial value setting step for setting an initial value of the parameter for correcting blur when an instruction to correct blur of the first image is given;
A second generation step of generating second correction data by correcting the original data so as to correct blur based on the initial value;
A second display control step for controlling the display of the second image based on the second correction data;
Obtaining a parameter for correcting the second image;
A third generation step of generating third correction data by correcting the original data so as to correct blur based on the acquired parameter;
A third display control step for controlling display of a third image based on the third correction data;
An update step of updating the parameter stored in correspondence with the original data with the parameter acquired in the acquisition step when storage of the third image is instructed. A recording medium on which a computer-readable program is recorded. A storage step of storing image original data and parameters for correcting an image based on the original data in association with the original data;
When displaying an image based on the stored original data, the first correction data is obtained by correcting the original data so as to correct the blur of the image based on the corresponding stored parameter. A first generating step to generate;
A first display control step for controlling display of the first image based on the first correction data;
A determination step of determining an instruction to correct blur of the first image;
An initial value setting step for setting an initial value of the parameter for correcting blur when an instruction to correct blur of the first image is given;
A second generation step of generating second correction data by correcting the original data so as to correct blur based on the initial value;
A second display control step for controlling the display of the second image based on the second correction data;
Obtaining a parameter for correcting the second image;
A third generation step of generating third correction data by correcting the original data so as to correct blur based on the acquired parameter;
A third display control step for controlling display of a third image based on the third correction data;
When the storage of the third image is instructed, the computer executes an update step of updating the parameter stored corresponding to the original data by the parameter acquired in the acquisition step. A featured program.

Images