WO2009110184A1 - Method of creating data relating to change factor information and signal processor - Google Patents

Method of creating data relating to change factor information and signal processor Download PDF

Info

Publication number
WO2009110184A1
WO2009110184A1 PCT/JP2009/000765 JP2009000765W WO2009110184A1 WO 2009110184 A1 WO2009110184 A1 WO 2009110184A1 JP 2009000765 W JP2009000765 W JP 2009000765W WO 2009110184 A1 WO2009110184 A1 WO 2009110184A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
factor information
change factor
signal
processing
Prior art date
Application number
PCT/JP2009/000765
Other languages
French (fr)
Japanese (ja)
Inventor
高橋史紀
Original Assignee
日東光学株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日東光学株式会社 filed Critical 日東光学株式会社
Priority to CN200980107860.2A priority Critical patent/CN101960488B/en
Publication of WO2009110184A1 publication Critical patent/WO2009110184A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations

Definitions

  • the present invention relates to a method of generating data of change factor information and a signal processing device.
  • Patent Document 1 In the past, the present applicant has proposed a method for restoring an image (signal) that has deteriorated due to camera shake during shooting (see Patent Document 1).
  • the processing unit uses data of change factor information that causes a signal change, and compares the data from an arbitrary signal. Data is generated, the original signal data to be processed is compared with the comparison data, and the restored data is generated using the obtained difference data. The restored data is used instead of the arbitrary signal data. In this case, the same process is repeated, and the process of generating restored data that approximates the original signal before the change is performed.
  • G 0 is given a constraint condition such that it does not become a negative value, and G 1 is obtained.
  • a 1 is given a constraint such that A 1 does not become a negative value, and A 2 is obtained.
  • G 2 is given a constraint condition such that G 2 does not become a negative value, and G 3 is obtained. Such calculation is repeated many times to obtain appropriate PSF and restored image data.
  • JP 2007-48257 A Takeda, Komatsu “Examination of non-negative constraint conditions in Fourier iteration algorithm for blind deconvolution", Optics, Japan Optical Society, 1996, Vol. 25, No. 5, p274-281
  • Patent Document 1 describes a signal restoration process using data of known change factor information.
  • a speed sensor, an acceleration sensor, or the like needs to be a component.
  • Non-Patent Document 1 If the technology described in Non-Patent Document 1 is employed, an image can be restored without using a speed sensor or an acceleration sensor even if the data of the change factor information is unknown. However, in this technique, it is difficult to set a constraint condition corresponding to any image, and since the processing time becomes long, it is difficult to adopt it for a general-purpose device.
  • an object of the present invention is to provide a method of generating data of change factor information and a signal processing device that enable practical signal restoration even if the data of change factor information is unknown.
  • the change factor information generation method of the present invention is the change factor information data for generating the change factor information data that causes the signal change of the original signal data that has undergone changes such as deterioration.
  • any change factor information data is set as the first change factor information data.
  • comparison data is generated from any signal data.
  • the “restored data” in (5) includes both the first restored data and the restored data obtained by repeating the process as necessary in (4).
  • the method of generating change factor information according to another invention uses the new change factor information data instead of the first change factor information data, and performs the same processing as (1) to (3). Repeat at least once. By adopting this method, more appropriate change factor information data can be generated.
  • the processing of (3) includes the data of the original signal, the data obtained by processing the original signal data, and the data of the signal elements constituting the restoration data Is performed by moving a part or all of the data, and it is determined whether or not the signal element data constituting the restoration data satisfies a predetermined standard. If the predetermined standard is not satisfied, the difference data is used. Then, the distribution value calculated in this way is corrected, and correction processing using the data obtained by the corrected distribution value instead of the unsatisfied data is performed.
  • the quality of the first restoration data or restoration data is improved, and more appropriate change factor information data can be generated.
  • the method of generating the change factor information according to another invention is a process of generating new change factor information data.
  • the origin of the obtained change factor information data is determined based on each signal element.
  • the kinetic energy is set to a position where the kinetic energy is Min or more and Min ⁇ 1.2 or less.
  • the method of generating change factor information according to another invention is data obtained by processing the original signal data is reduced data obtained by performing a reduction process for reducing the original signal, and arbitrary signal data is reduced data. And the same size data.
  • the method for generating the change factor information according to another invention includes an enlargement process for enlarging the new change factor information data after obtaining the new change factor information data from the reduced data.
  • the data of the change factor information that has been performed and expanded is used in place of the data of the new change factor information, and the processing similar to the processing of (1) to (3) is performed at least once, and the expansion processing is performed as necessary. The subsequent processing is repeated.
  • a method for generating change factor information according to another invention is obtained by obtaining new change factor information data from the reduced data, and then obtaining the obtained restored data and new data in (5). Performs enlargement processing to enlarge the data of the change factor information at the same ratio, uses the enlarged change factor information data instead of the new change factor information data, and uses the enlarged restoration data instead of any signal data.
  • the process similar to the process from (1) to (3) is performed once or more, and the processes after the enlargement process are repeated as necessary. By adopting this method, it is possible to estimate large change factor information data based on good quality change factor data obtained by performing the reduction process.
  • the method of generating change factor information according to another invention increases the value of the central part of the data of new change factor information. By adopting this method, it is possible to generate data of change factor information that places emphasis on estimation of changes corresponding to blurring.
  • the method of generating change factor information according to another invention is data having an intensity distribution of Gaussian distribution as the first change factor information data. By adopting this method, it is possible to estimate data of favorable change factor information regardless of any blur or blur in the change.
  • the signal processing apparatus of the present invention from the original signal data that has undergone changes such as degradation, the signal before the change, the signal that should have been originally acquired, or an approximate signal thereof (hereinafter, A processing unit for restoring the original signal), the processing unit sets arbitrary change factor information data as the first change factor information data, and (1) uses the data to generate an arbitrary signal. Comparison data is generated from the data, (2) the original signal data or the data obtained by processing the original signal data is compared with the comparison data, and (3) the obtained difference data is used as the first change factor.
  • the first restoration data is generated by allocating to the arbitrary signal data using the information data, and (4) the first restoration data is used in place of the arbitrary signal data, and if necessary, the same processing is performed.
  • a process for generating new change factor information data different from the first change factor information data from the restored data and the original signal data or the data obtained by processing the original signal data obtained as a result is performed.
  • the original signal is restored based on the generated data of the new change factor information or data obtained by processing the data of the change factor information.
  • the “restored data” in (5) includes both the first restored data and the restored data obtained by repeating the process as necessary in (4).
  • the process of restoring the original signal may be performed arbitrarily using the data of the change factor information or the data obtained by processing the data of the change factor information.
  • the comparison data is generated from the signal data, the original signal data to be processed is compared with the comparison data, and the obtained difference data is used as the change factor information data or the change factor information data.
  • the restoration data is generated by allocating the processed data to arbitrary signal data, and this restoration data is used in place of the arbitrary signal data, and the same processing is repeated. By adopting this configuration, more practical signal restoration is possible.
  • the processing of (3) is a part of the signal data constituting the original signal data, the data obtained by processing the original signal data, and the restoration data. Alternatively, it is performed by a method of moving all, and it is determined whether or not the signal element data constituting the restoration data satisfies a predetermined standard. If the predetermined standard is not satisfied, the difference data is used for calculation. The corrected distribution value is corrected, and correction processing is performed using the data obtained by the corrected distribution value instead of the unsatisfied data. By adopting this configuration, the quality of the first restoration data or restoration data is improved, and more appropriate change factor information data can be generated.
  • the signal processing device generates the new change factor information data by moving the data of each signal element to the origin position of the obtained change factor information data.
  • the minimum value of the total kinetic energy required for the above is Min
  • the kinetic energy is set to a position where the kinetic energy is Min or more and Min ⁇ 1.2 or less.
  • the signal processing apparatus generates the comparison data from arbitrary signal data by using the data of the change factor information that causes the processing unit to cause the signal change.
  • the original signal data to be processed is compared with the comparison data, and the obtained difference data is distributed to arbitrary signal data using the data of the change factor information in which the origin position is set.
  • This is means for generating reconstructed data, using the reconstructed data in place of arbitrary signal data, and repeatedly performing similar processing. If this repetitive processing is performed very many times, a signal such as an image in which no ringing has occurred can be restored. Therefore, by adopting this configuration, occurrence of ringing or the like can be more reliably suppressed.
  • FIG. 2 is a processing flowchart for explaining a processing routine according to a method for generating data of change factor information performed by a processing unit of the signal processing device shown in FIG. 1.
  • FIGS. 3A and 3B are diagrams illustrating a process for emphasizing the skeleton part of the PSF illustrated in FIG. 2, in which FIG. 2A illustrates an example of a PSF estimated before the process, and FIG. ing.
  • FIG. 3 is a flowchart for explaining a processing routine for determining validity of an update amount and correcting restored data shown in FIG. 2.
  • FIG. 5 is a diagram schematically illustrating an example of an update amount before and after performing correction related to the processing flow illustrated in FIG. 4.
  • FIG. 8 is an external perspective view showing an outline of the signal processing device shown in FIG. 1, for explaining which plane of the signal processing device 1 is indicated by the XY plane in FIG. 7. It is a figure explaining the process of step S112 of the processing routine which concerns on the production
  • FIG. 2 is a processing flowchart for explaining a processing routine related to an image restoration processing method (repetitive processing) performed by a processing unit of the signal processing device shown in FIG. 1.
  • FIG. 8 It is a figure for demonstrating the concept of the processing method shown in FIG. It is a figure for demonstrating concretely the processing method shown in FIG. 8 taking an example of camera shake, and is a table
  • FIG. 8 taking an example of camera shake, and is a figure for demonstrating the condition which produces
  • FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 using camera shake as an example, in which comparison data and a blurred original image to be processed are compared to generate difference data It is a figure for demonstrating.
  • FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 by taking an example of camera shake, and a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. is there.
  • FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 using camera shake as an example.
  • FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 by taking an example of camera shake, and a diagram for explaining a situation in which newly generated difference data is allocated and new restored data is generated. It is.
  • this signal processing device is a consumer camera, it may be a camera for other uses such as a surveillance camera, a television camera, a handy type video camera, an endoscopic camera, a microscope, a binocular, It can also be applied to devices other than cameras, such as diagnostic imaging apparatuses such as NMR imaging, printers that print images, and scanners that read images.
  • diagnostic imaging apparatuses such as NMR imaging, printers that print images, and scanners that read images.
  • FIG. 1 shows an outline of the configuration of the signal processing apparatus 1.
  • the signal processing device 1 includes an imaging unit 2 that captures an image of a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. Further, the signal processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4 and a factor information storage unit 7 that stores data of change factor information that causes image degradation and the like.
  • the photographing unit 2 is a part including a photographing optical system having a lens and a photographing element such as a CCD or a C-MOS that converts light passing through the lens into an electric signal.
  • the control system unit 3 controls each unit in the signal processing device, such as the photographing unit 2, the processing unit 4, the recording unit 5, and the factor information storage unit 7.
  • the processing unit 4 is composed of an image processing processor, and is composed of hardware such as ASIC (Application Specific Specific Integrated Circuit). Then, the processing unit 4 executes a process related to a data generation method of change factor information, which will be described later, and an image restoration process.
  • ASIC Application Specific Specific Integrated Circuit
  • the processing unit 4 may store image data that is a source for generating comparison data, which will be described later. Further, the processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software.
  • the recording unit 5 is composed of a semiconductor memory, but magnetic recording means such as a hard disk drive or optical recording means using a DVD or the like may be employed.
  • the factor information storage unit 7 is a recording unit that stores data of change factor information generated by the method of generating data of change factor information according to the embodiment of the present invention or data obtained by processing the data. Then, the factor information storage unit 7 stores a blurring locus (history) that forms the center of the data, and a time (weight) that has remained at each point on the locus.
  • the data recorded in the factor information storage unit 7 is from an original image (an image in which deterioration or the like has changed) to an original image (an image before the change, an image that should have been originally taken, or an approximation thereof) This is used by the processing unit 4 when the image is restored. Therefore, data can be exchanged between the processing unit 4 and the factor information storage unit 7.
  • FIG. 2 shows a flowchart of an example of a method for generating data of change factor information according to the embodiment of the present invention performed by the processing unit 4.
  • the processing unit 4 sets an arbitrary initial value (P 0 ) of the first change factor information data (PSF) (step S101).
  • the initial value is set as a Gaussian disk whose intensity distribution is Gaussian distribution data.
  • the processing unit 4 reduces the captured original image data (original signal data) Img ′ to 1 ⁇ 4 in parallel with the processing in step S101 or before or after the processing. That is, a reduction process (processing) is performed by thinning out the original signal data Img 'to a capacity value of 25% (step S102).
  • SImg ′ The reduced data that has undergone this reduction processing is denoted as SImg ′.
  • the processing unit 4 uses I 0 as arbitrary signal data having the same capacity as the reduced data SImg ′, and superimposes I 0 and P 0 to obtain blurred data I 0 ′ for comparison ( Step S103).
  • the arbitrary signal data I 0 is preferably reduced original image data SImg ′.
  • the processing unit 4 compares SImg ′ with I 0 ′, and obtains data ⁇ of the difference (Step S104). It is determined whether the difference data ⁇ is within a predetermined value (step S105). If it is within the predetermined value, the process proceeds to step S120, and if not within the predetermined value, the process proceeds to step S106.
  • step S106 data obtained by multiplying the difference data ⁇ by a coefficient k (coefficient based on the change factor information (PSF)) is distributed to I 0 to obtain first restored data I 0 + n (this time I 0 + 1 ). .
  • processing for determining the validity of the distribution value (update amount) is performed (step S107).
  • step S108 the process which corrects I0 + n according to the validity is performed.
  • step S109 it is determined whether or not this process has reached a predetermined number of times. If the predetermined number (for example, 10 times) has been reached, the process proceeds to step S120.
  • step S110 the processing unit 4 performs the process of step S103 using I 0 + 1 instead of I 0 (step S110), and performs steps S103, S104, S105, S106, S107, S108, S109, and S110. This process is repeated a predetermined number of times (10 times this time). In step S109, it is determined whether or not the predetermined number of times has been reached.
  • step S106 is a process of moving (distributing) part or all of the pixel data constituting the reduced original image data SImg ′ and the restored data I 0 + n . Details of step S107 and step S108 will be described later.
  • the processing unit 4 performs a process of calculating a PSF (step S120).
  • FIG. 3A shows the PSF estimated at the current stage on the XY plane.
  • This XY plane corresponds to an XY plane in which the signal processing apparatus 1 shown in FIG.
  • the PSF is composed of a portion (skeleton portion) A corresponding to the locus of blur that forms the center of the data, and a portion B (portion such as blur) having a small data value around the portion. This small portion B is considered as a blurred portion.
  • FIG. 3B shows the state of the PSF after performing the process of emphasizing the portion A (step S121).
  • step S120 the P data obtained in step S120 becomes P ′ data.
  • step S122 a process of setting the origin of P ′ so that the kinetic energy is minimized (details will be described later) is performed (step S122).
  • the processing unit 4 determines whether or not P ′ is a valid PSF. Specifically, the processing unit 4 determines whether or not the processing from step S120 to S122 has been repeated three times (step S123). If the number of repetitions, that is, the number of PSF calculations does not satisfy 3 (N), P ′ data is used instead of P 0 (step S124), and the processing from step S103 to S110 is repeated, and then again. Steps S120 to S122 are executed.
  • step S125 P ′ obtained last and I 0 + n are multiplied by 4/3. It is determined whether or not the enlargement process has reached a predetermined reduction rate, that is, whether or not the original image has been reduced (step S126). If it is determined that the predetermined reduction ratio has not been reached, the enlarged I 0 + n data is used in place of the I 0 data (step S127), and the enlarged P ′ data is used in place of the P 0 (step S124). .
  • a new reduced image SImg ′ obtained by reducing the original image data to 1/3 is obtained.
  • the image reduced to 1/3 is the same size as the data 4/3 times earlier.
  • the process part 4 repeats the process from step S103 to S110 using each new data, and repeats the process from step S120 to S124. If it is determined in step S126 that the predetermined reduction rate, for example, a reduction rate of 1/3 with respect to the original image data has been reached, the process proceeds to step S128.
  • the obtained P ′ data and I 0 + n data are enlarged at the same ratio, but only the P ′ data is enlarged, and instead of the enlarged I 0 + n data.
  • Arbitrary signal data may be used for step S125.
  • the arbitrary signal data is, for example, SImg ′ data.
  • step S125 the direction which the data of the data and I 0 + n of the resultant P 'as in the present embodiment by enlarging processed in the same ratio, for using data of the I 0 + n passing through the restoration process, good quality PSF May be generated.
  • the P ′ data enlargement process in step S125 is performed by the processing unit 4 inserting new signal element data having an average value of adjacent signal element values between adjacent signal elements. Further, the process of enlarging the I 0 + n data is performed by the processing unit 4 inserting a new pixel between adjacent pixels. The pixel value of the new pixel is a value obtained by averaging the pixel values of adjacent pixels. The processing unit 4 determines whether or not a predetermined reduction rate has been reached by this enlargement processing in step S126. When the predetermined reduction ratio is reached by the enlargement process (Y), the processing unit 4 enlarges P ′ obtained at the current stage to the actual size (step S128).
  • the processing unit 4 enlarges P ′ obtained at the current stage three times. This enlargement process is also performed by the processing unit 4 inserting a new pixel between adjacent pixels.
  • the pixel value of the new pixel is a value obtained by averaging the pixel values of adjacent pixels.
  • the processing unit 4 inserts a pixel having a pixel value of one of the adjacent pixels between adjacent pixels, in addition to the interpolation method described above. The interpolation method performed by the above may be adopted.
  • step S129 the method for generating data of change factor information according to the embodiment of the present invention is completed (step S129), and a PSF is generated.
  • the PSF is stored in the factor information storage unit 7 by the processing unit 4 and used in the signal restoration process.
  • the PSF obtained through the above process is represented as “G”.
  • FIG. 4 and 5 are diagrams for explaining the details of the processing in steps S107 and S108 in FIG.
  • the basic concept of this processing is as follows. This process is not performed for the restored data I 0 + n in a lump, but for each signal element constituting the restored data I 0 + n . This point is greatly different from the processing of the entire signal (image) in steps S103, S105, and S106.
  • the update amount based on the difference data ⁇ of the corresponding portion is unlikely to be an appropriate amount. This is because the difference between the pixel values of two pixels straddling the edge part is too large, so even if the pixel value is distributed from one pixel to the other pixel across the edge part, the distribution is appropriate. This is because it is difficult to. Therefore, for pixels whose change in pixel value is unnaturally larger than the surroundings due to the update, the update amount in the vicinity of the edge is brought close to an appropriate value by reducing the absolute value of the update amount.
  • the partial pixels (reference pixels) may be simply a plurality of adjacent pixels or a plurality of pixels included in a predetermined distance from a predetermined pixel as a center. The predetermined distance is determined using data G of change factor information, for example.
  • Step S214 is a step for obtaining a correction value Ia 'by the update amount Dp after correction. The arrow shown in FIG.
  • step S205 schematically shows the state after the processing of step S205 and step S214
  • the black circle of the arrow indicates the pixel value Ib before the pixel update
  • the triangular arrow indicates Dp Indicates the corrected value Ia ′ after updating
  • the left arrow indicates an example when the value of the update amount Dc before correction is a positive value
  • the right arrow indicates an example when the value of the update amount Dc before correction is a negative value.
  • the update amount Dc is changed to a value Dp obtained by subtracting the pixel value Ib before update from the maximum value Max (step S207).
  • the pixel value of the correction value Ia ′ becomes equal to the maximum value Max.
  • the arrow shown in FIG. 5B schematically shows the state after step S207 and step S214, as in FIG.
  • the left arrow indicates Ia when updated by Dc
  • the right arrow indicates a correction value Ia 'when updated by Dp.
  • the update amount Dc is changed to a value Dp obtained from the following equation (2) (step S209).
  • the pixel value of the correction value Ia ′ is a value obtained by adding 1 ⁇ 4 of the updated pixel value Ia exceeding the maximum value Max to the plus side to the maximum value Max.
  • the arrow shown in FIG. 5C schematically shows the state after step S209 and step S214, as in FIG. 5B.
  • Dp 0.25 (Ia ⁇ Max) + (Max ⁇ Ib) (2)
  • the update amount Dc is changed to a value Dp obtained from the following equation (3) (step S211).
  • the pixel value of the correction value Ia ′ is a value obtained by subtracting 1 ⁇ 4 of the updated pixel value Ia exceeding the minimum value Min to the minus side from the minimum value Min.
  • the arrow shown in (D) of FIG. 5 schematically represents the state after step S211 and step S214, as in FIG. 5 (B).
  • Dp ⁇ (Ib ⁇ Min) ⁇ 0.25 (Min ⁇ Ia) (3)
  • the arrow shown in (E) of FIG. 5 schematically represents the state after step S213 and step S214, as in FIG. 5 (B).
  • the determination processing shown in the above steps S204, S206, S208, and S210 is processing for determining whether or not a predetermined standard is satisfied. It should be noted that the order of determination based on the determination in steps S204, S206, S208, and S210 and whether or not the condition in step S212 is satisfied can be changed as appropriate. Even when the change is made, the first to fourth judgment is actually made. If none of the four conditions is met, the remaining conditions will always be met. Therefore, whether the remaining conditions are met is determined by whether or not the predetermined criteria are satisfied as in step S212 in FIG. It is not a judgment process.
  • step S214 the corrected value Ia ′ is obtained (step S214).
  • the factor information storage unit 7 stores the data at the stage of step S122 in FIG. That is, data that can be estimated as blurring of the signal processing device 1 shown in FIG. FIG. 6 shows the appearance of the signal processing apparatus 1 shown in FIG.
  • the rotation of the signal processing apparatus 1 about the XYZ direction axes shown in the drawing is blurred, but it is particularly likely to appear around the X and Y direction axes.
  • the data relating to the blur shown in FIG. 7 is data of change over time of the coordinate data on the XY plane in FIG.
  • the data includes information on the locus of blur as represented by the XY plane shown in FIG. 7 and information on how long it has been at each position on the locus.
  • the starting point A (X 1 , Y 1 ) on the XY plane shown in FIG. 7 is the imaging start position
  • the end point B (X N , Y N ) of the trajectory is the imaging end position.
  • Degradation of the image due to blurring is a phenomenon in which the light energy is not concentrated on one point but is scattered on the locus AB shown in FIG. Therefore, concentrating the dispersed light energy at one point restores the original image to the original image.
  • One point for concentrating the light energy can be freely determined. For example, it is possible to determine the points A, B, AB on the locus in FIG. 4 or points off the AB locus.
  • the point where the dispersed light energy is concentrated is referred to as an “origin position”, and the origin position is represented by a point 0 coordinate (0x, 0y) on the XY plane shown in FIG.
  • data (P ′) in which the skeleton part of the PSF calculated in step S120 described above is emphasized in the process of step S121 is represented by G (Xn, Yn).
  • G (Xn, Yn) This indicates “weight”, which is information indicating how long the position (Xn, Yn) has been, and satisfies the equation (4).
  • Equation (4) indicates that light energy is normalized to 1 and handled. Note that (Xn, Yn) is a coordinate on the XY plane shown in FIG. ... (4)
  • the origin position (0x, 0y) having the minimum value of the movement energy E (0x, 0y) is set.
  • This setting is performed by the processing unit 4. Further, the processing unit 4 stores the data in which the origin position is newly set in place of the data stored in the factor information storage unit 7 before the setting.
  • the restoration process of the original image to the original image when the imaging power is turned off, when the processing unit 4 is not operating, when the operating rate of the processing unit 4 is low, etc. It can be a time delayed from the time when the original image was taken.
  • the original image data stored in the recording unit 5 and the change factor information such as a transfer function for the original image stored in the factor information storage unit 7 are associated with each other for a long time. It is preserved over.
  • the advantage of delaying the time for executing the restoration process of the original image from the time of shooting the original image is that the burden on the processing unit 4 at the time of shooting involving various processes can be reduced.
  • I 0 is an arbitrary initial image and is image data stored in advance in the recording unit of the processing unit 4.
  • I 0 ′ indicates data of a degraded image of I 0 of the initial image data, and is comparison data for comparison.
  • Img ′ is data of the original image.
  • ⁇ is difference data between the original image data Img ′ and the comparison data I 0 ′.
  • K is an allocation ratio based on the data of the change factor information.
  • I 0 + n is restored image data (restored data) newly generated by allocating the difference data ⁇ to the initial image data I 0 based on the data G of the change factor information.
  • Img is data of the original image.
  • the relationship between Img and Img ′ is represented by the following equation (8).
  • Img ′ Img * G (8)
  • * is an operator representing a superposition integral.
  • the difference data ⁇ may be a simple difference between corresponding pixels, but in general, the difference data ⁇ differs depending on the data G of the change factor information, and is expressed by the following equation (9).
  • ⁇ f (Img ′, Img, G) (9)
  • the processing routine of the processing unit 4 first determines the origin position where the dispersed light energy is concentrated (step S300). This determination has already been made in step S122 shown in FIG. Therefore, detailed description of this process is omitted. Then, prepare the arbitrary image data I 0 (step S301). As the initial image data I 0 , the deteriorated original image data Img ′ may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. good. In step S302, data I 0 of an arbitrary image that is an initial image is input instead of Img in equation (8), and comparison data I 0 ′ that is a degraded image is obtained. Next, the original image data Img ′ and the comparison data I 0 ′ are compared to calculate difference data ⁇ (step S303).
  • step S306 the validity of the update amount (distribution value) used when generating the restored data I 0 + n is determined (step S306), and the restored data I 0 + n is corrected (step S307).
  • the processing in step S306 and step S307 is performed in the same manner as the processing for determining and correcting the validity of the distribution value performed in steps S106 and S107 shown in FIG. Therefore, detailed description of this process is omitted.
  • steps S302 to S307 in FIG. 8 are repeated.
  • the restored data I 0 + n in the middle of this repetition is restored data in the middle of processing.
  • step S304 when each absolute value of the difference data ⁇ of each pixel becomes less than a predetermined value, the iterative process is terminated. Then, the restored data I 0 + n at the time when the repetitive processing is completed is estimated as the original image data Img. That is, when the maximum value or average value of the absolute values of the difference data ⁇ of each pixel becomes smaller than a predetermined value, the restored data I 0 + n that is the basis of the comparison data I 0 + n ′ is the original image data Img. Therefore, the restored data I 0 + n is estimated as the original image data Img.
  • the initial image data I 0 and the change factor information data G may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary.
  • the present embodiment assumes the following conditions. That is, (1) The output corresponding to the input is uniquely determined. (2) If the output is the same, the input is the same. (3) The input is updated so that the outputs are the same, and the solution is converged by performing iterative processing while correcting the update amount to an appropriate value.
  • comparison data I 0 ′ (I 0 + n ′) that is approximate to the original image data Img ′ can be generated, the original data of the generation is generated.
  • the initial image data I 0 or the restored data I 0 + n is approximate to the original image data Img.
  • the value that is the determination criterion for the difference data ⁇ is “6” in this embodiment when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished.
  • FIG. 8 details of the camera shake restoration processing method (repetitive processing (restoration means) of steps S302, S303, S304, S305, S306, and S307) shown in FIG. 8 will be described in detail with reference to FIG. 10, FIG. 11, FIG. This will be described with reference to FIGS. 14, 15, 16 and 17.
  • FIG. 8 details of the camera shake restoration processing method (repetitive processing (restoration means) of steps S302, S303, S304, S305, S306, and S307) shown in FIG. 8 will be described in detail with reference to FIG. 10, FIG. 11, FIG. This will be described with reference to FIGS. 14, 15, 16 and 17. FIG.
  • Image restoration algorithm When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is a camera shake, the light energy is distributed to the blurred pixels during the exposure time. Further, if the blur during the exposure time is known, the manner in which the energy is dispersed during the exposure time can be understood, so that it is possible to create a blur-free image from the blurred image.
  • the pixels are designated as S-1, S, S + 1, S + 2, S + 3,...
  • S-1, S, S + 1, S + 2, S + 3,... In order from the left, and attention is paid to a certain pixel S.
  • the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG.
  • the imaging results at this time are shown in the table of FIG.
  • What is shown in FIG. 11 is the correct image data Img when no deterioration occurs.
  • Each data is represented by 8-bit (0 to 255) data.
  • the movement energy E (0x, 0y) is calculated.
  • the movement energy is E (0x).
  • the moving energy can be set to the minimum value “0.7” by concentrating the dispersed light energy on the pixel “S” or “S1”.
  • the sum of the kinetic energy to the pixel “S + 1” is the smallest. That is, the movement to the pixel “S” is “0.8”, the movement to the pixel “S + 1” is “0.7”, and the movement to the pixel “S + 2” is “1.2”.
  • ⁇ Blurry is uniform for all pixels and is recognized as a linear problem. If there is no upper blur (vertical blur), the blur situation is as shown in the table of FIG.
  • the data shown as “blurred image” in FIG. 13 becomes the degraded original image data Img ′.
  • “120” of the pixel “S-3” is in accordance with the distribution ratio of “0.5”, “0.3”, and “0.2” of the data G of the change factor information that is blur information. Dispersed in such a manner that “60” is distributed to the “S-3” pixel, “36” is distributed to the “S-2” pixel, and “24” is distributed to the “S-1” pixel.
  • “input” corresponds to the initial image data I 0 .
  • This data I 0, that is, Img ′ is superposed and integrated with the change factor information data G in step S302. That is, for example, “60” of the “S-3” pixel of the initial image data I 0 is “30” for the S-3 pixel, “18” for the “S-2” pixel, “12” is assigned to each pixel of “ ⁇ 1”.
  • the other pixels are similarly distributed, and comparison data I 0 ′ shown as “output I 0 ′” is generated.
  • “15” multiplied by “S-3” is distributed to the pixel “S-3”
  • “S-2” pixel data “15” is allocated to the “S-2” pixel.
  • “4.5” multiplied by 0.3 is allocated, and further, the data “9.2” of the pixel “S-1” is allocated to the pixel “S-1” by the distribution ratio.
  • “1.84” multiplied by a certain 0.2 is allocated.
  • the total amount (update amount Dc for each pixel) allocated to the pixels of “S-3” is “21.34”, and this value is the initial image data I 0 in FIG.
  • the updated pixel value Ia in FIG. 4 which is the restored data I 0 + 1 in FIG. 8 is calculated in addition to the original image data Img ′.
  • the updated pixel value Ia is “81.34”.
  • the difference data ⁇ is distributed to the arbitrary image data I 0 using the change factor information data G to generate the restored data I 0 + n shown as “next input” in FIG. .
  • this since this is the first time, it is represented as I 0 + 1 in FIG.
  • step S205 is performed as shown in FIG. 15, and the update amount “21.34” as Dc is used as it is as the update amount Dp after correction, and the correction value Ia ′ is equal to “81 before correction” “81 .34 ".
  • the same correction is performed on the updated pixel values Ia of “S ⁇ 2” to “S”, “S + 2”, and “S + 3”.
  • This correction value Ia ′ becomes the correction value of the restored data I 0 + n .
  • step S208 In order to satisfy “Ib (121.00)> Av (113.33)” and satisfy the condition “Ia (130.11)> Max (121.00)”, the condition of step S208 in FIG. You will be satisfied. Accordingly, the process of step S209 is performed as shown in FIG. 15, and “2.28” is used as the updated amount Dp after correction, and the correction value Ia ′ is “123.28”. The same correction is performed for Ia of the pixel “S + 4”. This Ia ′ becomes a correction value of the restored data I 0 + n .
  • the process proceeds to step S303 to obtain new difference data ⁇ .
  • the size of the difference data ⁇ is determined in step S304. If the difference data ⁇ is larger than the predetermined value, the new difference data ⁇ is distributed to the previously modified restored data I 0 + 1 in step S305 to generate new restored data I 0 + 2 . In this case, the new restoration data I 0 + 2 is corrected in the same manner as described in FIG. 15 (see FIG. 17).
  • the updated pixel value Ia of the pixels “S-3”, “S”, and “S + 3” is corrected in the same manner as the pixel “S-3” in FIG. That is, the updated pixel value Ia becomes the corrected value Ia ′ as it is.
  • step S208 the process of step S209 is performed as shown in FIG. 17, and the correction value Ia ′ is “122.11” using the value “1.16” according to the above-described equation (2) as the update amount Dp after correction. .
  • the same correction is performed on the updated pixel value Ia of the pixel “S + 4”.
  • This correction value Ia ′ becomes the correction value of the restored data I 0 + n .
  • step S210 For example, in the pixel “S-2”, the conditions of steps S204, S206, and S208 are not satisfied, and the process proceeds to step S210. Then, “Ib (77.30) ⁇ Av (87.67)” is satisfied, and the condition “Ia (76.97) ⁇ Min (77.30)” is satisfied, so the condition of step S210 in FIG. 4 is satisfied. Will be. Therefore, as shown in FIG. 17, the process of step S211 is performed, and the value “ ⁇ 0.082” according to the above equation (3) is used as the updated amount Dp after correction, and the correction value Ia ′ is “77.22”. Become. The same correction is performed on the updated pixel value Ia of the pixels “S ⁇ 1” and “S + 2”. This correction value Ia ′ becomes the correction value of the restored data I 0 + n .
  • step S302 by performing the step S302 by using the restored data I 0 + 2 that are fixed, a new comparison data I 0 + 2 'from the restored data I 0 + 2 that are fixed are generated.
  • steps S302 and S303 are executed, the process proceeds to step S304, and the process proceeds to step S305 based on the determination. Such a process is repeated.
  • the processing unit 4 can perform a process for determining and correcting the validity of the distribution value (steps S107 and S108), thereby suppressing an extreme change in the value of the data due to the distribution. PSF is generated. Further, during this process, the processing unit 4 performs a process of resetting the origin of P ′, whereby an extreme change in the value of data due to the distribution can be suppressed, and a more appropriate PSF is generated. Further, during this process, the processing unit 4 executes the reduction process (step S102), whereby the magnitude of the change is also reduced. Therefore, it becomes easier to estimate the PSF, and a more appropriate PSF is generated. .
  • the processing unit 4 repeatedly performs the enlargement process (step S125) by gradually increasing the enlargement ratio, thereby performing the process based on the PSF having an appropriate skeleton obtained by performing the reduction process. A high quality and large PSF can be estimated. Further, during this process, the processing unit 4 performs a process (step S121) of emphasizing the skeleton part of the PSF, thereby generating a PSF that puts weight on estimation of a change corresponding to blurring. In this process, the processing unit 4 sets a PSF initial value to a Gaussian disk (step S101), and estimates a good PSF regardless of any blur or blur. be able to.
  • the signal processing apparatus 1 can generate an appropriate PSF even if the PSF is unknown, and can perform practical signal restoration. Therefore, a speed sensor or an acceleration sensor that mechanically measures blur or the like can be omitted from the components of the signal processing apparatus 1.
  • the signal processing apparatus 1 repeats Steps S302 to S307 shown in FIG. 8 so that the difference data ⁇ gradually decreases.
  • the difference data ⁇ becomes smaller than a predetermined value
  • the original image data Img without blur is obtained.
  • the correction process (FIGS. 4 and 5) is performed, the image data estimated as the original image data Img obtained is reduced in the occurrence of ringing and has a good image restoration state. Become.
  • the correction process (FIGS. 4 and 5) corrects unnatural data among the signal element data constituting the restored data and suppresses a large change in the pixel value. Even if the data G of the change factor information estimated by the processing shown in (2) is low in reliability, it is possible to restore an appropriate image.
  • the processing unit 4 repeats the processing of steps S103 to S108 shown in FIG. 2 twice or more, but it can be performed only once. Furthermore, the processing unit 4 determines and corrects the validity of the distribution value (steps S107 and S108), resets the origin of P ′ (step S122), reduces (step S102), and expands (step S102).
  • step S125 all or part of the enlargement process (step S128) and the PSF skeleton part enhancement process (step S121) to be finally the same size as the PSF data for the initial image data, Can be omitted.
  • the initial value of PSF can be set to an arbitrary value without being set in the Gaussian disk (step S101).
  • the signal restoration process in the signal processing apparatus 1 employs the process shown in FIG. 8, but other processes such as a process using a Wiener filter can be employed. Even when the process shown in FIG. 8 is adopted, both or one of the process for setting the origin position (step S300) and the process for determining and correcting the validity of the update amount (steps S306 and S307) are omitted. Can do. Further, even when the process of determining and correcting the validity of the update amount (steps S306 and S307) is adopted, the method of correcting the restored data I 0 + n is not limited to the methods shown in FIGS. Particularly, the method of correcting the difference data ⁇ in steps S205, S207, S209, S211 and S213 can be changed as appropriate according to the case in steps S204, S206, S208, S210 and S212 shown in FIG.
  • the processing unit 4 repeats the processing of steps S103 to S108 shown in FIG. 2 ten times. In step S123, the processing unit 4 repeats steps S103 to S122. You can increase or decrease the number of times depending on the situation. Moreover, it is good also as the user of the signal processing apparatus 1 being able to set those frequency
  • the reduction / enlargement method in the reduction process (step S102) and the enlargement process (step S125) is performed by thinning out pixels and inserting a new pixel having a value obtained by averaging the pixel values of adjacent pixels.
  • other means for example, a reduction process using a pixel having an average pixel value of a plurality of adjacent pixels instead of the plurality of pixels, a pixel value of an adjacent pixel as it is as a pixel value of a new pixel, and the new pixel You may perform an expansion process by inserting.
  • the value of the skeleton part data is doubled, but may be 1.5 times, 3 times, 4 times, 5 times, or the like.
  • the processing unit 4 performs an enlargement process (step S128) to finally make the data the same size as the PSF data for the initial image.
  • this process does not require data having the same size as the PSF data for the initial image, and can be enlarged to data having a size different from that of the PSF data for the initial image.
  • the processing unit 4 performs a process of calculating a PSF (step S120).
  • the obtained restored data I 0 + n and the original image data Img ′ are each Fourier-transformed, and the PSF is obtained by division in the frequency space.
  • one pixel surrounding the affected range is included in addition to the affected range (affected range). It may be a large range or a range of a predetermined distance centering on a pixel to be corrected.
  • the maximum value, minimum value, and average value of the reference pixel are not used, but only the maximum value and the minimum value are used, and the correction value exceeds the maximum / minimum value. You may make it add the value of 1/4 or 1/3 to the maximum value or the minimum value.
  • the upper limit is set to 1.2 times the maximum value and the lower limit is set to 0.8 times the minimum value. That is, if the updated pixel value Ia is in the range of X times the maximum value and Y times the minimum value, the pixel value Ia may not be corrected.
  • the processing unit 4 determines whether to process the image once obtained in step S105 in FIG. 2 or step S304 in FIG. 8 for each of a plurality of pixels constituting the image. It is determined whether or not the absolute values of the difference data ⁇ and ⁇ are all less than a predetermined value or the average value of the absolute values is less than a predetermined value, and whether or not the entire image is to be processed.
  • the comparison target with the predetermined value may be the difference data for each of the plurality of pixels constituting the image, and it may be determined whether or not the repeated processing is stopped for each pixel.
  • the comparison target with the predetermined value may be the sum of the difference data ⁇ of each pixel, or the sum of the absolute values of the difference data ⁇ and ⁇ of each pixel, or two or more of the above four. it can. For example, it is determined whether or not the difference data ⁇ and ⁇ for each pixel farthest from zero and the sum of the difference data ⁇ for each pixel satisfy different criteria. You may do it. As described above, by appropriately selecting a value to be compared with the predetermined value, appropriate processing can be performed according to the type of original image, the state of change, or the status of restoration processing.
  • the restoration target is image data.
  • these restoration processing concepts and techniques can be applied to any digital data restoration processing.
  • it can be applied to restoration of digital audio data.
  • it is possible to efficiently suppress the occurrence of some inaccurate audio data, such as ringing, and even if the data of the change factor information is inaccurate, a restoration process that can obtain reasonable results Is possible.
  • the signal processing device 1 is a consumer camera.
  • the signal processing device 1 uses image data captured by a digital camera or the like as shown in FIG. 2, FIG. 4, FIG. 8, FIG. A printer device that performs printing after executing one or more of the processes shown in FIG.
  • the signal processing apparatus 1 is a computer in which software for operating a printer device while performing one or more of the processes shown in FIGS. 2, 4, 8, and 9 is installed.
  • FIG. 8, FIG. 9, or the like may be a computer installed with software that executes one or more of the processes shown in FIG.
  • each processing method described above may be programmed.
  • the program may be stored in a storage medium, such as a CD, DVD, or USB memory, and read by a computer.
  • the signal processing device 1 may be a program stored in the storage medium placed in an external server of the signal processing device 1 and downloaded and used as necessary.
  • the signal processing apparatus 1 has communication means for downloading the program in the storage medium.
  • the processing performed by the processing unit 4 is configured by software, but each of them is composed of parts that perform part of the processing. You may comprise with hardware.
  • the change factor information data G includes not only the deterioration factor information data but also information for simply changing an image and information for improving an image contrary to deterioration.
  • step S109, step S123, step S304 when the number of processing iterations (step S109, step S123, step S304) is set automatically or fixedly on the signal processing device 1, the set number of times is set as the PSF value (P, P ′). ) Or data G of change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of iterations may be increased, and when the variance is small, the number of iterations may be decreased.
  • the processing may be stopped when the difference data ⁇ , ⁇ diverges during the iterative processing, or when the energy of the image data after the energy has moved does not decrease but increases.
  • a method of determining whether or not the light is diverging can be determined by looking at the average value of the difference data ⁇ and ⁇ and determining that the average value is larger than the previous value.
  • the process may be stopped. For example, in the case of 8 bits, if the value to be changed is a value exceeding 255, the processing is stopped.
  • the value when an input that is new data is to be changed to an abnormal value, the value may not be used but may be set to a normal value. For example, when a value exceeding 255 in 8 bits of 0 to 255 is set as input data, it is processed as a maximum value of 255.
  • Table 1 shows the evaluation of restored images that have undergone the above-described processes.
  • a conventional example is an image (Img ') that has been photographed and remains degraded due to blurring or the like.
  • the PSF data obtained by omitting the predetermined steps shown in Table 1 from the processing shown in FIG. 2 is used, and the signals are restored by omitting the predetermined steps shown in Table 1 from the processing shown in FIG.
  • the images obtained in this way are referred to as Examples 2 to 12.
  • the twelfth embodiment is an image using a Wiener filter for the image restoration process instead of the process of FIG.
  • an image of a PSF obtained by measuring blur using a speed sensor instead of performing the process of FIG. 2 was also examined.
  • Example 2 Since the conventional example is a deteriorated image, blurring and blurring were clearly confirmed. In Example 1, neither blurring, ringing, or blur was observed. The blur disappeared because the information (part A in FIG. 3) corresponding to the blur information forming the center of the PSF was appropriate. It is considered that the blur has disappeared due to information other than information corresponding to the blur information that forms the center of the PSF (part B in FIG. 3). In the comparative example, since the PSF in the portion B in FIG. 3 cannot be obtained, this effect is not obtained and the blur cannot be eliminated.
  • Example 2 is an image in which the process of steps S103 to S108 in FIG. 2 is performed only once. In this image, blurring and blurring were improved as compared with the conventional image. In addition, there was a tendency that blurring and blurring were more improved as the number of repetitions of steps S103 to S108 in FIG. 2 was larger.
  • Example 3 is an image processed by omitting steps S107, S108, and S122 in FIG. In this image, blurring and blurring were improved as compared with the conventional image.
  • Example 4 is an image processed by omitting steps S107 and S108 in FIG.
  • Example 5 is an image processed by omitting step S122 in FIG. In these images, no blur or blur was observed.
  • Example 6 is an image obtained by omitting steps S102, S125, and S126 in FIG. 2 and gradually processing the image without enlarging the image.
  • Example 7 is an image in which step S123 in FIG. 2 is omitted and the number of repetitions is one. These images were improved in blurring and blurring as compared with the conventional image.
  • Example 8 is an image processed by omitting step S121 in FIG. In this image, some blurring was observed, but no blur was observed except for blurring. In order to obtain the effect of eliminating this blur, the process of step S121 in FIG. 2 is omitted or reduced (for example, the data of the skeleton part is multiplied by 1.5). In order to further obtain the effect of eliminating blurring, the processing in step S121 in FIG. 2 is further emphasized (for example, the data of the skeleton part is tripled).
  • Example 9 is an image processed by omitting steps S300, S306, and S307 in FIG. In this image, no blur or blur was observed, but ringing was observed. However, it has been confirmed that the occurrence of ringing is reduced or not observed by performing the repeated processing in which steps S300, S306, and S307 in FIG. 8 are omitted extremely many times (for example, 100 times or more).
  • Example 10 is an image processed by omitting step S300 in FIG. In this image, no blur or blur was observed, and no ringing was observed.
  • Example 11 is an image processed by omitting steps S306 and S307 in FIG. In this image, no blur or blur was observed, but some ringing was observed. That is, it is understood that the occurrence of ringing can be suppressed by executing any one of the process of determining the origin position (step S300) and the process of determining the validity of the update amount and correcting I 0 + n (steps S306 and S307). It was. Moreover, it turned out that the ringing suppression effect by step S306, S307 is a little superior to the ringing suppression effect by the process of step S300.
  • Example 12 is an image obtained by omitting all the processes in FIG. 8 and performing image restoration processing using a Wiener filter instead. In this image, ringing was observed although blurring and blurring were improved as compared with the conventional example.
  • step S300 processing in which steps S306 and S307 in FIG. 8 were omitted and the conditions in step S300 were changed was examined.
  • the pixel “S” is 0.85
  • the pixel “S + 1” is 0.3
  • the pixel “S + 2” is 0.25
  • the pixel “S”, “S + 1”, and “S + 2” are distributed.
  • Each of the light energy thus concentrated was used as the origin position, and the repetition process shown in FIG. 8 was performed.
  • the restored images in each of these cases were visually observed to determine the presence or absence of ringing.
  • the process in which the conditions of S300 in FIG. 8 are changed can be variously changed.
  • the origin position (0x, 0y) that minimizes the total sum of the kinetic energy E (0x, 0y) is set, but the origin that exceeds the minimum value of the kinetic energy E (0x, 0y) and falls below a predetermined value
  • the position (0x, 0y) may be set.
  • S 0.45
  • S + 1 0.3
  • not the pixel “S + 1” having the smallest kinetic energy but the pixel “S” is set to the origin position. (0x, 0y) may be used.
  • the kinetic energy at this time is 0.8 for the pixel “S”, 0.7 for the pixel “S + 1”, and 1.2 for the pixel “S + 2”.
  • the pixel “S” has a value smaller than 0.84, which is a value obtained by multiplying the minimum value 0.7 by 1.2.
  • the origin position is any position on the XY plane in FIG. Therefore, the origin position may be determined within a range on the trajectory AB in FIG. That is, for example, any one of the pixels “S”, “S + 1”, and “S + 2” in FIG. 12 that are within the range of the AB trajectory in FIG. 7 or the pixel “S ⁇ 1” or “S + 3” is the origin position It is preferable to do so if the total amount of kinetic energy is reduced. Further, a blur locus projected on the X-axis or Y-axis with the blur locus represented by the XY plane in FIG. 7 is defined as the blur locus, and the minimum value of the kinetic energy on the locus or the value as in the above-described embodiment. You may make it obtain

Abstract

Provided are a method of creating data relating to change factor information and a signal processor which allow practical signal restoration even when the data relating to the change factor information is unknown. The method of creating the data relating to change factor information sets data relating to an optional change factor information to be data relating to first change factor information, (1) by use of the data, creates data for comparison from optional signal data, (2) compares data relating to an original signal or data obtained by processing the data relating to the original signal with the data for comparison, (3), distributes the data relating to the obtained difference to the optional signal data by use of the data relating to the first change factor information to create first restoration data, (4) repeats similar processing by use of the first restoration data instead of the optional signal data according to necessity, (5) and from the resultant restoration data and data relating to the original signal or the data obtained by processing the data relating to the original signal, creates data relating to novel change factor information different from the data relating to the first change factor information.

Description

変化要因情報のデータの生成法および信号処理装置Method for generating data of change factor information and signal processing apparatus
　本発明は、変化要因情報のデータの生成法および信号処理装置に関する。 The present invention relates to a method of generating data of change factor information and a signal processing device.
　従来から、カメラ等の信号処理装置で撮影した際には、信号には時々劣化が生ずることが知られている。信号劣化の要因としては撮影時の手ブレ、光学系の各種の収差、レンズの歪み等がある。 Conventionally, it has been known that when a picture is taken by a signal processing device such as a camera, the signal sometimes deteriorates. Factors that cause signal degradation include camera shake during shooting, various aberrations of the optical system, lens distortion, and the like.
　本出願人は、過去に、撮影時の手ブレによって劣化した画像（信号）を復元する方法を提案した（特許文献１参照）。この特許文献１記載の技術は、信号を処理する処理部を有する信号処理装置において、処理部は、信号変化の要因となる変化要因情報のデータを利用して、任意の信号のデータから比較用データを生成し、処理対象となる原信号のデータと上記比較用データとを比較し、得られた差分のデータを利用して復元データを生成し、この復元データを上記任意の信号データの代わりに使用し、同様の処理を繰り返す繰り返し処理を行うことで、変化する前の原信号に近似する復元データを生成する処理を行うものである。 In the past, the present applicant has proposed a method for restoring an image (signal) that has deteriorated due to camera shake during shooting (see Patent Document 1). In the technology described in Patent Document 1, in a signal processing apparatus having a processing unit that processes a signal, the processing unit uses data of change factor information that causes a signal change, and compares the data from an arbitrary signal. Data is generated, the original signal data to be processed is compared with the comparison data, and the restored data is generated using the obtained difference data. The restored data is used instead of the arbitrary signal data. In this case, the same process is repeated, and the process of generating restored data that approximates the original signal before the change is performed.
　また、撮影時の手ブレ等によって劣化した画像を唯一の情報として、そのブレの原因となる点像分布関数（ＰＳＦ：Ｐｏｉｎｔ　Ｓｐｒｅａｄ　Ｆｕｎｃｔｉｏｎ）を推定し、元の物体画像を求める技術が提案されている（非特許文献１参照）。この技術は、被写体の変化前の画像データ（Ａ０）にＰＳＦ（Ｇ０）を乗じると、撮影された劣化画像のデータ（Ａ’）となることを前提としている。すなわち「Ａ０・Ｇ０＝Ａ’」が満たされることを前提とする。そして、Ａ０およびＧ０に任意の値を与えて、「Ｇ０＝Ａ’／Ａ０」の計算をする。そしてＧ０は負の値にならない等の拘束条件を与え、Ｇ１を得る。そして、「Ａ’／Ｇ１＝Ａ１」とし、Ａ０をＡ１へと変える。そして、Ａ１は負の値にならない等の拘束条件を与え、Ａ２を得る。そして、「Ａ’／Ａ２＝Ｇ２」とし、Ｇ１をＧ２へと変える。そして、Ｇ２は負の値にならない等の拘束条件を与え、Ｇ３を得る。このような計算を多数回繰り返して、妥当なＰＳＦおよび復元画像データを得る。 In addition, a technique for estimating an original object image by estimating a point spread function (PSF) that causes blurring using only an image deteriorated due to camera shake at the time of shooting as the only information has been proposed. (See Non-Patent Document 1). This technique is based on the premise that when the image data (A 0 ) before the change of the subject is multiplied by PSF (G 0 ), the imaged degraded image data (A ′) is obtained. That is, it is assumed that “A 0 · G 0 = A ′” is satisfied. Then, arbitrary values are given to A 0 and G 0 to calculate “G 0 = A ′ / A 0 ”. Then, G 0 is given a constraint condition such that it does not become a negative value, and G 1 is obtained. Then, “A ′ / G 1 = A 1 ” is set, and A 0 is changed to A 1 . A 1 is given a constraint such that A 1 does not become a negative value, and A 2 is obtained. Then, “A ′ / A 2 = G 2 ” is set, and G 1 is changed to G 2 . G 2 is given a constraint condition such that G 2 does not become a negative value, and G 3 is obtained. Such calculation is repeated many times to obtain appropriate PSF and restored image data.
　なお、一般の撮影画像以外にも、Ｘ線写真、顕微鏡画像等、種々の画像や信号が、ブレやその他の原因によって劣化したり、変化したりすることが知られている。 In addition to general captured images, it is known that various images and signals such as X-ray photographs and microscopic images are deteriorated or changed due to blurring or other causes.
特開２００７－４８２５７号公報JP 2007-48257 A
　特許文献１では、既知の変化要因情報のデータを利用する信号の復元処理についての記載がある。そのような復元処理を行うためには、速度センサまたは加速度センサ等を構成要素とする必要がある。 Patent Document 1 describes a signal restoration process using data of known change factor information. In order to perform such restoration processing, a speed sensor, an acceleration sensor, or the like needs to be a component.
　非特許文献１に記載されている技術を採用すると、速度センサまたは加速度センサ等を用いることなく、変化要因情報のデータが未知であっても画像を復元することができる。しかし、この技術はあらゆる画像に対応する拘束条件の設定が困難であり、また、処理時間が長くなることから汎用装置への採用は困難な状況となっている。 If the technology described in Non-Patent Document 1 is employed, an image can be restored without using a speed sensor or an acceleration sensor even if the data of the change factor information is unknown. However, in this technique, it is difficult to set a constraint condition corresponding to any image, and since the processing time becomes long, it is difficult to adopt it for a general-purpose device.
　そこで、本発明の目的は、変化要因情報のデータが未知であっても実用的な信号復元を可能とする変化要因情報のデータの生成法および信号処理装置を提供することである。 Therefore, an object of the present invention is to provide a method of generating data of change factor information and a signal processing device that enable practical signal restoration even if the data of change factor information is unknown.
　上記目的を達成するため、本発明の変化要因情報の生成法は、劣化等の変化が生じた原信号データの、その信号変化の要因となる変化要因情報のデータを生成する変化要因情報のデータの生成法において、任意の変化要因情報のデータを最初の変化要因情報のデータとして設定し、（１）そのデータを利用して、任意の信号データから比較用データを生成して、（２）原信号のデータまたはその原信号のデータを加工したデータと比較用データとを比較し、（３）得られた差分のデータを最初の変化要因情報のデータを利用して任意の信号データに配分することで最初の復元データを生成し、（４）その最初の復元データを任意の信号データの代わりに用い、必要に応じて、同様の処理を繰り返し、（５）その結果得られた、復元データと原信号のデータまたはその原信号のデータを加工したデータとから最初の変化要因情報のデータとは異なる新たな変化要因情報のデータを生成する。 In order to achieve the above object, the change factor information generation method of the present invention is the change factor information data for generating the change factor information data that causes the signal change of the original signal data that has undergone changes such as deterioration. In the generation method, any change factor information data is set as the first change factor information data. (1) Using the data, comparison data is generated from any signal data. (2) Compare the original signal data or the processed data of the original signal with the comparison data, and (3) distribute the obtained difference data to any signal data using the first change factor information data To generate the first restored data, (4) use the first restored data in place of any signal data, repeat the same process as necessary, and (5) the resulting restored Data and Generating data of a new change factor information that is different from a data signal of the data or the original signal from the processed data and the first change factor information data.
　この発明によれば、変化要因情報のデータが未知であっても妥当な変化要因情報のデータを生成でき、実用的な信号復元が可能となる。なお、（５）における「復元データ」は、最初の復元データおよび（４）において必要に応じて処理を繰り返して得られる復元データの両者を含む。 According to the present invention, even if the data of the change factor information is unknown, the appropriate change factor information data can be generated, and a practical signal restoration is possible. The “restored data” in (5) includes both the first restored data and the restored data obtained by repeating the process as necessary in (4).
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、新たな変化要因情報のデータを最初の変化要因情報のデータの代わりに用いて（１）から（３）と同様の処理を１回以上行う。この方法を採用することによって、より妥当な変化要因情報のデータを生成できる。 In addition to the above-described invention, the method of generating change factor information according to another invention uses the new change factor information data instead of the first change factor information data, and performs the same processing as (1) to (3). Repeat at least once. By adopting this method, more appropriate change factor information data can be generated.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、（３）の処理は、原信号のデータまたはその原信号のデータを加工したデータおよび復元データを構成する信号要素のデータの一部または全部を移動させる方法にて行い、復元データを構成する信号要素のデータが所定の基準を満たすか否かを判定し、所定の基準を満たさない場合には、差分のデータを利用して算出された配分値を修正し、その満たされないデータの代わりに修正された配分値によって得られたデータを使用する修正処理を行う。この方法を採用することによって、最初の復元データまたは復元データの品質が向上し、より妥当な変化要因情報のデータを生成できる。 In the method of generating change factor information according to another invention, in addition to the above-described invention, the processing of (3) includes the data of the original signal, the data obtained by processing the original signal data, and the data of the signal elements constituting the restoration data Is performed by moving a part or all of the data, and it is determined whether or not the signal element data constituting the restoration data satisfies a predetermined standard. If the predetermined standard is not satisfied, the difference data is used. Then, the distribution value calculated in this way is corrected, and correction processing using the data obtained by the corrected distribution value instead of the unsatisfied data is performed. By adopting this method, the quality of the first restoration data or restoration data is improved, and more appropriate change factor information data can be generated.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、新たな変化要因情報のデータを生成する処理は、得られた変化要因情報のデータの原点位置を、各々の信号要素のデータの移動に要する移動エネルギーの総和の最小値をＭｉｎとしたとき、その移動エネルギーがＭｉｎ以上で、Ｍｉｎ×１．２以下となる位置に設定する。この方法を採用することによって、最初の復元データまたは復元データの品質が向上し、より妥当な変化要因情報のデータを生成できる。 In addition to the above-described invention, the method of generating the change factor information according to another invention is a process of generating new change factor information data. The origin of the obtained change factor information data is determined based on each signal element. When the minimum value of the total kinetic energy required for data movement is Min, the kinetic energy is set to a position where the kinetic energy is Min or more and Min × 1.2 or less. By adopting this method, the quality of the first restoration data or restoration data is improved, and more appropriate change factor information data can be generated.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、原信号のデータを加工したデータは、原信号を小さくする縮小処理を行った縮小データで、任意の信号データは縮小データと同サイズのデータとする。この方法を採用することによって、変化の大きさも縮小されることから、変化要因情報のデータを推定し易くなり、より妥当な変化要因情報のデータを生成できる。 In addition to the above-described invention, the method of generating change factor information according to another invention is data obtained by processing the original signal data is reduced data obtained by performing a reduction process for reducing the original signal, and arbitrary signal data is reduced data. And the same size data. By adopting this method, since the magnitude of the change is also reduced, it becomes easier to estimate the data of the change factor information, and more appropriate change factor information data can be generated.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、縮小処理をしたデータから新たな変化要因情報のデータを得た後、新たな変化要因情報のデータを大きくする拡大処理を行い、拡大処理した前記変化要因情報のデータを前記新たな変化要因情報のデータの代わりに用い、（１）から（３）の処理と同様の処理を１回以上行い、必要に応じて拡大処理以降の処理を繰り返す。この方法を採用することによって、縮小処理を行って得た良質な変化要因情報のデータに基づいて大きな変化要因情報のデータの推定をすることができる。 In addition to the above-described invention, the method for generating the change factor information according to another invention includes an enlargement process for enlarging the new change factor information data after obtaining the new change factor information data from the reduced data. The data of the change factor information that has been performed and expanded is used in place of the data of the new change factor information, and the processing similar to the processing of (1) to (3) is performed at least once, and the expansion processing is performed as necessary. The subsequent processing is repeated. By adopting this method, it is possible to estimate large change factor information data based on good quality change factor data obtained by performing the reduction process.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、縮小処理をしたデータから新たな変化要因情報のデータを得た後、（５）における前記得られた復元データおよび新たな変化要因情報のデータをそれぞれ同じ比率で大きくする拡大処理を行い、拡大処理した変化要因情報のデータを新たな変化要因情報のデータの代わりに用い、拡大処理した復元データを任意の信号データの代わりに用い、（１）から（３）の処理と同様の処理を１回以上行い、必要に応じて拡大処理以降の処理を繰り返す。この方法を採用することによって、縮小処理を行って得た良質な変化要因情報のデータに基づいて大きな変化要因情報のデータの推定をすることができる。 In addition to the above-described invention, a method for generating change factor information according to another invention is obtained by obtaining new change factor information data from the reduced data, and then obtaining the obtained restored data and new data in (5). Performs enlargement processing to enlarge the data of the change factor information at the same ratio, uses the enlarged change factor information data instead of the new change factor information data, and uses the enlarged restoration data instead of any signal data The process similar to the process from (1) to (3) is performed once or more, and the processes after the enlargement process are repeated as necessary. By adopting this method, it is possible to estimate large change factor information data based on good quality change factor data obtained by performing the reduction process.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、新たな変化要因情報のデータを構成する中枢となる部分のデータの値を大きくする。この方法を採用することによって、ブレに相当する変化の推定に重きを置く変化要因情報のデータを生成できる。 In addition to the above-described invention, the method of generating change factor information according to another invention increases the value of the central part of the data of new change factor information. By adopting this method, it is possible to generate data of change factor information that places emphasis on estimation of changes corresponding to blurring.
　他の発明に係る変化要因情報の生成法は、上述した発明に加え、最初の変化要因情報のデータは、強度分布がガウス分布のデータである。この方法を採用することによって、変化がどのようなブレまたはボケを含んでいても、良好な変化要因情報のデータの推定をすることができる。 In addition to the above-described invention, the method of generating change factor information according to another invention is data having an intensity distribution of Gaussian distribution as the first change factor information data. By adopting this method, it is possible to estimate data of favorable change factor information regardless of any blur or blur in the change.
　上記目的を達成するため、本発明の信号処理装置は、劣化等の変化が生じた原信号データから、変化する前の信号若しくは本来取得されるべきであった信号またはそれらの近似信号（以下、元信号という）の復元をする処理部を有し、処理部は、任意の変化要因情報のデータを最初の変化要因情報のデータとして設定し、（１）そのデータを利用して、任意の信号データから比較用データを生成して、（２）原信号のデータまたはその原信号のデータを加工したデータと比較用データとを比較し、（３）得られた差分のデータを最初の変化要因情報のデータを利用して任意の信号データに配分することで最初の復元データを生成し、（４）その最初の復元データを任意の信号データの代わりに用い、必要に応じて、同様の処理を繰り返し、（５）その結果得られた、復元データと原信号のデータまたはその原信号のデータを加工したデータとから最初の変化要因情報のデータとは異なる新たな変化要因情報のデータを生成する処理を行い、（６）生成された新たな変化要因情報のデータまたはその変化要因情報のデータを加工したデータに基づいて元信号の復元を行う。 In order to achieve the above object, the signal processing apparatus of the present invention, from the original signal data that has undergone changes such as degradation, the signal before the change, the signal that should have been originally acquired, or an approximate signal thereof (hereinafter, A processing unit for restoring the original signal), the processing unit sets arbitrary change factor information data as the first change factor information data, and (1) uses the data to generate an arbitrary signal. Comparison data is generated from the data, (2) the original signal data or the data obtained by processing the original signal data is compared with the comparison data, and (3) the obtained difference data is used as the first change factor. The first restoration data is generated by allocating to the arbitrary signal data using the information data, and (4) the first restoration data is used in place of the arbitrary signal data, and if necessary, the same processing is performed. Repeat 5) A process for generating new change factor information data different from the first change factor information data from the restored data and the original signal data or the data obtained by processing the original signal data obtained as a result is performed. (6) The original signal is restored based on the generated data of the new change factor information or data obtained by processing the data of the change factor information.
　この発明によれば、変化要因情報のデータが未知であっても妥当な変化要因情報のデータを生成でき、実用的な信号復元が可能となる。なお、（５）における「復元データ」は、最初の復元データおよび（４）において必要に応じて処理を繰り返して得られる復元データの両者を含む。 According to the present invention, even if the data of the change factor information is unknown, the appropriate change factor information data can be generated, and a practical signal restoration is possible. The “restored data” in (5) includes both the first restored data and the restored data obtained by repeating the process as necessary in (4).
　他の発明に係る信号処理装置は、上述した発明に加え、元信号の復元をする処理は、処理部が変化要因情報のデータまたはその変化要因情報のデータを加工したデータを利用して、任意の信号データから比較用データを生成して、処理対象となる原信号のデータと比較用のデータとを比較し、得られた差分のデータを変化要因情報のデータまたはその変化要因情報のデータを加工したデータを利用して任意の信号データに配分することで復元データを生成し、この復元データを任意の信号データの代わりに使用し、同様の処理を繰り返す繰り返し処理を行うことである。この構成を採用することによって、より実用的な信号復元が可能となる。 In the signal processing device according to another invention, in addition to the above-described invention, the process of restoring the original signal may be performed arbitrarily using the data of the change factor information or the data obtained by processing the data of the change factor information. The comparison data is generated from the signal data, the original signal data to be processed is compared with the comparison data, and the obtained difference data is used as the change factor information data or the change factor information data. The restoration data is generated by allocating the processed data to arbitrary signal data, and this restoration data is used in place of the arbitrary signal data, and the same processing is repeated. By adopting this configuration, more practical signal restoration is possible.
　他の発明に係る信号処理装置は、上述した発明に加え、（３）の処理は、原信号のデータまたはその原信号のデータを加工したデータおよび復元データを構成する信号要素のデータの一部または全部を移動させる方法にて行い、復元データを構成する信号要素のデータが所定の基準を満たすか否かを判定し、所定の基準を満たさない場合には、差分のデータを利用して算出された配分値を修正し、その満たされないデータの代わりに修正された配分値によって得られたデータを使用する修正処理を行う。この構成を採用することによって、最初の復元データまたは復元データの品質が向上し、より妥当な変化要因情報のデータを生成できる。 In the signal processing apparatus according to another invention, in addition to the above-described invention, the processing of (3) is a part of the signal data constituting the original signal data, the data obtained by processing the original signal data, and the restoration data. Alternatively, it is performed by a method of moving all, and it is determined whether or not the signal element data constituting the restoration data satisfies a predetermined standard. If the predetermined standard is not satisfied, the difference data is used for calculation. The corrected distribution value is corrected, and correction processing is performed using the data obtained by the corrected distribution value instead of the unsatisfied data. By adopting this configuration, the quality of the first restoration data or restoration data is improved, and more appropriate change factor information data can be generated.
　他の発明に係る信号処理装置は、上述した発明に加え、新たな変化要因情報のデータを生成する処理は、得られた変化要因情報のデータの原点位置を、各々の信号要素のデータの移動に要する移動エネルギーの総和の最小値をＭｉｎとしたとき、その移動エネルギーがＭｉｎ以上で、Ｍｉｎ×１．２以下となる位置に設定する。この構成を採用することによって、最初の復元データまたは復元データの品質が向上し、より妥当な変化要因情報のデータを生成できる。 In addition to the above-described invention, the signal processing device according to another invention generates the new change factor information data by moving the data of each signal element to the origin position of the obtained change factor information data. When the minimum value of the total kinetic energy required for the above is Min, the kinetic energy is set to a position where the kinetic energy is Min or more and Min × 1.2 or less. By adopting this configuration, the quality of the first restoration data or restoration data is improved, and more appropriate change factor information data can be generated.
　他の発明に係る信号処理装置は、上述した発明に加え、復元手段は、処理部が信号変化の要因となる変化要因情報のデータを利用して、任意の信号データから比較用データを生成して、処理対象となる原信号のデータと比較用のデータとを比較し、得られた差分のデータを原点位置が設定された変化要因情報のデータを利用して任意の信号データに配分することで復元データを生成し、この復元データを任意の信号データの代わりに使用し、同様の処理を繰り返す繰り返し処理を行う手段である。この繰り返し処理を極めて多数回行えば、リンギングが発生していない画像等の信号を復元できる。よって、この構成を採用することで、より確実にリンギング等の発生を抑えることができる。 In addition to the above-described invention, the signal processing apparatus according to another invention generates the comparison data from arbitrary signal data by using the data of the change factor information that causes the processing unit to cause the signal change. The original signal data to be processed is compared with the comparison data, and the obtained difference data is distributed to arbitrary signal data using the data of the change factor information in which the origin position is set. This is means for generating reconstructed data, using the reconstructed data in place of arbitrary signal data, and repeatedly performing similar processing. If this repetitive processing is performed very many times, a signal such as an image in which no ringing has occurred can be restored. Therefore, by adopting this configuration, occurrence of ringing or the like can be more reliably suppressed.
　本発明では、変化要因情報のデータが未知であっても実用的な信号復元を可能とする変化要因情報のデータの生成法および信号処理装置を提供することができる。 In the present invention, it is possible to provide a method of generating data of change factor information and a signal processing apparatus that enable practical signal restoration even if the data of change factor information is unknown.
本発明の実施の形態に係る信号処理装置の主要構成を示すブロック図である。It is a block diagram which shows the main structures of the signal processing apparatus which concerns on embodiment of this invention. 図１に示す信号処理装置の処理部で行う変化要因情報のデータの生成法に係る処理ルーチンを説明するための処理フロー図である。FIG. 2 is a processing flowchart for explaining a processing routine according to a method for generating data of change factor information performed by a processing unit of the signal processing device shown in FIG. 1. 図２に示すＰＳＦの骨格部を強調する処理を説明する図で、（Ａ）は、その処理の前に推定されるＰＳＦ、（Ｂ）は、その処理の後に推定されるＰＳＦの一例を示している。FIGS. 3A and 3B are diagrams illustrating a process for emphasizing the skeleton part of the PSF illustrated in FIG. 2, in which FIG. 2A illustrates an example of a PSF estimated before the process, and FIG. ing. 図２に示す更新量の妥当性判断および復元データを修正する処理ルーチンを説明するためのフロー図である。FIG. 3 is a flowchart for explaining a processing routine for determining validity of an update amount and correcting restored data shown in FIG. 2. 図４に示す処理フローに係る修正を行う前後の更新量の一例を概略化して示す図である。FIG. 5 is a diagram schematically illustrating an example of an update amount before and after performing correction related to the processing flow illustrated in FIG. 4. 図１に示す信号処理装置の概要を示す外観斜視図で、図７におけるＸ－Ｙ平面が信号処理装置１のどの平面を示しているかを説明するための図である。FIG. 8 is an external perspective view showing an outline of the signal processing device shown in FIG. 1, for explaining which plane of the signal processing device 1 is indicated by the XY plane in FIG. 7. 図２に示す変化要因情報のデータの生成法に係る処理ルーチンのステップＳ１１２の処理を説明する図で、Ｘ－Ｙ平面で表されるブレの軌跡を示す図である。It is a figure explaining the process of step S112 of the processing routine which concerns on the production | generation method of the data of the change factor information shown in FIG. 2, and is a figure which shows the locus | trajectory of the blur represented on an XY plane. 図１に示す信号処理装置の処理部で行う画像復元処理方法（反復処理）に係る処理ルーチンを説明するための処理フロー図である。FIG. 2 is a processing flowchart for explaining a processing routine related to an image restoration processing method (repetitive processing) performed by a processing unit of the signal processing device shown in FIG. 1. 図８に示す処理方法の概念を説明するための図である。It is a figure for demonstrating the concept of the processing method shown in FIG. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、手ブレのないときのエネルギーの集中を示す表である。It is a figure for demonstrating concretely the processing method shown in FIG. 8 taking an example of camera shake, and is a table | surface which shows the concentration of energy when there is no camera shake. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、手ブレのないときの画像データを示す図である。It is a figure for demonstrating concretely the processing method shown in FIG. 8 as an example of camera shake, and is a figure which shows image data when there is no camera shake. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、手ブレが生じたときのエネルギーの分散を示す図である。It is a figure for demonstrating concretely the processing method shown in FIG. 8 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、任意の画像から比較用データを生成する状況を説明するための図である。It is a figure for demonstrating concretely the processing method shown in FIG. 8 taking an example of camera shake, and is a figure for demonstrating the condition which produces | generates the data for a comparison from arbitrary images. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、比較用データと、処理対象となるブレた原画像とを比較して、差分のデータを生成する状況を説明するための図である。FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 using camera shake as an example, in which comparison data and a blurred original image to be processed are compared to generate difference data It is a figure for demonstrating. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、差分のデータを配分し任意の画像に加えることで復元データを生成する状況を説明するための図である。FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 by taking an example of camera shake, and a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. is there. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、生成された復元データから新たな比較用データを生成し、そのデータと処理対象となるブレた原画像とを比較して差分のデータを生成する状況を説明するための図である。FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 using camera shake as an example. In this case, new comparison data is generated from the generated restored data, and the data and the blurred original image to be processed It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. 図８に示す処理方法を、手ブレを例にして具体的に説明するための図で、新たに生成された差分のデータを配分し、新たな復元データを生成する状況を説明するための図である。FIG. 8 is a diagram for specifically explaining the processing method shown in FIG. 8 by taking an example of camera shake, and a diagram for explaining a situation in which newly generated difference data is allocated and new restored data is generated. It is.
符号の説明Explanation of symbols
　１　信号処理装置
　４　処理部
　Ｓ１０１　最初の変化要因情報のデータ（を設定する処理）
　Ｉｏ　初期画像のデータ（任意の信号データ）
　Ｉｏ’　比較用データ
　Ｇ　変化要因情報のデータ
　Ｉｍｇ’　原画像のデータ（原信号のデータ）
　ＳＩｍｇ’　縮小された原画像のデータ
　Ｉ０＋ｎ　復元データ
　Ｉｍｇ　元画像（元信号）
　Δ、δ　差分のデータ
　Ｐ０　縮小データ
　Ａ　骨格部（新たな変化要因情報のデータを構成する中枢となる部分のデータ）
　Ｄｃ　更新量（配分値）
　Ｓ２０４，Ｓ２０６，Ｓ２０８，Ｓ２１０　所定の基準 DESCRIPTION OF SYMBOLS 1 Signal processing apparatus 4 Processing part S101 First change factor information data (Process to set)
Io Initial image data (arbitrary signal data)
Io 'Comparison data G Change factor information data Img' Original image data (original signal data)
SImg 'Reduced original image data I 0 + n Restored data Img Original image (original signal)
Δ, delta difference data P 0 reduced data A skeleton part (data of the central part constituting the data of new change factor information)
Dc Update amount (distributed value)
S204, S206, S208, S210 Predetermined criteria
（信号処理装置の構成）
　以下、本発明の実施の形態に係る信号処理装置を、図を参照しながら説明する。なお、この信号処理装置は、民生用のカメラとしているが、監視用カメラ、テレビ用カメラ、ハンディタイプのビデオカメラ、内視鏡カメラ、等他の用途のカメラとしたり、顕微鏡、双眼鏡、さらにはＮＭＲ撮影等の画像診断装置、画像を印刷するプリンタ、画像を読み込むスキャナ等、カメラ以外の機器にも適用できる。 (Configuration of signal processing device)
Hereinafter, a signal processing device according to an embodiment of the present invention will be described with reference to the drawings. Although this signal processing device is a consumer camera, it may be a camera for other uses such as a surveillance camera, a television camera, a handy type video camera, an endoscopic camera, a microscope, a binocular, It can also be applied to devices other than cameras, such as diagnostic imaging apparatuses such as NMR imaging, printers that print images, and scanners that read images.
　図１には信号処理装置１の構成の概要を示している。信号処理装置１は、人物等の画像を撮影する撮影部２と、その撮影部２を駆動する制御系部３と、撮影部２で撮影された画像を処理する処理部４と、を有している。また、この実施の形態に係る信号処理装置１は、さらに処理部４で処理された画像を記録する記録部５と、画像劣化等を生じさせる変化要因情報のデータを保存する要因情報保存部７を有する。 FIG. 1 shows an outline of the configuration of the signal processing apparatus 1. The signal processing device 1 includes an imaging unit 2 that captures an image of a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. Further, the signal processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4 and a factor information storage unit 7 that stores data of change factor information that causes image degradation and the like. Have
　撮影部２は、レンズを有する撮影光学系やレンズを通過した光を電気信号に変換するＣＣＤやＣ－ＭＯＳ等の撮影素子を備える部分である。制御系部３は、撮影部２，処理部４，記録部５，および要因情報保存部７等、信号処理装置内の各部を制御するものである。 The photographing unit 2 is a part including a photographing optical system having a lens and a photographing element such as a CCD or a C-MOS that converts light passing through the lens into an electric signal. The control system unit 3 controls each unit in the signal processing device, such as the photographing unit 2, the processing unit 4, the recording unit 5, and the factor information storage unit 7.
　処理部４は、画像処理プロセサで構成されており、ＡＳＩＣ(Application Specific Integrated Circuit)のようなハードウェアで構成されている。そして、処理部４は、後述する変化要因情報のデータの生成法に係る処理および画像の復元処理を実行する。 The processing unit 4 is composed of an image processing processor, and is composed of hardware such as ASIC (Application Specific Specific Integrated Circuit). Then, the processing unit 4 executes a process related to a data generation method of change factor information, which will be described later, and an image restoration process.
　また、この処理部４には、後述する比較用データを生成する際の元となる画像のデータが保管されることもある。さらに処理部４は、ＡＳＩＣのようなハードウェアとして構成されるのではなく、ソフトウェアで処理する構成としても良い。記録部５は、半導体メモリで構成されているが、ハードディスクドライブ等の磁気記録手段、またはＤＶＤ等を使用する光記録手段等を採用しても良い。 Also, the processing unit 4 may store image data that is a source for generating comparison data, which will be described later. Further, the processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 is composed of a semiconductor memory, but magnetic recording means such as a hard disk drive or optical recording means using a DVD or the like may be employed.
　要因情報保存部７は、本発明の実施の形態に係る変化要因情報のデータの生成法によって生成される変化要因情報のデータまたはそのデータを加工したデータを保存しておく記録部である。そして要因情報保存部７は、そのデータの中枢をなすブレの軌跡（履歴）と、その軌跡上の各点にとどまっていた時間（重み）を記憶する。要因情報保存部７で記録されたデータは、撮影された原画像（劣化等の変化が生じた画像）から元画像（変化する前の画像もしくは本来撮影されるべきであった画像またはそれらの近似画像）への復元処理の際に、処理部４で用いられる。よって、処理部４と要因情報保存部７との間では、相互にデータの受け渡しが可能となっている。 The factor information storage unit 7 is a recording unit that stores data of change factor information generated by the method of generating data of change factor information according to the embodiment of the present invention or data obtained by processing the data. Then, the factor information storage unit 7 stores a blurring locus (history) that forms the center of the data, and a time (weight) that has remained at each point on the locus. The data recorded in the factor information storage unit 7 is from an original image (an image in which deterioration or the like has changed) to an original image (an image before the change, an image that should have been originally taken, or an approximation thereof) This is used by the processing unit 4 when the image is restored. Therefore, data can be exchanged between the processing unit 4 and the factor information storage unit 7.
（変化要因情報のデータの生成）
　図２に、処理部４が行う本発明の実施の形態に係る変化要因情報のデータの生成法の一例のフローチャートを示す。 (Generation of change factor information data)
FIG. 2 shows a flowchart of an example of a method for generating data of change factor information according to the embodiment of the present invention performed by the processing unit 4.
　まず、処理部４は、最初の変化要因情報のデータ（ＰＳＦ）の任意の初期値（Ｐ０）を設定する（ステップＳ１０１）。ここでは、その初期値を強度分布がガウス分布のデータであるガウシアンディスクとして設定する。 First, the processing unit 4 sets an arbitrary initial value (P 0 ) of the first change factor information data (PSF) (step S101). Here, the initial value is set as a Gaussian disk whose intensity distribution is Gaussian distribution data.
　次に、処理部４は、ステップＳ１０１の処理と併行してまたはその処理の前若しくは後に、撮影した原画像のデータ（原信号のデータ）Ｉｍｇ’を１／４に縮小する。すなわち、原信号のデータＩｍｇ’のデータを間引いて２５％の容量値とする縮小処理（加工）を行う（ステップＳ１０２）。この縮小処理を経たデータである縮小データをＳＩｍｇ’と記す。 Next, the processing unit 4 reduces the captured original image data (original signal data) Img ′ to ¼ in parallel with the processing in step S101 or before or after the processing. That is, a reduction process (processing) is performed by thinning out the original signal data Img 'to a capacity value of 25% (step S102). The reduced data that has undergone this reduction processing is denoted as SImg ′.
　次に、処理部４は、その縮小データＳＩｍｇ’と同一容量の任意の信号データとしてＩ０を用い、Ｉ０とＰ０とを重畳積分して比較用のブレたデータＩ０’を得る（ステップＳ１０３）。なお、任意の信号データＩ０としては、縮小された原画像のデータＳＩｍｇ’が好ましい。 Next, the processing unit 4 uses I 0 as arbitrary signal data having the same capacity as the reduced data SImg ′, and superimposes I 0 and P 0 to obtain blurred data I 0 ′ for comparison ( Step S103). The arbitrary signal data I 0 is preferably reduced original image data SImg ′.
　次に、処理部４は、ＳＩｍｇ’とＩ０’とを比較し、その差分のデータΔを得る（ステップＳ１０４）。その差分のデータΔが所定値以内かを判定し（ステップＳ１０５）、所定値以内であればステップＳ１２０へ移行し、所定値以内でなければステップＳ１０６へ移行する。ステップＳ１０６では、その差分のデータΔに係数ｋ（変化要因情報（ＰＳＦ）に基づいた係数）を乗じたものをＩ０に配分して、最初の復元データＩ０＋ｎ（今回はＩ０＋１）を得る。ステップＳ１０６において、Ｉ０に配分する際には、その配分値（更新量）の妥当性を判断する処理を行う（ステップＳ１０７）。そして、その妥当性に応じてＩ０＋ｎを修正する処理を行う（ステップＳ１０８）。次に、ステップＳ１０９に移行し、この処理が所定回数に達したか否かを判断し、所定回数（たとえば１０回）に達したらステップＳ１２０へ移行し、所定回数に達していないときはステップＳ１１０へ移行する。 Next, the processing unit 4 compares SImg ′ with I 0 ′, and obtains data Δ of the difference (Step S104). It is determined whether the difference data Δ is within a predetermined value (step S105). If it is within the predetermined value, the process proceeds to step S120, and if not within the predetermined value, the process proceeds to step S106. In step S106, data obtained by multiplying the difference data Δ by a coefficient k (coefficient based on the change factor information (PSF)) is distributed to I 0 to obtain first restored data I 0 + n (this time I 0 + 1 ). . In step S106, when allocating to I0 , processing for determining the validity of the distribution value (update amount) is performed (step S107). And the process which corrects I0 + n according to the validity is performed (step S108). Next, the process proceeds to step S109, where it is determined whether or not this process has reached a predetermined number of times. If the predetermined number (for example, 10 times) has been reached, the process proceeds to step S120. Migrate to
　そして処理部４は、Ｉ０＋１をＩ０の代わりに用いて（ステップＳ１１０）ステップＳ１０３の処理を行い、ステップＳ１０３、ステップＳ１０４、ステップＳ１０５、ステップＳ１０６、ステップＳ１０７、ステップＳ１０８、ステップＳ１０９およびステップＳ１１０の処理を所定回数（今回は１０回）繰り返す。その所定回数に達したか否かの判断をステップＳ１０９にて行う。 Then, the processing unit 4 performs the process of step S103 using I 0 + 1 instead of I 0 (step S110), and performs steps S103, S104, S105, S106, S107, S108, S109, and S110. This process is repeated a predetermined number of times (10 times this time). In step S109, it is determined whether or not the predetermined number of times has been reached.
　ステップＳ１０６の処理は、縮小した原画像のデータＳＩｍｇ’および復元データＩ０＋ｎを構成する画素のデータの一部または全部を移動（配分）させる処理である。なお、ステップＳ１０７およびステップＳ１０８の詳細については後述する。 The process of step S106 is a process of moving (distributing) part or all of the pixel data constituting the reduced original image data SImg ′ and the restored data I 0 + n . Details of step S107 and step S108 will be described later.
　次に、ステップＳ１０９にて所定回数に達したと判断された場合は、処理部４は、ＰＳＦを算出する処理を行う（ステップＳ１２０）。その処理は、得られた復元データＩ０＋ｎと縮小された原画像のデータＳＩｍｇ’から新たなＰＳＦ（＝Ｐ）を得る処理である。その処理の具体例は、得られた復元データＩ０＋ｎと原画像の縮小データＳＩｍｇ’をそれぞれフーリエ変換し、周波数空間での割り算によってＰＳＦの周波数特性を算出し、その周波数特性をフーリエ逆変換することによってＰＳＦ（＝Ｐ）を得る。 Next, when it is determined in step S109 that the predetermined number of times has been reached, the processing unit 4 performs a process of calculating a PSF (step S120). The process is a process of obtaining a new PSF (= P) from the obtained restored data I 0 + n and the reduced original image data SImg ′. A specific example of the processing is to Fourier transform the obtained restoration data I 0 + n and the reduced data SImg ′ of the original image, calculate the frequency characteristic of the PSF by division in the frequency space, and inverse Fourier transform the frequency characteristic. To obtain PSF (= P).
　次に、処理部４は、Ｐのデータの中枢をなすブレの軌跡の部分（骨格部）のデータの値を２倍にして、骨格部を強調する処理を行う（ステップＳ１２１）。図３（Ａ）は、現段階の時点で推定されるＰＳＦをＸ－Ｙ平面上に示したものである。このＸ－Ｙ平面は、後述する図６に示す信号処理装置１のブレが生じやすいＸ－Ｙ平面に相当する。ＰＳＦは、そのデータの中枢をなすブレの軌跡に相当する部分（骨格部）Ａと、その周辺のデータ値の小さな部分Ｂ（ボケ等の部分）とからなる。この小さな値となる部分Ｂは、ボケの部分と考えられる。図３（Ｂ）は、Ａの部分を強調する処理（ステップＳ１２１）を行った後のＰＳＦの状態を示している。Ａの骨格部分が若干強調されていることがわかる。この処理に当っては、細線化処理が採用されているが、他の強調方法を採用してもよい。この処理の結果、ステップＳ１２０で得られたＰのデータは、Ｐ’のデータとなる。その後、移動エネルギーが最小となるようにＰ’の原点を設定する処理（詳細は後述する）を行う（ステップＳ１２２）。 Next, the processing unit 4 performs a process of emphasizing the skeleton part by doubling the data value of the blur locus part (skeleton part) forming the center of the P data (step S121). FIG. 3A shows the PSF estimated at the current stage on the XY plane. This XY plane corresponds to an XY plane in which the signal processing apparatus 1 shown in FIG. The PSF is composed of a portion (skeleton portion) A corresponding to the locus of blur that forms the center of the data, and a portion B (portion such as blur) having a small data value around the portion. This small portion B is considered as a blurred portion. FIG. 3B shows the state of the PSF after performing the process of emphasizing the portion A (step S121). It can be seen that the skeleton of A is slightly emphasized. In this processing, thinning processing is employed, but other emphasis methods may be employed. As a result of this processing, the P data obtained in step S120 becomes P ′ data. Thereafter, a process of setting the origin of P ′ so that the kinetic energy is minimized (details will be described later) is performed (step S122).
　次に、処理部４は、Ｐ’が妥当なＰＳＦか否かを判断する。具体的には、処理部４は、ステップＳ１２０からＳ１２２までの処理を３回繰り返したか否かを判断する（ステップＳ１２３）。繰り返しの回数、すなわちＰＳＦの算出回数が３回を満たしていなければ（Ｎ）、Ｐ’のデータをＰ０の代わりに用いて（ステップＳ１２４）、ステップＳ１０３からＳ１１０までの処理を繰り返し、その後再度ステップＳ１２０～Ｓ１２２を実行する。 Next, the processing unit 4 determines whether or not P ′ is a valid PSF. Specifically, the processing unit 4 determines whether or not the processing from step S120 to S122 has been repeated three times (step S123). If the number of repetitions, that is, the number of PSF calculations does not satisfy 3 (N), P ′ data is used instead of P 0 (step S124), and the processing from step S103 to S110 is repeated, and then again. Steps S120 to S122 are executed.
　そして、ステップＳ１２３の判断において、ＰＳＦ算出の回数、すなわち繰り返しの回数が３回を満たしていれば（Ｙ）、処理部４は、得られたＰ’のデータおよびＩ０＋ｎのデータを同じ比率で拡大処理する（ステップＳ１２５）。この実施の形態では、最後に得られたＰ’とＩ０＋ｎを４／３倍する。その拡大処理が予め定められていた縮小率、すなわち、原画像に対する縮小の程度に至ったか否かを判断する（ステップＳ１２６）。所定の縮小率に至っていないと判断すると、拡大処理したＩ０＋ｎのデータをＩ０のデータの代わりに用い（ステップＳ１２７）、拡大処理したＰ’のデータをＰ０の代わりに用いる（ステップＳ１２４）。一方、原画像のデータを１／３に縮小した新たな縮小画像ＳＩｍｇ’を得る。この１／３に縮小した画像は先に４／３倍したデータとは同じサイズの画像となる。そして、処理部４は、新たな各データを用い、ステップＳ１０３からＳ１１０までの処理を繰り返し、また、ステップＳ１２０からＳ１２４までの処理を繰り返す。この繰り返しは、ステップＳ１２６で所定の縮小率、たとえば原画像データに対して１／３の縮小率に達していると判定されたときは、ステップＳ１２８へ移行する。なお、ステップＳ１２５では、得られたＰ’のデータおよびＩ０＋ｎのデータを同じ比率で拡大処理しているが、拡大処理するのはＰ’のデータのみとし、拡大処理したＩ０＋ｎのデータの代わりに任意の信号データを用い、ステップＳ１２５を実行しても良い。その任意の信号データは、たとえばＳＩｍｇ’のデータ等である。但し、ステップＳ１２５では、本実施の形態のように得られたＰ’のデータおよびＩ０＋ｎのデータを同じ比率で拡大処理した方が、復元処理を経たＩ０＋ｎのデータを用いるため、良質なＰＳＦを生成できる場合がある。 If the number of PSF calculations, that is, the number of repetitions satisfies 3 in the determination of step S123 (Y), the processing unit 4 uses the same ratio for the obtained P ′ data and I 0 + n data. Enlargement processing is performed (step S125). In this embodiment, P ′ obtained last and I 0 + n are multiplied by 4/3. It is determined whether or not the enlargement process has reached a predetermined reduction rate, that is, whether or not the original image has been reduced (step S126). If it is determined that the predetermined reduction ratio has not been reached, the enlarged I 0 + n data is used in place of the I 0 data (step S127), and the enlarged P ′ data is used in place of the P 0 (step S124). . On the other hand, a new reduced image SImg ′ obtained by reducing the original image data to 1/3 is obtained. The image reduced to 1/3 is the same size as the data 4/3 times earlier. And the process part 4 repeats the process from step S103 to S110 using each new data, and repeats the process from step S120 to S124. If it is determined in step S126 that the predetermined reduction rate, for example, a reduction rate of 1/3 with respect to the original image data has been reached, the process proceeds to step S128. In step S125, the obtained P ′ data and I 0 + n data are enlarged at the same ratio, but only the P ′ data is enlarged, and instead of the enlarged I 0 + n data. Arbitrary signal data may be used for step S125. The arbitrary signal data is, for example, SImg ′ data. However, in step S125, the direction which the data of the data and I 0 + n of the resultant P 'as in the present embodiment by enlarging processed in the same ratio, for using data of the I 0 + n passing through the restoration process, good quality PSF May be generated.
　なお、ステップＳ１２５におけるＰ’のデータの拡大処理は、処理部４が、隣接する信号要素の間にその隣接する信号要素の値の平均値の新たな信号要素のデータを挿入することにより行う。また、Ｉ０＋ｎのデータの拡大処理は、処理部４が、隣り合う画素の間に新たな画素を挿入することにより行う。その新たな画素の画素値は、隣り合う画素の画素値を平均した値とする。この拡大処理によって所定の縮小率に達したか否かの判断は、処理部４がステップＳ１２６で行う。拡大処理によって所定の縮小率に達したら（Ｙ）、処理部４は、現段階で得られているＰ’を実寸大に拡大する（ステップＳ１２８）。すなわち、現段階で得られているＰ’は、実寸の１／３のものであるから、処理部４は、現段階で得られているＰ’を３倍に拡大する。この拡大処理も、処理部４が隣り合う画素の間に新たな画素を挿入することにより行う。その新たな画素の画素値は、隣り合う画素の画素値を平均した値とする。なお、ステップＳ１０５，Ｓ１２８の各拡大処理に当っては、上述の補間法以外に、処理部４が、隣り合う画素の間にその隣り合う画素の一の画素の画素値の画素を挿入することにより行う補間法を採用してもよい。 Note that the P ′ data enlargement process in step S125 is performed by the processing unit 4 inserting new signal element data having an average value of adjacent signal element values between adjacent signal elements. Further, the process of enlarging the I 0 + n data is performed by the processing unit 4 inserting a new pixel between adjacent pixels. The pixel value of the new pixel is a value obtained by averaging the pixel values of adjacent pixels. The processing unit 4 determines whether or not a predetermined reduction rate has been reached by this enlargement processing in step S126. When the predetermined reduction ratio is reached by the enlargement process (Y), the processing unit 4 enlarges P ′ obtained at the current stage to the actual size (step S128). That is, since P ′ obtained at the current stage is one third of the actual size, the processing unit 4 enlarges P ′ obtained at the current stage three times. This enlargement process is also performed by the processing unit 4 inserting a new pixel between adjacent pixels. The pixel value of the new pixel is a value obtained by averaging the pixel values of adjacent pixels. In each enlargement process in steps S105 and S128, the processing unit 4 inserts a pixel having a pixel value of one of the adjacent pixels between adjacent pixels, in addition to the interpolation method described above. The interpolation method performed by the above may be adopted.
　以上の処理によって、本発明の実施の形態に係る変化要因情報のデータの生成法が終了し（ステップＳ１２９）、ＰＳＦが生成される。このＰＳＦは、処理部４によって要因情報保存部７に記憶され、信号の復元処理の際に用いられる。以上の過程を経て得られたＰＳＦを「Ｇ」と表す。 Through the above processing, the method for generating data of change factor information according to the embodiment of the present invention is completed (step S129), and a PSF is generated. The PSF is stored in the factor information storage unit 7 by the processing unit 4 and used in the signal restoration process. The PSF obtained through the above process is represented as “G”.
（配分値の妥当性を判断する処理）
　上述のステップＳ１０７で行う、配分値の妥当性を判断する処理について図を用いて説明する。 (Process to determine the appropriateness of the distribution value)
Processing for determining the validity of the distribution value performed in step S107 described above will be described with reference to the drawings.
　図４および図５は、図２におけるステップＳ１０７およびＳ１０８の処理の詳細を説明するための図である。この処理の基本的な考え方は、次の通りである。この処理は、復元データＩ０＋ｎに対し一括して行うものではなく、復元データＩ０＋ｎを構成する信号要素毎に行うものである。この点はステップＳ１０３，Ｓ１０５およびＳ１０６の信号（画像）全体の処理とは大きく異なる。まず、Ｐ０によって一つの信号要素となる一つの画素が影響を受ける範囲の複数の画素のうち、更新によって画素値の変化が大きくなるものがある場合は、その複数の画素の範囲にエッジとなる部分が存すると考える。そのエッジとなる部分は、対応する部分の差分のデータΔに基づく更新量が適切な量となり難い。なぜならば、エッジ部分を跨ぐ２つの画素の画素値は、その違いが大きすぎるため、一方の画素から他方の画素へとエッジ部分を越えて画素値を配分しても、その配分は適切なものとはなりにくいためである。そこで、更新によって画素値の変化が周囲に比べ不自然に大きくなる画素については、更新量の絶対値を小さくすることで、エッジ近傍への更新量を適切なものに近づける。 4 and 5 are diagrams for explaining the details of the processing in steps S107 and S108 in FIG. The basic concept of this processing is as follows. This process is not performed for the restored data I 0 + n in a lump, but for each signal element constituting the restored data I 0 + n . This point is greatly different from the processing of the entire signal (image) in steps S103, S105, and S106. First, when there is a pixel whose change in pixel value is greatly increased by updating among a plurality of pixels in a range in which one pixel serving as one signal element is affected by P 0 , an edge and an edge are included in the range of the plurality of pixels. I think that there will be a part. For the portion serving as the edge, the update amount based on the difference data Δ of the corresponding portion is unlikely to be an appropriate amount. This is because the difference between the pixel values of two pixels straddling the edge part is too large, so even if the pixel value is distributed from one pixel to the other pixel across the edge part, the distribution is appropriate. This is because it is difficult to. Therefore, for pixels whose change in pixel value is unnaturally larger than the surroundings due to the update, the update amount in the vicinity of the edge is brought close to an appropriate value by reducing the absolute value of the update amount.
　そのためには、まず、所定の差分のデータΔからある画素の更新量（Ｄｃ）を算出する（ステップＳ２０１）。そして、変化要因情報Ｇによって一つの画素が影響を及ぼす範囲の複数の画素（一部の信号要素）およびその一つの画素との集合を参照し、その参照する各々の画素の更新前の画素値（＝Ｉｂ）の最小値（＝Ｍｉｎ）、最大値（＝Ｍａｘ）および平均値（＝Ａｖ）を算出する（ステップＳ２０２）。なお、この一部の画素（参照する画素）は、単純に隣接する複数の画素、あるいは所定の画素を中心としてそこから所定の距離に含まれる複数の画素としても良い。その所定の距離は、たとえば変化要因情報のデータＧを利用して決定される。 For this purpose, first, an update amount (Dc) of a certain pixel is calculated from predetermined difference data Δ (step S201). Then, reference is made to a set of a plurality of pixels (a part of signal elements) in the range in which one pixel is affected by the change factor information G and the one pixel, and the pixel value before the update of each pixel to be referred to A minimum value (= Min), a maximum value (= Max), and an average value (= Av) of (= Ib) are calculated (step S202). The partial pixels (reference pixels) may be simply a plurality of adjacent pixels or a plurality of pixels included in a predetermined distance from a predetermined pixel as a center. The predetermined distance is determined using data G of change factor information, for example.
　次に、参照する画素のうちの一つの画素（＝一つの信号要素）の更新後の画素値となるデータ（＝Ｉａ）を算出する（ステップＳ２０３）。この更新後の画素値Ｉａは、ある画素に対応した差分のデータδから算出した各画素毎の更新量（Ｄｃ）を用いて更新した場合の各画素毎の画素値であり、次の（１）式によって導かれる。
Ｉａ＝Ｉｂ＋Ｄｃ　……（１） Next, data (= Ia) that is the updated pixel value of one pixel (= one signal element) among the referenced pixels is calculated (step S203). This updated pixel value Ia is a pixel value for each pixel when updated using the update amount (Dc) for each pixel calculated from the difference data δ corresponding to a certain pixel. ) Guided by the formula.
Ia = Ib + Dc (1)
　次に、修正前の更新量（Ｄｃ）の妥当性を判断する。まず、復元データを構成する信号要素のデータとなる更新後の画素値Ｉａが最小値Ｍｉｎ以上であり最大値Ｍａｘ以下である場合には（ステップＳ２０４の判断が「Ｙ」（＝Ｙｅｓ））、その画素は、周囲の画素との画素値のバランスが自然であり、更新量Ｄｃは、修正せずにそのまま修正後の更新量（Ｄｐ）とし、修正値Ｉａ’を得る（ステップＳ２０５，Ｓ２１４）。なおステップＳ２１４は、修正後の更新量Ｄｐによって修正値Ｉａ’を得るステップである。図５の（Ａ）に示す矢印は、ステップＳ２０５およびステップＳ２１４の処理を経た状態を概略的に表しており、矢印の黒丸は画素の更新前の画素値Ｉｂを示し、三角の矢示はＤｐによって更新した後の修正値Ｉａ'を示し、矢印の長さは更新量（Ｄｃ＝Ｄｐ）を示している。左側の矢印は、修正前の更新量Ｄｃの値が正の値の場合、右側の矢印は、修正前の更新量Ｄｃの値が負の値の場合についての一例を示している。 Next, the validity of the update amount (Dc) before correction is determined. First, when the updated pixel value Ia, which is signal element data constituting the restored data, is not less than the minimum value Min and not more than the maximum value Max (determination in step S204 is “Y” (= Yes)), The pixel has a natural balance of pixel values with the surrounding pixels, and the update amount Dc is not corrected and is used as the update amount (Dp) after correction, thereby obtaining a correction value Ia ′ (steps S205 and S214). . Step S214 is a step for obtaining a correction value Ia 'by the update amount Dp after correction. The arrow shown in FIG. 5A schematically shows the state after the processing of step S205 and step S214, the black circle of the arrow indicates the pixel value Ib before the pixel update, and the triangular arrow indicates Dp Indicates the corrected value Ia ′ after updating, and the length of the arrow indicates the update amount (Dc = Dp). The left arrow indicates an example when the value of the update amount Dc before correction is a positive value, and the right arrow indicates an example when the value of the update amount Dc before correction is a negative value.
　そして、ステップＳ２０４の判断が「Ｎ」（＝Ｎｏ）の場合は、ステップＳ２０６へと進む。ここでは、更新後の画素値Ｉａが最大値Ｍａｘを超えてしまっている場合であって更新前の画素値Ｉｂが平均値Ａｖ以下である場合には（ステップＳ２０６の判断が「Ｙ」（＝Ｙｅｓ））、その画素は、周囲の画素との画素値のバランスが不自然であると判断する。そして、更新量Ｄｃは、最大値Ｍａｘから更新前の画素値Ｉｂを差し引いた値Ｄｐへと変更される（ステップＳ２０７）。この結果、修正値Ｉａ’の画素値は、最大値Ｍａｘと等しくなる。図５の（Ｂ）に示す矢印は、ステップＳ２０７およびステップＳ２１４を経た状態を、図５（Ａ）と同様に概略的に表している。左側の矢印は、Ｄｃによって更新した場合のＩａ、右側の矢印は、Ｄｐによって更新した場合の修正値Ｉａ’を示している。 And when judgment of step S204 is "N" (= No), it progresses to step S206. Here, when the updated pixel value Ia exceeds the maximum value Max and the updated pixel value Ib is equal to or less than the average value Av (determination in step S206 is “Y” (= Yes)), it is determined that the pixel value balance with the surrounding pixels is unnatural. Then, the update amount Dc is changed to a value Dp obtained by subtracting the pixel value Ib before update from the maximum value Max (step S207). As a result, the pixel value of the correction value Ia ′ becomes equal to the maximum value Max. The arrow shown in FIG. 5B schematically shows the state after step S207 and step S214, as in FIG. The left arrow indicates Ia when updated by Dc, and the right arrow indicates a correction value Ia 'when updated by Dp.
　そして、ステップＳ２０６の判断が「Ｎ」（＝Ｎｏ）の場合は、ステップＳ２０８へと進む。ここでは、更新後の画素値Ｉａが最大値Ｍａｘを超えてしまっている場合であって更新前の画素値Ｉｂが平均値Ａｖを越える場合には（ステップＳ２０８の判断が「Ｙ」（＝Ｙｅｓ））、その画素は、周囲の画素との画素値のバランスが不自然であると判断する。その場合、更新量Ｄｃは、以下の（２）式から得られる値Ｄｐへと変更される（ステップＳ２０９）。この結果、修正値Ｉａ’の画素値は、更新後の画素値Ｉａが最大値Ｍａｘをプラス側に超えた分の１／４を最大値Ｍａｘに加えた値となる。図５の（Ｃ）に示す矢印は、ステップＳ２０９およびステップＳ２１４を経た状態を、図５（Ｂ）と同様に概略的に表している。
Ｄｐ=０．２５（Ｉａ－Ｍａｘ）＋（Ｍａｘ－Ｉｂ）　…（２） When the determination in step S206 is “N” (= No), the process proceeds to step S208. Here, when the updated pixel value Ia exceeds the maximum value Max and the updated pixel value Ib exceeds the average value Av (the determination in step S208 is “Y” (= Yes). )), It is determined that the pixel value balance with the surrounding pixels is unnatural. In this case, the update amount Dc is changed to a value Dp obtained from the following equation (2) (step S209). As a result, the pixel value of the correction value Ia ′ is a value obtained by adding ¼ of the updated pixel value Ia exceeding the maximum value Max to the plus side to the maximum value Max. The arrow shown in FIG. 5C schematically shows the state after step S209 and step S214, as in FIG. 5B.
Dp = 0.25 (Ia−Max) + (Max−Ib) (2)
　そして、ステップＳ２０８の判断が「Ｎ」（＝Ｎｏ）の場合は、ステップＳ２１０へと進む。ここでは、更新後の画素値Ｉａが最小値Ｍｉｎ未満であって更新前の画素値Ｉｂが平均値Ａｖ以下である場合には（ステップＳ２１０の判断が「Ｙ」（＝Ｙｅｓ））、その画素は、周囲の画素との画素値のバランスが不自然であると判断する。そして、更新量Ｄｃは、以下の（３）式から得られる値Ｄｐへと変更される（ステップＳ２１１）。この結果、修正値Ｉａ’の画素値は、更新後の画素値Ｉａが最小値Ｍｉｎをマイナス側に超えた分の１／４を最小値Ｍｉｎから引いた値となる。図５の（Ｄ）に示す矢印は、ステップＳ２１１およびステップＳ２１４を経た状態を、図５（Ｂ）と同様に概略的に表している。
Ｄｐ=－（Ｉｂ－Ｍｉｎ）－０．２５（Ｍｉｎ－Ｉａ）　…（３） When the determination in step S208 is “N” (= No), the process proceeds to step S210. Here, when the pixel value Ia after update is less than the minimum value Min and the pixel value Ib before update is equal to or less than the average value Av (the determination in step S210 is “Y” (= Yes)), the pixel Determines that the balance of pixel values with surrounding pixels is unnatural. Then, the update amount Dc is changed to a value Dp obtained from the following equation (3) (step S211). As a result, the pixel value of the correction value Ia ′ is a value obtained by subtracting ¼ of the updated pixel value Ia exceeding the minimum value Min to the minus side from the minimum value Min. The arrow shown in (D) of FIG. 5 schematically represents the state after step S211 and step S214, as in FIG. 5 (B).
Dp = − (Ib−Min) −0.25 (Min−Ia) (3)
　そして、ステップＳ２１０の判断が「Ｎ」（＝Ｎｏ）の場合は、更新後の画素値Ｉａが最小値Ｍｉｎ未満であって更新前の画素値Ｉｂが平均値Ａｖを超えている場合となる（ステップＳ２１２）。その場合には、その画素は、周囲の画素との画素値のバランスが不自然である。そして、更新量Ｄｃは、最小値Ｍｉｎから更新前の画素値Ｉｂを差し引いた値Ｄｐへと変更される（ステップＳ２１３）。この結果、修正値Ｉａ’の画素値は、最小値Ｍｉｎと等しくなる。図５の（Ｅ）に示す矢印は、ステップＳ２１３およびステップＳ２１４を経た状態を、図５（Ｂ）と同様に概略的に表している。 If the determination in step S210 is “N” (= No), the updated pixel value Ia is less than the minimum value Min, and the updated pixel value Ib exceeds the average value Av ( Step S212). In that case, the pixel value balance with the surrounding pixels is unnatural. Then, the update amount Dc is changed to a value Dp obtained by subtracting the pixel value Ib before update from the minimum value Min (step S213). As a result, the pixel value of the correction value Ia ′ becomes equal to the minimum value Min. The arrow shown in (E) of FIG. 5 schematically represents the state after step S213 and step S214, as in FIG. 5 (B).
　以上のステップＳ２０４，Ｓ２０６，Ｓ２０８，Ｓ２１０に示す判断の処理は、所定の基準を満たすか否かを判定する処理となる。なお、このステップＳ２０４，Ｓ２０６，Ｓ２０８，Ｓ２１０の判断、およびステップＳ２１２の条件を満足するか否かを判断基準としたものの判断の順序は、適宜変更できる。その変更を行ったときも、実際に判断するのは最初から４つめまでである。その４つに該当しなかった場合は、必ず残りの条件に当てはまることとなるため、残りの条件に該当するか否かは、図４におけるステップＳ２１２と同様に所定の基準を満たすか否かを判定する処理とはならない。 The determination processing shown in the above steps S204, S206, S208, and S210 is processing for determining whether or not a predetermined standard is satisfied. It should be noted that the order of determination based on the determination in steps S204, S206, S208, and S210 and whether or not the condition in step S212 is satisfied can be changed as appropriate. Even when the change is made, the first to fourth judgment is actually made. If none of the four conditions is met, the remaining conditions will always be met. Therefore, whether the remaining conditions are met is determined by whether or not the predetermined criteria are satisfied as in step S212 in FIG. It is not a judgment process.
　このように、得られたＤｐを修正した更新量とし、修正値Ｉａ’が得られることとなる（ステップＳ２１４）。この結果、ある画素が更新される（ステップＳ２１５）。そして、全ての画素が更新されたか否かの判断を行う（ステップＳ２１６）。その判断が「Ｎ」（＝Ｎｏ）の場合は、参照する画素を変更して（ステップＳ２１７）、変更後の参照する画素（他の一部の信号要素）から所定の基準を求めるためおよび次の画素の修正値Ｉａ’を得るためにステップＳ２０２に戻り、ステップＳ２０２～２１７の処理を繰り返す。この処理では、一つの画素の更新量を修正したら、参照する画素を変更して別の一つの画素の更新量を変更している。そして、ステップＳ２１６の判断が「Ｙ」（＝Ｙｅｓ）の場合は、全ての画素の修正値Ｉａ’を用いて復元データＩ０＋ｎを修正する（ステップＳ２１８）。以上で図２におけるステップＳ１０７，１０８が終了する。 In this way, the obtained Dp is used as an updated amount, and the corrected value Ia ′ is obtained (step S214). As a result, a certain pixel is updated (step S215). Then, it is determined whether or not all the pixels have been updated (step S216). If the determination is “N” (= No), the pixel to be referred to is changed (step S217), and a predetermined standard is obtained from the changed reference pixel (other signal elements) and In order to obtain the correction value Ia ′ of the pixel, the process returns to step S202, and the processes of steps S202 to 217 are repeated. In this process, when the update amount of one pixel is corrected, the reference pixel is changed to change the update amount of another pixel. When the determination in step S216 is “Y” (= Yes), the restoration data I 0 + n is corrected using the correction value Ia ′ of all pixels (step S218). The steps S107 and 108 in FIG.
（Ｐ’の原点を設定する処理）
　上述の図２におけるステップＳ１２２で行う、Ｐ’の原点を設定する処理について図を用いて説明する。 (Process to set the origin of P ')
The process for setting the origin of P ′ performed in step S122 in FIG. 2 will be described with reference to the drawings.
　要因情報保存部７には、上述の図２におけるステップＳ１２２の段階のデータが保存される。すなわち、図６に示す信号処理装置１のブレと推定できるデータが要因情報保存部７に保存される。この図６には、図１に示す信号処理装置１の外観が示されている。この信号処理装置１の図に示すＸＹＺの各方向軸を中心とした回転がブレとなるが、特に現れやすいのはＸとＹの方向軸回りである。図７に示すブレに関するデータは、図６におけるＸ－Ｙ平面上における座標データの経時変化のデータである。そのデータは、図７に示すＸ－Ｙ平面で表されるようなブレの軌跡の情報およびその軌跡上の各位置にどの程度の期間とどまっていたかの情報を含む。図７に示すＸ－Ｙ平面の始点Ａ（Ｘ１，Ｙ１）は、撮影開始位置であり、軌跡の終点Ｂ（ＸＮ，ＹＮ）は、撮影終了位置である。 The factor information storage unit 7 stores the data at the stage of step S122 in FIG. That is, data that can be estimated as blurring of the signal processing device 1 shown in FIG. FIG. 6 shows the appearance of the signal processing apparatus 1 shown in FIG. The rotation of the signal processing apparatus 1 about the XYZ direction axes shown in the drawing is blurred, but it is particularly likely to appear around the X and Y direction axes. The data relating to the blur shown in FIG. 7 is data of change over time of the coordinate data on the XY plane in FIG. The data includes information on the locus of blur as represented by the XY plane shown in FIG. 7 and information on how long it has been at each position on the locus. The starting point A (X 1 , Y 1 ) on the XY plane shown in FIG. 7 is the imaging start position, and the end point B (X N , Y N ) of the trajectory is the imaging end position.
　ブレによる画像の劣化は、光エネルギーが１点に集中せずに、光エネルギーが図７に示す軌跡Ａ－Ｂ上に分散する現象である。よって分散した光エネルギーを１点に集中させることが、原画像を元画像へと復元することとなる。その光エネルギーを集中させる１点は、自由に決定できる。たとえば図４におけるＡ点、Ｂ点、Ａ－Ｂの軌跡上またはＡ－Ｂの軌跡を外れた点に決定できる。 Degradation of the image due to blurring is a phenomenon in which the light energy is not concentrated on one point but is scattered on the locus AB shown in FIG. Therefore, concentrating the dispersed light energy at one point restores the original image to the original image. One point for concentrating the light energy can be freely determined. For example, it is possible to determine the points A, B, AB on the locus in FIG. 4 or points off the AB locus.
　ここで、分散した光エネルギーを集中させる点を「原点位置」と言うこととし、原点位置は、図７に示すＸ－Ｙ平面上の点０座標（０ｘ，０ｙ）で表すこととする。また、上述したステップＳ１２０で算出したＰＳＦの骨格部をステップＳ１２１の処理で強調したデータ（Ｐ’）をＧ（Ｘｎ，Ｙｎ）で表すこととする。これは、各位置（Ｘｎ，Ｙｎ）においてどの程度の期間とどまっていたかの情報である「重さ」を示し、（４）式を満たす。（４）式は、光エネルギーを１と正規化して取り扱うことを示す。なお、（Ｘｎ，Ｙｎ）は、図７に示すＸ－Ｙ平面上の座標である。
Figure JPOXMLDOC01-appb-I000001
・・・（４） Here, the point where the dispersed light energy is concentrated is referred to as an “origin position”, and the origin position is represented by a point 0 coordinate (0x, 0y) on the XY plane shown in FIG. Further, data (P ′) in which the skeleton part of the PSF calculated in step S120 described above is emphasized in the process of step S121 is represented by G (Xn, Yn). This indicates “weight”, which is information indicating how long the position (Xn, Yn) has been, and satisfies the equation (4). Equation (4) indicates that light energy is normalized to 1 and handled. Note that (Xn, Yn) is a coordinate on the XY plane shown in FIG.
Figure JPOXMLDOC01-appb-I000001
... (4)
　また、分散した光エネルギーを原点位置である点０へ集中させる移動エネルギーをＥ（０ｘ，０ｙ）で表すこととする。すると分散したエネルギーを原点位置（０ｘ、０ｙ）に集中させる移動エネルギーは、移動距離と重さの関数で表現することができ、たとえば以下の式（５）（ｎ＝１，２，・・・Ｎ：Ｎは分散して広がった領域数）で表わすことができる。
Figure JPOXMLDOC01-appb-I000002
・・・（５）
　また、より移動距離に重きを置く場合は、以下の式（６）を用いることもできる。この（６）式の方が平方根の計算が無く、計算が楽になる利点がある。
Figure JPOXMLDOC01-appb-I000003
・・・（６）
　さらに、より計算を楽にするためには、以下の式（７）を用いることもできる。
Figure JPOXMLDOC01-appb-I000004
・・・（７） Further, the movement energy for concentrating the dispersed light energy at the point 0 which is the origin position is represented by E (0x, 0y). Then, the movement energy for concentrating the dispersed energy at the origin position (0x, 0y) can be expressed by a function of the movement distance and weight. For example, the following equation (5) (n = 1, 2,... N: N is the number of regions spread in a distributed manner).
Figure JPOXMLDOC01-appb-I000002
... (5)
Moreover, when placing more weight on the moving distance, the following equation (6) can also be used. This formula (6) has the advantage that there is no square root calculation and the calculation is easy.
Figure JPOXMLDOC01-appb-I000003
... (6)
Furthermore, in order to make calculation easier, the following equation (7) can be used.
Figure JPOXMLDOC01-appb-I000004
... (7)
　そして、移動エネルギーＥ（０ｘ，０ｙ）を最小値とする原点位置（０ｘ，０ｙ）を設定する。この設定は、処理部４で行われる。また、処理部４は、新たに原点位置が設定されたデータを、その設定の前に要因情報保存部７に保存されていたデータに代えて保存する。 Then, the origin position (0x, 0y) having the minimum value of the movement energy E (0x, 0y) is set. This setting is performed by the processing unit 4. Further, the processing unit 4 stores the data in which the origin position is newly set in place of the data stored in the factor information storage unit 7 before the setting.
（信号復元処理）
　次に、以上のように構成された本実施の形態に係る信号処理装置１の処理部４の画像復元処理方法（復元手段）の概要を、図８および図９に基づいて説明する。なお、画像復元処理方法は先に説明したステップＳ１０２，Ｓ１０３，Ｓ１０４，Ｓ１０５，Ｓ１０６，Ｓ１０７，Ｓ１０８，Ｓ１１０の処理とほぼ同様であり、繰り返し処理となっている。 (Signal restoration processing)
Next, an outline of an image restoration processing method (restoration means) of the processing unit 4 of the signal processing device 1 according to the present embodiment configured as described above will be described with reference to FIGS. 8 and 9. Note that the image restoration processing method is substantially the same as the processing of steps S102, S103, S104, S105, S106, S107, S108, and S110 described above, and is a repetitive processing.
　ここで、原画像の元画像への復元処理を実行する時期は、撮影用の電源がオフされている時、処理部４が稼働していない時、処理部４の稼働率が低い時等、原画像を撮影した時期から遅らせた時期とすることができる。その場合には、記録部５に保存された原画像データおよび、要因情報保存部７に保存された、その原画像についての伝達関数等の変化要因情報が、それぞれが関連づけられた状態で長期間に渡り保存される。このように、原画像の復元処理を実行する時期を、原画像を撮影した時期から遅らせる利点は、種々の処理を伴う撮影時の処理部４の負担を軽減できることである。 Here, when the restoration process of the original image to the original image is performed, when the imaging power is turned off, when the processing unit 4 is not operating, when the operating rate of the processing unit 4 is low, etc. It can be a time delayed from the time when the original image was taken. In that case, the original image data stored in the recording unit 5 and the change factor information such as a transfer function for the original image stored in the factor information storage unit 7 are associated with each other for a long time. It is preserved over. As described above, the advantage of delaying the time for executing the restoration process of the original image from the time of shooting the original image is that the burden on the processing unit 4 at the time of shooting involving various processes can be reduced.
　図８中、「Ｉ０」は、任意の初期画像であって、処理部４の記録部に予め保存されている画像のデータである。「Ｉ０’」は、その初期画像のデータのＩ０の劣化画像のデータを示し、比較のための比較用データである。「Ｇ」は、図２に示す処理で推定された変化要因情報（＝劣化要因情報（伝達関数））のデータで、要因情報保存部７に保存されていたものを処理部４が抽出し、処理部４の記録部に保存されるものである。「Ｉｍｇ’」は、原画像のデータである。 In FIG. 8, “I 0 ” is an arbitrary initial image and is image data stored in advance in the recording unit of the processing unit 4. “I 0 ′” indicates data of a degraded image of I 0 of the initial image data, and is comparison data for comparison. “G” is data of change factor information (= degradation factor information (transfer function)) estimated in the process shown in FIG. 2, and the processing unit 4 extracts the data stored in the factor information storage unit 7, It is stored in the recording unit of the processing unit 4. “Img ′” is data of the original image.
　「δ」は、原画像データＩｍｇ’と、比較用データＩ０’との差分のデータである。「ｋ」は、変化要因情報のデータに基づく配分比である。「Ｉ０＋ｎ」は、初期画像のデータＩ０に、差分のデータδを変化要因情報のデータＧに基づいて配分して新たに生成した復元画像のデータ（復元データ）である。「Ｉｍｇ」は、元画像のデータである。ここで、ＩｍｇとＩｍｇ’の関係は、次の（８）式で表されるものとする。
Ｉｍｇ’＝Ｉｍｇ＊Ｇ　……（８）
ここで、「＊」は、重畳積分を表す演算子である。 “Δ” is difference data between the original image data Img ′ and the comparison data I 0 ′. “K” is an allocation ratio based on the data of the change factor information. “I 0 + n ” is restored image data (restored data) newly generated by allocating the difference data δ to the initial image data I 0 based on the data G of the change factor information. “Img” is data of the original image. Here, the relationship between Img and Img ′ is represented by the following equation (8).
Img ′ = Img * G (8)
Here, “*” is an operator representing a superposition integral.
　なお、差分のデータδは、対応する画素の単純な差分でも良い場合もあるが、一般的には、変化要因情報のデータＧにより異なり、次の（９）式で表される。
δ＝ｆ（Ｉｍｇ’，Ｉｍｇ，Ｇ）　…（９） The difference data δ may be a simple difference between corresponding pixels, but in general, the difference data δ differs depending on the data G of the change factor information, and is expressed by the following equation (9).
δ = f (Img ′, Img, G) (9)
　処理部４の処理ルーチンは、まず、分散した光エネルギーを集中させる原点位置を決定する（ステップＳ３００）。この決定は、既に図２に示すステップＳ１２２で行っている。よって、この処理の詳細な説明は省略する。そして、任意の画像データＩ０を用意する（ステップＳ３０１）。この初期画像のデータＩ０としては、劣化している原画像のデータＩｍｇ’を用いても良く、また、黒ベタ、白ベタ、灰色ベタ、市松模様等どのような画像のデータを用いても良い。ステップＳ３０２で、（８）式のＩｍｇの代わりに初期画像となる任意の画像のデータＩ０を入れ、劣化画像である比較用データＩ０’を求める。次に、原画像データＩｍｇ’と比較用データＩ０’とを比較し、差分のデータδを算出する（ステップＳ３０３）。 The processing routine of the processing unit 4 first determines the origin position where the dispersed light energy is concentrated (step S300). This determination has already been made in step S122 shown in FIG. Therefore, detailed description of this process is omitted. Then, prepare the arbitrary image data I 0 (step S301). As the initial image data I 0 , the deteriorated original image data Img ′ may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. good. In step S302, data I 0 of an arbitrary image that is an initial image is input instead of Img in equation (8), and comparison data I 0 ′ that is a degraded image is obtained. Next, the original image data Img ′ and the comparison data I 0 ′ are compared to calculate difference data δ (step S303).
　そして、差分のデータδの各々の絶対値が所定値未満であるか否かを判断する（ステップＳ３０４）。ステップＳ３０４で差分のデータδが所定値以上であれば、ステップＳ３０５で新たな復元画像のデータ（＝復元データ）を生成する処理を行う。すなわち、個々の信号要素が得られた個々の差分のデータδを変化要因情報のデータＧに基づいて、任意の画像データＩ０に配分し、新たな復元データＩ０＋ｎを生成する。 Then, it is determined whether or not each absolute value of the difference data δ is less than a predetermined value (step S304). If the difference data δ is greater than or equal to the predetermined value in step S304, a process of generating new restored image data (= restored data) is performed in step S305. That is, the individual difference data δ from which individual signal elements are obtained is distributed to arbitrary image data I 0 based on the data G of the change factor information, and new restored data I 0 + n is generated.
　そして、復元データＩ０＋ｎを生成する際に用いる更新量（配分値）の妥当性を判断し（ステップＳ３０６）、復元データＩ０＋ｎを修正する（ステップＳ３０７）。このステップＳ３０６およびステップＳ３０７の処理は、上述した図２に示すステップＳ１０６、Ｓ１０７で行う、配分値の妥当性を判断し修正する処理と同様に行う。よって、この処理の詳細な説明は省略する。 Then, the validity of the update amount (distribution value) used when generating the restored data I 0 + n is determined (step S306), and the restored data I 0 + n is corrected (step S307). The processing in step S306 and step S307 is performed in the same manner as the processing for determining and correcting the validity of the distribution value performed in steps S106 and S107 shown in FIG. Therefore, detailed description of this process is omitted.
　その後、図８のステップＳ３０２～Ｓ３０７を繰り返す。この繰り返しの最中の復元データＩ０＋ｎは、処理が行われた途中段階の復元データとなる。ステップＳ３０４において、各画素の差分のデータδの各々の絶対値が所定値未満となったら、繰り返し処理を終了する。そして、繰り返し処理を終了した時点での復元データＩ０＋ｎを元画像のデータＩｍｇと推定する。すなわち、各画素の差分のデータδの各々の絶対値の最高値または平均値が所定値より小さくなった場合、比較用データＩ０＋ｎ’の元となった復元データＩ０＋ｎは元画像のデータＩｍｇと非常に近似したものとなることから、その復元データＩ０＋ｎを元画像のデータＩｍｇと推定するのである。なお、記録部５には、初期画像のデータＩ０、変化要因情報のデータＧ、を記録しておき、必要により処理部４に渡すようにしても良い。 Thereafter, steps S302 to S307 in FIG. 8 are repeated. The restored data I 0 + n in the middle of this repetition is restored data in the middle of processing. In step S304, when each absolute value of the difference data δ of each pixel becomes less than a predetermined value, the iterative process is terminated. Then, the restored data I 0 + n at the time when the repetitive processing is completed is estimated as the original image data Img. That is, when the maximum value or average value of the absolute values of the difference data δ of each pixel becomes smaller than a predetermined value, the restored data I 0 + n that is the basis of the comparison data I 0 + n ′ is the original image data Img. Therefore, the restored data I 0 + n is estimated as the original image data Img. The initial image data I 0 and the change factor information data G may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary.
　上述した繰り返し処理方法（復元手段）の考え方をまとめると以下のようになる。すなわち、この処理方法においては、処理の解を逆問題としては解かず、合理的な解を求める最適化問題として解くのである。逆問題として解く場合、理論上は可能であるが、現実問題としては困難である。 Summarizing the concept of the above-described iterative processing method (restoration means) is as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible, but it is difficult as a real problem.
　最適化問題として解く場合において、本実施の形態では、次のような条件を前提としている。
すなわち、
（１）入力に対する出力は、一意に決まる。
（２）出力が同じであれば、入力は同じである。
（３）出力が同じになるように、入力を更新し、その更新量を妥当な値に修正しながら反復処理することにより、解を収束させていく。 In the case of solving as an optimization problem, the present embodiment assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The input is updated so that the outputs are the same, and the solution is converged by performing iterative processing while correcting the update amount to an appropriate value.
　このことを換言すれば、図９（Ａ）（Ｂ）に示すように、原画像のデータＩｍｇ’と近似である比較用データＩ０’（Ｉ０＋ｎ’）を生成できれば、その生成の元データとなる初期画像のデータＩ０または復元データＩ０＋ｎは、元画像のデータＩｍｇに近似したものとなる。 In other words, as shown in FIGS. 9A and 9B, if comparison data I 0 ′ (I 0 + n ′) that is approximate to the original image data Img ′ can be generated, the original data of the generation is generated. The initial image data I 0 or the restored data I 0 + n is approximate to the original image data Img.
　なお、この実施の形態では、差分のデータδの判定基準となる値は、各データを８ビット（０～２５５）で表した場合に、この実施の形態では「６」としている。すなわち、６より小さい、つまり５以下の時は、処理を終了している。 In this embodiment, the value that is the determination criterion for the difference data δ is “6” in this embodiment when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished.
　次に、図８に示す手ブレの復元処理方法（ステップＳ３０２，Ｓ３０３，Ｓ３０４，Ｓ３０５，Ｓ３０６，Ｓ３０７の反復処理（復元手段））の詳細を、図１０，図１１，図１２，図１３，図１４，図１５，図１６および図１７に基づいて説明する。 Next, details of the camera shake restoration processing method (repetitive processing (restoration means) of steps S302, S303, S304, S305, S306, and S307) shown in FIG. 8 will be described in detail with reference to FIG. 10, FIG. 11, FIG. This will be described with reference to FIGS. 14, 15, 16 and 17. FIG.
　（手ブレの復元アルゴリズム）
　手ブレが無いとき、所定の画素に対応する光エネルギーは、露光時間中、その画素に集中する。また、手ブレがある場合、光エネルギーは、露光時間中にブレた画素に分散する。さらに、露光時間中のブレがわかれば、露光時間中のエネルギーの分散の仕方がわかるため、ブレた画像からブレの無い画像を作ることが可能となる。 (Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is a camera shake, the light energy is distributed to the blurred pixels during the exposure time. Further, if the blur during the exposure time is known, the manner in which the energy is dispersed during the exposure time can be understood, so that it is possible to create a blur-free image from the blurred image.
　以下、簡単のため、横一次元で説明する。画素を左から順にＳ－１，Ｓ，Ｓ＋１，Ｓ＋２，Ｓ＋３，・・・，とし、ある画素Ｓに注目する。ブレが無いとき、露光時間中のエネルギーは、その画素に集中するため、エネルギーの集中度は「１．０」である。この状態を図１０に示す。このときの撮影結果を、図１１の表に示す。図１１に示すものが、劣化しなかった場合の正しい画像データＩｍｇとなる。なお、各データは、８ビット（０～２５５）のデータで表している。 Hereafter, for the sake of simplicity, it will be described in one horizontal dimension. The pixels are designated as S-1, S, S + 1, S + 2, S + 3,... In order from the left, and attention is paid to a certain pixel S. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 11 is the correct image data Img when no deterioration occurs. Each data is represented by 8-bit (0 to 255) data.
　露光時間中にブレがあり、露光時間中の５０％の時間はＳ番目の画素に、３０％の時間はＳ＋１番目の画素に、２０％の時間はＳ＋２番目の画素にそれぞれブレていたとする。エネルギーの分散の仕方は、図１２に示す表のとおりとなる。これが変化要因情報のデータＧとなる。 Suppose that there is blurring during the exposure time, 50% of the exposure time is blurred to the Sth pixel, 30% of time to the S + 1th pixel, and 20% of time to the S + 2th pixel. The way of energy dispersion is as shown in the table of FIG. This becomes the data G of the change factor information.
　露光時間中にブレがあり、露光時間中の５０％の時間はＳ番目の画素に、３０％の時間はＳ＋１番目の画素に、２０％の時間はＳ＋２番目の画素にそれぞれブレていたとする。エネルギーの分散の仕方は、図１２に示す表のとおりとなる。これが変化要因情報のデータＧとなる。上述の式（５）における「Ｎ」の値は「３」となり、「重み」としての５０％、３０％、および２０％の総和が「１」となる。よって、この変化要因情報Ｇ（ここでは、横一次元で考えるため、Ｇ（Ｘｎ）となる）は、上述の式（４）を満たす。 Suppose that there is blurring during the exposure time, 50% of the exposure time is blurred to the Sth pixel, 30% of time to the S + 1th pixel, and 20% of time to the S + 2th pixel. The way of energy dispersion is as shown in the table of FIG. This becomes the data G of the change factor information. The value of “N” in the above formula (5) is “3”, and the sum of 50%, 30%, and 20% as “weight” is “1”. Therefore, the change factor information G (here, G (Xn) is considered in a one-dimensional horizontal direction) satisfies the above-described formula (4).
　この図１２および式（５）に基づいて、移動エネルギーＥ（０ｘ，０ｙ）を算出する。ここでは、横一次元で考えるため、移動エネルギーは、Ｅ（０ｘ）となる。また、移動距離は、画素一つ分の移動距離を「１」として計算する。すると、分散した光エネルギーを画素「Ｓ」に集中させる場合の移動エネルギーは、Ｅ（０ｘ）は、以下のように計算され、求められる。
（１×０）＋（０×０．５）＋（１×０．３）＋（２×０．２）＝０．７ Based on this FIG. 12 and Formula (5), the movement energy E (0x, 0y) is calculated. Here, since it is considered in one horizontal dimension, the movement energy is E (0x). Also, the movement distance is calculated by assuming that the movement distance for one pixel is “1”. Then, the movement energy in the case where the dispersed light energy is concentrated on the pixel “S” is obtained by calculating E (0x) as follows.
(1 × 0) + (0 × 0.5) + (1 × 0.3) + (2 × 0.2) = 0.7
　同様に、分散した光エネルギーを画素「Ｓ＋１」に集中させる場合の移動エネルギーは、Ｅ（０ｘ）は、以下のように計算され、求められる。
（１×０．５）＋（０×０．３）＋（１×０．２）＝０．７ Similarly, E (0x) is calculated and calculated as follows when the dispersed light energy is concentrated on the pixel “S + 1”.
(1 × 0.5) + (0 × 0.3) + (1 × 0.2) = 0.7
　同様に、分散した光エネルギーを画素「Ｓ＋２」に集中させる場合の移動エネルギーは、Ｅ（０ｘ）は、以下のように計算され、求められる。
（２×０．５）＋（１×０．３）＋（０×０．２）＝１．３ Similarly, E (0x) is calculated and obtained as follows when the dispersed light energy is concentrated on the pixel “S + 2”.
(2 × 0.5) + (1 × 0.3) + (0 × 0.2) = 1.3
　以上の結果から、図１２の場合は、分散した光エネルギーを画素「Ｓ」または「Ｓ１」に集中させることで、移動エネルギーを最小値の「０．７」とすることができる。また、図１２の代わりに「Ｓ＝０．４５」「Ｓ＋１＝０．３」「Ｓ＋２＝０．２５」の場合は、画素「Ｓ＋１」への移動エネルギーの総和が最も小さくなる。すなわち、画素「Ｓ」への移動は「０．８」となり画素「Ｓ＋１」への移動は「０．７」となり、画素「Ｓ＋２」への移動が「１．２」となるためである。以下、分散した光エネルギーを移動エネルギーが最も小さい位置、すなわち上述の図１２の例で画素「Ｓ」へと、集中させる場合の繰り返し処理の詳細について説明する。 From the above results, in the case of FIG. 12, the moving energy can be set to the minimum value “0.7” by concentrating the dispersed light energy on the pixel “S” or “S1”. Further, when “S = 0.45”, “S + 1 = 0.3”, and “S + 2 = 0.25” instead of FIG. 12, the sum of the kinetic energy to the pixel “S + 1” is the smallest. That is, the movement to the pixel “S” is “0.8”, the movement to the pixel “S + 1” is “0.7”, and the movement to the pixel “S + 2” is “1.2”. Hereinafter, the details of the iterative process when the dispersed light energy is concentrated on the position where the moving energy is the smallest, that is, on the pixel “S” in the above-described example of FIG.
　ブレは、全ての画素で一様であり、線形問題として把握される。そして、上ブレ（縦ブレ）が無いとすると、ブレの状況は、図１３に示す表のとおりとなる。図１３中の「ブレ画像」として示されるデータが、劣化している原画像のデータＩｍｇ’となる。具体的には、たとえば「Ｓ－３」の画素の「１２０」は、ブレ情報である変化要因情報のデータＧの「０．５」「０．３」「０．２」の配分比に従い、「Ｓ－３」の画素に「６０」、「Ｓ－２］の画素に「３６」、「Ｓ－１」の画素に「２４」というように分散する。同様に、「Ｓ－２」の画素データである「６０」は、「Ｓ－２」に「３０」、「Ｓ－１」に「１８」、「Ｓ」に「１２」として分散する。この劣化している原画像データＩｍｇ’と、図１２に示す変化要因情報のデータＧから元画像データＩｍｇを算出することとなる。 ¡Blurry is uniform for all pixels and is recognized as a linear problem. If there is no upper blur (vertical blur), the blur situation is as shown in the table of FIG. The data shown as “blurred image” in FIG. 13 becomes the degraded original image data Img ′. Specifically, for example, “120” of the pixel “S-3” is in accordance with the distribution ratio of “0.5”, “0.3”, and “0.2” of the data G of the change factor information that is blur information. Dispersed in such a manner that “60” is distributed to the “S-3” pixel, “36” is distributed to the “S-2” pixel, and “24” is distributed to the “S-1” pixel. Similarly, “60” which is the pixel data of “S-2” is distributed as “30” in “S-2”, “18” in “S-1”, and “12” in “S”. The original image data Img is calculated from the deteriorated original image data Img ′ and the change factor information data G shown in FIG.
　ステップＳ３０１に示す任意の画像データＩ０としては、どのようなものでも採用できるが、この説明に当たっては、原画像データＩｍｇ’を用いる。すなわち、Ｉ０＝Ｉｍｇ’として処理を開始する。図１４の表中に「入力」とされたものが初期画像のデータＩ０に相当する。このデータＩ０すなわちＩｍｇ’と、ステップＳ３０２で変化要因情報のデータＧとを重畳積分する。すなわち、たとえば、初期画像のデータＩ０の「Ｓ－３」の画素の「６０」は、Ｓ－３の画素に「３０」が、「Ｓ－２」の画素に「１８」が、「Ｓ－１」の画素に「１２」がそれぞれ割り振られる。他の画素についても同様に配分され、「出力Ｉ０’」として示される比較用データＩ０’が生成される。このため、ステップＳ３０３の差分のデータδは、図１４の最下欄に示すようになる。この差分のデータδの絶対値の最高値が所定値、たとえば１０未満となるか判断する（ステップＳ３０４）、この例では「Ｓ－３」の画素の差分のデータδが３０であり、ステップＳ３０４でＮｏ（＝Ｎ）となり、ステップＳ３０５へ移行する。 Any arbitrary image data I 0 shown in step S301 can be adopted, but in this description, original image data Img ′ is used. That is, the process starts with I 0 = Img ′. In the table of FIG. 14, “input” corresponds to the initial image data I 0 . This data I 0, that is, Img ′ is superposed and integrated with the change factor information data G in step S302. That is, for example, “60” of the “S-3” pixel of the initial image data I 0 is “30” for the S-3 pixel, “18” for the “S-2” pixel, “12” is assigned to each pixel of “−1”. The other pixels are similarly distributed, and comparison data I 0 ′ shown as “output I 0 ′” is generated. Therefore, the difference data δ in step S303 is as shown in the bottom column of FIG. It is determined whether the maximum absolute value of the difference data δ is a predetermined value, for example, less than 10 (step S304). In this example, the difference data δ of the pixel “S-3” is 30, and step S304 is performed. No (= N) and the process proceeds to step S305.
　差分のデータδの配分は、図１５に示すように、たとえば「Ｓ－３」の画素データ「３０」に、自分の所（＝「Ｓ－３」の画素）の配分比である０．５をかけた「１５」を「Ｓ－３」の画素に配分し、また「Ｓ－２」の画素のデータ「１５」にその「Ｓ－２」の画素にきているはずの配分比である０．３をかけた「４．５」を配分し、さらに、「Ｓ－１」の画素のデータ「９．２」に、その「Ｓ－１」の画素にきているはずの配分比である０．２をかけた「１．８４」を配分する。「Ｓ－３」の画素に配分された総量（各画素毎の更新量Ｄｃ）は、「２１．３４」となり、この値を図４における初期画像のデータＩ０である、図４におけるＩｂ（ここでは原画像データＩｍｇ’を使用）にプラスして、図８における復元データＩ０＋１である、図４における更新後の画素値Ｉａが算出される。この例では、図１５に示すように更新後の画素値Ｉａは「８１．３４」となる。このように、差分のデータδを変化要因情報のデータＧを使用して、任意の画像のデータＩ０に配分して、図１５中の「次回入力」として示される復元データＩ０＋ｎを生成する。この場合、第１回目であるため、図１５では、Ｉ０＋１と表している。 As shown in FIG. 15, the distribution of the difference data δ is 0.5, which is the distribution ratio of the pixel data “30” of “S-3” to the pixel data “30” (= the pixel of “S-3”). “15” multiplied by “S-3” is distributed to the pixel “S-3”, and “S-2” pixel data “15” is allocated to the “S-2” pixel. “4.5” multiplied by 0.3 is allocated, and further, the data “9.2” of the pixel “S-1” is allocated to the pixel “S-1” by the distribution ratio. “1.84” multiplied by a certain 0.2 is allocated. The total amount (update amount Dc for each pixel) allocated to the pixels of “S-3” is “21.34”, and this value is the initial image data I 0 in FIG. Here, the updated pixel value Ia in FIG. 4 which is the restored data I 0 + 1 in FIG. 8 is calculated in addition to the original image data Img ′. In this example, as shown in FIG. 15, the updated pixel value Ia is “81.34”. In this way, the difference data δ is distributed to the arbitrary image data I 0 using the change factor information data G to generate the restored data I 0 + n shown as “next input” in FIG. . In this case, since this is the first time, it is represented as I 0 + 1 in FIG.
　この後、図８におけるステップＳ３０６にて更新量の妥当性を判断する。具体的には、この復元データＩ０＋１を修正するために、画素毎の更新後の画素値（＝Ｉａ）を算出する。この算出は、上述の通り（１）式による。そこで、差分のデータδを各画素に配分する。そして、図１５に示すように、各画素ごとに、参照する各々の画素の更新前の画素値（＝Ｉｂ）の最小値（＝Ｍｉｎ）、最大値（＝Ｍａｘ）および平均値（＝Ａｖ）を算出する（図４におけるステップＳ２０２）。たとえば画素「Ｓ－３」は、画素「Ｓ－３」、画素「Ｓ－２」および画素「Ｓ－１」を参照している。よって、図１５に示すように、画素「Ｓ－３」、画素「Ｓ－２」および画素「Ｓ－１」の最小値（＝Ｍｉｎ）、最大値（＝Ｍａｘ）および平均値（＝Ａｖ）を算出する。図に示す例では、画素「Ｓ－３」に関しては、最大値が画素「Ｓ－１」に「８２．００」で、最小値が画素「Ｓ－３」の「６６．００」となり、平均値は、画素「Ｓ－３」「Ｓ－２」「Ｓ－１」の各値の合計を３で割った値で「６９．３３」となる。画素「Ｓ－２」～「Ｓ＋４」についても同様の算出を行う。 Thereafter, the validity of the update amount is determined in step S306 in FIG. Specifically, in order to correct the restored data I 0 + 1 , an updated pixel value (= Ia) for each pixel is calculated. This calculation is based on equation (1) as described above. Therefore, the difference data δ is distributed to each pixel. Then, as shown in FIG. 15, for each pixel, the minimum value (= Min), maximum value (= Max), and average value (= Av) of the pixel value (= Ib) before the update of each pixel to be referred to Is calculated (step S202 in FIG. 4). For example, the pixel “S-3” refers to the pixel “S-3”, the pixel “S-2”, and the pixel “S-1”. Therefore, as shown in FIG. 15, the minimum value (= Min), maximum value (= Max), and average value (= Av) of the pixel “S-3”, the pixel “S-2”, and the pixel “S-1”. Is calculated. In the example shown in the figure, regarding the pixel “S-3”, the maximum value is “82.00” in the pixel “S-1”, the minimum value is “66.00” in the pixel “S-3”, and the average The value is “69.33”, which is a value obtained by dividing the sum of the respective values of the pixels “S-3”, “S-2”, and “S-1” by 3. The same calculation is performed for the pixels “S−2” to “S + 4”.
　そして、ＩａおよびＩｂが図４におけるステップＳ２０４，Ｓ２０６，Ｓ２０８，Ｓ２１０およびＳ２１２のいずれかの条件を満足するのかを判定する。たとえば画素「Ｓ－３」のＩａは、「Ｍｉｎ（６０．００）≦Ｉａ（８１．３４）≦Ｍａｘ（８２．００）」の条件を満足するため、図４におけるステップＳ２０４の条件を満足する。よって、図１５に示すようにステップＳ２０５の処理を行い、Ｄｃである更新量「２１．３４」をそのまま修正後の更新量Ｄｐとして用い、修正値Ｉａ’は、修正前のＩａと等しい「８１．３４」となる。「Ｓ－２」～「Ｓ」、「Ｓ＋２」および「Ｓ＋３」の更新後の画素値Ｉａについても同様の修正を行う。この修正値Ｉａ’が復元データＩ０＋ｎの修正値となる。 Then, it is determined whether Ia and Ib satisfy any of the conditions in steps S204, S206, S208, S210, and S212 in FIG. For example, Ia of the pixel “S-3” satisfies the condition of “Min (60.00) ≦ Ia (81.34) ≦ Max (82.00)”, and therefore satisfies the condition of step S204 in FIG. . Therefore, the process of step S205 is performed as shown in FIG. 15, and the update amount “21.34” as Dc is used as it is as the update amount Dp after correction, and the correction value Ia ′ is equal to “81 before correction” “81 .34 ". The same correction is performed on the updated pixel values Ia of “S−2” to “S”, “S + 2”, and “S + 3”. This correction value Ia ′ becomes the correction value of the restored data I 0 + n .
　たとえば画素「Ｓ＋１」では、ステップＳ２０４、Ｓ２０６の条件を満足せず、ステップＳ２０８に移行する。そして、「Ｉｂ（１２１．００）＞Ａｖ（１１３．３３）」であり、「Ｉａ（１３０．１１）＞Ｍａｘ（１２１．００）」の条件を満足するため、図４におけるステップＳ２０８の条件を満足することになる。よって、図１５に示すようにステップＳ２０９の処理を行い、「２．２８」を修正後の更新量Ｄｐとして用い、修正値Ｉａ’は「１２３．２８」となる。画素「Ｓ＋４」のＩａについても同様の修正を行う。このＩａ’が復元データＩ０＋ｎの修正値となる。 For example, in the pixel “S + 1”, the conditions of steps S204 and S206 are not satisfied, and the process proceeds to step S208. In order to satisfy “Ib (121.00)> Av (113.33)” and satisfy the condition “Ia (130.11)> Max (121.00)”, the condition of step S208 in FIG. You will be satisfied. Accordingly, the process of step S209 is performed as shown in FIG. 15, and “2.28” is used as the updated amount Dp after correction, and the correction value Ia ′ is “123.28”. The same correction is performed for Ia of the pixel “S + 4”. This Ia ′ becomes a correction value of the restored data I 0 + n .
　図１６に示すように、この修正された復元データＩ０＋１（Ｉａ'）がステップＳ３０２の新たな入力画像のデータ（＝初期画像のデータＩ０に代わるもの）になり、ステップＳ３０２が実行され、ステップＳ３０３へと移行し、新しい差分のデータδを得る。その差分のデータδの大きさをステップＳ３０４で判断し、所定値より大きい場合、ステップＳ３０５で新しい差分のデータδを前回の修正された復元データＩ０＋１に配分し、新しい復元データＩ０＋２を生成することとなるが、その際に、図１５の説明と同様にして新しい復元データＩ０＋２を修正する（図１７参照）。たとえば、画素「Ｓ－３」、「Ｓ」および「Ｓ＋３」の更新後の画素値Ｉａについては、上述の図１５における画素「Ｓ－３」と同様の修正を行う。すなわち、更新後の画素値Ｉａがそのまま修正値Ｉａ’となる。 As shown in FIG. 16, the corrected restored data I 0 + 1 (Ia ′) becomes the new input image data (= replaces the initial image data I 0 ) in step S302, and step S302 is executed. The process proceeds to step S303 to obtain new difference data δ. The size of the difference data δ is determined in step S304. If the difference data δ is larger than the predetermined value, the new difference data δ is distributed to the previously modified restored data I 0 + 1 in step S305 to generate new restored data I 0 + 2 . In this case, the new restoration data I 0 + 2 is corrected in the same manner as described in FIG. 15 (see FIG. 17). For example, the updated pixel value Ia of the pixels “S-3”, “S”, and “S + 3” is corrected in the same manner as the pixel “S-3” in FIG. That is, the updated pixel value Ia becomes the corrected value Ia ′ as it is.
　また、たとえば画素「Ｓ＋１」では、ステップＳ２０４，Ｓ２０６の条件を、満足せず、ステップＳ２０８に移行する。そして「Ｉｂ（１２１．００）＞Ａｖ（１１６．１５）」であり、「Ｉａ（１２５．４３）＞Ｍａｘ（１２１．００）」の条件を満足するため、図４におけるステップＳ２０８の条件を満足することとなる。よって、図１７に示すようにステップＳ２０９の処理を行い、上述の（２）式による値「１．１０６」を修正後の更新量Ｄｐとして用い、修正値Ｉａ’は「１２２．１１」となる。画素「Ｓ＋４」の更新後の画素値Ｉａについても同様の修正を行う。この修正値Ｉａ’が復元データＩ０＋ｎの修正値となる。 For example, in the pixel “S + 1”, the conditions in steps S204 and S206 are not satisfied, and the process proceeds to step S208. Then, “Ib (121.00)> Av (116.15)” and the condition “Ia (125.43)> Max (121.00)” is satisfied, and therefore the condition of step S208 in FIG. 4 is satisfied. Will be. Therefore, the process of step S209 is performed as shown in FIG. 17, and the correction value Ia ′ is “122.11” using the value “1.16” according to the above-described equation (2) as the update amount Dp after correction. . The same correction is performed on the updated pixel value Ia of the pixel “S + 4”. This correction value Ia ′ becomes the correction value of the restored data I 0 + n .
　また、たとえば画素「Ｓ－２」では、ステップＳ２０４，Ｓ２０６，Ｓ２０８の条件を、満足せず、ステップＳ２１０に移行する。そして「Ｉｂ（７７．３０）≦Ａｖ（８７．６７）」であり、「Ｉａ（７６．９７）＜Ｍｉｎ（７７．３０）」の条件を満足するため、図４におけるステップＳ２１０の条件を満足することとなる。よって、図１７に示すようにステップＳ２１１の処理を行い、上述の（３）式による値「－０．０８２」を修正後の更新量Ｄｐとして用い、修正値Ｉａ’は「７７．２２」となる。画素「Ｓ－１」および「Ｓ＋２」の更新後の画素値Ｉａについても同様の修正を行う。この修正値Ｉａ’が復元データＩ０＋ｎの修正値となる。 For example, in the pixel “S-2”, the conditions of steps S204, S206, and S208 are not satisfied, and the process proceeds to step S210. Then, “Ib (77.30) ≦ Av (87.67)” is satisfied, and the condition “Ia (76.97) <Min (77.30)” is satisfied, so the condition of step S210 in FIG. 4 is satisfied. Will be. Therefore, as shown in FIG. 17, the process of step S211 is performed, and the value “−0.082” according to the above equation (3) is used as the updated amount Dp after correction, and the correction value Ia ′ is “77.22”. Become. The same correction is performed on the updated pixel value Ia of the pixels “S−1” and “S + 2”. This correction value Ia ′ becomes the correction value of the restored data I 0 + n .
　その後、修正された復元データＩ０＋２を用いてステップＳ３０２を遂行することにより、修正された復元データＩ０＋２から新しい比較用データＩ０＋２ ’が生成される。このように、ステップＳ３０２，Ｓ３０３が実行された後、ステップＳ３０４へ行き、そこでの判断によりステップＳ３０５へ移行する。このような処理を繰り返す。 Thereafter, by performing the step S302 by using the restored data I 0 + 2 that are fixed, a new comparison data I 0 + 2 'from the restored data I 0 + 2 that are fixed are generated. As described above, after steps S302 and S303 are executed, the process proceeds to step S304, and the process proceeds to step S305 based on the determination. Such a process is repeated.
（本実施の形態によって得られる主な効果）
　本実施の形態に係る変化要因情報の生成法の処理（図２）を実行することによって、変化要因情報のデータ（ＰＳＦ）が未知であっても妥当なＰＳＦを生成でき、実用的な信号復元が可能となる。よって、ブレ等を機械的に測定する速度センサまたは加速度センサをカメラ等の信号処理装置１の構成要素から省略することができる。また、この処理の際に、処理部４が、図２に示すステップＳ１０３からＳ１０８の処理を２回以上（たとえば１０回）繰り返していることによって、より妥当なＰＳＦが生成される。また、この処理の際に、処理部４が、配分値の妥当性を判断し修正する処理（ステップＳ１０７、Ｓ１０８）をすることによって、配分による極端なデータの値の変化を抑制でき、より妥当なＰＳＦが生成される。また、この処理の際に、処理部４が、Ｐ’の原点を再設定する処理をすることによって、配分による極端なデータの値の変化を抑制でき、より妥当なＰＳＦが生成される。また、この処理の際に、処理部４が、縮小処理（ステップＳ１０２）を実行することによって、変化の大きさも縮小されることから、ＰＳＦを推定し易くなり、より妥当なＰＳＦが生成される。 (Main effects obtained by this embodiment)
By executing the process of the change factor information generation method according to the present embodiment (FIG. 2), a valid PSF can be generated even if the data (PSF) of the change factor information is unknown, and practical signal restoration is possible. Is possible. Therefore, a speed sensor or an acceleration sensor that mechanically measures blur or the like can be omitted from the components of the signal processing device 1 such as a camera. In this process, the processing unit 4 repeats the processes of steps S103 to S108 shown in FIG. 2 two or more times (for example, 10 times), thereby generating a more appropriate PSF. Further, during this process, the processing unit 4 can perform a process for determining and correcting the validity of the distribution value (steps S107 and S108), thereby suppressing an extreme change in the value of the data due to the distribution. PSF is generated. Further, during this process, the processing unit 4 performs a process of resetting the origin of P ′, whereby an extreme change in the value of data due to the distribution can be suppressed, and a more appropriate PSF is generated. Further, during this process, the processing unit 4 executes the reduction process (step S102), whereby the magnitude of the change is also reduced. Therefore, it becomes easier to estimate the PSF, and a more appropriate PSF is generated. .
　また、この処理の際に、処理部４が、拡大処理（ステップＳ１２５）の拡大率を徐々に大きくして繰り返し行うことによって、縮小処理を行って得た妥当な骨格部を有するＰＳＦに基づいて良質でかつ大きなＰＳＦの推定をすることができる。また、この処理の際に、処理部４が、ＰＳＦの骨格部の強調をする処理（ステップＳ１２１）を行うことによって、ブレに相当する変化の推定に重きを置くＰＳＦを生成できる。また、この処理の際に、処理部４が、ＰＳＦ初期値をガウシアンディスクに設定することで（ステップＳ１０１）、変化がどのようなブレまたはボケを含んでいても、良好なＰＳＦの推定をすることができる。 Further, during this process, the processing unit 4 repeatedly performs the enlargement process (step S125) by gradually increasing the enlargement ratio, thereby performing the process based on the PSF having an appropriate skeleton obtained by performing the reduction process. A high quality and large PSF can be estimated. Further, during this process, the processing unit 4 performs a process (step S121) of emphasizing the skeleton part of the PSF, thereby generating a PSF that puts weight on estimation of a change corresponding to blurring. In this process, the processing unit 4 sets a PSF initial value to a Gaussian disk (step S101), and estimates a good PSF regardless of any blur or blur. be able to.
　本実施の形態に係る信号処理装置１は、ＰＳＦが未知であっても妥当なＰＳＦを生成でき、実用的な信号復元が可能である。よって、ブレ等を機械的に測定する速度センサまたは加速度センサをカメラ等は、信号処理装置１の構成要素から省略することができる。 The signal processing apparatus 1 according to the present embodiment can generate an appropriate PSF even if the PSF is unknown, and can perform practical signal restoration. Therefore, a speed sensor or an acceleration sensor that mechanically measures blur or the like can be omitted from the components of the signal processing apparatus 1.
　また、信号処理装置１は、図８に示すステップＳ３０２～ステップＳ３０７を繰り返すことで、差分のデータδが徐々に小さくなっていき、所定値より小さくなると、ブレていない元画像データＩｍｇが得られる。また、このとき、修正処理（図４，図５）を行っているため、得られる元画像データＩｍｇと推定される画像データは、リンギングの発生が軽減され、画像の復元状態が良好なものとなる。また、修正処理（図４,図５）によって、復元データを構成する信号要素のデータのうち、不自然と思われるものを修正し、画素値の大きな変化を抑制しているため、仮に図２に示す処理によって推定される変化要因情報のデータＧが信頼性の低いものであったとしても、妥当な画像の復元が可能となる。 Further, the signal processing apparatus 1 repeats Steps S302 to S307 shown in FIG. 8 so that the difference data δ gradually decreases. When the difference data δ becomes smaller than a predetermined value, the original image data Img without blur is obtained. . At this time, since the correction process (FIGS. 4 and 5) is performed, the image data estimated as the original image data Img obtained is reduced in the occurrence of ringing and has a good image restoration state. Become. In addition, the correction process (FIGS. 4 and 5) corrects unnatural data among the signal element data constituting the restored data and suppresses a large change in the pixel value. Even if the data G of the change factor information estimated by the processing shown in (2) is low in reliability, it is possible to restore an appropriate image.
（他の形態）
　以上、本実施の形態における変化要因情報の生成法および信号処理装置１について説明したが、本発明の要旨を逸脱しない限り種々変更実施可能である。たとえば、変化要因情報の生成法においては、処理部４が、図２に示すステップＳ１０３からＳ１０８の処理を２回以上繰り返しているが、１回のみとすることができる。さらに、処理部４が、配分値の妥当性を判断し修正する処理（ステップＳ１０７、Ｓ１０８）、Ｐ’の原点を再設定する処理（ステップＳ１２２）、縮小処理（ステップＳ１０２）、拡大処理（ステップＳ１２５）、最終的に初期の画像データ用のＰＳＦのデータと同じ大きさのデータとする拡大処理（ステップＳ１２８）およびＰＳＦの骨格部の強調をする処理（ステップＳ１２１）の全部または一部は、省略できる。さらに、ＰＳＦの初期値は、ガウシアンディスクに設定（ステップＳ１０１）せず、任意の値を設定できる。 (Other forms)
As described above, the generation method of the change factor information and the signal processing apparatus 1 in the present embodiment have been described, but various modifications can be made without departing from the gist of the present invention. For example, in the method of generating change factor information, the processing unit 4 repeats the processing of steps S103 to S108 shown in FIG. 2 twice or more, but it can be performed only once. Furthermore, the processing unit 4 determines and corrects the validity of the distribution value (steps S107 and S108), resets the origin of P ′ (step S122), reduces (step S102), and expands (step S102). S125), all or part of the enlargement process (step S128) and the PSF skeleton part enhancement process (step S121) to be finally the same size as the PSF data for the initial image data, Can be omitted. Furthermore, the initial value of PSF can be set to an arbitrary value without being set in the Gaussian disk (step S101).
　また、たとえば信号処理装置１における信号復元処理には、図８に示す処理を採用しているが、他の処理、たとえばウィナーフィルターを用いる処理等を採用できる。また、図８に示す処理を採用する場合でも、原点位置を設定する処理（ステップＳ３００）および更新量の妥当性を判断し修正する処理（ステップＳ３０６、Ｓ３０７）の双方または一方は、省略することができる。さらに更新量の妥当性を判断し修正する処理（ステップＳ３０６、Ｓ３０７）を採用する場合でも、復元データＩ０＋ｎを修正する方法は、図４および図５に示す方法に限定されない。特に図４に示すステップＳ２０４，Ｓ２０６，Ｓ２０８，Ｓ２１０およびＳ２１２における場合分け、ステップＳ２０５，Ｓ２０７，Ｓ２０９，Ｓ２１１およびＳ２１３における差分のデータδを修正する方法は、それぞれ適宜変更することができる。 For example, the signal restoration process in the signal processing apparatus 1 employs the process shown in FIG. 8, but other processes such as a process using a Wiener filter can be employed. Even when the process shown in FIG. 8 is adopted, both or one of the process for setting the origin position (step S300) and the process for determining and correcting the validity of the update amount (steps S306 and S307) are omitted. Can do. Further, even when the process of determining and correcting the validity of the update amount (steps S306 and S307) is adopted, the method of correcting the restored data I 0 + n is not limited to the methods shown in FIGS. Particularly, the method of correcting the difference data δ in steps S205, S207, S209, S211 and S213 can be changed as appropriate according to the case in steps S204, S206, S208, S210 and S212 shown in FIG.
　また、処理部４が、図２に示すステップＳ１０３からＳ１０８の処理を１０回繰り返すこととしており、また、ステップＳ１２３において、ステップＳ１０３からＳ１２２までを繰り返すこととしているが、処理部４の処理速度の状況等によってそれらの回数を増減できる。また、信号処理装置１のユーザがそれらの回数を任意に設定できることとしても良い。さらに、縮小処理（ステップＳ１０２）および拡大処理（ステップＳ１２５）の画像の大きさの比率、拡大処理（ステップＳ１２５）の画像の大きさの比率の変更回数は、復元画像の画質等によってそれらの回数を増減できる。 In addition, the processing unit 4 repeats the processing of steps S103 to S108 shown in FIG. 2 ten times. In step S123, the processing unit 4 repeats steps S103 to S122. You can increase or decrease the number of times depending on the situation. Moreover, it is good also as the user of the signal processing apparatus 1 being able to set those frequency | counts arbitrarily. Furthermore, the number of changes in the ratio of the image size in the reduction process (step S102) and the enlargement process (step S125) and the ratio of the image size in the enlargement process (step S125) depend on the image quality of the restored image. Can be increased or decreased.
　また、縮小処理（ステップＳ１０２）および拡大処理（ステップＳ１２５）における縮小・拡大の仕方は、画素の間引き・隣り合う画素の画素値を平均した値の新たな画素の挿入によって行っている。しかし、他の手段、たとえば隣接する複数の画素の平均画素値の画素をその複数の画素の代わりに用いる縮小処理、隣り合う画素の画素値をそのまま新たな画素の画素値としてその新たな画素を挿入することによって拡大処理を行っても良い。さらに、ＰＳＦの骨格部の強調をする処理（ステップＳ１２１）において、骨格部のデータの値を２倍にしているが、１．５倍、３倍、４倍、または５倍等としても良い。 Further, the reduction / enlargement method in the reduction process (step S102) and the enlargement process (step S125) is performed by thinning out pixels and inserting a new pixel having a value obtained by averaging the pixel values of adjacent pixels. However, other means, for example, a reduction process using a pixel having an average pixel value of a plurality of adjacent pixels instead of the plurality of pixels, a pixel value of an adjacent pixel as it is as a pixel value of a new pixel, and the new pixel You may perform an expansion process by inserting. Further, in the process of enhancing the skeleton part of PSF (step S121), the value of the skeleton part data is doubled, but may be 1.5 times, 3 times, 4 times, 5 times, or the like.
　また、処理部４は、最終的に初期画像用のＰＳＦのデータと同じ大きさのデータとする拡大処理（ステップＳ１２８）を行っている。しかしこの処理によって初期画像用のＰＳＦのデータと同じ大きさのデータとする必要はなく、初期画像用のＰＳＦのデータと異なる大きさのデータへと拡大処理することができる。 In addition, the processing unit 4 performs an enlargement process (step S128) to finally make the data the same size as the PSF data for the initial image. However, this process does not require data having the same size as the PSF data for the initial image, and can be enlarged to data having a size different from that of the PSF data for the initial image.
　また、処理部４は、ＰＳＦを算出する処理を行い（ステップＳ１２０）、その処理は、得られた復元データＩ０＋ｎと原画像のデータＩｍｇ’をそれぞれフーリエ変換し、周波数空間での割り算によってＰＳＦの周波数特性を算出し、その周波数特性をフーリエ逆変換することによってＰＳＦ（＝Ｐ）を得るものである。しかし、他の手段によって、得られる復元データＩ０＋ｎと原画像のデータＩｍｇ’から新たなＰＳＦ（＝Ｐ）を得る処理を採用できる。 Further, the processing unit 4 performs a process of calculating a PSF (step S120). In this process, the obtained restored data I 0 + n and the original image data Img ′ are each Fourier-transformed, and the PSF is obtained by division in the frequency space. PSF (= P) is obtained by calculating the frequency characteristic of the signal and inverse Fourier transforming the frequency characteristic. However, a process for obtaining a new PSF (= P) from the obtained restored data I 0 + n and the original image data Img ′ can be adopted by other means.
　更新量の妥当性を判断し修正する処理（ステップＳ３０６、Ｓ３０７）においては、たとえば、参照する画素としては、影響を及ぼす範囲（影響範囲）のもの以外に、その影響範囲を囲む画素一つ分大きくした範囲としたり、修正しようとする画素を中心として所定の距離の範囲としても良い。また、所定の基準としては、参照する画素の最大値、最小値、平均値を利用してのものではなく、最大値と最小値のみを利用し、修正値を最大・最小を超えた分の１／４や１／３の値を最大値や最小値に付加するようにしても良い。また、上限を最大値の１．２倍、下限を最小値の０．８倍とし、その範囲に入れば、修正しないようにしても良い。すなわち、最大値のＸ倍、最小値のＹ倍の範囲に更新後の画素値Ｉａが入れば修正しないようにしても良い。 In the process of determining and correcting the appropriateness of the update amount (steps S306 and S307), for example, as a pixel to be referenced, one pixel surrounding the affected range is included in addition to the affected range (affected range). It may be a large range or a range of a predetermined distance centering on a pixel to be corrected. In addition, as a predetermined standard, the maximum value, minimum value, and average value of the reference pixel are not used, but only the maximum value and the minimum value are used, and the correction value exceeds the maximum / minimum value. You may make it add the value of 1/4 or 1/3 to the maximum value or the minimum value. Further, the upper limit is set to 1.2 times the maximum value and the lower limit is set to 0.8 times the minimum value. That is, if the updated pixel value Ia is in the range of X times the maximum value and Y times the minimum value, the pixel value Ia may not be corrected.
　本実施の形態に係る繰り返し処理では、処理部４は、図２におけるステップＳ１０５や図８におけるステップＳ３０４の一旦得た画像を再度処理するかの判断では、画像を構成する複数の各画素毎の差分のデータΔ、δの絶対値が全て所定値未満または絶対値の平均値が所定値未満であるか否かを判断し、画像全体の処理を行うか判断している。しかし、所定値との比較の対象を、画像を構成する複数の各画素毎の差分のデータとし、各画素毎に繰り返し処理を停止するか否かを判断するようにしたりしても良い。また、所定値との比較対象を、各画素の差分のデータδの総和、もしくは各画素の差分のデータΔ、δの絶対値の総和、または以上の４つのうちの２つ以上とすることができる。たとえば、各画素毎の差分のデータΔ、δの中で零から最も離れた値と、各画素毎の差分のデータδの総和の値とが、別々の基準を共に満たすか否かを判断するようにしても良い。このように、所定値と比較する値を適宜選択することで、原画像の種類、変化の状態または復元処理の状況に応じて、適切な処理を行うことができる。 In the iterative process according to the present embodiment, the processing unit 4 determines whether to process the image once obtained in step S105 in FIG. 2 or step S304 in FIG. 8 for each of a plurality of pixels constituting the image. It is determined whether or not the absolute values of the difference data Δ and δ are all less than a predetermined value or the average value of the absolute values is less than a predetermined value, and whether or not the entire image is to be processed. However, the comparison target with the predetermined value may be the difference data for each of the plurality of pixels constituting the image, and it may be determined whether or not the repeated processing is stopped for each pixel. The comparison target with the predetermined value may be the sum of the difference data δ of each pixel, or the sum of the absolute values of the difference data Δ and δ of each pixel, or two or more of the above four. it can. For example, it is determined whether or not the difference data Δ and δ for each pixel farthest from zero and the sum of the difference data δ for each pixel satisfy different criteria. You may do it. As described above, by appropriately selecting a value to be compared with the predetermined value, appropriate processing can be performed according to the type of original image, the state of change, or the status of restoration processing.
　上述の実施の形態では、復元対象を画像データとしている。しかし、これらの復元処理の考え方および手法は、あらゆるデジタルデータの復元処理に適用できる。たとえば、デジタルの音声データの復元等への適用が可能である。その適用の結果、リンギングのように一部に不正確な音声データ等が発生することを効率よく抑制でき、また変化要因情報のデータが不正確であっても、妥当な結果が得られる復元処理が可能となる。 In the above-described embodiment, the restoration target is image data. However, these restoration processing concepts and techniques can be applied to any digital data restoration processing. For example, it can be applied to restoration of digital audio data. As a result of this application, it is possible to efficiently suppress the occurrence of some inaccurate audio data, such as ringing, and even if the data of the change factor information is inaccurate, a restoration process that can obtain reasonable results Is possible.
　また、上述の実施の形態では、信号処理装置１を民生用のカメラとしているが、信号処理装置１は、デジタルカメラ等で撮影した画像のデータを図２、図４、図８および図９等に示す処理の一つまたは複数を実行した上で印刷するプリンタ機器としても良い。また、信号処理装置１は、プリンタ機器に対して図２、図４、図８および図９等に示す処理の一つまたは複数を実行させつつ操作するソフトウェアがインストールされたコンピュータ、さらには図２、図４、図８および図９等に示す処理の一つまたは複数を実行するソフトウェアがインストールされたコンピュータ等としても良い。 In the above-described embodiment, the signal processing device 1 is a consumer camera. However, the signal processing device 1 uses image data captured by a digital camera or the like as shown in FIG. 2, FIG. 4, FIG. 8, FIG. A printer device that performs printing after executing one or more of the processes shown in FIG. In addition, the signal processing apparatus 1 is a computer in which software for operating a printer device while performing one or more of the processes shown in FIGS. 2, 4, 8, and 9 is installed. 4, FIG. 8, FIG. 9, or the like may be a computer installed with software that executes one or more of the processes shown in FIG.
　また、上述した各処理方法は、プログラム化されても良い。また、プログラム化されたものが記憶媒体、たとえばＣＤ、ＤＶＤ、ＵＳＢメモリに入れられ、コンピュータによって読みとり可能とされても良い。この場合、信号処理装置１は、その記憶媒体内のプログラム化されたものが信号処理装置１の外部サーバに入れられ、必要によりダウンロードされ、使用されるようにしても良い。この場合、信号処理装置１は、その記憶媒体内のプログラムをダウンロードする通信手段を持つこととなる。 Further, each processing method described above may be programmed. Alternatively, the program may be stored in a storage medium, such as a CD, DVD, or USB memory, and read by a computer. In this case, the signal processing device 1 may be a program stored in the storage medium placed in an external server of the signal processing device 1 and downloaded and used as necessary. In this case, the signal processing apparatus 1 has communication means for downloading the program in the storage medium.
　図２、図４、図８等に示す復元処理方法においては、処理部４で行った処理をソフトウェアで構成しているが、それぞれ、一部の処理を分担して行うようにした部品からなるハードウェアで構成しても良い。また、変化要因情報のデータＧとしては、劣化要因情報のデータのみではなく、単に画像を変化させる情報や、劣化とは逆に、画像を良くする情報を含むものとする。 In the restoration processing method shown in FIG. 2, FIG. 4, FIG. 8, and the like, the processing performed by the processing unit 4 is configured by software, but each of them is composed of parts that perform part of the processing. You may comprise with hardware. Further, the change factor information data G includes not only the deterioration factor information data but also information for simply changing an image and information for improving an image contrary to deterioration.
　また、処理の反復回数（ステップＳ１０９，ステップＳ１２３，ステップＳ３０４）が信号処理装置１側で自動的にまたは固定的に設定されている場合、その設定された回数をＰＳＦの値（Ｐ，Ｐ’）や変化要因情報のデータＧによって変更するようにしても良い。たとえば、ある画素のデータがブレにより多数の画素に分散している場合は、反復回数を多くし、分散が少ない場合は反復回数を少なくするようにしても良い。 Further, when the number of processing iterations (step S109, step S123, step S304) is set automatically or fixedly on the signal processing device 1, the set number of times is set as the PSF value (P, P ′). ) Or data G of change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of iterations may be increased, and when the variance is small, the number of iterations may be decreased.
　さらに、反復処理中に、差分のデータΔ、δが発散してきたり、エネルギーが移動した後の画像データのエネルギーが小さくならず大きくなってきたら、処理を中止させるようにしても良い。発散しているか否かは、たとえば差分のデータΔ、δの平均値を見てその平均値が前回より大きくなったら発散していると判断する方法を採用できる。また、反復処理中に、入力を異常な値に変更しようとしたときには、処理を中止させるようにしても良い。たとえば８ビットの場合、変更されようとする値が２５５を超える値であるときには、処理を中止させる。また、反復処理中、新たなデータである入力を異常な値に変更しようとしたとき、その値を使用せず、正常な値とするようにしても良い。たとえば、８ビットの０～２５５の中で、２５５を超える値を入力データとしようとした際は、マックスの値である２５５として処理するようにする。 Furthermore, the processing may be stopped when the difference data Δ, δ diverges during the iterative processing, or when the energy of the image data after the energy has moved does not decrease but increases. For example, a method of determining whether or not the light is diverging can be determined by looking at the average value of the difference data Δ and δ and determining that the average value is larger than the previous value. In addition, during an iterative process, if an input is to be changed to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed is a value exceeding 255, the processing is stopped. Further, during an iterative process, when an input that is new data is to be changed to an abnormal value, the value may not be used but may be set to a normal value. For example, when a value exceeding 255 in 8 bits of 0 to 255 is set as input data, it is processed as a maximum value of 255.
　また、出力画像となる復元データを生成する際、ＰＳＦの値（Ｐ，Ｐ’）や変化要因情報のデータＧによっては、復元させようとする画像の領域外へ出てしまうようなデータが発生する場合がある。このような場合、領域外へはみ出るデータは反対側へ入れる。また、領域外から入ってくるべきデータがある場合は、そのデータは反対側から持ってくるようにするのが好ましい。 In addition, when generating the restoration data to be the output image, depending on the PSF value (P, P ′) and the data G of the change factor information, data that goes out of the area of the image to be restored is generated. There is a case. In such a case, data that protrudes outside the area is input to the opposite side. Also, if there is data that should come from outside the area, it is preferable to bring that data from the opposite side.
　表１に、上述の各処理を実行した復元画像の評価を示す。撮影されブレなどにより劣化したままの画像（Ｉｍｇ’）を従来例とする。図２に示した処理によって得られたＰＳＦのデータを用い、図８に示した処理によって信号を復元して得た画像を実施例１とする。図２に示した処理から、表１に示す所定のステップを省略して得られたＰＳＦのデータを用い、図８に示した処理から、表１に示す所定のステップを省略して信号を復元して得た画像を実施例２から１２とする。なお、実施例１２は、図８の処理に代えて、画像復元処理にウィナーフィルターを用いた画像である。また、比較例として、図２の処理を行わず、代わりに速度センサによってブレを測定してＰＳＦとしたものの画像についても検討した。 Table 1 shows the evaluation of restored images that have undergone the above-described processes. A conventional example is an image (Img ') that has been photographed and remains degraded due to blurring or the like. An image obtained by restoring the signal by the process shown in FIG. 8 using the PSF data obtained by the process shown in FIG. The PSF data obtained by omitting the predetermined steps shown in Table 1 from the processing shown in FIG. 2 is used, and the signals are restored by omitting the predetermined steps shown in Table 1 from the processing shown in FIG. The images obtained in this way are referred to as Examples 2 to 12. Note that the twelfth embodiment is an image using a Wiener filter for the image restoration process instead of the process of FIG. In addition, as a comparative example, an image of a PSF obtained by measuring blur using a speed sensor instead of performing the process of FIG. 2 was also examined.
Figure JPOXMLDOC01-appb-T000001
Figure JPOXMLDOC01-appb-T000001
　従来例および実施例１から１２ならびに比較例の画像について、「ブレの有無」「リンギングの有無」「ボケ（いわゆるピンボケ）の有無」について目視による検査を行った。その検査結果を表１に併せて示す。検査を行った者は、色弱、色盲ではなく視力が両目とも１．０の者である。「有り」と判断する場合は、一見してブレ等があると判断できたものである。「若干有り」と判断する場合は、５秒以下目視して、他の画像と比較しながらブレ等が有ることが判断できるものである。「無い」と判断する場合は、５秒を超えて目視して、他の画像と比較しながらブレ等が無いことが判断できるものである。画像の種類は、風景および人物の２種類である。画像は、市販のカラープリンターで印刷したもので、印刷条件による画像への影響を排除して印刷したものである。 The images of the conventional example, Examples 1 to 12 and the comparative example were visually inspected for “presence / absence of blurring”, “presence / absence of ringing”, and “presence / absence of blur (so-called out-of-focus)”. The test results are also shown in Table 1. Those who have been examined are those who have 1.0 eyesight in both eyes, not color weakness or color blindness. When it is judged as “present”, it can be judged at first glance that there is a blur or the like. When it is determined that “slightly exists”, it can be determined that there is a blur or the like while visually comparing with other images for 5 seconds or less. When it is determined that there is no image, it is possible to determine that there is no blur or the like while comparing with other images by visually checking for more than 5 seconds. There are two types of images: landscape and person. The image is printed with a commercially available color printer, and is printed while eliminating the influence on the image due to printing conditions.
　従来例は、劣化している画像であるため、ブレおよびボケが明確に確認できた。実施例１は、ブレ、リンギング、ボケが共に確認されなかった。ブレが消えていたのは、ＰＳＦの中枢をなすブレ情報に相当する情報（図３のＡの部分）が妥当なものだったためと考えられる。ボケが消えていたのは、ＰＳＦの中枢をなすブレ情報に相当する情報以外の情報（図３のＢの部分）による作用と考えられる。比較例では、図３のＢの部分のＰＳＦを得ることができないため、この作用が得られておらず、ボケを消すことはできていない。 Since the conventional example is a deteriorated image, blurring and blurring were clearly confirmed. In Example 1, neither blurring, ringing, or blur was observed. The blur disappeared because the information (part A in FIG. 3) corresponding to the blur information forming the center of the PSF was appropriate. It is considered that the blur has disappeared due to information other than information corresponding to the blur information that forms the center of the PSF (part B in FIG. 3). In the comparative example, since the PSF in the portion B in FIG. 3 cannot be obtained, this effect is not obtained and the blur cannot be eliminated.
　実施例２は、図２におけるステップＳ１０３からＳ１０８の過程を１回のみとした画像である。この画像は、従来例の画像よりもブレおよびボケが改善されていた。なお、図２におけるステップＳ１０３からＳ１０８の過程の繰り返し数は、多い程ブレおよびボケがより改善される傾向がみられた。 Example 2 is an image in which the process of steps S103 to S108 in FIG. 2 is performed only once. In this image, blurring and blurring were improved as compared with the conventional image. In addition, there was a tendency that blurring and blurring were more improved as the number of repetitions of steps S103 to S108 in FIG. 2 was larger.
　実施例３は、図２におけるステップＳ１０７、Ｓ１０８、Ｓ１２２を省略して処理した画像である。この画像は、従来例の画像よりもブレおよびボケが改善されていた。 Example 3 is an image processed by omitting steps S107, S108, and S122 in FIG. In this image, blurring and blurring were improved as compared with the conventional image.
　実施例４は、図２におけるステップＳ１０７、Ｓ１０８を省略して処理した画像である。実施例５は、図２におけるステップＳ１２２を省略して処理した画像である。これらの画像は、ブレおよびボケが観測されなかった。 Example 4 is an image processed by omitting steps S107 and S108 in FIG. Example 5 is an image processed by omitting step S122 in FIG. In these images, no blur or blur was observed.
　実施例６は、図２におけるステップＳ１０２、Ｓ１２５、Ｓ１２６を省略して、徐々に画像を拡大処理せずに処理した画像である。実施例７は、図２におけるステップＳ１２３を省略し、繰り返しの回数を１回として処理した画像である。これらの画像は、従来例の画像よりもブレおよびボケが改善されていた。 Example 6 is an image obtained by omitting steps S102, S125, and S126 in FIG. 2 and gradually processing the image without enlarging the image. Example 7 is an image in which step S123 in FIG. 2 is omitted and the number of repetitions is one. These images were improved in blurring and blurring as compared with the conventional image.
　実施例８は、図２におけるステップＳ１２１を省略して処理した画像である。この画像は、ブレが若干観測されたが、ボケは、ブレに起因するものを除き観測されなかった。なお、このボケを消す効果をより得るためには、図２におけるステップＳ１２１の処理を省略または軽減する（たとえば骨格部のデータを１．５倍にする等）。そして、ブレを消す効果をより得るためには、図２におけるステップＳ１２１の処理をより強調する（たとえば骨格部のデータを３倍にする等）。 Example 8 is an image processed by omitting step S121 in FIG. In this image, some blurring was observed, but no blur was observed except for blurring. In order to obtain the effect of eliminating this blur, the process of step S121 in FIG. 2 is omitted or reduced (for example, the data of the skeleton part is multiplied by 1.5). In order to further obtain the effect of eliminating blurring, the processing in step S121 in FIG. 2 is further emphasized (for example, the data of the skeleton part is tripled).
　実施例９は、図８におけるステップＳ３００、Ｓ３０６、Ｓ３０７を省略して処理した画像である。この画像は、ブレおよびボケは観測されなかったが、リンギングが観測された。しかし、図８におけるステップＳ３００、Ｓ３０６、Ｓ３０７を省略した繰り返し処理を極めて多数回（たとえば１００回以上）行うことによって、そのリンギングの発生が軽減されまたは観測されないことが確認された。 Example 9 is an image processed by omitting steps S300, S306, and S307 in FIG. In this image, no blur or blur was observed, but ringing was observed. However, it has been confirmed that the occurrence of ringing is reduced or not observed by performing the repeated processing in which steps S300, S306, and S307 in FIG. 8 are omitted extremely many times (for example, 100 times or more).
　実施例１０は、図８におけるステップＳ３００を省略して処理した画像である。この画像は、ブレおよびボケは観測されず、リンギングも観測されなかった。実施例１１は、図８におけるステップＳ３０６、Ｓ３０７を省略して処理した画像である。この画像は、ブレおよびボケは観測されなかったが、リンギングが若干観測された。すなわち、原点位置を決定する処理（ステップＳ３００）、更新量の妥当性を判断しＩ０＋ｎを修正する処理（ステップＳ３０６、Ｓ３０７）のいずれかを実行することによって、リンギングの発生を抑制できることがわかった。また、ステップＳ３０６、Ｓ３０７によるリンギング抑制効果は、ステップＳ３００の処理によるリンギング抑制効果よりも若干優れていることがわかった。 Example 10 is an image processed by omitting step S300 in FIG. In this image, no blur or blur was observed, and no ringing was observed. Example 11 is an image processed by omitting steps S306 and S307 in FIG. In this image, no blur or blur was observed, but some ringing was observed. That is, it is understood that the occurrence of ringing can be suppressed by executing any one of the process of determining the origin position (step S300) and the process of determining the validity of the update amount and correcting I 0 + n (steps S306 and S307). It was. Moreover, it turned out that the ringing suppression effect by step S306, S307 is a little superior to the ringing suppression effect by the process of step S300.
　実施例１２は、図８における全ての処理を省略し、代わりにウィナーフィルターにて画像の復元処理した画像である。この画像は、従来例にくらべブレおよびボケが改善されていたが、リンギングが観測された。 Example 12 is an image obtained by omitting all the processes in FIG. 8 and performing image restoration processing using a Wiener filter instead. In this image, ringing was observed although blurring and blurring were improved as compared with the conventional example.
　次に、図８におけるステップＳ３０６、Ｓ３０７を省略し、なおかつステップＳ３００の条件を変更した処理について検討した。図１２の変形例となる画素「Ｓ」が０．８５、画素「Ｓ＋１」が０．３、画素「Ｓ＋２」が０．２５の場合、画素「Ｓ」、「Ｓ＋１」、「Ｓ＋２」に分散した光エネルギーを各々集中させ原点位置とし、図８に示す繰り返し処理を行った。それらの各場合の復元画像を目視で観察し、リンギングの有無を判定した。また、式（５）の計算範囲である「Ｎ＝３」を若干拡大し、分散した光エネルギーを図９における画素「Ｓ－１」、「Ｓ＋３」に各々集中させ原点位置とし、図８に示す繰り返し処理を行った場合の、復元画像のリンギングの有無を同様に判定した。表２に判定結果を示した。 Next, processing in which steps S306 and S307 in FIG. 8 were omitted and the conditions in step S300 were changed was examined. In the modified example of FIG. 12, when the pixel “S” is 0.85, the pixel “S + 1” is 0.3, and the pixel “S + 2” is 0.25, the pixel “S”, “S + 1”, and “S + 2” are distributed. Each of the light energy thus concentrated was used as the origin position, and the repetition process shown in FIG. 8 was performed. The restored images in each of these cases were visually observed to determine the presence or absence of ringing. Further, “N = 3”, which is the calculation range of Equation (5), is slightly expanded, and the dispersed light energy is concentrated on the pixels “S−1” and “S + 3” in FIG. The presence or absence of ringing of the restored image when the repetitive processing shown was performed was similarly determined. Table 2 shows the determination results.
　なお、分散した光エネルギーを画素「Ｓ－１」に集中させる場合の移動エネルギーは、画素「Ｓ」、「Ｓ＋１」、「Ｓ＋２」の場合と略同様に以下のように計算され、求められる。
（０×０）＋（１×０．４５）＋（２×０．３）＋（３×０．２５）＋（４×０）＝１．８０
また、同様に、分散した光エネルギーを画素「Ｓ＋３」に集中させる場合の移動エネルギーは、以下のように計算され、求められる。
（４×０）＋（３×０．４５）＋（２×０．３）＋（１×０．２５）＋（０×０）＝２．２０ Note that the movement energy in the case where the dispersed light energy is concentrated on the pixel “S−1” is calculated and obtained as follows in substantially the same manner as in the case of the pixels “S”, “S + 1”, and “S + 2”.
(0 × 0) + (1 × 0.45) + (2 × 0.3) + (3 × 0.25) + (4 × 0) = 1.80
Similarly, the movement energy when the dispersed light energy is concentrated on the pixel “S + 3” is calculated and obtained as follows.
(4 × 0) + (3 × 0.45) + (2 × 0.3) + (1 × 0.25) + (0 × 0) = 2.20
Figure JPOXMLDOC01-appb-T000002
Figure JPOXMLDOC01-appb-T000002
　表２の結果および多数の他の例から、移動エネルギー値が所定値を超えるとリンギングが観測され、移動エネルギー値を所定値以内に抑えるとリンギングの発生を抑えることができることがわかった。すなわち、式（５）から導かれる移動エネルギーの総和の最小値をＭｉｎとしたとき、その移動エネルギーがＭｉｎを超えた場合であってもＭｉｎ×１．２以下となる値であればリンギングの発生を従来に比べかなり抑えることができた。 From the results in Table 2 and many other examples, it was found that ringing was observed when the kinetic energy value exceeded a predetermined value, and generation of ringing could be suppressed by keeping the kinetic energy value within the predetermined value. That is, when the minimum value of the total amount of kinetic energy derived from Equation (5) is Min, even if the kinetic energy exceeds Min, ringing occurs if the value is Min × 1.2 or less. Was able to be suppressed considerably compared with the past.
　表２に示すような、図８におけるＳ３００の条件を変更した処理は、種々変更実施可能である。たとえば、移動エネルギーＥ（０ｘ，０ｙ）の総和を最小とする原点位置（０ｘ，０ｙ）を設定しているが、移動エネルギーＥ（０ｘ，０ｙ）の最小値を超え、所定値以下となる原点位置（０ｘ，０ｙ）を設定しても良い。図１２の変形例である「Ｓ＝０．４５」「Ｓ＋１＝０．３」「Ｓ＋２＝０．２５」の場合、移動エネルギーが最も小さい画素「Ｓ＋１」ではなく、画素「Ｓ」を原点位置（０ｘ，０ｙ）としても良い。このときの各移動エネルギーは、画素「Ｓ」が０．８、画素「Ｓ＋１」が０．７、画素「Ｓ＋２」が１．２となる。画素「Ｓ」は最小値である０．７に１．２を乗じた値である０．８４より小さい値となっている。 As shown in Table 2, the process in which the conditions of S300 in FIG. 8 are changed can be variously changed. For example, the origin position (0x, 0y) that minimizes the total sum of the kinetic energy E (0x, 0y) is set, but the origin that exceeds the minimum value of the kinetic energy E (0x, 0y) and falls below a predetermined value The position (0x, 0y) may be set. In the case of “S = 0.45”, “S + 1 = 0.3”, and “S + 2 = 0.25”, which are the modified examples of FIG. 12, not the pixel “S + 1” having the smallest kinetic energy but the pixel “S” is set to the origin position. (0x, 0y) may be used. The kinetic energy at this time is 0.8 for the pixel “S”, 0.7 for the pixel “S + 1”, and 1.2 for the pixel “S + 2”. The pixel “S” has a value smaller than 0.84, which is a value obtained by multiplying the minimum value 0.7 by 1.2.
　また、本実施例では原点位置を図７におけるＸ－Ｙ平面のいずれかの位置としている。よって、原点位置は、図７のＡ－Ｂの軌跡上の範囲内で決定するものとしても良い。すなわち、たとえば図７におけるＡ－Ｂの軌跡上の範囲内である図１２における画素「Ｓ」「Ｓ＋１」「Ｓ＋２」のいずれかとすること、または画素「Ｓ－１」もしくは「Ｓ＋３」を原点位置とすることが、移動エネルギーの総和を小さくするのであれば、そのようにすることが好ましい。また、図７のＸ－Ｙ平面で表されるブレの軌跡をＸ軸またはＹ軸に投影したものをブレの軌跡とし、その軌跡上で移動エネルギーの最小値や上述の実施例のような値となる位置を求めるようにしても良い。 In the present embodiment, the origin position is any position on the XY plane in FIG. Therefore, the origin position may be determined within a range on the trajectory AB in FIG. That is, for example, any one of the pixels “S”, “S + 1”, and “S + 2” in FIG. 12 that are within the range of the AB trajectory in FIG. 7 or the pixel “S−1” or “S + 3” is the origin position It is preferable to do so if the total amount of kinetic energy is reduced. Further, a blur locus projected on the X-axis or Y-axis with the blur locus represented by the XY plane in FIG. 7 is defined as the blur locus, and the minimum value of the kinetic energy on the locus or the value as in the above-described embodiment. You may make it obtain | require the position which becomes.

Claims (13)

  1. 　劣化等の変化が生じた原信号データの、その信号変化の要因となる変化要因情報のデータを生成する変化要因情報のデータの生成法において、
    　任意の変化要因情報のデータを最初の変化要因情報のデータとして設定し、
    （１）そのデータを利用して、任意の信号データから比較用データを生成して、
    （２）上記原信号のデータまたはその原信号のデータを加工したデータと上記比較用データとを比較し、
    （３）得られた差分のデータを上記最初の変化要因情報のデータを利用して上記任意の信号データに配分することで最初の復元データを生成し、
    （４）その最初の復元データを上記任意の信号データの代わりに用い、必要に応じて、同様の処理を繰り返し、
    （５）その結果得られた、上記復元データと上記原信号のデータまたはその原信号のデータを加工したデータとから上記最初の変化要因情報のデータとは異なる新たな変化要因情報のデータを生成することを特徴とする変化要因情報のデータの生成法。     In the method of generating data of change factor information that generates data of change factor information that is a factor of the signal change of the original signal data in which a change such as deterioration has occurred,
    Set any change factor information data as the first change factor information data,
    (1) Using the data, generate comparison data from arbitrary signal data,
    (2) Compare the original signal data or data obtained by processing the original signal data with the comparison data,
    (3) First restore data is generated by allocating the obtained difference data to the arbitrary signal data using the first change factor information data,
    (4) The first restoration data is used in place of the arbitrary signal data, and if necessary, the same processing is repeated,
    (5) New change factor information data different from the first change factor information data is generated from the restored data and the original signal data or the processed data of the original signal obtained as a result. A method of generating change factor information data characterized by:
  2. 　前記新たな変化要因情報のデータを前記最初の変化要因情報のデータの代わりに用いて前記（１）から（３）と同様の処理を１回以上行うことを特徴とする請求項１記載の変化要因情報のデータの生成法。 2. The change according to claim 1, wherein the new change factor information data is used in place of the first change factor information data, and the same processing as (1) to (3) is performed once or more. Generation method of factor information data.
  3. 　前記（３）の処理は、前記原信号のデータまたはその原信号のデータを加工したデータおよび前記復元データを構成する信号要素のデータの一部または全部を移動させる方法にて行い、前記復元データを構成する信号要素のデータが所定の基準を満たすか否かを判定し、所定の基準を満たさない場合には、前記差分のデータを利用して算出された配分値を修正し、その満たされないデータの代わりに前記修正された配分値によって得られたデータを使用する修正処理を行うこと、を特徴とする請求項１または２記載の変化要因情報のデータの生成法。 The processing of (3) is performed by a method of moving part or all of the original signal data or data obtained by processing the original signal data and signal element data constituting the restored data, and the restored data It is determined whether or not the data of the signal elements constituting the signal satisfies a predetermined criterion, and if the predetermined criterion is not satisfied, the distribution value calculated using the difference data is corrected and is not satisfied 3. The method of generating data of change factor information according to claim 1, wherein a correction process using data obtained by the corrected distribution value instead of data is performed.
  4. 　前記新たな変化要因情報のデータを生成する処理は、得られた前記変化要因情報のデータの原点位置を、各々の前記信号要素のデータの移動に要する移動エネルギーの総和の最小値をＭｉｎとしたとき、その移動エネルギーがＭｉｎ以上で、Ｍｉｎ×１．２以下となる位置に設定することを特徴とする請求項１または２記載の変化要因情報のデータの生成法。 In the process of generating the data of the new change factor information, the origin position of the obtained data of the change factor information is set to Min, and the minimum value of the total sum of the kinetic energy required to move the data of each signal element is set to Min. 3. The method of generating data of change factor information according to claim 1, wherein the movement energy is set to a position where the energy is not less than Min and is not more than Min × 1.2.
  5. 　前記原信号のデータを加工したデータは、前記原信号を小さくする縮小処理を行った縮小データで、前記任意の信号データは上記縮小データと同サイズのデータとすることを特徴とする請求項１から４のいずれか１項に記載の変化要因情報のデータの生成法。 2. The data obtained by processing the original signal data is reduced data obtained by performing a reduction process for reducing the original signal, and the arbitrary signal data is data having the same size as the reduced data. 5. A method of generating data of change factor information according to any one of items 1 to 4.
  6. 　前記縮小処理をしたデータから前記新たな変化要因情報のデータを得た後、前記新たな変化要因情報のデータを大きくする拡大処理を行い、上記拡大処理した前記変化要因情報のデータを前記新たな変化要因情報のデータの代わりに用い、前記（１）から（３）の処理と同様の処理を１回以上行い、必要に応じて上記拡大処理以降の処理を繰り返すことを特徴とする請求項５記載の変化要因情報のデータの生成法。 After obtaining the new change factor information data from the reduced data, an enlargement process for enlarging the new change factor information data is performed, and the enlarged change factor information data is converted into the new change factor information data. 6. The change factor information is used in place of data, and the same processes as the processes (1) to (3) are performed once or more, and the processes after the enlargement process are repeated as necessary. A method of generating the data of the change factor information described.
  7. 　前記縮小処理をしたデータから前記新たな変化要因情報のデータを得た後、前記（５）における前記得られた復元データおよび前記新たな変化要因情報のデータをそれぞれ同じ比率で大きくする拡大処理を行い、上記拡大処理した前記変化要因情報のデータを前記新たな変化要因情報のデータの代わりに用い、上記拡大処理した復元データを前記任意の信号データの代わりに用い、前記（１）から（３）の処理と同様の処理を１回以上行い、必要に応じて上記拡大処理以降の処理を繰り返すことを特徴とする請求項５記載の変化要因情報のデータの生成法。 After obtaining the data of the new change factor information from the data subjected to the reduction process, an enlargement process for increasing the obtained restored data and the data of the new change factor information in (5) at the same ratio, respectively. The expansion factor data is used instead of the new change factor information data, and the expanded restoration data is used instead of the arbitrary signal data. 6. The method of generating data of change factor information according to claim 5, wherein the processing similar to the processing of) is performed once or more, and the processing after the enlargement processing is repeated as necessary.
  8. 　前記新たな変化要因情報のデータを構成する中枢となる部分のデータの値を大きくすることを特徴とする請求項１から７のいずれか１項に記載の変化要因情報のデータの生成法。 The method for generating data of change factor information according to any one of claims 1 to 7, wherein a value of data of a central part constituting the data of the new change factor information is increased.
  9. 　前記最初の変化要因情報のデータは、強度分布がガウス分布のデータであることを特徴とする請求項１から８のいずれか１項に記載の変化要因情報のデータの生成法。 9. The method of generating change factor information data according to claim 1, wherein the first change factor information data is data having a Gaussian intensity distribution.
  10. 　劣化等の変化が生じた原信号データから、変化する前の信号若しくは本来取得されるべきであった信号またはそれらの近似信号（以下、元信号という）の復元をする処理部を有し、
    　上記処理部は、任意の変化要因情報のデータを最初の変化要因情報のデータとして設定し、
    （１）そのデータを利用して、任意の信号データから比較用データを生成して、
    （２）上記原信号のデータまたはその原信号のデータを加工したデータと上記比較用データとを比較し、
    （３）得られた差分のデータを上記最初の変化要因情報のデータを利用して上記任意の信号データに配分することで最初の復元データを生成し、
    （４）その最初の復元データを上記任意の信号データの代わりに用い、必要に応じて、同様の処理を繰り返し、
    （５）その結果得られた、上記復元データと上記原信号のデータまたはその原信号のデータを加工したデータとから上記最初の変化要因情報のデータとは異なる新たな変化要因情報のデータを生成する処理を行い、
    （６）生成された上記新たな変化要因情報のデータまたはその変化要因情報のデータを加工したデータに基づいて上記元信号の復元を行うことを特徴とする信号処理装置。     From the original signal data in which a change such as deterioration has occurred, a processing unit that restores a signal before the change or a signal that should have been originally acquired or an approximate signal thereof (hereinafter referred to as an original signal),
    The processing unit sets arbitrary change factor information data as the first change factor information data,
    (1) Using the data, generate comparison data from arbitrary signal data,
    (2) Compare the original signal data or data obtained by processing the original signal data with the comparison data,
    (3) First restore data is generated by allocating the obtained difference data to the arbitrary signal data using the first change factor information data,
    (4) The first restoration data is used in place of the arbitrary signal data, and if necessary, the same processing is repeated,
    (5) New change factor information data different from the first change factor information data is generated from the restored data and the original signal data or the processed data of the original signal obtained as a result. Process
    (6) A signal processing apparatus that restores the original signal based on the generated data of the new change factor information or data obtained by processing the data of the change factor information.
  11. 　前記元信号の復元をする処理は、前記処理部が前記変化要因情報のデータまたはその変化要因情報のデータを加工したデータを利用して、任意の信号データから比較用データを生成して、処理対象となる前記原信号のデータと比較用のデータとを比較し、得られた差分のデータを前記変化要因情報のデータまたはその変化要因情報のデータを加工したデータを利用して上記任意の信号データに配分することで復元データを生成し、この復元データを上記任意の信号データの代わりに使用し、同様の処理を繰り返す繰り返し処理を行うことを特徴とする請求項１０記載の信号処理装置。 The process of restoring the original signal is performed by generating comparison data from arbitrary signal data using the data of the change factor information or the data of the change factor information processed by the processing unit. The original signal data to be compared and the comparison data are compared, and the obtained difference data is used as the above arbitrary signal using the data of the change factor information or the data of the change factor information. 11. The signal processing apparatus according to claim 10, wherein reconstructed data is generated by allocating the data, the reconstructed data is used in place of the arbitrary signal data, and the same processing is repeated to repeat.
  12. 　前記（３）の処理は、前記原信号のデータまたはその原信号のデータを加工したデータおよび前記復元データを構成する信号要素のデータの一部または全部を移動させる方法にて行い、前記復元データを構成する信号要素のデータが所定の基準を満たすか否かを判定し、所定の基準を満たさない場合には、前記差分のデータを利用して算出された配分値を修正し、その満たされないデータの代わりに前記修正された配分値によって得られたデータを使用する修正処理を行うこと、を特徴とする請求項１０または１１記載の信号処理装置。 The processing of (3) is performed by a method of moving part or all of the original signal data or data obtained by processing the original signal data and signal element data constituting the restored data, and the restored data It is determined whether or not the data of the signal elements constituting the signal satisfies a predetermined criterion, and if the predetermined criterion is not satisfied, the distribution value calculated using the difference data is corrected and is not satisfied The signal processing apparatus according to claim 10 or 11, wherein a correction process using data obtained by the corrected distribution value instead of data is performed.
  13. 　前記新たな変化要因情報のデータを生成する処理は、得られた前記変化要因情報のデータの原点位置を、各々の前記信号要素のデータの移動に要する移動エネルギーの総和の最小値をＭｉｎとしたとき、その移動エネルギーがＭｉｎ以上で、Ｍｉｎ×１．２以下となる位置に設定することを特徴とする請求項１０または１１記載の信号処理装置。 In the process of generating the data of the new change factor information, the origin position of the obtained data of the change factor information is set to Min, and the minimum value of the total sum of the kinetic energy required to move the data of each signal element is set to Min. 12. The signal processing apparatus according to claim 10, wherein the signal is set at a position where the kinetic energy is Min or more and Min × 1.2 or less.
PCT/JP2009/000765 2008-03-04 2009-02-23 Method of creating data relating to change factor information and signal processor WO2009110184A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN200980107860.2A CN101960488B (en) 2008-03-04 2009-02-23 Method of creating data relating to change factor information and image processor

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2008052798A JP5495500B2 (en) 2008-03-04 2008-03-04 Method for generating data of change factor information and signal processing apparatus
JP2008-052798 2008-03-04

Publications (1)

Publication Number Publication Date
WO2009110184A1 true WO2009110184A1 (en) 2009-09-11

Family

ID=41055749

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2009/000765 WO2009110184A1 (en) 2008-03-04 2009-02-23 Method of creating data relating to change factor information and signal processor

Country Status (3)

Country Link
JP (1) JP5495500B2 (en)
CN (1) CN101960488B (en)
WO (1) WO2009110184A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104091341B (en) * 2014-07-18 2018-03-30 厦门美图之家科技有限公司 A kind of image fuzzy detection method based on conspicuousness detection

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1124122A (en) * 1997-07-03 1999-01-29 Ricoh Co Ltd Method and device for correcting camera shake image, and recording medium with recorded program for executing the same method by computer and capable of being read by computer
JPH11272866A (en) * 1998-03-23 1999-10-08 Olympus Optical Co Ltd Image restoring method
JP2000004363A (en) * 1998-06-17 2000-01-07 Olympus Optical Co Ltd Image restoring method
JP2002300459A (en) * 2001-03-30 2002-10-11 Minolta Co Ltd Image restoring device through iteration method, image restoring method and its program, and recording medium
JP2004343483A (en) * 2003-05-16 2004-12-02 Acutelogic Corp Device and method for correcting camera-shake and device for detecting camera shake
JP2005167484A (en) * 2003-12-01 2005-06-23 Japan Science & Technology Agency Image configuration apparatus and image configuration method
JP2007129354A (en) * 2005-11-01 2007-05-24 Nitto Kogaku Kk Image processing apparatus
JP2007179517A (en) * 2005-12-28 2007-07-12 Kao Corp Image generation method and device, and makeup simulation method and device
WO2007077733A1 (en) * 2005-12-28 2007-07-12 Nittoh Kogaku K.K Image processing device
WO2007083621A1 (en) * 2006-01-18 2007-07-26 Nittoh Kogaku K.K Image processing device
JP2008042874A (en) * 2006-07-14 2008-02-21 Eastman Kodak Co Image processing device, method for restoring image and program

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100440933C (en) * 2005-12-31 2008-12-03 北京中星微电子有限公司 Anti-fluttering method and apparatus for hand-held digital camera equipment

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1124122A (en) * 1997-07-03 1999-01-29 Ricoh Co Ltd Method and device for correcting camera shake image, and recording medium with recorded program for executing the same method by computer and capable of being read by computer
JPH11272866A (en) * 1998-03-23 1999-10-08 Olympus Optical Co Ltd Image restoring method
JP2000004363A (en) * 1998-06-17 2000-01-07 Olympus Optical Co Ltd Image restoring method
JP2002300459A (en) * 2001-03-30 2002-10-11 Minolta Co Ltd Image restoring device through iteration method, image restoring method and its program, and recording medium
JP2004343483A (en) * 2003-05-16 2004-12-02 Acutelogic Corp Device and method for correcting camera-shake and device for detecting camera shake
JP2005167484A (en) * 2003-12-01 2005-06-23 Japan Science & Technology Agency Image configuration apparatus and image configuration method
JP2007129354A (en) * 2005-11-01 2007-05-24 Nitto Kogaku Kk Image processing apparatus
JP2007179517A (en) * 2005-12-28 2007-07-12 Kao Corp Image generation method and device, and makeup simulation method and device
WO2007077733A1 (en) * 2005-12-28 2007-07-12 Nittoh Kogaku K.K Image processing device
WO2007083621A1 (en) * 2006-01-18 2007-07-26 Nittoh Kogaku K.K Image processing device
JP2008042874A (en) * 2006-07-14 2008-02-21 Eastman Kodak Co Image processing device, method for restoring image and program

Also Published As

Publication number Publication date
JP5495500B2 (en) 2014-05-21
JP2009211337A (en) 2009-09-17
CN101960488A (en) 2011-01-26
CN101960488B (en) 2013-08-07

Similar Documents

Publication Publication Date Title
EP2574038B1 (en) Image capturing apparatus, image processing apparatus, image processing method, and image processing program
JP5615393B2 (en) Image processing apparatus, imaging apparatus, image processing method, image processing program, and storage medium
JP5007241B2 (en) Image processing device
US8396318B2 (en) Information processing apparatus, information processing method, and program
JP2008176735A (en) Image processing apparatus and method
JP4847633B2 (en) Imaging apparatus and image restoration method
JP2013192224A (en) Method and apparatus for deblurring non-uniform motion blur using multi-frame including blurred image and noise image
JP4856293B2 (en) Imaging apparatus and image restoration method
JP2011128978A (en) Information processor, information processing method and program
JP2011134204A (en) Image processing device, image processing method and program
JP4927005B2 (en) Method for generating data of change factor information and signal processing apparatus
JP4965179B2 (en) Image processing device
JP4885150B2 (en) Image processing device
JP5495500B2 (en) Method for generating data of change factor information and signal processing apparatus
JP5065099B2 (en) Method for generating data of change factor information and signal processing apparatus
CN115496673A (en) Image processing method, image processing apparatus, image processing system, and storage medium
JP2009289079A (en) Image output device
JP4974176B2 (en) Method for generating data of change factor information and signal processing apparatus
JP4606976B2 (en) Image processing device
JP5005319B2 (en) Signal processing apparatus and signal processing method
JP5007245B2 (en) Signal processing device
JP5005553B2 (en) Signal processing device
JP6566780B2 (en) Image processing apparatus, imaging apparatus, image processing method, image processing program, and storage medium
JP2017028583A (en) Image processor, imaging apparatus, image processing method, image processing program, and storage medium
JP2007048276A (en) Image processor

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200980107860.2

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09716914

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 09716914

Country of ref document: EP

Kind code of ref document: A1