JP5005553B2 - Signal processing device - Google Patents

Description

  The present invention relates to a signal processing apparatus.

  Conventionally, it has been known that a signal sometimes deteriorates when it is photographed by a signal processing device such as a camera. Factors that cause signal degradation include camera shake during shooting, various aberrations of the optical system, lens distortion, and the like.

  In the past, the present applicant has proposed a method of restoring an image (signal) that has deteriorated due to camera shake during shooting (see Patent Document 1). In the technology described in Patent Document 1, in a signal processing apparatus having a processing unit that processes a signal, the processing unit uses data of change factor information that causes a signal change, and compares the data from an arbitrary signal. Data is generated, the original signal data to be processed is compared with the comparison data, the restored data is generated using the obtained difference data, and the restored data is used instead of the arbitrary signal data. In this case, the same process is repeated, and the process of generating restored data that approximates the original signal before the change is performed.

  Further, when image restoration processing is performed including signal processing as described in Patent Document 1, ringing is likely to occur in the vicinity of an edge portion existing in an image. This ringing is caused by a frequency component attenuated by image degradation. The following techniques have been proposed to prevent this ringing. In other words, the edge determination unit determines whether an edge exists from the state of the image obtained by restoring the encoded image for each block, and for a block having an edge, the plane approximation unit performs planar approximation on both sides of the edge. Then, an approximate image is created. Then, the approximate image is subjected to DCT (Discrete Cosine Transform) and converted to the frequency domain. Then, this is supplied to the calculation unit and added to the decoded image in the frequency domain at a predetermined internal division ratio. That is, the high frequency region lost by the quantization process is added in the arithmetic unit. Since the lack of this high-frequency region is the cause of ringing, ringing in the reproduced image can be prevented by processing in the computing unit (see Patent Document 2).

  In addition to general captured images, it is known that various images and signals such as X-ray photographs and microscopic images are degraded or changed due to blurring or other causes. In addition, a phenomenon similar to ringing in an image also occurs in a general signal.

JP 2007-48257 A JP-A-6-70173 (see abstract)

  In the signal processing apparatus described in Patent Document 1, it is known that a signal such as an image in which ringing does not occur can be restored if the iterative process is performed extremely many times. However, in this method, it takes too much time to eliminate ringing, that is, to obtain a signal having a good restoration state. Further, in the ringing removal technique described in Patent Document 2, the approximate image is converted into the frequency domain, and this is supplied to the calculation unit, and is added to the decoded image in the frequency domain at a predetermined internal ratio. If the high frequency that was present in the image becomes zero in the approximate image, it cannot be restored after all. That is, if the frequency domain of the approximate image includes zero or a band close to zero, even if it is added at a predetermined internal ratio, the portion of the band near zero or near zero is also determined by the predetermined internal ratio. The high frequency that was present in the original image cannot be recovered if the amount of change is zero or close to zero and is not substantially added. Therefore, even if the ringing removal technique described in Patent Document 2 is adopted, it is insufficient for suppressing the occurrence of ringing.

  Accordingly, an object of the present invention is to provide a signal processing apparatus that can efficiently suppress the occurrence of ringing or the like when restoring a signal.

  In order to solve the above-described problem, the signal processing apparatus of the present invention, from the original signal data that has undergone changes such as degradation, the signal before the change, the signal that should have been originally acquired, or an approximate signal thereof (hereinafter, A processing unit for restoring the original signal), the processing unit generates comparison data from arbitrary signal data using the data of the change factor information that causes the signal change, and The original signal data and the comparison data are compared, and the obtained difference data is distributed to arbitrary signal data using the data of the change factor information to generate restored data. It is used in place of arbitrary signal data and repeats the same process repeatedly, and the processing unit determines whether or not the signal element data constituting the restored data satisfies a predetermined standard, and the predetermined standard is used. Filled If have, correct the distribution value calculated by using the data of the difference, performs a correction process that uses the data obtained by the modified allocation values in place of the not satisfied data.

  According to the present invention, the restored data can satisfy a predetermined standard. Adopting a standard that keeps the predetermined standard away from surrounding pixels can effectively suppress the occurrence of ringing. Further, by correcting the data of the signal elements constituting the restoration data, it is possible to appropriately distribute the image edge portions and the like. Therefore, occurrence of ringing or the like can be efficiently suppressed. Here, the pixel value refers to an arbitrary numerical value relating to an image or a pixel such as a color number and a luminance value.

  In the signal processing device according to another invention, in addition to the above-described invention, the predetermined reference is generated from the data of the signal element to be determined and the data of the signal elements around it. By adopting this configuration, an appropriate determination can be made in comparison with surrounding pixels, and ringing can be more effectively suppressed.

  In a signal processing device according to another invention, in addition to the above-described invention, the peripheral data is a set of a plurality of signal elements included in a range in which one signal element is influenced by the change factor information. By adopting this configuration, the value of the signal element in the range affected by the change can be individually determined and examined, so that it can be corrected to a more appropriate value.

  In the signal processing device according to another invention, in addition to the above-described invention, the case where the predetermined criterion is not satisfied means that the data of the signal element constituting the restored data is outside the predetermined range. By adopting this configuration, the restored data obtained can be leveled and ringing can be efficiently suppressed.

  In the signal processing device according to another invention, in addition to the above-described invention, the correction process is a process of reducing the absolute value of the distribution value when the predetermined standard is not satisfied. By adopting this configuration, even when the value to be distributed may take both positive and negative values, such as when the signal data is pixel data, the signal is used to reduce the absolute value of the distribution value. Appropriate correction processing can be performed so that the value of the element does not change abruptly.

  In the signal processing device according to another invention, in addition to the above-described invention, the processing unit stops the repetition processing when the difference data becomes equal to or smaller than the predetermined value or smaller than the predetermined value, or when the number of repetitions reaches the predetermined number. To perform the process. By adopting this configuration, the process is stopped regardless of whether the difference becomes “0”, so that it is possible to prevent the processing from taking a long time. Further, when the processing is continued up to a predetermined number of times, the restored data to be approximated is closer to the original signal before the original signal is deteriorated. Furthermore, when there is noise or the like, a situation in which the difference does not become “0” tends to occur in reality, but even in such a case, the process can be ended in a predetermined number of times, so that the processing can be performed infinitely. It will not be repeated.

  In the signal processing device according to another invention, in addition to the above-described invention, the processing unit may perform the repetition processing when the number of repetitions reaches a predetermined number and the difference data is equal to or less than the predetermined value or smaller than the predetermined value. If it stops and exceeds the predetermined value or exceeds the predetermined value, a process is repeated for a predetermined number of times. By adopting this configuration, iterative processing is performed in combination with the number of processing times and difference data, so when simply limiting the number of processing times or using difference data as a reference In comparison, it is possible to achieve a process that balances good signal restoration accuracy and short processing time.

  In the present invention, it is possible to provide a signal processing apparatus capable of efficiently suppressing the occurrence of ringing and the like.

  Hereinafter, a signal processing device according to an embodiment of the present invention will be described with reference to the drawings. Although this signal processing device is a consumer camera, it may be a camera for other uses such as a surveillance camera, a television camera, a handy type video camera, an endoscopic camera, a microscope, a binocular, The present invention can also be applied to devices other than cameras, such as diagnostic imaging devices such as NMR imaging, and devices that form part of the image adjustment function of a printer that prints images.

  FIG. 1 shows an outline of the configuration of the signal processing apparatus 1. The signal processing device 1 includes an imaging unit 2 that captures an image of a person, a control system unit 3 that drives the imaging unit 2, and a processing unit 4 that processes an image captured by the imaging unit 2. ing. The signal processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4 and an angular velocity sensor, and detects change factor information that causes a change such as image degradation. And a factor information storage unit 7 for storing known change factor information that causes image degradation and the like.

  The photographing unit 2 includes a photographing optical system having a lens and a photographing element such as a CCD or C-MOS that converts light passing through the lens into an electric signal. The control system unit 3 controls each unit in the signal processing device, such as the photographing unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 generates a sampling frequency for detecting vibrations such as camera shake to be detected and supplies the sampling frequency to the detection unit 6. The processing unit 4 controls the start and end of vibration detection.

  Further, the processing unit 4 may store image data that is a source for generating comparison data, which will be described later. Further, the processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 is composed of a semiconductor memory, but magnetic recording means such as a hard disk drive or optical recording means using a DVD or the like may be employed.

  As shown in FIG. 2, the detection unit 6 includes two angular velocity sensors that detect the speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the signal processing device 1. is there. By the way, camera shake at the time of photographing with the camera also moves in each direction of the X direction, Y direction, and Z direction, and rotates about the Z axis. Rotation around the X axis. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, only two angular velocity sensors around the X axis and the Y axis in FIG. 2 are arranged. However, for the sake of completeness, an angular velocity sensor around the Z axis may be further added, or a sensor for detecting movement in the X direction or the Y direction may be added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor.

  The factor information storage unit 7 is a recording unit that stores change factor information such as known deterioration factor information, such as a transfer function calculated based on aberrations of the optical system and / or detected vibrations. The transfer function recorded in the factor information storage unit 7 is, for example, the original image (the image before the change or the image before the change or the image originally taken) from the original image (the image in which the deterioration or the like has been changed) taken immediately after the calculation. Are used by the processing unit 4 in the restoration process. Here, when the restoration process of the original image to the original image is performed, when the imaging power is turned off, when the processing unit 4 is not operating, when the operating rate of the processing unit 4 is low, etc. It can be a time delayed from the time when the original image was taken. In that case, the original image data stored in the recording unit 5 and the change factor information such as a transfer function for the original image stored in the factor information storage unit 7 are associated with each other for a long time. It is preserved over. As described above, the advantage of delaying the time for executing the restoration process of the original image from the time of shooting the original image is that the burden on the processing unit 4 at the time of shooting involving various processes can be reduced.

  Next, an outline of the image restoration processing method of the processing unit 4 of the signal processing apparatus 1 according to the present embodiment configured as described above will be described with reference to FIGS. 3, 4, and 5.

In FIG. 3, “I ₀ ” is an arbitrary initial image and is image data stored in advance in the recording unit of the processing unit 4. “I ₀ ′” indicates data of a degraded image of I ₀ of the initial image data, and is comparison data for comparison. “G” is data of change factor information (= deterioration factor information (transfer function)) detected by the detection unit 6 and is stored in the recording unit of the processing unit 4. “Img ′” is data of the original image.

“Δ” is difference data between the original image data Img ′ and the comparison data I ₀ ′. “K” is an allocation ratio based on the data of the change factor information. “I _{0 + n} ” is restored image data (restored data) newly generated by allocating the difference data δ to the initial image data I ₀ based on the data G of the change factor information. As a result of this “distribution”, the data I _{0 of the} initial image is “updated” to restored data. The “distributed value” at this time may be expressed as “update amount”. “Img” is data of the original image. Here, it is assumed that the relationship between Img and Img ′ is expressed by the following equation (1).
Img ′ = Img * G (1)
Here, “*” is an operator representing a superposition integral.

The difference data δ may be a simple difference between corresponding pixels, but in general, the difference data δ varies depending on the data G of the change factor information, and is expressed by the following equation (2).
δ = f (Img ′, Img, G) (2)

Processing routine of the processing unit 4 first begins to prepare any image data _{I 0} (step S101). As the initial image data I ₀ , the deteriorated original image data Img ′ may be used, and any image data such as black solid, white solid, gray solid, or checkered pattern may be used. good. In step S102, data I ₀ of an arbitrary image that is an initial image is inserted instead of Img in equation (1), and comparison data I ₀ ′ that is a degraded image is obtained. Next, the original image data Img ′ and the comparison data I ₀ ′ are compared to calculate difference data δ (step S103).

Then, it is determined whether or not each absolute value of the difference data δ is less than a predetermined value (step S104). The determination performed in step S104 is performed on the restored image itself, and the difference data δ is a value that is the highest or average value of the absolute values of the individual differences derived from the individual signal elements. If the difference data δ is greater than or equal to a predetermined value in step S104, a process of generating new restored image data (= restored data) is performed in step S105. That is, the individual difference data δ from which individual signal elements are obtained is distributed to arbitrary image data I ₀ based on the data G of the change factor information, and new restored data I _{0 + n} is generated.

Then, the validity of the update amount (distribution value) used when generating the restored data I _{0 + n} is determined (step S106), and the restored data I _{0 + n} is corrected (step S107). The process of steps S106 and S107 will be described with reference to FIGS. The basic concept of this processing is as follows. This process is not performed on the duplicate data I _{0 + n} at once, but is performed for each signal element constituting the duplicate data I _{0 + n} . This point is significantly different from the determination of the entire image in step S104. First, among the plurality of pixels in the range in which one pixel serving as one signal element is affected by the change factor information G, if there is a pixel whose change in pixel value increases due to the update, the range of the plurality of pixels is included. I think there is an edge. For the portion serving as the edge, the update amount based on the difference data δ of the corresponding portion is unlikely to be an appropriate amount. This is because the difference between the pixel values of two pixels straddling the edge part is too large, so even if the pixel value is distributed from one pixel to the other pixel across the edge part, the distribution is appropriate. This is because it is difficult to. As a result of such inappropriate allocation, ringing is likely to occur like a wave image in an area where there is originally no change near the edge. Therefore, for pixels whose change in pixel value is unnaturally larger than the surroundings due to the update, the update amount in the vicinity of the edge is brought close to an appropriate value by reducing the absolute value of the update amount.

  For this purpose, first, an update amount (Dc) of a certain pixel is calculated from predetermined difference data δn (step S201). Then, reference is made to a set of a plurality of pixels (a part of signal elements) in the range in which one pixel is affected by the change factor information G and the one pixel, and the pixel value before the update of each pixel to be referred to A minimum value (= Min), a maximum value (= Max), and an average value (= Av) of (= Ib) are calculated (step S202). The partial pixels (reference pixels) may be simply a plurality of adjacent pixels or a plurality of pixels included in a predetermined distance from a predetermined pixel as a center. The predetermined distance is determined using data G of change factor information, for example.

Next, data (= Ia) that is the updated pixel value of one pixel (= one signal element) among the referenced pixels is calculated (step S203). The updated pixel value Ia is a pixel value for each pixel when updated using the update amount (Dc) for each pixel calculated from the difference data δ corresponding to a certain pixel. ) Guided by the formula.
Ia = Ib + Dc (3)

  Next, the validity of the update amount (Dc) before correction is determined. First, when the updated pixel value Ia, which is signal element data constituting the duplicate data, is not less than the minimum value Min and not more than the maximum value Max (determination in step S204 is “Y” (= Yes)). The pixel has a natural balance of pixel values with surrounding pixels, and the update amount Dc is not corrected, but is used as the update amount (Dp) after correction to obtain the correction value Ia ′ (steps S205 and S214). ). Step S214 is a step for obtaining a correction value Ia 'by the update amount Dp after correction. The arrow shown in FIG. 5A schematically shows the state after the processing of step S205 and step S214, the black circle of the arrow indicates the pixel value Ib before the pixel update, and the triangular arrow indicates Dp Indicates the corrected value Ia ′ after updating, and the length of the arrow indicates the update amount (Dc = Dp). The left arrow indicates an example when the value of the update amount Dc before correction is a positive value, and the right arrow indicates an example when the value of the update amount Dc before correction is a negative value.

  When the determination in step S204 is “N” (= No), the process proceeds to step S206. Here, when the updated pixel value Ia exceeds the maximum value Max and the updated pixel value Ib is equal to or less than the average value Av (determination in step S206 is “Y” (= Yes)), it is determined that the pixel value balance with the surrounding pixels is unnatural. Then, the update amount Dc is changed to a value Dp obtained by subtracting the pixel value Ib before update from the maximum value Max (step S207). As a result, the pixel value of the correction value Ia ′ becomes equal to the maximum value Max. The arrow shown in FIG. 5B schematically shows the state after step S207 and step S214, as in FIG. The left arrow indicates Ia when updated by Dc, and the right arrow indicates a correction value Ia 'when updated by Dp.

When the determination in step S206 is “N” (= No), the process proceeds to step S208. Here, when the updated pixel value Ia exceeds the maximum value Max and the updated pixel value Ib exceeds the average value Av (the determination in step S208 is “Y” (= Yes). )), It is determined that the pixel value balance with the surrounding pixels is unnatural. In this case, the update amount Dc is changed to a value Dp obtained from the following equation (4) (step S209). As a result, the pixel value of the correction value Ia ′ is a value obtained by adding ¼ of the updated pixel value Ia exceeding the maximum value Max to the plus side to the maximum value Max. The arrow shown in FIG. 5C schematically shows the state after step S209 and step S214, as in FIG. 5B.
Dp = 0.25 (Ia−Max) + (Max−Ib) (4)

When the determination in step S208 is “N” (= No), the process proceeds to step S210. Here, when the pixel value Ia after update is less than the minimum value Min and the pixel value Ib before update is equal to or less than the average value Av (the determination in step S210 is “Y” (= Yes)), the pixel Determines that the balance of pixel values with surrounding pixels is unnatural. Then, the update amount Dc is changed to a value Dp obtained from the following equation (5) (step S211). As a result, the pixel value of the correction value Ia ′ is a value obtained by subtracting ¼ of the updated pixel value Ia exceeding the minimum value Min to the minus side from the minimum value Min. The arrow shown in FIG. 5D schematically shows the state after step S211 and step S214, as in FIG. 5B.
Dp = − (Ib−Min) −0.25 (Min−Ia) (5)

  If the determination in step S210 is “N” (= No), the updated pixel value Ia is less than the minimum value Min, and the updated pixel value Ib exceeds the average value Av ( Step S212). In that case, the pixel value balance with the surrounding pixels is unnatural. Then, the update amount Dc is changed to a value Dp obtained by subtracting the pixel value Ib before update from the minimum value Min (step S213). As a result, the pixel value of the correction value Ia ′ becomes equal to the minimum value Min. The arrow shown in (E) of FIG. 5 schematically represents the state after step S213 and step S214, as in FIG. 5 (B).

  The determination processing shown in the above steps S204, S206, S208, and S210 is processing for determining whether or not a predetermined standard is satisfied. It should be noted that the order of determination based on the determination in steps S204, S206, S208, and S210 and whether or not the condition in step S212 is satisfied can be changed as appropriate. Even when the change is made, the first to fourth judgment is actually made. If none of the four conditions is met, the remaining conditions will always be met. Therefore, whether the remaining conditions are met is determined by whether or not the predetermined criteria are satisfied as in step S212 in FIG. It is not a judgment process.

In this way, the obtained Dp is used as an updated amount, and the corrected value Ia ′ is obtained (step S214). As a result, a certain pixel is updated (step S215). Then, it is determined whether or not all the pixels have been updated (step S216). If the determination is “N” (= No), the pixel to be referred to is changed (step S217), and a predetermined standard is obtained from the changed reference pixel (other signal elements) and In order to obtain the correction value Ia ′ of the pixel, the process returns to step S202, and the processes of steps S202 to 217 are repeated. In this process, when the update amount of one pixel is corrected, the reference pixel is changed to change the update amount of another pixel. When the determination in step S216 is “Y” (= Yes), the restoration data I _{0 + n} is corrected using the correction value Ia ′ of all pixels (step S218). Thus, steps S106 and 107 in FIG. 3 are completed.

Thereafter, steps S102 to S107 in FIG. 3 are repeated. The restored data I _{0 + n in} the middle of this repetition is restored data in the middle of processing. In step S104, when each absolute value of the difference data δ of each pixel becomes less than a predetermined value, the iterative process is terminated. Then, the restored data I _{0 + n} at the time when the repetitive processing is completed is estimated as the original image data Img. That is, when the maximum value or average value of the absolute values of the difference data δ of each pixel becomes smaller than a predetermined value, the restored data I _{0 + n that} is the basis of the comparison data I _{0 + n} ′ is the original image data Img. Therefore, the restored data I _{0 + n} is estimated as the original image data Img. The initial image data I ₀ and the change factor information data G may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary.

  The concept of the above-described iterative processing method is summarized as follows. That is, in this processing method, the processing solution is not solved as an inverse problem, but is solved as an optimization problem for obtaining a rational solution. When solving as an inverse problem, it is theoretically possible, but it is difficult as a real problem.

In the case of solving as an optimization problem, the present embodiment assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the output is the same, the input is the same.
(3) The input is updated so that the outputs are the same, and the solution is converged by performing iterative processing while correcting the update amount to an appropriate value.

In other words, as shown in FIGS. 6A and 6B, if comparison data I ₀ ′ (I _{0 + n} ′) that is approximate to the original image data Img ′ can be generated, the original data of the generation is generated. The initial image data I ₀ or the restored data I _{0 + n} is approximate to the original image data Img.

  In this embodiment, the sampling rate of the angular velocity detection sensor is set within 60 Hz to 240 Hz. However, the angular velocity may be detected every 5 μsec so that a high frequency can be detected. In addition, in this embodiment, the value serving as a determination criterion for the difference data δ is “6” when each data is represented by 8 bits (0 to 255). That is, when it is less than 6, that is, 5 or less, the processing is finished. Further, the shake raw data detected by the angular velocity detection sensor does not correspond to the actual shake when the sensor itself is insufficiently calibrated. Therefore, in order to deal with actual blurring, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

  Next, details of the camera shake restoration processing method shown in FIG. 3 (repetitive processing of steps S102, S103, S104, S105, S106, and S107) will be described in detail with reference to FIGS. 7, 8, 9, 10, 11, and 12. , Based on FIG. 13 and FIG.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how to disperse the energy during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. Let the pixels be S-1, S, S + 1, S + 2, S + 3,... In order from the left, and pay attention to a certain pixel S. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 8 is the correct image data Img when no deterioration occurs. Each data is represented by 8-bit (0 to 255) data.

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the Sth pixel, 30% of time is shifted to the S + 1th pixel, and 20% of time is shifted to the S + 2th pixel. The way of energy dispersion is as shown in the table of FIG. This becomes the data G of the change factor information.

  The blur is uniform for all pixels and is recognized as a linear problem. If there is no upper blur (vertical blur), the blur situation is as shown in the table of FIG. The data shown as “blurred image” in FIG. 10 becomes the data Img ′ of the degraded original image. Specifically, for example, “120” of the pixel “S-3” is in accordance with the distribution ratio of “0.5”, “0.3”, “0.2” of the data G of the change factor information that is the blur information, Dispersed in such a manner that “60” is distributed to the “S-3” pixel, “36” is distributed to the “S-2” pixel, and “24” is distributed to the “S-1” pixel. Similarly, “60” which is the pixel data of “S-2” is distributed as “30” in “S-2”, “18” in “S-1”, and “12” in “S”. The original image data Img is calculated from the deteriorated original image data Img 'and the change factor information data G shown in FIG.

The arbitrary image data I ₀ shown in step S101, can be adopted also What, When this description, using the original image data Img '. That is, the process starts with I ₀ = Img ′. In the table of FIG. 11, “input” corresponds to the initial image data I ₀ . This data I _0, that is, Img ′ is superposed and integrated with the data G of the change factor information in step S102. That is, for example, “60” of the “S-3” pixel of the initial image data I ₀ is “30” for the S-3 pixel, “18” for the “S-2” pixel, “12” is assigned to each pixel of “−1”. The other pixels are similarly distributed, and comparison data I ₀ ′ shown as “output I ₀ ′” is generated. Therefore, the difference data δ in step S103 is as shown in the bottom column of FIG. It is determined whether the maximum absolute value of the difference data δ is a predetermined value, for example, less than 10 (step S104). In this example, the difference data δ of the pixel “S-3” is 30, and step S104 is performed. No (= N) and the process proceeds to step S105.

As shown in FIG. 12, the distribution of the difference data δ is 0.5, which is the distribution ratio of the pixel data “30” of “S-3” to the pixel data “30” (= the pixel of “S-3”). “15” multiplied by “S-3” is distributed to the pixel of “S-3”, and the data “15” of the pixel of “S-2” is allocated to the pixel of “S-2”. “4.5” multiplied by 0.3 is allocated, and further, the data “9.2” of the pixel “S-1” is allocated to the pixel “S-1” by the distribution ratio. “1.84” multiplied by a certain 0.2 is allocated. The total amount (update amount Dc for each pixel) allocated to the pixels of “S-3” is “21.34”, and this value is the data I ₀ of the initial image in FIG. Here, the updated pixel value Ia in FIG. 4, which is the restored data I _{0 + 1} in FIG. 3, is calculated in addition to the original image data Img ′. In this example, as shown in FIG. 12, the updated pixel value Ia is “81.34”. In this way, the difference data δ is distributed to the arbitrary image data I ₀ using the change factor information data G to generate the restored data I _{0 + n} shown as “next input” in FIG. . In this case, since this is the first time, it is represented as I _{0 + 1 in} FIG.

Thereafter, the validity of the update amount is determined in step S106 in FIG. Specifically, in order to correct the restored data I _{0 + 1} , an updated pixel value (= Ia) for each pixel is calculated. This calculation is based on equation (3) as described above. Therefore, the difference data δ is distributed to each pixel. Then, as shown in FIG. 12, for each pixel, the minimum value (= Min), the maximum value (= Max), and the average value (= Av) of the pixel value (= Ib) before the update of each pixel to be referred to Is calculated (step S202 in FIG. 4). For example, the pixel “S-3” refers to the pixel “S-3”, the pixel “S-2”, and the pixel “S-1”. Therefore, as shown in FIG. 12, the minimum value (= Min), the maximum value (= Max), and the average value (= Av) of the pixel “S-3”, the pixel “S-2”, and the pixel “S-1”. Is calculated. In the example shown in the figure, regarding the pixel “S-3”, the maximum value is “82.00” in the pixel “S-1”, the minimum value is “66.00” in the pixel “S-3”, and the average The value is “69.33”, which is a value obtained by dividing the sum of the values of the pixels “S-3”, “S-2”, and “S-1” by 3. The same calculation is performed for the pixels “S−2” to “S + 4”.

Then, it is determined whether Ia and Ib satisfy any of the conditions in steps S204, S206, S208, S210, and S212 in FIG. For example, Ia of the pixel “S-3” satisfies the condition of “Min (60.00) ≦ Ia (81.34) ≦ Max (82.00)”, and therefore satisfies the condition of step S204 in FIG. . Therefore, as shown in FIG. 12, the process of step S205 is performed, and the update amount “21.34” as Dc is used as it is as the update amount Dp after correction, and the correction value Ia ′ is equal to “81 before correction” “81 .34 ". The same correction is performed for the updated pixel values Ia of “S−2” to “S”, “S + 2”, and “S + 3”. This correction value Ia ′ becomes the correction value of the restored data I _{0 + n} .

For example, in the pixel “S + 1”, the conditions of steps S204 and S206 are not satisfied, and the process proceeds to step S208. In order to satisfy “Ib (121.00)> Av (113.33)” and satisfy the condition “Ia (130.11)> Max (121.00)”, the condition of step S208 in FIG. You will be satisfied. Therefore, the process of step S209 is performed as shown in FIG. 12, and “2.28” is used as the updated amount Dp after correction, and the correction value Ia ′ is “123.28”. The same correction is performed for Ia of the pixel “S + 4”. This Ia ′ becomes a correction value of the restored data I _{0 + n} .

As shown in FIG. 13, the corrected restored data I _{0 + 1} (Ia ′) becomes the new input image data (= replaces the initial image data I ₀ ) in step S102, and step S102 is executed. The process proceeds to step S103 to obtain new difference data δ. The size of the difference data δ is determined in step S104. If the difference data δ is larger than the predetermined value, the new difference data δ is allocated to the previously modified restored data I _{0 + 1} in step S105 to generate new restored data I _{0 + 2} . In this case, the new restoration data I _{0 + 2} is corrected in the same manner as described in FIG. 12 (see FIG. 14). For example, the updated pixel value Ia of the pixels “S-3”, “S”, and “S + 3” is corrected in the same manner as the pixel “S-3” in FIG. That is, the updated pixel value Ia becomes the corrected value Ia ′ as it is.

For example, in the pixel “S + 1”, the conditions in steps S204 and S206 are not satisfied, and the process proceeds to step S208. Then, “Ib (121.00)> Av (116.15)” and the condition “Ia (125.43)> Max (121.00)” is satisfied, and therefore the condition of step S208 in FIG. 4 is satisfied. Will be. Therefore, the process of step S209 is performed as shown in FIG. 14, and the correction value Ia ′ is “122.11” using the value “1.16” according to the above-described equation (4) as the update amount Dp after correction. . The same correction is performed on the updated pixel value Ia of the pixel “S + 4”. This correction value Ia ′ becomes the correction value of the restored data I _{0 + n} .

For example, in the pixel “S-2”, the conditions of steps S204, S206, and S208 are not satisfied, and the process proceeds to step S210. Then, “Ib (77.30) ≦ Av (87.67)” is satisfied, and the condition “Ia (76.97) <Min (77.30)” is satisfied, so the condition of step S210 in FIG. 4 is satisfied. Will be. Therefore, the process of step S211 is performed as shown in FIG. 14, and the value “−0.082” according to the above equation (5) is used as the update amount Dp after correction, and the correction value Ia ′ is “77.22”. Become. The same correction is performed on the updated pixel value Ia of the pixels “S−1” and “S + 2”. This correction value Ia ′ becomes the correction value of the restored data I _{0 + n} .

Thereafter, by performing the step S102 by using the restored data I _{0 + 2} that are fixed, a new comparison data I _{0 + 2} ^'from the restored data I _{0 + 2} that are fixed are generated. As described above, after steps S102 and S103 are executed, the process proceeds to step S104, and the process proceeds to step S105 based on the determination. Such a process is repeated.

  As described above, by repeating step S102 to step S107, the difference data δ gradually decreases. When the difference data δ becomes smaller than a predetermined value, the original image data Img that is not blurred is obtained. At this time, since correction processing (FIGS. 4 and 5) is performed, the occurrence of ringing is reduced in the image data estimated as the obtained original image data Img, and the restored state of the image is good. In addition, the correction process (FIGS. 4 and 5) corrects unnatural data among the signal element data constituting the restored data and suppresses large changes in pixel values. Even if the information data G has low reliability, it is possible to restore an appropriate image.

  In the camera shake restoration processing method shown in FIG. 3 described above (repetitive processing of steps S102, S103, S104, S105, S106, and S107), the processing performed by the processing unit 4 is configured by software. You may comprise with the hardware which consists of components which respectively performed a part of processing divided. Further, the change factor information data G includes not only the deterioration factor information data but also information for simply changing an image and information for improving an image contrary to deterioration.

  Further, when the number of processing iterations is set automatically or fixedly on the signal processing device 1 side, the set number of times may be changed by the data G of the change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of repetitions may be increased, and when the dispersion is small, the number of repetitions may be decreased.

  Furthermore, during the iterative process, if the difference data δ diverges or the energy of the image data after the energy has moved does not decrease but increases, the process may be stopped. For example, it is possible to adopt a method of determining whether or not the light is diverging by observing the average value of the difference data δ and determining that the light is diverging if the average value is larger than the previous value. In addition, during an iterative process, if an input is to be changed to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed is a value exceeding 255, the processing is stopped. Further, during an iterative process, when an input that is new data is to be changed to an abnormal value, the value may not be used but may be set to a normal value. For example, when a value exceeding 255 within the 8-bit range of 0 to 255 is used as input data, it is processed as a maximum value of 255.

  In addition, when generating restoration data to be an output image, depending on the data G of the change factor information, there may be data that goes out of the area of the image to be restored. In such a case, data that protrudes outside the area is input to the opposite side. Also, if there is data that should come from outside the area, it is preferable to bring that data from the opposite side. For example, when data allocated to a lower pixel is generated from the data of the pixel XN1 positioned at the bottom in the area, the position is outside the area. Therefore, the data is processed to be allocated to the pixel X11 located at the top right above the pixel XN1. Similarly, the pixel N2 adjacent to the pixel XN1 is assigned to the topmost pixel X12 (= next to the pixel X11) directly above.

The signal processing apparatus 1 according to the present embodiment has been described above, but various modifications can be made without departing from the gist of the present invention. For example, the method of correcting the restored data I _{0 + n} is not limited to the method shown in FIGS. Particularly, the method of correcting the difference data δ in steps S205, S207, S209, S211 and S213 can be changed as appropriate according to the case in steps S204, S206, S208, S210 and S212 shown in FIG.

  For example, as a pixel to be referred to, in addition to the range of influence (influence range), a range that is larger by one pixel surrounding the range of influence, or the range of the distance a centered on the pixel to be corrected It is also good. In addition, as a predetermined standard, the maximum value, minimum value, and average value of the reference pixel are not used, but only the maximum value and the minimum value are used, and the correction value exceeds the maximum / minimum value. You may make it add the value of 1/4 or 1/3 to the maximum value or the minimum value. Further, the upper limit is set to 1.2 times the maximum value and the lower limit is set to 0.8 times the minimum value. That is, if the updated pixel value Ia is in the range of X times the maximum value and Y times the minimum value, the pixel value Ia may not be corrected.

  In the iterative process according to the present embodiment, the processing unit 4 determines the absolute value of the difference data δ for each of a plurality of pixels constituting the image when determining whether to process the image once obtained in step S104 in FIG. It is determined whether or not all the values are less than a predetermined value or the average value of the absolute values is less than the predetermined value, and whether or not the entire image is to be processed. However, the comparison target with the predetermined value may be the difference data for each of the plurality of pixels constituting the image, and it may be determined whether or not the repeated processing is stopped for each pixel. The comparison target with the predetermined value may be the sum of the difference data δ of each pixel, the sum of the absolute values of the difference data δ of each pixel, or two or more of the above four. For example, it is determined whether the value farthest from zero in the difference data δ for each pixel and the sum of the difference data δ for each pixel satisfy different criteria. May be. As described above, by appropriately selecting a value to be compared with the predetermined value, appropriate processing can be performed according to the type of original image, the state of change, or the status of restoration processing.

  In the above-described embodiment, the restoration target is image data. However, these restoration processing concepts and techniques can be applied to any digital data restoration processing. For example, it can be applied to restoration of digital audio data. As a result of this application, it is possible to efficiently suppress the occurrence of some inaccurate audio data, such as ringing, and even if the data of the change factor information is inaccurate, a restoration process that can obtain reasonable results Is possible.

  In the above-described embodiment, the signal processing device 1 is a consumer camera. However, the signal processing device 1 executes the processing shown in FIGS. 3 and 4 on the data of an image taken by a digital camera or the like. It is also possible to use a printer device for printing with Further, the signal processing apparatus 1 is installed with a computer in which software for operating the printer device while performing the processing shown in FIGS. 3 and 4 is installed, and further, software for executing the processing shown in FIGS. 3 and 4 is installed. It may be a computer etc.

  Moreover, each processing method mentioned above may be programmed. Alternatively, the program may be stored in a storage medium, such as a CD, DVD, or USB memory, and read by a computer. In this case, the signal processing device 1 may be a program stored in the storage medium placed in an external server of the signal processing device 1 and downloaded and used as necessary. In this case, the signal processing apparatus 1 has communication means for downloading the program in the storage medium.

It is a block diagram which shows the main structures of the signal processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the signal processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. FIG. 2 is a processing flowchart for explaining a processing routine related to an image restoration processing method (repetitive processing) performed by a processing unit of the signal processing device shown in FIG. 1. FIG. 4 is a flowchart for explaining a processing routine for determining validity of an update amount and correcting restored data shown in FIG. 3. FIG. 5 is a diagram schematically illustrating an example of an update amount before and after performing correction related to the processing flow illustrated in FIG. 4. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows image data when there is no camera shake. It is a figure for demonstrating concretely the processing method shown in FIG. 3 taking an example of camera shake, and is a figure which shows dispersion | distribution of energy when camera shake occurs. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which comparison data is generated from an arbitrary image. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example. The situation in which the comparison data is compared with the blurred original image to be processed to generate difference data. It is a figure for demonstrating. FIG. 4 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 by taking an example of camera shake, in which new comparison data is generated from the generated restored data, the data and the blurred original image to be processed, It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 3 is a diagram for specifically explaining the processing method shown in FIG. 3 using camera shake as an example, and a diagram for explaining a situation in which newly generated difference data is allocated and new restoration data is generated. is there.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Signal processing apparatus 4 Processing part Io Initial image data (arbitrary signal data)
Io 'Comparison data G Change factor information data Img' Original image data (original signal data)
I _{0 + n} restored data Img Original image (original signal)
δ Difference data Dc Update amount (distributed value)
S204, S206, S208, S210 Predetermined criteria

Claims (6)

From a change in the deterioration has occurred raw signal data, changing the signal before or originally should have been acquired signals or their approximation signal (hereinafter, referred to as the original signal) has a processing unit for restoring the previous serial processor utilizes the data in the change factor information that causes the signal change, to generate comparison data from any signal data, and the data for the data and the previous SL comparison of the original signal to be processed compare, obtained by the data of the difference using the data of the previous SL change factor information generates restored data by allocating before Symbol arbitrary signal data, signal data of the restored data before Symbol optionally In a signal processing device that uses it instead and repeats the same process repeatedly,
Calculated pre Symbol processor, a data signal-element constituting the front Symbol restore data to determine whether they meet a predetermined criterion, if not satisfy the predetermined criteria, by using the data of the previous SL differential is modified allocation values, it performs a correction process that uses the data obtained by the previous SL modified allocation values in place of the not satisfied data,
The predetermined criterion is generated from the data of the signal element to be determined and the data of the signal element in the vicinity thereof, and the value of the updated signal element obtained by the calculated distribution value is the maximum of the peripheral signal element. A signal processing apparatus characterized by determining that a predetermined criterion is not satisfied when the value exceeds or when the value is less than the minimum value of the peripheral signal elements . The signal processing apparatus according to claim 1, wherein the data of the peripheral signal element is a set of a plurality of the signal elements included in a range in which one of the signal elements is influenced by the change factor information. When the updated signal element value does not satisfy the predetermined standard, the updated signal element value is compared with the average value of the peripheral signal elements, and the distribution is corrected depending on whether the value exceeds the average value. Different values
The signal processing apparatus according to claim 1 or 2 .   The signal processing apparatus according to claim 1, wherein the correction process is a process of reducing an absolute value of the distribution value when the predetermined criterion is not satisfied. 2. The processing unit according to claim 1, wherein when the difference data is equal to or less than a predetermined value or smaller than the predetermined value, or when the number of repetitions reaches a predetermined number, the processing unit stops the repetition processing. 5. The signal processing device according to any one of 1 to 4 . Wherein the processing unit, during the iterative process, the data of the difference when the number of the repetition reaches a predetermined number of times is less than a predetermined value or less or before Symbol predetermined value stops, more than than the previous SL predetermined value or before SL is equal to or larger than the predetermined value, the signal processing apparatus according to claim 1, any one of 5, wherein the performing further predetermined number of times to repeat the process.

Images