JP4718618B2 - Signal processing device - Google Patents

Description

  The present invention relates to a signal processing apparatus.

  Conventionally, when a subject is photographed by a signal (image) processing device such as a camera, it is known that the image sometimes deteriorates. Factors of image degradation include camera shake during shooting, various aberrations of the optical system, lens distortion, and the like.

  In order to correct a photographed image that has deteriorated due to camera shake during photographing, a method of moving a lens and a method of circuit processing are known. For example, as a method of moving a lens, a method is known in which camera shake is detected and corrected by moving a predetermined lens in accordance with the detected camera shake (see Patent Document 1). In addition, as a circuit processing method, a change in the optical axis of the camera is detected by an angular acceleration sensor, a transfer function indicating a blurring state at the time of shooting is acquired from the detected angular velocity, etc., and the acquired transfer function is obtained for a shot image. Is known that corrects a deteriorated image by performing inverse transformation (see Patent Document 2).

  In addition to general captured images, it is known that various images such as X-ray photographs and microscopic images are deteriorated or changed due to blurring or other causes.

JP-A-6-317824 (see abstract) Japanese Patent Laid-Open No. 11-24122 (see abstract)

  The camera employing the camera shake correction described in Patent Document 1 requires a hardware space for driving a lens, such as a motor, and becomes large. In addition, such hardware itself and a drive circuit for operating the hardware are necessary, which increases costs. In addition, in the case of camera shake correction described in Patent Document 2, the above-described problems are eliminated, but there are the following problems. That is, it is theoretically possible that image restoration is performed by inverse transformation of the acquired transfer function, but as a practical problem, image restoration is difficult for the following two reasons.

  First, the transfer function to be acquired is very weak against noise, blur information error, etc., and the value fluctuates greatly due to these slight fluctuations. For this reason, the corrected image obtained by the inverse transformation is far from an image photographed with no camera shake, and cannot be used in practice. Second, when performing inverse transformation considering noise, etc., a method of estimating the solution by singular value decomposition etc. of the solution of simultaneous equations can be adopted, but the calculated value for the estimation becomes astronomical size. In practice, there is a high risk of being unable to solve.

  The above-mentioned problems that occur in images also appear in various general signal data, and it is accurate to restore the signal by inverse transformation of the transfer function, not to mention that the acquired transfer function is inaccurate. Even so, it has become difficult. In addition, obtaining a transfer function that is 100% accurate is a situation that is not possible in the natural world.

  As described above, an object of the present invention is to provide a signal processing apparatus that has a realistic circuit processing method while preventing an increase in size of the apparatus when restoring a signal.

  In order to solve the above problems, the signal processing device of the present invention is based on the original signal data that has undergone changes such as deterioration, the signal data before the change, the signal data that should have been originally acquired, or the approximate signal thereof. Data (hereinafter, these “signal data” are collectively referred to as original signal data), or similar signal data obtained by reducing or expanding the original signal data is generated using the data of the change factor information at the time of change. A processing unit that performs reduction processing to reduce the capacity value of the original signal data, reduced original signal data generated from arbitrary signal data using data of change factor information, and Distributing the difference data between the two data of the reduced data for comparison having the same capacity and degrading the arbitrary signal data to the arbitrary signal data based on the data of the change factor information, Generates small restoring data, using this reduced restored data in place of the reduced original signal data, has a function of performing iterative process repeats the same processing.

  According to the present invention, it is possible to generate reduced / restored data required when generating original signal data that is data before change from original signal data that has undergone changes such as deterioration. In that case, there is almost no increase in hardware, and the apparatus does not increase in size. Further, reduced data for comparison is created from arbitrary signal data, and difference data between the reduced data for comparison and the reduced original signal data to be processed is distributed to the reduced original signal data using the data of the change factor information. Since the process is repeated and the reduced restoration signal data is gradually obtained, the signal processing apparatus having a realistic circuit processing method can be obtained in restoring the signal. In addition, since the target of the iterative processing is not the original signal data but the reduced original signal data whose capacity is reduced, the processing speed is increased.

  In the signal processing device according to another invention, in addition to the above-described invention, the processing unit expands the reduced / restored data obtained at the end of the iterative processing, and expands the same capacity as the original signal data or the similar signal data. Get the process. By adopting this configuration, it is possible to obtain original signal data or similar signal data having a capacity comparable to that of the original signal data.

  In order to solve the above problems, the signal processing apparatus of the present invention is a factor for changing the original signal data or the similar signal data obtained by reducing or enlarging the original signal data from the original signal data in which a change such as deterioration has occurred. A processing unit that generates information using the data of the information, and the processing unit uses the reduced original signal data subjected to reduction processing for reducing the capacity value of the original signal data and the data of the change factor information from arbitrary signal data. The data of the difference between the two data of the reduced data for comparison, which has the same capacity as the reduced original signal data generated in this way and has deteriorated the arbitrary signal data, is distributed to the arbitrary signal data based on the data of the change factor information. Reduced-restored data is generated, the reduced-restored data is used in place of arbitrary signal data, and the first repeated process and the first repeated process are repeated. The enlargement process for enlarging the capacity value of the obtained reduced and restored data, and the enlarged signal data for comparison that has degraded the enlarged signal data is generated from the enlarged signal data obtained by the enlargement process, and the capacity of the original signal data or the original signal data The second difference data between the original signal similar signal data reduced or expanded to have the same capacity as the comparative enlarged data and the comparative enlarged data is distributed to the enlarged signal data based on the data of the change factor information. Thus, there is a function of generating enlarged restoration data, using the enlarged restoration data instead of the enlarged signal data, and performing a second iterative process that repeats the same process.

  According to the present invention, it is possible to restore original signal data or similar signal data from original signal data. In that case, there is almost no increase in hardware, and the apparatus does not increase in size. Further, in the first repetitive processing, comparison reduced data is created from arbitrary signal data, the first difference data between the comparison reduced data and the reduced original signal data to be processed is used as the change factor information data. In the second repetition process, comparison enlarged data is created from the reduced and restored data, and the comparison enlarged data and the original signal data to be processed or the original signal similar signal data are used. The process of distributing the second difference data to the enlarged signal data using the data of the change factor information is repeated. The first repetitive process and the second repetitive process gradually obtain enlarged restored signal data that is close to the original signal data or the similar signal data. Therefore, a signal processing apparatus having a realistic circuit processing method for restoring the signal. It can be. In the first iterative process, the object of the iterative process is not the original signal data, but the reduced original signal data whose capacity is reduced, so that the processing speed can be increased. Therefore, if the first repetition process can be shortened and the process within a predetermined time, it can be repeated a larger number of times, and the restoration quality of the reduced restoration data becomes good. Then, by performing the second iterative process using the reduced restoration data of good quality, the restoration quality of the enlarged signal is also good.

  In addition to the above-described invention, the signal processing apparatus according to another invention is configured such that the reduction process sets the x / y value to 0 when the capacity value of the reduced original signal data is x and the capacity value of the original signal data is y. .9 or less. By setting the value of this ratio to 0.9 or less, the first iterative process can be performed quickly. Therefore, the first iterative process can be shortened and the process within a predetermined time can be achieved. , The restoration quality of the reduced restoration data becomes good.

  In the signal processing device according to another invention, in addition to the above-described invention, the processing unit may perform the reduction process, the first iterative process, the enlargement process, and the first process only when a change obtained from the data of the change factor information is equal to or greater than a predetermined value. It is assumed that the process 2 is repeated. By adopting this configuration, only when the change is relatively large, the original signal data or the similar signal data is restored from the original signal data with the reduction process and the enlargement process. For this reason, since only the thing which tends to take time becomes a process target, it becomes more advantageous at the surface of time reduction.

  In the signal processing apparatus according to another invention, in addition to the above-described invention, the processing unit compares the reduced original signal data with the reduced data for comparison, and based on this comparison result, or the first difference data is a predetermined value. When the number of repetitions of the first repetition process is equal to or greater than a predetermined value, the first repetition process is terminated, and the original signal data or the original signal similar signal data is expanded for comparison. Contrast with data, based on these comparison results, or when the second difference data is less than or equal to a predetermined value, or when the number of repetitions of the second iteration is greater than or equal to a predetermined value, The second iterative process is ended. By adopting this configuration, when restoring the original signal data to the original signal data or similar signal data, a condition for ending the first repetition process and the second repetition process can be made suitable. Here, “control” refers to comparing two things together.

  In the signal processing device according to another invention, in addition to the above-described invention, the processing unit obtains the expanded restoration data as original signal data or similar signal data. By adopting this configuration, it is possible to obtain original signal data or similar signal data in which the first repetitive processing and the second repetitive processing are completed under suitable conditions.

  In addition to the above-described invention, a signal processing device according to another invention is configured such that the number of times of the second iterative process is smaller than the number of times of the first iterative process. The signal processing apparatus adopting this configuration focuses on the first iterative process having a high processing speed to improve the restoration quality of the reduced restoration data, and uses the reduced restoration data having the good quality to perform the second process. By performing iterative processing, the restoration quality of the enlarged signal can be improved.

  In addition to the above-described invention, a signal processing apparatus according to another invention uses signal data as image data. By adopting this configuration, when image deterioration due to camera shake occurs, original image data that is an image before change or an image that should have been originally acquired from an original image in which change such as deterioration has occurred, or It is possible to restore the approximate image data of the original image or similar image data obtained by reducing or enlarging the original image data.

  According to the present invention, it is possible to provide a signal processing apparatus having a realistic circuit processing method as well as preventing an increase in size of the apparatus when restoring a signal.

It is a block diagram which shows the main structures of the signal processing apparatus which concerns on embodiment of this invention. It is an external appearance perspective view which shows the outline | summary of the signal processing apparatus shown in FIG. 1, and is a figure for demonstrating the arrangement position of an angular velocity sensor. It is a figure for demonstrating the concept of the reduction process method performed with the process part of the signal processing apparatus shown in FIG. It is a figure for demonstrating the concept of the expansion process method performed with the process part of the signal processing apparatus shown in FIG. It is a processing flowchart for demonstrating the processing routine which concerns on the common process part of each repetition process performed in the process part of the signal processing apparatus shown in FIG. It is a figure for demonstrating the concept of the processing method shown in FIG. FIG. 6 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking hand shake as an example, and is a table showing energy concentration when there is no hand shake. FIG. 6 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking an example of camera shake, and is a diagram showing image data when there is no camera shake. FIG. 6 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking hand shake as an example, and is a diagram showing dispersion of energy when hand shake occurs. FIG. 6 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking an example of camera shake, and is a diagram for explaining a situation in which comparison data is generated from an arbitrary image. FIG. 5 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking hand shake as an example, and comparing the comparison data with the blurred original image to be processed to generate the first difference data It is a figure for demonstrating. FIG. 6 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking an example of camera shake, and is a diagram for explaining a situation in which restored data is generated by allocating difference data and adding it to an arbitrary image. . FIG. 5 is a diagram for specifically explaining the common processing shown in FIG. 5 by using hand shake as an example. In this case, new comparison data is generated from the generated restored data, the data and the blurred original image to be processed It is a figure for demonstrating the condition which produces | generates the data of a difference by comparing. FIG. 5 is a diagram for specifically explaining the common processing shown in FIG. 5 by taking an example of camera shake, and is a diagram for explaining a situation in which newly generated first difference data is allocated and new restoration data is generated. It is. It is a flowchart which shows the whole flow of each process performed by the process part of the signal processing apparatus shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Signal processing apparatus 2 Imaging | photography part (receiver)
DESCRIPTION OF SYMBOLS 3 Control system part 4 Processing part 5 Recording part 6 Detection part 7 Factor information storage part G Change function calculated from change factor information Img 'Original image data (captured image)
I _{0 + n} restored data Img Original image data (image before change or image data that should have been originally acquired)
I ₀ Arbitrary image data h Target image data (reduced and restored data obtained at the end, reduced and restored data obtained at the end, or original image data)
J Image data to be processed (original image data, reduced original image data, or enlarged image data)

  Hereinafter, a signal processing device 1 according to an embodiment of the present invention will be described with reference to the drawings. Although this signal processing device 1 is a consumer camera as an image processing device, it may be a camera for other uses such as a surveillance camera, a television camera, a handy type video camera, an endoscope camera, The present invention can also be applied to devices other than cameras, such as microscopes, binoculars, and diagnostic imaging apparatuses such as NMR imaging.

  FIG. 1 shows an outline of the configuration of the signal processing apparatus 1. The signal processing apparatus 1 includes a photographing unit 2 that captures an image of a person or the like, a control system unit 3 that drives the photographing unit 2, and a processing unit 4 that processes an image captured by the photographing unit 2. is doing. The signal processing apparatus 1 according to this embodiment further includes a recording unit 5 that records an image processed by the processing unit 4 and an angular velocity sensor, and detects change factor information that causes a change such as image degradation. And a factor information storage unit 7 for storing change factor information that causes image degradation and the like. Note that when the signal processing device 1 is applied as a device other than an image processing device, the photographing unit 2 is a receiving unit 2 that receives various input signals such as an audio signal (hereinafter, the photographing unit 2 and the receiving unit 2 are appropriately used). And will be used properly.)

  The photographing unit 2 includes a photographing optical system having a lens and a photographing element such as a CCD or a C-MOS that converts light passing through the lens into an electric signal. The control system unit 3 controls each unit in the signal processing device, such as the photographing unit 2, the processing unit 4, the recording unit 5, the detection unit 6, and the factor information storage unit 7.

  The processing unit 4 is configured by an image processing processor, and is configured by hardware such as an ASIC (Application Specific Integrated Circuit). The processing unit 4 generates a sampling frequency for detecting vibrations such as camera shake to be detected and supplies the sampling frequency to the detection unit 6. The processing unit 4 controls the start and end of vibration detection. Further, the processing unit 4 performs a reduction process for reducing the capacity value of the original signal data, an enlargement process for expanding the capacity value of the reduced / restored data, and first and second iterative processes. Further, the processing unit 4 determines whether to perform the reduction process, the first repetition process, the enlargement process, and the second repetition process according to the degree of change obtained from the data of the change factor information detected by the detection unit 6. To do. When the signal processing device 1 is applied as a device other than the image processing device, the reception sensitivity of the receiving unit 2 can be changed according to the magnitude of the input signal or the like.

  In addition, the processing unit 4 may store original image data (original signal data) or the like that is a source for generating comparison reduced data and comparison enlarged data. Further, the processing unit 4 is not configured as hardware such as an ASIC, but may be configured to perform processing by software. The recording unit 5 is composed of a semiconductor memory, but magnetic recording means such as a hard disk drive or optical recording means using a DVD or the like may be employed.

  As shown in FIG. 2, the detection unit 6 detects speeds around the X axis and the Y axis that are perpendicular to the Z axis that is the optical axis of the signal processing apparatus 1. If there is a camera shake at the time of shooting, the shot image becomes a blurred image. Such camera shake also causes movement in each direction in the X, Y, and Z directions and rotation around the Z axis. However, the most affected by each variation is the rotation around the Y axis and the X axis. Rotation around. These two variations are only slightly varied, and the captured image is greatly blurred. For this reason, in this embodiment, a PITCH (motion in the up and down (Y) direction) detection sensor and a YAW (movement in the left and right (X) direction) are used to detect blurring around the X axis and the Y axis in FIG. ) Two detection sensors are used. However, for the sake of completeness, a ROLL detection sensor for detecting rotation around the Z axis may be further added, or a sensor for detecting movement in the X direction or Y direction may be further added. The sensor used may be an angular acceleration sensor instead of an angular velocity sensor. When the signal processing apparatus 1 is applied to a signal other than an image signal and the reception characteristic or the response characteristic of the signal processing system is affected by, for example, temperature or humidity, the detection unit 6 includes a thermometer or A hygrometer can be included. In this way, the detection unit 6 observes a change factor that degrades the signal.

  The factor information storage unit 7 stores change factor information such as a change factor that degrades the signal, for example, aberration of the optical system and / or vibration detected by the detection unit 6, a point spread function calculated based on the vibration, and the like. It is a recording part. The change factor information recorded in the factor information storage unit 7 is, for example, original image data (change) that is original signal data from original image data (image data that has undergone changes such as deterioration) that is original signal data photographed most recently. In the restoration process to the similar image data which is the similar signal data obtained by reducing or enlarging the original image data or the image data before the image data or the original image data which should have been taken) Used. In addition, when the signal processing device 1 is applied as a device other than the image processing device, the temperature, humidity, and the like may change the reception characteristics of the receiving unit 2 and the characteristics of the entire system. It can be used as change factor information. Further, the response characteristic function of the system that is known in advance, such as the impulse response of the system, can be stored in the factor information storage unit 7.

  Here, the time when the original image data is restored to the original image data or the similar image data is the time when the original image is taken when the power supply for photographing is turned off or when the operation rate of the processing unit 4 is low. It can be a time delayed from. In this case, the original image data stored in the recording unit 5 and the change factor information (such as a point spread function) about the original image data stored in the factor information storage unit 7 are associated with each other. Stored for a long time. As described above, the advantage of delaying the timing of executing the restoration processing of the original image data from the timing of shooting the original image is that the burden on the processing unit 4 at the time of shooting involving various processes can be reduced.

  Next, the outline of the main processing (reduction processing, enlargement processing, each repetitive processing) performed by the signal processing apparatus 1 according to the present embodiment configured as described above will be described, and the overall processing flow will be described. Will be described with reference to FIG.

(Reduction processing)
FIG. 3 shows an outline of the reduction process. One unit of a square cell in the figure represents one of the pixels constituting the image data. The left side of the arrow is the pixel before the reduction process, and the right side is the pixel after the reduction process. FIG. 3A shows an example of a reduction process in which the average value of each pixel data of four adjacent pixels that form a square as a whole is replaced with the pixel data of one pixel. In this case, the reduction rate is 25%. FIG. 3B shows an example of a reduction process in which the average value of each pixel data of 16 adjacent pixels forming a square as a whole is replaced with the pixel data of one pixel. In this case, the reduction ratio is 6.25%.

  FIG. 3C shows an example of a reduction process in which nine adjacent pixels forming a square as a whole are converted into pixel data of four pixels. In this case, the reduction rate is about 44.4%. In this reduction processing method, the average value of the pixel data of the pixels A, B, D and E is replaced with the pixel data of the pixel a, and the average value of the pixel data of the pixels B, C, E and F is changed to the pixel b. Replace with pixel data, replace the average value of the pixel data of the pixels D, E, G and H with the pixel data of the pixel c, and replace the average value of the pixel data of the pixels E, F, H and I with the pixel d This is replaced with pixel data.

  FIG. 3 merely shows an outline of the reduction process by extracting a part of the image data. Actually, the reduction process shown in FIG. 3 is performed for all the pixels of the image data. In the signal processing device 1 according to the present embodiment, the reduction processing shown in FIG. The following explanation is based on this assumption. In the reduction process, one pixel is generated from a plurality of pixels by using pixel data values of pixels constituting the original image data, and the one pixel is used instead of the plurality of pixels. It is processing. Here, the method of “using the pixel data values of the pixels constituting the original image data” is not limited to the method of obtaining the average value of the pixel data of a plurality of pixels, but the method of calculating one pixel data of the plurality of pixels. A method of using the value as it is, a method of obtaining an average value of pixel data of a part of a plurality of pixels, a method of obtaining a value obtained by multiplying any of these values by a predetermined coefficient, or the like can be used. By this reduction processing, reduced original image data that does not change greatly in appearance can be obtained from the original image data (captured image data). In addition, when an average value of pixel data of a plurality of pixels is used, it is possible to restore an image with reduced noise such as white noise by generating one pixel using the average value. The reduced original image data obtained by the reduction process can also be used as image data displayed on a monitor such as a digital camera.

(Enlargement processing)
FIG. 4 shows an outline of the enlargement process. One unit of a square cell in the figure represents one of the pixels constituting the image data. The pixels a, b, c, and d correspond to the pixels a, b, c, and d shown in FIG. Each pixel has pixel data of a code assigned to each pixel. A pixel having pixel data of the average value ((a + b) / 2) of the pixel data of the pixel a and the pixel data of the pixel b is inserted between the pixels a and b shown in FIG. . Similar pixel insertion is performed between the pixels a and c, between the pixels c and d, between the pixels b and d, and between the pixels a and d. Here, the pixel data of the center pixel inserted between the pixels a and d may be an average value ((b + c) / 2) of the pixel data of the pixel b and the pixel data of the pixel c. This is because the central pixel exists between the pixels b and c. From the same viewpoint, this central pixel may be a pixel having pixel data of the average value ((a + b + c + d) / 4) of the pixel data of the pixels a, b, c and d. By this enlargement process, an enlargement process for returning to the number of pixels of the original image data before the reduction process shown in FIG. FIG. 4 merely shows a state of the enlargement process by extracting a part of the image data. Actually, the enlargement process shown in FIG. 4 is performed for all the pixels of the image data. The signal processing apparatus 1 according to the present embodiment employs the enlargement process shown in FIG. The following explanation is based on this assumption.

  In the enlargement process, a new pixel generated based on the pixel data to be enlarged is added to a plurality of pixels in the pixel constituting the reduced restoration data finally obtained through the first iterative process described later. It is a process to insert in between. Here, the “new pixel data generated based on the pixel data to be enlarged” is not limited to the average value of the pixel data of a plurality of adjacent pixels. An average value of pixel data of a part of a plurality of adjacent pixels, or any of these average values multiplied by a predetermined coefficient can be used. By this enlargement processing, enlarged image data that is enlarged can be obtained, although the appearance pattern does not change greatly from reduced original image data described later.

(Repeat process)
The iterative process performed by the signal processing apparatus 1 according to the present embodiment includes a first iterative process as a signal restoring unit for the reduced image data and a second iterative process as a signal restoring unit for the enlarged image data. There is. In addition, as will be described later, there is a third iterative process as a signal restoration unit for original image data without performing a reduction process and an enlargement process. First, a common process common to these three repetitive processes will be described below.

  In the common processing, the processing unit 4 generates comparison data from arbitrary image data using the data of the change factor information, and changes the difference data between the image data to be processed and the comparison data. This is a process of generating restored data by allocating to the arbitrary image data using the factor information data, using the restored data instead of the arbitrary image data, and repeating the same process.

An outline of the common processing will be described with reference to FIG. FIG. 5 is a processing flowchart for explaining a processing routine relating to repetitive processing as a signal (image) restoration means. In FIG. 5, “I ₀ ” is arbitrary image data, “J” is image data to be processed, and these are image data stored in the recording unit of the processing unit 4. Here, since “I ₀ ” is arbitrary image data, the image data “J” to be processed may be used as arbitrary image data. “I ₀ ′” indicates data of a change image of the arbitrary image data I ₀ , and this is comparison data for comparison. “G” is a change function (point image function) calculated from change factor information (= deterioration factor information) stored in the factor information storage unit 7.

“Δ” is difference data between the image data J to be processed and the comparison data I ₀ ′. “K” is an allocation ratio based on the data of the change factor information. “I _{0 + n} ” is data (restoration image) newly generated by allocating difference data δ to arbitrary image data I ₀ based on a change function (point spread function) G calculated from change factor information. Data). Here, assuming that “h” is the target image data without deterioration that is finally obtained, the relationship between the target image data h and the image data J to be processed is expressed by the following equation (1). be able to.
J = h * G (1)
Here, “*” is an operator representing a superposition integral (the same applies hereinafter).

Processing routine of the processing unit 4 first begins to prepare any image data _{I 0} (step S101). In step S102, obtains the equation (1) object image put any image data I ₀ instead of the data h, comparison data I ₀ is the change image _'. Next, the image data J to be processed is compared with the comparison data I ₀ ′ to calculate difference data δ (step S103).

In step S104, it is determined whether or not the absolute value of each difference data δ is less than a predetermined value. If the absolute value is greater than or equal to the predetermined value, new restored image data (= restored data) is generated in step S105. Process. That is, the difference data δ when the image data J to be processed and the comparison data I ₀ ′ are compared are distributed to arbitrary image data I ₀ based on the change function G, and new restored data I _{0 + 1} Is generated. Thereafter, steps S102, S103, S104, and S105 are repeated.

If the absolute value of the difference data δ is less than the predetermined value in step S104, the iterative process is terminated. Then, the restored data I _{0 + n} at the time when the iterative process is completed is estimated as the target image data h. That is, when the absolute value of the difference data δ becomes smaller than a predetermined value, it is determined that the restored data I _{0 + n that} is the basis of the comparison data I _{0 + n} ′ is sufficiently approximate to the target image data h, and the restored data I _{0 + n} is estimated as target image data h. Note that arbitrary image data I ₀ and change function G may be recorded in the recording unit 5 and transferred to the processing unit 4 as necessary.

  The concept of the common processing described above is summarized as follows. That is, in this processing method, the solution to be processed is not solved as an inverse problem, but as an optimization problem for obtaining a reasonable solution. Although it is theoretically possible to solve as an inverse problem, it is difficult as a real problem.

In the case of solving as an optimization problem, the common processing according to the present embodiment assumes the following conditions.
That is,
(1) The output corresponding to the input is uniquely determined.
(2) If the compared outputs are the same, their inputs are the same.
(3) The solution is converged by iteratively processing while updating the input so that the compared outputs are the same.

In other words, as shown in FIGS. 6A and 6B, if comparison data I ₀ ′ (I _{0 + n} ′) approximate to the image data J to be processed can be generated, the generation source Image data I ₀ (I _{0 + n} ) as data is approximated to the target image data h.

  Next, details of the blur image restoration (repetitive processing of steps S102, S103, S104, and S105) by the common processing shown in FIG. 5 are performed. The target image data h is the original image data Img, and the image data J to be processed is the original. Description will be made based on FIGS. 7, 8, 9, 10, 11, 12, 13, and 14 by taking the case of image data Img ′ (third repetitive processing described later) as an example.

(Image restoration algorithm)
When there is no camera shake, the light energy corresponding to a given pixel is concentrated on that pixel during the exposure time. In addition, when there is camera shake, light energy is dispersed to pixels that are shaken during the exposure time. Further, if the blur during the exposure time is known, it is possible to know how to disperse the energy during the exposure time, so that it is possible to create a blur-free image from the blurred image.

  Hereinafter, for the sake of simplicity, the description will be made in one horizontal dimension. The pixels are designated as S-1, S, S + 1, S + 2, S + 3,. When there is no blur, the energy during the exposure time is concentrated on the pixel, so the energy concentration is “1.0”. This state is shown in FIG. The imaging results at this time are shown in the table of FIG. What is shown in FIG. 8 is the correct image data Img when no deterioration occurs. Each data is represented by 8-bit (0 to 255) data.

  It is assumed that there is blurring during the exposure time, 50% of the exposure time is blurred to the Sth pixel, 30% of time is shifted to the S + 1th pixel, and 20% of time is shifted to the S + 2th pixel. The way of energy dispersion is as shown in the table of FIG. This is a change function (point spread function) G calculated from the change factor information.

  Blur is uniform for all pixels and is perceived as a shift invariant problem. If there is no upper blur (vertical blur), the blur situation is as shown in the table of FIG. The data shown as “blurred image” in FIG. 10 is the deteriorated original image data Img ′. Specifically, for example, “120” of the pixel “S-3” is “0.5”, “0.3”, ““ of the change function (point spread function) G calculated from the change factor information that is blur information. According to the distribution ratio of 0.2, “60” is distributed to the “S-3” pixel, “36” to the “S-2” pixel, and “24” to the “S-1” pixel. Similarly, “60” which is the pixel data of “S-2” is distributed as “30” in “S-2”, “18” in “S-1”, and “12” in “S”. The original image data Img is calculated from the deteriorated original image data Img 'and the change function (point spread function) G calculated from the change factor information shown in FIG.

Any image data _{I 0} shown in step S101 in FIG. 5, when this description uses the original image data Img '. That is, the process starts with I ₀ = Img ′. In the table of FIG. 11, “input” corresponds to arbitrary image data I ₀ . This data I _0, that is, Img ′ is integrated with the change function (point spread function) G calculated from the change factor information in step S102. That is, for example, “60” of the “S-3” pixel of the predetermined image data I ₀ is “30” for the “S-3” pixel, “18” for the “S-2” pixel, “12” is allocated to each pixel of “S-1”. The other pixels are similarly distributed, and comparison data I ₀ ′ shown as “output I ₀ ′” is generated. Therefore, the difference data δ in step S103 is as shown in the bottom column of FIG.

Thereafter, in step S104 of FIG. 5, it is determined whether or not the absolute value of the difference data δ of the plurality of pixels constituting the image of the original image data Img ′ and the comparison data I ₀ ′ is less than a predetermined value. To do. If the result of the determination is “N”, the process proceeds to step S105. That is, the difference data δ is distributed to the arbitrary image data I ₀ using the change factor information data G to generate restored data I _{0 + n} shown as “next input” in FIG. In this case, since this is the first time, it is represented as I _{0 + 1 in} FIG.

The distribution of the difference data δ is performed by using, for example, the pixel data “30” of “S-3” (= “S-3”) using the change function (point spread function) G calculated from the change factor information. "15" multiplied by 0.5, which is the distribution ratio of "S"), is distributed to the "S-3" pixel, and "S-2" is assigned to the data "15" of the "S-2" pixel. “4.5” multiplied by 0.3, which is the distribution ratio that should have arrived at the pixel of “S-1”, is further allocated to the data “9.2” of the pixel “S-1”, "1.84" multiplied by 0.2, which is the distribution ratio that should have come to the pixel "." The total amount (update amount) allocated to the pixels of “S-3” is “21.34”, and this value is added to the data I ₀ of the predetermined image (here, the original image data Img ′ is used). The restoration data I _{0 + 1} is generated.

As shown in FIG. 13, the restored data I _{0 + 1} becomes the input image data (= predetermined image data I ₀ ) in step S102 in FIG. 5, step S102 is executed, and the process proceeds to step S103. The difference data δ is obtained. Thereafter, the determination in step S104 is made in the same manner as described above. If “N” as a result of the determination, the process proceeds to step S105, where the new difference data δ is distributed to the previous restoration data I _{0 + 1} to generate new restoration data I _{0 + 2} (see FIG. 14). Thereafter, new comparison data I _{0 + 2} ′ is generated from the restored data I _{0 + 2} by performing step S102. As described above, after steps S102 and S103 are executed, if the determination in step S104 is “N”, the process proceeds to step S105. Such a process is repeated.

  As described above, the absolute value of the difference data δ gradually decreases as a result of repeated processing. When the absolute value of the difference data δ becomes smaller than a predetermined value, the determination in step S104 becomes “Y”, and the original image with reduced blurring is obtained. Data close to the image data Img is obtained.

  In the blurred image restoration processing method shown in FIG. 5 described above (repetitive processing (common processing) of steps S102, S103, S104, and S105), the processing performed by the processing unit 4 is configured by software. However, each of them may be configured by hardware made up of parts that share part of the processing. Further, the change factor information data G includes not only the deterioration factor information data but also information for simply changing an image and information for improving an image contrary to deterioration.

  The number of processing repetitions may be set automatically or fixedly on the signal processing device 1 side. In that case, the set number of times may be changed by a change function (point spread function) G calculated from the change factor information. For example, when the data of a certain pixel is distributed over many pixels due to blurring, the number of repetitions may be increased, and when the dispersion is small, the number of repetitions may be decreased.

  Furthermore, during the repetitive processing, when the difference data δ diverges or the energy of the image data after the energy has moved does not decrease but increases, the processing may be stopped. For example, it is possible to adopt a method of determining whether or not the light is diverging by observing the average value of the difference data δ and determining that the light is diverging if the average value is larger than the previous value. In addition, during an iterative process, if an input is to be changed to an abnormal value, the process may be stopped. For example, in the case of 8 bits, if the value to be changed is a value exceeding 255, the processing is stopped. Also, during an iterative process, when an input that is new data is to be changed to an abnormal value, that value may be used instead of the normal value. For example, when a value exceeding 255 within the 8-bit range of 0 to 255 is used as input data, it is processed as 255, which is the upper limit value.

  Further, when generating restoration data to be an output image, depending on the change function (point spread function) G calculated from the change factor information, data that goes out of the area of the image to be restored is generated. There is a case. In such a case, data that protrudes outside the area is input to the opposite side. If there is data that should come from outside the area, the data may be brought from the opposite side. For example, when data allocated to a lower pixel is generated from the data of the pixel XN1 positioned at the bottom in the area, the position is outside the area. Therefore, the data is processed to be allocated to the pixel X11 located at the top right above the pixel XN1. Similarly, the pixel N2 adjacent to the pixel XN1 is assigned to the topmost pixel X12 (= next to the pixel X11) directly above.

  In the common processing, the sampling rate of the angular velocity detection sensor is set within 60 Hz to 240 Hz. However, the angular velocity may be detected every 5 μsec so that a high frequency can be detected. In addition, in this embodiment, the value serving as a determination criterion for the difference data δ is “6” when each data is represented by 8 bits (0 to 255). That is, when it is smaller than 6, the process is finished. Further, the shake raw data detected by the angular velocity detection sensor does not correspond to the actual shake when the sensor itself is insufficiently calibrated. Therefore, in order to deal with actual blurring, when the sensor is not calibrated, correction is required to multiply the raw data detected by the sensor by a predetermined magnification.

The above is the processing method of the common processing. In the first iterative process, which is a processing method based on the same concept as this common process, “arbitrary image data I ₀ ” in the common process is reduced to the reduced original image data obtained by reducing the captured original image data Img ′. And Further, the “comparison data I ₀ ′” in the common processing uses the change function (point image function) G1 calculated from the arbitrary factor of the image data I ₀ (reduced original image data subjected to the reduction process). The comparison reduced data has the same capacity as the reduced reduced original image data. Further, “image data J to be processed” in the common processing is reduced original image data. Further, “difference data δ” in the common process is data of the first difference between the reduced original image data and the reduced data for comparison. Further, “restoration data I _{0 + n} ” in the common processing is generated by allocating the first difference data to the reduced original image data using the change function (point spread function) G1 calculated from the change factor information. Reduced restoration data. In addition, “target image data h” in the common processing is the reduced restoration data obtained at the end.

In the second iterative process, the “arbitrary image data I ₀ ” in the common process is an enlarged image obtained by enlarging the last reduced restoration data (target image data h) obtained by performing the first iterative process. Data. In addition, the “comparison data I ₀ ′” in the common processing uses the change function (point image function) G2 calculated from the change factor information for the arbitrary image data I ₀ (enlarged image data that has been enlarged). The enlarged data for comparison is deteriorated. Further, “image data J to be processed” in the common process is the original image similar image data having the same capacity as the original image data Img ′ or the comparison enlarged data. Further, “difference data δ” in the common processing is data of the second difference between the enlarged image data and the comparative enlarged data. In addition, “restoration data I _{0 + n} ” in the common processing is generated by allocating the second difference data to the enlarged image data using the change function (point spread function) G2 calculated from the change factor information. Enlarged restoration data. Further, the “target image data h” in the common processing is the enlarged restoration data (original image data Img or similar image data) obtained at the end.

The third repetitive process is a common process performed on the predetermined image data. That is, in the third repetitive processing, the original image data Img ′ may be used for “arbitrary image data I ₀ ” in the common processing, and black solid, white solid, gray solid, checkered pattern, etc. Simple image data may be used. In addition, the “comparison data I ₀ ′” in the common process is a change function (point spread function) calculated from the change factor information in the arbitrary image data I ₀ (arbitrary image data) in the third repetition process. ) Comparison data deteriorated by using G3. Further, “image data J to be processed” in the common process becomes the original image data Img ′ in the third repetition process. Further, the “difference data δ” in the common process becomes the third difference data between the original image data Img ′ and the comparison data in the third repetition process. Further, the “restoration data I _{0 + n} ” in the common process is an arbitrary one using the change function (point spread function) G3 calculated from the change factor information in the third difference data in the third repetition process. The restored data is generated by distributing the image data. Further, the “target image data h” in the common process becomes the original image data Img in the third repetition process.

(Overall process flow)
FIG. 15 shows a flowchart of overall processing mainly performed by the processing unit 4 of the signal processing device 1 according to the present embodiment. First, information Ga related only to shake is obtained from the change function (point spread function) G calculated from the change factor information obtained by the detection unit 6, and whether or not Ga is equal to or less than a predetermined value, that is, whether or not the degree of shake is present. It is determined whether it is larger (step S201). If Ga is less than or equal to a predetermined value and the degree of blurring is small, the third iterative process is performed (step S202), the original image data is obtained as the finally obtained restored image data, and the image restoration process ends. To do. Here, the reason for adopting the third iterative process when the degree of blurring is small is that the data of the original image Img ′ in a state where the blurring is not so much when the reduction process and the enlargement process are performed on the image data with small blurring. This is because reducing or enlarging the image increases the risk that the image will deteriorate or the processing time will increase. As described above, the reduction process, the first repetition process, the enlargement process, and the second repetition process are performed only when the change (Ga in this case) obtained from the change factor information exceeds a predetermined value, The processing method of performing the third repetitive processing in this case is not necessarily required, and a processing flow that does not include steps S201 and S202 depending on the specifications of the signal processing device 1 or the image to be processed can be used. Note that in step S201, information including information other than blur may be used instead of the information Ga regarding blur.

  When it is determined that Ga exceeds a predetermined value (step S201), that is, when the degree of blurring is large, a reduction process for reducing the capacity value of the original image data Img ′ obtained by photographing (FIG. 3). (See (C)) (step S203). Next, a first iterative process is performed on the reduced original image data (step S204). Then, an enlargement process (see FIG. 4) is performed to enlarge the capacity value of the last reduced / restored data obtained by performing the first repetition process so as to be the same as the capacity value of the original image data Img ′ (step S205). ). A second iterative process is performed on the enlarged image data subjected to the enlargement process (step S206), and finally restored image data is obtained as original image data, and the signal processing apparatus 1 according to the present embodiment The image restoration process performed by the processing unit 4 ends.

  When the processing time is limited, the number of times of the second repetition process is preferably smaller than the number of times of the first repetition process. By doing so, the first iterative process can be performed many times, and the restoration quality of the reduced restoration data becomes good. At the same time, the second repetitive processing is performed supplementarily (confirmatively) using the reduced restoration data of good quality, so that the restoration quality of the enlarged image data having a large capacity by the enlargement processing is also good. Can be.

  In the first, second, and third iterations, the accuracy of image restoration usually increases as the number of iterations increases (not proportionally). On the other hand, the processing time required for the entire image restoration becomes longer in proportion to the number of repetitions. If the number of repetitions is the same, the time required for one repetition process increases as the amount of image data increases.

  Here, it is assumed that the image data of the original image data Img ′ is 5 million pixels, and the reduced original image data is about 300,000 pixels equivalent to VGA (Video Graphics Array) (reduction ratio is about 6%). explain. In this experiment, the processing time for performing the first repetitive processing 50 times is equivalent to the time for performing the third repetitive processing three times for the original image data Img ′. Thereafter, when the second iterative process is performed twice, the processing time is equivalent to performing the third iterative process twice for the original image data Img ′. A restored image when the third repetition process is performed five times on the original image data Img ′, and an enlarged restored image when the second repetition process is performed twice after the first repetition process is performed 50 times Clearly, the latter enlarged restored image had higher accuracy and quality of image restoration. Moreover, it was equivalent to the restored image when the third repetitive processing was performed 50 times.

  As another method for increasing the speed of the iterative process, a method of increasing the feedback gain (“k” shown in FIG. 5: distribution ratio based on data of change factor information) in the iterative process can be considered. However, if the value of the feedback gain “k” is increased too much, “kδ” shown in FIG. 5 becomes too large and diverges in the iterative process, making it impossible to converge (the value of δ is made sufficiently small). There is a case. Therefore, it is generally not appropriate to increase the value of the feedback gain and perform the third iterative process. Further, by increasing the number of repetitions, noise such as white noise that may be included in the original image data Img ′ may be emphasized. However, in the present embodiment, the noise included in the pixel data is averaged (distributed) during each reduction and enlargement process, and the influence is reduced. Further, since the enlarged image data obtained from the first iterative process is close to a good restoration quality, the difference data δ in the second iterative process is sufficiently small, and therefore the feedback gain “ Even if the value of “k” is increased, kδ does not become too large. Therefore, the image restoration process performed by the signal processing device 1 according to the present embodiment increases the feedback gain value in the second iterative process as compared with the case where the first iterative process and the third iterative process are performed. The adverse effect of this is small. Therefore, in the signal processing device 1 according to the present embodiment, it is possible to further speed up the image restoration process compared to the case where the third iterative process is performed.

  Since the signal processing apparatus 1 in the present embodiment repeatedly generates a restored image and improves its quality, the apparatus does not increase in size. In addition, the signal processing device 1 according to the present embodiment generates comparison data from predetermined image data using the data of the change factor information, and the difference between the image data to be processed and the comparison data The restoration data is generated by allocating the data to the predetermined image data using the data of the change factor information, and this restoration data is used in place of the predetermined image data. Since image data or similar image data is obtained, a realistic circuit processing method is used for image restoration processing.

  The signal processing apparatus 1 according to the embodiment of the present invention has been described above. However, the present invention can be variously modified without departing from the gist thereof. For example, the restored image data finally obtained may be similar image data that becomes similar signal data obtained by reducing or enlarging the original image data, instead of the original image data. In order to obtain similar image data, enlarged image data (original image similar image data) different from the capacity of the original image is generated during enlargement processing. Then, the original image data Img ′ is subjected to reduction processing or enlargement processing so as to have the same capacity as the original image similarity image data, thereby obtaining capacity-adjusted original image data. Then, the second iterative process is executed using the original image data whose capacity has been adjusted instead of the image data J to be processed in step S103 of FIG.

  Another method for obtaining the similar image data is, for example, a method in which only the reduction process and the first repetition process are performed, and the final reduced / restored data obtained by performing the first repetition process is obtained as the similar image data. is there. By adopting this method, the second iterative process can be omitted, so that the entire image restoration process can be speeded up. When this method is adopted, the capacity of the reduced original image data obtained by the reduction process is adjusted to be the same as the capacity of the image data that has been subjected to the restoration process finally obtained. Alternatively, the similar image data obtained by such a method may be processed by enlarging the reduced restoration data obtained at the end of the first iterative process to obtain enlarged data having the same capacity as the similar image data. good. The enlargement method in this case may be a method other than the enlargement process (see FIG. 4) according to the present embodiment.

  As the original image data Img ′ has a higher resolution and better image quality, the restoration quality of the image data finally obtained by performing the image restoration process becomes better. Therefore, the advantage of obtaining similar image data is, for example, that after obtaining high-quality original image data Img ′ due to high resolution, reduction processing and various repetition processing (image restoration processing) are performed, and finally For example, it is possible to obtain image data with less blurring and good handleability with a smaller capacity than the original image data Img ′. Note that the resolution of the similar image data finally obtained by the signal processing device 1 may be fixedly set, or a plurality of types of resolution may be set. The digital camera as the signal processing apparatus 1 capable of obtaining similar image data is preferably provided with a mechanism for selecting or adjusting the resolution of the image that the operator wants to finally obtain.

  As shown in FIG. 15, the image restoration process performed by the signal processing device 1 in the present embodiment includes a reduction process (step S203), a first repetition process (step S204), an enlargement process (step S205), and a second process. The process is repeated in the order of the repetitive processing (step S206). However, for example, after the second iterative process (step S206), the reduction process and the first iterative process may be performed again and then terminated. Furthermore, the entire image restoration process performed in the order of the reduction process (step S203), the first repetition process (step S204), the enlargement process (step S205), and the second repetition process (step S206) shown in FIG. May be repeated multiple times. In addition, before or after the image restoration process or at any time between various processes (step S203 to step S206), a process unrelated to the image restoration process such as so-called γ correction can be performed.

  The reduction processing shown in FIG. 3C performed by the signal processing device 1 in the present embodiment is x / y where x is the capacity value of the reduced original image data and y is the capacity value of the original image data. This is a process for setting the value (reduction ratio) to about 0.444. The reduction ratio of the reduction processing performed by the signal processing device 1 is preferably 0.05 or more. The reason is to avoid excessive loss of pixel data of the original image data Img ′. The reduction ratio is preferably 0.9 or less, and more preferably 0.5 or less. The reason is to perform the first repetition process quickly.

  In the first and second iterative processes performed by the signal processing device 1 in the present embodiment, as shown in FIG. 5, the absolute value of the first difference data δ between the reduced original image data and the reduced data for comparison, When the absolute value of the second difference data δ between the enlarged image data and the comparative enlarged data is less than the predetermined value (step S104), the first and second iterative processes are terminated. However, the condition for ending the first iterative process is that the reduced original image data is compared with the reduced data for comparison, and based on this comparison result or when the first difference data is equal to or less than a predetermined value, Or it can be set as the case where the frequency | count of repetition of a 1st repetition process becomes a predetermined value. Further, the condition for ending the second iterative process is to compare the original image similar image data having the same capacity as the original image data or the comparative enlarged data and the comparative enlarged data, and based on these comparison results or A case where the second difference data becomes equal to or smaller than a predetermined value or a case where the number of repetitions of the second repetitive processing becomes a predetermined value can be used. Further, the determination criterion in step S201 may be “<” instead of “≦”.

Here, the above-mentioned “control” includes comparing a predetermined correspondence relationship between a plurality of data to be compared. For example, the processing unit 4 performs a process of ending the iterative process shown in FIG. 5 if a specific portion of both data satisfies a certain relationship even if the image data to be compared with each other does not approximate. It's also good. In addition, the first and second difference data δ may be a simple difference between corresponding pixels, but generally varies depending on a change function (point spread function) G calculated from change factor information. . For example, the first and second difference data δ is expressed by the following equation (2).
δ = f (J, I ₀ , G) (2)

  In the present embodiment, the restoration target is image data. However, these restore processing concepts and techniques can be applied to any data restore process. For example, it can be applied to restoration of digital audio data. In the case of audio signal data, data obtained by digitizing sound at regular intervals can be handled as signal element data in the same manner as pixel data in image data.

  Each signal restoration method described above may be programmed. Alternatively, the program may be stored in a storage medium, such as a CD, DVD, or USB memory, and read by a computer. In this case, the signal processing device 1 may be a program stored in the storage medium placed in an external server of the signal processing device 1, downloaded as needed, and used. In this case, the signal processing apparatus 1 has communication means for downloading the program in the storage medium.

Claims (7)

From the original signal data that has undergone changes such as deterioration, the data of the signal before the change, the data of the signal that should have been originally acquired, or the data of its approximate signal (hereinafter these “signal data” are collectively referred to as In the signal processing apparatus having a processing unit that generates similar signal data obtained by reducing or expanding the original signal data using the data of the change factor information at the time of the change,
The processing unit has the same capacity as the reduced original signal data reduced by reducing the capacity value of the original signal data and the reduced original signal data generated from arbitrary signal data using the data of the change factor information. Further, the reduced data for comparison is distributed to the arbitrary signal data based on the data of the change factor information to generate reduced restoration data. Using the reduced restoration data instead of the arbitrary signal data, and repeating the same process,
An enlargement process for enlarging the capacity value of the reduced restoration data obtained by performing the first iterative process, and a comparison enlarged data in which the enlarged signal data is deteriorated is generated from the enlarged signal data obtained by the enlargement process. The original signal data or the original signal similar signal data obtained by reducing or enlarging the original signal data to have the same capacity as the comparative enlarged data, and the second difference data between the comparative enlarged data A second iteration of repeating the same processing by allocating to the enlarged signal data based on the data of the change factor information to generate enlarged restored data, using the enlarged restored data instead of the enlarged signal data And a signal processing device. The reduction process is characterized in that the value of x / y is 0.9 or less, where x is the capacity value of the reduced original signal data and y is the capacity value of the original signal data. The signal processing apparatus according to claim 1 . The processing unit performs the reduction process, the first iterative process, the enlargement process, and the second iterative process only when the change obtained from the data of the change factor information is a predetermined value or more. The signal processing apparatus according to claim 1 . The processing unit compares the reduced original signal data with the reduced data for comparison, based on the comparison result, or when the first difference data is equal to or less than a predetermined value, or the first When the number of repetitions of the repetition process is equal to or greater than a predetermined value, the first repetition process is terminated,
The original signal data or the original signal similar signal data is compared with the comparative enlarged data, based on the comparison result, or when the second difference data is equal to or less than a predetermined value, or 2. The signal processing apparatus according to claim 1 , wherein when the number of repetitions of the second iterative process is equal to or greater than a predetermined value, the second iterative process is terminated. The signal processing apparatus according to claim 1 , wherein the processing unit obtains the expansion restoration data as the original signal data or the similar signal data.   The signal processing apparatus according to claim 1, wherein the number of times of the second iterative process is smaller than the number of times of the first iterative process. The signal processing apparatus according to any one of claims 1 to 6 you, characterized in that the data of the image data of said signal.

Images