JP2008225657A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method to much more easily remove movement blur since it is difficult to tune such image processing as movement blur removal by one line. <P>SOLUTION: A variable shift register 51-2 extracts the set of first pixels and second pixels adjacent in a moving direction, that is, a set in which the pixels are isolated only by movement quantity V as a unit from the sets of pixels to be extracted by a variable shift register 51-2. As for the set of i(i is a positive integer)-th set of N(N is a positive integer) pieces of set, an approximation arithmetic part 54-1 subtracts a result acquired by multiplying a value acquired by subtracting a value acquired by subtracting 1/(N-i) from the movement quantity V and a coefficient α by the pixel data of the second pixels from a result acquired by multiplying the pixel data of the first pixels by a coefficient α and the movement quantity V for calculating a value for predicting the pixel values from which movement blur has been removed. The coefficient α is calculated by a formula 25. This invention may be applied to an image processor. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program capable of removing motion blur.

  Video cameras using an image sensor such as a CCD (Charge Coupled Device) are widely used.

  In an image obtained by imaging a moving object with such a video camera, motion blur occurs when the moving speed of the object is relatively fast.

  Various proposals have been made to remove motion blur by image processing. Many image processes for removing motion blur use polynomials using predetermined coefficients.

  This coefficient is calculated in advance by a method having a large calculation amount such as a least square method.

  Conventionally, a first signal that is a real-world signal having a first dimension is projected onto a sensor, and a second signal having a second dimension that is smaller than the first dimension obtained by detection by the sensor is obtained. In some cases, significant information buried by projection is extracted from the second signal by performing signal processing based on the second signal (see, for example, Patent Document 1).

JP 2001-250119 A

  However, it is not easy to change the coefficients used in the polynomial. When classifying into classes and using coefficients for each class, the same number of coefficient sets as the number of classes is stored in ROM (Read Only Memory) or the like. There is a need.

  In such a case, it is difficult to tune the image processing for removing motion blur online.

  The present invention has been made in view of such a situation, and makes it possible to remove motion blur more easily.

により求められる。 An image processing apparatus according to one aspect of the present invention is an image processing apparatus that processes an image including pixel data for each of a predetermined number of pixels acquired by an imaging element having a time integration effect, and includes: a first pixel; Extraction means for extracting a pair that is a pair of second pixels adjacent to the first pixel in the direction of motion and that is separated from the other pair by a motion amount V in units of pixels; From the result obtained by multiplying the pixel data of the first pixel by the coefficient α and the motion amount V for the i (i is a positive integer) -th set of (positive integer) pairs, from the motion amount V By subtracting the result obtained by multiplying the pixel data of the second pixel by the value obtained by subtracting 1 / (N−i) and the coefficient α, the pixel value from which motion blur is removed is predicted. First calculating means for calculating a value, the coefficient α is

Is required.

  For each of the N sets, a predicted value calculation unit that calculates a predicted value of a pixel value from which motion blur has been removed by adding the values calculated by the first calculation unit can be further provided.

により求められるようにすることができる。 The pixel of the second pixel is obtained by multiplying the pixel data of the first pixel by the coefficient β and the motion amount V for the k-th group (k is a positive integer) among the N groups. Subtracting the result obtained by multiplying the data by the coefficient β and the motion amount V further includes second calculating means for calculating a value for predicting a pixel value from which motion blur is removed, and the coefficient β

As required.

  Predicted value calculating means for calculating predicted values of pixel values from which motion blur has been removed by adding the values calculated by the first calculating means or the second calculating means for each of the N sets. Further, it can be provided.

  Selection means for selecting a predicted value close to the median value of the plurality of predicted values among the plurality of predicted values predicted from the values calculated by the plurality of first calculating means may be further provided.

  Sorting means for sorting a plurality of predicted values predicted from values calculated by the plurality of first calculating means, and a predicted value close to the median of the plurality of predicted values from the sorted predicted values. In addition, a selection unit that selects a predetermined number of prediction values and an average value calculation unit that calculates an average value of the selected prediction values can be further provided.

により求められる。 An image processing method according to an aspect of the present invention is an image processing method for processing an image including pixel data for each of a predetermined number of pixels acquired by an image sensor having a time integration effect, the first pixel, A pair with the second pixel that is adjacent to the first pixel in the direction of motion, and is separated from the other pair by a motion amount V in units of pixels, and N (N is a positive integer) ) Out of the sets, the result of multiplying the pixel data of the first pixel by the coefficient α and the amount of motion V for the first group (i is a positive integer) N−i) is subtracted from the result obtained by multiplying the pixel data of the second pixel by the value obtained by subtracting the coefficient α and a value for predicting the pixel value from which motion blur has been removed is calculated. The coefficient α is

Is required.

により求められる。 A program according to one aspect of the present invention is a program for causing a computer to perform image processing for processing an image including pixel data for each of a predetermined number of pixels acquired by an imaging device having a time integration effect. And a second pixel that is adjacent to the first pixel in the direction of movement, and is separated from the other group by a motion amount V in units of pixels, and N (N Is a positive integer) of the i (i is a positive integer) number of sets, the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V is the motion amount V. In order to predict a pixel value from which motion blur has been removed by subtracting a result obtained by multiplying a value obtained by subtracting 1 / (N−i) from the pixel data of the second pixel from the pixel data of the second pixel. The computer performs image processing including a step of calculating the value of

Is required.

により求められる係数αと動き量Vとを前記第１の画素の画素データに乗算した結果から、動き量Vから１／（Ｎ−ｉ）を減算して得られた値と係数αとを前記第２の画素の画素データに乗算した結果を減算することにより、動きボケを除去した画素値を予測するための値が演算される。 In one aspect of the present invention, a set of a first pixel and a second pixel adjacent to the first pixel in the direction of movement, the set amount being separated from the other set by a movement amount V in units of pixels. For the i-th (i is a positive integer) -th set of N (N is a positive integer) sets,

The value obtained by subtracting 1 / (N−i) from the motion amount V and the coefficient α from the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V obtained by By subtracting the result obtained by multiplying the pixel data of the second pixel, a value for predicting the pixel value from which motion blur is removed is calculated.

  As described above, according to one aspect of the present invention, motion blur can be removed.

  Also, according to one aspect of the present invention, motion blur can be removed more easily.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

により求められる。 An image processing apparatus according to one aspect of the present invention is an image processing apparatus that processes an image including pixel data for each of a predetermined number of pixels acquired by an imaging element having a time integration effect, and includes: a first pixel; Extraction means (for example, in FIG. 13) that extracts a pair that is a pair with the second pixel that is adjacent to the first pixel in the direction of motion and that is separated from the other pair by a motion amount V in units of pixels. Of the variable shift register 51-1) and the N (N is a positive integer) set, the coefficient α and the motion of the pixel data of the first pixel for the i (i is a positive integer) -th set By subtracting the result obtained by multiplying the pixel data of the second pixel by the value obtained by subtracting 1 / (N−i) from the amount of motion V and the coefficient α from the result of multiplying the amount V. First calculation means for calculating a value for predicting a pixel value from which motion blur is removed (for example, FIG. With 3 the approximate calculation section 54-1) and, the coefficient alpha,

Is required.

  Prediction value calculation means (for example, the prediction value in FIG. 13) that calculates the predicted value of the pixel value from which motion blur is removed by adding the values calculated by the first calculation means for each of the N sets. A computing unit 55) can be further provided.

により求められるようにすることができる。 The pixel of the second pixel is obtained by multiplying the pixel data of the first pixel by the coefficient β and the motion amount V for the k-th group (k is a positive integer) among the N groups. Second calculation means for calculating a value for predicting a pixel value from which motion blur is removed by subtracting the result obtained by multiplying the data by the coefficient β and the motion amount V (for example, the equivalent conversion calculation unit in FIG. 13). 53-1), and the coefficient β is

As required.

  For each of the N sets, a predicted value calculation unit that calculates a predicted value of a pixel value from which motion blur has been removed by adding the values calculated by the first calculation unit or the second calculation unit ( For example, the predicted value calculation unit 55) of FIG. 13 can be further provided.

  Selection means (for example, selection unit 72 in FIG. 14) that selects a predicted value close to the median value of the plurality of predicted values among a plurality of predicted values predicted from the values calculated by the plurality of first calculating means. Can be further provided.

  Sort means (for example, the sorting unit 71 in FIG. 14) for sorting a plurality of predicted values predicted from the values calculated by the plurality of first calculating means, and a plurality of predictions from the sorted plurality of predicted values. A selection unit (for example, the selection unit 72 in FIG. 14) that selects a predetermined number of prediction values that are close to the median value, and an average value calculation unit that calculates an average value of the selected prediction values (For example, an average value calculation unit 73 in FIG. 14) can be further provided.

により求められる。 Image processing to be performed by a computer according to an image processing method or program of one aspect of the present invention is a set of a first pixel and a second pixel adjacent to the first pixel in the direction of motion, Is extracted from the other set by the motion amount V in units of (for example, step S34 in FIG. 16), and i (i is a positive integer) among N (N is a positive integer) sets. ) For the second set, the value obtained by subtracting 1 / (N−i) from the motion amount V and the coefficient α from the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V. And subtracting the result obtained by multiplying the pixel data of the second pixel by a value to predict a pixel value from which motion blur has been removed (for example, step S37 in FIG. 16), α is

Is required.

  FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus includes a motion detection unit 11, an affine transformation unit 12, a blur removal parameter generation unit 13, a blur removal computation unit 14, a computation result comparison / selection unit 15, and an inverse affine transformation unit 16.

  Image data from the outside is supplied to the motion detection unit 11 and the affine transformation unit 12. The motion detection unit 11 detects a motion vector of the image based on the image data, and outputs the detected motion vector.

  When a motion vector of an image of image data is detected externally, the motion vector detected from the outside is supplied. When a motion vector is supplied from the outside, the motion vector supplied from the outside is supplied to the blur removal parameter generation unit 13 via a switch provided between the motion detection unit 11 and the blur removal parameter generation unit 13. The

  On the other hand, when no motion vector is supplied from the outside, the motion vector detected in the motion detection unit 11 is converted to a blur removal parameter generation unit via a switch provided between the motion detection unit 11 and the blur removal parameter generation unit 13. 13 is supplied.

  The blur removal parameter generation unit 13 generates a blur removal parameter indicating a motion amount and a motion angle from the motion vector. The blur removal parameter generation unit 13 supplies the blur removal parameter indicating the amount of motion and the angle of motion to the affine transformation unit 12 and the inverse affine transformation unit 16. Also, the blur removal parameter generation unit 13 supplies the blur removal parameter indicating the amount of motion to the blur removal calculation unit 14.

  The affine transformation unit 12 refers to the angle of motion indicated by the blur removal parameter supplied from the blur removal parameter generation unit 13 and affine transforms the image data so that the direction of motion is the horizontal direction of the image. The affine transformation unit 12 extracts a pixel row that is a row of pixels in the horizontal direction of the image from the affine transformed image data, and supplies the extracted pixel row to the blur removal calculation unit 14.

  In other words, the pixel column supplied from the affine transformation unit 12 to the blur removal calculating unit 14 is composed of pixels arranged in a row in the movement direction.

  For example, the affine transformation unit 12 supplies a pixel column including a predetermined number of pixels such as 10 to 20 to the blur removal calculation unit 14.

  Alternatively, the affine transformation unit 12 is a pixel row having a length corresponding to the amount of motion indicated by the blur removal parameter from the image data subjected to affine transformation, and is a row of one pixel in the horizontal direction of the image. The column is extracted, and the extracted pixel column is supplied to the blur removal calculation unit 14. For example, the affine transformation unit 12 supplies a pixel row including a number of pixels corresponding to the motion amount, such as a multiple of the motion amount, to the blur removal calculation unit 14.

  The blur removal calculation unit 14 calculates a predicted value of a pixel value that does not include motion blur (a pixel value from which motion blur has been removed) by applying a calculation described later to the pixel row supplied from the affine transformation unit 12. . The blur removal calculating unit 14 calculates one or a plurality of predicted values for one pixel value.

  The blur removal calculation unit 14 supplies the predicted value obtained by the calculation to the calculation result comparison / selection unit 15.

  The calculation result comparison / selection unit 15 selects a more likely prediction value from the prediction values supplied from the blur removal calculation unit 14, and obtains a pixel value from the selected prediction value. The calculation result comparison / selection unit 15 supplies the obtained pixel value to the inverse affine transformation unit 16.

  The inverse affine transformation unit 16 stores the pixel values of pixels not including motion blur supplied from the calculation result comparison / selection unit 15 in a buffer provided therein. Then, the inverse affine transformation unit 16 refers to the amount of motion and the angle of motion indicated by the blur removal parameter supplied from the blur removal parameter generation unit 13 to make the same as the image based on the input image data. In this way, image data composed of pixel values of pixels not including motion blur is affine transformed.

  The inverse affine transformation unit 16 outputs image data that does not include motion blur, that is, image data of an image from which motion blur has been removed.

  Next, with reference to FIGS. 2 to 11, calculation of a predicted value of a pixel value that does not include motion blur will be described.

In the following description, the motion amount V indicates the amount of motion of an image based on input image data. Image data D ₀ ,..., D _n indicate pixel values of the pixels of the input image data. Each of the image data J ₀ ,..., J _m is a value indicating the intensity of light from the subject in the real world.

That is, the image data J ₀ ,..., J _m are images that do not include motion blur caused by an imaging element (photoelectric conversion element) having a time integration effect such as a CCD or CMOS (Complementary Metal-Oxide Semiconductor) sensor, respectively. Indicates the pixel value. It can be said that the image data J ₀ ,..., J _m indicate pixel values of pixels when a real-world image is represented by virtual pixels.

Hereinafter, the image data D _0, · · ·, the D _n, referred blurred image data D _0, · · ·, and D _n.

In the following, the image data J _0, ..., a J _m, real world image data J _0, referred ..., and J _m.

Figure 2 is a blurred image data D _0, · · ·, D _n and the real world image data J _0, · · ·, a diagram showing the relationship between J _m.

  For example, an image sensor such as a CCD or CMOS sensor outputs image data of an image composed of 30 frames per second. The exposure time of the image sensor can be 1/30 second. The exposure time is a period from the start of the conversion of the incident light into the electric charge until the end of the conversion of the incident light into the electric charge. Hereinafter, the exposure time is also referred to as shutter time.

  For example, an imaging device that is a CCD converts incident light into electric charges for a period corresponding to a shutter time, and accumulates the converted electric charges. The amount of charge is approximately proportional to the intensity of the incident light and the time during which the light is incident. The imaging element adds the charge converted from the incident light to the already accumulated charge during the period corresponding to the shutter time. That is, the image sensor integrates incident light for a period corresponding to the shutter time, and accumulates an amount of charge corresponding to the integrated light. It can be said that such an image sensor has an integration effect with respect to time.

  The electric charge accumulated in the image sensor is converted into a voltage value, and the voltage value is further converted into a pixel value such as image data that is digital data and output. Therefore, it can be said that each pixel value output from the image sensor is a result of integrating light from a subject in the real world, which is spread over time, with respect to the shutter time.

  FIG. 2 is a model diagram in which pixel values of pixels arranged in a line adjacent to the moving direction of the subject image in the image data obtained by capturing the moving subject are developed in the time direction.

  The vertical direction in FIG. 2 indicates the elapsed time from the top to the bottom in the figure. The position of the upper side of the upper rectangle in FIG. 2 corresponds to the time when the image sensor starts to convert the incident light into charges, and the position of the lower side of the upper rectangle in FIG. This corresponds to the time when the conversion of the converted light into electric charge is completed. That is, the distance from the upper side to the lower side of the upper rectangle in FIG. 2 corresponds to the shutter time.

  The horizontal direction in FIG. 2 corresponds to the spatial direction. More specifically, in the example shown in FIG. 2, the upper rectangle in FIG. 2 indicates 18 pixels that are continuous in the spatial direction, and each square surrounded by a thick line indicates one pixel. .

  Here, the period corresponding to the shutter time is divided into two or more periods having the same length. A virtual division number, which is a number for dividing the period corresponding to the shutter time, is set corresponding to the amount of movement V of the subject within the shutter time.

  For example, when the motion amount V in units of pixels is 3, the number of virtual divisions is 3 corresponding to the motion amount V being 3, and the period corresponding to the shutter time is divided into 3. .

  The top row of the upper rectangle in FIG. 2 corresponds to the first divided period after the shutter is opened. The second row from the top of the upper rectangle in FIG. 2 corresponds to the second divided period from when the shutter has opened. The third row from the top of the upper rectangle in FIG. 2 corresponds to the third divided period from when the shutter has opened.

  Since one frame is a short time, it can be assumed that the subject is a rigid body and is moving at a constant speed.

  When the direction of motion of the subject image is from the left to the right in FIG. 2 and the amount of motion V of the subject image is 3, the subject image is 3 in the next frame with reference to a certain frame. Move so that it is displayed on the right side by the amount of pixels.

  When the amount of motion V of the subject image is 3, when attention is paid to one frame, the subject image moves to the right across three pixels in one frame. That is, it can be said that the image of the subject moves by the movement amount V in one frame.

  Considering motion blur, motion blur does not occur in an image of a stationary subject, and motion blur occurs in an image of a moving subject.

  Looking at the relationship between the light from the subject and the pixel, the light from one point of the stationary subject is incident on one pixel of the image sensor, whereas the light from one point of the moving subject. Is incident on a plurality of pixels of the image sensor in one frame, that is, in the shutter time.

  An image in which light from one point of the subject does not move beyond the range of one pixel of the image sensor, in other words, an image of a subject in which light from one point of the subject moves within the range of one pixel, Does not occur.

  Therefore, if an image of the subject can be obtained in a period in which light from one moving subject is incident on one pixel of the image sensor, motion blur should not occur in the image.

  Since the image of the subject moves by the amount of movement V in one frame, if the number of virtual divisions is the same as the amount of movement V, light from one point of the subject is within the range of one pixel during the divided period. Therefore, the subject image in the divided period does not include motion blur.

Accordingly, as shown in FIG. 2, when the same as the motion amount V virtual division number, the subject of the image in the divided period, the real world image data J _0, · · ·, be represented by J _m it can.

Explaining this from a different aspect, for example, as shown in FIG. 2, light from a predetermined point of the subject and represented by real-world image data J ₀ is 1 pixel in the shutter time. Move by 3 pixels in the direction of the line.

In the first divided period after the shutter is opened, the light represented by the real-world image data J ₀ is incident on the leftmost pixel of the upper rectangle in FIG. 2, and the second, In the divided period, the light represented by the real world image data J ₀ is incident on the second pixel from the left of the upper rectangle in FIG. Further, in the third divided period after the shutter is opened, the light represented by the real world image data J ₀ is incident on the third pixel from the left of the upper rectangle in FIG.

Similarly, in the first divided period after the shutter is opened, the light represented by the real world image data J ₁ is incident on the second pixel from the left of the upper rectangle in FIG. 2, and the shutter is opened. in a second, divided period Te, the light represented by the real world image data J ₁ is incident on the third pixel from the left of a rectangular upper side in FIG. The shutter is the third open, in divided period, the light represented by the real world image data J ₁ is incident on the fourth pixel from the left of the upper rectangle in FIG.

In the first divided period after the shutter is opened, the light represented by each of the real world image data J _{2 to the} real world image data J ₁₇ is the third from the left of the upper rectangle in FIG. It enters each of the pixel through the 18th pixel. In the second divided period after the shutter is opened, the light represented by each of the real world image data J _{2 to the} real world image data J ₁₆ is the fourth pixel from the left of the upper rectangle in FIG. It enters each of the 18th to 18th pixels. Further, in the third divided period after the shutter is opened, the light represented by each of the real world image data J _{2 to the} real world image data J ₁₅ is the fifth from the left of the upper rectangle in FIG. Are incident on each of the first through eighteenth pixels.

  As described above, since the image sensor has an integration effect with respect to time, each pixel of the image sensor integrates light from the subject with respect to the shutter time, and outputs a pixel value corresponding to the result of integration. .

For example, as shown in FIG. 2, the blurred image data D ₀ is obtained by integrating the real world image data J ₀ and the real world image data J ₁ , and the blurred image data D ₁ is the real world image data J 1. ₀ , real world image data J ₁ and real world image data J ₂ are integrated.

Similarly, the blurred image data D ₂ is obtained by integrating the real world image data J ₁ , the real world image data J _2, and the real world image data J ₃ , and the blurred image data D ₃ is the real world image data J 3. ₂ and real world image data J ₃ and real world image data J ₄ are integrated.

In the example shown in FIG. 2, each of the blurred image data D _{4 to the} blurred image data D ₁₅ is obtained by integrating three consecutive ones of the real world image data J _{3 to the} real world image data J ₁₆ .

  In general, the relationship between the blurred image data D and the real world image data J is expressed by Expression (1).

In the following description, the blur removal processing result image data K _0, · · ·, the K _m, the image processing apparatus, the blurred image data D _0, · · ·, a value obtained from the D _n, the real world image Data J ₀ ,..., J _m are predicted values.

Further, the prediction errors E ₀ ,..., _Em are errors between the blur removal processing result image data K and the real world image data J. That is, when i = 0,..., M, the prediction error E _i = K _i −J _i .

  Further, the index i and the index j are used to specify the blurred image data D and the like.

Now, focusing on the blurred image data D ₁ and the blurred image data D ₂ in FIG. 2, as shown in FIG. 3, the blurred image data D ₁ includes real world image data J ₀ and real world image data J ₁ . Real world image data J ₂ is integrated, and blur image data D ₂ is an integration of real world image data J ₁ , real world image data J _2, and real world image data J _3. the result of subtracting the blurred image data D ₂ from the data D ₁ is equal to the result of subtracting the real world image data J ₃ from the real world image data J _0.

  The difference value of the pixel values of adjacent pixels in the blurred image data matches the difference value of the pixel values of pixels separated by the motion amount V in the real world image data J divided by the motion amount V. When this is generalized for the adjacent blurred image data D, Expression (2) is established.

In addition, the pixel values of the blurred image data D _{(n × V)} and the blurred image data D _{(n × V + 1)} , which are adjacent pixels of the blurred image data D, are [1,-(i−1) / i]. That is, the result of the calculation of D _{(n × V)} -(i-1) / i × D _{(n × V + 1)} is separated by the motion amount V in the real world image data J. The result obtained by dividing the difference value of the pixel values of the obtained pixels by the motion amount V is multiplied by (i-1) / i, and 1 / i × D _{(n × V)} is added.

  When this is generalized for the adjacent blurred image data D, Expression (3) is established.

In the formula (3), considered 1 / i × D a _{(n × V)} as the prediction error E, for example, as shown in FIG. 4, the blurred image data D _1, 1/2 to blurred image data D ₂ By subtracting the multiplied value, J ₀ × 1/2 + (J ₀ -J ₃ ) × 1/2 can be obtained as a value approximating the real world image data J ₀ .

  It should be noted that, in equation (3), when 1 / i × D (n × V) is moved to the left side, it can be transformed into equation (2).

  Hereinafter, the operation of ax + by is represented by (a, b) (x, y) or (a, b).

  Moreover, Formula (4) is materialized.

  From equation (4), as shown in FIG. 5, 1- (N-1) / N = 1 / N, (N-1) / N (1- (N-2) / (N-1)) = 1 / N, (N-1) / N (N-2) / (N-1) (1- (N-3) / (N-2)) = 1 / N is generalized to formula (5 ) Holds.

1からNまでのインデックスiについて、式（４）のサメンションを取ると、その総和は、必ず1になる。

For the index i from 1 to N, if the summation of equation (4) is taken, the sum is always 1.

  In the blur removal calculation unit 14, two types of calculation units are used in combination. Here, the two types of calculators are referred to as an approximate calculator and an equivalent converter, respectively.

These two types of computing units are small-scale filters that are applied to adjacent blurred image data D _i and blurred image data D _{(i + 1)} .

The approximation calculator applies the calculation of Expression (6) to the blurred image data D _i and the blurred image data D _{(i + 1)} .

That is, the approximate arithmetic unit calculates a _i ×× from the blur image data D _i and the blur image data D _{(i + 1)} located on the right side with respect to the blur removal processing result image data K to be output at that time. V × D _i -a _i × (V-1 / (Ni)) × D _{(i + 1)} is calculated, and it is located on the left side with respect to the blur removal processing result image data K to be output at that time From the blurred image data D _i and the blurred image data D _{(i + 1)} , −a _i × (V−1 / (Ni)) × D _i + a _i × V × D _{(i + 1)} is calculated.

N is the total number of computing units. Further, the coefficient a _i is determined by the equation (8) described later.

  Hereinafter, the calculation represented by Expression (6) is also referred to as an approximate calculation.

The equivalent converter applies the calculation of Expression (7) to the blurred image data D _i and the blurred image data D _{(i + 1)} .

That is, the equivalent converter calculates a _i ×× from the blur image data D _i and the blur image data D _{(i + 1)} located on the right side with respect to the blur removal processing result image data K to be output at that time. V × D _i −a _i × V × D _{(i + 1)} is calculated, and the blur image processing result image data K to be output at that time is subjected to blur image data D _i and a blur image located on the left side. From the data D _{(i + 1)} , -a _i × V × D _i + a _i × V × D _{(i + 1)} is calculated.

  Hereinafter, the operation represented by Expression (7) is also referred to as equivalent conversion operation.

Since the gain of the approximate calculator is a _i / (Ni) and the gain of the equivalent converter is 0, the approximate calculator and the equivalent converter in the blur removal calculating unit 14 are appropriately combined by appropriately combining the approximate calculators. The total gain can be unity.

In order to systematically set the total gain by the approximate calculator and the equivalent converter in the blur removal calculating unit 14 to 1, a _i is determined by Expression (8).

なお、N=1およびi=0である場合、係数a_iは、1とする。

When N = 1 and i = 0, the coefficient a _i is 1.

When the coefficient a _i is determined by the equation (8), the total gain by the approximate calculator and the equivalent converter in the blur removal calculating unit 14 is 1 as shown in the equation (9) from the equation (4). Become.

Here, when the blur removal processing result image data K ₀ is obtained from the blur image data D ₀ and the blur image data D _{1 by} one approximate arithmetic unit, the blur removal processing result image data K ₀ is expressed by the equation (10). The value indicated by.

In this case, the prediction error E ₀ is D ₁ −J _{(V / 2)} .

Also, if the blur removal processing result image data K ₀ is obtained from the blur image data D ₀ and the blur image data D ₁ with one approximate calculator and one equivalent converter, the blur removal processing result image data K _0. Is a value represented by equation (11).

The overall prediction error E _{total in} this case is −J _{(V + V / 2)} + D _{(V + 1)} .

As can be seen, all the prediction errors E are generated by the approximation calculator, and each prediction error Ei generated by the approximation calculator is J _{(iV / 2)} −D _{i + 1} × (gain of the approximation calculator) ).

  As described above, the blur removal processing result image data K is calculated by combining one or a plurality of approximate calculators and zero or more equivalent converters.

  Considering generalization of one or a plurality of approximate arithmetic units to be combined and zero or more equivalent converters, an integrated expression of the approximate arithmetic unit and the equivalent converter is expressed by Expression (12).

  In the following, the number of approximate arithmetic units is represented by W, and the number of equivalent converters is represented by X.

  The index S (w) indicates the w-th element (w> = 0) of the set of indices i of the approximate arithmetic unit. For example, in the integrated expression f (i), f (0) is an approximation operation, f (1) is an equivalent conversion operation, f (2) is an approximation operation, and f (3) is an equivalent conversion operation. When f (4) is an approximate operation, S (0) = 0, S (1) = 2, S (2) = 4 holds. That is, the index S (w) indicates the index i of the wth approximate arithmetic unit.

The blur removal processing result image data K ₀ that is the predicted value to be obtained is obtained by an operation represented by a prediction operation equation (13) in which one or a plurality of approximate operations and zero or more equivalent conversion operations are combined. be able to.

In this case, the prediction error E _i generated in the i-th arithmetic unit indicated by the index i, which is an approximate arithmetic unit or an equivalent converter, is expressed by Expression (14).

As shown in the equation (15), if the scalar value b _i is 0 when f (i) is an equivalent conversion operation, and f (i) is an approximation operation, the coefficient a _i is a prediction error. E _total is _{calculated |} required by Formula (16).

  Next, a combination of one or a plurality of approximate arithmetic units and zero or more equivalent converters for obtaining a predicted value will be described.

  In order to stably obtain the predicted value, it is desirable that the gain of the arithmetic unit, that is, the gain of the predicted value is 1.

As described above, the gain of the approximate calculator is a _i / (Ni), and the gain of the equivalent converter is 0.

  When a predicted value is obtained by combining two types of arithmetic units, that is, an approximate calculator and an equivalent converter, a combination of one or more approximate calculators and zero or more equivalent converters for calculating the predicted value is comprehensive. In order to set the gain to 1, the combination may be determined by paying attention to the approximate calculator.

  Here, we propose a systematic method to make the gain of the overall prediction value 1 while distributing the error evenly. A stable prediction value can be obtained by evenly distributing the error with respect to the blurred image data D used in the calculation.

  First, the gain of the equivalent converter is 0, and the equivalent converter does not contribute to the gain of the overall prediction value. Therefore, how to combine approximate calculators, specifically, the weight of the output of each calculator Determine as follows.

  From equation (4), equation (17) holds, and from equation (17), equation (18) holds.

  From equation (8), equation (19) holds.

  When formula (20) is expanded, it can be seen that the gain in the computing unit is always 1 / N. From this, it can be seen that when the gain is added to the index i which is 1 to N, it becomes 1.

  Note that, from the equation (20), the above-described equation (9) is obtained.

Therefore, the gain of each approximate arithmetic unit is 1 / N, and as described above, each error Ei generated in each approximate arithmetic unit is J _{(iV / 2)} -D _i × (approximation unit gain) ), The error gains are equal in each approximate computing unit.

  That is, it can be said that the error is generated according to the value of the blur image data D to which the approximate calculation unit applies the approximate calculation, and is distributed to the blur image data D to which the approximate calculation is applied.

  Here, with reference to FIG. 6 thru | or FIG. 10, the specific example of the calculation of the predicted value by an approximate calculator and an equivalent converter is demonstrated.

Figure 6 is similar to FIG. 2, the unsharp image data D _0, · · ·, D _n and the real world image data J _0, · · ·, a diagram showing the relationship between J _m.

First, as shown in FIG. 6, when the real world image data J ₀ indicated by A in FIG. 6 is obtained, the blurred image data D ₁ and the blurred image data D adjacent to the blurred image data D ₁ in the direction of motion. ₂ is extracted. Further, from the blurred image data D ₁ , the blurred image data D _{4 that} is separated by the pixel of the motion amount V in the direction of motion and the blurred image data D ₅ that is adjacent to the blurred image data D ₄ in the direction of motion are extracted. from the image data D _4, the blurred image data D ₈ adjacent in the direction of the motion blurred image data D ₇ and the blurred image data D ₇ separated by the pixels of the motion amount V in the direction of motion is extracted.

As shown in FIG. 7, the difference obtained by subtracting the blurred image data D ₂ from the blurred image data D ₁ is that the real world image data J ₁ and the real world image data J ₂ are offset as can be seen from FIG. Therefore, it is equal to the difference obtained by subtracting the real world image data J ₃ from the real world image data J ₀ .

Similarly, the difference obtained by subtracting the blurred image data D ₅ from the blurred image data D ₄ is equal to the difference obtained by subtracting the real world image data J ₆ from the real world image data J _3, blurred image from the blurred image data D ₇ difference obtained by subtracting the data D ₈ is equal to the difference obtained by subtracting the real world image data J ₉ from the real world image data J _6.

Real world image data J ₀ from the difference obtained by subtracting the real world image data J _3, real-world image data J ₃ from the difference obtained by subtracting the real world image data J _6, and real world image data J ₆ real world image data from J ₉ Since the real world image data J ₃ and the real world image data J ₆ are offset when the sum of the differences obtained by subtracting is subtracted, the difference obtained by subtracting the real world image data J ₉ from the real world image data J ₀ It will be required.

That is, the difference obtained by subtracting the blurred image data D ₂ from the blurred image data D _1, the difference obtained by subtracting the blurred image data D ₅ from the blurred image data D _4, the difference obtained by subtracting the blurred image data D ₈ from the blurred image data D ₇ When obtaining the sum, difference obtained by subtracting the real world image data J ₉ from the real world image data J ₀ is obtained.

Thus, for example, the difference obtained by subtracting the blurred image data D ₂ from the blurred image data D ₁ is equal to the difference obtained by subtracting the real world image data J ₃ from the real world image data J ₀ , and is expressed by Expression (7). It can be seen that a value obtained by multiplying the difference between the two real world image data J by a predetermined constant is obtained by one equivalent conversion operation.

Then, a difference obtained by subtracting the blurred image data D ₂ from the blurred image data D _1, the difference obtained by subtracting the blurred image data D ₂ from the blurred image data D _1, the difference obtained by subtracting the blurred image data D ₅ from the blurred image data D ₄ , And the sum of the differences obtained by subtracting the blurred image data D ₈ from the blurred image data D ₇ , and the difference obtained by subtracting the real world image data J ₃ from the real world image data J ₀ and the real world image data J, respectively. _The difference obtained by subtracting the real world image data J ₉ from ₀ is obtained, and the number to be subtracted is the same as the real world image data J ₀ , and the subtraction number is the real world image data J ₃ to the real world image data It has changed to J _9.

  That is, when the difference between the adjacent blur image data D separated by the pixel of the motion amount V in the motion direction is added to the difference between the blur image data D adjacent in the motion direction, the subtracted number is the motion direction by the motion amount V. And the difference between the two real world image data J can be obtained.

  A value obtained by multiplying a difference between two real world image data J obtained by moving the subtraction number in the movement direction by the movement amount V by a predetermined constant is obtained by combining a plurality of equivalent conversion operations represented by Expression (7). I understand.

Then, the blurred image data D ₁ and blurred image data D _2, and the blurred image data D ₄ and the unsharp image data D _5, by using the blurred image data D ₇ and the unsharp image data D _8, will be described approximate operation .

As shown in FIG. 8, the blurred image data D _1, the subtracting the result of multiplying blurred image data D ₂ to 2/3, blurred image data D ₁ is, the real world image data J ₀ the real world image data Since J ₁ and real world image data J ₂ are integrated, blur image data D ₂ is an integration of real world image data J ₁ , real world image data J _2, and real world image data J ₃ . The result obtained by multiplying the real world image data J ₃ by 2/3 is subtracted from the real world image data J _0, and the sum of the real world image data J ₁ and the real world image data J ₂ is multiplied by 1/3. A value obtained by adding the result obtained to the subtraction result is obtained.

Moreover, the blurred image data D _4, Subtracting the results obtained by multiplying 1/2 to the blurred image data D _5, blurred image data D ₄ is, real world image data J ₃ and the real world image data J ₄ real world image is obtained by integrating the data J _5, the blurred image data D _5, since the real world image data J ₄ the real world image data J ₅ such that the integral of the real world image data J _6, the real world image data J ₆ result of multiplying 1/2 subtracted from the real world image data J _3, real-world image data J ₄ and the real world and the image data J ₅ multiplied by 1/2 in the sum addition result of the subtraction A value added to the result is obtained.

Moreover, the blurred image data D _7, Subtracting the result of multiplying 0 to blur the image data D _8, blurred image data D ₇ is the real world image data J ₆ the real world image data J ₇ the real world image data J since ₈ and such a the integral, the real world image data J ₆ and the real world image data J ₇ value obtained by adding the real world image data J ₈ is obtained.

The result obtained by subtracting the result obtained by multiplying the blurred image data D ₁ by 2/3 from the blurred image data D ₂ is obtained by multiplying the result obtained by multiplying the real world image data J ₃ by 2/3 from the real world image data J _0. Subtract the real world image data J ₁ and the real world image data J ₂ and add the result of 1/3 to the result of the subtraction to remove the components of the real world image data J ₃ real Given that, from the blurred image data D _4, obtained by subtracting the result of multiplying 1/2 blurred image data D _5, the result obtained by multiplying 1/2 to the real world image data J ₆ subtracted from world image data J _3, real-world images included the results obtained by multiplying 1/2 to the value obtained by adding to the subtraction result to the sum obtained by adding the real world image data J ₄ and the real world image data J ₅ data for component J ₃ may be utilized.

That is, the blurred image data D _4, by multiplying 2/3 to the result obtained by subtracting the result of multiplying 1/2 blurred image data D _5, its value, the blurred image data D _1, subtracting from the result obtained by subtracting the result of multiplying the blurred image data D ₂ to 2/3.

As a result, the result obtained by subtracting the result obtained by multiplying the blur image data D ₂ by 2/3 from the blur image data D ₁ is obtained by multiplying the result obtained by multiplying the real world image data J ₃ by 2/3. From the _{value obtained} by subtracting the result of subtracting from J ₀ and adding 1/3 to the sum of real world image data J ₁ and real world image data J ₂ to the result of the subtraction, the real world image data J ₃ Ingredients are erased.

Then, from the result of erasing the components of the real world image data J ₃ , it is considered to further erase the components of the real world image data J ₆ .

The result obtained by subtracting the result obtained by multiplying the blurred image data D ₈ by 0 from the blurred image data D ₇ is multiplied by 1/2, and the value is deleted from the component of the real world image data J ₃ When subtracting from the result, the component of the real world image data J _{3 and} the component of the real world image data J ₆ are deleted.

  In this way, unnecessary real world image data J can be deleted in order, and the necessary real world image data J can be approximated.

This approximation simply, as shown in FIG. 9, the real world image data J _0, the real world image data J ₀ the real world image data J ₁ the real world image data J ₂ and the added value, That is, the image is approximated by the result obtained by multiplying the blurred image data D ₁ by 1/3, and the real world image data J ₃ is converted into the real world image data J ₃ , the real world image data J _4, and the real world image data J ₅ . The approximate value is obtained by multiplying the added image, that is, the blurred image data D ₄ by 1/3, and the real world image data J ₆ is replaced with the real world image data J ₆ , the real world image data J _7, and the real world image data J. _The approximation is based on the value obtained by adding ₈ and the result obtained by multiplying the blurred image data D ₇ by 1/3.

In this case, the error E ₀ indicates an error when the real world image data J ₀ is approximated by the result of multiplying the blurred image data D ₁ by 1/3.

The error E ₃ indicates an error when the real world image data J ₃ is approximated by the result of multiplying the blurred image data D ₄ by 1/3, and the error E ₆ is 1/3 in the blurred image data D ₇ . the indicating an error when approximating the real world image data J ₆ with the result of multiplying.

  Such an approximate calculation and an equivalent conversion calculation are combined to obtain a predicted value.

Figure 10 is an equivalent transform operation is applied to the blurred image data D ₁ and the blurred image data D _2, to apply the approximate calculation in the blurred image data D ₄ and the blurred image data D _5, blurred image data D ₇ and the blurred image data apply the equivalent transformation operation on D _8, to apply the approximate calculation in the blurred image data D ₁₀ and the blurred image data D _11, the equivalent conversion operation applied to the blurred image data D ₁₃ and the blurred image data D _14, the blurred image data in the case of applying the approximate calculation in D ₁₆ and the blurred image data D _17, it is a diagram illustrating an example of a combination of approximation calculation equivalent conversion operation.

  In FIG. 10, the solid line underline indicates the equivalent conversion calculation term, and the dotted line underline indicates the approximation calculation term.

  For each of the terms of the arithmetic expression for the blurred image data D shown on the upper side of FIG. 10, if the equivalent conversion arithmetic term is replaced with the real world image data J and the approximate arithmetic term is replaced with the approximate value, FIG. 10 can be rewritten as an expression corresponding to the second figure from the top.

  Furthermore, when the approximate value is replaced with the real world image data J and the error E, it can be rewritten into an expression corresponding to the third figure from the top of FIG. The underline of the alternate long and short dash line indicates a term replaced with real world image data J and error E.

When this is expanded and the coefficients related to the error E are arranged, finally, the blur removal processing result image data K ₀ that is the predicted value is converted into the real world image data J ₃ by the prediction error E ₃ and the prediction error E ₉ . prediction error E ₁₅ result multiplied by 1/3 to a value obtained by adding understood to be a value obtained by adding.

Thus, as shown in FIG. 11, the set of the blurred image data D ₁ adjacent thereto and the blurred image data D _0, the set of the blurred image data D ₀ and the blurred image data D ₁ in the direction of movement the set of the blurred image data D _V separated by the pixels of the motion amount V with blurred image data D _{V + 1} adjacent thereto, from the set of the blurred image data D _V and the blurred image data D _{V + 1} in the direction of movement the set of the blurred image data D _2V separated by pixels of the motion amount V with blurred image data D _{2V + 1} adjacent thereto, from the set of the blurred image data D _2V and the blurred image data D _{2V + 1} in the direction of movement the set of the blurred image data D _3V separated by pixels of the motion amount V with blurred image data D _{3V + 1} adjacent thereto, from the set of the blurred image data D _3V and the blurred image data D _{3V + 1} in the direction of movement Approximate operation for each set of blurred image data D _4V separated by pixels of motion amount V and blurred image data D _{4V + 1} adjacent to this Alternatively, either equivalent conversion processing is applied.

For example, by applying a set to approximate calculation of the unsharp image data D ₀ and the blurred image data D _1, the equivalent conversion operation applied to the set of the blurred image data D _V and the blurred image data D _{V + 1,} blurred image data applying the set to approximate calculation of D _2V and the blurred image data D _{2V + 1,} to apply the set to approximate calculation of the unsharp image data D _3V and the blurred image data D _{3V + 1,} blurred image data D _4V and blur The equivalent transformation operation is applied to the pair with the image data D _{4V + 1} .

  The predicted values are obtained by adding the results of the respective approximation calculations and the equivalent conversion calculations.

  In the example of the operation shown in FIG. 11, in the integrated expression f (i), f (0) is an approximate operation, f (1) is an equivalent conversion operation, f (2) is an approximate operation, Since f (3) is an approximate operation and f (4) is an equivalent conversion operation, index S (0) is 0, index S (1) is 2, and index S (2) is 3.

  Thus, the predicted value is obtained according to the combination of the approximate calculation and the equivalent conversion calculation. Specifically, the number of terms in the predicted value operation (total number of approximate operations and equivalent conversion operations), the number of terms in the approximate operation, the number of terms in the equivalent conversion operation, and the approximate operation in the formula for the calculation of the predicted value By changing the position of the term, different combinations of the approximate calculation and the equivalent conversion calculation are obtained, and a predicted value is obtained for each different combination.

  That is, a predicted value is obtained for each combination in which the total number N of arithmetic units, the number W of approximate arithmetic units, the number X of equivalent converters, and the index S (w) are different.

  Hereinafter, a specific approximate arithmetic unit determined by a total number N of specific arithmetic units, a specific number of approximate arithmetic units W, a specific equivalent converter number X, and a specific index S (w) A combination with an equivalent converter, that is, a combination of a specific approximate operation and an equivalent conversion operation is also referred to as a prediction pattern. One predicted value is obtained from a combination of an approximate calculator and an equivalent calculator of one prediction pattern (a combination of an approximate calculation and an equivalent conversion calculation).

  The blur removal calculation unit 14 in FIG. 1 calculates predicted values for all the different combinations of the approximate calculation and the equivalent conversion calculation.

  For example, the maximum value of the total number N of arithmetic units is determined in advance, and for each of the total number N of arithmetic units in the range of 1 or more and the maximum value or less, the number W of one or more approximate arithmetic units and the equivalent converter of 0 or more From the number X and the respective indexes S (w), all the different combinations of the approximate calculation and the equivalent conversion calculation are obtained.

  The blur removal calculation unit 14 calculates predicted values for all the different combinations of the approximate calculation and the equivalent conversion calculation obtained in this way.

  The calculation result comparison / selection unit 15 selects a more probable prediction value from one or a plurality of prediction values for all the different combinations of the approximate calculation and the equivalent conversion calculation supplied from the blur removal calculation unit 14. The pixel value is obtained from the selected predicted value.

  Here, selection of a predicted value in the calculation result comparison / selection unit 15 will be described.

  Each of the plurality of predicted values obtained according to the combination of the approximate calculation and the equivalent conversion calculation includes the error E with the same gain as described above. The magnitude of the error E in each predicted value depends on the shape of the waveform in the image represented by the blurred image data D used in the calculation of the predicted value.

  There is no bias in the edge of the image represented by the blurred image data D used in the calculation of the predicted value, and only the difference in how each of the approximate computing units is arranged (the value of the index S (w) is different) Only), ideally, the error E of the predicted value is scattered symmetrically around 0.

  The calculation result comparison / selection unit 15 selects, for example, a prediction value close to the median value as a pixel value from a plurality of prediction values supplied from the blur removal calculation unit 14. By doing so, ringing can be removed more robustly from the blur removal processing result image data K.

  In the following, the maximum likelihood values for a plurality of predicted values will be described more specifically.

In the prediction formula of Formula (12), for example, the number of terms in the prediction formula is 20 (the total number N of computing units is 20), and the number of terms in the approximate computation is 10 (the number of approximate computing units is 10). Then, there are ₂₀ C ₁₀ combinations of prediction formulas.

_Assuming that the prediction error E _total follows a Gaussian distribution G (0, σ), what number of predicted values is closest to the true value of real-world image data J when R predictive values are selected at random? Is obtained by the equations (22) to (24). Expression (22) represents an operation for calculating the probability that the largest predicted value is closest to the true value of the real world image data J, and Expression (23) represents that the second largest predicted value is the true value of the real world image data J. The calculation for calculating the probability closest to the value is shown, and Expression (24) indicates the calculation for calculating the probability that the Kth largest predicted value is closest to the true value of the real world image data J.

  From the equations (22) to (24), it can be seen that the median value has the maximum probability close to the true value of the real world image data J.

  For example, when the number of terms in the prediction equation is 20 (the total number N of arithmetic units is 20) and the number of approximate calculation terms is 10 (the number of approximate arithmetic units is 10), the calculation result comparison / selection unit 15 Selects 10 prediction values close to the median value from the plurality of prediction values, and outputs the average value of the selected prediction values as a final pixel value from which motion blur is removed.

  Next, the configuration of the blur removal calculating unit 14 will be described with reference to FIGS. 12 and 13.

  FIG. 12 is a block diagram illustrating a configuration of the blur removal calculating unit 14. The blur removal calculation unit 14 includes calculation units 31-1 to 31-Z.

  The number Z of the calculation units 31-1 to 31-Z is the total number of different combinations of approximate calculation and equivalent conversion calculation, that is, the total number of prediction patterns that are combinations of approximate calculation and equivalent conversion calculation. be equivalent to.

  The calculation unit 31-1 is a combination of an approximate calculation according to one prediction pattern and an equivalent conversion calculation based on the motion amount V supplied from the blur removal parameter generation unit 13 and the pixel sequence supplied from the affine transformation unit 12. Calculate the predicted value.

  The calculation unit 31-2 responds to one prediction pattern different from the prediction pattern in the calculation unit 31-1 from the motion amount V supplied from the blur removal parameter generation unit 13 and the pixel string supplied from the affine transformation unit 12. The predicted value is calculated by a combination of the approximate calculation and the equivalent conversion calculation.

  The calculation units 31-3 to 31-Z respectively calculate the calculation unit 31-1 or the calculation unit 31 from the motion amount V supplied from the blur removal parameter generation unit 13 and the pixel sequence supplied from the affine transformation unit 12. -2 is a prediction pattern different from the prediction pattern in -2, and a prediction value is calculated by a combination of an approximation calculation and an equivalent conversion calculation according to one different prediction pattern.

  The calculation units 31-1 to 31-Z each output the calculated predicted value. That is, the same number of predicted values as the number of predicted patterns Z are output from the blur removal calculating unit 14.

  Hereinafter, when it is not necessary to individually distinguish the calculation units 31-1 to 31 -Z, they are simply referred to as calculation units 31.

  FIG. 13 is a block diagram illustrating a configuration of the calculation unit 31.

  The calculation unit 31 includes variable shift registers 51-1 to 51-N, a switch 52, an equivalent conversion calculation unit 53-1 to an equivalent conversion calculation unit 53-X, and an approximate calculation unit 54-1 to an approximate calculation unit 54-. W and a predicted value calculation unit 55.

  The variable shift register 51-1 shifts the pixel row, which is one row of pixels in the direction of motion, supplied from the affine transformation unit 12, according to the amount of motion V. The two adjacent pixels on the left side are supplied to the switch 52, and the shifted pixel column is supplied to the variable shift register 51-2.

For example, when the pixel sequence of the blurred image data D _{0 to the} blurred image data D ₁₅ shown in FIG. 2 is supplied from the affine transformation unit 12, it is variable when trying to obtain the predicted value of the real world image data J _0. The shift register 51-1 shifts the pixel columns of the blurred image data D _{0 to the} blurred image data D ₁₅ to the left by one pixel, and two adjacent blurred image data D ₁ and blurred pixels on the left side of the shifted pixel column. The image data D ₂ is supplied to the switch 52. The variable shift register 51-1 supplies the entire shifted pixel column to the variable shift register 51-2.

  The variable shift register 51-2 shifts the pixel row that is supplied from the variable shift register 51-1, which is a row of pixels in the direction of motion, according to the amount of motion V, and is shifted. Two adjacent pixels on the left side of the column are supplied to switch 52. The variable shift register 51-2 supplies the entire shifted pixel column to the variable shift register 51-3 (not shown).

For example, when the pixel sequence of the blurred image data D _{0 to the} blurred image data D ₁₅ shown in FIG. 2 is supplied from the affine transformation unit 12, it is variable when trying to obtain the predicted value of the real world image data J _0. The shift register 51-2 shifts the pixel columns of the blurred image data D _{0 to the} blurred image data D ₁₅ shifted in the variable shift register 51-1 further to the left according to the motion amount V of 3. After shifting by 3 pixels, two adjacent blur image data D ₄ and blur image data D ₅ on the left side of the shifted pixel column are supplied to the switch 52. The variable shift register 51-2 supplies the entire shifted pixel column to the variable shift register 51-3.

  Similarly, each of the variable shift registers 51-3 to 51-N is shifted by shifting a pixel column that is a column of one pixel in the direction of motion according to the motion amount V. Two adjacent pixels on the left side of the pixel column are supplied to the switch 52.

  The variable shift register 51-3 through variable shift register 51- (N-1) (not shown) are respectively shifted from the entire shifted pixel column to the variable shift register 51-4 (not shown) through variable shift register 51-. To each of N.

For example, the variable shift registers 51-1 through variable shift register 51-N, respectively, as shown in FIG. 6, the blurred image data D ₁ and blurred image data D _2, the blurred image data D ₄ and the unsharp image data D ₅ The blurred image data D ₇ and the blurred image data D ₈ , the blurred image data D ₁₀ and the blurred image data D ₁₁ , the blurred image data D ₁₃ and the blurred image data D ₁₄ are extracted and supplied to the switch 52.

  That is, each of the variable shift registers 51-1 to 51-N is a set of the blurred image data D and the blurred image data D adjacent to the blurred image data D in the direction of motion, A set that is separated from other sets by the amount of motion V as a unit is extracted and supplied to the switch 52.

  The switch 52 is a combination of the blurred image data D supplied from each of the variable shift registers 51-1 to 51-N and the blurred image data D adjacent to the blurred image data D in the direction of motion. The equivalent conversion calculation unit 53-1 through equivalent conversion calculation unit 53 -X and the approximate calculation unit 54-1 through approximate calculation unit 54 -W are supplied.

  For example, the switch 52 assigns each set of adjacent blurred image data D to all of the equivalent conversion calculation units 53-1 to 53-X and the approximate calculation units 54-1 to 54-W. Supply.

  For example, the switch 52 converts one set of the adjacent blurred image data D to any one of the equivalent conversion calculation unit 53-1 to the equivalent conversion calculation unit 53-X and the approximate calculation unit 54-1 to the approximate calculation unit 54-W. Supply to one.

  The equivalent conversion calculation unit 53-1 corresponds to one equivalent converter and applies the equivalent conversion calculation to one set of adjacent blurred image data D supplied from the switch 52.

  The equivalent conversion calculation unit 53-2 to the equivalent conversion calculation unit 53-X correspond to one equivalent converter, and perform an equivalent conversion calculation on one set of adjacent blurred image data D supplied from the switch 52. Apply.

  The equivalent conversion calculation unit 53-1 to the equivalent conversion calculation unit 53-X respectively supply the calculation result of the equivalent conversion calculation to the predicted value calculation unit 55.

  The approximate calculation unit 54-1 corresponds to one approximate calculator and applies the approximate calculation to one set of adjacent blurred image data D supplied from the switch 52.

  Each of the approximate calculation units 54-2 to 54-W corresponds to one approximate calculation unit, and applies the approximate calculation to one set of adjacent blurred image data D supplied from the switch 52.

  The approximate calculation unit 54-1 to the approximate calculation unit 54-W supply the calculation result of the approximate calculation to the predicted value calculation unit 55, respectively.

  The predicted value calculation unit 55 calculates the predicted value by using the calculation result of the equivalent conversion calculation and the calculation result of the approximate calculation for each term of the calculation formula determined by the prediction pattern. For example, the predicted value calculation unit 55 adds the calculation results supplied from the equivalent conversion calculation unit 53-1 to the equivalent conversion calculation unit 53-X and the approximate calculation unit 54-2 to the approximate calculation unit 54-W, and performs prediction. The value is calculated and the calculated predicted value is output.

  Hereinafter, when it is not necessary to distinguish the variable shift registers 51-1 to 51-N from each other, they are simply referred to as variable shift registers 51. When it is not necessary to individually distinguish the equivalent conversion calculation unit 53-1 to the equivalent conversion calculation unit 53 -X, they are simply referred to as an equivalent conversion calculation unit 53. Further, when it is not necessary to distinguish the approximate calculation unit 54-1 to the approximate calculation unit 54-W individually, they are simply referred to as the approximate calculation unit 54.

  FIG. 14 is a block diagram illustrating a configuration of the calculation result comparison / selection unit 15. The calculation result comparison / selection unit 15 includes a sorting unit 71, a selection unit 72, and an average value calculation unit 73.

  The sorting unit 71 sorts the Z predicted values supplied from the computing units 31-1 to 31-Z according to their sizes. The sorting unit 71 supplies the predicted values sorted in order of size to the selection unit 72.

  The selecting unit 72 selects a predetermined number of predicted values close to the center in the sorted order from the predicted values sorted by the size supplied from the sorting unit 71, and the selected predicted values are averaged. It supplies to the calculating part 73.

  Alternatively, the selecting unit 72 selects a predetermined number of predicted values close to the median from the predicted values sorted by the size supplied from the sorting unit 71, and the selected predicted values are averaged calculating units. 73.

  Note that the selection unit 72 may select a predicted value based on the average value of all predicted values or the average value of the maximum predicted value and the minimum predicted value.

  Also, which of the center, median, average value of all predicted values, and average value of the maximum and minimum predicted values in the sorted order is used as the reference, the motion blur is based on each. An image from which the image is removed may be displayed and selected by the user.

  The average value calculation unit 73 obtains an average value of the predicted values selected by the selection unit 72 and outputs the average value as a pixel value.

  Next, a process for removing motion blur will be described with reference to a flowchart.

  FIG. 15 is a flowchart for explaining an example of blur removal processing. In step S11, the blur removal parameter generation unit 13 acquires the motion amount V of the input image data from the motion detection unit 11 or a motion vector supplied from the outside.

  In step S <b> 12, the blur removal parameter generation unit 13 acquires the angle of the direction of motion of the input image data from the motion detection unit 11 or a motion vector supplied from the outside.

  In step S13, the affine transformation unit 12 refers to the motion amount V and the angle of the motion direction, and affine transforms the image data so that the motion direction is the horizontal direction of the image. The affine transformation unit 12 extracts a pixel row that is a row of pixels in the horizontal direction of the image from the affine transformed image data, and supplies the extracted pixel row to the blur removal calculation unit 14.

  In step S14, the blur removal calculation unit 14 performs a blur removal calculation process. Details of the blur removal calculation process will be described later with reference to the flowchart of FIG.

  In step S15, the blur removal calculating unit 14 determines whether or not the prediction values based on all the prediction patterns have been obtained. If it is determined that the prediction values based on all the prediction patterns have not been obtained, the process returns to step S14. Repeat the removal operation.

  If it is determined in step S15 that the prediction values based on all the prediction patterns have been obtained, the procedure proceeds to step S16. In step S16, the sorting unit 71 of the operation result comparison / selection unit 15 performs the blur removal operation in step S14. The predicted values obtained by the processing are sorted in the order of their sizes.

  In step S <b> 17, the selection unit 72 of the calculation result comparison / selection unit 15 selects a predetermined prediction value from the sorted prediction values. For example, the selection unit 72 selects a predetermined number of predicted values that are close to the center in the sorted order. For example, the selection unit 72 selects a predetermined number of predicted values close to the median value.

  In step S18, the average value calculation unit 73 of the calculation result comparison / selection unit 15 calculates the average value of the selected prediction values as the pixel value.

  In step S19, the blur removal calculating unit 14 determines whether or not the process has been completed for the entire screen. If it is determined that the process has not been completed for the entire screen, the procedure returns to step S11, and the next pixel is determined. The above process is repeated.

  If it is determined in step S19 that the process has been completed for all screens, the blur removal process ends.

  FIG. 16 is a flowchart for explaining an example of details of the blur removal calculation process corresponding to step S14 of FIG. In step S31, the variable shift register 51-1 extracts the pixel at the left end of the pixel row supplied from the affine transformation unit 12 and the pixel adjacent thereto. Further, the variable shift register 51-2 includes a reference pixel that is a pixel on the right side by the same number of pixels as the motion amount V in units of pixels from the left end of the pixel row, and an adjacent pixel that is adjacent to the right side of the reference pixel. Extract pixels.

  In step S <b> 32, the variable shift register 51-3 sets, as a new reference pixel, a pixel on the right side of the previous reference pixel by the same number of pixels as the motion amount V in units of pixels.

  In step S <b> 33, the blur removal calculation unit 14 determines whether the new reference pixel and the pixel adjacent to the right side thereof are in the pixel row supplied from the affine transformation unit 12.

  If it is determined in step S33 that the new reference pixel and the pixel adjacent to the right side thereof are in the pixel column, the process proceeds to step S34, and the variable shift register 51-3 is adjacent to the reference pixel and the right side of the reference pixel. The adjacent pixels that are pixels to be extracted are extracted. After step S33, the procedure returns to step S32 and repeats the above-described processing.

  Thereafter, when the processes in steps S32 to S34 are repeated, the processes in steps S32 and S34 are performed in the order of the variable shift register 51-4 to the variable shift register 51-N, in the order of the variable shift register 51-4 to the variable shift register. Executed by each of 51-N.

  If it is determined in step S33 that the new reference pixel and the pixel adjacent to the right side thereof are not in the pixel column, the procedure proceeds to step S35, and the equivalent conversion calculation unit 53-1 is supplied via the switch 52. In addition, an equivalent conversion operation is applied to a set of reference pixels and adjacent pixels.

  In step S36, the blur removal calculation unit 14 determines whether or not the equivalent conversion calculation is applied to all the reference pixels and the adjacent pixels, and does not apply the equivalent conversion calculation to all the reference pixels and the adjacent pixels. If it is determined, the process returns to step S35 and the equivalent conversion calculation is repeated.

  When the equivalent conversion calculation in step S35 is repeated, the equivalent conversion calculation in step S35 is performed in the order of the equivalent conversion calculation unit 53-2 to equivalent conversion calculation unit 53-X, in the order of the equivalent conversion calculation unit 53-2 to equivalent conversion. It is executed by each of the arithmetic units 53-X.

  If it is determined in step S36 that the equivalent conversion calculation has been applied to all reference pixels and adjacent pixels, the procedure proceeds to step S37, and the approximate calculation unit 54-1 is supplied with the 1 An approximation operation is applied to a set of reference pixels and adjacent pixels.

  In step S38, the blur removal calculating unit 14 determines whether or not the approximation calculation is applied to all the reference pixels and the adjacent pixels, and determines that the approximation calculation is not applied to all the reference pixels and the adjacent pixels. If so, the process returns to step S37 to repeat the approximate calculation.

  When the approximate calculation in step S37 is repeated, the approximate calculation in step S37 is performed by the approximate calculation unit 54-2 through the approximate calculation unit 54-W in the order of the approximate calculation unit 54-2 through the approximate calculation unit 54-W. Executed by each.

  If it is determined in step S38 that the approximation calculation is applied to all the reference pixels and adjacent pixels, the procedure proceeds to step S39, and in step S39, the predicted value calculation unit 55 calculates the result of the equivalent conversion calculation and the approximation calculation. The predicted value is calculated using the result of the above in a predetermined term of the arithmetic expression, and the process of the blur removal calculation ends. That is, in step S39, the predicted value calculation unit 55 calculates the predicted value using the result of the equivalent conversion calculation and the result of the approximate calculation for each term determined by the prediction pattern.

  Note that the flowchart of FIG. 16 illustrates the process of blur removal calculation when moving from left to right. The process of blur removal calculation when moving from right to left is the right side in steps S31 to S34. Except for the difference that “is left”, it is the same as the case shown in the flowchart of FIG.

  Next, another example of blur removal calculation processing will be described.

  FIG. 17 is a flowchart illustrating another example of the details of the blur removal calculation process corresponding to step S14 of FIG. In step S51, the blur removal calculating unit 14 sets 0 to the index q. The index q is a variable for specifying the prediction pattern.

  In step S52, the blur removal calculating unit 14 sets 0 to the index k and the index i, respectively. The index k is a variable for specifying the argument of the index S.

In step S53, the blur removal calculating unit 14 sets 0 to the predicted value K in the predicted pattern _Pq . In step S54, the variable shift register 51 acquires the adjacent pixel D _{(i × V)} and pixel D _{(i × V + 1)} . That is, the variable shift register 51 acquires the adjacent blurred image data D _{(i × V)} and blurred image data D _{(i × V + 1)} .

In step S55, the blur removal calculating unit 14 determines whether or not the index i is equal to the index s (k) of the approximate calculator in the prediction pattern _Pq . If it is determined in step S55 that the index i is equal to the index s (k) of the approximate calculator in the prediction pattern P _q , the procedure proceeds to step S56, and the approximate calculation unit 54 determines the adjacent pixel D _{(i × An} approximate operation r (i × V) is applied to _V) and the pixel D _{(i × V + 1)} .

  In step S57, the blur removal calculating unit 14 increments the index k by 1, and the procedure proceeds to step S59.

On the other hand, if it is determined in step S55 that the index i is not equal to the index s (k) of the approximate calculator in the prediction pattern P _q , the procedure proceeds to step S58, and the equivalent transformation calculation unit 53 determines that the adjacent pixel The equivalent transformation operation e (i × V) is applied to D _{(i × V)} and the pixel D _{(i × V + 1)} . After step S58, the procedure proceeds to step S59.

In step S59, the predicted value calculation unit 55 adds the result of the calculation in step S56 or step S58 to the predicted value K in the predicted pattern _Pq .

  In step S60, the blur removal calculating unit 14 determines whether or not the index i is equal to the sum of the number of approximate calculators W and the number of equivalent converters X, and the index i is the number of approximate calculators W. If it is determined that the sum is not equal to the sum of the number of equivalent converters X, the procedure proceeds to step S61, and the blur removal calculating unit 14 increments the index i by one. After step S61, the procedure returns to step S54 and repeats the above-described processing.

  If it is determined in step S60 that the index i is equal to the sum of the number W of approximate arithmetic units and the number X of equivalent converters, the procedure proceeds to step S62, and the blur removal calculating unit 14 predicts that the index q is predicted. It is determined whether or not it is equal to the number of patterns.

  If it is determined in step S62 that the index q is not equal to the number of prediction patterns, the procedure proceeds to step S63, and the blur removal calculating unit 14 increments the index q by 1. After step S63, the procedure returns to step S52 and repeats the above-described processing.

  If it is determined in step S62 that the index q is equal to the number of prediction patterns, the blur removal calculation process ends.

  Next, a case where a prediction pattern is selected according to input image data and a prediction value is calculated using the selected prediction pattern will be described.

  FIG. 18 is a block diagram showing another example of the configuration of the image processing apparatus according to the embodiment of the present invention. In FIG. 18, the same parts as those shown in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

  The image processing apparatus in FIG. 18 includes a motion detection unit 11, an affine transformation unit 12, a blur removal parameter generation unit 13, a calculation result comparison / selection unit 15, an inverse affine transformation unit 16, a blur removal calculation unit 101, and a prediction pattern selection unit 102. Consists of

  The affine transformation unit 12 supplies the pixel sequence to the blur removal calculation unit 101 and supplies the image data subjected to the affine transformation to the prediction pattern selection unit 102.

  The blur removal calculation unit 101 is a pixel pattern supplied from the affine transformation unit 12 with a prediction pattern corresponding to an equivalent transformation calculation assignment instruction signal for instructing assignment of equivalent transformation computation supplied from the prediction pattern selection unit 102. Then, the predicted value of the pixel value not including motion blur is calculated.

  The prediction pattern selection unit 102 generates an equivalent conversion calculation assignment instruction signal from the image data supplied from the affine conversion unit 12 and supplies the equivalent conversion calculation assignment instruction signal to the blur removal calculation unit 101.

  The predicted pattern selection unit 102 includes an edge detection unit 111. The edge detection unit 111 is configured by a Sobel filter, a Prewitt filter, a Laplacian filter, or the like, and detects an edge of an image from image data.

  The prediction pattern selection unit 102 generates an equivalent conversion calculation assignment instruction signal so as to assign an equivalent conversion calculation to the edge pixels detected by the edge detection unit 111.

  FIG. 19 is a block diagram illustrating a configuration of the blur removal calculating unit 101. The blur removal calculation unit 101 includes calculation units 131-1 to 131-T.

  The number T of the calculation units 131-1 to 131-T is equal to the number of prediction patterns to be selected.

  The calculation unit 131-1 is based on the equivalent conversion calculation assignment instruction signal supplied from the prediction pattern selection unit 102, the motion amount V supplied from the blur removal parameter generation unit 13, and the pixel sequence supplied from the affine conversion unit 12. A predicted value is calculated by a combination of an approximate calculation according to one prediction pattern and an equivalent conversion calculation.

  The calculation unit 131-2 includes the equivalent conversion calculation assignment instruction signal supplied from the prediction pattern selection unit 102, the motion amount V supplied from the blur removal parameter generation unit 13, and the pixel sequence supplied from the affine conversion unit 12. A predicted value is calculated by a combination of an approximate calculation according to one prediction pattern different from the prediction pattern in the calculation unit 131-1 and an equivalent conversion calculation.

  The calculation units 131-3 to 131-T respectively include an equivalent conversion calculation assignment instruction signal supplied from the prediction pattern selection unit 102, a motion amount V supplied from the blur removal parameter generation unit 13, and an affine conversion unit 12. Is a prediction pattern different from the prediction pattern in the calculation unit 131-1 or the calculation unit 131-2, and is a combination of an approximation calculation and an equivalent conversion calculation according to one different prediction pattern, Calculate the predicted value.

  The calculation units 131-1 to 131-T each output the calculated predicted value. That is, the same number of prediction values as the number T of prediction patterns to be selected are output from the blur removal calculation unit 101.

  Hereinafter, when it is not necessary to individually distinguish the calculation units 131-1 to 131 -T, they are simply referred to as calculation units 131.

  FIG. 20 is a block diagram illustrating a configuration of the calculation unit 131.

  The calculation unit 131 includes a variable shift register 151-1 to variable shift register 151-N, a switch 152, an equivalent conversion calculation unit 153-1 to an equivalent conversion calculation unit 153-X, and an approximate calculation unit 154-1 to an approximate calculation unit 154. W and a predicted value calculation unit 155.

  Each of the variable shift registers 151-1 to 151 -N, like the variable shift register 51, is a pixel column that is supplied from the affine transformation unit 12 and that is a column of one pixel in the direction of motion. The shift is performed in accordance with the amount of motion V, and two adjacent pixels on the left side of the shifted pixel column are supplied to the switch 152, and the shifted pixel column is supplied to the subsequent stage.

  That is, each of the variable shift registers 151-1 to 151 -N is a set of the blurred image data D and the blurred image data D adjacent to the blurred image data D in the direction of motion, A group separated from the other groups by the amount of motion V as a unit is extracted and supplied to the switch 152.

  In response to the equivalent conversion calculation assignment instruction signal supplied from the prediction pattern selection unit 102, the switch 152 includes the blurred image data D supplied from each of the variable shift registers 151-1 to 151-N, Pairs of the blur image data D adjacent to the blur image data D in the direction of motion are transferred to the equivalent conversion operation units 153-1 to 153-X and the approximate operation units 154-1 to 154-W. Supply.

  For example, when the equivalent conversion calculation assignment instruction signal indicates that one set of adjacent blur image data D is an edge pixel, the switch 152 converts the one set of blur image data D to the equivalent conversion calculation unit 153- 1 to any one of equivalent conversion operation units 153 -X. For example, when the equivalent conversion calculation assignment instruction signal indicates that one set of adjacent blur image data D is not an edge pixel, the switch 152 converts the one set of blur image data D to the equivalent conversion calculation unit 153-1. Thru | or the equivalent conversion calculating part 153-X and the approximate calculating part 154-1 thru | or the approximate calculating part 154-W, it supplies to any one.

  The equivalent conversion calculation unit 153-1 to the equivalent conversion calculation unit 153-X correspond to one equivalent converter, are configured in the same manner as the equivalent conversion calculation unit 53, and are adjacent blur images supplied from the switch 152. The equivalent conversion operation is applied to one set of data D.

  The equivalent conversion calculation unit 153-1 to the equivalent conversion calculation unit 153-X supply the calculation result of the equivalent conversion calculation to the predicted value calculation unit 155, respectively.

  Each of the approximate calculation units 154-1 to 154-W corresponds to one approximate calculation unit, and is configured in the same manner as the approximate calculation unit 54. An approximation operation is applied to one set.

  The approximate calculation units 154-1 to 154-W supply the calculation results of the approximate calculation to the predicted value calculation unit 155, respectively.

  The predicted value calculation unit 155 is configured to calculate the calculation result of the equivalent conversion calculation and the calculation result of the approximation calculation according to the equivalent conversion calculation assignment instruction signal supplied from the prediction pattern selection unit 102 according to the prediction pattern. The predicted value is calculated using this term.

  Hereinafter, the variable shift registers 151-1 to 151 -N are simply referred to as variable shift registers 151 when it is not necessary to distinguish them individually. When it is not necessary to individually distinguish the equivalent conversion calculation unit 153-1 to the equivalent conversion calculation unit 153-X, they are simply referred to as an equivalent conversion calculation unit 153. Further, when it is not necessary to distinguish the approximate calculation units 154-1 to 154-W from each other, they are simply referred to as approximate calculation units 154.

  FIG. 21 is a flowchart illustrating another example of the blur removal process. Steps S111 to S113 are the same as steps S11 to S13 in FIG.

  In step S114, the prediction pattern selection unit 102 selects a prediction pattern. For example, the prediction pattern selection unit 102 detects an edge of the image from the image data, and selects a prediction pattern in which an equivalent conversion operation is assigned to the pixel of the edge.

  In step S115, the blur removal calculation unit 101 performs a blur removal calculation process. Details of the blur removal calculation process will be described later with reference to the flowchart of FIG.

  In step S116, the blur removal calculating unit 101 determines whether or not the prediction values based on all the selected prediction patterns have been obtained, and when it is determined that the prediction values based on all the selected prediction patterns have not been obtained. Returning to step S115, the blur removal calculation process is repeated.

  If it is determined in step S116 that the predicted values for all the selected prediction patterns have been obtained, the procedure proceeds to step S117.

  Steps S117 to S120 are the same as steps S16 to S19 in FIG.

  FIG. 22 is a flowchart for explaining an example of details of the blur removal calculation process corresponding to step S115 of FIG. Steps S131 to S134 are respectively executed by the variable shift register 151 or the blur removal calculating unit 101 and are the same as steps S31 to S34 in FIG.

  In step S <b> 135, the equivalent conversion calculation unit 153-1 applies the equivalent conversion calculation to a set of reference pixels and adjacent pixels supplied via the switch 152.

  In step S136, the blur removal calculation unit 14 determines whether all equivalent conversion calculation results required in the selected prediction pattern have been obtained, and all of the required prediction patterns are selected. If it is determined that the result of the equivalent conversion operation has not been obtained, the process returns to step S135 and the equivalent conversion operation is repeated.

  When the equivalent conversion calculation in step S135 is repeated, the equivalent conversion calculation in step S135 is performed in the order of equivalent conversion calculation unit 153-2 to equivalent conversion calculation unit 153-X, in order of equivalent conversion calculation unit 153-2 to equivalent conversion. It is executed by each of the calculation units 153 -X.

  If it is determined in step S136 that all equivalent conversion operations required in the selected prediction pattern have been obtained, the procedure proceeds to step S137, and the approximate calculation unit 154-1 passes the switch 152. An approximation operation is applied to the supplied set of reference pixels and adjacent pixels.

  In step S138, the blur removal calculation unit 14 determines whether or not the results of all the approximate calculations required in the selected prediction pattern are obtained, and all the approximations required in the selected prediction pattern. If it is determined that the result of the calculation is not obtained, the process returns to step S137 and the approximate calculation is repeated.

  When the approximate calculation in step S137 is repeated, the approximate calculation in step S137 is performed by the approximate calculation unit 154-2 through the approximate calculation unit 154-W in the order of the approximate calculation unit 154-2 through the approximate calculation unit 154-W. Executed by each.

  If it is determined in step S138 that the results of all approximate calculations required for the selected prediction pattern have been obtained, the procedure proceeds to step S139, and in step S139, the predicted value calculation unit 155 selects the selected prediction pattern. The predicted value is calculated using the result of the equivalent conversion calculation and the result of the approximation calculation for the predetermined term in the calculation expression of the prediction pattern, and the blur removal calculation ends. That is, in step S139, the predicted value calculation unit 155 calculates a predicted value using the result of the equivalent conversion calculation and the result of the approximate calculation for each term determined by the selected prediction pattern.

  Note that the flowchart of FIG. 22 describes the process of blur removal calculation when moving from left to right. The process of blur removal calculation when moving from right to left is the right side in steps S131 to S134. Except for the difference that “is left”, it is the same as the case shown in the flowchart of FIG.

  FIG. 23 is a flowchart for explaining another example of details of the blur removal calculation corresponding to step S115 of FIG. In step S151, the blur removal calculating unit 101 focuses on one of the selected prediction patterns.

  In step S152, the blur removal calculating unit 101 sets 0 to each of the index k and the index i.

In step S153, the blur removal calculating unit 101 sets 0 to the prediction value K in the prediction pattern of interest. In step S154, the variable shift register 151 acquires the adjacent pixel D _{(i × V)} and pixel D _{(i × V + 1)} . That is, the variable shift register 151 acquires the adjacent blurred image data D _{(i × V)} and the blurred image data D _{(i × V + 1)} .

In step S155, the blur removal calculating unit 101 determines whether the index i is equal to the index s (k) of the approximate calculator in the predicted pattern of interest. If it is determined in step S155 that the index i is equal to the index s (k) of the approximate calculator in the predicted pattern of interest, the procedure proceeds to step S156, and the approximate calculator 154 determines that the adjacent pixel D _{(i × An} approximate operation r (i × V) is applied to _V) and the pixel D _{(i × V + 1)} .

  In step S157, the blur removal calculating unit 101 increments the index k by 1, and the procedure proceeds to step S159.

On the other hand, if it is determined in step S155 that the index i is not equal to the index s (k) of the approximate calculator in the predicted pattern of interest, the procedure proceeds to step S158, and the equivalent transformation calculation unit 153 The equivalent transformation operation e (i × V) is applied to D _{(i × V)} and the pixel D _{(i × V + 1)} . After step S158, the procedure proceeds to step S159.

In step S159, the predicted value calculation unit 155 adds the result of the calculation in step S156 or step S158 to the predicted value K in the predicted pattern _Pq .

  In step S160, the blur removal calculating unit 101 determines whether or not the index i is equal to the sum of the number of approximate calculators W and the number of equivalent converters X, and the index i is the number of approximate calculators W. Is determined to be not equal to the sum of the number of equivalent converters X, the procedure proceeds to step S161, and the blur removal calculating unit 101 increments the index i by one. After step S161, the procedure returns to step S154 and repeats the above-described processing.

  If it is determined in step S160 that the index i is equal to the sum of the number W of approximate calculators and the number X of equivalent converters, the procedure proceeds to step S162, and the blur removal calculation unit 101 is still paying attention. It is determined whether there is any prediction pattern.

  If it is determined in step S162 that there is a prediction pattern that has not been noticed yet, the procedure proceeds to step S163, and the blur removal calculation unit 101 pays attention to the next prediction pattern from among the selected prediction patterns. After step S163, the procedure returns to step S152 and repeats the above-described processing.

  If it is determined in step S162 that there is no prediction pattern not focused on, that is, it is determined that the prediction values have been calculated for all of the selected prediction patterns, the blur removal calculation process ends.

  In this way, blurring can be removed by combining two operations of the approximate calculation and the equivalent conversion calculation.

  By changing the combination of the approximate calculation and the equivalent conversion calculation, the calculation for removing the blur can be easily changed.

  When a plurality of combinations of operations are prepared, the accuracy of prediction can be increased by comparing the magnitudes of predicted values of the combinations.

  A combination of a plurality of operations can be equivalent to performing many variations of processing.

  As described above, since the least square method and its coefficients are not used, the blur removal calculation can be easily changed.

  Since an approximation operation is not applied to a large edge on an image, ringing can be reduced in an image from which blur is removed.

  Ringing can be reduced when the pixel value is obtained by taking the median of the predicted value.

により求められるようにした場合には、より簡単に、動きボケを除去することができる。 In this way, when a predetermined calculation is applied to image data, motion blur can be removed. In addition, a set of the first pixel and the second pixel adjacent to the first pixel in the direction of movement is extracted from the other set by a motion amount V in units of pixels, From the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V for the i (i is a positive integer) -th set of N (N is a positive integer) sets, the motion A pixel value from which motion blur has been removed is predicted by subtracting the result obtained by multiplying the value α obtained by subtracting 1 / (N−i) from the amount V and the pixel data of the second pixel. And the coefficient α is

Therefore, motion blur can be removed more easily.

  Note that the present invention can be applied to devices that handle moving images, such as playing, recording, or displaying moving images. For example, a television receiver, a set-top box, a tuner, a video recorder, a video player, a video The present invention can be applied to a camera, a home game machine, a mobile phone, or a portable player.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 24 is a block diagram illustrating an example of a hardware configuration of a computer that executes the series of processes described above according to a program.

  In a computer, a central processing unit (CPU) 201, a read only memory (ROM) 202, and a random access memory (RAM) 203 are connected to each other by a bus 204.

  An input / output interface 205 is further connected to the bus 204. The input / output interface 205 includes an input unit 206 composed of a keyboard, mouse, microphone, etc., an output unit 207 composed of a display, a speaker, etc., a storage unit 208 composed of a hard disk or nonvolatile memory, and a communication unit 209 composed of a network interface. A drive 210 for driving a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 201 loads, for example, the program stored in the storage unit 208 to the RAM 203 via the input / output interface 205 and the bus 204 and executes the program. Is performed.

  The program executed by the computer (CPU 201) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 211 that is a package medium composed of a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the computer by loading the removable medium 211 in the drive 210 and storing it in the storage unit 208 via the input / output interface 205. Further, the program can be installed in a computer by being received by the communication unit 209 via a wired or wireless transmission medium and stored in the storage unit 208. In addition, the program can be installed in the computer in advance by storing the program in the ROM 202 or the storage unit 208 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram which shows the example of a structure of the image processing apparatus of one embodiment of this invention. FIG. 4 is a diagram illustrating a relationship between blurred image data D and real world image data J. Is a diagram illustrating the blurred image data D ₁ value obtained by subtracting the blurred image data D ₂ from. It is a figure explaining the example of an approximation calculation. It is a figure explaining Formula (5). It is a figure explaining the specific example of the calculation of the predicted value by an approximate calculator and an equivalent converter. It is a figure explaining the specific example of an equivalent conversion calculation. It is a figure explaining the specific example of an approximation calculation. It is a figure explaining the view of approximate calculation. It is a figure explaining the specific example of the calculation of the predicted value by an approximate calculator and an equivalent converter. FIG. 10 is a diagram for explaining an example of applying an approximation operation or an equivalent conversion operation to a set of adjacent blurred image data D. It is a block diagram which shows the structure of a blur removal calculating part. It is a block diagram which shows the structure of a calculating part. It is a block diagram which shows the structure of a calculation result comparison selection part. 10 is a flowchart for explaining an example of a blur removal process. It is a flowchart explaining the example of the detail of a process of a blurring removal calculation. 12 is a flowchart for explaining another example of details of the blur removal calculation process. It is a block diagram which shows the other example of a structure of the image processing apparatus of one embodiment of this invention. It is a block diagram which shows the structure of a blur removal calculating part. It is a block diagram which shows the structure of a calculating part. 12 is a flowchart for explaining another example of a blur removal process. 14 is a flowchart for explaining still another example of details of processing for blur removal calculation. 14 is a flowchart for explaining still another example of details of processing for blur removal calculation. It is a block diagram which shows the structural example of the hardware of a computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Motion detection part, 12 Affine transformation part, 13 Blur removal parameter production | generation part, 14 Blur removal calculation part, 15 Computation result comparison selection part, 16 Reverse affine transformation part, 31-1 thru | or 31-Z or 31 calculation part, 51- 1 to 51-N or 51 variable shift register, 52 switch, 53-1 to 53-X or 53 equivalent conversion calculation unit, 54-1 to 54-W or 54 approximation calculation unit, 55 prediction value calculation unit, 71 sort unit , 72 selection unit, 73 average value calculation unit, 101 blur removal calculation unit, 102 prediction pattern selection unit, 111 edge detection unit, 131-1 to 131-T or 131 calculation unit, 151-1 to 151-N or 151 variable Shift register, 152 switch, 153-1 to 153-X or 153 equivalent Conversion calculation unit, 154-1 to 154-W or 154 approximation calculation unit, 155 prediction value calculation unit, 201 CPU, 202 ROM, 203 RAM, 208 storage unit, 211 a removable media

Claims (8)

In an image processing apparatus for processing an image composed of pixel data for each of a predetermined number of pixels acquired by an image sensor having a time integration effect,
Extraction means for extracting a set of a first pixel and a second pixel that is adjacent to the first pixel in the direction of motion and that is separated from the other set by a motion amount V in units of pixels. When,
From the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V for the i (i is a positive integer) -th set of N (N is a positive integer) sets, By subtracting the result obtained by subtracting the value α obtained by subtracting 1 / (N−i) from the motion amount V and the pixel data of the second pixel, the pixel value from which motion blur is removed is obtained. First calculating means for calculating a value for prediction, and
The coefficient α is

An image processing apparatus required by The prediction value calculation means which calculates the predicted value of the pixel value from which motion blur was removed by adding the value calculated by the first calculation means for each of the N sets. Image processing apparatus. The pixel of the second pixel is obtained by multiplying the pixel data of the first pixel by the coefficient β and the motion amount V for the k-th group (k is a positive integer) among the N groups. A second computing means for computing a value for predicting a pixel value from which motion blur is removed by subtracting a result obtained by multiplying the data by a coefficient β and a motion amount V;
The coefficient β is

The image processing apparatus according to claim 1, which is obtained by: Predicted value calculating means for calculating predicted values of pixel values from which motion blur has been removed by adding the values calculated by the first calculating means or the second calculating means for each of the N sets. The image processing apparatus according to claim 3. The image according to claim 1, further comprising: a selecting unit that selects a predicted value close to a median value of the plurality of predicted values among a plurality of predicted values predicted from the values calculated by the plurality of first calculating units. Processing equipment. Sorting means for sorting a plurality of predicted values predicted from values calculated by the plurality of first calculating means;
A selection unit that selects a predetermined number of predicted values that are close to the median of the plurality of predicted values from the plurality of sorted predicted values;
The image processing apparatus according to claim 1, further comprising: an average value calculating unit that calculates an average value of the selected predicted values. In an image processing method for processing an image composed of pixel data for each of a predetermined number of pixels acquired by an image sensor having a time integration effect,
A set of a first pixel and a second pixel that is adjacent to the first pixel in the direction of motion and that is separated from the other set by a motion amount V in units of pixels;
From the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V for the i (i is a positive integer) -th set of N (N is a positive integer) sets, By subtracting the result obtained by subtracting the value α obtained by subtracting 1 / (N−i) from the motion amount V and the pixel data of the second pixel, the pixel value from which motion blur is removed is obtained. Including a step of calculating a value for prediction,
The coefficient α is

Image processing method required by In a program for causing a computer to perform image processing for processing an image made up of pixel data for each of a predetermined number of pixels acquired by an image sensor having a time integration effect,
A set of a first pixel and a second pixel that is adjacent to the first pixel in the direction of motion and that is separated from the other set by a motion amount V in units of pixels;
From the result of multiplying the pixel data of the first pixel by the coefficient α and the motion amount V for the i (i is a positive integer) -th set of N (N is a positive integer) sets, By subtracting the result obtained by subtracting the value α obtained by subtracting 1 / (N−i) from the motion amount V and the pixel data of the second pixel, the pixel value from which motion blur is removed is obtained. Causing the computer to perform image processing including a step of calculating a value for prediction;
The coefficient α is

The program required by.

Images