Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/70—Denoising; Smoothing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/73—Deblurring; Sharpening
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning

Definitions

• Prediction coefficient computing device and method image data computing device and method, program, and recording medium
• the present invention relates to a prediction coefficient calculation device and method, an image data calculation device and method, a program, and a recording medium, and more particularly to a prediction coefficient calculation device and a correction coefficient calculation device capable of correcting image blur more accurately.
• the present invention relates to a method, an image data calculation device and method, a program, and a recording medium.
• the present invention also relates to an image data calculation device and method, a prediction coefficient calculation device and method, a program, and a recording medium that can generate a naturally fluctuating image or calculate its prediction coefficient. .
• Fig. 1 shows an example of such an image. Since the background is in focus, the foreground flower image, which is the original subject, is out of focus.
• the feature of the image is detected, and the model formula for calculating the image with the blur corrected is changed according to the feature of the image.
• faithful correction can be performed at the edge portion and the detail portion.
• Patent Document 2 discloses generating an image in which an image of an object moving to the water surface is shaken in accordance with the shaking of the water surface.
• Patent Document 1 Japanese Patent Laid-Open No. 2005-63097
• Patent Document 2 JP 2006-318388 Disclosure of the invention
• Fig. 2 shows that one class of pixels in which many of the pixels that make up a focused background (landscapes other than flowers and leaves) are classified is 1, and pixels that are classified in other classes are classified as pixels.
• the focus is correct and the background is in focus!
• the numerous pixels that make up the foreground (flowers and leaves) are in focus! / Many of the pixels constituting the are classified into classes. This means that it is difficult to correct blur even if the focus is corrected using the prediction coefficient obtained by class classification from only normal images.
• Patent Document 2 since the technique of Patent Document 2 generates an image reflected on the water surface, the image generated thereby is a distorted image. Therefore, for example, it is an image in which a relatively detailed original state can be confirmed as it is when a person looks at an object in the air from a distance, and the ambient air temperature, It was difficult to generate images that fluctuate naturally due to changes in humidity.
• the present invention has been made in view of such a situation, and makes it possible to correct force S for correcting image blur accurately.
• the present invention also makes it possible to generate an image that fluctuates naturally.
• One aspect of the present invention is a blur adding means for generating student image data by adding blur to parent image data based on blur data of a blur model, and an image for constructing an image prediction tap from the student image data Based on the prediction tap construction means, the parent image data, and the image prediction tap, from the image data corresponding to the student image data, the parent image
• a prediction coefficient calculation device includes prediction coefficient calculation means for calculating a prediction coefficient for generating image data corresponding to data.
• Image class tap construction means for constructing an image class tap from the student image data
• blur data class tap construction means for constructing a blur data class tap from the blur data, the image class tap, and the blur data class tap
• a class classification means for classifying the class of the student image data
• the prediction coefficient calculation means can calculate the prediction coefficient for each of the further classified classes.
• the blur adding unit adds blur to the parent image data with characteristics according to a blur parameter specified by a user, and the prediction coefficient calculation unit further performs the prediction for each blur parameter. Coefficients can be calculated.
• the image processing apparatus further includes blur noise adding means for adding noise to the blur data with characteristics according to a noise parameter specified by a user, the blur adding means based on the blur data to which noise is added.
• the blur is added to the parent image data
• the blur data class tap constructing unit constructs the blur data class tap from the blur data to which noise is added
• the prediction coefficient computing unit further includes a blur parameter for each blur parameter. The above prediction coefficient can be calculated.
• a blur data scaling unit for scaling the blur data based on a scaling parameter specified by a user is further provided, wherein the blur noise adding unit adds noise to the scaled blur data, and
• the prediction coefficient calculation means can further calculate the prediction coefficient for each scaling parameter.
• the image class tap construction means further includes image noise adding means for adding noise to the student image data with characteristics according to an image noise parameter specified by a user, and the image class tap construction means includes the noise added
• image class tap is constructed from student image data
• image prediction tap construction means constructs the image prediction tap from the student image data to which noise has been added
• prediction coefficient calculation means further comprises the image noise parameter. The prediction coefficient can be calculated for each data.
• the image noise adding means is Noise is added to the student image data that has been subjected to the ceiling
• the prediction coefficient calculation means can further calculate the prediction coefficient for each scaling parameter.
• the blur data further includes a blur data prediction tap constructing unit that constructs the blur data prediction tap from the blur data, and the prediction coefficient calculation unit includes the parent image data, the image prediction tap, and Based on the blur data prediction tap, a prediction coefficient for generating image data corresponding to the student image data can be calculated for each of the classified classes.
• the blur data may be data to which noise is added.
• One aspect of the present invention is also a prediction coefficient calculation method of a prediction coefficient calculation device that calculates a prediction coefficient, wherein the blur adding unit adds blur to the parent image data based on the blur data of the blur model.
• Student image data is generated, an image prediction tap construction means constructs an image prediction tap from the student image data, and a prediction coefficient calculation means is based on the parent image data and the image prediction tap, and
• a blur adding step for generating student image data by adding blur to parent image data based on blur data of a blur model, and constructing an image prediction tap from the student image data
• An image prediction tap constructing step for generating image data corresponding to the parent image data from image data corresponding to the student image data
• This program can be recorded on a recording medium.
• Another aspect of the present invention provides a prediction coefficient providing unit that provides a prediction coefficient corresponding to a parameter specified by a user and relating to a blur of image data, and constructs an image prediction tap from the image data.
• Image data calculation means comprising: image prediction tap construction means for performing image data calculation means for calculating image data in which blur is corrected by applying the image prediction tap and the provided prediction coefficient to a prediction calculation formula Device.
• a blur data class tap constructing unit that constructs a blur data class tap from data; and a class classifying unit that classifies the class of the image data based on the image class tap and the blur data class tap;
• the prediction coefficient providing unit may further provide the prediction coefficient corresponding to the classified class.
• the prediction coefficient providing means includes a blur parameter that defines a blur characteristic, a parameter that defines a class based on noise included in the image data, a parameter that defines a class based on noise included in the blur data, Alternatively, the prediction coefficient can be provided based on motion information.
• the prediction coefficient providing means is further a parameter designated by a user and based on a parameter that defines a class based on the scaling of the image data or the blur data! A coefficient can be provided.
• the blur data further includes the blur data prediction tap construction means for constructing the blur data prediction tap from the blur data, and the image data calculation means includes the image prediction tap, the blur data prediction tap, In addition, it is possible to calculate the image data in which the blur is corrected by applying the provided prediction coefficient to the prediction calculation formula.
• the prediction coefficient providing means is a parameter designated by a user, and the image data A prediction coefficient corresponding to a parameter relating to blur is provided
• an image prediction tap construction unit constructs an image prediction tap from the image data
• an image data computation unit includes the image prediction tap and the provided prediction coefficient.
• another aspect of the present invention is a prediction coefficient providing step for providing a prediction coefficient corresponding to a parameter designated by a user and relating to a blur of image data, and an image from the image data.
• Image prediction tap construction step for constructing a prediction tap; and an image data computation step for computing image data in which blur is corrected by applying the image prediction tap and the provided prediction coefficient to a prediction computation expression.
• This is a program that causes a computer to execute.
• This program can be recorded on a recording medium.
• Still another aspect of the present invention is a parameter acquisition unit that acquires a parameter, a noise calculation unit that calculates blur noise of a blur model based on the acquired parameter !, An image data computing device comprising image data computing means for computing image data to which blur model noise is added.
• the image data calculation means can calculate the image data by adding noise to the blurred point spread function.
• the noise calculating means calculates depth data obtained by adding noise to depth data, and the image data calculating means is based on the depth data added with noise! Noise can be added.
• the noise calculation means can calculate the deviation, phase, sharpness of the blur point spread function, or noise that is a combination thereof.
• the noise calculating means calculates the amount of movement, the direction of movement, or a noise S that is a combination of them with a force S.
• the noise calculation means can add noise to the position of the interpolation pixel when calculating the pixel value of the interpolation pixel in the direction of movement.
• the image processing device further includes setting means for setting a processing area, and the image data calculation means can add noise to the set image data of the processing area.
• Still another aspect of the present invention is that, in the image data calculation method of the image data calculation device for calculating image data, the parameter acquisition unit acquires a parameter, and the noise calculation unit acquires the acquired parameter.
• This is an image data calculation method in which the blur noise of the blur model is calculated based on the parameters, and the image data calculation means calculates the image data to which the noise of the blur model is added.
• another aspect of the present invention provides a parameter acquisition step for acquiring a parameter, a noise calculation step for calculating blur noise of a blur model based on the acquired parameter, and the blur
• This is a program for causing a computer to execute a process including an image data calculation step for calculating image data to which model noise is added.
• the blur adding means generates student image data by adding blur to the parent image data based on the blur data of the blur model
• the image prediction tap construction means includes the student An image prediction tap is constructed from the image data
• the prediction coefficient calculation means corresponds to the parent image data from the image data corresponding to the student image data based on the parent image data and the image prediction tap. The prediction coefficient for generating the image data to be calculated is calculated.
• the prediction coefficient providing means provides a prediction coefficient corresponding to a parameter specified by a user and is a parameter designated by a user
• the image prediction tap construction means Then, an image prediction tap is constructed from the image data, and image data calculation means calculates the image data with the blur corrected by applying the image prediction tap and the provided prediction coefficient to a prediction calculation formula.
• the parameter acquisition unit acquires a parameter
• the noise calculation unit calculates a blur noise of the blur model based on the acquired parameter
• Data calculation means calculates image data to which the noise of the blur model is added.
• blurring of an image can be corrected accurately.
• a blurred image and a non-blurred image are classified into the same class, It is suppressed that it is difficult to correct image blur accurately.
• an image that fluctuates naturally can be generated.
• FIG. 1 is a diagram showing an example of a captured image.
• FIG. 2 is a diagram showing a classification result of the image of FIG.
• FIG. 3 is a block diagram showing a configuration of an embodiment of a learning device to which the present invention is applied.
• FIG. 4 is a diagram for explaining the addition of blur.
• FIG. 5 is another diagram for explaining the addition of blur.
• FIG. 6 is a graph showing a function of blur characteristics.
• FIG. 7 is a diagram for explaining a noise addition method.
• FIG. 8 is a flowchart for explaining learning processing of the learning device in FIG. 3;
• FIG. 9 is a block diagram showing a configuration of an embodiment of a prediction device to which the present invention is applied.
• FIG. 10 is a flowchart illustrating a prediction process of the prediction device in FIG.
• FIG. 12 is a flowchart illustrating a learning process of the learning device in FIG.
• FIG. 14 is a block diagram showing the configuration of still another embodiment of the learning device.
• FIG. 15 is a flowchart illustrating a learning process of the learning device in FIG.
• FIG. 16 is a block diagram showing a configuration of still another embodiment of the learning device.
• FIG. 17 is a flowchart illustrating a learning process of the learning device in FIG.
• FIG. 18 is a block diagram showing a configuration of still another embodiment of the prediction device.
• FIG. 19 is a flowchart for describing the prediction processing of FIG.
• FIG. 20 is a block diagram showing a configuration of an embodiment of an image generation device.
• FIG. 21 is a block diagram showing the configuration of another embodiment of the image generation apparatus.
• FIG. 22 A block diagram showing a functional configuration of an embodiment of the noise adding unit of FIG.
• FIG. 23 is a block diagram showing a functional configuration of an embodiment of a blur adding unit in FIG.
• FIG. 24 is a flowchart for explaining image generation processing for out-of-focus noise due to distance.
• FIG. 25 is a diagram illustrating an image generation process.
• FIG. 26 shows a function
• FIG. 27 is a flowchart illustrating an image generation process of defocus noise due to deviation.
• FIG. 28 is a flowchart for describing an image generation process of out-of-focus noise due to a phase.
• FIG. 29 is a diagram for explaining a phase shift of a function.
• FIG. 30 shows a function
• FIG. 32 shows a function
• FIG. 33 is a diagram for explaining imaging by a sensor.
• FIG. 34 is a diagram illustrating the arrangement of pixels.
• FIG. 35 is a diagram for explaining the operation of the detection element.
• FIG. 36 is a diagram of a model in which pixel values of pixels arranged in a row adjacent to each other are expanded in the time direction.
• FIG. 37 is a diagram of a model in which pixel values are expanded in the time direction and the period corresponding to the shatter time is divided.
• FIG. 38 is a diagram of a model in which pixel values are expanded in the time direction and the period corresponding to the shatter time is divided.
• FIG. 39 is a diagram of a model in which pixel values are expanded in the time direction and the period corresponding to the shatter time is divided.
• FIG. 40 is a flowchart illustrating an image generation process of motion blur noise based on a motion amount.
• FIG. 41 is a diagram for explaining an interpolation pixel.
• FIG. 42 is a diagram for explaining an interpolation pixel calculation method.
• FIG. 43 is a flowchart illustrating an image generation process of motion blur noise depending on an angle.
• FIG. 44 is a block diagram showing a configuration of another embodiment of the prediction device.
• FIG. 45 is a block diagram showing a configuration of still another embodiment of the prediction device.
• FIG. 46 is a block diagram showing a configuration of an embodiment of a computer to which the present invention is applied.
• FIG. 3 is a block diagram showing a configuration of an embodiment of the learning device 1 to which the present invention is applied.
• the learning device 1 as the prediction coefficient calculation device in FIG. 3 includes a blur adding unit 11, a noise adding unit 12, a noise adding unit 13, a tap building unit 14, a tap building unit 15, a class classification unit 16, and a tap building unit. 17, Prediction coefficient calculation unit 18, and coefficient memory 19 are used to predict a blur-corrected image that is the same size as the blur-corrected image from the blurred image that is the blurred image by class classification adaptive processing The prediction coefficient used when performing the prediction process is learned.
• the blur adding unit 11 receives parent image data, which is a pixel value of each pixel of the parent image corresponding to the blur corrected image after the prediction process, from a device (not shown).
• the blur adding unit 11 acquires the blur parameter P designated by the user, and based on the depth data z after noise addition supplied from the noise adding unit 12, has characteristics according to the blur parameter P and the parent parameter P. Add blur to the image data.
• the depth data z is the three-dimensional position data of the real world object corresponding to the image, and is calculated by stereo measurement using a plurality of images captured by a plurality of cameras or the like.
• the camera power and the distance for each pixel to the subject are used.
• the distance data for each pixel corresponding to each pixel can be obtained by, for example, the method disclosed in Japanese Patent Application Laid-Open No. 2005-70014.
• the blur adding unit 11 converts the parent image data after blur addition into the pre-prediction processing. This is supplied to the noise adding unit 13 as student image data which is the pixel value of each pixel of the student image corresponding to the blurred image.
• the noise adding unit 12 receives depth data z from a device (not shown).
• the noise adding unit 12 acquires a noise parameter Nz that is a noise parameter to be added to the depth data z designated by the user, and adds noise to the depth data z with characteristics according to the noise parameter Nz. Then, the noise adding unit 12 calculates the depth data z after adding the noise. This is supplied to the blur adding unit 11 and the tap building unit 15.
• the noise adding unit 13 acquires a noise parameter Ni that is specified by the user and is a noise parameter to be added to the student image data.
• the noise addition unit 13 is a noise parameter
• Noise is added to the student image data supplied from the blur adder 11 with characteristics according to Ni. Then, the noise adding unit 13 supplies the student image data after the noise addition to the tap building unit 14 and the tap building unit 17.
• the noise adding unit 12 and the noise adding unit 13 are provided with a force S that can obtain a prediction coefficient in consideration of noise removal from the blurred image, and can be omitted if the noise is not taken into consideration.
• Abbreviated power S Abbreviated power S
• the tap constructing unit 14 sequentially sets the pixels constituting the parent image as the pixel of interest, and extracts some of the pixel values constituting the student image used for classifying the pixel of interest into a class. Build an image class tap from student image data.
• the tap construction unit 14 supplies the image class taps to the class classification unit 16.
• the tap constructing unit 15 constructs a depth class tap from the depth data z by extracting the depth data z of some pixels used to classify the pixel of interest into a class.
• the tap construction unit 15 supplies the depth class tap to the class classification unit 16.
• the class classification unit 16 classifies the pixel of interest into a class based on the image class tap supplied from the tap construction unit 14 and the depth class tap supplied from the tap construction unit 15.
• Class classification is realized by using a feature code calculated from a plurality of data constituting a class tap as a classification code.
• ADRC Adaptive Dynamic Range Coding
• ADRC Adaptive Dynamic Range Coding
• the pixel value constituting the image class tap and the depth data z constituting the depth class tap are each subjected to ADRC processing, and the class of the pixel of interest is determined according to the two AD RC codes obtained as a result. Is determined.
• Multiple of And a plurality of pixel values as image class taps are re-quantized to K bits based on the dynamic range DR. That is, from each pixel value as an image class tap, the minimum value MIN is subtracted, and the subtracted value is divided (quantized) by DR / 2 K.
• the K-bit pixel value ADRC code in which the K-bit pixel values as the image cluster obtained as described above are arranged in a predetermined order is used.
• each pixel value as the image class tap is obtained by subtracting the minimum value MIN and then the maximum value MAX and the minimum value MIN. Is divided by 1/2 of the difference (rounded down), so that each pixel value is 1 bit (binarized).
• a bit string in which the 1-bit pixel values are arranged in a predetermined order is used as an ADRC code.
• a bit string in which depth data z of K-bit pixels as a depth class tap are arranged in a predetermined order is used as an ADRC code.
• the method of classifying based on the image class tap and the method of classifying based on the depth class tap may be different.
• the above-mentioned ADRC is adopted as a method for classification based on image class taps, and the depth data z constituting the depth class taps is smoothed not as ADRC as a method for classification based on depth class taps.
• a method for classifying into classes and a method for classifying into classes by edges in pixels corresponding to the depth data z constituting the depth class tap may be adopted.
• the sum of all the depth data z composing the depth class tap is the number of pixels corresponding to the depth data z composing the depth class tap.
• the value obtained by division and multiplication by a predetermined constant is used as the class code, and the class is determined according to the class code.
• the difference between the depth data z of adjacent pixels is calculated from the depth data z constituting the depth class tap, and based on the difference.
• the edge position is recognized.
• a template indicating the position of the edge is selected from templates prepared in advance, the number of the template is set as a class code, and the class is determined according to the class code.
• the class classification unit 16 supplies the class in which the target pixel is classified to the prediction coefficient calculation unit 18.
• the class classification unit 16 classifies the class of the pixel of interest based on the depth class tap that is performed only by the image class tap, the blurred image and the non-blurred image are classified into the same class. Can be suppressed.
• the tap constructing unit 17 constructs an image prediction tap from the student image data by extracting some of the pixel values constituting the student image used for predicting the pixel value of the target pixel.
• the tap construction unit 17 supplies the image prediction tap to the prediction coefficient calculation unit 18.
• an image prediction tap an image class tap, or a depth class tap
• an arbitrary pixel value can be selected. Select a pixel value of a target pixel and / or a predetermined pixel around the target pixel. Can do.
• the prediction coefficient calculation unit 18 is supplied to the noise parameter Nz supplied to the noise addition unit 12, the noise parameter Ni supplied to the noise addition unit 13, and the blur addition unit 11 specified by the user. Get the blur parameter P.
• the prediction coefficient calculation unit 18 is based on the parent image data supplied from a device (not shown) and the image prediction tap supplied from the tap construction unit 17, and the class supplied from the class classification unit 16 and the noise parameter Nz, For each noise parameter Ni and blur parameter P, the prediction coefficient is calculated and supplied to the coefficient memory 19 to make E5 self fe 0.
• an image prediction tap is extracted from a blurred image, and the pixel value of the blur-corrected image is obtained by a predetermined prediction calculation using the image prediction tap and the prediction coefficient ( Predict).
• the pixel value y of the pixel of the blur corrected image (hereinafter, referred to as blur correction pixel as appropriate) It is calculated by the formula.
• Equation (1) X constitutes an image prediction tap for the blur correction pixel y.
• the image prediction tap is composed of pixel values X 1, X 2,.
• the pixel value y of the blur correction pixel may be obtained by a higher-order expression of the second order or higher than the linear first-order expression shown in the expression (1). That is, any function can be used as the estimation formula regardless of the linear function or the nonlinear function.
• Equation (3) X is the image prediction value for the blur correction pixel of the kth sample.
• the optimal prediction coefficient w is the sum of the square errors expressed by the following equation E
• K is the pixel value y of the blur correction pixel and the blur correction pixel.
• Equation (7) is a normal equation shown in Equation (8).
• the normal equation of equation (8) can be solved for the prediction coefficient w by using, for example, a sweep-out method (Gauss-Jordan elimination method).
• the prediction coefficient calculation unit 18 solves the normal equation of Equation (8) for each class, noise parameter Nz, noise parameter Ni, and blur parameter P, thereby obtaining an optimal prediction coefficient (this value) , The prediction coefficient that minimizes the sum of squared errors E) w, class, noise parameter Nz
• Equation (1) X and the calculation of Equation (1) are performed to convert the blurred image into a blurred image.
• the coefficient memory 19 stores the prediction coefficient w supplied from the prediction coefficient calculation unit 18.
• the learning device 1 in Fig. 3 can prevent the blurred image and the non-blurred image from being classified into the same class, and therefore the prediction described later with reference to Fig. 9.
• the prediction process is performed using the prediction coefficient w learned for each classified class n
• the blur of the blurred image can be corrected accurately and converted into a high-quality blurred image.
• f represents the focal length of the lens 52
• V represents the distance between the lens 52 and the sensor 53
• L represents the distance between the object 51 and the lens 52.
• the distance L when the in-focus is not generated that is, when the in-focus is set as the depth data ⁇
• the distance L when the in-focus is generated that is, when the in-focus is not acquired
• the difference between the magnitude ⁇ 1 when the blur force S occurs and the magnitude ⁇ 0 when the blur does not occur is expressed by the following equation (11).
• F represents an F number, that is, ⁇ / r.
• Equation (11) when the magnitude ⁇ ⁇ when no blur occurs is 0, the magnitude ⁇ when the object 51 is at a distance d away from the in-focus position is It is expressed by the following formula (12)
• equation (12) is expressed by the following equation (13).
• equation (13) is expressed by equation (14) below.
• the function i (d) is a function that represents an additional characteristic of blur and is represented by the following equation (15).
• i (d) (k / ⁇ ) X d / (d + z0)
• FIG. 6 is a graph showing the function f (d), the function g (d), and the function h (d). As shown in Fig. 6, the function f (d) converges to a certain value as the distance d increases.
• the blur adding unit 11 performs a function f (d), a function g (d), and a function according to a blur parameter P, which is a parameter specified by the user and is used to select a function representing the characteristic of blur addition. Any one of h (d) is selected, and the parent image data is blurred with the characteristics represented by the selected function to generate student image data.
• the blur adding unit 11 generates student image data for each pixel according to the following equation (18).
• Y (x, y) represents the pixel value of the pixel that constitutes the student image, where the x coordinate is x and the y coordinate is y, X (x + k, y + l) is the position where the X coordinate is x + k and the y coordinate is y + 1 (from the pixel position of interest (x, y)). This represents the pixel value of the pixel at a position separated by (k, l).
• WT (k, l) is a blurred point spread function (Gaussian PSF (Point Spread Function)), and is represented by Equation (19) below.
• Equation (19) S (x + k, y + l) is the distance d, the depth data z of the pixel at the position where the x coordinate is x + k and the y coordinate is y + 1 Represents a selected function of the function f (d), the function g (d), and the function h (d) when the depth data ⁇ is subtracted from!
• Equation (18) and Equation (19) from the pixel whose X coordinate is x + k and y coordinate is y + 1, X coordinate is X and y coordinate force is By accumulating the pixel values diffused to the target pixel, the pixel value of the target pixel after adding the blur is obtained.
• the noise parameter Ni is a value from 0 to j. The same applies to B in Fig. 7 described later.
• the noise parameter Ni is a parameter that specifies the amplitude level of noise.
• the amplitude level of the noise added to the student image is increased stepwise. Amplitude level noise is added. That is, as shown in FIG. 7A, in the first method, when the value of the noise parameter Ni is 0, no noise is added to the student image, and the noise As the value of the parameter Ni increases, the amplitude level of noise added to the student image increases. When the value of the noise parameter Ni is j, noise with the maximum amplitude level is added to the student image.
• Equation (23) an equation of R ⁇ mseq [m] represented by the product of the coefficient R and the function mseq [m] that generates a pseudorandom number is used.
• noise parameter Ni is a parameter that specifies the mixing ratio of noise.
• the noise parameter Ni increases, the student image after 100 noises are added, the noise is not added, the number of student images decreases, and the noise is reduced.
• the value of the noise parameter Ni is j, 30 student images without noise and 70 student images with added noise are 100 noises. Generated as a student image after addition.
• the prediction coefficient calculation unit 18 in Fig. 3 calculates the prediction coefficient according to the equation (8) using one parent image and 100 student images as one sample. That is, the prediction coefficient calculation unit 18 solves the normal equation of the following equation (20) for each class, noise parameter Nz, noise parameter Ni, and blur parameter P, thereby obtaining the optimal prediction coefficient w.
• Equation (20) x represents the pixel value of the nth pixel of the qth student image, which constitutes the image prediction tap for the pixel of the kth defocus corrected image.
• Q is the number of student images for one sample, which is 100 in the example of B in Fig. 7.
• the noise adding unit 13 adds noise to the student image by the first method or the second method described above. Although description is omitted, noise addition in the noise adding unit 12 is performed in the same manner. In this case, for example, random noise caused by the imaging device, influence of extraneous light, difference in reflectance of the object surface, measurement noise, and other random noise are added to the depth data I and XX.
• the noise adding unit 12 adds the confusion noise caused by the influence of confusion due to reflection, smoothing due to measurement accuracy, etc., to the depth data z using a function similar to the function representing the characteristic of adding blur. You may do it.
• the learning device 1 in Fig. 3 describes the learning process in which the prediction coefficient w is learned.
• This learning process is started, for example, when the parent image data and the depth data z are input to the learning device 1 in FIG.
• step S1 the noise adding unit 12 acquires the noise parameter Nz specified by the user.
• step S2 the noise adding unit 12 uses the first method and the second method described with reference to FIG. 7 to generate noise in the depth data z with characteristics according to the noise parameter Nz. Is added.
• step S3 the blur adding unit 11 acquires the blur parameter P designated by the user.
• step S4 the blur adding unit 11 applies the parent image data input from a device (not shown) with characteristics according to the blur parameter P based on the noise-added depth data z supplied from the noise adding unit 12. Add blur.
• the blur adding unit 11 selects the function f (d), the function g (d), or the function h (d) according to the blur parameter P.
• the blur adding unit 11 performs pixel value Y (x, y) of the pixel of interest based on the depth data z, that is, according to Equations (18) and (19) in which the selected function is applied to S, that is, The pixel values of the pixels constituting the student image are obtained.
• the blur adding unit 11 supplies the pixel value of each pixel constituting the student image to the noise adding unit 13 as student image data.
• step S5 the noise adding unit 13 acquires the noise parameter Ni designated by the user.
• step S6 the noise adding unit 13 adds noise to the student image data supplied from the blur adding unit 11 with the characteristics according to the noise parameter Ni by the first method and the second method described in FIG.
• the student image data after adding the noise is supplied to the tap construction unit 14 and the tap construction unit 17.
• step S7 the tap constructing unit 14 constructs an image class tap by extracting predetermined ones from the student image data, and supplies the image class tap to the class classifying unit 16.
• step S8 the tap constructing unit 15 constructs a depth class tap by extracting a predetermined one from the depth data z, and supplies the depth class tap to the class classifying unit 16.
• step S9 the class classification unit 16 classifies the target pixel into a class based on the image class tap supplied from the tap construction unit 14 and the depth class tap supplied from the tap construction unit 15. .
• step S10 the tap construction unit 17 constructs an image prediction tap by extracting a predetermined one from the student image data, and supplies the image prediction tap to the prediction coefficient calculation unit 18.
• step S11 the prediction coefficient calculation unit 18 classifies based on the parent image data supplied from a device (not shown) and the image prediction tap supplied from the tap construction unit 17. For each class supplied from class 16 and noise parameter Nz, noise parameter Ni, and blur parameter P, calculate prediction coefficient w according to equation (8) or equation (20) above.
• n is supplied to the coefficient memory 19.
• step S12 the coefficient memory 19 stores the prediction coefficient w supplied from the prediction coefficient computing unit 18, and the process ends.
• FIG. 9 shows a predictor that performs a prediction process using the prediction coefficient w learned by the learning device 1 of FIG.
• the prediction device 81 in Fig. 9 includes a tap construction unit 91, a tap construction unit 92, a class classification unit 93, a coefficient memory 94, a tap construction unit 95, and a prediction calculation unit 96.
• the prediction device 81 in Fig. 9 receives, from an unillustrated device, blurred image data that is a pixel value of each pixel constituting the blurred image and corresponding depth data z.
• the blurred image data is supplied to the tap construction unit 91 and the tap construction unit 95, and the depth data z is supplied to the tap construction unit 92.
• the tap construction unit 91 sequentially uses the pixels constituting the blur-corrected image as the pixel of interest, and is used to classify the pixel of interest into a class.
• An image class tap is constructed from blurred image data by extracting some of the pixel values that make up the image.
• the tap construction unit 91 supplies the image class tap to the class classification unit 93.
• the tap constructing unit 92 extracts several depth data z used to classify the pixel of interest into a class, thereby converting the depth class tap into the depth data z. Build from.
• the tap construction unit 92 supplies the depth class tap to the class classification unit 93.
• the class classification unit 93 classifies the pixel of interest based on the image class tap supplied from the tap construction unit 91 and the depth class tap supplied from the tap construction unit 92. And the class is supplied to the coefficient memory 94.
• the coefficient memory 94 stores the prediction coefficient w for each class, noise parameter Nz, noise parameter Ni, and blur parameter P stored in the coefficient memory 19 of FIG. Coefficient
• the memory 94 acquires a noise parameter Nz, a noise parameter Ni, and a blur parameter P specified by the user.
• the coefficient memory 94 is based on the class supplied from the class classification unit 93 and the noise parameter Nz, noise parameter Ni, and blur parameter P specified by the user, the class, the noise parameter Nz, and the noise parameter. Read the prediction coefficient w corresponding to Ni and the blur parameter P from the stored prediction coefficient w, and the prediction coefficient w
• w is provided to the prediction computation unit 96.
• the tap constructing unit 95 extracts some of the pixels constituting the blurred image, which are used to predict the pixel value of the target pixel, so that the image predicting tap is blurred. Build from data.
• the tap construction unit 95 supplies the image prediction tap to the prediction calculation unit 96.
• the prediction calculation unit 96 uses the image prediction tap supplied from the tap construction unit 95 and the prediction coefficient w provided from the coefficient memory 94 to calculate a prediction value for the pixel value of the target pixel.
• the prediction calculation unit 96 performs a prediction calculation that is a calculation of the linear linear expression of the above-described expression (1). As a result, the prediction calculation unit 96 obtains the predicted value of the pixel value of the target pixel, that is, the pixel value of the pixels constituting the blur corrected image. Then, the prediction calculation unit 96 outputs the pixel value of each pixel constituting the blur corrected image as blur corrected image data.
• step S31 the tap constructing unit 91 constructs an image class tap from the blurred image data, and supplies the image class tap to the class classifying unit 93.
• step S32 the tap constructing unit 92 constructs a depth class tap from the depth data z, and supplies the depth class tap to the class classifying unit 93.
• step S33 the class classification unit 93 classifies the pixel of interest into a class based on the image class tap supplied from the tap construction unit 91 and the depth class tap supplied from the tap construction unit 92, and The class is supplied to the coefficient memory 94.
• step S34 the coefficient memory 94 acquires a noise parameter Nz, a noise parameter Ni, and a blur parameter P specified by the user.
• step S35 the coefficient memory 94 stores the class, noise based on the class supplied from the class classification unit 93 and the noise parameter Nz, noise parameter Ni, and blur parameter P specified by the user. Prediction coefficient w corresponding to parameter Nz, noise parameter Ni, and blur parameter P is selected from the stored prediction coefficients w.
• step S 36 the tap construction unit 95 constructs an image prediction tap from the blurred image data, and supplies the image prediction tap to the prediction calculation unit 96.
• step S37 the prediction calculation unit 96 uses the image prediction tap supplied from the tap construction unit 95 and the prediction coefficient w supplied from the coefficient memory 94 to the line of equation (1) described above.
• Prediction calculation which is a linear equation calculation, is performed to determine the pixel value of each pixel constituting the blur-corrected image and output it as blur-corrected image data. Then, the process ends.
• FIG. 11 is a block diagram showing a configuration of another embodiment of the learning device 1.
• the learning device 1 in FIG. 11 includes a blur adding unit 11, a noise adding unit 12, a noise adding unit 13, a tap building unit 14, a tap building unit 15, a class classification unit 16, a tap building unit 17, a coefficient memory 19 ,
• the prediction coefficient w used when performing the prediction process for predicting the corrected image is learned.
• FIG. 11 the same components as those in learning device 1 in FIG. 3 are denoted by the same reference numerals. That is, learning device 1 in FIG. 11 is similar to learning device 1 in FIG. In addition, a prediction coefficient calculation unit 102 is provided instead of the prediction coefficient calculation unit 18.
• the downscaling unit 101 receives depth data z from a device (not shown).
• the downscaling unit 101 is a horizontal scaling parameter H that indicates the horizontal size of the parent image corresponding to the depth data z after downscaling specified by the user.
• a scaling parameter (H, V) consisting of a vertical scaling parameter V representing the vertical size.
• the downscaling unit 101 Based on the scaling parameters (H, V), the downscaling unit 101, for example, the size power of the parent image corresponding to the depth data z, the parent image data input to the blur addition unit 11
• the depth data z is downscaled so as to be the same as the image size, and the downscaled depth data z is supplied to the noise adding unit 12.
• the prediction coefficient calculation unit 102 acquires a noise parameter Nz, a noise parameter Ni, a blur parameter P, and a scaling parameter (H, V) specified by the user.
• the prediction coefficient calculation unit 102 is constructed from the image class tap and the down-scaled depth data z based on the parent image data supplied from a device (not shown) and the image prediction tap supplied from the tap construction unit 17. For each class classified based on the selected depth class tap, noise parameter Nz, noise parameter Ni, blur parameter P, and scaling parameter (H, V), the prediction coefficient w is calculated and the coefficient memory 19 To supply.
• the learning device 1 in Fig. 11 downscales the input depth data z, so the size power of the parent image corresponding to the depth data z input to the downscaling unit 101, At the same time, even when the size of the parent image corresponding to the parent image data input to the blur adding unit 11 is larger, the parent image data input to the blur adding unit 11 using the downscaled depth data z. It is possible to learn a prediction coefficient w used when performing a prediction process using a blurred image having the same size as the parent image corresponding to, and the corresponding depth data z.
• the learning device 1 in FIG. 11 uses an image obtained by reducing a captured image having a size larger than the standard as a parent image, and uses it for a prediction process that predicts a blur-corrected image from a standard-size blurred image.
• the prediction coefficient w obtained can be learned.
• the learning device 1 in FIG. 11 performs learning processing for learning the prediction coefficient w.
• This learning process is started, for example, when parent image data and depth data z are input to the learning device in FIG.
• step S61 the downscaling unit 101 acquires the scaling parameters (H, V).
• step S62 the downscaling unit 101 sends the blur adding unit 11 Based on the scaling parameters (H, V), the depth data z is downscaled to match the size of the parent image corresponding to the input parent image data, and the downscaled depth data z is added to the noise addition unit 12 To supply.
• step S63 to step S72 is the same as the processing from step S1 to step S10 in Fig. 8, and a description thereof will be omitted.
• step S73 the prediction coefficient calculation unit 102 is based on the parent image data supplied from a device (not shown) and the image prediction tap supplied from the tap construction unit 17, and the class classification unit 16 is also supplied with the class power.
• the noise parameter Nz noise parameter Ni, blur parameter P, and scaling parameter (H, V)
• step S74 the coefficient memory 19 stores the prediction coefficient w supplied from the prediction coefficient calculation unit 102, as in step S12, and the process ends.
• FIG. 13 shows a plan in which prediction processing is performed using the prediction coefficient w learned by the learning device 1 in FIG.
• FIG. 3 is a block diagram showing a configuration of a measuring device 81.
• the prediction device 81 in FIG. 13 includes a tap construction unit 91, a tap construction unit 92, a class classification unit 93, a tap construction unit 95, a prediction calculation unit 96, and a coefficient memory 111.
• the same components as those of the prediction device 81 of FIG. 9 are denoted by the same reference numerals. That is, the prediction device 81 of FIG. 13 is provided with a coefficient memory 111 instead of the coefficient memory 94 of the prediction device 81 of FIG.
• the coefficient memory 111 stores the prediction coefficient w for each class, noise parameter N z, noise parameter Ni, blur parameter P, and scaling parameter (H, V) stored in the coefficient memory 19 of FIG. It is remembered.
• the coefficient memory 111 stores the noise parameters specified by the user.
• the coefficient memory 111 is based on the class supplied from the class classification unit 93 and the noise parameter Nz, noise parameter Ni, blur parameter P, and scaling parameter (H, V) specified by the user.
• the prediction coefficient w corresponding to the class, noise parameter Nz, noise parameter Ni, blur parameter P, and scaling parameter (H, V).
• the prediction device 81 in FIG. 13 performs a prediction process similar to the prediction process in FIG. 10, and thus description thereof is omitted.
• the coefficient memory 111 stores the class supplied from the class classification unit 93, the noise parameter Nz, the noise parameter Ni, the blur parameter P, and the scaling specified by the user. Based on the parameter (H, V), the prediction coefficient w corresponding to the class, noise parameter Nz, noise parameter Ni, blur parameter P, and scaling parameter (H, V) is already stored. Person in charge
• the number w is read out and the prediction coefficient w is provided to the prediction calculation unit 96.
• FIG. 14 is a block diagram showing a configuration of still another embodiment of the learning device 1.
• the learning device 1 in FIG. 14 includes a blur adding unit 11, a noise adding unit 12, a noise adding unit 13, a tap building unit 14, a tap building unit 15, a class classification unit 16, a tap building unit 17, a coefficient memory 19 , A downscaling unit 101, a prediction coefficient calculation unit 102, and a downscaling unit 121, which performs prediction processing for predicting a blur-corrected image with higher resolution than the blurred image from the blurred image by class classification adaptation processing. Learn the prediction coefficient w used sometimes.
• the same components as those in learning device 1 in FIG. 11 are denoted by the same reference numerals. That is, the learning device 1 in FIG. 14 is obtained by further providing the downscaling unit 121 force S to the learning device 1 in FIG.
• the downscaling unit 121 Based on the scaling parameters (H, V) specified by the user, the downscaling unit 121, for example, blurs so that the size of the student image is the same as the size of the blur image to be predicted.
• the student image data supplied from the adding unit 11 is downscaled, and the downscaled student image data is supplied to the noise adding unit 13.
• the downscaling unit 101 in Fig. 14 is based on the scaling parameters (H, V), for example, a high-resolution blur correction image corresponding to the depth data z as compared with the blur image. Downsize the depth data z so that the size of the parent image is the same as the size of the blurred image.
• H, V the scaling parameters
• the prediction coefficient calculation unit 102 is constructed from parent image data supplied from a device (not shown) and student image data after downscaling supplied from the tap construction unit 17. Based on the image prediction taps, the class tap constructed from the student image data after downscaling and the class classified based on the depth class tap constructed from the depth data z after downscaling, and the noise parameter Nz , Calculate the prediction coefficient w for each of the noise parameter Ni, blur parameter P, and scaling parameter (H, V)
• the coefficient memory 19 is supplied.
• the learning device 1 in FIG. 14 performs downscaling on the student image data and the depth data z, so that the resolution of the student image and the parent image corresponding to the depth data z is
• the parent image corresponding to the parent image data input to the learning device 1 in FIG. 14 can be converted to a low resolution.
• the learning device 1 in FIG. 14 uses the student image after conversion, the depth data z, and the parent image data, thereby predicting a blur-corrected image with higher resolution than the blurred image from the blurred image.
• the learning device 1 in FIG. 14 uses a prediction coefficient w used for prediction processing for predicting a blur corrected image that is an HD (High Definition) image from a blur image that is an SD (Standard Definition) image. Can learn.
• the learning device 1 in FIG. 14 performs learning processing for learning the prediction coefficient w.
• This learning process is started, for example, when the parent image data and the depth data z are input to the learning device 1 in FIG.
• step S101 to step S106 is the same as the processing from step S61 to step S66 in FIG.
• step S107 the downscaling unit 121 acquires the scaling parameters (H, V).
• step S108 the downscaling unit 121 downscales the student image data supplied from the blur adding unit 11 based on the scaling parameters (H, V), and the downscaled student image data is a noise adding unit. Supply to 13.
• step S109 to step S116 is the same as the processing from step S67 to step S74, and thus the description thereof is omitted.
• the device 81 Since the device 81 is the same as the prediction device 81 of FIG. 13, its description is omitted. [0204] In addition, the calculation of the prediction coefficient w may use data other than just pixels.
• the configuration of learning device 1 is shown in FIG.
• the learning device 1 in FIG. 16 includes a blur adding unit 11, a noise adding unit 12, a noise adding unit 13, a tap building unit 14, a tap building unit 15, a class classification unit 16, a tap building unit 17, and a coefficient memory 19 , The tap construction unit 131, and the prediction coefficient calculation unit 132.
• the depth data z is used to obtain the same size from the blurred image and the corresponding depth data z. It learns the prediction coefficient w used when performing the prediction process that predicts the blur-corrected image by the classification adaptation process.
• learning device 1 in FIG. 16 further includes learning device 1 in FIG. A construction unit 131 is provided, and a prediction coefficient calculation unit 132 is provided instead of the prediction coefficient calculation unit 18.
• Depth data z after noise addition is supplied from the noise addition unit 12 to the tap construction unit 131.
• the tap constructing unit 131 constructs a depth prediction tap by extracting several forces of the depth data z used for predicting the pixel value of the target pixel from the depth data z.
• the tap construction unit 131 supplies the depth prediction tap to the prediction coefficient calculation unit 132.
• the prediction coefficient calculation unit 132 acquires the noise parameter Nz, the noise parameter Ni, and the blur parameter P specified by the user. Further, the prediction coefficient calculation unit 132 is based on parent image data supplied from a device (not shown), an image prediction tap supplied with the tap construction unit 17 force, and a depth prediction tap supplied from the tap construction unit 131. Thus, the prediction coefficient w is calculated for each class supplied from the class classification unit 16 and each of the noise parameter Nz, the noise parameter Ni, and the blur parameter P.
• the prediction coefficient calculation unit 132 calculates n, k as X of the normal equation of the above equation (8) established for each class, noise parameter Nz, noise parameter Ni, and blur parameter P.
• Depth data Z that constitutes a depth prediction tap consisting of only the blurred pixels of the kth sample As a result, the prediction coefficient w corresponding to the number of pixels corresponding to the image prediction tap and the depth prediction tap is calculated for each class, noise parameter Nz, noise parameter Ni, and blur parameter P.
• the prediction coefficient calculation unit 132 obtains the class, noise parameter n
• Predictive coefficient w for each of Nz, noise parameter Ni, and blur parameter P is stored in coefficient memory 19.
• the learning device 1 in FIG. 16 uses the depth prediction tap constructed from the depth data z and also uses the pixel corresponding to the image prediction tap and the depth prediction tap in consideration of the depth data z. Since the prediction coefficient w for several times is calculated, by using this prediction coefficient w,
• the predicting device 81 in FIG. 18 described below can predict a blur-corrected image more accurately.
• the learning apparatus 1 in FIG. 16 performs learning processing for learning the prediction coefficient w.
• This learning process is started, for example, when the parent image data and the depth data z are input to the learning device 1 in FIG.
• step S121 to step S130 is the same as the processing from step S1 to step S10 in FIG.
• step S131 the tap constructing unit 131 constructs a depth prediction tap by extracting a predetermined one from the noise-added depth data z supplied from the noise adding unit 12, and selects the depth prediction tap. This is supplied to the prediction coefficient calculation unit 132.
• the prediction coefficient calculation unit 132 includes parent image data supplied from a device (not shown), an image prediction tap supplied from the tap construction unit 17, and a tap construction unit.
• the prediction coefficient w is calculated for each of the classes supplied from the class classification unit 16 and the noise parameter Nz, noise parameter Ni, and blur parameter P, and is stored in the coefficient memory 19. Supply.
• step S133 the coefficient memory 19 stores the prediction coefficient w supplied from the prediction coefficient computing unit 132, as in step S12 of Fig. 8, and the process ends.
• Fig. 18 shows a plan for performing the prediction process using the prediction coefficient w learned by the learning device 1 of Fig. 16.
• FIG. 3 is a block diagram showing a configuration of a measuring device 81.
• the prediction device 81 in FIG. 18 includes a tap construction unit 91, a tap construction unit 92, a class classification unit 93, a coefficient memory 94, a tap construction unit 95, a tap construction unit 141, and a prediction calculation unit 142. It is.
• FIG. 18 the same components as those of the prediction device 81 of FIG. 9 are denoted by the same reference numerals. That is, the prediction device 81 of FIG. 18 is newly provided with a tap construction unit 141, and a prediction calculation unit 142 is provided instead of the prediction calculation unit 96 of the prediction device 81 of FIG.
• the tap construction unit 141 extracts the depth prediction tap by extracting several forces of the depth data z used to predict the pixel value of the target pixel. Build from depth data z.
• the tap construction unit 141 supplies the depth prediction tap to the prediction calculation unit 144.
• the prediction calculation unit 142 uses the image prediction tap supplied from the tap construction unit 95, the depth prediction tap supplied from the tap construction unit 141, and the prediction coefficient w provided from the coefficient memory 94 to generate the target pixel. A prediction calculation for obtaining a predicted value of the pixel value is performed.
• the prediction calculation unit 142 calculates the image prediction n as X of the linear primary expression of the above-described expression (1).
• Depth data z that forms the depth prediction tap consisting of only the blur pixels that make up the measurement tap is also applied, and w is the image prediction tap and depth prediction n learned by the learning device 1 in FIG.
• the prediction coefficients for the number of pixels corresponding to the measurement tap By applying the prediction coefficients for the number of pixels corresponding to the measurement tap, the pixel values of the pixels constituting the blur corrected image are obtained.
• the prediction calculation unit 142 outputs the pixel value of each pixel constituting the blur corrected image as blur corrected image data.
• This prediction process is started, for example, when blurred image data and depth data z are input to the prediction device 81.
• step S141 to step S146 is the same as the processing from step S31 to step S36 in Fig. 10, and a description thereof will be omitted.
• step S147 the tap constructing unit 141 constructs a depth prediction tap from the depth data z, and supplies the depth prediction tap to the prediction computation unit 142.
• step S148 the prediction calculation unit 142 uses the image prediction tap supplied from the tap construction unit 95, the depth prediction tap supplied from the tap construction unit 141, and the prediction coefficient w provided from the coefficient memory 94. To calculate the predicted value of the pixel value of the pixel of interest, A pixel value of each pixel constituting the normal image is obtained and output as blur corrected image data. Then, the process ends.
• the noise described above can be considered including fluctuations added to the parameter.
• the fluctuation includes a fluctuation from a spatial or temporal average value of a quantity having a spread or intensity such as energy, density and voltage.
• the function that gives fluctuations is arbitrary.
• the 1 / f fluctuation can be generated by Fourier transforming the noise SWN, processing the power spectrum to 1 / f in the frequency domain, and performing inverse Fourier transform. Add 1 / f to the power spectrum related to fluctuations in the time direction of the noise amplitude to be added to the parameter, and add individual 1 / f fluctuations for each pixel parameter. For the frame as well, the power spectrum related to fluctuations in the time direction is set to 1 / f.
• an image with noise added thereto is generated by adding noise to the blur data of a preset blur model.
• FIG. 20 is a block diagram showing a configuration of an embodiment of an image generation apparatus that generates image data of an image to which noise is added.
• the basic configuration of the image generation device 301 is the same as that of the learning device 1 of FIG. 316, tap construction unit 317, prediction coefficient calculation unit 318, and coefficient memory 319 are the blur addition unit 11, noise addition unit 12, noise addition unit 13, tap construction unit 14, tap construction unit 15, It has the same functions as the class classification unit 16, tap construction unit 17, prediction coefficient calculation unit 18, and coefficient memory 19. Therefore, the explanation is omitted.
• the noise parameter N is supplied instead of the noise parameter Nz.
• the noise coefficient N and motion information are supplied to the prediction coefficient calculation unit 318! /.
• This image generating apparatus 301 has a function of generating image data of an image with added noise. In addition, it has a function of learning a prediction coefficient when performing a process of correcting noise from an image to which noise is added. That is, the image generation device 301 has a function as an image data generation device and a function as a prediction coefficient calculation device. For this reason, the image data generated by the noise adding unit 313 is output to other devices as image data of an image to which noise has been added, and is also supplied to the tap building unit 314 and the tap building unit 317 as student image data.
• the image data generated by the noise adding unit 313 is output to other devices as image data of an image to which noise has been added, and is also supplied to the tap building unit 314 and the tap building unit 317 as student image data.
• An image with noise added is generated as a blurred image by adding a noise component to the focused state or motion of the image.
• an image generation apparatus that generates image data of an image with noise added may have a configuration corresponding to the learning apparatus shown in FIG. An embodiment in this case is shown in FIG.
• the basic configuration of the image generating apparatus 400 is the same as that of the learning apparatus 1 in FIG. That is, the blur addition unit 311, noise addition unit 312, noise addition unit 313, tap construction unit 314, tap construction unit 315, class classification unit 316, tap construction unit 317, prediction coefficient calculation unit 402, coefficient memory 319,
• the downscaling unit 401, the prediction coefficient calculation unit 402, and the downscaling unit 421 are the blur addition unit 11, the noise addition unit 12, the noise addition unit 13, the tap construction unit 14, the tap construction unit 15, the class classification unit in FIG. 16, the tap construction unit 17, the coefficient memory 19, the downscaling unit 101, the prediction coefficient calculation unit 102, and the downscaling unit 121. Therefore, the description is omitted.
• the down-scaling unit 401 is supplied with motion information and parent image data.
• a noise parameter N is supplied to the noise adding unit 312 instead of the noise parameter Nz.
• the prediction coefficient calculation unit 402 is supplied with motion information, and is also supplied with a noise parameter N instead of the noise parameter Nz. .
• In-focus noise (out-of-focus noise) is added to the distance information, the deviation ⁇ of the blurred Gaussian function, the phase of the blurred Gaussian function, or the sharpness of the blurred Gaussian function. Or a combination of certain of them.
• noise is added to the depth data z as blur data.
• the noise SWNd is added to the depth data z before adding noise, so that Depth data Zswn is calculated.
• the noise SWNd is represented by the sum of a component SWNd (frame) that changes in units of frames and a component SWNd (pixel) that changes in units of pixels, as shown in the following equation.
• the noise SWNd can be expressed by the following equation, for example. This function mseq generates a pseudorandom number.
• the noise SWNd is expressed by the following equation.
• the subscript d on the right side of the following equation indicates that the coefficient R and the function mseq are related to distance.
• the coefficient Rd as a parameter for determining the noise SWNd is set corresponding to the noise parameter N.
• the noise adding unit 312 performs the above processing as shown in FIG.
• the setting unit 331 sets a processing area based on a user instruction.
• the acquisition unit 332 acquires the noise parameter N and motion information.
• the determination unit 333 determines the coefficient of the noise equation.
• the calculation unit 334 performs various calculations including noise.
• the blur adding unit 311 has a functional configuration of an acquisition unit 351, a selection unit 352, and a calculation unit 353.
• the acquiring unit 351 acquires the blur parameter P.
• the selection unit 352 selects the weight w.
• the calculation unit 353 performs various calculations.
• step S201 the setting unit 331 sets a processing area based on a user instruction. In this case, the user can set a part or all of the image as a processing area. If the entire image is always processed, this processing can be omitted.
• step S202 the acquisition unit 332 acquires the noise parameter N specified by the user.
• step S203 the determination unit 333 determines the coefficient Rd of the noise SWNd in Expression (24) corresponding to the noise parameter N.
• step S204 the computing unit 334 computes the noise SWNd. That is, the noise SWNd is calculated according to the equation (24).
• step S205 the computing unit 334 computes depth data to which the noise SWNd is added for the set processing region. Specifically, according to the equation (21), the noise SWNd calculated in step S204 is added to the acquired depth data z, and the depth data Zswn after adding the noise SW Nd is calculated. The depth data Zswn to which the noise SWNd is added is output to the blur adding unit 311 as a parameter that gives noise to the blur model.
• step S206 the computing unit 353 of the blur adding unit 311 calculates pixel data to which noise has been added. That is, as described above, the blur adding unit 311 calculates the blur point spread function WT (k, l) of Equation (19) as a blur model based on the depth data Zswn to which noise is added, and the equation ( Based on 18), a blur is added to the parent image data, and a still image in which the focus state is shaken is generated. This noise varies from frame to frame and from pixel to pixel.
• In-focus noise can be given based on the deviation ⁇ of the Gaussian function as a blur function.
• y-direction component S (x + k, y + l) are independent y
• equation (19) can be rewritten as
• WT (k, 1) 27T S x (x + k, y + l) S y (x + k, y + l) e (,) s y ( x + k,)
• Noise is given to the functions S (x + k, y + l) and S (x + k, y + l) as blur data independently.
• the X component and y component of the noise SWNs are SWNsx and SWNsy, respectively, and the functions S (x + k, y + l) and S (x + k, y + l) after adding the noise are calculated by the following equations:
• the X and y components that are changed in units of frames are SWNsx (frame) and S WNsy (frame)
• the x and y components that are changed in units of pixels are SWNsx (pixel) and SWNsy ( pixel)
• the x component SWNsx and y component SWNsy of the noise SWNs are expressed by the following equations.
• SWNsx SWNsx (frame) + SWNsx (pixel)
• SWNsy SWNsy (frame) + SWNsy (pixel)
• the noise SWNs is expressed by the above-described equation (23). If the component that changes in each frame is R ⁇ mseq [m] (frame) and the component that changes in pixels is R ⁇ mseq [m] (pixel), the X component SWNsx and y component SWNsy of the noise SWNs Is expressed by the following equation.
• step S231 the setting unit 331 sets a processing region based on a user instruction. In this case, the user can set a part or all of the image as a processing area. If the entire image is always processed, this processing can be omitted.
• step S232 the acquisition unit 332 acquires the noise parameter N specified by the user.
• step S233 the determination unit 333 determines the coefficients R 1 and R 2 in Expression (29) based on the noise parameter N.
• step S234 the calculation unit 334 calculates noise SWNsx SWNsy. That is, the noise SWNsx SWNsy is calculated from Equation (29) based on the coefficients R 1, R 2 corresponding to the noise parameter N acquired in step S232.
• step S235 the calculation unit 334 calculates a blurred point spread function WT (k, l) swn to which noises SWNsx and SWNsy are added. That is, the blurred point spread function WT (k, l) swn to which the noises SWN sx and SWNsy calculated in step S234 are added is calculated according to equation (28).
• the blur point spread function WT (k, l) swn to which the noise SWNsx and SWNsy are added is output to the blur adding unit 311 as a parameter that gives noise to the blur model.
• step S236 the calculation unit 353 of the blur adding unit 311 calculates pixel data to which the noise SWNsx and SWNsy are added for the set processing region. Specifically, the parent image data X (x + k, y + l) is acquired, and the noise SWNsx calculated in step S235 is obtained for the acquired parent image data X (x + k, y + l). The pixel data Y (x, y) force S is calculated using the blurred point spread function WT (k, l) s wn to which SWNsy is added, according to equation (18).
• Each pixel of the image of the image data generated in this way is added with a noise component that differs from frame to frame and from pixel to pixel. Therefore, if one frame of still image is generated by changing the noise component of each frame and pixel to generate an image of multiple frames, a kind of moving image that makes the image appear to shake is generated. can do.
• noise SWNk (x, y) and SWNl (x, y) are added to the X component k and y component 1 as blur data of the blur point spread function WT (k, l), and X
• the component kswn and y component lswn are as shown in the following equation.
• equation (19) is rewritten as the following equation.
• Noise SWNk (x, y) and SWNl (x, y) are also represented by the following equation, and noise components SWNk (x, y) (frame), SWN1 (frame), It is composed of the sum of noise components SWNk (x, y) (pixel) and SW Nl (pixel) in pixel units.
• SWNk (x, y) SWNk (x, y) (frame) + SWNk (x, y) (pixel)
• SWNl (x, y) SWNl (x, y) (frame) + SWNl (x, y) (pixel)
• Noise SWNk (x, y) and SWNl (x, y) are represented by the above-described equation (23). Then, the component that changes in each frame unit is R ⁇ mseq [m] (frame), R ⁇ mseq [m] (frame), pixel k k 1 1
• Nk (x, y) and SWM (x, y) are expressed by the following equations.
• step S261 the setting unit 331 sets a processing region based on a user instruction. In this case, the user can set a part or all of the image as a processing area. If the entire image is always processed, this processing can be omitted.
• step S262 the acquisition unit 332 acquires the noise parameter N specified by the user.
• step S 263 the determination unit 333 determines the coefficients R and R of the noise SWNk (x, y) and SWNl (x, y) in Expression (33) based on the noise parameter N.
• step S264 the calculation unit 334 calculates noises SWNk (x, y) and SWNl (x, y).
• the calculation unit 334 calculates a blurred point spread function WT (k, l) swn to which noises SWNk (x, y) and SWNl (x, y) are added. That is, the blurred point spread function WT (k, l) swn with the noise SWNk (x, y) and SWNl (x, y) calculated in step S264 is calculated according to equation (31). .
• the blur point spread function WT (k, l) swn to which the noise SWNk (x, y) and SWNl (x, y) are added is output to the blur adding unit 311 as a parameter that gives noise in the blur model. .
• step S266 the calculation unit 353 of the blur adding unit 311 calculates pixel data to which noise SWNk (x, y) and SWNl (x, y) are added, regarding the set processing region. Specifically, from the input parent image data X (x + k, y + l), the blur points with the noise SWNk (x, y) and SWNl (x, y) calculated in step S265 are added. Pixel data Y (x, y) is calculated according to Equation (18) using the spread function WT (k, l) swn.
• giving noise to the phase as described above is, for example, when the value of the X coordinate that gives the peak value of the blurred point spread function WT represented by the X coordinate is So
• WT which is a phase function whose X coordinate gives the peak value of
• In-focus noise can be added to an image by adding noise to the sharpness of the blur point spread function WT (k, l) as a blur model.
• Fig. 30 shows the function WT with the highest sharpness, the medium function WT, and the lowest function WT.
• the sharpness can be lowered by increasing the spacing between the sharpening points and increased by increasing the distance between the points.
• Level normalization is performed after calculating the addition characteristics of different deviations ⁇ for the target pixel and integrating them.
• the state in which the sharpness changes is noise in the depth direction (ie distance direction) within one pixel. It can be considered equivalent to a state in which there is an occurrence (that is, a state in which movement occurs back and forth within the integration time of one pixel).
• the blurred point spread function is expressed by the following mixed normal distribution formula.
• Noise SWNp is expressed by equation (23).
• the coefficient of the noise SWNp (x, y) R force is set corresponding to the S noise parameter N.
• step S271 the setting unit 331 sets a processing area based on a user instruction. To do. In this case, the user can set a part or all of the image as a processing area. If the entire image is always processed, this processing can be omitted.
• step S272 the acquisition unit 332 acquires the noise parameter N specified by the user.
• step S273 the determination unit 333 determines the coefficient R of the noise SWNp (x, y) in Expression (38) based on the noise parameter N.
• step S274 the calculation unit 334 calculates the noise SWNp (x, y). That is, based on the coefficient R corresponding to the noise parameter N obtained in step S272,
• Noise SWNp (x, y) is calculated from equation (38).
• step S275 the calculation unit 334 calculates a blurred point spread function WT (k, l) swn to which the noise SWNp (x, y) is added. That is, the blurred point spread function WT (k, l) swn to which the noise SWNp (x, y) calculated in step S274 is added is calculated according to equation (35).
• the blur point spread function WT (k, l) swn to which the noise SWNp (x, y) is added is output to the blur adding unit 311 as a parameter that gives noise in the blur model.
• step S276 the computing unit 353 of the blur adding unit 311 calculates pixel data to which the noise SWNp (x, y) has been added, regarding the set processing region. Specifically, the blurred point spread function WT (k, k, with the noise SWNp (x, y) calculated in step S275 added from the input parent image data X (x + k, y + l). l) Pixel data Y ( x , y) is calculated according to equation (18) using SW n.
• the blurring point spread function WT (k, l) in Equation (19) as a blur model can be changed to the functions WT and WT that also distort the Gaussian function WT force.
• FIG. 33 is a diagram illustrating imaging by a sensor.
• Sensor 391 for example, solid-state imaging It consists of a CCD video camera equipped with a CCD (Charge-Coupled Device) area sensor.
• the object corresponding to the foreground in the real world moves horizontally between the object corresponding to the background and the sensor 391 in the real world, for example, from the left side to the right side in the figure.
• CCD Charge-Coupled Device
• the sensor 391 configured by, for example, a video camera or the like images an object corresponding to the foreground together with an object corresponding to the background.
• the sensor 391 outputs the captured image in units of one frame.
• sensor 391 outputs an image consisting of 30 frames per second.
• the exposure time of sensor 391 can be 1/30 seconds.
• the exposure time is a period from when the sensor 391 starts converting input light to electric charge until the conversion of input light to electric charge ends.
• the exposure time is also referred to as a shatter time.
• FIG. 34 is a diagram illustrating the arrangement of pixels.
• a through I indicate individual pixels.
• the pixels are arranged on a plane corresponding to the image.
• One detection element corresponding to one pixel is arranged on the sensor 391.
• one detection element outputs a pixel value corresponding to one pixel constituting the image.
• the position of the detection element in the X direction corresponds to the horizontal position on the image
• the position of the detection element in the Y direction corresponds to the vertical position on the image.
• a detection element that is a CCD converts input light into electric charges and accumulates the converted electric charges during a period corresponding to the shatter time.
• the amount of charge is approximately proportional to the intensity of the input light and the time during which the light is input.
• the detecting element adds the electric charge converted from the input light to the already accumulated electric charge in a period corresponding to the shatter time. That is, the detection element integrates the input light for a period corresponding to the shatter time, and accumulates an amount of charge corresponding to the integrated light. It can be said that the detection element has an integration effect with respect to time.
• Fig. 36 shows, in the time direction, pixel values of pixels arranged in a row adjacent to each other in an image of an object corresponding to a stationary foreground and an object corresponding to a stationary background.
• the pixels arranged on one line of the screen can be selected as the pixels arranged in a row adjacent to each other.
• the pixel values F01 to F04 shown in FIG. 36 are pixel values corresponding to the foreground object that is stationary.
• the pixel values B01 to B04 shown in FIG. 36 are the pixel values corresponding to the background object that is stationary.
• the horizontal direction in Fig. 36 corresponds to the spatial direction X. More specifically, in the example shown in FIG. 36, the distance from the left side of the rectangle indicated as “F01” in FIG. 36 to the right side of the rectangle indicated as “B04” is 8 times the pixel pitch, That is, it corresponds to the interval between eight consecutive pixels.
• the period corresponding to the shatter time is divided into two or more periods of the same length.
• the model diagram shown in Fig. 36 can be represented as the model shown in Fig. 37.
• the number of virtual divisions is set according to the amount of movement V of the object corresponding to the foreground within the shirt time.
• the virtual division number is set to 4 corresponding to the motion amount V being 4, and the period corresponding to the shirt time is divided into four.
• the top line in the figure corresponds to the first divided period after the shatter opens.
• the second line from the top in the figure corresponds to the second divided period after the shatter opens.
• the third row from the top corresponds to the third divided period since the shatter opened.
• the 4th row from the top in the figure corresponds to the 4th divided period after the shatter opens.
• the shatter time divided according to the amount of movement v is also referred to as shatter time / v.
• the foreground component FOl / v is equal to the pixel value F01 divided by the virtual division number.
• the foreground component F02 / v is equal to the pixel value F02 divided by the virtual division number
• the foreground component F03 / v is the pixel value F03.
• the foreground component F04 / v is equal to the value obtained by dividing the pixel value F04 by the virtual division number.
• the background component BOl / v is equal to the value obtained by dividing the pixel value B01 by the virtual division number.
• the background component B02 / v is equal to the pixel value B02 divided by the virtual division number
• B03 / v is the pixel value B03 divided by the virtual division number.
• B04 / v equal to the value divided by is equal to the pixel value B04 divided by the number of virtual divisions.
• Fig. 38 shows a covered background area (when the object corresponding to the foreground moves toward the right side of the figure).
• This is a model diagram in which the pixel values of pixels on one line are expanded in the time direction, including the foreground component and background component mixed region, and the region where the background component is covered by the foreground over time) is there.
• the foreground motion V is 4. Since one frame is a short time, it can be assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed.
• the image of the object corresponding to the foreground moves so that it is displayed on the right by 4 pixels in the next frame with reference to a certain frame.
• the leftmost pixel through the fourth pixel from the left belong to the foreground area.
• the fifth through seventh pixels from the left belong to the mixed area, which is the covered background area.
• the rightmost pixel belongs to the background area.
• the component included in the pixel value of the pixel belonging to the covered background area corresponds to the shatter time. At some point in the period, the background component is replaced by the foreground component.
• the pixel value M with a thick frame in FIG. 38 is expressed by Expression (39).
• M B02 / v + B02 / v + F07 / v + F06 / v (39)
• the fifth pixel from the left includes a background component corresponding to one shatter time / V, and includes a foreground component corresponding to three shatter times / V.
• the elementary mixture ratio ⁇ (the ratio of the foreground component to the value of one pixel, which is the sum of the foreground and background components) is 1/4.
• the sixth pixel from the left contains a background component corresponding to two shatter times / V and a foreground component corresponding to two shatter times / V, so the mixture ratio of the sixth pixel from the left ⁇ Is 1/2.
• the seventh pixel from the left contains a background component corresponding to three shatter times / ⁇ and a foreground component corresponding to one shatter time / ⁇ , so the mixture ratio ⁇ of the seventh pixel from the left ⁇ Is 3/4.
• the object corresponding to the foreground is a rigid body, and it can be assumed that the foreground image moves at a constant speed so that it is displayed 4 pixels to the right in the next frame.
• the foreground component F07 / v of the first pixel after the shatter opens, corresponding to the second shatter time / V of the fifth pixel from the left in Fig. 38. Equal to foreground components.
• the foreground component F07 / v is the sixth pixel from the left in FIG. 38, the foreground component corresponding to the third shatter time / V when the shatter opens, and the seventh pixel from the left in FIG. Are equal to the foreground component corresponding to the fourth shot time / V when the shatter is open.
• the object corresponding to the foreground is a rigid body, and it can be assumed that the foreground image moves at a constant speed so that it is displayed 4 pixels to the right in the next frame.
• the foreground component F06 / V of the first shotta time / V when the shatter is open is the foreground corresponding to the second shatter time / V of the fourth pixel from the left in Fig. 38. Is equal to Similarly, the foreground component F06 / v is the fifth pixel from the left in FIG. 38, the foreground component corresponding to the third shatter time / V when the shatter opens, and the sixth pixel from the left in FIG. Is equal to the foreground component corresponding to the fourth shatter time / V of the second shatter.
• the object corresponding to the foreground is a rigid body, and it can be assumed that the foreground image moves at a constant speed so that it is displayed 4 pixels to the right in the next frame.
• the foreground component F05 / V of the first pixel when the shatter is open corresponds to the second shatter time / V of the third pixel from the left in Fig. 38. Equal to foreground components.
• the foreground component F05 / v is the fourth pixel from the left in FIG. 38, the foreground component corresponding to the third shatter time / V when the shatter opens, and the fifth pixel from the left in FIG. Are equal to the foreground component corresponding to the fourth shot time / V when the shatter is open.
• the object corresponding to the foreground is a rigid body, and it can be assumed that the foreground image moves at a constant speed so that it is displayed 4 pixels to the right in the next frame.
• the foreground component F04 / v of the first shatter time / V when the shatter opens is the second pixel from the left in FIG. Equal to the corresponding foreground component.
• the foreground component F04 / v is the third pixel from the left in FIG. 38, the foreground component corresponding to the third shatter time / V when the shatter opens, and the left force in FIG.
• the fourth pixel is equal to the foreground component corresponding to the fourth shatter time / V when the shatter is open.
• the foreground area corresponding to the moving object includes motion blur in this way, it can be said to be a distortion area.
• FIG. 39 shows an uncovered background area when the foreground moves toward the right side of the figure.
• FIG. 3 is a model diagram in which pixel values of pixels on one line are expanded in the time direction including a foreground component and background component mixed region, and a region where a background component appears corresponding to the passage of time).
• the foreground motion V is 4. Since one frame is a short time, it can be assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed.
• the image of the object corresponding to the foreground moves to the right by 4 pixels in the next frame with reference to a certain frame.
• the leftmost pixel through the fourth pixel from the left belong to the background area.
• the pixel value M ′ indicated by the thick line frame in FIG. 39 is expressed by Expression (40).
• M ' F02 / v + F01 / v + B26 / v + B26 / v (40)
• the fifth pixel from the left contains the background components corresponding to three shatter times / V, and the foreground component corresponding to one shatter time / V, so the fifth image from the left
• the elementary mixing ratio ⁇ is 3/4.
• the sixth pixel from the left contains a background component corresponding to two shatter times / V and a foreground component corresponding to two shatter times / V.
• the mixing ratio ⁇ of the sixth pixel from the left is 1/2.
• the seventh pixel from the left contains the background component corresponding to one shirter time / V and the foreground component corresponding to three shirter times / V, so the mixture ratio ⁇ of the seventh pixel from the left ⁇ Is 1/4.
• a is the mixing ratio.
• B is the background pixel value, and Fi / v is the foreground component.
• the object corresponding to the foreground is a rigid body and can be assumed to move at a constant speed, and the amount of movement is V force, for example, the fifth pixel from the left in FIG.
• the foreground component FOl / v of the shatter time / V is equal to the foreground component of the sixth pixel from the left in FIG. 39 corresponding to the second shatter time / V when the shatter is opened.
• FOl / v is the foreground component of the seventh pixel from the left in Fig. 39 corresponding to the third shatter time / V when the shatter opens and the eighth pixel from the left in Fig. 39.
• the foreground component corresponding to the fourth shatter time / v when the shatter is open is equal to each.
• the object corresponding to the foreground is a rigid body and can be assumed to move at a constant speed, and the virtual division number is 4, for example, the shirta of the sixth pixel from the left in FIG. 39 is opened.
• the first foreground component F02 / v of shatter time / V is equal to the foreground component of the seventh pixel from the left in Fig. 39 corresponding to the second shatter time / V when the shutter is opened.
• the foreground component F02 / v is equal to the foreground component of the eighth pixel from the left in FIG. 39 corresponding to the third shutter time / V when the shutter is opened! /.
• the object corresponding to the foreground is a rigid body and can be assumed to move at a constant speed, and the amount of movement is V force, for example, the seventh pixel from the left in FIG.
• the foreground component F03 / v of the shatter time / V is equal to the foreground component corresponding to the second shatter time / V of the eighth pixel from the left in FIG.
• the force virtual division number described as the virtual division number being 4 corresponds to the motion amount V.
• the amount of movement V is generally determined by the object corresponding to the foreground. Corresponds to the moving speed. For example, when the object corresponding to the foreground is moving so that it is displayed on the right by 4 pixels in the next frame with reference to a certain frame, the amount of movement V is 4. Corresponding to the amount of motion V, the number of virtual divisions is 4. Similarly, for example, when the object corresponding to the foreground is moving so that it is displayed on the left by 6 pixels in the next frame with reference to a certain frame, the motion amount V is set to 6, and the number of virtual divisions Is 6
• V v + SWNv (42)
• equation (41) is rewritten as follows, and each pixel value M is calculated based on the following equation.
• the noise SWN is a component SWN
• the noise SWNv is expressed by the above-described equation (23).
• the component that changes in each frame unit is R ⁇ mseq [m] (frame), and the component that changes in pixel units is R ⁇ mseq.
• the coefficient R in formula (45) is determined according to the S noise parameter N.
• step S291 the setting unit 331 sets an area designated by the user as a processing area. In this case, part or all of the image can be set as the processing area. If the entire image is always processed, this processing can be omitted.
• the acquiring unit 332 acquires motion information of each pixel in the processing region set in step S291. This movement information includes the movement amount V! /
• step S293 the acquisition unit 332 acquires the noise parameter N specified by the user.
• step S294 the determination unit 333 determines the coefficient Rv of Expression (45) based on the acquired noise parameter N.
• step S295 the calculation unit 334 calculates the noise SWN. That is, based on the coefficient Rv determined in step S294.
• the noise SWN is calculated according to the equation (45).
• step S296 the calculation unit 334 calculates the motion amount v with the noise SWN added.
• step S297 the calculation unit 353 of the blur adding unit 311 calculates pixel data to which the noise SWN is added in the set processing region. Specifically, the parent image data
• Pixel value M force S based on equation (43) using the mixture ratio ⁇ , background pixel value ⁇ ⁇ ⁇ , foreground pixel value Fi, and motion amount V with the calculated noise SWN added.
• V swn swn Calculated.
• a predetermined coefficient is applied to the pixel values of other pixels in the processing area WA on the line where the target pixel of interest is located.
• the weighted and summed value is added to the pixel value of the target pixel as a blur component.
• the direction of movement is the vertical direction
• the value obtained by multiplying the pixel values of other pixels in the processing area WA in the vertical line where the target pixel of interest is located by weighting with a predetermined coefficient is summed. It is added to the pixel value of the pixel of interest as a blur component.
• a range of a predetermined width centered on the line L in the direction of motion where the target pixel of interest is located is a processing region. It is called WA.
• interpolation pixels at positions separated by the same distance as the horizontal and vertical pitches of the pixels are calculated.
• Fig. 42 shows the principle of interpolation pixel calculation. As shown in the figure, the pixel value DPwa at the interpolation position Pwa is calculated based on the following equation, the pixel values DPwl to DPw4 at the four surrounding positions Pwl to Pw4 closest to the position Pwa. .
• Noise for the angle ⁇ (direction of motion) as blur data is decomposed and added to ⁇ ⁇ ⁇ .
• the noise for / 3h and / 3 V as blur data is S WN and SWN, respectively, ⁇ hswn and ⁇ vswn that are / 3 h and / 3 V after noise addition are respectively
• This equation is used to calculate the interpolation pixel when calculating the pixel value DPwa at the interpolation position Pwa. This means adding noise to the position.
• DPwaswn ⁇ (1— / 3 hswn) (1— / 3 vswn) / v ⁇ DPwl
• the pixel value DPwswn of the target pixel obtained by adding noise to the pixel value DPwl of the target pixel is calculated by the following equation.
• w is a weighting coefficient for each interpolation pixel, and is selected and determined based on the blur parameter P.
• step S361 the setting unit 331 sets a processing area based on an instruction from the user. In this case, part or all of the image can be set as the processing area.
• step S362 the acquisition unit 332 acquires motion information of each pixel in the processing region.
• This motion information includes information indicating the direction of motion in addition to the amount of motion V.
• step S363 the calculation unit 334 calculates an interpolated pixel along the direction of motion.
• the acquisition unit 332 acquires the noise parameter N based on the input from the user.
• the determination unit 333 determines the noise SWN and the coefficients R and R of SWN in equation (48).
• step S366 operation unit 334 calculates noise SWN and SWN based on equation (48).
• step S367 the calculation unit 334 adds the noise SWN and SWN.
• step S368 the acquisition unit 351 of the blur adding unit 311 acquires the blur parameter P based on the input from the user.
• step S369 the selection unit 352 selects a corresponding weight w from the weights w stored in advance based on the acquired blur parameter P.
• step S370 the calculation unit 353 adds the noise SWN and SWN.
• each noise SWN noise SWNd, SWNsx, SWNsy, SWNk (x, y), SWNl (x, y)
• rand is a function that generates pseudo-random numbers.
• noise SWN can also be expressed by the following equation.
• the image data to which noise is added by the blur adding unit 311 is supplied to a device (not shown) as image data to which an effect is added after further noise is added by the noise adding unit 313 as necessary.
• This image data is supplied to the tap construction unit 314 and used for the learning process.
• Information necessary for learning (the same information supplied to the blur adding unit 311) is also supplied from the noise adding unit 312 to the tap building unit 315.
• the prediction coefficient calculation unit 318 has noise.
• the parameter N, noise parameter Ni, blur parameter P, and motion information (motion amount and direction) are supplied.
• the learning process performed by the image generation device 301 in Fig. 20 and the image generation unit 400 in Fig. 21 is the same as that in the learning device 1 in Fig. 3 and the learning device 1 in Fig. 14, and the description thereof will be repeated.
• the prediction coefficient for generating an image with corrected shaking is obtained from the image with added shaking.
• the class used is an arbitrary force.
• the class D corresponding to the blur parameter P can be determined based on the following equation.
• a represents the x coordinate component of the motion vector in the specified region
• n represents the y coordinate component
• A represents the X coordinate component of the offset value input by the user
• N represents the y coordinate component.
• Nmax means the total number of classes of y-coordinate components
• the blur parameter stored corresponding to the image data is expressed by ((a + A),
• class D can be computed.
• Classification can also be made based on the amount of motion v and the direction of motion (angle ⁇ ).
• class taps can be extracted from the image according to the amount of motion and angle, and can be classified by 1-bit ADRC, or can be classified based on the amount of motion and angle itself. wear.
• the difference value between the class classVc using the integer of the motion amount V as it is and the pixel of interest and the eight neighboring pixels around it is classified into three classes: positive, negative, and equivalent.
• classVdiff ⁇ ⁇ classVdiffJ X 3 J ⁇
• size_Vc in Equation (55) is 30.
• FIG. 44 is a block diagram showing a configuration of an embodiment of a prediction apparatus that corrects an image including blur using a prediction coefficient generated by learning of the image generation apparatus 301 in FIG. .
• This prediction device 681 has basically the same configuration as the prediction device 81 of FIG.
• the tap construction unit 691, the tap construction unit 692, the class classification unit 693, the coefficient memory 694, the tap construction unit 695, and the prediction calculation unit 696 included in the prediction device 681 of FIG. 44 are the prediction device of FIG. 81 has basically the same functions as the tap construction unit 91, tap construction unit 92, class classification unit 93, coefficient memory 94, tap construction unit 95, and prediction computation unit 96.
• the tap construction unit 692 is input with motion information that only passes through the depth data z.
• motion information is also input to the coefficient memory 694.
• the noise parameter N is input instead of the noise parameter Nz.
• the prediction processing of the prediction device 681 is the same as that shown in Fig. 10 except that the information used for the processing is different, and the description thereof is omitted. However, in this case, in step S32 of FIG. 10, the tap construction unit 692 constructs a class tap from the depth data z or the motion information.
• step S35 the coefficient memory 701 is based on the class supplied from the class classification unit 693, the motion information, the noise parameter N specified by the user, the noise parameter Ni, and the blur parameter P. Prediction coefficient w corresponding to the class, motion information, noise parameter N, noise parameter Ni, and blur parameter P
• the prediction coefficient w is read from the coefficient w and provided to the prediction calculation unit 696.
• FIG. 45 is a block diagram showing a configuration of an embodiment of a prediction apparatus that corrects an image including blur using a prediction coefficient generated by learning of the image generation apparatus 400 in FIG. .
• This prediction device 681 has basically the same configuration as the prediction device 81 in FIG. That is, the tap construction unit 691, the tap construction unit 692, the class classification unit 693, the coefficient memory 701, the tap construction unit 695, and the prediction calculation unit 696 included in the prediction device 681 of FIG.
• the device 81 has basically the same functions as the tap construction unit 91, the tap construction unit 92, the class classification unit 93, the coefficient memory 111, the tap construction unit 95, and the prediction calculation unit 96 that the device 81 has.
• the tap construction unit 692 is input with motion information that only passes through the depth data z.
• motion information is also input to the coefficient memory 701.
• a noise parameter N is input instead of the noise parameter Nz.
• the prediction processing of the prediction device 681 is the same as that shown in Fig. 10 except that the information used for the processing is different, and the description thereof is omitted. However, in this case, in step S32 of FIG. 10, the tap construction unit 692 constructs a class tap from the depth data z or the motion information.
• step S35 the coefficient memory 701 stores the class supplied from the class classification unit 693, the motion information, the noise parameter N specified by the user, the noise parameter Ni, the blur parameter P, and the scaling parameter ( Based on (H, V), the prediction coefficient w corresponding to the class, motion information, noise parameter N, noise parameter Ni, blur parameter P, and scaling parameter (H, V) is already stored. Read from coefficient w
• the prediction coefficient w is provided to the prediction calculation unit 696.
• FIG. 46 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program.
• a CPU Central Processing Unit
• a ROM Read Only Memory
• a RAM Random Access Memory
• the CPU 521, ROM 522, and RAM 523 are connected to each other by a bus 524.
• the CPU 521 is also connected to an input / output interface 525 via the bus 524.
• the input / output interface 525 is connected to an input unit 526 including a keyboard, a mouse, and a microphone, and an output unit 527 including a display and a speaker.
• CPU 521 executes various processes in response to commands input from the input unit 526. Then, the CPU 521 outputs the processing result to the output unit 527.
• the storage unit 528 connected to the input / output interface 525 includes, for example, a hard disk, and stores programs executed by the CPU 521 and various data.
• the communication unit 529 communicates with an external device via a network such as the Internet or a local area network. Further, the communication unit 529 may acquire a program and store it in the storage unit 528.
• the drive 530 connected to the input / output interface 525, when a removable medium 531 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is mounted, drives them and records there. Get the programs and data that are being used. Acquired programs and data are transferred to and stored in the storage unit 528 as necessary.
• a removable medium 531 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory
• the step of describing the program stored in the recording medium is not necessarily processed in time series, as well as processing performed in time series in the order described. This includes processing executed in parallel or individually.

Landscapes

• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Image Processing (AREA)
• Editing Of Facsimile Originals (AREA)
• Facsimile Image Signal Circuits (AREA)

Priority Applications (1)

Applications Claiming Priority (4)

Publications (1)

Family

ID=39230182

Family Applications (1)

Country Status (3)

Cited By (1)

Families Citing this family (18)

Citations (8)

Family Cites Families (4)

• 2007
  • 2007-09-19 JP JP2007242558A patent/JP4872862B2/ja not_active Expired - Fee Related
  • 2007-09-28 WO PCT/JP2007/068924 patent/WO2008038748A1/ja not_active Ceased
  • 2007-09-28 US US12/441,404 patent/US20100061642A1/en not_active Abandoned

Patent Citations (8)

Also Published As

Similar Documents

Legal Events