JP4692401B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing program for processing an image (still image or moving image), and in particular, an image processing apparatus and an image processing for performing image interpolation processing mainly for changing the resolution and magnification of an image. Regarding the program.

  Regarding image processing using a neural network, many research and development and technical proposals have been made from the 1980s to the 1990s. For example, in Patent Documents 1 to 3 below, there is a technique for performing line interpolation for progressively interlaced by performing processing using a hierarchical neural network that has been learned by the back-propagation method. It is disclosed.

On the other hand, Patent Documents 4 to 7 below disclose techniques for performing interpolation for changing resolution and magnification using a hierarchical neural network in order to improve resolution and smooth interpolation performance.
Japanese Patent Laid-Open No. 5-37910 JP-A-7-240901 JP-A-8-18926 JP-A-6-12486 JP-A-9-51430 JP-A-9-319866 Japanese Patent Laid-Open No. 10-164519

  However, each of the techniques disclosed in Patent Documents 1 to 7 described above has the following problems. For example, the technique disclosed in Patent Document 4 has a problem that it is necessary to prepare statistical data of an image in advance.

  The technique disclosed in Patent Document 5 substitutes a complicated logic selection by a neural network in a pattern process by controlling a combination coefficient of a filter used conventionally by a neural network. However, here, since the filter selection method for the original image is learned and applied to other images, the learning pattern is actually limited to an upper limit, and any image can be handled. There is a problem that it is difficult to construct a processing mechanism.

  Further, the techniques disclosed in Patent Documents 6 and 7 estimate a gradation curved surface of one frame (field) of an image using a neural network having pixel values and pixel positions as input signals, and perform interpolation using this. However, it is necessary to update the load of the neural network for each frame (field), and it is considered that there is a limit to the shape of the curved surface that can be reproduced by the neural network. There is a problem that it is not suitable for processing.

  Furthermore, in the technique disclosed in Patent Document 1, when the teacher signal is a real image, it cannot be said that the performance of the input image other than the image used for learning can be sufficiently exhibited.

  On the other hand, in the techniques disclosed in Patent Documents 2 and 3, since a combination of peripheral pixel values to be learned including both input and output is artificially generated, it is possible to learn various patterns uniformly. It becomes possible to do. However, even in this case, when a pattern that has not been learned is input, correct interpolation is not performed even with the generalization ability of the neural network, and as a result, a noisy interpolation signal is output. There is.

  In order to solve the above problems, an object of the present invention is to provide an image processing apparatus and an image processing program for processing an image by effectively using a neural network.

  In order to achieve the above object, in the present invention, the suitability of the result of image processing using the first neural network is judged by the second neural network, and the first neural network is determined based on the judgment result. A mixing ratio between the result of the used image processing and the result of the other image processing is determined, and a result of mixing these image processing results is output as a result of the image processing.

That is, according to the present invention, an image processing apparatus having pixel data of a plurality of pixels as an input value and pixel data of an interpolation pixel for interpolating the plurality of pixels as an output value,
The first neural network is a means configured to output first pixel data of interpolated pixels for interpolating the plurality of pixels with respect to pixel data of the plurality of pixels, and the first neural network is generated in advance. A first image processing means that is a neural network in which a relationship between pixel data of a plurality of pixels and pixel data of an interpolation pixel corresponding to the pixel data of the plurality of pixels is learned;
Second image processing means for executing image processing different from the first image processing means and outputting second pixel data of interpolated pixels for interpolating the plurality of pixels with respect to pixel data of the plurality of pixels;
Means configured by a second neural network for determining suitability of image processing by the first image processing means for pixel data of the plurality of pixels and outputting a value indicating suitability of the first image processing means; The second neural network includes first pixel data of the plurality of previously generated pixels used for learning of the first neural network, and first information indicating that the first neural network has been learned. A relationship with a specific value is learned in advance, and a relationship between pixel data of a plurality of randomly generated pixels and a specific value indicating that learning has not been performed by the first neural network is learned. A fitness determination means that is a neural network;
Based on the value indicating the suitability of the first image processing means output from the suitability judging means, the first pixel data outputted from the first image processing means and the output from the second image processing means Pixel value mixing means for mixing the second pixel data and interpolating the plurality of pixels to output pixel data of an interpolation pixel as the output value,
An image processing apparatus is provided.

According to the present invention, there is provided an image processing program for causing a computer to execute an image processing method using pixel data of a plurality of pixels as an input value and pixel data of an interpolation pixel for interpolating the plurality of pixels as an output value. ,
Realization of a first image processing mechanism that realizes a first image processing mechanism configured by a first neural network that outputs first pixel data of interpolation pixels that interpolate the plurality of pixels with respect to pixel data of the plurality of pixels. Steps,
A first learning step in which the first image processing mechanism learns a relationship between pixel data of a plurality of pixels generated in advance and pixel data of an interpolation pixel corresponding to the pixel data of the plurality of pixels;
A second image processing mechanism that executes image processing different from the first image processing means and outputs second pixel data of interpolated pixels for interpolating the plurality of pixels with respect to pixel data of the plurality of pixels is realized. A second image processing mechanism realizing step,
Consistency constituted by a second neural network that determines the suitability of image processing by the first image processing mechanism for pixel data of the plurality of pixels and outputs a value indicating the suitability of the first image processing mechanism. A compatibility judgment mechanism realization step for realizing the judgment mechanism;
The conformity determination mechanism includes pixel data of the plurality of previously generated pixels used for learning of the first neural network, and a first specific value indicating that the first neural network has been learned. Is learned in advance, and learns a relationship between pixel data of a plurality of randomly generated pixels and a second specific value indicating that the first neural network has not been learned. Learning steps,
The first pixel data output from the first image processing mechanism and the output from the second image processing mechanism based on a value indicating the compatibility of the first image processing mechanism output from the compatibility determination mechanism. A pixel value mixing step of mixing the second pixel data and interpolating the plurality of pixels to output pixel data of an interpolation pixel as the output value,
An image processing program for causing a computer to execute the image processing method is provided.

  The present invention has the above-described configuration, and by determining whether or not an image processing result using a neural network is appropriate, a natural high-quality image can be obtained by effectively using the neural network. This has the effect of realizing processing for generation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the configuration of the image processing apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to an embodiment of the present invention. In the following description, the image processing apparatus illustrated in FIG. 1 uses a 4 × 4 pixel as illustrated in FIG. 2 as a peripheral pixel (input pixel) and is located near the center of these peripheral pixels. A case where image processing for obtaining five existing interpolation pixels is performed will be described. In FIG. 2, ◯ (open circles) are peripheral pixels, and ● (black circles) are interpolation pixels. According to the relationship between the peripheral pixels and the interpolated pixels shown in FIG. 2, the horizontal and vertical directions of the peripheral pixels arranged in a square lattice pattern (vertical two-axis directions forming a square lattice) This is equivalent to performing resolution conversion processing to double the number of 4 × 4 peripheral pixels.

  In FIG. 1, the peripheral pixels (for example, 4 × 4 pixels shown in FIG. 2) are input pixels, and the pixel values of the peripheral pixels are the first neural network unit (first NN unit) 101, The second neural network unit (second NN unit) 102 and the bicubic interpolation processing unit 103 are input.

  The first neural network unit 101 has a function of using 4 × 4 peripheral pixels as input pixels and outputting pixel values of five interpolation pixels (5-value NN interpolation result) using a neural network. Yes. The first neural network unit 101 is subjected to learning specialized for oblique line interpolation, which will be described later, for example.

  On the other hand, the second neural network unit 102 uses 4 × 4 peripheral pixels as input pixels, and uses the neural network to distinguish the conformity of the first neural network unit 101 (by the first neural network unit 101). (The validity of the NN interpolation result). The second neural network unit 102 is subjected to learning specialized for distinguishing the suitability of the first neural network unit 101 as will be described later. That is, the second neural network unit 102 can determine whether a pixel pattern learned for the first neural network unit 101 is input as an input pixel.

  The bicubic interpolation processing unit 103 has a function of executing a bicubic interpolation method using 4 × 4 peripheral pixels as input pixels and obtaining pixel values of the interpolated pixels related to these peripheral pixels (5-value bicubic interpolation result). have.

  Both the NN interpolation result by the first neural network unit 101 and the bicubic interpolation result by the bicubic interpolation processing unit 103 are five values as shown in FIG. 2 and are supplied to the mixture ratio determining unit 104. . Further, the discrimination result of the suitability of the first neural network unit 101 calculated by the second neural network unit 102 is also supplied to the mixture ratio determining unit 104.

  Then, the mixture ratio determination unit 104 determines a mixture ratio based on the distinction result of the second neural network unit 102 for each of the five values of the NN interpolation result and the five values of the bicubic interpolation result. Based on this, a quinary interpolation result obtained by mixing the NN interpolation result and the bicubic interpolation result is output. That is, the mixing ratio determination unit 104 is limited to the case where the advantages of the image processing by the first neural network unit 101 can be exploited, and the image processing by the first neural network unit 101 is added to the final output interpolation result. The result can be reflected.

  Both the first and second neural network units 101 and 102 are well-known neural networks having a layered structure (three layers of an input layer, an intermediate layer, and an output layer) using, for example, back propagation as a learning algorithm. It is a combination of many units (neurons) connected by variable weights.

  For example, both the first and second neural network units 101 and 102 have 16 units of input layers corresponding to peripheral pixels. For the intermediate layer, for example, the first neural network unit 101 includes 24 units, and the second neural network unit 102 includes 16 units, and the output layer includes 5 units each. And one unit. As in the known neural network, each unit is a conversion system having nonlinear input / output characteristics, and each unit is obtained by multiplying an output value from the previous layer by an independent weight. The sum is input, and the sum is nonlinearly transformed and output to the subsequent layer.

  Then, the first and second neural network units 101 and 102 use the learning information signal (that is, the relationship between the pixel to be interpolated and the surrounding pixels, etc.) as described later, and Learning is performed, and a neural network in which weights between the units are predetermined is learned and constructed. That is, the output signal of the neural network is compared with the teacher signal for learning, and the weight between the units is changed based on, for example, the back propagation learning algorithm so that the difference is reduced. The weight between is determined.

  Next, an example of the learning operation of the first neural network unit 101 in the embodiment of the present invention will be described. In the following description, operations related to the flowchart shown in FIG. 3 will be described with reference to the input / output data shown in FIG.

  FIG. 3 is a flowchart for explaining an example of the learning operation of the first neural network unit in the embodiment of the present invention. FIG. 4 shows an artificial generation pattern in the embodiment of the present invention. It is a figure which shows an example of the combination of the input-output data in the area | region divided | segmented.

  In FIG. 3, when the peripheral pixels i00 to i15 are input, the first neural network unit 101 displays an area including the peripheral pixels i00 to i15 and the interpolation pixels o0 to o4 to be output as lines (horizontal in the pixel array). The image is divided into a plurality of areas by line segments oblique to the direction and the vertical direction (step S301).

  That is, as shown in FIG. 4, the first neural network unit 101 divides an area including peripheral pixels i00 to i15 and interpolation pixels o0 to o4 into three areas (areas) A and B with two line segments. , C. These line segments are diagonal lines with respect to the line (horizontal pixel line), for example, a line segment (straight line y) at the position of the width (distance) t on both sides centered on the straight line y = ax + b indicated by the dotted line = Ax + b is a line segment with a slope a of t / 2).

  For the sake of explanation, in FIG. 4, a field screen including peripheral pixels i00 to i15 and interpolation pixels o0 to o4 is displayed with coordinates of x axis and y axis centering on o0, and variables a, b, and t are , -1.5 ≦ a ≦ 1.5, −2 ≦ b ≦ 2, 0 ≦ t ≦ 4, the region including the peripheral pixels i00 to i15 and the interpolation pixels o0 to o4 is divided. Shall. These variables a, b, and t are randomly generated by generating random numbers within the above range, and the region including the peripheral pixels i00 to i15 and the interpolation pixels o0 to o4 is randomly generated. It is divided by various diagonal line segments. Note that the ranges of the variables a, b, and t are examples, and are not limited to these.

  The first neural network unit 101 also performs learning on a line segment close to the vertical direction (a line segment having a very large inclination with respect to the horizontal direction), so that x = ay + b in which x and y are interchanged. Also learn about patterns based on.

  As a result of the region division in step S301, the peripheral pixels i00 to i15 and the interpolation pixels o0 to o4 belong to one of the divided regions. For example, in the example illustrated in FIG. 4, the peripheral pixels i00 and i01 belong to the region (area) A, and the peripheral pixels i02 to i09 and the interpolation pixels o00 to o03 belong to the region (area) B. Peripheral pixels i10 to i15 and interpolation pixel o04 belong to C.

  Subsequently, the first neural network unit 101 generates pixel data in each region so as to be common in each divided region (step S302). That is, when the pixel data is represented by an 8-bit luminance level, a random value is generated from 0 to 255, and three level values are assigned as pixel levels belonging to each of the three regions (areas A to C). Is generated.

  For example, the pixel level of the area (area) A is “40”, the pixel level of the area (area) B is “55”, the pixel level of the area (area) C is “100”, and the like. Are randomly generated. In the above case, the levels of the peripheral pixels i00 and i01 belonging to the region (area) A are “40”, and the levels of the peripheral pixels i02 to i09 and the interpolation pixels o00 to o03 belonging to the region (area) B are “55”. And the levels of the peripheral pixels i10 to i15 and the interpolation pixel o04 belonging to the region (area) C are “100”.

  Then, the pixel data of the peripheral pixels set for each region in step S302 is input to the neural network in the first neural network unit 101, and the output value of the neural network corresponding to the input value is output. Learning of the neural network is executed based on the relationship between the pixel data of the peripheral pixels input to the first neural network unit 101 and the pixel data of the interpolation pixels o0 to o4 in step S302 (step S303). .

  That is, the pixel levels of the peripheral pixels i00 and i01 belonging to the region (area) A are “40”, the levels of the peripheral pixels i02 to i09 and the interpolation pixels o00 to o03 belonging to the region (area) B are “55”, and the region (area) ) The levels of the peripheral pixels i10 to i15 and the interpolation pixel o04 belonging to C are “100”, and an output value is output from the neural network in the first neural network unit 101 based on the weight set so far. The difference between the output value related to the interpolation pixels o00 to o03 and the pixel level “55” of the interpolation pixels o00 to o03, or the difference between the output value related to the interpolation pixel o04 and the pixel level “100” of the interpolation pixel o04 is small. The weight is calculated and changed so that

  Further, the learning by the processes in steps S301 to S303 described above is repeatedly performed for various division examples as illustrated in FIGS. 5A to 5D, for example. As a result, learning is performed on combinations of pixel data of various peripheral pixels. Note that learning by the above-described steps S301 to S303 is performed, for example, a predetermined number of times of learning (step S304), and the neural network in the first neural network unit 101 is gradually constructed by learning. Go.

  As described above, the first neural network unit 101 efficiently learns the interpolation pattern for each area divided by the diagonal line segments and constructs the neural network, so that the horizontal direction and the vertical direction Natural interpolation can be performed with little aliasing of the slanted line segment having a small inclination with respect to each of the above.

  Next, an example of the learning operation of the second neural network unit 102 in the embodiment of the present invention will be described. FIG. 6 is a flowchart for explaining an example of the learning operation of the second neural network unit in the embodiment of the present invention.

  The second neural network unit 102 has a counter for managing the number of learning repetitions, and first resets the value of the counter k (k = 0) with the start of the learning operation (step S401).

  The second neural network unit 102 performs different processing based on the value of the counter k. That is, the second neural network unit 102 determines whether the value of the counter k is even or odd, and when the counter k is an even number (here, k = 0 is determined to be an even number), Similar to the processing in the first neural network unit 101 (steps S301 and S302 in FIG. 3), the signal of the pattern used in the interpolation by the oblique line segment is generated and set (steps S403 and S404), and the output side "1" is given as a teacher signal for learning the neural network (step S405). Then, the counter k is incremented by 1 (k = k + 1) (step S406), and it is determined whether or not the counter k has reached a predetermined number (step S408), and the counter k has not reached the predetermined number. In that case, the process returns to step S402 again.

  On the other hand, when the counter k is an odd number, all input signals (16 values) are set to pixel levels from 0 to 255, which are random numbers generated, and “0” is given as an output teacher signal, Learning is performed (step S409). Then, the counter k is incremented by 1 (k = k + 1) (step S406), and it is determined whether or not the counter k has reached a predetermined number (step S408), and the counter k has not reached the predetermined number. In that case, the process returns to step S402 again. When the counter k reaches a predetermined number of times, the learning operation ends.

  In this way, the second neural network unit 102 causes the neural network to alternately learn an output signal of “1” for interpolation by an oblique line segment and an output signal of “0” for a random pattern. Thus, it is possible to have a distinction ability for determining the adaptability to the input signal pattern by the first neural network unit 102. That is, when the second neural network unit 102 can determine that the suitability of the NN interpolation result by the first neural network unit 101 is high, the second neural network unit 102 outputs an output signal having a value close to “1”. When it is determined that the adaptability of the NN interpolation result by the neural network unit 101 is low, an output signal having a value close to “0” is output.

  The output signal from the second neural network unit 102 is supplied to the mixing ratio determining unit 104, and the mixing ratio determining unit 104 determines the mixing ratio of the NN interpolation result and the bicubic interpolation result based on this output signal. Is done. For example, when the mixture ratio determination unit 104 receives the value “m” (where 0 ≦ m ≦ 1) of the output signal from the second neural network unit 102, “m” is used as the internal division ratio for each interpolation pixel. It is possible to output (1-m) × {NN interpolation result} + m × {bicubic interpolation result} as an interpolation result. The mixing of the NN interpolation result and the bicubic interpolation result is not limited to the calculation using the above-described internal division, and any mixing method can be used.

  As described above, in the second neural network unit 102, a neural network for determining the suitability of the NN interpolation result by the first neural network unit 101 is constructed, and the suitability (discrimination result) of the NN interpolation result is obtained. Will be output. As a result, based on the distinction result from the second neural network unit 102, the mixture ratio determining unit 104 determines that the NN interpolation result by the first neural network unit 101, the bicubic interpolation result by the bicubic interpolation processing unit 103, Can be mixed at an appropriate mixing ratio, and a natural and high-quality interpolation image can be obtained.

  In the above-described embodiment, the mixture ratio determining unit 104 outputs a mixture of the NN interpolation result by the first neural network unit 101 and the bicubic interpolation result by the bicubic interpolation processing unit 103. As described above, based on the suitability of the NN interpolation result output from the second neural network unit 102, the NN interpolation result by the first neural network unit 101 and the bicubic interpolation by the bicubic interpolation processing unit 103 are explained. Either one of the results may be selected and output as an interpolation result. In the above-described embodiment, the bicubic image that executes the bicubic interpolation method is used as the image processing unit that calculates the pixel data of the interpolation pixel by an image processing method different from the image processing method by the first neural network unit 101. Although the processing unit 103 is provided, the processing unit 103 is not necessarily limited to the bicubic interpolation method, and an image processing unit capable of executing any other image interpolation method may be provided.

  In the above-described embodiment, the functions of the image processing apparatus according to the present invention are illustrated by blocks, but these functions can be realized by hardware and / or software (program). The processing of each flowchart used in the above description can also be realized by causing a CPU (Central Processing Unit) to execute a predetermined program, for example. The program for realizing the functions of the image processing apparatus according to the present invention may be read from a recording medium and loaded into a computer, or may be transmitted via a communication network and loaded into a computer.

  The present invention has an effect of making it possible to perform image processing by effectively using a neural network, and image processing technology (especially for mainly changing the resolution and magnification of an image). (Image interpolation technology).

It is a block diagram which shows an example of a structure of the image processing apparatus in embodiment of this invention. It is a figure which shows typically an example of the surrounding pixel input into the image processing apparatus in embodiment of this invention, and the interpolation pixel output from the image processing apparatus. 5 is a flowchart for explaining an example of a learning operation of a first neural network unit in the embodiment of the present invention. In an embodiment of the invention, it is a figure showing an example of a combination of input-output data in a field divided by an artificial generation pattern. In an embodiment of the invention, it is a figure showing the 1st example of the example of division to an input pixel. In an embodiment of the invention, it is a figure showing the 2nd example of the example of division to an input pixel. In an embodiment of the invention, it is a figure showing the 3rd example of the example of division to an input pixel. In an embodiment of the invention, it is a figure showing the 4th example of the example of division to an input pixel. 6 is a flowchart for explaining an example of a learning operation of a second neural network unit in the embodiment of the present invention.

Explanation of symbols

101 First neural network section (first NN section)
102 Second neural network section (second NN section)
103 Bicubic interpolation processing unit 104 Mixing ratio determining unit

Claims (2)

An image processing apparatus having pixel data of a plurality of pixels as an input value and pixel data of an interpolation pixel for interpolating the plurality of pixels as an output value,
The first neural network is a means configured to output first pixel data of interpolated pixels for interpolating the plurality of pixels with respect to pixel data of the plurality of pixels, and the first neural network is generated in advance. A first image processing means that is a neural network in which a relationship between pixel data of a plurality of pixels and pixel data of an interpolation pixel corresponding to the pixel data of the plurality of pixels is learned;
Second image processing means for executing image processing different from the first image processing means and outputting second pixel data of interpolated pixels for interpolating the plurality of pixels with respect to pixel data of the plurality of pixels;
Means configured by a second neural network for determining suitability of image processing by the first image processing means for pixel data of the plurality of pixels and outputting a value indicating suitability of the first image processing means; The second neural network includes first pixel data of the plurality of previously generated pixels used for learning of the first neural network, and first information indicating that the first neural network has been learned. A relationship with a specific value is learned in advance, and a relationship between pixel data of a plurality of randomly generated pixels and a specific value indicating that learning has not been performed by the first neural network is learned. A fitness determination means that is a neural network;
Based on the value indicating the suitability of the first image processing means output from the suitability judging means, the first pixel data outputted from the first image processing means and the output from the second image processing means Pixel value mixing means for mixing the second pixel data and interpolating the plurality of pixels to output pixel data of an interpolation pixel as the output value,
An image processing apparatus. An image processing program for causing a computer to execute an image processing method having pixel data of a plurality of pixels as input values and pixel data of interpolation pixels for interpolating the plurality of pixels as output values,
Realization of a first image processing mechanism that realizes a first image processing mechanism configured by a first neural network that outputs first pixel data of interpolation pixels that interpolate the plurality of pixels with respect to pixel data of the plurality of pixels. Steps,
A first learning step in which the first image processing mechanism learns a relationship between pixel data of a plurality of pixels generated in advance and pixel data of an interpolation pixel corresponding to the pixel data of the plurality of pixels;
A second image processing mechanism that executes image processing different from the first image processing means and outputs second pixel data of interpolated pixels for interpolating the plurality of pixels with respect to pixel data of the plurality of pixels is realized. A second image processing mechanism realizing step,
Consistency constituted by a second neural network that determines the suitability of image processing by the first image processing mechanism for pixel data of the plurality of pixels and outputs a value indicating the suitability of the first image processing mechanism. A compatibility judgment mechanism realization step for realizing the judgment mechanism;
The conformity determination mechanism includes pixel data of the plurality of previously generated pixels used for learning of the first neural network, and a first specific value indicating that the first neural network has been learned. Is learned in advance, and learns a relationship between pixel data of a plurality of randomly generated pixels and a second specific value indicating that the first neural network has not been learned. Learning steps,
The first pixel data output from the first image processing mechanism and the output from the second image processing mechanism based on a value indicating the compatibility of the first image processing mechanism output from the compatibility determination mechanism. A pixel value mixing step of mixing the second pixel data and interpolating the plurality of pixels to output pixel data of an interpolation pixel as the output value,
An image processing program for causing a computer to execute an image processing method.

Images