JPWO2020062846A5 - - Google Patents

Description

Mutual reference of related applications This application has priority based on Chinese patent application No. 2018111552522.6 filed on September 30, 2018 and Chinese patent application No. 201811155326.6 filed on September 30, 2018. Priority based on, priority based on Chinese patent application No. 201811155147.2 filed on September 30, 2018, and priority based on Chinese patent application No. 201811155930.9 filed on September 30, 2018. Is claimed, the content of which is incorporated herein by reference in its entirety.

The present disclosure relates generally to the field of deep learning techniques, and more particularly to image processing techniques based on deep learning, including devices and methods for image processing discrimination networks and computer readable media.

Deep learning techniques based on artificial neural networks have made great strides in fields such as image processing. The advantage of deep learning techniques lies in the solution of different technical problems utilizing general-purpose structures and relatively similar systems.

An embodiment of the present disclosure is an apparatus that generates a plurality of correlation images. The apparatus includes a feature extraction unit configured to receive a training image, extract at least one feature from the training image, and generate a first feature image based on the training image, and the first feature extraction unit. A normalizer configured to normalize the feature image and generate a second feature image, and a plurality of translational shifts on the second feature image to generate a plurality of shifted images. It may include a shift correlation unit configured to correlate each of the plurality of shifted images with the second feature image to generate a plurality of correlation images.

In at least some embodiments, the shift correlation unit places the leftmost or rightmost pixel in column a in the pixel block of the second feature image on the rightmost or first of the pixel block, respectively. Shifting to the left column, and shifting the pixels in the lowest or highest b row in the pixel block of the second feature image to the highest or lowest row of the pixel block, respectively. Can be configured to perform the plurality of translational shifts on the second feature image. In at least some embodiments, 0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, where Y is the total sequence of pixels in the pixel block of the second feature image. A number, where X is the total number of rows of pixels in the pixel block of the second feature image, and a and b are the same or different.

In at least some embodiments, the shift correlation unit deletes the leftmost or rightmost pixel in column a in the pixel block of the second feature image, and the rightmost or rightmost pixel block. Add the pixels in column a to the leftmost position, delete the lowest or top b row pixels in the pixel block of the second feature image, and delete the top or bottom of the pixel block. By adding each of the b-row pixels at the position of , the second feature image may be configured to perform the translational shift a plurality of times. In at least some embodiments, 0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, where Y is the total sequence of pixels in the pixel block of the second feature image. A number, where X is the total number of rows of pixels in the pixel block of the second feature image, and each of the added pixels has a pixel value of 0 .

In at least some embodiments, the shift correlation unit is a pixel that corresponds positionally within the pixel block of the second feature image to the pixel value of each pixel in each pixel block of the plurality of shifted images. By multiplying the pixel values of, each of the plurality of shifted images may be configured to correlate with the second feature image. In at least some embodiments, the first feature image can be a luminance feature image. In at least some embodiments, the feature extraction unit may include a luminance detector configured to extract luminance information from the training image to generate the luminance feature image.

In at least some embodiments, in order to generate the luminance feature image, the luminance detector determines the luminance value of the pixel at a given position in the luminance feature image by the following equation (1). Consists of

I = 0.299R + 0.587G + 0.114B (1)

I is the luminance value. R is the red component value of the pixel corresponding to the position in the training image. G is a green component value of the pixel corresponding to the position in the training image. B is the blue component value of the pixel corresponding to the position in the training image.

In at least some embodiments, the normalizer may be configured to normalize the luminance feature image by the following equation (2).

N is the first feature image. I represents the luminance value of the pixel at a given position in the luminance feature image. Blur (I) is an image obtained by applying a Gaussian filter to the luminance feature image. Blur (I ² ) is an image obtained by squaring each pixel value in the luminance feature image and then applying a Gaussian filter to the luminance feature image.

In at least some embodiments, the second feature image may include pixel blocks having a first size. Each of the plurality of shifted images and each of the plurality of correlated images may include a pixel block having the first size. In each of the plurality of shifted images, a pixel having a non-zero pixel value may have a corresponding pixel having the same non-zero pixel value in the second feature image.

Another embodiment of the present disclosure is a method of generating a plurality of correlated images. The method includes a step of generating a first feature image based on a training image, a step of normalizing the first feature image and generating a second feature image, and a plurality of translations with respect to the second feature image. It may include a step of performing a shift to generate a plurality of shifted images, and a step of correlating each of the plurality of shifted images with the second feature image to generate a plurality of correlated images.

In at least some embodiments, the step of correlating each of the plurality of shifted images with the second feature image is said to the pixel value of each pixel in each pixel block of the plurality of shifted images. The second feature may include a step of multiplying the pixel values of the corresponding pixels in position within the pixel block of the image.

In at least some embodiments, the step of performing the multiple translational shifts places the leftmost or rightmost pixel in column a in the pixel block of the second feature image at the top of the pixel block, respectively. The step of shifting to the right or leftmost column and the lowest or highest b row of pixels in the pixel block of the second feature image are the highest or lowest row of the pixel block, respectively. It may include a step that shifts to. In at least some embodiments, 0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, where Y is the total sequence of pixels in the pixel block of the second feature image. A number, where X is the total number of rows of pixels in the pixel block of the second feature image, and a and b are the same or different. In at least some embodiments, at least one of a and b may change at least once during the execution of the plurality of translational shifts.

In at least some embodiments, the step of performing the plurality of translational shifts removes the leftmost or rightmost column a pixel in the pixel block of the second feature image and removes the pixel of the pixel block. The step of adding the pixel of column a at the rightmost or leftmost position, and the pixel of the lowest or highest row b in the pixel block of the second feature image are deleted, and the pixel of the pixel block is deleted. It may include a step of adding rows b of pixels at the top or bottom positions, respectively. In at least some embodiments, 0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, where Y is the total sequence of pixels in the pixel block of the second feature image. It is a number, and X is the total number of rows of pixels in the pixel block of the second feature image. In at least some embodiments, each of the added pixels can have a pixel value of 0. In at least some embodiments, at least one of a and b may change at least once during the execution of the plurality of translational shifts.

In at least some embodiments, the method may further include a step of performing XY translational shifts, where Y is the total number of rows of pixels in the pixel block of the second feature image and X is said. The total number of rows of pixels in the pixel block of the second feature image.

In at least some embodiments, the method may further include receiving the training image prior to generating the first feature image. In at least some embodiments, the step of generating the first feature image may include the step of generating a luminance feature image based on the luminance information of the training image.

In at least some embodiments, the method may further comprise the step of determining the luminance value of a pixel at a given position in the luminance feature image by the following equation (1).

I = 0.299R + 0.587G + 0.114B (1)

I is the luminance value. R is the red component value of the pixel corresponding to the position in the training image. G is a green component value of the pixel corresponding to the position in the training image. B is the blue component value of the pixel corresponding to the position in the training image.

In at least some embodiments, the method may further include the step of normalizing the luminance feature image by the following equation (2).

N is the first feature image. I represents the luminance feature image. Blur (I) is an image obtained by applying a Gaussian filter to the luminance feature image. Blur (I ² ) is an image obtained by squaring each pixel value in the luminance feature image and then applying a Gaussian filter to the luminance feature image.

In at least some embodiments, the first feature image may include pixel blocks having a first size. In at least some embodiments, each of the plurality of shifted images and each of the plurality of correlated images may include a pixel block having the first size. In at least some embodiments, in each of the plurality of shifted images, a pixel having a non-zero pixel value may have a corresponding pixel having the same non-zero pixel value in the first feature image.

Another embodiment of the present disclosure is a non-temporary computer-readable medium that stores instructions that cause a computer to perform a method of generating multiple correlated images. The method may be as described above.

Another embodiment of the present disclosure is a system for training a hostile generation network. The system may include a generated network microprocessor configured to be trained by a differential network microprocessor and a hostile generated network processor including a differential network microprocessor coupled to the generated network.

In at least some embodiments, the differential network microprocessor has a plurality of input ends coupled to a plurality of devices, each of which produces a plurality of correlated images, each of which may be as described above. A plurality of analysis modules coupled to one of the input ends and a pooling module in which each stage of the cascade is coupled to a pooling module in one of the plurality of analysis modules and the previous stage of the cascade. The cascade may include a plurality of connected pooling modules and a discriminant network coupled to the pooling modules at the last stage of the cascade.

The subject matter considered to be the invention is specifically noted and expressly claimed in the claims at the end of the specification. The aforementioned and other objectives, features and advantages of the present disclosure will become more apparent from the following detailed description, which will proceed in conjunction with the accompanying drawings. The drawings are as follows.

The block diagram of the apparatus for image processing which concerns on embodiment of this disclosure is shown. The schematic diagram of the 3 * 3 pixel block in the 1st feature image which concerns on embodiment of this disclosure is shown. Shown are 3 * 3 pixel blocks in each of the nine shifted images obtained by shifting the first feature image illustrated in FIG. 2 according to an embodiment of the present disclosure. Shown are 3 * 3 pixel blocks in each of the nine shifted images obtained by shifting the first feature image illustrated in FIG. 2 according to another embodiment of the present disclosure. The discrimination network according to the embodiment of this disclosure which can be coupled to the apparatus for image processing which concerns on this disclosure is shown. The flowchart of the method for image processing which concerns on embodiment of this disclosure is shown. A flowchart of a method for image processing according to another embodiment of the present disclosure is shown. A block diagram of a system for training a neural network according to an embodiment of the present disclosure is shown.

The various features of the drawings are not drawn to a certain scale, as the illustrations are intended to provide clarity in facilitating the understanding of the invention, along with detailed explanations by those skilled in the art.

Next, the embodiments of the present disclosure will be described clearly and concretely together with the accompanying drawings briefly described above. The subject matter of this disclosure is described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors may otherwise embody the claimed subject matter in conjunction with current or future techniques to include different steps or elements similar to the steps or elements described in this document. think of.

Although the technique has been described in relation to various embodiments of the drawings, it is important to understand that other similar embodiments have been used to perform the same function of the technique without departing from the technique. Changes and additions to the embodiments obtained or described may be implemented. Therefore, the art should not be limited to any single embodiment, but should be construed in accordance with the breadth and scope of the appended claims. In addition, all other embodiments obtained on the basis of the embodiments described in this document by those having ordinary knowledge in the art are considered to be within the scope of this disclosure.

Deep learning techniques based on artificial neural networks have made great strides in fields such as image processing. Deep learning is a learning method based on data characterization in machine learning methods. Observations (eg, images) can be represented by a variety of methods as vectors of intensity values of various pixels, or more abstractly, as a series of edges, regions with a particular shape, and the like. The advantage of deep learning techniques lies in the solution of different technical problems utilizing general-purpose structures and relatively similar systems. The advantage of deep learning is to replace manual feature acquisition with an efficient unsupervised or semi-supervised algorithm for feature learning and hierarchical feature extraction.

Images in nature can be easily distinguished from images created synthetically by humans or randomly by computers. Natural images are characteristic because they contain at least certain structures and are very non-random. For example, images generated synthetically and randomly by a computer contain few natural scenes or objects.

Image processing systems such as compression algorithms, analog storage media, and even human own visual systems work for real-world images. Generative adversarial networks (GANs) are a solution for generating realistic samples of natural images. GAN can be an approach to production modeling in which two models are trained simultaneously or cross-trained.

The learning system is configured to adjust the parameters based on a particular target and can be represented by a loss function. In GAN, the loss function is replaced by another machine learning system that can independently learn difficult tasks. The GAN usually includes an adversarial network that opposes the differential network. The generation network receives the input of the low resolution data image, upscales the low resolution data image, and sends the upscaled image to the discrimination network. The discriminant network is tasked with classifying whether its input is the output of the generated network (ie, a "fake" upscaled data image) or an actual image (ie, the original high resolution data image). Will be. The discrimination network outputs a score between "0" and "1" that measures the probability that the input is an upscaled image and an original image. When the discrimination network outputs a score approaching "0" or "0", the discrimination network determines that the image is the output of the generation network. When the discrimination network outputs a numerical value approaching "1" or "1", the discrimination network determines that the image is an original image. Such an adversarial network opposes the discriminative network, and thus the "hostile" way takes advantage of the competition between the two networks until the images produced by the adversarial network are indistinguishable from the original. Drives to improve those methods.

The discrimination network may be trained to score the input as "real" or "fake" with data having a given score. The "fake" data can be a high resolution image generated by the generation network and the "real" data can be a given reference image. To train the discriminant network, whenever the discriminant network receives "real" data, it outputs a score that approaches "1", and whenever it receives "fake" data, it outputs a score that approaches "0". Until, the parameters of the discrimination network are adjusted. To train the generation network, the parameters of the generation network are adjusted until the output of the generation network receives a score as close as possible to "1" from the discrimination network.

The universal analogy of GAN is counterfeiters and police. The generation network is analogized by counterfeiters and attempts to manufacture counterfeit money and use it without detection, whereas the discrimination network is analogized by police and may attempt to detect the counterfeit money. Competition between counterfeiters and police encourages both sides to improve their methods until they are indistinguishable from the real thing.

Both the generation network and the discrimination network try to optimize the objective function, that is, the loss function, which is different and conflicts in the zero-sum game. Maximize the output of the discriminant network through "cross-training", the spawning network improves the images produced by the spawning network, and the discriminant network improves the accuracy of the distinction between the original high resolution image and the image produced by the spawning network. .. The generation network and the discrimination network compete to generate better images and raise the criteria for evaluating the images.

In order to train to improve the generation network at certain parameters, it remains necessary to increase the accuracy of the differential network in distinguishing between the original high resolution image and the image generated by the generation network. For example, I'm interested in the task of generating an image that is perceived as real and undamaged. This can be applied to problems such as blur removal, denoising, demosaic processing, decompression, contrast enhancement, image superresolution, and so on. In such problems, the corrupted image is visually impaired and a machine learning system can be designed to repair it. However, the goal of recovering the original image is often unrealistic, leading to images that do not look real. GAN is designed to produce "real" images. A typical configuration takes a color output image and uses a machine learning system (eg, a convolutional network) to output a single number that measures how realistic the image is. Although this system can improve perceptual quality, the output of hostile systems is still insufficient to be recognized as a natural image by human viewers.

FIG. 1 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

The block diagram of FIG. 1 is not intended to show that the device 100 includes only the components shown in FIG. Rather, device 100 is an arbitrary number of additional accessories and / or not shown in FIG. 1, known to those of ordinary skill in the art, depending on the specific implementation details. Can include components.

As shown in FIG. 1, the device 100 includes a feature extraction unit 110 and a shift correlation unit 120.

The feature extraction unit 110 is configured to extract one or more features from a training image input to or received by the device 100 and generate a feature image based on the extracted features. .. The feature image represents one or more features of the training image. The training image can be an image generated by the generation network or a predetermined reference image.

In some embodiments, the feature extraction unit 110 may include a luminance detector 111, as shown in FIG.

The luminance detector 111 is configured to generate a first feature image of the training image, for example, by extracting information about the luminance in the training image from the training image. Therefore, the first feature image may also be referred to as a luminance feature image.

In some embodiments, the feature extraction unit 110 may include a normalizer 112, as shown in FIG.

The normalizer 112 is configured to generate a second feature image by normalizing the first feature image. In the embodiment in which the first feature image is a luminance feature image, the normalizer 112 is configured to normalize the luminance feature image. Normalization allows the pixel values of an image to fall within a smaller range of values and removes outlier pixel values that are too high or too low. This can eventually facilitate the calculation of correlations, as discussed below.

The image processing apparatus 100 according to the present disclosure includes a general-purpose computer, a microprocessor, a digital electronic circuit, an integrated circuit, a particularly designed ASIC (application specific integrated circuit), computer hardware, firmware, software, and / or. It can be implemented in a computing device in the form of that combination.

The second feature image generated by the feature extraction unit 110 is output to the shift correlation unit 120 for further processing. The shift correlation unit 120 is configured to perform a plurality of translational shifts of the second feature image to generate a plurality of shifted images. The shift correlation unit 120 is further configured to generate a plurality of correlation images based on a set of correlations between each of the second feature image and the plurality of shifted images. The shift correlation unit 120 is further configured to transmit the plurality of correlation images to the deep learning network in order to train the deep learning network. For example, in some embodiments, the plurality of correlation images may be transmitted to the differential network in the hostile generation network to iteratively train the discrimination network with the generation network in the hostile generation network.

The second feature image has a first size pixel block defined by pixels in a first quantity of rows and pixels in a first quantity of columns. The second feature image occupies a first region corresponding to the first size prior to the plurality of translational shifts. Translation shifts can be achieved in several ways. In some embodiments, the translational shift moves the pixels in the second feature image from the initial region in the row (or horizontal) or column (or vertical) direction. In some embodiments, the translational shift removes the rows and / or columns of the pixels shifted outside the first region and values "0" for the pixels in the space vacated by the shifted pixels. Can include assigning. In some embodiments, the translational shift may include rearranging or rearranging the rows and / or columns of pixels.

Each of the plurality of shifted images has a pixel block having the same size as the first size of the pixel block in the second feature image. Each of the plurality of shifted images has the same number of rows of pixels and the same number of columns of pixels as the second feature image.

Each pixel with a non-zero value in each shifted image has a corresponding pixel with the same non-zero value in the second feature image. In at least some embodiments, a pixe having no corresponding pixel in the second feature image is assigned a value of "0". As an exemplary example, the pixel values in the first two rows of the shifted image are the same as the corresponding pixel values in the last two rows of the first feature image, respectively, and the other in the shifted image. All pixels are assigned a value of "0". Each pixel in the shifted image having the corresponding pixel in the second feature image has the same pixel value as the corresponding pixel.

In the present disclosure, the "corresponding pixel" is not limited to the pixel corresponding to the position, but may include the pixel occupying a different position. "Corresponding pixel" refers to a pixel having the same pixel value.

In the present disclosure, the image is treated as a pixel block. The value of the pixel in the block represents the value of the pixel in the image that corresponds positionally to the pixel in the block.

The correlation between two images can be calculated by pixel-to-pixel multiplication of the pixel blocks of the two images. For example, the pixel values in the i-th row and j-th column (i, j) of the correlation image are shifted to the pixel values at the (i, j) position in the second feature image (in the image). i, j) It can be determined by multiplying the value of the pixel at the position.

As shown in FIG. 1, in some embodiments, the feature extraction unit 110 includes a luminance detector 111 and a normalizer 112.

The luminance detector 111 generates a first characteristic image by extracting information on the luminance in the training image from the training image received by the feature extraction unit 110, and the luminance feature image is based on the extracted luminance information. Is configured to generate. Therefore, the first feature image is also referred to as a luminance feature image. The human eye tends to be more sensitive to the brightness of the image than other features. By extracting the luminance information, the apparatus of the present disclosure can remove unnecessary information from the training image and reduce the processing load.

The number of rows and columns of pixels in the luminance feature image is the same as in the training image. The luminance value I of the pixel in the i-th row and the j-th column (i, j) of the luminance feature image can be calculated by the following equation (1).

I = 0.299R + 0.587G + 0.114B (1)

In the formula (1), R represents the red component value of the pixel (i, j) in the training image. G represents a green component value. B represents the blue component value. Both i and j are integers. The value of i is 1 ≦ i ≦ X. The value of j is 1 ≦ j ≦ Y. X is the total number of rows in the training image, and Y is the total number of columns in the training image.

In some embodiments, the training image is a color image. In some embodiments, the training image comprises an R component, a G component, and a B component, wherein the apparatus of the present disclosure comprises the R component, the G component, and the B component being the brightness detector. Is configured to process the training image so that it is input to the Y, U, and V components, respectively, and then input to the Y, U, and V channels, respectively. obtain. The Y component, the U component, and the V component are components of the training image in the YUV space. The Y channel, the U channel, and the V channel indicate that the outputs from these channels are the Y component output, the U component output, and the V component output, respectively. In an embodiment in which the RGB component of the training image is converted to a YUV component, the luminance value I corresponds to the value of the Y component.

In some embodiments, the training image has a Y component, a U component, and a V component. In that case, the apparatus of the present disclosure processes the Y component of the training image via the Y channel of the luminance detector, processes the U component of the training image via the U channel of the luminance detector, and the said. It may be configured to process the V component of the training image via the luminance detector V channel.

In some embodiments, using YUV space is to perform chroma sampling on the training image. The Y component of the training image enters the Y channel. The U component of the training image enters the U channel. The V component of the training image enters the V channel. By dividing the input signal of the training image into three groups, each channel processing signal in the components from the Y component, the U component, and the V component group reduces the calculation load and improves the processing speed. Can be. Since the U component and the V component have a relatively low effect on the display effect of the image, processing different components on different channels does not significantly affect the image display.

The normalizer 112 is configured to generate a second feature image by normalizing the first feature image. In an embodiment in which the feature extraction unit 110 includes a luminance detector 111 and the first feature image is a luminance feature image, the normalizer 112 is configured to normalize the luminance feature image. Normalization allows the pixel values of an image to fall within a smaller range of values and removes outlier pixel values that are too high or too low. This can, in the end, facilitate the calculation of correlations.

More specifically, the normalizer 112 is configured to perform normalization according to the following equation (2) to obtain a second feature image.

In the formula (2), N represents a second feature image. I represents a luminance feature image obtained from the training image. Blur represents Gaussian blur. Blur (I) represents a Gaussian blur filter performed on the luminance feature image. Blur (I ² ) represents an image obtained by squaring each pixel value in the luminance feature image and then performing a Gaussian blur filter on the luminance feature image. μ represents the output image obtained using the Gaussian blur filter. σ ² represents a locally distributed normalized image.

In some embodiments of the present disclosure, the translational shift of the second feature image comprises shifting the pixels in the last a column of the second feature image before the pixels in the remaining columns to obtain an intermediate image. .. Then, the pixel in the last b row in the intermediate image is shifted before the pixel in the remaining row to obtain a shifted image. The value of a is 0 ≦ a <Y. The value of b is 0 ≦ b <X. Both a and b are integers. X represents the total number of rows of pixels in the second feature image. Y represents the total number of columns of pixels in the second feature image. The value of a and the value of b may be the same or different. When both a and b are zero, the shifted image is the second feature image. In some embodiments, at least one value of a and b changes in any two given image shift processes. It is understandable that the order in which the shifts are made is not particularly limited. For example, in some embodiments, the pixels in a row can be shifted to obtain an intermediate image, and the pixels in a column can be shifted to obtain a shifted image.

The value of each pixel in the shifted image corresponds to the value of the pixel in the second feature image. The value of the pixel (i, j) in each of the plurality of shifted images is derived from different pixels at different positions in the second feature image.

In some embodiments, the translational shift of the first feature image comprises shifting the pixels in the last b row in the second feature image before the pixels in the remaining rows to obtain an intermediate image. Then, the pixel in the last row a in the intermediate image is shifted before the pixel in the remaining row to obtain a shifted image.

In some embodiments, the second feature image is subjected to XY translational shifts to obtain XY correlation images. Even if both a and b are zero, this is also counted as one translational shift.

The block diagram of FIG. 1 is not intended to show that the device 100 includes only the components shown in FIG. Rather, device 100 is an arbitrary number of additional accessories and / or not shown in FIG. 1, known to those of ordinary skill in the art, depending on the specific implementation details. Can include components.

FIG. 2 shows a schematic diagram of a 3 * 3 pixel block in the second feature image according to the embodiment of the present disclosure. In FIG. 2, “p1” ... “p9” each represent a value of one of nine pixels. FIG. 3 shows a 3 * 3 pixel block in each of the nine shifted images obtained by shifting the second feature image illustrated in FIG. 2 according to the embodiment of the present disclosure.

In the embodiments of the present disclosure, the second feature image comprises a pixel block having a first size. Each of the plurality of shifted images and each of the plurality of correlated images includes a pixel block having the first size.

For the purposes of the present disclosure, the pixels in the top row in the block illustrated in FIG. 2 are in the first row, and the pixels in the leftmost column in the block illustrated in FIG. 2 are in the first column. Is. When a = 1 and b = 1, the shifted image shown in the center of the second row in FIG. 3 is obtained, and the pixels in the last column (that is, the rightmost column) in the second feature image. Is moved before the pixels in the first column (ie, the leftmost column), and the pixels in the last row (ie, the bottom row) are the pixels in the first row (ie, the top row). Move in front of.

In the embodiments illustrated in FIGS. 2 and 3, a pixel can occupy one of nine positions within a block, with nine shifts in the likelihood that each pixel will appear in each of the nine positions. It is reflected in the image. The nine correlated images then include not only the correlation of each pixel with itself, but also the correlation of each pixel with the other pixels in the image. In an exemplary example of a hostile generated network, where the generated network produces an image in which the value of one pixel differs from the high resolution original (“real”) image, it is based on the synthetically generated image. Each of the obtained correlated images shows a discrepancy with the correlated image of the high resolution original image. This discrepancy causes the differential network to score synthetically generated images closer to "0" (ie, "fake" classification), and the generated network produces a more realistic and perceptually convincing output. Drive to update and improve.

It is understandable that the present disclosure does not limit the translational shifts that can be applied to the image. FIG. 4 shows a 3 * 3 pixel block in each of the nine shifted images obtained after shifting the second feature image illustrated in FIG. 2 according to another embodiment of the present disclosure.

In FIGS. 2 and 4, the last pixel in column a in the second feature image is removed, and the pixel in column a is added in front of the pixel in the remaining column to obtain an intermediate image. Each pixel in the added column a has a value of "0". Next, in the intermediate image, the pixel in the last b row is removed, and the pixel in the b row is added before the pixel in the remaining row to obtain a shifted image. Each pixel in the added column b has a value of "0". More specifically, 0 ≦ a <Y, 0 ≦ b <X, and both a and b are integers. X represents the total number of rows of pixels in the second feature image. Y represents the total number of columns of pixels in the second feature image. The value of a and the value of b may be the same or different. In some embodiments, at least one value of a and b changes in any two given image shift processes.

The shift correlation unit 120 is configured to generate a correlation image by multiplying the values of the pixels at the corresponding positions in the two images. In the correlation image, the pixel value at the (i, j) position is multiplied by the pixel (i, j) value in the second feature image and the pixel (i, j) value in the shifted image. Obtained by that. The value of i is 1 ≦ i ≦ X. The value of j is 1 ≦ j ≦ Y. Both i and j are integers. X represents the total number of rows of pixels in the second feature image. Y represents the total number of columns of pixels in the second feature image.

The image processing apparatus 100 according to the present disclosure includes a general-purpose computer, a microprocessor, a digital electronic circuit, an integrated circuit, a particularly designed ASIC (application specific integrated circuit), computer hardware, firmware, software, and / or. It can be implemented in a computing device in the form of that combination. These diverse implementations include implementations in one or more computer programs that are executable and / or interpretable in a programmable system that includes at least one programmable processor, wherein the at least one programmable processor may be dedicated or general purpose. And coupled to receive data and instructions from the storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, at least one input device, and at least one output device. obtain.

These computer programs (also referred to as programs, software, software applications or codes) may include programmable processor machine instructions and may be implemented in high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine readable medium", "computer readable medium" are machine instructions and / or to programmable processors including machine readable media that receive machine instructions as machine readable signals. Refers to any computer program product, device and / or device (eg, magnetic disk, disk disk, memory, programmable logic device (PLD)) used to provide data. The term "machine readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

To provide interaction with the user, the devices, systems, processes, functions, and techniques described in this document are display devices (eg, CRTs) or LCDs (eg, CRTs) or LCDs for displaying information to the user. It can be performed on a computer with a liquid crystal display) monitor), as well as a keyboard and pointing device (eg, a mouse or trackball) that allows the user to provide input to the computer. Other types of accessories and / or devices may be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback). Inputs from the user can be received in any form, including acoustic, audio or tactile inputs.

The devices, systems, processes, functions, and techniques described above include back-end components (eg, as data servers), or include middleware components (eg, application servers), or front-end components (eg, the user said above). A computing that includes (a client computer with a graphical user interface or web browser capable of performing and interacting with its equipment, systems, processes, functions, and techniques), or a combination of such backend, middleware, or frontend components. Can be implemented in the system. The components of the system may be interconnected by any form or medium of digital data communication (communication network, etc.). Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), and the Internet.

The computing system may include a client and a server. The client and the server are usually separated from each other and generally interact with each other via a communication network. The client-server relationship arises from computer programs that run on their respective computers and have a client-server relationship with each other.

The image processing apparatus according to the present disclosure may be coupled to a neural network and may be configured to train the neural network. In some embodiments, the apparatus according to the present disclosure is configured to train a hostile generation network (GAN). The GAN may include a generation network and a discrimination network.

The discrimination network is known to those having ordinary knowledge in the art as long as the discrimination network can classify the degree of matching between the image received as an input and a predetermined reference image having the same resolution as the input image. It can be constructed and configured in any suitable way. FIG. 5 shows the discrimination network 200 according to the embodiment of the present disclosure. The discrimination network 200 may include a plurality of input terminals In1, In2, In3, a plurality of analysis modules 210, a plurality of pooling modules 220, and a discrimination module 230.

Each of the plurality of analysis modules 210 is coupled to the corresponding one of the plurality of input ends In1, In2, In3. The analysis module 210 receives a plurality of correlation images generated by the apparatus according to the present disclosure via the input terminals In1, In2, and In3. The analysis module 210 is configured to generate a plurality of corresponding third feature images based on the plurality of correlation images. Each of the plurality of third feature images is a multi-channel image representing different dimensions of the corresponding correlation image. Each of the plurality of third feature images has a larger number of channels than the corresponding correlated image. For example, the input correlation image may have 3 channels and the output 3rd feature image may have 64 channels, 128 channels or any number of other channels. Each of the plurality of third feature images is generated at the same resolution as the corresponding correlated image.

Each of the plurality of analysis modules 210 is coupled to one of the plurality of pooling modules 220. The plurality of pooling modules 220 are cascaded. The pooling module 220 is configured to receive a plurality of input images, generate a merged image by concatenating the plurality of input images, and reduce the resolution of the merged image to generate a downscaled merged image. Will be done. More specifically, the plurality of input images include a third feature image received from the corresponding analysis module 210 and a reference image. As shown in FIG. 5, in the first stage of the cascade, the third feature image from the analysis module 210 also serves as a reference image for the corresponding pooling module 220. In the subsequent stages of the cascade, the reference image is a downscaled merged image produced by the pooling module in the previous stage of the cascade.

The discrimination module 230 receives a merged image downscaled from the pooling module 220 in the final stage of the cascade and matches the received image with a predetermined reference image having the same resolution as the received image. It is configured to classify the downscaled merged images received by generating a score representing the degree.

The generation network can be constructed and configured by any suitable method known to those of ordinary skill in the art, as long as the generation network can upscale and generate images.

The device 100 may be coupled to the discrimination network via the input end of the discrimination network. The discrimination network does not have to directly receive the output image from the generation network or the high resolution original sample image. Rather, the discrimination network may be configured to receive, classify, and score the output images from the generation network, or high resolution original sample images, after being preprocessed by the device 100. In other words, the discrimination network may be configured to receive, classify and score the output from device 100.

The conventional method of training GANs is to send the output image from the generation network or the original sample image directly to the discrimination network for classification. As a result, for the purpose of classification, the discrimination network is limited to relying on the information in the output image or the original sample image.

In the image processing apparatus according to the present disclosure, the shift correlation unit processes the output image from the generation network and / or the high resolution original image to generate a plurality of correlation images. For example, the shift correlation unit may include not only information specific to the output image and / or the original sample image, but also information about the correlation between those images and the shifted or otherwise converted image. It is configured to generate multiple correlation images, including. Compared to conventional methods, the discriminant network in the system of the present disclosure comprises, for example, a set of correlations between the output image from the generation network and the converted image, and the original sample image and the converted image. It provides additional information for making the classification by comparing with a set of correlations between. In addition, the natural image evaluation (NIQE) non-reference quality score suggests that the correlation between the output image (or the original sample image) and the converted image may affect the perceptual quality.

Compared with the conventional method, the classification based on the output from the image processing apparatus of the present disclosure improves the accuracy of the classification, improves the accuracy of the classification result, and is very similar to the actual image, so that it is classified by the discrimination network. Train the parameters of the generated network towards the creation of less likely solutions. This encourages a perceptually superior solution.

The present disclosure further provides a method for image processing. FIG. 6 shows a flowchart of the method for image processing according to the embodiment of the present disclosure.

Step S1 includes, for example, obtaining a first feature image by generating a luminance feature image based on the extracted luminance information of the training image.

Step S2 includes a step of normalizing the first feature image to obtain a second feature image.

Step S3 includes a step of performing a plurality of translational shifts on the second feature image to obtain a plurality of shifted images. Each shifted image has the same number of rows and columns of pixels as the second feature image. Each pixel with a non-zero value in each shifted image has a corresponding pixel with the same non-zero value in the second feature image. Pixels that do not have a corresponding pixel in the second feature image may be assigned a value of "0". In other words, each pixel with a non-zero value in the shifted image has a corresponding pixel in the second feature image.

Step S4 includes generating a plurality of correlated images based on the correlation between the second feature image and the plurality of shifted images. Each correlation image has the same number of rows and columns of pixels as the second feature image.

Step S5 includes, for example, transmitting the plurality of correlation images to a neural network such as a discrimination network of a hostile generation network.

The method according to the present disclosure may be configured to train a neural network. In some embodiments, the methods according to the present disclosure are configured to train a hostile generation network (GAN). The GAN may include a generation network and a discrimination network. The conventional method of training GANs is to send the output image from the generation network or the original sample image directly to the discrimination network for classification. As a result, for the purpose of classification, the discrimination network is limited to relying on the information in the output image or the original sample image.

Compared to conventional techniques, the methods of the present disclosure do not send the output image from the generation network or the high resolution original image directly to the discrimination network. Rather, the image is processed by the above-mentioned device, including a feature extraction unit and a shift correlation unit, before being sent to the discrimination network for classification. The shift correlation unit produces a plurality of transformed images. For example, the shift correlation unit produces a plurality of correlation images that include not only information specific to the output image and / or the original sample image, but also information about the correlation between those images and the converted image. It is configured to do. This additional information is based on the similarity between the two sets of correlations of the differential network, i.e., one set of correlations between the output image from the generation network and the transformed image. The classification is to be based on the similarity between the other set of correlations between the original sample image and the transformed image. In addition, the natural image rating (NIQE) non-reference quality score suggests that the correlation between the output image (or the original sample image) and the converted image may affect the perceptual quality.

Classification based on the output from the equipment of the present disclosure enhances the accuracy of classification, improves the accuracy of classification results, and creates a network towards the creation of solutions that are so similar to real images that they are difficult to classify by a discriminant network. To train the parameters of. This encourages a perceptually superior solution.

FIG. 7 shows a flowchart of a method for image processing according to another embodiment of the present disclosure.

Step S1 includes a step of obtaining a first feature image. The first feature image may be a luminance feature image obtained by extracting the luminance information of the training image.

Therefore, the step of obtaining the first feature image may include step S11 including a step of obtaining a luminance feature image based on the luminance information in the training image.

The luminance feature image has the same number of rows and columns of pixels as the training image. The luminance value I of the pixel in the i-th row and the j-th column (i, j) of the luminance feature image can be calculated by the following equation (1).

I = 0.299R + 0.587G + 0.114B (1)

In the formula (1), R represents the red component value of the pixel (i, j) in the training image. G represents a green component value. B represents the blue component value. Both i and j are integers. The value of i is 1 ≦ i ≦ X. The value of j is 1 ≦ j ≦ Y. X is the total number of rows in the training image, and Y is the total number of columns in the training image.

In step S12, the luminance feature image is normalized to obtain a second feature image. Normalization allows the pixel values of an image to fall within a smaller range of values and removes outlier pixel values that are too high or too low. This can, in the end, facilitate the calculation of correlations.

More specifically, in step S12, normalization is performed by the following equation (2).

In the formula (2), N represents the second feature image. I represents the luminance value of the pixel at a given position in the luminance feature image obtained from the training image. Blur represents Gaussian blur. Blur (I) represents a Gaussian blur filter performed on the luminance feature image. Blur (I ² ) represents an image obtained by squaring each pixel value in the luminance feature image and then performing a Gaussian blur filter on the luminance feature image. μ represents the output image obtained using the Gaussian blur filter. σ ² represents a locally dispersed image.

Step S2 includes a step of performing a plurality of translational shifts on the second feature image to obtain a plurality of shifted images. Each shifted image has the same number of rows and columns of pixels as the second feature image.

In some embodiments of the present disclosure, the multiple translational shift steps shift the pixels in the last column a in the second feature image in front of the pixels in the remaining columns to obtain an intermediate image. It includes a step of shifting the pixels in the last b row in the intermediate image before the pixels in the remaining rows to obtain a shifted image.

In another embodiment of the present disclosure, the multiple translational shift steps shift the pixels in the last b row in the second feature image before the pixels in the remaining rows to obtain an intermediate image, and said. It involves shifting the pixels in the last row of rows of the intermediate image before the pixels in the remaining rows to obtain a shifted image.

The value of a is ≦ a <Y. The value of b is 0 ≦ b <X. Both a and b are integers. X represents the total number of rows of pixels in the second feature image. Y represents the total number of columns of pixels in the second feature image. In some embodiments, at least one value of a and b changes in any two given image shift processes.

Each pixel with a non-zero value in each shifted image has a corresponding pixel with the same non-zero value in the second feature image. Pixels that do not have a corresponding pixel in the second feature image may be assigned a value of "0". In other words, each pixel with a non-zero value in the shifted image has a corresponding pixel in the second feature image.

Step S3 includes generating a plurality of correlated images based on the correlation between the second feature image and the plurality of shifted images. Each correlation image has the same number of rows and columns of pixels as the second feature image.

The step of generating the plurality of correlated images includes a step of multiplying the value of each pixel in the second feature image by the value of the positionally corresponding pixel in the shifted image. In other words, the value of the pixel (i, j) in the second feature image is multiplied by the value of the pixel (i, j) in the shifted image to obtain the pixel at the (i, j) position in the correlation image. Generate a value. The value of i is 1 ≦ i ≦ X. The value of j is 1 ≦ j ≦ Y. Both i and j are integers. X represents the total number of rows of pixels in the second feature image. Y represents the total number of columns of pixels in the second feature image.

Step S4 includes, for example, transmitting the plurality of correlation images to a neural network such as a discrimination network of a hostile generation network.

The methods for image processing according to the present disclosure include general purpose computers, microprocessors, digital electronic circuits, integrated circuits, specifically designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or the like. It can be implemented in a combination of computing devices. These diverse implementations include implementations in one or more computer programs that are executable and / or interpretable in a programmable system that includes at least one programmable processor, wherein the at least one programmable processor may be dedicated or general purpose. And coupled to receive data and instructions from the storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, at least one input device, and at least one output device. obtain.

These computer programs (also referred to as programs, software, software applications or codes) may include programmable processor machine instructions and may be implemented in high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine readable medium", "computer readable medium" are machine instructions and / or to programmable processors including machine readable media that receive machine instructions as machine readable signals. Refers to any computer program product, device and / or device (eg, magnetic disk, disk disk, memory, programmable logic device (PLD)) used to provide data. The term "machine readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

FIG. 8 shows a block diagram of a system for training a neural network according to an embodiment of the present disclosure.

As shown in FIG. 8, the device 100 may be coupled to the discrimination network 200 via the input end In. The structure and configuration of the discrimination network 200 are not particularly limited. The discrimination network 200 can be constructed and configured as described above, or the degree of matching between an image received as an input by the discrimination network and a predetermined reference image having the same resolution as the input image. As long as it can be classified, it can be constructed and constructed by any suitable method known to those having ordinary knowledge in the art.

The embodiments of the present disclosure do not directly transmit the output image from the generation network and / or the high resolution original image to the discrimination network. Rather, the image is processed by the above-mentioned device, including a feature extraction unit and a shift correlation unit, before being sent to the discrimination network for classification. The shift correlation unit is configured to process the output image from the generation network and / or the high resolution original image to generate a plurality of converted images. For example, the shift correlation unit produces a plurality of correlation images that include not only information specific to the output image and / or the original sample image, but also information about the correlation between those images and the converted image. It is configured to do. This additional information is based on the similarity between the two sets of correlations of the differential network, i.e., one set of correlations between the output image from the generation network and the transformed image. The classification is to be based on the similarity between the other set of correlations between the original sample image and the transformed image. In addition, the natural image rating (NIQE) non-reference quality score suggests that the correlation between the output image (or the original sample image) and the converted image may affect the perceptual quality.

Classification based on the output from the device according to the present disclosure is generated towards the creation of solutions that improve the accuracy of classification, improve the accuracy of classification results, and are so similar to real images that they are difficult to classify by a discriminant network. Train network parameters. This encourages a perceptually superior solution.

In some embodiments, the apparatus according to the present disclosure may be configured to train a hostile generation network, eg, as shown in FIG. FIG. 8 shows a system according to an embodiment of the present disclosure that trains a hostile generation network including one device 100 coupled to a discrimination network 200 via one input end In. However, the present disclosure is not limited to the embodiment shown in FIG. For example, in an embodiment in which the generation network produces a plurality of images having different resolutions, the discrimination network may include a plurality of input terminals In, each coupled to the device 100. Each image from the generation network is transmitted to one of a plurality of image processing devices 100. Each device 100 generates a plurality of correlated images based on the received image, and transmits the plurality of correlated images to the discrimination network 200. A plurality of correlated images from one device 100 may represent feature images of a particular channel of images to be classified. The discrimination network 200 is configured to receive correlated images from the plurality of devices 100 via the plurality of input ends and set the image with the highest resolution from the generation network as the image to be classified. , The discrimination network 200 is configured to score the degree of matching between the image to be classified and a predetermined reference image having the same resolution.

The block diagram of FIG. 8 is not intended to show that the discrimination network contains only the components shown in FIG. The discrimination network according to the present disclosure is an arbitrary number of additional accessories known to those of ordinary skill in the art but not shown in FIG. 8, depending on the specific implementation details. / Or may include components.

The present disclosure provides a computer-readable medium that stores instructions for performing a method of preprocessing an image that trains a hostile generation network as described above.

As used herein, the term "computer-readable medium" is used to provide machine instructions and / or data to programmable processors that include machine-readable media that receive machine instructions as machine-readable signals. Refers to any computer program product, device and / or device (eg, magnetic disk, disk disk, memory, programmable logic device (PLD)). The term "machine readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor. The computer-readable medium according to the present disclosure includes a random access memory (RAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), a programmable read-only memory (PROM), and an erasable programmable read-only memory (EPROM). ), Electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, compact discs (CDs) or DVDs (digital versatile discs) optical storage media and discs such as other non-temporary media. Or, but not limited to, including, but not limited to, tape.

References herein include "some embodiments," "some embodiments," and "exemplary embodiments," "examples," and "specific examples," or "some examples." , It is intended that a particular feature and structure, material or property has been described in connection with an embodiment or example contained in at least some of the embodiments or examples of the present disclosure. Schematic representations of terms do not necessarily refer to the same embodiment or example. In addition, the particular features, structures, materials or properties described may be included in any one or more embodiments or examples in any suitable manner. Further, for a person having ordinary knowledge in the art, what is disclosed is not limited to a specific combination of technical features with respect to the scope of the present disclosure, and the technique does not deviate from the concept of the present invention. Other technical schemes formed by combining technical features or equivalent features of technical features should also be covered. Moreover, the terms "1st" and "2nd" are for illustration purposes only and are interpreted as an explicit or implied reference to the relative importance of the technical features shown and as an implied reference to the quantity. Should not be. Thus, the features defined by the terms "first" and "second" may explicitly or implicitly include one or more features. In the description of the present disclosure, the meaning of "plurality" is two or more unless specifically defined.

The principles and embodiments of the present disclosure are described herein. The description of embodiments of the present disclosure is used solely to aid in understanding the methods of the present disclosure and its core ideas. On the other hand, for those who have ordinary knowledge in the art, what is disclosed is the scope of the present disclosure, and the technical plan is not limited to a particular combination of technical features and does not deviate from the concept of the invention. Other technical schemes formed by combining technical features or equivalent features of technical features should also be covered. For example, a technical plan may be obtained by substituting (but not limited to) the above features as disclosed in the present disclosure with similar features.

Claims (21)

A device that generates multiple correlation images
A feature extraction unit configured to receive a training image, extract at least one feature from the training image, and generate a first feature image based on the training image.
A normalizer configured to normalize the first feature image and generate a second feature image,
A plurality of translational shifts are performed on the second feature image to generate a plurality of shifted images, and each of the plurality of shifted images is correlated with the second feature image to generate a plurality of correlation images. A device that includes a shift correlation unit that is configured to produce. The shift correlation unit shifts the leftmost or rightmost a column of pixels in the pixel block of the second feature image to the rightmost or leftmost column of the pixel block, respectively. Then, by shifting the lowest or highest b row of pixels in the pixel block of the second feature image to the highest or lowest row of the pixel block, respectively, the second feature image is obtained. On the other hand, it is configured to perform the above-mentioned multiple translational shifts.
0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, Y is the total number of columns of pixels in the pixel block of the second feature image, and X is the first. 2 The total number of rows of pixels in the pixel block of the feature image.
The device according to claim 1, wherein a and b are the same or different . The shift correlation unit deletes the leftmost or rightmost a-column pixel in the pixel block of the second feature image, and a-column at the rightmost or leftmost position of the pixel block. Add the pixels of the second feature image, delete the lowest or top b row of pixels in the pixel block of the second feature image, and add the b row of pixels to the top or bottom position of the pixel block. By adding each of them, it is configured to perform the translational shift a plurality of times with respect to the second feature image.
0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, Y is the total number of columns of pixels in the pixel block of the second feature image, and X is the first. 2 The total number of rows of pixels in the pixel block of the feature image.
The device of claim 1 or 2 , wherein each of the added pixels has a pixel value of 0 . The shift correlation unit multiplies the pixel value of each pixel in each pixel block of the plurality of shifted images by the pixel value of the positionally corresponding pixel in the pixel block of the second feature image. The apparatus according to any one of claims 1 to 3 , wherein each of the plurality of shifted images is configured to correlate with the second feature image. The first feature image is a luminance feature image.
The feature extraction unit is
The apparatus according to any one of claims 1 to 4, comprising a luminance detector configured to extract luminance information from the training image and generate the luminance feature image. In order to generate the luminance feature image, the luminance detector is configured to determine the brightness value of the pixel at a given position in the luminance feature image by the following equation (1).
I = 0.299R + 0.587G + 0.114B (1)
I is the luminance value, and is
R is the red component value of the pixel corresponding to the position in the training image, and is
G is a green component value of the pixel corresponding to the position in the training image, and is
The device according to claim 5 , wherein B is a blue component value of a pixel corresponding to the position in the training image. The normalizer is configured to normalize the luminance feature image by the following equation (2).

N is the first feature image, and is
I represents the luminance value of the pixel at a given position in the luminance feature image.
Blur (I) is an image obtained by applying a Gaussian filter to the luminance feature image.
The Blur (I ² ) is an image obtained by squaring each pixel value in the luminance feature image and then applying a Gaussian filter to the luminance feature image , according to claim 5 or 6. Device. The second feature image includes a pixel block having a first size.
Each of the plurality of shifted images and each of the plurality of correlated images comprises a pixel block having the first size.
One of claims 1 to 7, wherein in each of the plurality of shifted images, the pixel having the non-zero pixel value has the corresponding pixel having the same non-zero pixel value in the second feature image. The device described in the section. A method of generating multiple correlation images
The method is
Steps to generate the first feature image based on the training image,
The step of normalizing the first feature image and generating the second feature image,
A step of performing a plurality of translational shifts on the second feature image to generate a plurality of shifted images, and a step of generating a plurality of shifted images.
A method comprising a step of correlating each of the plurality of shifted images with the second feature image to generate a plurality of correlated images. The step of correlating each of the plurality of shifted images with the second feature image is to the pixel value of each pixel in each pixel block of the plurality of shifted images within the pixel block of the second feature image. 9. The method of claim 9, comprising the step of multiplying the pixel values of the positionally corresponding pixels of. The step of performing the multiple translation shifts is
A step of shifting the leftmost or rightmost column a pixel in the pixel block of the second feature image to the rightmost or leftmost column of the pixel block, respectively.
A step of shifting the lowest or highest b row of pixels in the pixel block of the second feature image to the highest or lowest row of the pixel block, respectively, is included.
0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, Y is the total number of columns of pixels in the pixel block of the second feature image, and X is the first. 2 The total number of rows of pixels in the pixel block of the feature image.
The method according to claim 9 or 10, wherein a and b are the same or different . 11. The method of claim 11, wherein at least one of a and b changes at least once during the execution of the plurality of translational shifts. The step of performing the multiple translation shifts is
The leftmost or rightmost a-column pixel in the pixel block of the second feature image is deleted, and the a-column pixel is added at the rightmost or leftmost position of the pixel block, respectively. Steps and
The second feature image includes a step of deleting the lowest or highest b row of pixels in the pixel block and adding b row of pixels to the highest or lowest position of the pixel block, respectively.
0 ≦ a <Y, 0 ≦ b <X, both a and b are integers, Y is the total number of columns of pixels in the pixel block of the second feature image, and X is the first. 2 The total number of rows of pixels in the pixel block of the feature image.
The method of claim 9 or 10, wherein each of the added pixels has a pixel value of 0. 13. The method of claim 13, wherein at least one of a and b changes at least once during the execution of the plurality of translational shifts. Including the step of performing XY translation shifts,
Any of claims 9 to 14 , where Y is the total number of columns of pixels in the pixel block of the second feature image and X is the total number of rows of pixels in the pixel block of the second feature image. The method described in paragraph 1. The step of receiving the training image is further included before generating the first feature image.
The method according to any one of claims 9 to 15, wherein the step of generating the first feature image includes a step of generating a luminance feature image based on the luminance information of the training image. Further comprising the step of determining the luminance value of the pixel at a given position in the luminance feature image by the following equation (1).
I = 0.299R + 0.587G + 0.114B (1)
I is the luminance value, and is
R is the red component value of the pixel corresponding to the position in the training image, and is
G is a green component value of the pixel corresponding to the position in the training image, and is
The method according to claim 16 , wherein B is a blue component value of a pixel corresponding to the position in the training image. Further including a step of normalizing the luminance feature image by the following equation (2).

N is the first feature image, and is
I represents the luminance feature image.
Blur (I) is an image obtained by applying a Gaussian filter to the luminance feature image.
The Blur (I ² ) is an image obtained by squaring each pixel value in the luminance feature image and then applying a Gaussian filter to the luminance feature image , according to claim 16 or 17. Method. The first feature image comprises a pixel block having a first size.
Each of the plurality of shifted images and each of the plurality of correlated images comprises a pixel block having the first size.
One of claims 9 to 18, wherein in each of the plurality of shifted images, a pixel having a non-zero pixel value has a corresponding pixel having the same non-zero pixel value in the first feature image. The method described in the section. A non-temporary computer-readable medium that stores instructions that cause a computer to perform the method of any one of claims 9-19. A system that trains hostile generation networks
A hostile generation network processor comprising a generation network microprocessor configured to be trained by a discrimination network microprocessor and a discrimination network microprocessor coupled to said hostile generation network.
The discrimination network microprocessor is
A plurality of input terminals coupled to the apparatus according to any one of claims 1 to 8.
Multiple analysis modules, each coupled to one of the plurality of input ends,
A plurality of pooling modules connected by the cascade, wherein each stage of the cascade comprises a pooling module coupled to a pooling module in one of the plurality of analysis modules and a stage prior to the cascade.
A system that includes a discrimination network coupled to a pooling module in the final stage of the cascade.