JP6612473B2 - Image generation using neural networks - Google Patents

Description

This application claims priority to US Provisional Patent Application No. 62 / 286,915, filed Jan. 25, 2016. The disclosure of the previous application is considered in its entirety as part of this application and is incorporated by reference into the disclosure of this application.

  The present application relates to image generation using neural networks.

  A neural network is a machine learning model that employs one or more layers of nonlinear units and predicts the output for a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current value of the respective parameter set.

  Some neural networks are recursive neural networks. A recursive neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, recursive neural networks can use some or all of the internal state of the network from previous time steps in calculating the output at the current time step.

  An example of a recursive neural network is a Long Short-Term Memory (LSTM) neural network, which includes one or more LSTM memory blocks. Each LSTM memory block can contain one or more cells, allowing each cell to store the previous status of the cell for use, for example, to generate a current activation Or an input gate, a forgetting gate, and an output gate that allow the cell to be provided to other components of the LSTM neural network.

Aaron van den Oord and Benjamin Schrauwen, `` The Student-t Mixture as a Natural Image Patch Prior with Application to Image Compression, available at http://www.jmlr.org/papers/volume15/vandenoord14a/vandenoord14a.pdf "

  This application describes how a system implemented as a computer program on one or more computers at one or more locations can generate an output image from neural network input.

  For a system of one or more computers, being configured to perform a particular operation or action means that in operation, software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action Means that the system has been installed. For one or more computer programs, being configured to perform a particular operation or action means that one or more instructions, when executed by a data processing device, cause the device to perform the operation or action. Means that the program contains.

  Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages. A neural network system as described herein can generate images more accurately from neural network inputs. In particular, the training of the neural network can be improved by modeling the color values for the pixels in the output image as discrete values rather than continuous values. That is, the neural network can be trained more quickly, and the quality of the output image generated by the trained neural network can be improved. Per pixel, per color value, i.e., the color value for a given color channel for a given pixel is the color value of both the previous pixel in the given pixel and any previous color channel. By generating the output image so as to satisfy the conditions, the quality of the generated output image can be improved. Generating images in this manner using the neural network system described herein neuralizes the general interpixel dependence completely without introducing the independent assumptions required for existing models. The network can be captured.

  The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the present subject matter will be apparent from the description, drawings, and claims.

1 is a diagram illustrating an exemplary neural network system. FIG. 2 is a flowchart of an exemplary process for generating an output image from a neural network input. 4 is a flow diagram of an example process for generating color values for a given color channel for a given pixel in an output image.

  Like reference numbers and symbols in the various drawings indicate like elements.

  FIG. 1 shows an exemplary neural network system 100. Neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations that can implement the systems, components, and techniques described below. is there.

  Neural network system 100 receives a neural network input and generates an output image from the neural network input. For example, the neural network system 100 can receive the neural network input 102 and generate an output image 152 from the neural network 102.

  In some implementations, the neural network system 100 can be used for lossless compression of images or generation of new images that have similar characteristics to the images for which the system has been trained.

  In particular, with lossless compression, the neural network input may be an image, and the neural network system 100 can generate an output image that is a reconstruction of the input image.

  The neural network system 100 can now store at least a portion of the score distribution generated by the output layer of the neural network system 100 for use in arithmetic coding of images, as described below. An example technique for using score distributions generated by machine learning models for arithmetic encoding and decoding is available at http://www.jmlr.org/papers/volume15/vandenoord14a/vandenoord14a.pdf Possible Aaron van den Oord and Benjamin Schrauwen, “The Student-t Mixture as a Natural Image Patch Prior with Application to Image Compression”.

  In image generation, during the training period, the neural network input may be an image, and the neural network system 100 can generate an output image that is a reconstruction of the input image.

  After training, the neural network system 100 can generate an output image for each pixel without any input as a condition.

  In particular, for a given input, the neural network system 100 includes an output image that includes a predetermined number of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels. Generate. For example, the neural network system 100 can generate an image that includes a red channel, a green channel, and a blue channel. As a different example, the neural network system 100 can generate an image that includes a cyan color channel, a magenta color channel, a yellow color channel, and a black color channel. The plurality of color channels are arranged according to a predetermined order, for example, red, green, and blue, or blue, red, and green.

  In general, the neural network system 100 generates a color value in the output image for each pixel in a sequence of pixels taken from the output image. That is, the neural network system 100 orders the pixels in the output image into a sequence and then generates a color value for each pixel in the output image one by one in the order according to the sequence.

  For example, the sequence starts at the upper left corner of the output image and proceeds row by row across the output image, and the last pixel in the sequence may be the pixel at the lower right corner of the output image. In this example, the neural network system 100 first generates color values for the upper left corner pixel and then proceeds to the next pixel in the upper row of the image.

  In particular, for a given color channel for a given pixel in the output image, the neural network system 100 determines (i) the color value for the pixel before the pixel in the sequence, and (ii) the color in the order of the color channel. Generate a color value for the color channel of a given pixel, conditional on the color value for the pixel for any color channel before the channel. In training periods, or image compression, these color values can be taken from the corresponding pixels in the input image, not from the output image, because the output image is a neural network input, ie, reconstruction of the input image. .

  In particular, neural network system 100 includes one or more initial neural network layers 110 and one or more output layers 120.

  After a given color value for a given color channel for a given pixel in the output image has been generated, the initial neural network layer 110 can generate a current output to generate an alternative representation of the current output image. The image is configured to process an output image that includes color values that have already been generated for the output image.

  For example, the initial neural network layer 110 can process the current output image 140 to generate an alternative representation 142 of the current output image 140.

  As shown in FIG. 1, the shaded portion of the current output image 140 represents pixels whose color values have already been generated by the neural network system 100, while the non-shaded portion of the current output image 140 is colored. Represents a pixel whose value has not yet been generated.

  One or more output layers 120 receive the alternative representation and generate a score distribution over a set of discrete possible color values for the next color channel in the image. For example, the set of discrete possible color values may be a set of integers including zero and 255 from zero to 255, and the score distribution includes a respective score for each of the integers in the set. The score in the score distribution represents, for each possible pixel value, the likelihood, for example, the probability that the pixel value should be the value of a given color channel for the task that the system is configured to perform. it can.

  If the given color channel referred to above is the last color channel in the predetermined order of the color channels, the output layer 120 will receive the first in the next pixel in the sequence after the given pixel. A score distribution is generated for the color channel. In the example of FIG. 1, the output layer 120 generates a score distribution 146 for the first color channel of the next pixel 142 in the output image 140.

  If the given color channel referred to above is not the last color channel in a given order, the output layer 120 will be the next after the given color channel in the order of the color channels for the given pixel. A score distribution is generated for the color channel. For example, if the color channel order is red, green, and blue, and the last color value generated was for the green channel of a given pixel, the score distribution generated by output layer 120 is given by Is the score distribution for the blue channel of the pixels.

  In some embodiments, the neural network system 100 includes a single output layer, such as a single softmax layer, that generates a score distribution for all of the color channels.

  In some other embodiments, the neural network system 100 includes a respective output layer corresponding to each of the color channels, eg, a respective softmax layer, each output layer having a score distribution for the corresponding color channel. Is generated.

  In some embodiments, as described in more detail below, the alternative representation is a feature map that includes features for each color channel of each pixel in the output image. In these implementations, when generating color values for a given channel for a given pixel, the output layer uses the corresponding portion of the alternative representation. That is, use the portion of the alternative representation that contains the features of the given color channel for the given pixel.

  The neural network system 100 then determines from the generated score distribution the current color channel, i.e. the first color channel in the next pixel in the sequence after the given pixel, or the color channel for the given pixel. Select the value for the next color channel after a given color channel in the order. For example, the neural network system 100 can sample the color values according to the score distribution or select the color value with the highest score according to the score distribution.

  The initial neural network layer 110 generates an alternative representation where the layer 110 is conditional on the current output image, i.e. is not conditional on any color value in the output image not generated by the neural network system 100. Can be configured in any of a variety of ways.

  In some implementations, the initial neural network layer 110 is a complete convolutional neural network consisting of a plurality of convolutional neural network layers each holding the spatial resolution of the input to the initial neural network layer 110. That is, the spatial resolution of the input to the initial neural network layer 110 and the output of each of the convolutional neural network layers have the same spatial resolution, i.e., have the same number of pixels as the output image, while each pixel by the convolutional neural network layer. The number of features generated for can vary.

  However, throughout the process, the features at each input location, ie, at each pixel, at every layer in the network are divided into multiple portions, each corresponding to one of the color channels.

  Thus, the alternative representation generated by the initial neural network layer 110 includes a respective portion for each of the color channel values for a given pixel, and when generating a score distribution for the given color channel, the output layer 120 is configured to process a portion corresponding to a given color channel.

  To ensure that the convolutional neural network is conditioned only on output values that have already been generated, the portion of the alternative representation corresponding to a given color channel for a given pixel is (i) a pixel in the sequence And (ii) convolution masked to be generated based solely on color channel data for pixels for the color channel prior to a given color channel in the order of color channels Each convolutional neural network layer is configured to apply

  In the first convolution layer, i.e. the layer that receives the current output image as input, the neighboring pixels in the current output image that precede the given pixel in the sequence, and in the current output image already generated The mask restricts the connection to a given pixel in the output feature map of the first convolution layer for the corresponding pixel color.

  In a further convolution layer, adjacent pixels in the input feature map for the further convolution layer preceding a given pixel in the sequence, features corresponding to the color of the corresponding pixel in the already generated input feature map, and input For features corresponding to a given color of the corresponding pixel in the feature map, the mask limits connections at the given pixel in the output feature map of the further convolution layer.

  The neural network system 100 can implement this masking in any of a variety of ways. For example, each convolution layer may have a corresponding weighted kernel.

  In some other implementations, the initial neural network layer 110 includes a plurality of LSTM layers arranged one after the other. Similar to the convolutional neural network layer, the LSTM layer preserves the spatial dimensions of the input, and the features generated by each LSTM layer for each input location in every layer in the network are divided into parts, each of which is a color Corresponds to one of the channels.

  Each of these LSTM layers applies an input-to-state component by applying a convolution to the input feature map for the LSTM layer, i.e., the hidden state of the preceding LSTM layer or the current output image. ) And apply a convolution to the previous hidden situation of the layer to produce a State-to-State Recurrent Component. The LSTM layer then generates a gate value for the LSTM layer from the input vs. status component and the status vs. status component, and generates an updated hidden status and updated cell status for the layer from the gate value and preceding cell status To do.

  In some of these implementations, the LSTM layer is a row LSTM layer that processes the input feature map row by row from top to bottom and calculates features for all rows at once.

  That is, for each row of the input feature map, the row LSTM layer is configured to calculate the input versus status component of the row LSTM layer for the entire input feature map, for example using a one-dimensional convolution, and input for the entire input feature map After calculating the pairwise situation component, the input pairwise situation component is used to process the input feature map row by row from top to bottom to calculate features for all rows at once.

  To ensure that the row LSTM layer is not conditioned on the output of color values that have not yet been generated, the convolution used by the row LSTM layer to generate the input-to-situation component is the same as for the convolutional neural network layer. Masked as described in.

  In other of these implementations, the LSTM layer is a Diagonal Bidirectional LSTM Layer.

  In general, a bidirectional LSTM layer generates an output map for one direction and an output map for the other direction, and combines the two output maps to produce the final output map for the layer. Composed. That is, the bi-directional LSTM layer calculates the situation-to-situation component and the input-to-situation component for each of the two directions, and then the situation-to-situation component for each direction and the output map for that direction from the input-to-situation component Is generated.

  In particular, each diagonal BiLSTM layer scans the input feature map in a diagonal fashion along the first direction and a diagonal style along the second direction to generate an output feature map for the layer Configured as follows.

  More specifically, each diagonal BiLSTM layer can easily apply convolutions along the diagonal, for example, by offsetting each row in the input feature map by one position relative to the preceding row. Is configured to skew the input feature map into the space to be

  For each of the two directions, the diagonal BiLSTM layer then calculated and skewed the input versus situation component for the diagonal BiLSTM layer in the main direction by applying a 1x1 convolution to the skewed input feature map. It is configured to calculate the situation-to-situation component for the diagonal BiLSTM layer in the main direction by applying column-wise convolution to the input feature map. In some implementations, the column-wise convolution has a 2x1 size kernel.

  The diagonal BiLSTM layer is a skewed output feature for each direction, eg, a left-skewed output feature map and a right-skewed output feature map, from the situation-to-situation component and the input-to-situation component for the direction as described above. Each skewed output feature map is further configured to skew back to match the spatial dimension of the input feature map by generating a map and removing the offset location. The diagonal BiLSTM layer then shifts the right output map down one row and adds the shifted right output map to the left output map to produce the final output map for the layer.

  As with the row LSTM layer, the convolution applied to the diagonal BiLSTM layer to generate the situation versus situation component can also be masked as described above.

  In some implementations, the initial neural network layer 110 receives the current output image as input, an adjacent pixel in the current output image that precedes a given pixel in the sequence, and already generated, 1 For a color at the corresponding pixel in the current output image followed by one or more row LSTM layers or one or more diagonal BiLSTM layers for a given in the output feature map of the first convolution layer A first convolution layer is included where the mask limits connections at the pixels.

  In some implementations, the initial neural network layer 110 includes a skip connection between layers, a residual connection between layers, or both.

  FIG. 2 is a flowchart of an exemplary process 200 for generating an output image from a neural network input. For convenience, the process 200 will be described as being performed by a system of one or more computers located at one or more locations. For example, a properly programmed neural network system, such as the neural network system 100 of FIG.

  Process 200 may be performed during a neural network training period to generate an output image. For example, process 200 may be a forward pass of the training process. Process 200 can also be implemented as part of compressing a neural network input, ie, an input image.

  The system receives neural network input (step 202). As described above, the neural network input may be an input image.

  The system generates an output image for each pixel from the neural network input into a sequence of pixels taken from the output image (step 204). That is, the system generates one color value for each pixel in the output image, in order according to the sequence, so that the color value for the earlier pixel in the sequence precedes the later color value in the sequence. Is generated. Within each pixel, the system generates one color value for each color channel of the pixels, according to a predetermined order of color channels. In particular, the system provides (i) a color value for a pixel for the pixel before the pixel in the sequence, and (ii) a color value for the pixel for any color channel before the color channel in the order of color channels. Each color value for each pixel is generated as a condition. In training periods, or image compression, these color values can be taken from the corresponding pixels in the input image, not from the output image, because the output image is a neural network input, ie, reconstruction of the input image. .

  FIG. 3 is a flow diagram of an example process 300 for generating color values for a given color channel for a given pixel in the output image. For convenience, process 300 will be described as being performed by a system of one or more computers located at one or more locations. For example, a properly programmed neural network system, such as neural network system 100 of FIG.

  The system processes the current output image through the initial neural network layer and generates an alternative representation (step 302). The current output image is the color value for each of the color channels for the pixel before the given pixel in the sequence, and for any color channel before the given color channel in order for the given pixel. It is an image including the color value of. As described above, the initial neural network layer is configured to apply masked convolution so that the alternative representation is conditional on already generated color values and any color values that have not yet been generated. Is not a requirement.

  The alternative representation includes a respective portion corresponding to each of the color channels of a given pixel.

  The system uses the output layer corresponding to the given color channel, for example the softmax layer, to process the portion of the alternative representation corresponding to the given color channel, and possible color values for the given color channel A score distribution is generated over (step 304). As described above, in some implementations, a single output layer corresponds to all of the color channels, while in other implementations, each color channel has a different and corresponding output layer.

  The system uses the score distribution to select a color value for a given color channel for a given pixel, for example by selecting the highest-scoring color channel or sampling from the score distribution (step 306).

  The system can repeat process 300 for each color channel for each pixel in the output image to generate a color value for each pixel in the output image.

  The system can perform processes 200 and 300 for the desired output, ie, the neural network input for which the output image that must be generated by the system for input is not known.

  The system trains the initial neural network layer and, if the output layer has parameters, the output layer, i.e., determines the trained values for the initial neural network layer and optionally the parameters of the output layer, Processes 200 and 300 can also be performed on neural network inputs with a set of training data, ie, a set of inputs for which the output images that must be generated by the system are known. Processes 200 and 300 are repeated for inputs selected from a training data set as part of a conventional machine learning training technique for training initial neural network layers, such as stochastic gradient descent and back propagation techniques, for example. Can be implemented.

  During the training period, the output image that must be generated is known in advance, thus reducing the amount of time and computational resources required to process a given training neural network input, and therefore training The computations performed by the initial neural network layer can be accelerated in order to reduce the time required for training, improve the performance of the trained neural network, or both.

  For example, when the initial neural network layer is a complete convolutional neural network, the processing required for the initial neural network layer to generate an alternative representation is sequential because all output images are available from the beginning of the computation. Can be done in parallel. That is, as described above, the system can use the color value for the input image instead of the color value of the output image pixel already generated. Because the convolution is masked, the system can generate all alternative representations in parallel based on the input image.

  Embodiments of the subject matter and functional operations described herein include digital electronic circuits, tangibly embodied computer software, or structures that include the structures disclosed herein and their structural equivalents. It can be implemented in firmware, computer hardware, or a combination of one or more thereof.

  Embodiments of the subject matter described herein are directed to one or more computer programs, i.e., a tangible, non-transitory program, for execution by a data processing device or for controlling operation of a data processing device. It can be implemented as one or more modules of computer program instructions encoded on a carrier. Alternatively or additionally, the program instructions are generated to encode information for transmission to a suitable receiving device for execution by a data processing device, such as an artificially generated electronic, optical, or electromagnetic signal. Can be encoded on the generated propagation signal. The computer storage medium may be a machine readable storage device, a machine readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

  The term “data processing apparatus” encompasses all types of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The device can include dedicated logic circuitry, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the device executes for the subject computer program such as, for example, processor firmware, protocol stack, database management system, operating system, or code comprising one or more combinations thereof. It can also contain code that creates the environment.

  A computer program (called or described as a program, software, software application, module, software module, script, or code) can be any programming language, such as a compiled or interpreted language, or a declarative or procedural language And can be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but need not correspond. A program can be part of a file that holds other programs or data, for example one or more scripts stored in a markup language document, a single file dedicated to the program of interest, or for example one or more Can be stored in a plurality of tailored files, such as files that store modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one location or distributed across multiple locations and interconnected by a communication network.

  The process and logic flows described herein are one or more programs that execute one or more computer programs to perform functions by operating on input data and generating output. It can be implemented by a computer capable. Processes and logic flows can also be implemented by dedicated logic circuits, such as, for example, FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the device can be, for example, FPGA, or ASIC It can also be implemented as a dedicated logic circuit.

  Computers suitable for the execution of computer programs can be based on, for example, general purpose or special purpose microprocessors or both, or include any other type of central processing unit. Generally, a central processing unit receives instructions and data from read-only memory or random access memory or both. The essential elements of a computer are a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical disks, or optical disks, or receives data from one or more mass storage devices. It will be operably coupled to receive or transfer data or both. However, the computer need not have such a device. In addition, a computer may be, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a universal serial bus, for example. It can be incorporated into another device such as a portable storage device such as a (USB) flash drive.

  Computer readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CDs Includes all forms of non-volatile memory, media, and memory devices, including ROM and DVD-ROM discs. The processor and the memory can be supplemented by, or incorporated in, dedicated logic circuitry.

  To achieve user interaction, embodiments of the subject matter described herein include a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, As well, it can be implemented on a computer having a keyboard and a pointing device such as a mouse or trackball that allows the user to provide input to the computer. Similarly, other types of devices can be used to achieve user interaction. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, acoustic feedback, or haptic feedback. Also, input from the user can be received in any form including acoustic, voice, or haptic input. In addition, the computer sends a document to a device used by the user and receives a document from the device, eg, a web page on a web browser on the user's client device in response to a request received from the web browser By interacting with the user.

  Embodiments of the subject matter described herein include, for example, a backend component as a data server, or include a middleware component such as, for example, an application server, or a client computer having a graphical user interface, for example. A computer comprising a front-end component such as a web browser with which a user can interact with an implementation of the subject matter described in the document, or any combination of one or more such back-ends, middleware, or front-end components. Can be implemented in a storage system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”) such as the Internet.

  The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client-server relationship is caused by computer programs that run on each computer and have a client-server relationship with each other.

  This specification contains many specific implementation details, which should not be considered as limiting the scope of any invention or claimable, but rather specific to a particular embodiment of a particular invention. It should be considered as a description of possible features. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Further, features are described above as working in particular combinations, and may even be so claimed initially, but one or more features from the claimed combination may in some cases Then, it can be deleted from the combination, and the claimed combination can be directed to a sub-combination or a sub-combination variant.

  Similarly, operations are illustrated in a particular order, but this may be performed in the particular order in which they are shown or in a sequence to obtain the desired result. Or should be understood to require performing all the described operations. In certain situations, multitasking and parallel processing may be advantageous. Further, the division of the various system modules and components in the embodiments described above should not be understood as requiring such divisions in all embodiments, and the program components and systems described are It should be understood that, in general, it can be integrated into a single software product or packaged into multiple software products.

  Particular embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequence shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

100 neural network system
102 Neural network input
110 Initial neural network layer
120 Output layer
140 Current output image
142 Alternative representation, pixel
144 Alternative expressions
146 Score distribution
152 Output image
200 processes
300 processes

Claims (24)

A neural network system implemented by one or more computers, wherein the neural network system is configured to receive a neural network input and generate an output image from the neural network input, the output image being two-dimensional Including a plurality of pixels arranged in a map, each pixel having a respective color value for each of a plurality of color channels;
One or more initial neural network layers configured to receive the neural network input and process the neural network input to generate an alternative representation of the neural network input;
One or more output layers, including, for each pixel in the output image, generating a respective score distribution over a set of discrete possible color values for each of the plurality of color channels; A neural network system comprising: an output layer configured to receive the alternative representation and to generate the output image for each pixel from a sequence of pixels taken from the output image. The plurality of color channels are ordered, and the one or more output layers include respective output layers corresponding to each of the plurality of color channels, each of the output layers for each pixel of the output image,
(i) a color value for a pixel for a pixel before the pixel in the sequence, and (ii) the for any color channel before the color channel corresponding to the output layer in the order of color channels. The configuration of claim 1, configured to generate the respective score distribution over a set of discrete possible color values for the color channel corresponding to the output layer, conditional on color values for pixels. Neural network system.   For each pixel, each of the output layers is (i) a color value for the pixel for the pixel before the pixel in the sequence, and (ii) the color corresponding to the output layer in the order of color channels. The neural network system of claim 2, configured to receive a portion of an alternate representation corresponding to the color channel as well as contextual information based on a color value for the pixel for any color channel prior to the channel. .   For the pixel for the color channel before the color channel corresponding to the output layer in the sequence in which the portion of the alternative representation corresponding to the color channel corresponds to the output layer in the order of the color channels The neural network system is configured to apply a mask to an output of a neural network layer in the one or more initial neural network layers so that the neural network system is generated based only on the color channel data of 3. The neural network system according to 3.   5. The neural network system according to claim 1, wherein each of the output layers is a softmax layer. The neural network input is an image;
The one or more initial neural network layers include a row length-short term memory (LSTM) layer, the row LSTM layer comprising:
6. The neural network system according to any one of claims 1 to 5, configured to process the input image line by line from top to bottom and calculate features for all lines at once.   The neural network system of claim 6, wherein the row LSTM layer calculates the features using a one-dimensional convolution. The row LSTM layer is
Calculate the input versus status component of the row LSTM layer for the entire input image;
After calculating the input versus situation component for the entire input image, use the input versus situation component to process the input image row by row from top to bottom to calculate features for all rows at once 8. A neural network system according to claim 6 or 7, configured to: The neural network input is an image;
The one or more initial neural network layers include a diagonal bidirectional LSTM (BiLSTM) layer, and the diagonal BiLSTM layer
The input image map is configured to scan an input image map in a diagonal manner along a first direction and a diagonal manner along a second direction to generate features of the input image map. The neural network system according to any one of claims. The diagonal BiLSTM layer is
Skewing the input image map into a space that allows easy application of convolution along a diagonal;
For each of the first direction and the second direction,
By applying a 1x1 convolution to the skewed input image map, calculating the input bi-situational component of the diagonal BiLSTM layer for the direction,
10. The neural network system of claim 9, configured to calculate a situation-to-situation recursive component of the diagonal BiLSTM layer for the direction by applying a column direction convolution to the skewed input image map. .   The neural network system of claim 10, wherein the column-wise convolution comprises a 2 × 1 size kernel.   The initial neural network layer includes a plurality of LSTM layers, and the plurality of LSTM layers are configured with residual connections from one LSTM layer to another LSTM layer in the plurality of LSTM layers. The neural network system according to any one of the above. The input is an image;
13. The neural network system according to any one of claims 1 to 12, wherein the one or more initial neural network layers include one or more convolutional neural network layers.   14. The neural network system according to any one of claims 1 to 13, wherein the neural network input is an input image and the output image is a reconstructed version of the input image.   The neural network input is an input image, and the neural network system uses at least a portion of the score distribution for use in arithmetically encoding the input image for lossless compression of the input image. 15. A neural network system according to any one of claims 1 to 14, configured to store.   16. A neural network system according to any one of the preceding claims, wherein the pixels in the sequence of pixels are taken row by row from the output image.   Encoding with instructions that, when executed by one or more computers, cause the one or more computers to perform operations implementing each neural network system according to any one of claims 1 to 16. One or more computer recording media. Receiving a neural network input;
17. A method comprising: processing a neural network input image using the neural network system according to any one of claims 1 to 16 and generating an output image from the neural network input. A computer-implemented method for generating an output image from a neural network input, wherein the output image includes a plurality of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels. Have
Receiving the neural network input at one or more initial network layers of the neural network system and processing the neural network input to generate an alternative representation of the neural network input;
One or more outputs of the neural network input, including, for each pixel in the output image, generating a respective score distribution over a set of discrete possible color values for each of the plurality of color channels Receiving the alternative representation at a layer and generating the output image for each pixel from a sequence of pixels taken from the output image.   20. A system comprising one or more computers configured to perform the method of claim 19. When executed by one or more computers and at least one processor,
Receiving neural network input at one or more initial network layers of the neural network system and processing the neural network input to generate an alternative representation of the neural network input;
For each pixel in the output image, including generating a respective score distribution over a set of discrete possible color values for each of multiple color channels, one of said neural network input or output layer Receiving the alternative representation at and generating the output image for each pixel from a sequence of pixels taken from the output image, thereby generating an output image comprising a plurality of pixels arranged in a two-dimensional map. And at least one memory comprising computer program code that causes the apparatus to cause each pixel to have a respective color value for each of a plurality of color channels. A method for generating an output image, wherein the output image includes a plurality of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels,
Generating the output image pixel by pixel from a sequence of pixels taken from the output image;
The generating step comprises: for each color channel of each pixel in the output image
Processing a current output image using one or more initial neural network layers to generate an alternative representation, wherein the current output image is (i) prior to the pixel in the sequence; And (ii) only the color value for the pixel for any color channel before the color channel in the order of the color channels, and
Processing the alternative representation using an output layer to generate a score distribution over a set of discrete possible color values for the color channel.   23. A system comprising one or more computers configured to perform the method of claim 22.   23. One or more computer recording media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the method of claim 22.

Images