US20210081796A1 - Neural architecture search for dense image prediction tasks - Google Patents
Neural architecture search for dense image prediction tasks Download PDFInfo
- Publication number
- US20210081796A1 US20210081796A1 US17/107,745 US202017107745A US2021081796A1 US 20210081796 A1 US20210081796 A1 US 20210081796A1 US 202017107745 A US202017107745 A US 202017107745A US 2021081796 A1 US2021081796 A1 US 2021081796A1
- Authority
- US
- United States
- Prior art keywords
- neural network
- dense
- architecture
- architectures
- candidate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001537 neural effect Effects 0.000 title description 10
- 238000013528 artificial neural network Methods 0.000 claims abstract description 102
- 238000012549 training Methods 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 66
- 230000008569 process Effects 0.000 claims abstract description 38
- 238000003860 storage Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 25
- 238000011176 pooling Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 abstract description 15
- 238000010200 validation analysis Methods 0.000 description 13
- 238000003709 image segmentation Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000010801 machine learning Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000002787 reinforcement Effects 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 230000003993 interaction Effects 0.000 description 3
- 238000013441 quality evaluation Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000026676 system process Effects 0.000 description 2
- 241000009334 Singa Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 235000000332 black box Nutrition 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/15—Correlation function computation including computation of convolution operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G06K9/6256—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G06N3/0454—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Definitions
- This specification relates to determining architectures for neural networks that perform image processing tasks.
- Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input.
- Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer.
- Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
- This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an architecture for a neural network that is configured to perform a dense image prediction task.
- a dense image prediction task is a task that requires making a respective prediction for each of a large number of pixels in an input image.
- Many dense image prediction task require assigning a label or otherwise making a prediction for each pixel of the image.
- dense image prediction tasks generally require the neural network to operate on high-resolution images and to maintain the resolution of the images throughout the processing pipeline.
- the system obtains training data for the dense image prediction task.
- the system determines an architecture for a neural network that is configured to perform the dense image prediction task, i.e., to receive inputs and generate outputs that conform to the requirements of the dense prediction task, using the training data.
- each candidate architecture in the space of candidate architectures includes (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task.
- each candidate architecture includes the same neural network backbone as each other candidate architecture but has a different dense prediction cell from each other candidate architecture.
- the system determines the architecture for the neural network based on the best performing candidate architectures.
- the architecture search techniques described in this specification can determine a high-performing architecture for a dense prediction task in a computationally efficient manner.
- dense prediction tasks require generating predictions for a large number of the pixels in the input image, such tasks require the network to operate on high resolution imagery.
- existing architecture search techniques e.g., techniques geared for image classification or other, non-dense image processing tasks, ill-suited for use for these tasks.
- many of these tasks rely on low-resolution proxy tasks that would not be representative of the final dense image prediction task or require a search space to be searched that is so large as to make such searching computationally infeasible when operating on high-resolution images.
- the described techniques effectively limit the search space to identifying the best architecture for a dense prediction cell, resulting in architectures that have performance that exceeds the previous state-of-the-art in multiple dense image prediction tasks.
- the resulting architecture can be more computationally efficient than the previously state-of-the-art models while exceeding their performance.
- the resulting architectures can achieve state-of-the-art performance on several dense prediction tasks, including achieving 82.7% mIOU accuracy on the Cityscapes data set (street scene parsing), 71.3% mIOU accuracy on the PASCAL-Person-Part data set (person-part segmentation), and 87.9% mIOU accuracy on the PASCAL VOC 2012 data set (semantic image segmentation).
- the resulting architectures are more computationally efficient, requiring approximately half the parameters and half the computational cost as previous state of the art systems to achieve these levels of performance on these data sets.
- the amount of resources consumed by the search process is reduced. Additionally, by pre-training the backbone, e.g., on an object segmentation task, and then holding the backbone fixed during the search, the amount of resources consumed by the search process is reduced. Additionally, by pre-computing and then caching the feature maps generated by the pre-trained backbone, the amount of resources consumed by the search is reduced.
- the system can perform the search with much less latency and much greater data efficiency (i.e., using much less memory), than techniques that rely on training candidate neural networks without pre-caching inputs and without decreasing the size of the
- FIG. 1 shows an example neural architecture search system.
- FIG. 2 shows an example candidate architecture
- FIG. 3 is a flow diagram of an example process for determining a final architecture.
- FIG. 4 is a flow diagram of an example process for searching the space of candidate architectures.
- This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an architecture for a neural network that is configured to perform a dense image prediction task.
- a dense image prediction task is a task that requires making a respective prediction for each of a large number of pixels in an input image.
- Many dense image prediction tasks require assigning a label or otherwise making a prediction for each pixel of the image.
- dense image prediction tasks generally require the neural network to operate on high-resolution images.
- One example of a dense image prediction task is an image segmentation task.
- the input is an image and the output is a respective label for every pixel in the image that classifies the content depicted at that pixel in the image.
- an image segmentation task is a person-part segmentation task, where the inputs are images of one or more people and the output is a respective label for every pixel in the image such that the labels classify which pixels correspond to which person parts (e.g., head, torso, legs, and so on) and which correspond to the background (i.e., do not depict any people).
- person parts e.g., head, torso, legs, and so on
- the background i.e., do not depict any people
- an image segmentation task is a semantic image segmentation task.
- the input is an image and the output is a respective label for every pixel in the image that identifies which object class the pixel belongs to, e.g., from a set of multiple foreground object classes and one or more background object classes.
- an image segmentation task is a scene parsing task.
- the input is an image and the output is a respective label for every pixel in the image that identifies which portion of a scene depicted in the image the pixel belongs to.
- a dense image prediction task is an object detection task.
- the input is an image and the output is data that specifies which pixels of the image are parts of an image of an object.
- the output may be a label for each pixel in the image that identifies whether the pixel is part of an image of an object.
- the output may be a score for each of a large number of bounding boxes in the image that indicates whether the bounding box is part of an image of an object.
- FIG. 1 shows an example neural architecture search system 100 .
- the neural architecture search system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.
- the neural architecture search system 100 is a system that obtains training data 102 for training a neural network to perform a dense image prediction task and a validation set 104 for evaluating the performance of the neural network on the dense image prediction task. The system then uses the training data 102 and the validation set 104 to determine an architecture for a neural network that is configured to perform the dense image prediction task i.e., to receive inputs and generate outputs that conform to the requirements of the dense prediction task.
- the architecture defines the number of layers in the neural network, the operations performed by each of the layers, and the connectivity between the layers in the neural network, i.e., which layers receive inputs from which other layers in the neural network.
- the training data 102 and the validation set 104 both include a set of training inputs and, for each training input, a respective target output that should be generated by the neural network to perform the dense image prediction task.
- a larger set of training data may have been randomly partitioned to generate the training data 102 and the validation set 104 .
- each training input is an image and the neural network output for the training input identifies the labels that should be assigned to some or all of the pixels in the training input.
- the system 100 can receive the training data 102 and the validation set 104 in any of a variety of ways.
- the system 100 can receive training data as an upload from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system 100 , and randomly divide the uploaded data into the training data 102 and the validation set 104 .
- API application programming interface
- the system 100 can receive an input from a user specifying which data that is already maintained by the system 100 should be used for training the neural network, and then divide the specified data into the training data 102 and the validation set 104 .
- the system 100 determines the architecture for the neural network by searching a space of candidate architectures to identify one or more best performing architectures.
- Each candidate architecture in the space of candidate architectures includes (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task.
- each candidate architecture includes the same neural network backbone as each other candidate architecture but has a different dense prediction cell from each other candidate architecture.
- FIG. 2 shows a candidate neural network architecture 200 .
- the architecture 200 receives a training input 220 , i.e., one of the three images shown in FIG. 2 , and generates a neural network output 250 that assigns labels to the training input.
- the neural network output 250 is represented as an overlay over the corresponding input image, with pixels assigned the same label being in the same shade in the overlay.
- the candidate neural network architecture includes a backbone 230 that is made up of multiple neural networks layers, e.g., convolutional layers optionally combined with other types of layers (batch normalization, pooling, and so on) and receives the training input 220 and generates feature maps that are provided as input to a dense prediction cell (DPC) 240 .
- the dense prediction cell 240 processes the feature maps to generate the neural network output 250 .
- each candidate neural network in the search space has the same backbone 230 but all of the candidate neural networks have different DPCs 240 .
- the search space for the architecture search is the space of possible architectures for the DPC 240 .
- the DPC can have B branches, where B is a fixed integer greater than one.
- Each of the B branches of the DPC maps an input tensor to the branch to an output tensor by applying an operation to the input tensor.
- the DPC can then combine, e.g., concatenate, the output tensors generated by all of the B branches to generate a combined tensor output or use the output tensor generated by a designated one of the B branches, e.g., the last of the B branches in a processing order, as the combined tensor output.
- B can be, e.g., an integer in the range of 3 to 10, inclusive, with larger values of B allowing more flexibility and a larger search space but increasing the computational cost of the search process.
- the combined tensor output can be used as the output of the DPC 240 for the dense prediction task.
- the DPC processes the combined tensor output through one or more output layers, e.g., a sigmoid output layer, to generate a final output.
- the different architectures in the search space each specify (i) an input tensor from a set of input tensors to be provided as input to each of the B branches, and (ii) an operation from a set of operations to be performed by each of the B branches to generate the output tensor from the input tensor.
- the input tensor for a given branch can be selected from a set that includes (i) the feature maps generated by the backbone and (ii) output tensors generated by any branches before the given branch in the processing order of the B branches.
- the set includes only the feature maps generated by the backbone while for the last of the B branches in the processing order, the set includes (i) the feature maps generated by the backbone and (ii) output tensors generated by any of the B ⁇ 1 other branches.
- the operator space i.e., the space of possible operations from which the operation performed by each of the B blocks is selected, can include one or more of the following: (1) a convolution with a 1 ⁇ 1 kernel, (2) one or more atrous separable convolutions, each having a different sampling rate, and (3) one or more spatial pyramid pooling operations, each having a respective grid size.
- each of the atrous separable convolutions will have a different sampling rate.
- the sampling rate can be defined as r_h ⁇ r_w, where each of r_h and r_w is selected from the set of ⁇ 1, 2, 6, 9, . . . , 21 ⁇ .
- each of the spatial pyramid pooling operations will have a different grid size.
- the grid size can be defined as g_h ⁇ g_w, where each of g_h and g_w is selected from the set of ⁇ 1, 2, 4, 8 ⁇ .
- search spaces can include different operator spaces and different possible values for the sampling rates and grid sizes of the operators in the space.
- the system 100 includes a candidate selection engine 130 , a training engine 140 , and a quality evaluation engine 150 .
- the system To search the space of candidate architectures, the system repeatedly performs a set of architecture search operations.
- the candidate selection engine 130 selects one or more candidate architectures from the space of possible candidate architectures using quality measures for candidate architectures that have already been evaluated.
- the training engine 140 then trains the selected candidate architectures on at least some of the training data 102 and the quality evaluation engine 150 evaluates the trained candidate architectures using the validation set 104 .
- Selecting candidate architectures, training the candidate architectures, and evaluating the quality of the architectures is described in more detail below with reference to FIGS. 3 and 4 .
- the system 100 can identify as the best performing candidate architectures a threshold, fixed number of candidate architectures evaluated during the search that had the best quality measures.
- the system 100 determines the final architecture for the neural network based on the best performing candidate architectures.
- the system 100 can generate, from each of the identified best performing candidates, a final architecture, and then train the final architectures to convergence on the dense prediction task.
- the system can then select the best performing, e.g., as determined based on a quality measure on the validation set 104 , trained architecture as the architecture of the neural network.
- the candidate architecture and the corresponding final architecture are the same.
- the system 100 replaces the neural network backbone with a different, larger neural network backbone that has more parameters and that allows the final neural network to perform better on the dense prediction task.
- the system uses a smaller backbone for the architecture search than is employed by the final architecture.
- the system can employ an Xception architecture for the backbone in the final architecture while employing the MobileNet-v2 architecture for the candidate architectures, i.e., during the search.
- the MobileNet-v2 architecture requires roughly one twentieth the computational cost of the Xception architecture and cuts down the number of channels in the backbone feature maps from 2048 to 320 dimensions.
- the Xception architecture is described in more detail in F. Chollet.
- Xception Deep learning with depthwise separable convolutions.
- CVPR 2017 while the MobileNet-v2 architecture is described in more detail in M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.
- Mobilenetv2 Inverted residuals and linear bottlenecks. In CVPR, 2018.
- Using the smaller backbone can allow the system to make architecture search more computationally efficient while still providing a quality signal for how well the final architecture having the larger backbone will perform on the dense image prediction task.
- the neural network search system 100 can then output architecture data 152 that specifies the final architecture of the neural network, i.e., data specifying the layers that are part of the neural network, the connectivity between the layers, and the operations performed by the layers.
- the neural network search system 100 can output the architecture data 152 to the user that submitted the training data.
- the system 100 uses the trained neural network having the final architecture to process requests received by users, e.g., through the API provided by the system. That is, the system 100 can receive inputs to be processed, use the trained neural network to process the inputs, and provide the outputs generated by the trained neural network or data derived from the generated outputs in response to the received inputs.
- FIG. 3 is a flow diagram of an example process 300 for determining a final architecture for a dense image prediction task.
- the process 300 will be described as being performed by a system of one or more computers located in one or more locations.
- a neural architecture search system e.g., the neural architecture search system 100 of FIG. 1 , appropriately programmed, can perform the process 300 .
- the system receives training data for the dense image prediction task (step 302 ).
- the system searches a space of candidate architectures to identify one or more best performing architectures using the training data (step 304 ). Searching the space of candidate architectures is described in more detail below with reference to FIG. 4 . Once the search has completed, the system can select, as the best performing architectures, a threshold number of candidate architectures that had the best quality evaluation during the search.
- the system determines the architecture for the neural network based on the one or more best performing candidate architectures (step 306 ). For example, the system can generate, from each of the identified best performing candidates, a final architecture, and then train the final architectures to convergence on the dense prediction task. The system can then select the best performing trained architecture, e.g., as determined based on a quality measure on the validation set, as the architecture of the neural network.
- the quality measure that is used can be the same as the one described below with reference to step 410 .
- the candidate architecture and the corresponding final architecture are the same.
- the system replaces the neural network backbone with a different, larger neural network backbone that has more parameters and that allows the final neural network to perform better on the dense prediction task.
- the system uses a smaller backbone for the architecture search than is employed by the final architecture.
- FIG. 4 is a flow diagram of an example process 400 for searching the space of candidate architectures.
- the process 400 will be described as being performed by a system of one or more computers located in one or more locations.
- a neural architecture search system e.g., the neural architecture search system 100 of FIG. 1 , appropriately programmed, can perform the process 400 .
- the system obtains data specifying a pre-trained backbone for use in searching the space of candidate architectures (step 402 ).
- the system pre-trains the backbone along with a placeholder DPC on a placeholder dense image prediction task to determine trained values of the parameters of the backbone, i.e., so that the system only needs train the backbone once for the entire search process.
- the system obtains data specifying the trained values of the backbone parameters from another system that has already pre-trained the backbone.
- the system processes at least some of the training inputs in the training data using the pre-trained backbone to generate feature maps for the training inputs (step 404 ).
- the system processes each of the training inputs using the pre-trained backbone, i.e., in accordance with the trained values of the parameters of the backbone, to generate feature maps and then stores the generated feature maps for use during the search process.
- the system is using the smaller backbone for the search, because the feature maps generated by the smaller backbone have many fewer channels than the feature maps generated by the larger backbone (that will be used in the final architecture), storing the feature maps requires much less storage space than would be required to store feature maps generated by the larger backbone.
- the system then repeatedly performs steps 406 - 410 until termination criteria for the search are satisfied, e.g., until a threshold number of candidate architectures have been evaluated, until the highest performing candidate architecture reaches a threshold accuracy, or until a threshold amount of time has elapsed.
- the system selects one or more candidate architectures from the space of candidate architectures (step 406 ).
- the system can use any of a variety of techniques for searching the space to select the candidate architectures.
- the system can use a random search strategy.
- a random search strategy at each iteration of the process 400 , the system selects one or more architectures from the space of candidate architectures uniformly at random while also selecting one or more architectures that are close to, i.e., are similar to, the currently best observed architectures, i.e., the architectures already evaluated as part of the search that have been found to perform best on the dense prediction task.
- Random search strategies that can be employed by the system are described in more detail in D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. Karro, and D. Sculley. Google muler: A service for black - box optimization .
- SIGKDD 2017 and in B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition .
- CVPR 2018.
- the system can use a reinforcement learning guided search strategy to select the candidate actions.
- the system can select architectures that are output by a recurrent neural network that is being trained through reinforcement learning to output candidate architectures that perform well on the task. Examples of reinforcement learning guided search strategies that can be employed by the system are described in more detail in B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning . In ICLR, 2017.
- the system trains the selected one or more candidate architectures on at least a portion of the training data (step 408 ). That is, for each selected candidate, the system trains a neural network having the architecture until criteria for stopping the training are satisfied. During the training, the system holds the trained values of the backbone parameters fixed while updating the values of the parameters of the DPC in the neural network.
- step 404 was performed to pre-process the training data using the pre-trained backbone
- the system does not process the training inputs using the backbone during step 408 and instead accesses the pre-generated feature maps from memory and provides the pre-generated feature maps as input to the DPC. This greatly decreases the processing power and time consumed by the training, since training inputs do not need to be processed through backbone to generate the feature maps before the feature maps can processed by the DPC.
- the system can perform this training using an early stopping criterion, i.e., can train each candidate for a fixed number of iterations instead of to convergence.
- the system evaluates the performance of each of the trained candidate architectures (step 410 ).
- the system uses an evaluation metric that measures the performance of the trained neural network having the candidate architecture on the validation set to evaluate the performance.
- any evaluation metric that is appropriate for dense prediction tasks can be used.
- the system can use the pixel-wise mean intersection-over-union (mIOU) of the trained neural network over the validation data set as the evaluation metric.
- the system can use the mean pixel accuracy over the validation data set of the trained neural network as the evaluation metric.
- Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
- Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus.
- the computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
- the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
- data processing apparatus refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
- the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
- the apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
- a computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
- a program may, but need not, correspond to a file in a file system.
- a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.
- a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
- the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations.
- the index database can include multiple collections of data, each of which may be organized and accessed differently.
- engine is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions.
- an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
- the processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
- the processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
- Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.
- a central processing unit will receive instructions and data from a read only memory or a random access memory or both.
- the essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
- the central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices.
- a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
- PDA personal digital assistant
- GPS Global Positioning System
- USB universal serial bus
- Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
- semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
- magnetic disks e.g., internal hard disks or removable disks
- magneto optical disks e.g., CD ROM and DVD-ROM disks.
- embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
- a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
- keyboard and a pointing device e.g., a mouse or a trackball
- Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser.
- a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
- Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.
- Machine learning models can be implemented and deployed using a machine learning framework, .e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.
- a machine learning framework .e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
- the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
- LAN local area network
- WAN wide area network
- the computing system can include clients and servers.
- a client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
- a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client.
- Data generated at the user device e.g., a result of the user interaction, can be received at the server from the device.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Optimization (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Pure & Applied Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
Abstract
Description
- This application claims priority to U.S. application Ser. No. 16/425,900, filed May 29, 2019, which claims priority to Greek Application No. No. 20180100232, filed on May 29, 2018. The disclosure of the prior applications are considered part of and are incorporated by reference in the disclosure of this application.
- This specification relates to determining architectures for neural networks that perform image processing tasks.
- Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
- This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an architecture for a neural network that is configured to perform a dense image prediction task.
- Generally, a dense image prediction task is a task that requires making a respective prediction for each of a large number of pixels in an input image. Many dense image prediction task require assigning a label or otherwise making a prediction for each pixel of the image. Thus, dense image prediction tasks generally require the neural network to operate on high-resolution images and to maintain the resolution of the images throughout the processing pipeline.
- To determine the architecture, the system obtains training data for the dense image prediction task. The system then determines an architecture for a neural network that is configured to perform the dense image prediction task, i.e., to receive inputs and generate outputs that conform to the requirements of the dense prediction task, using the training data.
- In particular, the system searches a space of candidate architectures to identify one or more best performing architectures using the training data. Each candidate architecture in the space of candidate architectures includes (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task. Thus, each candidate architecture includes the same neural network backbone as each other candidate architecture but has a different dense prediction cell from each other candidate architecture.
- The system then determines the architecture for the neural network based on the best performing candidate architectures.
- Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.
- The architecture search techniques described in this specification can determine a high-performing architecture for a dense prediction task in a computationally efficient manner. In particular, because dense prediction tasks require generating predictions for a large number of the pixels in the input image, such tasks require the network to operate on high resolution imagery. This makes existing architecture search techniques, e.g., techniques geared for image classification or other, non-dense image processing tasks, ill-suited for use for these tasks. This is because many of these tasks rely on low-resolution proxy tasks that would not be representative of the final dense image prediction task or require a search space to be searched that is so large as to make such searching computationally infeasible when operating on high-resolution images.
- The described techniques, on the other hand, effectively limit the search space to identifying the best architecture for a dense prediction cell, resulting in architectures that have performance that exceeds the previous state-of-the-art in multiple dense image prediction tasks.
- Moreover, by making use of the described techniques, the resulting architecture can be more computationally efficient than the previously state-of-the-art models while exceeding their performance. As examples of the kinds of results that the described techniques can achieve, the resulting architectures can achieve state-of-the-art performance on several dense prediction tasks, including achieving 82.7% mIOU accuracy on the Cityscapes data set (street scene parsing), 71.3% mIOU accuracy on the PASCAL-Person-Part data set (person-part segmentation), and 87.9% mIOU accuracy on the PASCAL VOC 2012 data set (semantic image segmentation). At the same time, the resulting architectures are more computationally efficient, requiring approximately half the parameters and half the computational cost as previous state of the art systems to achieve these levels of performance on these data sets.
- Additionally, by making use of a smaller backbone during the search than will be included in the final architecture, the amount of resources consumed by the search process is reduced. Additionally, by pre-training the backbone, e.g., on an object segmentation task, and then holding the backbone fixed during the search, the amount of resources consumed by the search process is reduced. Additionally, by pre-computing and then caching the feature maps generated by the pre-trained backbone, the amount of resources consumed by the search is reduced. As a particular example, when the backbone is smaller and the feature maps generated by the pre-trained backbone are pre-computed and cached, the system can perform the search with much less latency and much greater data efficiency (i.e., using much less memory), than techniques that rely on training candidate neural networks without pre-caching inputs and without decreasing the size of the
- The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
-
FIG. 1 shows an example neural architecture search system. -
FIG. 2 shows an example candidate architecture. -
FIG. 3 is a flow diagram of an example process for determining a final architecture. -
FIG. 4 is a flow diagram of an example process for searching the space of candidate architectures. - Like reference numbers and designations in the various drawings indicate like elements.
- This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an architecture for a neural network that is configured to perform a dense image prediction task.
- Generally, a dense image prediction task is a task that requires making a respective prediction for each of a large number of pixels in an input image. Many dense image prediction tasks require assigning a label or otherwise making a prediction for each pixel of the image. Thus, dense image prediction tasks generally require the neural network to operate on high-resolution images.
- One example of a dense image prediction task is an image segmentation task. In an segmentation task, the input is an image and the output is a respective label for every pixel in the image that classifies the content depicted at that pixel in the image.
- One example of an image segmentation task is a person-part segmentation task, where the inputs are images of one or more people and the output is a respective label for every pixel in the image such that the labels classify which pixels correspond to which person parts (e.g., head, torso, legs, and so on) and which correspond to the background (i.e., do not depict any people).
- Another example of an image segmentation task is a semantic image segmentation task. In a semantic image segmentation task, the input is an image and the output is a respective label for every pixel in the image that identifies which object class the pixel belongs to, e.g., from a set of multiple foreground object classes and one or more background object classes.
- Another example of an image segmentation task is a scene parsing task. In a scene parsing task, the input is an image and the output is a respective label for every pixel in the image that identifies which portion of a scene depicted in the image the pixel belongs to.
- Another example of a dense image prediction task is an object detection task. In an object detection task, the input is an image and the output is data that specifies which pixels of the image are parts of an image of an object. For example, the output may be a label for each pixel in the image that identifies whether the pixel is part of an image of an object. As another example, the output may be a score for each of a large number of bounding boxes in the image that indicates whether the bounding box is part of an image of an object.
-
FIG. 1 shows an example neural architecture search system 100. The neural architecture search system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. - The neural architecture search system 100 is a system that obtains
training data 102 for training a neural network to perform a dense image prediction task and a validation set 104 for evaluating the performance of the neural network on the dense image prediction task. The system then uses thetraining data 102 and the validation set 104 to determine an architecture for a neural network that is configured to perform the dense image prediction task i.e., to receive inputs and generate outputs that conform to the requirements of the dense prediction task. The architecture defines the number of layers in the neural network, the operations performed by each of the layers, and the connectivity between the layers in the neural network, i.e., which layers receive inputs from which other layers in the neural network. - Generally, the
training data 102 and the validation set 104 both include a set of training inputs and, for each training input, a respective target output that should be generated by the neural network to perform the dense image prediction task. For example, a larger set of training data may have been randomly partitioned to generate thetraining data 102 and the validation set 104. In the dense image prediction task setting, each training input is an image and the neural network output for the training input identifies the labels that should be assigned to some or all of the pixels in the training input. - The system 100 can receive the
training data 102 and the validation set 104 in any of a variety of ways. For example, the system 100 can receive training data as an upload from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system 100, and randomly divide the uploaded data into thetraining data 102 and the validation set 104. As another example, the system 100 can receive an input from a user specifying which data that is already maintained by the system 100 should be used for training the neural network, and then divide the specified data into thetraining data 102 and the validation set 104. - Generally, the system 100 determines the architecture for the neural network by searching a space of candidate architectures to identify one or more best performing architectures.
- Each candidate architecture in the space of candidate architectures includes (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task.
- Thus, each candidate architecture includes the same neural network backbone as each other candidate architecture but has a different dense prediction cell from each other candidate architecture.
-
FIG. 2 shows a candidateneural network architecture 200. As described above, thearchitecture 200 receives atraining input 220, i.e., one of the three images shown inFIG. 2 , and generates aneural network output 250 that assigns labels to the training input. As shown inFIG. 2 , theneural network output 250 is represented as an overlay over the corresponding input image, with pixels assigned the same label being in the same shade in the overlay. - The candidate neural network architecture includes a
backbone 230 that is made up of multiple neural networks layers, e.g., convolutional layers optionally combined with other types of layers (batch normalization, pooling, and so on) and receives thetraining input 220 and generates feature maps that are provided as input to a dense prediction cell (DPC) 240. Thedense prediction cell 240 processes the feature maps to generate theneural network output 250. - As indicated above, each candidate neural network in the search space has the
same backbone 230 but all of the candidate neural networks havedifferent DPCs 240. Thus, the search space for the architecture search is the space of possible architectures for theDPC 240. - An example of a space of possible architectures for the
DPC 240 that can be searched by the system 100 follows. - For example, the DPC can have B branches, where B is a fixed integer greater than one. Each of the B branches of the DPC maps an input tensor to the branch to an output tensor by applying an operation to the input tensor. The DPC can then combine, e.g., concatenate, the output tensors generated by all of the B branches to generate a combined tensor output or use the output tensor generated by a designated one of the B branches, e.g., the last of the B branches in a processing order, as the combined tensor output. B can be, e.g., an integer in the range of 3 to 10, inclusive, with larger values of B allowing more flexibility and a larger search space but increasing the computational cost of the search process.
- In some cases, the combined tensor output can be used as the output of the
DPC 240 for the dense prediction task. In other cases, the DPC processes the combined tensor output through one or more output layers, e.g., a sigmoid output layer, to generate a final output. - Thus, the different architectures in the search space each specify (i) an input tensor from a set of input tensors to be provided as input to each of the B branches, and (ii) an operation from a set of operations to be performed by each of the B branches to generate the output tensor from the input tensor.
- The input tensor for a given branch can be selected from a set that includes (i) the feature maps generated by the backbone and (ii) output tensors generated by any branches before the given branch in the processing order of the B branches. Thus, for the first branch in the processing order, the set includes only the feature maps generated by the backbone while for the last of the B branches in the processing order, the set includes (i) the feature maps generated by the backbone and (ii) output tensors generated by any of the B−1 other branches.
- The operator space, i.e., the space of possible operations from which the operation performed by each of the B blocks is selected, can include one or more of the following: (1) a convolution with a 1×1 kernel, (2) one or more atrous separable convolutions, each having a different sampling rate, and (3) one or more spatial pyramid pooling operations, each having a respective grid size.
- When the operator space includes multiple atrous separable convolutions, each of the atrous separable convolutions will have a different sampling rate. For example, the sampling rate can be defined as r_h×r_w, where each of r_h and r_w is selected from the set of {1, 2, 6, 9, . . . , 21}.
- When the operator space includes multiple spatial pyramid pooling operations, each of the spatial pyramid pooling operations will have a different grid size. For example, the grid size can be defined as g_h×g_w, where each of g_h and g_w is selected from the set of {1, 2, 4, 8}.
- Other examples of search spaces can include different operator spaces and different possible values for the sampling rates and grid sizes of the operators in the space.
- Returning to the description of
FIG. 1 , the system 100 includes acandidate selection engine 130, atraining engine 140, and aquality evaluation engine 150. - To search the space of candidate architectures, the system repeatedly performs a set of architecture search operations.
- At each iteration of the operations, the
candidate selection engine 130 selects one or more candidate architectures from the space of possible candidate architectures using quality measures for candidate architectures that have already been evaluated. - The
training engine 140 then trains the selected candidate architectures on at least some of thetraining data 102 and thequality evaluation engine 150 evaluates the trained candidate architectures using the validation set 104. - Selecting candidate architectures, training the candidate architectures, and evaluating the quality of the architectures is described in more detail below with reference to
FIGS. 3 and 4 . - After the architecture search operations have been repeatedly performed, the system 100 can identify as the best performing candidate architectures a threshold, fixed number of candidate architectures evaluated during the search that had the best quality measures.
- Once the system 100 has identified the best performing candidate architectures, the system 100 determines the final architecture for the neural network based on the best performing candidate architectures.
- For example, the system 100 can generate, from each of the identified best performing candidates, a final architecture, and then train the final architectures to convergence on the dense prediction task. The system can then select the best performing, e.g., as determined based on a quality measure on the validation set 104, trained architecture as the architecture of the neural network.
- In some implementations, the candidate architecture and the corresponding final architecture are the same.
- In other implementations, however, the system 100 replaces the neural network backbone with a different, larger neural network backbone that has more parameters and that allows the final neural network to perform better on the dense prediction task.
- In other words, in some cases, the system uses a smaller backbone for the architecture search than is employed by the final architecture.
- For example, the system can employ an Xception architecture for the backbone in the final architecture while employing the MobileNet-v2 architecture for the candidate architectures, i.e., during the search. The MobileNet-v2 architecture requires roughly one twentieth the computational cost of the Xception architecture and cuts down the number of channels in the backbone feature maps from 2048 to 320 dimensions. The Xception architecture is described in more detail in F. Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017 while the MobileNet-v2 architecture is described in more detail in M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
- Using the smaller backbone can allow the system to make architecture search more computationally efficient while still providing a quality signal for how well the final architecture having the larger backbone will perform on the dense image prediction task.
- Additional techniques that can be employed by the system to improve the computational efficiency of the search are described below with reference to
FIG. 4 . - The neural network search system 100 can then
output architecture data 152 that specifies the final architecture of the neural network, i.e., data specifying the layers that are part of the neural network, the connectivity between the layers, and the operations performed by the layers. For example, the neural network search system 100 can output thearchitecture data 152 to the user that submitted the training data. - In some implementations, instead of or in addition to outputting the
architecture data 152, the system 100 uses the trained neural network having the final architecture to process requests received by users, e.g., through the API provided by the system. That is, the system 100 can receive inputs to be processed, use the trained neural network to process the inputs, and provide the outputs generated by the trained neural network or data derived from the generated outputs in response to the received inputs. -
FIG. 3 is a flow diagram of an example process 300 for determining a final architecture for a dense image prediction task. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural architecture search system, e.g., the neural architecture search system 100 ofFIG. 1 , appropriately programmed, can perform the process 300. - The system receives training data for the dense image prediction task (step 302).
- The system searches a space of candidate architectures to identify one or more best performing architectures using the training data (step 304). Searching the space of candidate architectures is described in more detail below with reference to
FIG. 4 . Once the search has completed, the system can select, as the best performing architectures, a threshold number of candidate architectures that had the best quality evaluation during the search. - The system determines the architecture for the neural network based on the one or more best performing candidate architectures (step 306). For example, the system can generate, from each of the identified best performing candidates, a final architecture, and then train the final architectures to convergence on the dense prediction task. The system can then select the best performing trained architecture, e.g., as determined based on a quality measure on the validation set, as the architecture of the neural network. The quality measure that is used can be the same as the one described below with reference to step 410.
- In some implementations, the candidate architecture and the corresponding final architecture are the same.
- In other implementations, however, the system replaces the neural network backbone with a different, larger neural network backbone that has more parameters and that allows the final neural network to perform better on the dense prediction task.
- In other words, in some cases, the system uses a smaller backbone for the architecture search than is employed by the final architecture.
-
FIG. 4 is a flow diagram of anexample process 400 for searching the space of candidate architectures. For convenience, theprocess 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural architecture search system, e.g., the neural architecture search system 100 ofFIG. 1 , appropriately programmed, can perform theprocess 400. - The system obtains data specifying a pre-trained backbone for use in searching the space of candidate architectures (step 402). In some cases, the system pre-trains the backbone along with a placeholder DPC on a placeholder dense image prediction task to determine trained values of the parameters of the backbone, i.e., so that the system only needs train the backbone once for the entire search process. In some other cases, the system obtains data specifying the trained values of the backbone parameters from another system that has already pre-trained the backbone.
- Optionally, the system processes at least some of the training inputs in the training data using the pre-trained backbone to generate feature maps for the training inputs (step 404). In other words, the system processes each of the training inputs using the pre-trained backbone, i.e., in accordance with the trained values of the parameters of the backbone, to generate feature maps and then stores the generated feature maps for use during the search process. When the system is using the smaller backbone for the search, because the feature maps generated by the smaller backbone have many fewer channels than the feature maps generated by the larger backbone (that will be used in the final architecture), storing the feature maps requires much less storage space than would be required to store feature maps generated by the larger backbone.
- The system then repeatedly performs steps 406-410 until termination criteria for the search are satisfied, e.g., until a threshold number of candidate architectures have been evaluated, until the highest performing candidate architecture reaches a threshold accuracy, or until a threshold amount of time has elapsed.
- The system selects one or more candidate architectures from the space of candidate architectures (step 406). The system can use any of a variety of techniques for searching the space to select the candidate architectures.
- For example, the system can use a random search strategy. In a random search strategy, at each iteration of the
process 400, the system selects one or more architectures from the space of candidate architectures uniformly at random while also selecting one or more architectures that are close to, i.e., are similar to, the currently best observed architectures, i.e., the architectures already evaluated as part of the search that have been found to perform best on the dense prediction task. Random search strategies that can be employed by the system are described in more detail in D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. Karro, and D. Sculley. Google vizier: A service for black-box optimization. In SIGKDD, 2017 and in B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. - As another example, the system can use a reinforcement learning guided search strategy to select the candidate actions. In particular, at each iteration of the
process 400, the system can select architectures that are output by a recurrent neural network that is being trained through reinforcement learning to output candidate architectures that perform well on the task. Examples of reinforcement learning guided search strategies that can be employed by the system are described in more detail in B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017. - The system trains the selected one or more candidate architectures on at least a portion of the training data (step 408). That is, for each selected candidate, the system trains a neural network having the architecture until criteria for stopping the training are satisfied. During the training, the system holds the trained values of the backbone parameters fixed while updating the values of the parameters of the DPC in the neural network.
- In particular, in implementations where
step 404 was performed to pre-process the training data using the pre-trained backbone, the system does not process the training inputs using the backbone duringstep 408 and instead accesses the pre-generated feature maps from memory and provides the pre-generated feature maps as input to the DPC. This greatly decreases the processing power and time consumed by the training, since training inputs do not need to be processed through backbone to generate the feature maps before the feature maps can processed by the DPC. - The system can perform this training using an early stopping criterion, i.e., can train each candidate for a fixed number of iterations instead of to convergence.
- The system evaluates the performance of each of the trained candidate architectures (step 410). The system uses an evaluation metric that measures the performance of the trained neural network having the candidate architecture on the validation set to evaluate the performance.
- Any evaluation metric that is appropriate for dense prediction tasks can be used. For example, the system can use the pixel-wise mean intersection-over-union (mIOU) of the trained neural network over the validation data set as the evaluation metric. As another example, the system can use the mean pixel accuracy over the validation data set of the trained neural network as the evaluation metric.
- This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
- Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
- The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
- A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
- In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.
- Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
- The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
- Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
- Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
- To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
- Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.
- Machine learning models can be implemented and deployed using a machine learning framework, .e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
- The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.
- While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
- Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
- Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/107,745 US20210081796A1 (en) | 2018-05-29 | 2020-11-30 | Neural architecture search for dense image prediction tasks |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GR20180100232 | 2018-05-29 | ||
GR20180100232 | 2018-05-29 | ||
US16/425,900 US10853726B2 (en) | 2018-05-29 | 2019-05-29 | Neural architecture search for dense image prediction tasks |
US17/107,745 US20210081796A1 (en) | 2018-05-29 | 2020-11-30 | Neural architecture search for dense image prediction tasks |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/425,900 Continuation US10853726B2 (en) | 2018-05-29 | 2019-05-29 | Neural architecture search for dense image prediction tasks |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210081796A1 true US20210081796A1 (en) | 2021-03-18 |
Family
ID=66998484
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/425,900 Active US10853726B2 (en) | 2018-05-29 | 2019-05-29 | Neural architecture search for dense image prediction tasks |
US17/107,745 Pending US20210081796A1 (en) | 2018-05-29 | 2020-11-30 | Neural architecture search for dense image prediction tasks |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/425,900 Active US10853726B2 (en) | 2018-05-29 | 2019-05-29 | Neural architecture search for dense image prediction tasks |
Country Status (4)
Country | Link |
---|---|
US (2) | US10853726B2 (en) |
EP (1) | EP3732631A1 (en) |
CN (1) | CN111819580A (en) |
WO (1) | WO2019232099A1 (en) |
Families Citing this family (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019101836A1 (en) * | 2017-11-22 | 2019-05-31 | Deepmind Technologies Limited | Population based training of neural networks |
CN110136136B (en) * | 2019-05-27 | 2022-02-08 | 北京达佳互联信息技术有限公司 | Scene segmentation method and device, computer equipment and storage medium |
US11829871B2 (en) * | 2019-08-20 | 2023-11-28 | Lg Electronics Inc. | Validating performance of a neural network trained using labeled training data |
CN110826696B (en) * | 2019-10-30 | 2023-06-27 | 北京百度网讯科技有限公司 | Super-network search space construction method and device and electronic equipment |
WO2021094526A1 (en) * | 2019-11-15 | 2021-05-20 | Ariel Ai Ltd | Image generation using surface-based neural synthesis |
CN111144407A (en) * | 2019-12-22 | 2020-05-12 | 浪潮(北京)电子信息产业有限公司 | Target detection method, system, device and readable storage medium |
US11620487B2 (en) * | 2019-12-31 | 2023-04-04 | X Development Llc | Neural architecture search based on synaptic connectivity graphs |
CN111401517B (en) * | 2020-02-21 | 2023-11-03 | 华为技术有限公司 | Method and device for searching perceived network structure |
CN111325328B (en) * | 2020-03-06 | 2023-10-24 | 上海商汤临港智能科技有限公司 | Neural network generation method, data processing method and device |
CN115335830A (en) * | 2020-03-23 | 2022-11-11 | 谷歌有限责任公司 | Neural architecture search with weight sharing |
CN113469352A (en) * | 2020-03-31 | 2021-10-01 | 上海商汤智能科技有限公司 | Neural network model optimization method, data processing method and device |
CN111695673B (en) * | 2020-05-09 | 2024-05-24 | 华为技术有限公司 | Method for training neural network predictor, image processing method and device |
CN111582478B (en) * | 2020-05-09 | 2023-09-22 | 北京百度网讯科技有限公司 | Method and device for determining model structure |
CN111914613B (en) * | 2020-05-21 | 2024-03-01 | 淮阴工学院 | Multi-target tracking and facial feature information recognition method |
CN111767988A (en) * | 2020-06-29 | 2020-10-13 | 北京百度网讯科技有限公司 | Neural network fusion method and device |
CN111950702A (en) * | 2020-07-16 | 2020-11-17 | 华为技术有限公司 | Neural network structure determining method and device |
CN111898683B (en) * | 2020-07-31 | 2023-07-28 | 平安科技(深圳)有限公司 | Image classification method and device based on deep learning and computer equipment |
CN112070729B (en) * | 2020-08-26 | 2023-07-07 | 西安交通大学 | Anchor-free remote sensing image target detection method and system based on scene enhancement |
KR20220032861A (en) * | 2020-09-08 | 2022-03-15 | 삼성전자주식회사 | Neural architecture search method and attaratus considering performance in hardware |
JP7423484B2 (en) * | 2020-09-16 | 2024-01-29 | 株式会社東芝 | Learning devices, methods and programs |
CN112200223A (en) * | 2020-09-22 | 2021-01-08 | 北京迈格威科技有限公司 | Image recognition network construction method, device, equipment and medium |
CN112132810B (en) * | 2020-09-24 | 2023-09-12 | 西安电子科技大学 | Image significance detection method based on perception logic and feature contrast |
CN112183627A (en) * | 2020-09-28 | 2021-01-05 | 中星技术股份有限公司 | Method for generating predicted density map network and vehicle annual inspection mark number detection method |
US11972348B2 (en) * | 2020-10-30 | 2024-04-30 | Apple Inc. | Texture unit circuit in neural network processor |
US20220207350A1 (en) * | 2020-12-30 | 2022-06-30 | International Business Machines Corporation | Identifying related messages in a natural language interaction |
TWI764542B (en) * | 2021-01-27 | 2022-05-11 | 國立臺灣大學 | Autonomous intelligent vehicle real-time visual localization and uncertainty estimation system |
US11983239B2 (en) | 2021-06-08 | 2024-05-14 | Lemon Inc. | Lightweight transformer for high resolution images |
US20220391636A1 (en) * | 2021-06-08 | 2022-12-08 | Lemon Inc. | Multi-resolution neural network architecture search space for dense prediction tasks |
CN114781499B (en) * | 2022-04-08 | 2024-05-31 | 上海人工智能创新中心 | Method for constructing ViT model-based intensive prediction task adapter |
Citations (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9524450B2 (en) * | 2015-03-04 | 2016-12-20 | Accenture Global Services Limited | Digital image processing using convolutional neural networks |
US20170262736A1 (en) * | 2016-03-11 | 2017-09-14 | Nec Laboratories America, Inc. | Deep Deformation Network for Object Landmark Localization |
US20180060701A1 (en) * | 2016-08-31 | 2018-03-01 | Adobe Systems Incorporated | Deep-learning network architecture for object detection |
US20180218502A1 (en) * | 2017-01-27 | 2018-08-02 | Arterys Inc. | Automated segmentation utilizing fully convolutional networks |
US20180253622A1 (en) * | 2017-03-06 | 2018-09-06 | Honda Motor Co., Ltd. | Systems for performing semantic segmentation and methods thereof |
US20180338159A1 (en) * | 2017-05-17 | 2018-11-22 | Samsung Electronics Co,. Ltd. | Super-resolution processing method for moving image and image processing apparatus therefor |
US20180341872A1 (en) * | 2016-02-02 | 2018-11-29 | Beijing Sensetime Technology Development Co., Ltd | Methods and systems for cnn network adaption and object online tracking |
US20190012578A1 (en) * | 2017-07-07 | 2019-01-10 | Carnegie Mellon University | 3D Spatial Transformer Network |
US20190043203A1 (en) * | 2018-01-12 | 2019-02-07 | Intel Corporation | Method and system of recurrent semantic segmentation for image processing |
US20190057507A1 (en) * | 2017-08-18 | 2019-02-21 | Samsung Electronics Co., Ltd. | System and method for semantic segmentation of images |
US20190114510A1 (en) * | 2017-10-12 | 2019-04-18 | Lawrence Livermore National Security, Llc | Multi-scale deep learning system |
US20190156178A1 (en) * | 2017-11-22 | 2019-05-23 | Massachusetts Institute Of Technology | Systems and methods for optimization of a data model network architecture for target deployment |
US20190172223A1 (en) * | 2017-12-03 | 2019-06-06 | Facebook, Inc. | Optimizations for Dynamic Object Instance Detection, Segmentation, and Structure Mapping |
US20190243928A1 (en) * | 2017-12-28 | 2019-08-08 | Dassault Systemes | Semantic segmentation of 2d floor plans with a pixel-wise classifier |
US20190294970A1 (en) * | 2018-03-23 | 2019-09-26 | The Governing Council Of The University Of Toronto | Systems and methods for polygon object annotation and a method of training an object annotation system |
US20190303762A1 (en) * | 2018-03-30 | 2019-10-03 | Xilinx, Inc. | Methods of optimization of computational graphs of neural networks |
US20190354837A1 (en) * | 2018-05-18 | 2019-11-21 | Baidu Usa Llc | Resource-efficient neural architects |
US20200005074A1 (en) * | 2017-03-27 | 2020-01-02 | Intel Corporation | Semantic image segmentation using gated dense pyramid blocks |
US10614574B2 (en) * | 2017-10-16 | 2020-04-07 | Adobe Inc. | Generating image segmentation data using a multi-branch neural network |
US20200143205A1 (en) * | 2017-08-10 | 2020-05-07 | Intel Corporation | Convolutional neural network framework using reverse connections and objectness priors for object detection |
US20200219268A1 (en) * | 2018-03-06 | 2020-07-09 | Beijing Sensetime Technology Development Co., Ltd. | Target tracking methods and apparatuses, electronic devices, and storage media |
US20200293899A1 (en) * | 2017-10-27 | 2020-09-17 | Deepmind Technologies Limited | Using Hierarchical Representations for Neural Network Architecture Searching |
US20200410347A1 (en) * | 2018-04-24 | 2020-12-31 | Robert Bosch Gmbh | Method and device for ascertaining a network configuration of a neural network |
US20210012183A1 (en) * | 2018-04-24 | 2021-01-14 | Robert Bosch Gmbh | Method and device for ascertaining a network configuration of a neural network |
US20210133540A1 (en) * | 2018-05-29 | 2021-05-06 | The Trustees Of Princeton University | System and method for compact, fast, and accurate lstms |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9165243B2 (en) * | 2012-02-15 | 2015-10-20 | Microsoft Technology Licensing, Llc | Tensor deep stacked neural network |
US20160013773A1 (en) * | 2012-11-06 | 2016-01-14 | Pavel Dourbal | Method and apparatus for fast digital filtering and signal processing |
US20140181171A1 (en) * | 2012-12-24 | 2014-06-26 | Pavel Dourbal | Method and system for fast tensor-vector multiplication |
US9373057B1 (en) * | 2013-11-01 | 2016-06-21 | Google Inc. | Training a neural network to detect objects in images |
US9471377B2 (en) * | 2013-11-13 | 2016-10-18 | Reservoir Labs, Inc. | Systems and methods for parallelizing and optimizing sparse tensor computations |
US20170032285A1 (en) * | 2014-04-09 | 2017-02-02 | Entrupy Inc. | Authenticating physical objects using machine learning from microscopic variations |
US9881234B2 (en) * | 2015-11-25 | 2018-01-30 | Baidu Usa Llc. | Systems and methods for end-to-end object detection |
US10372968B2 (en) * | 2016-01-22 | 2019-08-06 | Qualcomm Incorporated | Object-focused active three-dimensional reconstruction |
US10354159B2 (en) * | 2016-09-06 | 2019-07-16 | Carnegie Mellon University | Methods and software for detecting objects in an image using a contextual multiscale fast region-based convolutional neural network |
US10657364B2 (en) * | 2016-09-23 | 2020-05-19 | Samsung Electronics Co., Ltd | System and method for deep network fusion for fast and robust object detection |
DE102017125256A1 (en) * | 2016-10-28 | 2018-05-03 | Google Llc | Search for a neural architecture |
US9939272B1 (en) * | 2017-01-06 | 2018-04-10 | TCL Research America Inc. | Method and system for building personalized knowledge base of semantic image segmentation via a selective random field approach |
CN107688849B (en) * | 2017-07-28 | 2021-04-13 | 赛灵思电子科技(北京)有限公司 | Dynamic strategy fixed-point training method and device |
CN109145911A (en) * | 2017-11-15 | 2019-01-04 | 中国石油大学(华东) | A kind of street is taken a picture target person extracting method |
US11995538B2 (en) * | 2018-05-10 | 2024-05-28 | Microsoft Technology Licensing, Llc | Selecting a neural network architecture for a supervised machine learning problem |
-
2019
- 2019-05-29 CN CN201980013495.2A patent/CN111819580A/en active Pending
- 2019-05-29 US US16/425,900 patent/US10853726B2/en active Active
- 2019-05-29 WO PCT/US2019/034467 patent/WO2019232099A1/en unknown
- 2019-05-29 EP EP19732165.6A patent/EP3732631A1/en active Pending
-
2020
- 2020-11-30 US US17/107,745 patent/US20210081796A1/en active Pending
Patent Citations (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9524450B2 (en) * | 2015-03-04 | 2016-12-20 | Accenture Global Services Limited | Digital image processing using convolutional neural networks |
US20180341872A1 (en) * | 2016-02-02 | 2018-11-29 | Beijing Sensetime Technology Development Co., Ltd | Methods and systems for cnn network adaption and object online tracking |
US20170262736A1 (en) * | 2016-03-11 | 2017-09-14 | Nec Laboratories America, Inc. | Deep Deformation Network for Object Landmark Localization |
US20180060701A1 (en) * | 2016-08-31 | 2018-03-01 | Adobe Systems Incorporated | Deep-learning network architecture for object detection |
US20180218502A1 (en) * | 2017-01-27 | 2018-08-02 | Arterys Inc. | Automated segmentation utilizing fully convolutional networks |
US20180253622A1 (en) * | 2017-03-06 | 2018-09-06 | Honda Motor Co., Ltd. | Systems for performing semantic segmentation and methods thereof |
US20200005074A1 (en) * | 2017-03-27 | 2020-01-02 | Intel Corporation | Semantic image segmentation using gated dense pyramid blocks |
US20180338159A1 (en) * | 2017-05-17 | 2018-11-22 | Samsung Electronics Co,. Ltd. | Super-resolution processing method for moving image and image processing apparatus therefor |
US20190012578A1 (en) * | 2017-07-07 | 2019-01-10 | Carnegie Mellon University | 3D Spatial Transformer Network |
US20200143205A1 (en) * | 2017-08-10 | 2020-05-07 | Intel Corporation | Convolutional neural network framework using reverse connections and objectness priors for object detection |
US20190057507A1 (en) * | 2017-08-18 | 2019-02-21 | Samsung Electronics Co., Ltd. | System and method for semantic segmentation of images |
US20190114510A1 (en) * | 2017-10-12 | 2019-04-18 | Lawrence Livermore National Security, Llc | Multi-scale deep learning system |
US10614574B2 (en) * | 2017-10-16 | 2020-04-07 | Adobe Inc. | Generating image segmentation data using a multi-branch neural network |
US20200293899A1 (en) * | 2017-10-27 | 2020-09-17 | Deepmind Technologies Limited | Using Hierarchical Representations for Neural Network Architecture Searching |
US20190156178A1 (en) * | 2017-11-22 | 2019-05-23 | Massachusetts Institute Of Technology | Systems and methods for optimization of a data model network architecture for target deployment |
US20190172223A1 (en) * | 2017-12-03 | 2019-06-06 | Facebook, Inc. | Optimizations for Dynamic Object Instance Detection, Segmentation, and Structure Mapping |
US20190243928A1 (en) * | 2017-12-28 | 2019-08-08 | Dassault Systemes | Semantic segmentation of 2d floor plans with a pixel-wise classifier |
US20190043203A1 (en) * | 2018-01-12 | 2019-02-07 | Intel Corporation | Method and system of recurrent semantic segmentation for image processing |
US20200219268A1 (en) * | 2018-03-06 | 2020-07-09 | Beijing Sensetime Technology Development Co., Ltd. | Target tracking methods and apparatuses, electronic devices, and storage media |
US20190294970A1 (en) * | 2018-03-23 | 2019-09-26 | The Governing Council Of The University Of Toronto | Systems and methods for polygon object annotation and a method of training an object annotation system |
US20190303762A1 (en) * | 2018-03-30 | 2019-10-03 | Xilinx, Inc. | Methods of optimization of computational graphs of neural networks |
US20200410347A1 (en) * | 2018-04-24 | 2020-12-31 | Robert Bosch Gmbh | Method and device for ascertaining a network configuration of a neural network |
US20210012183A1 (en) * | 2018-04-24 | 2021-01-14 | Robert Bosch Gmbh | Method and device for ascertaining a network configuration of a neural network |
US20190354837A1 (en) * | 2018-05-18 | 2019-11-21 | Baidu Usa Llc | Resource-efficient neural architects |
US20210133540A1 (en) * | 2018-05-29 | 2021-05-06 | The Trustees Of Princeton University | System and method for compact, fast, and accurate lstms |
Non-Patent Citations (26)
Title |
---|
Ahmed et Torresani, "Connectivity Learning in Multi-Branch Networks" 7 Dec 2017, arXiv: 1709.09582v2, pp. 1-17. (Year: 2017) * |
Badrinarayanan et al., "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation" 10 Oct 2016, arXiv: 1511.00561v3, pp. 1-14. (Year: 2018) * |
Baker et al., "Accelerating Neural Architecture Search using Performance Prediction" 8 Nov 2017, arXiv: 1705.10823v2, pp. 1-14. (Year: 2017) * |
Brock et al., "SMASH: One-Shot Architecture Search through HyperNetworks" 17 Aug 2017, arXiv: 1708.05344v1, pp. 1-21. (Year: 2017) * |
Cai et al., "Efficient Architecture Search by Network Transformation" 21 Nov 2017, arXiv: 1707.04873v2, pp. 1-8. (Year: 2017) * |
Chen et al., "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs" 2 Jun 2016, arXiv: 1606.00915v1, pp. 1-14. (Year: 2016) * |
Czarnowski et al., "Semantic Texture for Robust Dense Tracking" 29 Aug 2017, arXiv: 1708.08844v1, pp. 1-9. (Year: 2017) * |
Elsken et al., "Multi-objective Architecture Search" 24 Apr 2018, arXiv: 1804.09081v1, pp. 1-10. (Year: 2018) * |
Gao et Ji, "Efficient and Invariant Convolutional Neural Networks for Dense Prediction" 24 Nov 2017, arXiv: 1711.09064v1, pp. 1-6. (Year: 2017) * |
Gastaldi, Xavier "Shake-Shake regularization" 23 May 2017, arXiv: 1705.07485v2, pp. 1-10. (Year: 2017) * |
Huang et al., "GNAS: A Greedy Neural Architecture Search Method for Multi-Attribute Learning" 19 Apr 2018, arXiv: 1804.06964v1, pp. 1-9. (Year: 2018) * |
Jegou et al., "The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation" 31 Oct 2017, arXiv: 1611.09326v3, pp. 1-9. (Year: 2017) * |
Lin et al., "Feature Pyramid Networks for Object Detection" 9 Dec 2016, arXiv: 1612.03144v1, pp. 1-10. (Year: 2016) * |
Liu et al., "Hierarchical Representations for Efficient Architecture Search" 1 Nov 2017, arXiv: 1711.00436v1, pp. 1-13. (Year: 2017) * |
Liu et al., "Hierarchical Representations for Efficient Architecture Search" 22 Feb 2018, arXiv: 1711.00436v2, pp. 1-13. (Year: 2018) * |
Lou et al., "PT-Net: Improve Object and Face Detection Via a Pre-Trained CNN Model" 08 Mar 2018, pp. 1280-1284. (Year: 2018) * |
Negrinho et Gordon, "DeepArchitect: Automatically Designing and Training Deep Architectures" 28 Apr 2017, pp. 1-12. (Year: 2017) * |
Shin et al., "Differentiable Neural Network Architecture Search" 12 Feb 2018, pp. 1-5. (Year: 2018) * |
Suganuma et al., "A Genetic Programming Approach to Designing Convolutional Neural Network Architectures" 11 Aug 2017, arXiv: 1704.00764v2, pp. 1-9. (Year: 2017) * |
Wang et al., "Multi-scale Location-aware Kernel Representation for Object Detection" 2 Apr 2018, arXiv: 1804.00428v1, pp. 1-10. (Year: 2018) * |
Wistuba et al., "Finding Competitive Network Architectures Within a Day Using UCT" 20 Dec 2017, arXiv: 1712.07420v1, pp. 1-9. (Year: 2017) * |
Xie et al., "Vortex Pooling: Improving Context Representation in Semantic Segmentation" 22 Apr 2018, arXiv: 1804.06242v2, pp. 1-8. (Year: 2018) * |
Zhang et al., "Context Encoding for Semantic Segmentation" 23 Mar 2018, arXiv: 1803.08904v1, pp. 1-11. (Year: 2018) * |
Zhou et al., "Action Unit Selective Feature Maps in Deep Networks for Facial Expression Recognition" Jul 2017, pp. 2031-2038. (Year: 2017) * |
Zhou et al., "Resource-Efficient Neural Architect" 18 May 2018, US Provisional 62/673,309, pp. 1-19. (Year: 2018) * |
Zoph et al., "Neural Architecture Search with Reinforcement Learning" 5 Nov 2016, arXiv: 1611.01578v1, pp. 1-15. (Year: 2016) * |
Also Published As
Publication number | Publication date |
---|---|
WO2019232099A1 (en) | 2019-12-05 |
CN111819580A (en) | 2020-10-23 |
EP3732631A1 (en) | 2020-11-04 |
US10853726B2 (en) | 2020-12-01 |
US20190370648A1 (en) | 2019-12-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10853726B2 (en) | Neural architecture search for dense image prediction tasks | |
US11087201B2 (en) | Neural architecture search using a performance prediction neural network | |
US11669744B2 (en) | Regularized neural network architecture search | |
US11544536B2 (en) | Hybrid neural architecture search | |
US11847541B2 (en) | Training neural networks using data augmentation policies | |
KR20200110400A (en) | Learning data augmentation policy | |
US20220121906A1 (en) | Task-aware neural network architecture search | |
US11922281B2 (en) | Training machine learning models using teacher annealing | |
CN111652378B (en) | Learning to select vocabulary for category features | |
US11144782B2 (en) | Generating video frames using neural networks | |
WO2020102797A1 (en) | Multimodal image classifier using textual and visual embeddings | |
US20210407042A1 (en) | Generating super-resolution images using neural networks | |
CN113424199A (en) | Composite model scaling for neural networks | |
US20220383036A1 (en) | Clustering data using neural networks based on normalized cuts | |
WO2020106871A1 (en) | Image processing neural networks with dynamic filter activation | |
EP4182850A1 (en) | Hardware-optimized neural architecture search | |
US20220036172A1 (en) | Olfactory predictions using neural networks | |
US11900222B1 (en) | Efficient machine learning model architecture selection | |
CN117726889A (en) | Model training method, sampling strategy determining method, device and computing equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
AS | Assignment |
Owner name: GOOGLE LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZOPH, BARRET;SHLENS, JONATHON;ZHU, YUKUN;AND OTHERS;SIGNING DATES FROM 20190614 TO 20190617;REEL/FRAME:055201/0205 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |