DE102020206506A1 - Method for testing different architectures of an artificial neural network - Google Patents

Abstract

The invention relates to a method for testing different architectures (132) of an artificial neural network (108), the artificial neural network (108) comprising a defined sequence of filter layers (110), each of the filter layers (110) having a defined number of trainable filters (116) of a defined filter size for converting an input (112) into an output (114). The method comprises the following steps: generating test requests (122) by a test generation module (118) and sending the test requests (122) from the test generation module (118) to a test system (104) for executing one based on the artificial neural network (108) Algorithm; Carrying out at least two series (128) of tests (130) based on the test queries (122) by the test system (104), a different architecture (132) of the artificial neural network (108) being used for each series (128), wherein the architectures (132) of different rows (128) differ from one another at least in a sequence of the filter sizes, with different settings being determined for at least one further parameter (134a, 134b, 134c) of the artificial neural network (108), with each test ( 130) a different setting is used within a row (128), inputs (112) being entered into the artificial neural network (108) in each of the tests (130) and a test result (136) being determined which indicates a performance of the test system (104) in processing the inputs (112) by the artificial neural network (108); Sending the test results (136) from the test system (104) to a test evaluation module (120); and evaluating the test results (136) by the test evaluation module (120).

Description

The invention relates to a method for testing different architectures of an artificial neural network. The invention also relates to a test generation and evaluation system, a test system, a computer system, a computer program and a computer-readable medium for carrying out the method.

Artificial neural networks such as convolutional neural networks (CNNs) can be implemented on hardware systems with specially adapted accelerator chips. Examples of this are the Versal ™ platform and the Xavier ™ platform.

A given model of an artificial neural network can, for example, be adapted to such a hardware system with special porting software. The porting software is usually provided by the manufacturer of the hardware system. The adaptation of the model to the hardware system can therefore be subject to certain restrictions.

Alternatively, it is possible to have the hardware manufacturer adapt the hardware system specifically to the model. For reasons of confidentiality, however, the release of the model is often not desired.

It is also possible that manufacturer information regarding an achievable number of operations per second, also called int8 TOPS, does not allow a reliable conclusion about the actual performance of the artificial neural network, e.g. about a latency when processing a single input or a throughput of inputs.

In order to measure latencies and throughputs, benchmarks can be carried out on architectures such as ResNet-18, Resnet-50 or MobileNet, such as the MLPerf Inference Benchmark. However, these benchmarks only allow a rough assessment of the capabilities of the underlying platform.

Another approach are automated methods for generating artificial neural networks, also known as Neural Architecture Search (NAS). However, even with this it is hardly possible to draw more precise conclusions about certain capabilities of the underlying platform.

The object of the invention is to simplify the adaptation of an artificial neural network to a computer system. In particular, it is the object of the invention to create an improved method for testing different architectures of an artificial neural network on a computer system.

This problem is solved by the subject matter of the independent claims. Further embodiments of the invention emerge from the dependent claims and from the following description.

A first aspect of the invention relates to a computer-implemented method for testing different architectures of an artificial neural network. The method can for example be carried out automatically by an appropriate computer system. The computer system can comprise software and / or hardware modules. Among other things, the computer system can include a processor, for example in the form of a CPU, GPU or TPU (= Tensor Processing Unit), a memory and a bus system for data communication between the processor and the memory. In addition, the computer system can comprise one or more interfaces for data communication with peripheral devices.

The artificial neural network comprises a defined sequence of filter layers. Each of the filter layers comprises a defined number of trainable filters of a defined filter size for converting an input into an output.

An artificial neural network can be a convolutional neural network, for example. A filter layer can be a convolutional layer, for example.

Depending on the architecture, the artificial neural network can for example have more than ten filter layers, in particular about 18, 34 or 50 filter layers. Architectures with more than 50 filter layers are also possible.

An input can be a two- or three-dimensional input matrix of numerical values. The numerical values can, for example, colors and / or brightnesses of picture elements (pixels) of an image or of an image sensor. A depth of the input matrix can correspond, for example, to a number of color channels of the image or of the image sensor.

A filter can be a two- or three-dimensional weight matrix made up of trainable weights. The weights can be trainable, for example, by backpropagation. The weight matrix, which can also be referred to as the kernel, can, for example, be moved step by step with a defined step size in the x and y directions via an input. For example, an output value can be calculated in each step by multiplying each of the weights with a corresponding value of a currently underlying section of the input, adding up the resulting products and adding the resulting sum to a bias value.

The filter layers can be arranged one after the other, so that each of the filter layers, with the exception of the foremost filter layer, receives outputs of a respective preceding filter layer as inputs.

A filter layer can have several different filters, for example. The filters can differ from one another in terms of their weights. Each of the different filters can represent a filter channel or output channel of the filter layer. Different filter layers can have the same number or a different number of filters. For example, the filter layers can be arranged in ascending order with regard to their number of filters, starting with a filter layer with the smallest number of filters. However, any other desired sequences of the numbers of filters are also possible. For example, depending on the architecture, the filter layers can each have between 64 and 2048 output channels. However, filter layers with fewer than 64 or more than 2048 output channels are also possible.

According to one embodiment of the method, test queries are generated by a test generation module and sent from the test generation module to a test system for executing an algorithm based on the artificial neural network. Based on the test requests, at least two series of tests are carried out by the test system.

A test system can be understood to mean a system to be tested. The test system can comprise hardware and / or software modules. For example, the test system can be equipped with a special hardware accelerator for executing the algorithm.

For example, a test request can be a request for a single test, for a series of tests, or for multiple series of tests. The test request can include, for example, commands that cause the test system to execute a computer program that executes the algorithm.

For example, the algorithm can be configured to classify the inputs, recognize objects in the inputs, or generate a machine translation of the inputs.

According to one embodiment, a different architecture of the artificial neural network is used for each row, the architectures of different rows differing from one another at least in a sequence of the filter sizes.

The filter size can be defined, for example, by a number of weights per column and / or per row of the filter. In addition, the filter size can be defined in the depth direction of the filter.

For example, an architecture may only include filter layers with filters of a single filter size. Architectures with groups of filter layers are also possible, the filter layers of different groups differing in their filter size. Furthermore, architectures are possible in which filter layers of two filter sizes alternate. Another possibility are architectures in which the filter sizes are arranged in descending order, starting with the largest filter size. In addition, any other sequences of the filter sizes are conceivable.

The architectures can be stored in architecture definition data which define a number, type and linkage of the filter layers or other layers of the artificial neural network. For example, the architecture definition data for each of the filter layers can define parameters such as number of input channels, number of output channels, width and height of the filters, input source, step size, padding, dilation and pooling. Padding can be understood to mean surrounding an input matrix with a single-row or multiple-row frame made of zeros. Under dilation, the enlargement of a Filters can be understood by inserting spaces. Pooling can be understood to mean an optional downsampling of outputs from a filter layer by means of a subsequent layer, also called a pooling layer.

It is conceivable, for example, that the test system adapts the architecture definition data in accordance with the test queries and / or loads corresponding stored architecture definition data from a memory. In addition, the test system can read in inputs corresponding to the test queries, for example from a memory on which data sets of predefined inputs are stored, or from a camera that records images and provides them as inputs. For example, the test system can be designed to preprocess the inputs, for example to scale, crop or normalize them.

According to one embodiment, different settings are determined for at least one further parameter of the artificial neural network, a different setting being used for each test within a series.

Another parameter can be, for example, an influencing variable with which the performance of the test system, for example its latency or its data throughput, can be specifically influenced. Examples of further parameters are a resolution, i. H. an image size of the inputs, a number of output channels per filter layer and a stack size, d. H. a number of inputs that are processed in a batch, i.e. in parallel, by the test system. A setting can be understood as a predefined value for one or more parameters. The different settings can have a defined relationship to one another. For example, the settings can differ from one another by a defined percentage, for example by 50, 100 or 200 percent. In principle, the settings should differ from one another sufficiently.

Different series can contain the same number of tests, for example. The tests of different series can agree with one another, for example, insofar as the same settings of the further parameter or parameters are tested in each case in corresponding tests.

According to one embodiment, inputs are entered into the artificial neural network in each of the tests and a test result is determined which indicates a performance of the test system when the inputs are processed by the artificial neural network.

For example, the test result can indicate a latency in processing a single input; H. a processing time required by the artificial neural network to convert an input into an output with a certain accuracy. Alternatively or additionally, the test result can indicate a throughput, i. H. indicate how many entries per unit of time were processed by the test system. The test result can, however, also include other indicators that are suitable for describing a performance of the test system.

According to one embodiment, the test results are sent from the test system to a test evaluation module and are evaluated by the test evaluation module.

For example, the test evaluation module can use the test results to determine with which architectures or architecture variants and with which settings a particularly high performance of the test system was achieved. This knowledge can then be used, for example, to implement a model of an artificial neural network specially tailored to the test system.

The method makes it possible to test a large number of standard architectures of an artificial neural network on a given test system in comprehensible steps. By varying the artificial neural network in small but sufficiently meaningful steps, precise conclusions can be drawn about the strengths and weaknesses of the test system.

For example, the method can be used to test test systems with different hardware configurations with regard to their performance and then to compare them with one another. It is thus possible to select from a set of given hardware configurations those which are particularly well suited for a given model of an artificial neural network.

According to one embodiment, at least three different settings are determined for the at least one further parameter.

According to one embodiment, different settings are determined for at least two further parameters of the artificial neural network. At least two different settings are determined for each of the at least two further parameters. Furthermore, possible setting combinations between the respective settings of the at least two further parameters are determined. A different combination of settings is then used for each test within a series.

According to one embodiment, only one of the at least two further parameters is changed in its setting between tests that follow one another within a series. In this way, the effects of several parameters on the performance of the test system can be examined separately from one another.

Alternatively, the setting of two or more than two further parameters can be changed between the tests. This allows interactions between these parameters to be investigated.

According to one embodiment, the different settings are determined for at least one of the following parameters: a number of inputs that are input into the artificial neural network within an update step; a respective matrix size of the inputs; a respective number of output channels of the filter layers. At least one filter of the filter layer can be assigned to each of the output channels of a filter layer. A matrix size can be understood to mean, for example, a number of elements per row and / or per column of an input matrix. In addition, the matrix size can define a depth of the input matrix, such as a number of color channels of the input. The matrix size can also be referred to as (image) resolution or image size, for example. The number of output channels can be the same as the number of different filters in a filter layer.

According to one embodiment, the artificial neural network is a convolutional neural network and each of the filter layers is a convolutional layer. A convolutional layer can be a filter layer with one or more weight matrices, as described above. The artificial neural network comprises an activation layer and / or a batch normalization. Activation function is, for example, the rectified linear unit (ReLU).

According to one embodiment, at least ten series of tests based on the test requests are carried out by the test system.

According to one embodiment, a ranking of the architectures is determined based on the test results. A recommendation is then made based on the ranking. The recommendation can, for example, contain information about one or more architectures with which a comparatively high performance of the test system was achieved.

A second aspect of the invention relates to a test generation and evaluation system. The test generation and evaluation system comprises a test generation module which is designed to generate test queries and to send them to a test system according to an embodiment of a third aspect of the invention. Furthermore, the test generation and evaluation system comprises a test evaluation module which is designed to receive and evaluate test results from the test system. The test requests include commands which, when the commands are executed, cause the test system to carry out tests in a method according to an embodiment of the first aspect of the invention and to send corresponding test results to the test evaluation module.

Features of the method described above and below can also be features of the test generation and evaluation system and vice versa.

A third aspect of the invention relates to a test system which is designed to receive test requests from a test generation and evaluation system according to an embodiment of the second aspect of the invention, to carry out tests in a method according to an embodiment of the first aspect of the invention and to display corresponding test results to send the test generation and evaluation system.

Features of the method described above and below can also be features of the test system and vice versa.

A fourth aspect of the invention relates to a computer system which comprises a test generation and evaluation system according to an embodiment of the second aspect of the invention and a test system according to an embodiment of the third aspect of the invention.

Features of the method described above and below can also be features of the computer system and vice versa.

Further aspects of the invention relate to a computer program which, when it is executed on a processor, executes the method described above and below, as well as a computer-readable medium on which such a computer program is stored. The computer readable medium can be volatile or non-volatile data storage. For example, the computer-readable medium can be a hard disk, a USB storage device, a RAM, ROM, EPROM or flash memory. The computer-readable medium can also be a data communication network that enables a program code to be downloaded, such as the Internet or a data cloud.

Features of the method described above and below can also be features of the computer program and / or the computer-readable medium.

In the following, exemplary embodiments of the invention are described in detail with reference to the accompanying figures.

The reference symbols used in the figures and their meaning are summarized in the list of reference symbols. In principle, identical or similar parts are provided with the same reference symbols.

1 shows schematically a computer system 100 according to an embodiment of the invention.

The computer system 100 comprises a test generation and evaluation system 102 and a test system 104 . The test system 104 includes a processor 106 , the one on an artificial neural network 108 based algorithm. The network 108 consists of several successive filter layers 110 constructed that are configured to take an input 112 in one issue 114 to convert. This includes each of the filter layers 110 multiple filters 116 a defined filter size, for example a defined width, height and / or depth. The filters 116 move step by step in the x and y directions over the input 112 and key the input 112 over their entire depth. It is also possible that only certain depth layers of the input 112 through the filters 116 be scanned. With the exception of the foremost filter layer 110 preserve the filter sheets 110 in each case the output of a preceding filter layer 110 as input. The rearmost filter layer (not shown) can then, for example, the output 114 output. The edition 114 can also be from another layer of the network 108 output, for example from a fully connected layer, the elements of which are connected to all elements of a previous layer.

The filter sheets 110 can differ from one another in their filter size and / or their number of filters, ie in the number of their output channels. The filter sheets 110 are arranged in such a way that there is a defined sequence of filter sizes. By varying the sequence of the filter sizes, different architectures of the network 108 implemented.

The network 108 can for example be a convolutional neural network, with each of the filter layers 110 can be a convolutional layer of the convolutional neural network.

The test generation and evaluation system 102 comprises a test generation module 118 and a test evaluation module 120 that can be implemented as hardware and / or software modules. The test generation module 118 generates test requests 122 and sends it to the test system 102 . Any of the test requests 122 includes instructions that the processor 106 cause the algorithm to be run in several test runs with different configurations. The test system reads 104 according to the test requests 122 Inputs 112 from a memory 124 and puts it into the network 108 a.

On the attic 124 can have multiple records 126 predefined inputs 112 be saved. The inputs 112 different data sets 126 can differ from one another, for example, in terms of their resolution.

Alternatively or additionally, the entries 112 can be read in by a camera. Before entering the network 108 can be entered 122 for example through the test system 104 be preprocessed.

When the processor executes the instructions 106 become two or more rows 128 of tests 130 on the test system 104 carried out (the ranks 128 are here as lists, the tests 130 represented as lines of the lists). In each of the rows 128 will be a different architecture 132 of the network 108 tested. The architectures 132 different ranks 128 differ at least in the sequence of their filter sizes.

The architectures 132 will be in each of the tests 130 tested with a different combination of settings. For example, for each of three additional parameters 134a , 134b , 134c of the network 108 two or more settings as well as possible setting combinations between the respective settings can be determined. In each of the tests 130 one of the possible setting combinations can then be tested. The tests 130 different ranks 128 can agree, for example, that the same setting combinations are tested in each case.

The other parameters 134a , 134b , 134c can, for example, a respective number of output channels of the filter layers 110 , a resolution of the inputs 112 or be a stack size. The latter indicates how many inputs simultaneously, more precisely within an update step, into the network 108 can be entered.

For example, it is possible that between successive tests 130 a number 128 only one of the three other parameters 134a , 134b , 134c will be changed. This enables the influences of individual settings to be better understood. However, two or more than two further parameters can also be used 134a , 134b , 134c can be changed at the same time in order to investigate interactions between the relevant parameters.

In each of the tests 130 corresponding entries are made 112 into the network 108 entered. There is also a test result 136 that generates a performance of the test system 104 , such as latency or throughput, when processing the inputs 112 through the network 108 indicates. The test results 136 are either collected or sent individually to the test evaluation module 120 that sent the test results 136 then evaluates.

For example, the test evaluation module 120 based on the test results 136 a hierarchy of architectures 132 determine and based on the determined ranking a recommendation 138 that shows which of the architectures 132 with which settings for the relevant test system 104 is particularly suitable.

It is possible that the network 108 contains only particularly computationally intensive layers that are used for feature extraction, such as convolutional layers. Different types of layers, for example layers for region-of-interest pooling or fully-connected layers, can be used in the tests 130 be excluded.

For example, the ResNet-34 architecture without skip connections and without a first layer with a 7 × 7 filter can be used as a test basis. In other words, such a test base can contain, for example, 34 filter layers, the number of output channels of the filter layers and the sequence of the filter sizes, i. H. the downsizing of the activation layers, based on ResNet-34.

Examples of different architectures 132 of the network 108 , here in the form of a convolutional neural network, are summarized in Table 1 below. Each line in the "Filter size" column represents a network with several, in the case of ResNet-34 for example 34 filter layers 110 .

Successive filter layers 110 different filter sizes are separated with semicolons. Filter sizes with an "and" link mean that the relevant filters are active at the same time and their outputs are merged.

Examples of possible settings for the other parameters 134a , 134b , 134c are summarized in Table 2 below. Table 1: Convolution type description Filter size Standard convolution default 1 × 1 3 × 3 5 × 5 7 × 7 Bottleneck 1 × 1; 3 × 3; 1 × 1 1 × 1; 5 × 5; 1 × 1 1 × 1; 7 × 7; 1 × 1 Spatially Separable Convolution default 1 × 3; 3 × 1 1 × 5; 5 × 1 1 × 7; 7 × 1 Bottleneck 1 × 1; 1 × 3 and 3 × 1; 1 × 1 1 × 1; 1 × 5 and 5 × 1; 1 × 1 1 × 1; 1 × 7 and 7 × 1; 1 × 1 Depthwise Separable Convolution default 3 × 3 5 × 5 7 × 7 Bottleneck 1 × 1; 3 × 3; 1 × 1 1 × 1; 5 × 5; 1 × 1 1 × 1; 7 × 7; 1 × 1 Dilated Convolution ("Dilated Convolution") default 5 × 5 with I = 2 (= 1 space between elements of the filter) Table 2: parameter setting Batch size 1 4th 8th Resolution of the entries 224 × 224 1280 × 720 (HD) 1920 × 1080 (FHD) Number n of output channels n = x × 2 n = x n = x × 0.5

With 17 different architectures and 27 possible setting combinations, there are a total of 459 variations to be tested.

In addition, it should be noted that “comprehensive” does not exclude any other elements or steps and “a” or “a” does not exclude a plurality. It is further pointed out that features or steps that have been described with reference to one of the above exemplary embodiments can also be used in combination with other features or steps of other of the above exemplary embodiments. Reference signs in the claims are not to be regarded as a restriction.

List of reference symbols

100

Computer system

102

Test generation and evaluation system

104

Test system

106

processor

108

artificial neural network

110

Filter layer

112

input

114

output

116

filter

118

Test generation module

120

Test evaluation module

122

Test request

124

Storage

126

record

128

line

130

test

132

architecture

134a

first further parameter

134b

second further parameter

134c

third further parameter

136

Test result

138

recommendation

Claims (13)

A method for testing different architectures (132) of an artificial neural network (108), the artificial neural network (108) comprising a defined sequence of filter layers (110), each of the filter layers (110) having a defined number of trainable filters (116) a defined filter size for converting an input (112) to an output (114), the method comprising: Generating test requests (122) by a test generation module (118) and sending the test requests (122) from the test generation module (118) to a test system (104) for executing an algorithm based on the artificial neural network (108); Carrying out at least two series (128) of tests (130) based on the test queries (122) by the test system (104), wherein a different architecture (132) of the artificial neural network (108) is used for each row (128), the architectures (132) of different rows (128) differing from one another at least in a sequence of the filter sizes, different settings for at least one further parameter (134a, 134b, 134c) of the artificial neural network (108) being determined, a different setting being used for each test (130) within a row (128), wherein in each of the tests (130) inputs (112) are entered into the artificial neural network (108) and a test result (136) is determined which indicates a performance of the test system (104) when processing the inputs (112) by the artificial neural network Displays network (108); Sending the test results (136) from the test system (104) to a test evaluation module (120); and Evaluation of the test results (136) by the test evaluation module (120). Procedure according to Claim 1 , at least three different settings for the at least one further parameter (134a, 134b, 134c) being determined. Method according to one of the preceding claims, wherein different settings are determined for at least two further parameters (134a, 134b, 134c) of the artificial neural network (108), with at least two different settings for each of the at least two further parameters (134a, 134b, 134c) Settings are determined; possible setting combinations between the respective settings of the at least two further parameters (134a, 134b, 134c) being determined; a different combination of settings being used for each test (130) within a row (128). Procedure according to Claim 3 wherein between tests (130) which follow one another within a series (128), only one of the at least two further parameters (134a, 134b, 134c) is changed in its setting. Method according to one of the preceding claims, wherein the different settings are determined for at least one of the following parameters (134a, 134b, 134c): a number of inputs (112) which are input into the artificial neural network (108) within an update step; a respective matrix size of the inputs (112); a respective number of output channels of the filter layers (110). Method according to one of the preceding claims, wherein the artificial neural network (108) is a convolutional neural network and each of the filter layers (110) is a convolutional layer. Method according to one of the preceding claims, wherein at least ten series (128) of tests (130) based on the test queries (122) are carried out by the test system (104). Method according to one of the preceding claims, wherein a ranking of the architectures (132) is determined based on the test results (136); wherein a recommendation (138) is issued based on the ranking. A test generation and evaluation system (102) comprising: a test generation module (118) which is designed to generate test requests (122) and to send them to a test system (104) for executing an algorithm based on an artificial neural network (108) wherein the artificial neural network (108) comprises a defined sequence of filter layers (110) and each of the filter layers (110) comprises a defined number of trainable filters (116) of a defined filter size for converting an input (112) into an output (114) ; and a test evaluation module (120) which is designed to receive and evaluate test results (136) from the test system (104); wherein the test requests (122) comprise commands which, when executing the commands, cause the test system (104) to perform at least two series (128) of tests (130) and to send test results (136) to the test evaluation module (120), wherein for each row (128) a different architecture (132) of the artificial neural network (108) is used, the architectures (132) of different rows (128) differing from one another at least in a sequence of the filter sizes, wherein different settings are determined for at least one further parameter (134a, 134b, 134c) of the artificial neural network (108), a different setting being used for each test (130) within a row (128), wherein in each of the tests ( 130) inputs (112) are entered into the artificial neural network (108) and a test result (136) is determined which indicates a performance of the test system (104) when processing the inputs (112) by the artificial neural network (108). Test system (104) for executing an algorithm based on an artificial neural network (108), the artificial neural network (108) comprising a defined sequence of filter layers (110) and each of the filter layers (110) having a defined number of trainable filters (116) a defined filter size for converting an input (112) into an output (114), wherein the test system (104) is configured to receive test requests (122) from a test generation module (118), to perform at least two series (128) of tests (130) based on the test requests (122), and To send test results (136) to a test evaluation module (120), wherein a different architecture (132) of the artificial neural network (108) is used for each row (128), the architectures (132) of different rows (128) differing from one another at least in a sequence of the filter sizes, different settings for at least one further parameter (134a, 134b, 134c) of the artificial neural network (108) being determined, a different setting being used for each test (130) within a row (128), wherein in each of the tests (130) inputs (112) are entered into the artificial neural network (108) and a test result (136) is determined which indicates a performance of the test system (104) when processing the inputs (112) by the artificial neural network Network (108). A computer system (100) comprising: a test generation and evaluation system (102) according to Claim 9 ; and a test system (104) according to Claim 10 . Computer program, comprising instructions which, when the computer program is executed by a computer system (100) Claim 11 cause the computer system (100) to carry out the method according to one of the Claims 1 until 8th to execute. Computer-readable medium on which a computer program is based Claim 12 is stored.

Images