WO2025181663A1

WO2025181663A1 - Method for optimizing the computational resources of a deep neural network

Info

Publication number: WO2025181663A1
Application number: PCT/IB2025/051998
Authority: WO
Inventors: Fabio CERMELLI; Giuseppe Bruno AVERTA; Antonio TAVERA; Carlo MASONE; Barbara CAPUTO
Original assignee: Politecnico di Torino
Current assignee: Politecnico di Torino
Priority date: 2024-02-28
Filing date: 2025-02-25
Publication date: 2025-09-04
Anticipated expiration: 2026-08-28
Also published as: IT202400004381A1

Abstract

Described herein is a computer-implemented method (100) for selecting an architecture of a deep neural network by means of an optimization algorithm comprising a backbone-type macro-component, an encoder-type macro-component, and a decoder-type macro-component, said deep neural network being adapted to execute a computer vision task on a target dataset while fulfilling at least one processing requirement representative of a constraint of available resources on which said selected deep neural network executes said computer vision task, the method comprising the steps of: - receiving (110) said at least one processing requirement representative of a constraint of an available resource of the system; - receiving (120) efficiency data representative of macro-components adapted to define architectures of deep neural networks capable of executing a computer vision task, each one of said macro-components being of the backbone type, or of the encoder type, or of the decoder type; - selecting (130), within a search space comprising a plurality of backbone-type, encoder-type or decoder-type macro-components, a deep neural network architecture suitable for executing said computer vision task which has the highest validation score among all the examined candidate deep neural networks, said validation score being computed on the basis of a metric considering both said at least one processing requirement based on said efficiency data and an accuracy value for each one of said examined candidate deep neural networks.

Description

METHOD FOR OPTIMIZING THE COMPUTATIONAL RESOURCES OF A

DEEP NEURAL NETWORK

DESCRIPTION

[FIELD OF THE INVENTION]

The present invention relates to a method for optimizing the computational resources of a deep neural network. More specifically, the present invention relates to a method for optimizing the computational resources necessary when using a deep neural network for the execution of a computer vision task in the presence of constraints in terms of computational resources.

[PRIOR ART]

Deep learning is a branch of machine learning developed to emulate the learning capabilities of the human brain through the use of multiple abstraction levels, so that information can be extracted whose complexity increases with the number of abstraction levels, resulting in the possibility to analyze and interpret data in the form of images, audio, video or time series, and to solve very complex problems including automated driving, fault monitoring, flow control, etc.

One of the main problems encountered when using deep neural networks comes from their extensive resource requirements in terms of hardware availability and energy consumption.

This problem arises from the fact that the solutions currently known in the art are based on neural networks using oversized architectures, i.e. architectures needing a very large number of operations and parameters to perform at best, to the detriment of energy consumption and hardware requirements.

It is clear that such an approach results in non-negligible costs when solutions based on deep neural networks are adopted.

Generating efficient neural networks is a challenge that is arousing increasing interest, and several solutions are known in the art which are aimed at mitigating the abovedescribed problem. The most important solutions currently known in the art comprise model compression techniques, model compilation techniques, and techniques for designing architectures of efficient neural networks.

Model compression and compilation techniques are focused on the optimization of pretrained models. In particular, the compression technique uses pruning, distillation and quantization techniques to reduce the number of parameters of the network without changing its structure.

Notwithstanding a considerable efficiency gain, however, the loss of accuracy compared to the original model prevents the use of this technique for real applications. On the other hand, the compilation technique is not aimed at changing the performance of the neural network, but tries to provide the best implementation of its operations for a given hardware, so as to speed up the execution of such operations.

However, although accuracy remains unchanged from the original model, this technique can only provide marginal improvements.

Moreover, both techniques provide sub-optimal results when the neural network comprises mathematical operations that are not supported by the reference hardware. This is due to the fact that neither of these techniques can modify the type of the operations included in an architecture, since they can only remove some parameters or find a better implementation thereof.

The goal of the third technique of designing efficient neural network architectures, also known as Neural Architecture Search (NAS), is to simplify and automate the process of developing deep learning models, as well as to generate, in a fast and efficient manner, customized deep neural networks capable of meeting specific production requirements.

NAS is composed of three main constituent elements, identified as: the search space, i.e. the set of all possible neural architectures or, in other words, the set of all operations that can be used for designing a deep neural network, which can be evaluated; the evaluation strategy, aimed at evaluating the network before it is trained. the search strategy/algorithm, i.e. metric optimization based on the approach adopted to explore the search space in order to identify the best neural network for a given purpose.

Many implementations of NAS techniques for automatically designing efficient neural networks are known in the art, but these have a number of limitations, such as: the solutions currently known in the art are limited to image classification tasks that cannot be easily extended to more resource-intensive tasks like image segmentation or object and keypoint localization in images; such solutions are costly in terms of search time, because the architecture is broken up into numerous possible choices that cause the search space to become unmanageable. An idea of what “unmanageable” means in relation to a search space is given by a simple example in which a search space is composed of a set of one hundred possible operations, each one of which can be implemented in two different ways. This results in a search space formed of 2¹⁰⁰ possible neural architectures. Such a large size implies a long search time that may vary from weeks to months when applied for searching an architecture for the execution of a complex task like image segmentation; the main goal of such solutions is to maximize the accuracy of a generated neural network, without however providing any solution to the problem of optimizing the resources for a specific hardware device.

[OBJECTS AND SUMMARY OF THE INVENTION]

It is one object of the present invention to provide a method for optimizing the computational resources of a deep neural network which makes it possible to select, for the execution of a computer vision task, a deep neural network that ensures a balance between conflicting requirements, such as accuracy and computational cost in terms of hardware and energy resources.

It is another object of the present invention to provide a method for optimizing the computational resources of a deep neural network which reduces the time necessary to identify a deep neural network architecture capable of executing computer vision tasks with the maximum degree of accuracy that can be reached by a system with limitations in terms of available resources.

It is a further object of the present invention to provide a method for optimizing the computational resources of a deep neural network which can provide a customized deep neural network architecture capable of executing computer vision tasks while meeting constraints in terms of available resources of the system whereon it is executed.

More particularly, the method of the present invention can overcome the abovedescribed problems of the prior art by permitting, through an optimization algorithm, the selection of a deep neural network architecture comprising a backbone-type macrocomponent, an encoder-type macro-component, and a decoder-type macro-component, and adapted to execute a computer vision task on a target dataset while fulfilling at least one processing requirement representative of a constraint of available resources on which said selected deep neural network executes said computer vision task, said method comprising the steps of: receiving said at least one processing requirement representative of a constraint of an available resource of the system; receiving efficiency data representative of macro-components adapted to define architectures of deep neural networks capable of executing a computer vision task, each one of said macro-components being of the backbone type, or of the encoder type, or of the decoder type; selecting, within a search space comprising a plurality of backbone-type, encoder-type or decoder-type macro-components, a deep neural network architecture suitable for executing said computer vision task which has the highest validation score among all the examined candidate deep neural networks, said validation score being computed on the basis of a metric considering both said at least one processing requirement, based on said efficiency data, and an accuracy value for each one of said candidate deep neural networks.

The deep neural network architecture resulting from the selection made in accordance with the steps of said method is particularly suited for executing a given computer vision task in the presence of specific constraints in terms of computational resources, due to the fact that the criteria for the selection of said deep neural network also comprise criteria considering any constraints that may apply, thus providing a balance between two conflicting requirements. Indeed, while the need for executing a computer vision task with a high degree of accuracy implies an increased availability of computational resources, the necessity of executing the computer vision task within a context of limited resources results in the accuracy of the computer vision task being necessarily reduced.

According to a first embodiment of the present invention, the degree of accuracy of the computer vision task is evaluated by means of metrics that require the candidate neural network to be trained on the reference dataset, wherein said metrics are dependent on the computer vision task and are selected among: mean Intersection over Union for semantic segmentation tasks, mean Average Precision for instance segmentation and object localization, Panoptic Quality for panoptic segmentation.

According to a second embodiment of the present invention, the degree of accuracy for the computer vision task is computed by means of metrics not requiring the training of the candidate neural network and based on statistics such as, for example, logarithmic synaptic flow, linear region computation, and entropic score. According to one embodiment of the present invention, the accuracy value of the examined candidate deep neural networks is computed after a training step using a reference dataset.

According to one embodiment of the present invention, the optimization algorithm is a genetic algorithm or an evolutionary algorithm.

According to one embodiment of the present invention, the at least one processing requirement that the selected deep neural network must fulfil in the system while executing the computer vision task is one of: network latency; memory usage; energy consumption; number of images that can be processed per second; number of floatingpoint operations.

According to one embodiment of the present invention, the search space comprises a plurality of predefined architecture modules divided into a plurality of macro-categories from combinations of which a deep neural network can be obtained which is capable of executing, on static images and/or videos, the following tasks: image classification; semantic segmentation; instance segmentation; panoptic segmentation; object localization; object pose identification; human pose identification; keypoint localization; distance estimation; surface estimation; object tracking; video segmentation.

Further advantageous features of the present invention are set out in the appended claims, which are an integral part of the present description.

[BRIEF DESCRIPTION OF THE DRAWINGS]

These features as well as further advantages of the present invention will become more apparent in the light of the following description of a preferred embodiment thereof as shown in the annexed drawings, which are provided herein merely by way of nonlimiting example, wherein:

Fig. 1 shows a block diagram of macro-phases of the method according to the present invention for selecting a deep neural network architecture suitable for executing a computer vision task;

Fig. 2 shows a block diagram of the method according to the present invention for selecting a deep neural network architecture suitable for executing a computer vision task;

Fig. 3 shows, by way of example, one possible implementation of a step of selecting the deep neural network architecture, wherein the optimization algorithm is a genetic algorithm.

The references below are used only for simplicity’s sake, and shall not limit the protection scope or extension of the various embodiments.

[DETAILED DESCRIPTION OF THE INVENTION]

Recent advancements in the fields of artificial intelligence and machine learning have led to the development of evolved systems capable of solving increasingly complex problems by learning how to make autonomous decisions. Deep learning is, for example, a technique that permits extracting increasingly abstract information from a dataset in order to solve a given problem directly from such data. To do so, models are trained whose task is to produce, as a result, an information type which is consistent with the input data.

One example of application of the deep learning technique is image comprehension, e.g. classification, recognition and segmentation of objects, etc. in images, videos or other visual inputs.

Neural networks have made it possible to develop artificial systems that, being inputted large numbers of images, can extract therefrom significant information to learn how to recognize and discern the various objects contained in the image, their mutual distance, whether they are moving or not, and many other things as well, so as to be able to solve complex problems within contexts such as automated driving, fault monitoring, flow control, etc.

As previously described herein, today one of the most important needs is to be able to create an automation process which can generate architecture models capable of solving complex problems and ensuring a high degree of accuracy while also taking into account any constraints related to the computational resources available in a given system.

According to the present invention, this goal is achieved by a method of selecting an architecture of a deep neural network that, given a vision problem to be solved with machine learning methods, at least one constraint related to hardware resources, and a dataset representative of the target dataset whereon the computer vision task will be executed, makes it possible to identify that neural network architecture which represents the best compromise between the available computational resources (hardware and energy resources) and the accuracy of the selected deep neural network. Therefore, given a computer vision task of interest to be solved, one or more hardware and/or energy consumption limitations, and a validation dataset, the architecture identified at the end of the method according to the present invention will be the most efficient architecture among all possible architectures that are present within a NAS (Neural Architecture Search) search space. The term “search space” refers to the set of all possible solutions, including the desired one, that defines the neural network architectures that can be used in order to solve a given problem, which in the case of the present invention is a computer vision task.

The most efficient architecture is meant to be that architecture which can ensure the best accuracy in the execution of a computer vision task on the target data distribution, the performance of which is measurable in terms of the latency in obtaining the response for an input datum, the number of floating-point operations, and the number of parameters of the model to be executed, while meeting at least one constraint of available resources for at least one reference hardware component included in the system in which said architecture will have to execute said task. Further examples of measurable performance parameters are the number of inferences that can be executed per second on the hardware of the reference system, or the difference in energy consumption or cost in comparison with alternative hardware capable of offering the same performance.

Fig. 1 shows a block diagram of the macro-phases of the method according to the present invention that allows selecting an architecture of a deep neural network, hereafter also referred to as model, DNN OUT, suitable for executing a computer vision task with the best possible accuracy given the input constraints. Some examples of computer vision tasks that can be requested from a selected deep neural network architecture DNN OUT are: image classification, semantic segmentation, instance segmentation, panoptic segmentation, object localization, keypoint localization, object pose identification, human pose identification, distance estimation, surface estimation, object tracking, video segmentation.

An image classification task generally consists of identifying the class to which an object belongs.

Image segmentation in Computer Vision systems involves partitioning an image into different regions representing the various objects, and is based on the extraction, from the image domain, of one or more interconnected regions, i.e. pixel sets. A semantic segmentation task generally involves identifying similar objects belonging to the same class on a pixel level. An instance segmentation task generally involves identifying different instances of an object within an image. A panoptic segmentation task provides a more complete analysis of an image, because it classifies different classes for different instances, combining semantic segmentation and instance segmentation.

A keypoint localization task allows identifying and locating specific points of interest within an image.

An object localization task locates one or more instances of one or more object categories within an image. Object pose identification and human pose identification are tasks that detect and track the position and orientation of an object or a person in association with the location of specific keypoints, e.g. parts of the human body, like the shoulders and knees of a person, or specific elements of an object, like the headlights of a car. A distance estimation task identifies the depth of the objects in an image, and a surface estimation task makes it possible to determine the orientation of the surfaces of objects in an image.

An object tracking task makes it possible to determine the position and trajectory of an object in a video. Lastly, video segmentation allows partitioning a video into a plurality of areas according to specific features like object contours, motion, colour, texture or other visual characteristics, so as to separate the different objects from one another and from the background.

A step of selecting the architecture of a deep neural network occurs, as aforementioned, in consideration of some constraints that the selected deep neural network architecture will have to meet during the execution of a computer vision task.

In particular, said constraints comprise: a plurality of images with respective annotations representative of a target dataset, INPUT 1, whereon said selected deep neural network will have to execute the computer vision task; at least one processing requirement, INPUT 2; a type of computer vision task, INPUT 3, consistent with INPUT 1, i.e. consistent with a dataset that is coherent with a statistic distribution of the data contained in the reference system in which said neural network will have to execute the computer vision task.

The at least one processing requirement may comprise hardware constraints, such as features of a specific device included in the reference system, as well as energy consumption constraints.

In addition, hardware constraints may refer to different hardware components that may be taken into account during the selection process, e.g. for the purpose of assessing the best architecture configuration capable of ensuring the highest accuracy or the highest speed depending on the computer vision task to be executed.

The three blocks shown in Fig. 1 represent the main blocks that make it possible to select the deep neural network architecture in accordance with the present invention. In particular, a first neural design block, ND, provides the best neural network architecture, DNN SEL, capable of executing a given computer vision task. The best neural network architecture is meant to be that candidate neural network architecture, among all possible candidate neural network architectures selected within the search space comprising all possible solutions to different computer vision tasks, which can execute a given computer vision task, INPUT 3, on a target dataset with an accuracy value which exceeds those provided by any other examined candidate neural network architecture, and which can fulfil one or more hardware-related or performance-related processing requirements, INPUT 2.

Unlike the solutions currently known in the art, in which the search space is defined as a plurality of micro-operations and the searched architecture is broken up into numerous possible choices of mathematical operations, leading to long search times and difficult management of said search space, the search space according to the present invention, within which the deep neural network architecture is selected, comprises a plurality of backbone-type, encoder-type and decoder-type macro-components.

Said macro-components are predefined architectural modules, each one having a size and/or an implementation of mathematical operations which differ from those of the other architectural modules that are present within the search space for the same task type, with which an encoding is also associated which is given by a pair of values indicating the type and size thereof. Said encoding allows distinguishing among macrocomponents of the same type. For example, references B(A,1), B(B,2) indicate two different backbone-type macro-components, whereas references E(C,1),E(E,4) indicate two different encoder-type macro-components. This type of encoding is only one of the possible ways of precisely identifying each macro-component. In the example proposed herein, the letter outside the round brackets identifies the macro-component type (B for backbone, D for decoder, E for encoder), while the letter and the number inside the round brackets identify a specific macro-component, along with its size, belonging to the type designated by the letter outside the round brackets. For example A,1 identifies a ResNet with size 18, whereas A, 3 identifies a ResNet with size 50, both belonging to the backbone type of macro-components designated by letter B. The same goal can also be easily achieved by using encodings which are different from the one described in this example, which is proposed herein merely for explanatory purposes.

In particular: the backbone type is a predefined architectural module capable of extracting a plurality of characteristics from an input reference dataset, INPUT 1; the encoder type is a predefined architectural module capable of encoding a semantic content from the reference dataset according to a representation based on the characteristics extracted by the predefined backbone-type architectural module; the decoder type is a predefined architectural module capable of using a representation generated by a predefined encoder-type architectural module to execute the assigned computer vision task, INPUT 3.

The selected deep neural network resulting from the execution of the method according to the present invention comprises a backbone-type macro-component, an encoder-type macro-component, and a decoder-type macro-component.

The selected neural network architecture and the candidate neural network architectures are identified by a network vector comprising the three value pairs used for encoding the respective macro-components. For example, a first candidate neural network architecture is identified by network vector B(A,1)D(E,2)E(E,4), and a second candidate neural network architecture is identified by network vector B(B,2)D(B,3)E(C,1), in accordance with the encoding logics described above.

The network vector as described below will be used during the mutation applied by an optimization algorithm performing the task of selecting the deep neural network architecture according to the present invention.

The step of selecting the best deep neural network architecture is carried out on the basis of a validation score calculated according to a metric considering both the at least one constraint and an accuracy value of the candidate deep neural network. The accuracy score of the candidate deep neural network may be determined on the basis of statistic values according to metrics not requiring a step of training the candidate architecture. As an alternative, the accuracy score may be calculated more precisely after a step of training the candidate deep neural network on the reference dataset. The first option has the advantage that it shortens the time necessary to select a candidate deep neural network, whereas the second option permits computing a more precise accuracy score because it is carried out using a subset, INPUT 1, of the target data on which the neural network will have to execute the computer vision task, INPUT 3. Still with a view to reducing the time necessary to select a deep neural network, that part of the score which concerns the metric of the at least one constraint may, as will be described in more detail below, be computed on the basis of efficiency data, INPUT 4, already available upstream of the search step, for the purpose of statistically estimating the performance of a possible candidate architecture on a given hardware device without having to compute said metrics during the search step.

A second neural training block, NT, provides a network architecture, DNN_TRA, whose accuracy is maximized in relation to the target data distribution. This is done by identifying those network weights which maximize the accuracy of the selected model, DNN_SEL, using the reference dataset, INPUT 1, in the training step.

A third model assessment block, MA, provides the final deep neural network, DNN OUT, by executing a final validation task, using a validation dataset VAL DT, on the selected and trained deep neural network, DNN TRA, for the reference application scenario, the latter referring to the computer vision task to be executed on the target data distribution while meeting the at least one computational resource constraint and ensuring an adequate level of accuracy.

The level of accuracy achieved by the network DNN_OUT may however, as will be explained in more detail in the description of the method according to the present invention, be insufficient.

Therefore, as indicated by feedback arrows Rl, R2, an additional training step may be necessary to try to further improve the accuracy of the selected architecture DNN OUT, feedback case Rl, or, in the event that the achieved accuracy is not in any way satisfactory even after an additional training step, to re-execute from the start the selection of the neural network architecture, feedback case R2.

Fig. 2 shows a block diagram of the method 100 according to the present invention for selecting a deep neural network architecture suitable for executing a computer vision task.

In a first step 110, at least one processing requirement is received, said requirement being representative of a constraint of a resource available in a system in which the deep neural network executes the computer vision task. Said at least one processing requirement may be related to a particular hardware device included in the system. This aspect is important because a neural network architecture internally comprises a plurality of mathematical operations that, in general, can be implemented in a more or less optimized manner on reference hardware by adjusting specific parameters.

However, there is also the possibility that a given architecture, implemented according to certain mathematical operations, is not supported by a specific hardware device, and that parameter optimization alone is not enough.

The at least one processing requirement may also refer to mutually alternative hardware devices or to the system as a whole, defined according to terms of performance or available energy that said system can guarantee.

According to one embodiment of the present invention, the processing requirements may comprise, for example: compatibility with a given hardware device, number of operations to be executed in a given time interval, memory size of the architecture, latency of the system, available memory, and energy consumption.

In a second step 120, efficiency data are received which concern a plurality of macrocomponents of the backbone, encoder and decoder types, the various combinations of which can define deep neural network architectures capable of executing a computer vision task.

It is known in the art the need for exporting a neural network architecture to a given hardware device for the purpose of evaluating its performance. However, this is a timeconsuming process that strongly affects the speed of convergence of a process of selection of one neural network architecture among all possible candidate architectures obtainable from a search space. As a matter of fact, in order to evaluate the performance of each candidate architecture it is necessary to connect, via a physical or network connection, the hardware device of interest to the computer that is executing the selection process, so that the candidate neural network architecture can be exported to said hardware device, where specialized software will then assess its performance.

According to the method of the present invention, the assessment of the performance levels of all candidate architectures can be sped up by determining, upstream of the search step, efficiency data of all macro-components within the search space from which candidate architectures can be selected that are capable of executing a given computer vision task.

The preparatory work is even more advantageous when one wants to assess the performance of multiple hardware devices, in order to identify the one which is most suitable for a specific computer vision task.

Therefore, each macro-component is exported and tested on one or more hardware devices in order to determine a plurality of efficiency data, which are made available prior to conducting the search in accordance with the method of the present invention. Such data may, for example, be stored as follows:

B(A,3): 20ms - 3.3W - 45MB

B(B,1): 15ms - 2.12W - 16MB where B(A,3)/B(B,1) are identifiers of a specific macro-component among all macrocomponents of the same type.

As previously described herein, and merely by way of explanatory example, identifier B(A,3) refers to a particular backbone-type macro-component characterized by a specific size and a specific implementation of mathematical operations, different from those of other backbone-type macro-components, e.g. B(C,1), B(A,2), that are present within the search space.

The values after the identifier define the efficiency data that characterize the efficiency performance behaviour of a specific macro-component on a given hardware device. In particular, such values refer to response latency, energy consumption and memory usage, respectively.

The availability of efficiency data during the selection step makes the above-described export operations unnecessary.

The goal of a subsequent step 130 is to select, by means of an optimization algorithm, one deep neural network architecture among all possible candidate neural network architectures selectable within the search space, which, as aforesaid, comprises a plurality of backbone-type, encoder-type and decoder-type macro-components, and wherein the candidate deep neural network architecture is generated using a Neural Architecture Search, NAS, technique, the components of which have been described above.

According to one embodiment of the present invention, the optimization algorithm is any genetic or evolutionary algorithm capable of providing a solution to the optimization problem of minimizing a structural error function of a neural network architecture in relation to the parameters of said network.

According to one embodiment of the present invention, said first type of macrocomponents comprises architectures that, upon reception of an input image, returns as a result a set of spatial descriptors like those of the VIT, MiT, DEiT, ResNet, SWIN, MobileNet, EfficientNet, EdgeNext, Xception, DenseNet, NasNet, VGGNet type families.

According to one embodiment of the present invention, said second type of macrocomponents comprises any module capable of providing spatial semantic descriptors starting from the characteristics extracted from a backbone-type macro-component, such as Feature Pyramid Network (FPN), Atrous Spatial Pyramid Pooling (ASPP), and Transformer Encoders modules.

According to one embodiment of the present invention, said third type of macrocomponents comprises modules for the execution of a semantic image segmentation task, such as linear (pixel-level) classifiers or transformer-based modules, like MaskF ormer; or, for the execution of an instance segmentation task, anchor-based modules, like Mask R-CNN, or transformer-based modules, like MaskF ormer, DETR, k-Max Deeplab, YOSO, or hybrid modules, like Panoptic Deeplab and UPSNet; or, for the execution of an object localization task, anchor-based modules, like RCNN, Faster RCNN, or non-anchor-based modules, like YOLO, FCOS, CenterNet, or transformerbased modules, like DETR, DeformableDETR.

Said optimization algorithm is configured to provide a deep neural network suitable for executing the computer vision task which has obtained the highest validation score among all examined candidate network architectures that can execute the same computer vision task, INPUT 3, and meet the constraints due to the presence of the at least one processing requirement, INPUT 2, received in step 110.

The validation score is computed on the basis of a metric that comprises both an evaluation of the degree of accuracy attainable by said candidate deep neural network architecture for the requested computer vision task, INPUT 3, and an efficiency evaluation expressed in terms of compliance with the at least one processing requirement, INPUT 2.

The solutions known in the art are mainly focused on metrics that rely on NAS to determine the degree of accuracy of a selected candidate architecture.

However, even when the efficiency factor is taken into account, evaluation metrics are mostly based on two types of approach: a first approach uses a metric whose complexity is hardware-independent, like, for example, the number of floating-point operations executed by the neural network, also known as FLOP; a second approach involves exporting every possible candidate architecture to the reference hardware in order to compute its performance.

The criticality of the first approach lies in the use of metrics that inaccurately correlate the network’s actual performance on a specific hardware device, due to the fact that the various operations in the neural network can be implemented in a more or less efficient way, resulting in different characteristics even with equal values in the hardwareindependent metric. The criticalities of the second approach are due, in the first place, to the compilation step to be executed on specific reference hardware, which step takes a long time to complete and also requires the availability of specialized software, in addition to needing a constant connection to said reference hardware throughout the duration of the search step within the search space.

On the contrary, the metric used in the present invention makes it possible to consider both the accuracy of the candidate neural network architecture and its efficiency on the reference hardware. Said metric allows computing the validation score of each candidate neural network architecture with the formula: where ^s represents an accuracy metric for the specific computer vision task, INPUT 3, with values comprised between 0 and 1, where 1 represents the best results, e.g. mean

Intersection over Union (mloU) for a semantic segmentation task, represents the i-th processing requirement, INPUT 2, e.g. network latency, memory usage, energy consumption, with values comprised between 0 and 1, where value 0 indicates that a totally fulfilled requirement. Instead of being computed by connecting the hardware devices of interest to the computer that is executing the architecture selection, resulting in a longer time taken to reach the convergence that completes the selection step, these values can be easily retrieved from the efficiency data received from step 120.

5 represents the weight associated with the i-th processing requirement, INPUT 2, so that the importance of a given application requirement can be varied, and wherein the r > sum of the weights I i is less than or equal to 1. By way of example, if latency is a more important requirement than memory usage for a given system, it is possible to assign to the first weight a value which is higher than that of the second weight.

According to one embodiment of the present invention, the accuracy metric & for the specific computer vision task, INPUT 3, is computed using metrics of a proprietary combination of training-free techniques, based on a logarithmic synaptic flow metric (described in “FreeREA: Training-Free Evolution-based Architecture Search", Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, by Cavagnero, Niccold, et al.), a linear regions computation metric and an entropic score metric that do not require the neural network to be trained, being based on architecture statistics correlating the validation of the accuracy of the trained network, such as, for example, metrics based on the gradient flow within the various architectural macro-components, given an input value.

This correlation provides approximations of certain characteristics of the candidate architecture under examination, like easiness of training and expressivity, i.e. the ability to model complex relations in the training data. These characteristics provide a good approximation as to which networks can offer the best performance in terms of final accuracy.

The logarithmic synaptic flow metric is a metric computed on untrained architectures that provides a measure of the easiness of propagation of the weight gradient during the training. This evaluation provides an indirect assessment of the architecture’s capability of being trained in a short time. The linear regions computation metric is a metric of the number of independent input patterns that can be discriminated by the untrained network. This metric provides an assessment of the minimum difference between two similar inputs that result in distinguishable activations of the architecture, providing an indirect measure of the network’s expressivity power. The entropic score metric is computed as the elementwise average of the entropy of the normalized activations of the network in response to a random input supplied to the network. This score provides an indirect measure of the network’s expressivity, in that it is positively correlated with the amount of information that can flow through the network during the inference process.

According to one embodiment of the present invention, the step of selecting 130 is preceded by a step 125 of receiving a reference dataset comprising a plurality of images and associated annotations representative of a target dataset on which said selected deep neural network executes the computer vision task.

According to a second embodiment, wherein said reference dataset is available, in step 130 the accuracy metric & for the specific computer vision task is computed according to training-based metrics, i.e. techniques that require the candidate network architecture to be trained on customer-supplied data, INPUT 1, to be able to compute an accuracy metric for that specific computer vision task, INPUT 3.

According to a preferred embodiment of the present invention, the validation score is calculated through the use of training-free techniques that compute accuracy and efficiency data useful for computing the hardware efficiency score, resulting in a selection process that comes to convergence very quickly, being able to reach convergence in just a few minutes.

Should the neural network architecture selected through the use of training-free metrics for accuracy assessment not meet the final accuracy expectations, it will be possible to re-execute the selection by using training-based metrics in order to improve the accuracy value, thus being able to reach convergence in a few hours or days, depending on the specificity of the searched architecture.

The step of selecting 130 ends when the deep neural network architecture with the highest validation score is identified among all examined candidate deep neural networks suited to execute the computer vision task, INPUT 3, while fulfilling the at least one processing requirement, INPUT 2.

In addition, the step of selecting 130 provides a list of all examined candidate deep neural networks, comprising the selected network, each one having the validation score obtained during the examination.

This list proves particularly useful when the examined deep neural network architectures have obtained a validation score which is very close to the score obtained by the selected neural network architecture. In fact, since the validation score is obtained from the sum of the scores relating to accuracy and compliance with processing requirements, it may happen that the selected neural network architecture has obtained a better accuracy score than a second examined neural network architecture, but the latter has obtained a better score for processing requirements.

In such a case, a user may, this information being available, decide to use said second examined neural network architecture instead of the selected neural network architecture, if the user thinks that fulfilling processing requirements is more important. Therefore, the availability of this information allows for further evaluations, comprising testing other examined deep neural networks in addition to the one identified during the selection step because such networks have obtained higher accuracy or compliance scores, or because, for example, one wants to evaluate the examined neural network architectures on different hardware devices in order to find that network which guarantees the best performance. Thanks to said information, such further evaluations can be made without having to execute new search steps.

During a subsequent step 140, at least one training iteration is executed by the optimization algorithm for the selected deep neural network architecture by means of the reference dataset, INPUT 1, in order to increase the accuracy value of said neural network in relation to the target dataset on which the computer vision task, INPUT 3, will be executed.

Said training step 150 is executed through the use of known optimization parameters for neural network training, such as, for example, update algorithm, learning rate, regularization function, number of images per iteration, and augmentation applied to images.

The at least one iteration, or the number of iterations actually made, will depend on the size of the reference dataset used for the training. The bigger the dataset, the greater the number of iterations. For example, if a dataset comprising 200 images is used, said step will execute about 20 training iterations. If a bigger dataset is used, e.g. comprising 100,000 images, the number of training iterations will be about 10,000.

The last step 150 of the method 100 validates the selected deep neural network architecture by means of the target dataset in order to verify that the computer vision task, INPUT 3, is executed in compliance with the at least one processing requirement, INPUT 2.

Should the validation step 150 be unsuccessful, due to the fact that the selected deep neural network does not reach an expected score for executing the specific computer vision task, INPUT 3, while fulfilling the at least one processing requirement, INPUT 2, it will be possible to execute a new training step by using a variation of the parameters used in the training step 140 for the purpose of increasing the obtained validation score. If, even after a new training step 140, the obtained validation score is still unsuitable for the computer vision task to be executed and for the at least one processing requirement, then the method of the present invention can be re-executed while relaxing the constraints of the processing requirements imposed on the search algorithm, e.g. by modifying the energy consumption constraint or the latency constraint, so as to obtain candidate neural network architectures that are more accurate than those obtained from the previous execution of the method of the present invention.

Fig. 3 shows, by way of example, one possible implementation of the step of selecting 130 of the method 100, wherein the optimization algorithm is a genetic algorithm.

Since the computer vision task, the at least one processing requirement, the efficiency data and, optionally, the reference dataset are known from the previous steps 110,120,125 of the method 100, a first goal of said algorithm is to generate an initial population of candidate deep neural networks by means of the steps comprised in a step GEN POP INI.

In order to obtain said population of an arbitrary dimension P, the first step GEN CAND randomly generates a candidate neural network architecture comprising a backbone-type macro-component, a decoder-type macro-component, and an encodertype macro-component suitable for executing the computer vision task, INPUT 3, starting from the plurality of macro-components that are present within the search space.

Each candidate neural network architecture is identified by a respective network vector comprising the three value pairs used for encoding its macro-components. Some examples of network vectors are: B(A,2)D(B,3)E(A,7), B(C,5)D(A,1)E(B,5), B(A,1)D(H,7)E(I,1), etc.

In a subsequent step CRIT, it is verified whether the backbone-type, decoder-type and encoder-type macro-components that characterize the candidate neural network architecture fulfil the at least one processing requirement, INPUT 2, on the basis of the efficiency data, INPUT 4, which may comprise, for example, network latency, memory usage, and energy consumption.

If said requirements are not fulfilled, the candidate architecture is discarded and a new generation step GEN_CAND is executed.

If, on the contrary, the processing requirements are fulfilled, during the next step CALC MTR the validation score is determined by means of the metric expressed by the previously described formula: which takes into account both the accuracy of the candidate neural network architecture and its efficiency on the reference hardware. As previously described herein, the accuracy metric can be computed by means of training-free techniques or, subject to a training step, by means of training-based techniques using a reference dataset, INPUT 1, for the training. In this step, a score determined by said metric is associated with each candidate deep neural network as in the following example:

- B(A,2)D(B,3)E(A,7):45.6%

- B(C,5)D(A,1)E(B,5):12.8%

- B(A,1)D(H,7)E(I,1):44.8%

The candidate deep neural network is then added to the initial population of candidate neural networks in step ADD POP.

The operations of steps GEN CAND, CRIT, CALC METR, ADD POP are repeated until the number of candidate deep neural network architectures reaches the arbitrary dimension P.

During the subsequent steps ALG EVO, the genetic algorithm will execute a plurality of steps to converge towards the best deep neural network architecture for a specific computer vision task, taking into account the at least one processing requirement.

In a step SAMPL CAND, an arbitrary number S of candidate deep neural network architectures, belonging to the initial population, are selected randomly.

Among such S architectures, in the next step BEST SEL that candidate deep neural network architecture is chosen which has the highest validation score among all S selected architectures.

A subsequent step MUT COD applies a random mutation of one or more values within the network vector that identifies the candidate deep neural network chosen in step BEST SEL. This random mutation of one or more values makes it possible to explore all possible solutions within the search space, resulting in the selection of a new candidate deep neural network architecture that differs from the architecture identified in step BEST SEL in the type and/or size of one or more macro-components.

For example, if the network identified by network vector B(A,2)D(B,3)E(A,7):45.6% is the network with the highest validation score among the S selected architectures, a first possible mutation may be, for example, B(A,4)D(B,3)E(C,7), which identifies a deep neural network architecture to be searched for within the search space whose backbone-type macro-components has been mutated from B(A,2) to B(A,4) to have a bigger size, while the encoder-type macro-component has been mutated from E(A,7) to E(C,7) to have a different component type.

In step CRIT, much in the same way as in step CRIT, it is verified whether the backbone-type, decoder-type and encoder-type macro-components that characterize the mutated neural network architecture fulfil the at least one processing requirement, INPUT 2, on the basis of the efficiency data, INPUT 4, as in the previously described step CRIT.

If said requirements are not fulfilled, the mutated architecture is discarded, and a new step SAMP CAND is executed.

If, on the contrary, the processing requirements are fulfilled, during the next step CALC MTR2 the validation score is determined according to the same logics previously described for step CALC METR.

The new mutated neural network architecture is then added to the population in step ADD POP2.

The operations described for steps SAMPL CAND, EST SEL, MUT COD, CRIT2, CALC METR2 and ADD POP2 are executed an arbitrary number C of times.

In a subsequent step DEL POP, a number C of candidate neural network architectures are eliminated from the population, such networks being those that were generated earliest in time, according to a biological aging evolution mechanism.

The operations described in steps ALG EVO are executed an arbitrary number A of times.

According to one embodiment of the present invention, when the value of A exceeds 32 the solutions start converging always towards the same neural network architectures. This value depends on numerous factors, one of which is the size of the search space. When the genetic algorithm reaches convergence towards a solution, the selected candidate neural network architecture is, as shown in Fig. 1 and previously described herein, inputted to the neural training block, NT, to maximize the accuracy of said selected deep neural network in relation to the target data distribution.

Furthermore, at the end of the selection step 130 a list is made available which comprises the examined candidate deep neural networks, each one associated with the respective validation score obtained during the examination, so that it is possible, based on said list, to make further evaluations without having to execute a new search for a deep neural network. The present invention makes it possible to overcome the problems suffered by the prior art, which cannot be solved with known solutions. In particular, through a new definition of the search space, which comprises three predefined types of architectural modules, wherein, for each type, each predefined architectural module is defined by a size and an implementation of mathematical operations that differ from those of the other predefined architectural modules of the same type, thus being optimized for different hardware devices, and an indication about the size that the searched macrocomponents must have to be suited to define at least one candidate neural network architecture capable of executing a given computer vision task, it is possible to limit the search space and reduce the time necessary to identify the best neural network that offers the highest degree of accuracy in the execution of a given computer vision task while meeting the resource constraints of a system in which said neural network will have to execute said given task.

In particular, this search time reduction is due to the availability of efficiency data that allow verifying the compliance of a deep neural network with the at least one processing requirement, without any hardware devices having to be connected to the computer during the search step.

Times are further shortened when, in addition to the efficiency data, the accuracy of the candidate deep neural network is computed by means of training-free metrics.

The present invention is not limited to the above-described embodiments, since it may be subject to many modifications, improvements or replacements of equivalent parts and elements without departing from the inventive idea, as clearly specified in the following claims.

Claims

1 . Computer-implemented method (100) for selecting an architecture of a deep neural network by means of an optimization algorithm comprising a backbone-type macro-component, an encoder-type macro-component and a decoder-type macrocomponent, said deep neural network being adapted to execute a computer vision task on a target dataset while fulfilling at least one processing requirement representative of a constraint of available resources wherein said selected deep neural network executes said computer vision task, the method comprising the steps of: receiving (110) said at least one processing requirement representative of a constraint of an available resource of the system; receiving (120) efficiency data representative of macro-components adapted to define architectures of deep neural networks capable of executing a computer vision task, each one of said macro-components being of the backbone type, or of the encoder type, or of the decoder type; selecting (130), within a search space comprising a plurality of backbone-type, encoder-type or decoder-type macro-components, a deep neural network architecture suitable for executing said computer vision task which has the highest validation score among all the examined candidate deep neural networks, said validation score being computed on the basis of a metric considering both said at least one processing requirement based on said efficiency data and an accuracy value for each one of said examined candidate deep neural networks.

2 . Method according to claim 1, wherein the step of selecting (130) is preceded by the step of: receiving (125) a reference dataset comprising a plurality of images and associated annotations representative of the target dataset on which said selected deep neural network executes the computer vision task.

3 . Method according to claim 2, wherein the accuracy value in the step of selecting (130) is computed after a step of training the candidate deep neural network by using the reference dataset.

4 . Method according to at least one of the preceding claims, wherein the step of selecting (130) further generates a list comprising a plurality of examined candidate deep neural networks, each one having associated therewith the validation score obtained in the step of selecting (130).

5 . Method according to claim 2, wherein said method further comprises the steps of: executing (140) at least one training iteration for the selected deep neural network architecture by means of the reference dataset, in order to increase the accuracy value obtainable from said network in relation to the target dataset; validating (150) the selected deep neural network architecture by means of the target dataset, in order to verify that the computer vision task is executed in compliance with the at least one processing requirement.

6 . Method according to claim 1, wherein the metric used for computing the validation score in the step of selecting (130) is expressed by the formula where ® represents an accuracy metric for the computer vision task, 4 represents the i-th processing requirement with values comprised between 0 and 1, where the value 0 indicates a totally fulfilled requirement, ^? represents the weight associated with the i- th processing requirement, so that the importance of a given application requirement can be varied, and wherein the sum of the weights n is less than or equal to 1.

7 . Method according to claim 6, wherein the accuracy metric for the computer vision task is computed by means of metrics not requiring the training of the candidate neural network and based on statistics such as logarithmic synaptic flow, linear region computation, entropic score.

8 . Method according to claims 3 and 6, wherein the accuracy metric for the computer vision task is computed by means of metrics that are specific for trained neural networks, said metrics being dependent on the computer vision task and being one of: mean Intersection over Union; mean Average Precision; Panoptic Quality.

9 . Method according to claim 1, wherein the optimization algorithm comprises a genetic algorithm or an evolutionary algorithm.

10 . Method according to claim 1, wherein the backbone-type macro-component is one of the architectures belonging to the VIT, MiT, DEiT, ResNet, SWIN, MobileNet, EfficientNet, EdgeNext, Xception, DenseNet, NasNet, VGGNet type families.

11 . Method according to claim 1, wherein the encoder-type macro-component is one of: Atrous Spatial Pyramid Pooling (ASPP), Feature Pyramid Network (FPN), Transformer Encoders.

12 . Method according to claim 1, wherein the decoder-type macro-component is one of: MaskFormer, Mask R-CNN, DETR, k-Max, Deeplab, YOSO, Panoptic Deeplab and UPSNet, RCNN, Faster RCNN, YOLO, FCOS, CenterNet, DETR, DeformableDETR.

13 . Method according to claim 1, wherein the at least one processing requirement that the selected deep neural network must fulfil while executing the computer vision task is one of: network latency; memory usage; energy consumption; number of images that can be processed per second; number of floating-point operations; number of inferences that can be executed per second on a hardware device, belonging to the system, wherein the computer vision task is executed.

14 . Method according to the preceding claims, wherein the computer vision task is one of: image classification; semantic segmentation; instance segmentation; panoptic segmentation; object localization; keypoint localization; object pose identification; human pose identification; distance estimation; surface estimation; object tracking; video segmentation.