Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/098—Distributed learning, e.g. federated learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/004—Artificial life, i.e. computing arrangements simulating life
  • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks

Definitions

• the present disclosure generally pertains to circuitry for a plurality of edge devices and a method.
• an artificial intelligence such as a neural network
• a neural network For example, a well-trained neural network, a large amount of data may be needed.
• speech recognition may be trained with voice data from millions of users performing a voice-based web search, or the like.
• a data collection architecture or training architecture may be designed by hand (e.g., by a programmer), such that the architecture may be configured to collect large amounts of data depending on the objective of the training.
• NAS neural architecture search
• federated learning is known as a method to combine updates from several devices into one network training.
• the disclosure provides circuitry for a plurality of edge devices, each edge device including at least one sensor of at least one sensor type, the circuitry being configured to: determine a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtain sensor data from each edge device of the subset of edge devices for obtaining task training data; update, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.
• the disclosure provides a method for a plurality of edge devices, each edge device including at least one sensor of at least one sensor type, the method including: determining a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtaining sensor data from each edge device of the subset of edge devices for obtaining task training data; updating, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.
• Fig. 1 depicts a system including circuitry according to the present disclosure
• Fig. 2 depicts a method according to the present disclosure
• Fig. 3 depicts a further embodiment of a method according to the present disclosure in which a task algorithm is updated by changing a weight parameter
• Fig. 4 depicts a further embodiment of a method according to the present disclosure in which a task algorithm is updated by determining an updated edge device weight parameter
• Fig. 5 depicts a further embodiment of a method according to the present disclosure in which a task algorithm is updated by determining an updated backbone weight parameter.
• training methods are generally known.
• the training architecture may depend on a training objective. Accordingly, if the training objective is changed, the architecture may need to be changed, too.
• NAS neural architecture search
• the training data may need to be close or similar to the application data and still, NAS may lack the ability to dynamically change the architecture when the training objective is changed (dynamically) since a training designer (e.g., a programmer) may still need to be involved for searching the most suitable architecture.
• edge devices e.g., terminal devices, such as mobile phones, drones, cars, or the like
• processing powers and storage capabilities are more and more increasing, such that a part of the training and the adaption of the network may be carried out directly on a particular edge device (also referred to as on-device training).
• Each edge device may be configured to collect its own training data which may be sufficiently similar to application data and may carry out a training based on that training data. Hence, a privacy may be secured since the training data may not necessarily be uploaded to a server. Moreover, in case an edge device has free capabilities, training may be carried out incidentally.
• federated learning may not allow true swarm behavior since in federated learning, all devices may need to have an identical configuration (at least for that parts which participate in the training) and train on the same task. In other words, in federated learning the devices which acquire training data may all be copies of the same instance.
• circuitry for a plurality of edge devices each edge device including at least one sensor of at least one sensor type, the circuitry being configured to: determine a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtain sensor data from each edge device of the subset of edge devices for obtaining task training data; update, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.
• the circuitry may be any entity or multitude of entities which may be configurable for determining the subset of edge devices, obtain sensor data, and develop and/ or update a task algorithm as will be described herein.
• the circuitry may include or may be constituted of one or multiple processors (e.g., CPU (central processing unit), GPU (graphics processing unit)), FPGA(s) (field-programmable gate array), an Al (artificial intelligence) sensor, or the like.
• processors e.g., CPU (central processing unit), GPU (graphics processing unit)), FPGA(s) (field-programmable gate array), an Al (artificial intelligence) sensor, or the like.
• the circuitry may be included in one or more computer(s), server(s), smartphone(s), or the like.
• the circuitry may be included in one or more of the edge devices.
• the edge devices may include any type of device which may be controllable or programmable to collect any type of data, such as a smartphone, a camera, a car, a drone, smart glasses, or the like.
• an edge device may include at least one sensor of at least one sensor type.
• a sensor type may include a light sensor (e.g., any type of camera, such as RGB, time-of-flight, multispectral, infrared, or the like), a movement sensor, a gyroscopic sensor, an acceleration sensor, a distance sensor, a pressure sensor, a hygrometer, a microphone, an X-ray sensor, an ultrasound sensor, or the like. Accordingly, one or more of such sensors of a sensor type may be provided in an edge device.
• a smartphone may include one or more cameras, microphones, or the like.
• a car may include one or more pressure sensors, lidar or radar sensors, time-of-flight sensors, or the like.
• a drone may have one or more sensors, as well, such as one or more cameras, hygrometers, velocity sensors, or the like.
• Different edge devices may be suitable for different tasks.
• tasks for different edge devices.
• Drones may be used for semantic segmentation of landscapes, object detection (cars, wind turbines, or the like). Accordingly, drones may be used with different sensor types for their tasks.
• Cars may be equipped with a joint set of sensors (e.g., RGB and radar), but may also have additional sensor(s), such as lidar, cameras pointing in specific directions or angles, or the like.
• Tasks may include pedestrian detection, lane detection, or the like.
• Surveillance systems may use different sensors, as well, such as RGB, infrared, event sensors, and according to the present disclosure, performance of surveillance systems may be improved during operation, e.g., by contributing surveillance data to a general system (e.g., a server), which may train on that training data.
• Tasks may include security breach classification, detection of workers without helmets, or the like.
• edge devices are used for the same training task, such as a car and a smart phone, e.g., for a pedestrian detection task.
• the sensors of the edge devices may be similar, i.e. of the same type (e.g., all are light sensors), but the present disclosure is not limited to that case.
• one sensor may be a time-of-flight sensor, whereas another one may be a pressure sensor.
• the circuitry is further configured to determine a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices. For example, if the task includes object recognition, only such edge devices may be determined which include sensors suitable for object recognition, e.g., light sensors, cameras, time-of-flight sensors, lidar or radar sensors, or the like.
• the subset may include at least one edge device of the plurality of edge devices or, in case every edge device includes a suitable sensor, all edge devices of the plurality of edge devices may be determined as the subset of edge devices.
• the subset is not limited to be a “real” subset in a mathematical sense.
• the circuitry is further configured to obtain sensor data from each edge device of the subset of edge devices for obtaining task training data.
• the task training data may correspond to the data which is acquired by the sensor(s) or may be altered such that the circuitry may be able to process the data.
• the task training data may be labeled (e.g., the circuitry labels them and/ or the edge devices label them depending on the capabilities of each edge device), but the present disclosure is not only applicable to labeled data since a completely unsupervised way of training the task may also be envisaged.
• the circuitry is further configured to update, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.
• the training data may be used, for example, for changing the task algorithm in some way in order to update it.
• a weight parameter of a neural network may be changed, such that carrying out the task may lead to a different result than before the update.
• steps of the task algorithm may be removed, added, or changed (in their order) for updating the task algorithm.
• both types of updates may be carried out.
• the present disclosure is not limited to such update types.
• the task may be carried out by any edge device of the plurality of edge devices. For example, if an edge device was not included in the subset due to an unstable internet connection although that edge devices would have had suitable sensors, the updates may still be provided to that edge device, such that the edge device may carry out the task and/ or participate in further task training.
• the updates may be provided to edge devices which were not included in the subset, as well.
• the task is an object recognition task based only on image data
• different edge devices may be suitable for carrying out that task, e.g., smartphones and drones.
• drone image data depicting the object to be recognized from above is already considered as sufficient. Therefore, only the smartphones are determined as the subset of edge devices which provide the task training data. Since the training based on the task training data provided from the smartphones may lead to different weights of the task algorithm also for the drones, the updated weights may also be provided to the drones.
• the set of predetermined tasks includes classification, segmentation, detection, and recognition, wherein the present disclosure is not limited to these tasks since any task may be adopted.
• the task may include multiple tasks, for example classification and recognition, or any other combination of tasks of the set of predetermined tasks.
• the circuitry is further configured to: change a weight parameter of the task algorithm to an updated weight parameter for updating the task algorithm, as discussed herein.
• the weight parameter includes an edge device weight parameter and a backbone weight parameter.
• the task algorithm may be developed and/ or updated for the edge devices and for a (remote) server.
• the subset of edge devices may constitute a so-called task-head in which the edge devices may constitute a communication network for transmitting their updated weight parameters to each other.
• the backbone which may include a majority of weights of a neural network, a similar training as in the edge devices may be carried out and the weight parameters may be updated similarly and then transmitted to each edge device of the plurality of edge devices.
• the backbone may be configured to produce features out of the data and the task-head may produce a final task result, such as classification, segmentation, or the like.
• the architecture of the backbone may be shared between the tasks while the task-head may vary from task to task.
• the taskhead may be shared between the edge devices which are responsible for the task.
• the weight parameters of the edge devices of the task-head may be averaged, such that each edge device has the same weight parameter and they all carry out the task in (roughly) the same way.
• the circuitry is further configured to: determine an updated edge device weight parameter based on an average value averaged from a plurality of edge device weight parameters determined by the subset of edge devices.
• the circuitry is further configured to: provide the updated edge device weight parameter to the subset of edge devices.
• the edge devices may average the values by themselves or they may be averaged by a central instance (e.g., a cloud).
• the circuitry is further configured to: determine an updated backbone weight parameter, as discussed herein. The updated backbone weight parameter may be distributed to all edge devices, whereas the task-head weight parameter may only be shared in the task-head.
• the circuitry is further configured to: provide the updated backbone weight parameter to the plurality of edge devices, as discussed herein.
• the task algorithm is based on a neural network, as discussed herein. Any type of neural network may be envisaged, but it may depend on the type of task and/ or the sensor type(s) which are available in the plurality of edge devices. However, the present disclosure is not limited to a neural network as any type of artificial intelligence or machine learning algorithm may be utilized.
• the subset of edge devices constitutes a swarm for training the neural network to the predetermined task.
• algorithms for a swarm may be utilized in controlling the swarm and in obtaining the task training data.
• Some embodiments pertain to a method for a plurality of edge devices, each edge device including at least one sensor of at least one sensor type, the method including: determining a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtaining sensor data from each edge device of the subset of edge devices for obtaining task training data; updating, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices, as discussed herein.
• the method may be carried out by circuitry according to the present disclosure.
• the set of predetermined tasks includes classification, segmentation, detection, and recognition, as discussed herein.
• the method further includes: changing a weight parameter of the task algorithm to an updated weight parameter for updating the task algorithm, as discussed herein.
• the weight parameter includes an edge device weight parameter and a backbone weight parameter, as discussed herein.
• the method further includes: determining an updated edge device weight parameter based on an average value averaged from a plurality of edge device weight parameters determined by the subset of edge devices, as discussed herein.
• the method further includes: providing the updated edge device weight parameter to the subset of edge devices, as discussed herein.
• the method further includes: determining an updated backbone weight parameter, as discussed herein. In some embodiments, the method further includes: providing the updated backbone weight parameter to the plurality of edge devices, as discussed herein.
• the task algorithm is based on a neural network, as discussed herein. In some embodiments, the subset of edge devices constitutes a swarm for training the neural network to the predetermined task, as discussed herein.
• the methods as described herein are also implemented in some embodiments as a computer program causing a computer and/ or a processor to perform the method, when being carried out on the computer and/ or processor.
• a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.
• the edge devices may constitute a swarm. Furthermore, they may participate in federated learning (with the difference that the edge devices and sensors may be allowed to be of different types).
• the plurality of edge devices is connected to a central instance (e.g., a server, a local network, or the like), such that they may upload and download weight updates from that instance.
• a central instance e.g., a server, a local network, or the like
• Each edge device may have different capabilities (e.g., processing power, storage, or the like), but in some embodiments, each edge device is assumed to perform inference of an artificial intelligence (e.g., neural network) and perform a (small) number of network updates based on incoming sensor data.
• the edge devices may not all perform the same task, as discussed herein, but they may need to be able to collect data for their own task.
• the following method may be carried out by the edge devices and circuitry according to the present disclosure:
• the data may be labeled or unlabeled. Labeling may be envisaged depending on the purpose of the device. For example, the operator may be an end-user and thus, the data may be labelled (e.g., the device is a smartphone and the person labels photos that he or she takes, or the like). In some embodiments, there are applications of the devices where the label is clear and/ or may be assumed.
• the devices may perform task head updates only and fine-tune to their task and do no other updates.
• the backbone is updated from server downloads only.
• the architecture may be dynamically adaptable.
• NAS may be carried out in which the architecture may be trained, such that the most suitable architecture for each task may be determined.
• each update may change at least one weight parameter and the architecture structure (if necessary and/ or possible).
• one of the task-head and the backbone architecture is fixed (static), while the other may be changeable due to training.
• both the task-head and the backbone are fixed or changeable.
• a neural network may be provided including three parts, namely base features, backbone and task-head, i.e. compared to the embodiments described above, a base features part which feeds the backbone (which may feed the edge devices or the task-head) may be introduced.
• the base features part may be adapted to transform from sensor data into more general features, such that the different sets of training data may be present in the same domain (or have the same data structure, or the like).
• base features such as base RGB, base depth, base X-ray, or the like
• base features may be updated and trained in a similar way as the task-head.
• those edge devices may also train for base RGB for providing features in the RGB domain
• edge devices of a task-head having depth sensors those edge devices may train for base depth.
• circuitry 2 there is depicted a system 1 including circuitry 2 according to the present disclosure. It should be noted that, for illustrational purposes, the circuitry 2 is depicted as two entities 2a and 2b for illustrating the logical separation between the task-head and the backbone, but the circuitry may be provided in one entity (e.g., a server) or multiple entities (e.g., distributed among the edge devices and/or server (s)).
• entity e.g., a server
• multiple entities e.g., distributed among the edge devices and/or server (s)
• Fig. 1 depicts a plurality of edge devices 3a to 3d (four, in this embodiment), which are all implemented as cars, in this embodiment.
• edge devices 3a to 3d four, in this embodiment
• the present disclosure is not limited to edge devices being cars.
• a Task-Zero head 4a is trained by the car 3a
• a Task-One head 4b is trained by the cars 3b and 3c
• a Task-Two head 4c is trained by the car 3d.
• the Task-Zero head 4a and the Task-Two head 4c are each only trained by one car (i.e. the subset of edge devices includes one edge device for these two tasks), there is no need that edge device weight parameters are transmitted to other edge devices.
• the Task-One head 4b is trained by two edge devices, namely the cars 3b and 3c, which are communicating via a network 2a (i.e. via the circuitry 2a according to the present disclosure).
• the car 3b uploads its task training data to the network 2a and downloads the car 3c’s edge device weight parameters from the network 2a.
• the car 3c uploads its task training data to the network 2a and downloads the car 3b’s edge device weight parameters from the network 2a.
• the cars 3b and 3c each average their respective other’s edge device training parameters for updating the Task-1 head.
• a backbone 5 is updated by the cars 3a to 3d and task training data is transmitted to the network 2b which carries out a similar training as the task-heads 4a to 4c do. Accordingly, an updated backbone is provided to the cars 3a to 3d.
• Fig. 2 depicts an embodiment of a method 10 according to the present disclosure.
• a subset of edge devices of the plurality of edge devices is determined based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices, as discussed herein.
• sensor data is obtained from each edge device of the subset of edge devices for obtaining task training data, as discussed herein.
• a task algorithm is updated based on the task training data for carrying out the task by at least one edge device of the plurality of edge devices, as discussed herein.
• Fig. 3 depicts a further embodiment of a method 20 according to the present disclosure, in which the task algorithm is updated by changing a weight parameter.
• a subset of edge devices of the plurality of edge devices is determined based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices, as discussed herein.
• sensor data is obtained from each edge device of the subset of edge devices for obtaining task training data, as discussed herein.
• a weight parameter of the task algorithm is changed to an updated weight parameter for updating the task algorithm, as discussed herein.
• Fig. 4 depicts a further embodiment of a method 30 according to the present disclosure, in which the task algorithm is updated by determining an updated edge device weight parameter.
• a subset of edge devices of the plurality of edge devices is determined based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices, as discussed herein.
• sensor data is obtained from each edge device of the subset of edge devices for obtaining task training data, as discussed herein.
• an updated edge device weight parameter of the task algorithm is determined based on an average value averaged from a plurality of edge device weight parameters determined by the subset of edge devices, as discussed herein.
• the updated edge device weight parameter is provided to the subset of edge devices.
• Fig. 5 depicts a further embodiment of a method 40 according to the present disclosure, in which the task algorithm is updated by determining an updated backbone weight parameter.
• a subset of edge devices of the plurality of edge devices is determined based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices, as discussed herein.
• sensor data is obtained from each edge device of the subset of edge devices for obtaining task training data, as discussed herein.
• an updated backbone weight parameter of the task algorithm is determined, as discussed herein.
• the updated backbone weight parameter is provided to the plurality of edge devices.
• an updated edge device weight parameter may be determined as well as an updated backbone weight parameter, such that a method according to be present disclosure may include both embodiments of updating the weight parameter.
• control 2 could be implemented by a respective programmed processor, field programmable gate array (FPGA) and the like.
• FPGA field programmable gate array
• a non-transitory computer-readable recording medium stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed.
• Circuitry for a plurality of edge devices each edge device including at least one sensor of at least one sensor type, the circuitry being configured to: determine a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtain sensor data from each edge device of the subset of edge devices for obtaining task training data; update, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.
• circuitry of (1) or (2) further configured to: change a weight parameter of the task algorithm to an updated weight parameter for updating the task algorithm.
• the circuitry of (4) further configured to: determine an updated edge device weight parameter based on an average value averaged from a plurality of edge device weight parameters determined by the subset of edge devices.
• the circuitry of (5) further configured to: provide the updated edge device weight parameter to the subset of edge devices.
• circuitry of anyone of (4) to (6) further configured to: determine an updated backbone weight parameter.
• the circuitry of (7) further configured to: provide the updated backbone weight parameter to the plurality of edge devices.
• each edge device including at least one sensor of at least one sensor type
• the method including: determining a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtaining sensor data from each edge device of the subset of edge devices for obtaining task training data; updating, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.
• (20) The method of (19), wherein the subset of edge devices constitutes a swarm for training the neural network to the predetermined task.
• (21) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (20), when being carried out on a computer.
• (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (20) to be performed.
• a system including: a plurality of edge devices, each edge device including at least one sensor of at least one sensor type; and circuitry configured to: determine a subset of edge devices of the plurality of edge devices based on a task of a set of predetermined tasks and based on the at least one sensor of each edge device of the plurality of edge devices; obtain sensor data from each edge device of the subset of edge devices for obtaining task training data; update, based on the task training data, a task algorithm for carrying out the task by at least one edge device of the plurality of edge devices.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• General Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• Computational Linguistics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Molecular Biology (AREA)
• Computing Systems (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Health & Medical Sciences (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=78709341

Family Applications (1)

Country Status (2)

Family Cites Families (1)

• 2022
  • 2022-10-26 EP EP22805895.4A patent/EP4433944A1/de not_active Withdrawn
  • 2022-10-26 WO PCT/EP2022/079923 patent/WO2023088650A1/en not_active Ceased

Also Published As

Similar Documents

Legal Events