Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/088—Non-supervised learning, e.g. competitive learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/0464—Convolutional networks [CNN, ConvNet]

Definitions

• the present application discloses a method, a system and computer program for processing image data and adapting them to different domains.
• the present application further discloses a vehicle comprising such a system.
• Machine learning has great potential for modern driver assistance systems and automated driving.
• Functions based on deep neural networks process raw sensor data (e.g. from camera, radar, lidar) to derive relevant information. This includes, for example, the type and position of objects in the vehicle environment, their behaviour, roadway geometries and topologies as well as the detection and classification of free path for the vehicles.
• CNNs convolutional neural networks
• ViT visual transformer networks
• Machine learning algorithms must undergo a training process in order to be reliable.
• the machine learning algorithms are fed with a large amount of data: based on which type of data are used during training, machine learning methods can be divided in supervised machine learning methods, unsupervised machine learning methods and semi-supervised machine learning methods.
• Unsupervised machine learning systems use machine learning algorithms to analyse and cluster unlabelled (not annotated) datasets. Those algorithms do not require the intervention of a human user during the training process.
• the data clustering and analyse can been done for example through k-means clustering or probabilistic clustering methods.
• a drawback of unsupervised machine learning systems is that they are non-transparent for humans as humans have no influence on the way data are clustered and classified.
• Semi-supervised machine learning algorithms use a smaller labelled data set to guide classification and feature extraction from a large, unlabelled data set.
• Semi-supervised machine learning has the drawbacks that the clustering assumptions of the small labelled data set not always hold for the full unlabelled data set.
• Supervised machine learning methods are based on the use of labelled (annotated) data sets for the training process.
• human agents have to manually annotate the data by using annotation computer programs, where the images to be annotated have been uploaded.
• annotation computer programs where the images to be annotated have been uploaded.
• the human agent draws geometric figures, like for example bounding boxes or polygons, around each relevant object in an image, which has to be annotated.
• a label is then assigned to the object in the geometric figure.
• a label describes the class to which an object belongs. Examples of classes are vehicles, pedestrian, traffic signs, traffic lights and so on.
• a drawbacks of supervised machine learning system is that the process of manually annotating data is quite expensive and time consuming.
• Data augmentation allows to artificially modify existing labelled data sets by applying small modifications to them.
• existing labelled data sets can be randomly rotated, translated, cropped or the colours can be modified and/or altered. Additionally random objects could be added to the data.
• synthetic data generation scenes can be generated that would be too dangerous to create in real life, e.g., traffic accidents.
• the domain gap to real data typically leads to decreased performance of the machine learning algorithm.
• UDA unsupervised domain adaptation
• unsupervised domain adaptation methods aim at training a network to generalize well towards unseen/unknown target domains, but those target domains must already be employed during source domain training: as a consequence, unsupervised domain adaptation methods are not suitable to the task of adapting an already trained model to a new domain.
• the low-level spectrum of an image in particular the low frequency amplitude can vary in a significant way. This variation does not affect the perception of the object represented.
• the appearance of an object such as a vehicle or a pedestrian or a traffic sign in an image is independent from the low-level spectrum of the image itself. It is then possible to alter the low-level spectrum of an image without altering its high semantic, leaving the perception of the objects unaltered.
• the method of the present application comprises the following steps
• step (vii) Generating a new image X DT ®S based on the frequency spectrum of step (vi)
• the new image is generated directly from the target data, i.e. an online domain adaptation is performed.
• the generation of the new image X° T ⁇ ,s takes place in real time, as the target image D T are continuously modified according to the method of claim 1. This has the advantage to increase the performance on multiple unseen target domains, since those target data do not have to be available during training.
• target data taken by one or more vehicle’s sensors can undergo a style transfer according to the method of claim 1 before to be used as input by the trained machine learning algorithm.
• the trained machine learning system can be used also for target domains that differ from the source domain without undergoing an additional training phase.
• the source image data are labelled data.
• the machine learning algorithm which is trained using such source image data is a semantic segmentation algorithm. Semantic segmentation refers to the process of linking each pixel in an image to a class. Examples of classes are vehicles, pedestrian, traffic signs, traffic lights and so on.
• the machine learning algorithm is preferably a neuronal network, for example a convolutional network.
• the machine learning algorithm which is trained using the source data is an object detection algorithm.
• An object detection algorithm is a machine learning algorithm which takes as input an image comprises several objects and gives as output one or more bounding boxes around each object and assigns a class annotation to each bounding box.
• the machine learning algorithm which is trained using the source data is an object classification algorithm.
• An object classification algorithm is a machine learning algorithm which takes as input an image comprises one object and assigns as output a class annotation (also called class label) to said object.
• the method of the present invention further comprises the steps of computing and storing the frequency spectrum of the source image data D s .
• the storing of the frequency spectrum of the source image data D s takes place during the training process.
• DFT two- dimensional Fourier transform
• the amplitude spectrum of the source domain image is given by
• This set will be called in the following codebook.
• the codebook is generated using the Linde-Buzo-Gray (LBG) algorithm employing the same source image data that are used for the training of the machine learning method.
• LBG Linde-Buzo-Gray
• the frequency spectrum can be computed on more than one image: an average is then calculated and stored in the codebook.
• the frequency spectrum is divided in several frequency ranges or frequencies subset. Preferably only frequency spectra corresponding to chosen subset of frequencies are stored in the codebook. Preferably the frequency spectrum corresponding to the low-level spectrum of the image are stored.
• the amplitude spectra of the low-level spectrum are stored in the codebook.
• the method of the present invention further comprises the step of obtaining target image data D T .
• the target image data are taken by one or more vehicle’s sensors.
• the sensor can be a camera, a stereocamera, a radar, a lidar or a combination thereof.
• the frequency spectrum is computed for the target image data D T by using a two- dimensional Fourier transform (DFT) in the same way as explained above for the source image data D s .
• DFT two- dimensional Fourier transform
• phase spectrum ⁇ While the phase spectrum ⁇ ) is kept unaltered, the amplitude spectrum ⁇ X Dt ⁇ is modified according to the method of claim 1.
• the stored frequency spectrum of the source image data having similarities with frequency spectrum of the target image is extracted from the codebook.
• the selected codebook entry R ⁇ l is then mixed with the target domain amplitude spectrum . using the following masking technique: with ° denoting element-wise multiplication, 1 being an all-ones tensor, and M denoting the employed mask.
• the amplitude spectra should be shifted, since they are in non-centralized form.
• the first and second quadrant are swapped with the third and the fourth quadrant, respectively.
• X° T >s (X DT®S c ,h,w) is computed via a two dimensional inverse discrete Fourier transform by using the unaltered phase spectrum f° t and the mixed amplitude spectrum ⁇ X DT >s ⁇ as follows:
• (X DT >s c ,h,w) is then used as input for a previous trained machine learning algorithm, said machine learning algorithm having being trained on the source image data only.
• the machine learning algorithm is a semantic segmentation neural network that has been trained on the source domain only.
• the new image or style- transferred image X° T > s (X DT >s C:Kw ) is used as training image for training a machine learning algorithm.
• This has the advantage of improving the robustness and precision of the machine learning algorithm, especially if different target image data are used.
• Preferably both the new image and the source image data are used for training a machine learning algorithm.
• machine learning algorithms trained using both the new image and the source image data perform better in domain adaptation.
• the present application discloses a system according to claim 11.
• the system comprises an interface, one or more storage units and one or more processing devices capable to perform the method for generating the style transferred image X° T >s .
• the interface may correspond to one or more inputs and/or outputs for receiving and/or transmitting information, which may be in digital (bit) values according to a specified code, within a module, between modules or between modules of different entities.
• the interface may comprise interface circuitry configured to receive and/or transmit information.
• the one or more processing devices may be implemented using one or more processing units or any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software.
• a processor such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software.
• the described function of the one or more processing devices may as well be implemented in software, which is then executed on one or more programmable hardware components.
• Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc.
• DSP Digital Signal Processor
• the one or more storage units may comprise at least one element of the group of a computer readable storage medium, such as an magnetic or optical storage medium, e.g. a hard disk drive, a flash memory, Floppy-Disk, Random Access Memory (RAM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM), or a network storage.
• a computer readable storage medium such as an magnetic or optical storage medium, e.g. a hard disk drive, a flash memory, Floppy-Disk, Random Access Memory (RAM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM), or a network storage.
• the storage unit may also be an external cloud-server.
• the one or more storage units store the codebook: in such a way the frequency spectra and particularly the amplitude spectra of the source data can be retrieved at any time.
• the present application discloses a vehicle according to claim 13.
• the vehicle may be equipped with one or more sensors. Examples of sensors are radar, LiDAR, camera, stereocamera or a combination thereof.
• the target data are data captured by one or more sensors mounted on the vehicle.
• the target data are then processed according to the method disclosed above and used for performing object detection, object classification or image segmentation for use in a driving assistance feature (e.g. an autonomous or semi- autonomous driving operation) of the vehicle.
• a driving assistance feature e.g. an autonomous or semi- autonomous driving operation
• the method of the present application can be implemented as a computer program according to claim 15.
• Fig. 1 shows a flow chart of a method for training a machine learning algorithm using source image data D s and for generating a codebook Jl Ds
• Fig. 2 shows a flow chart of a method for domain adaptation between target domain data D T and source domain data D s
• Fig 3 shows a preferred embodiment of the present application
• Fig. 4 describes the function that transforms
• Fig. 5 shows a vehicle equipped with a system according to the present application
• source image data D s are obtained.
• the source image data D s comprises a set of image x Ds .
• Those images are used in step 102 as training images for a machine learning algorithm.
• the machine learning algorithm transforms the images into outputs y Ds .
• the outputs y Ds give a posteriori probability that a given pixel belongs to a given class on which the semantic segmentation is trained.
• step 103 a two dimensional Fourier transform is applied to the source image data D s in order to compute the frequency spectrum X Ds .
• the step 103 can be performed at the same time of the training process in step 102 as well as before or after step 102. In other words, step 102 and step 103 are not connected to each other and as such can be performed independently form each other.
• step 104 the frequency spectrum X° s are saved in the codebook. In a preferred embodiment only the amplitude spectra
• a method for domain adaptation between target domain data D T and source domain data D s is shown.
• the method is an online method.
• step 201 a set of target image data are obtained.
• the target image data are captured directly by one or more vehicle’s sensors.
• step 202 a two dimensional Fourier transform is applied to the source image data D T in order to compute the frequency spectrum X° T .
• the frequency spectrum presents a phase f° t and a spectral amplitude
• the spectral amplitude ⁇ X° T ⁇ is replaced by the most similar spectral amplitude from the codebook computed in Fig.
• step 203 is the new amplitude spectrum
• step 204 a new image or style-transferred image is computed via a two dimensional inverse discrete Fourier transform by using the unaltered phase spectrum f° t from step 202 and the mixed amplitude spectrum from step 203.
• the new image or style transferred image is then used as input to the trained machine learning of Fig. 1.
• step 301a source domain data D s are obtained while in step 301b target domain data D T are obtained.
• step 302a the two dimensional Fourier transform of the source domain data D s is computed while in step 302b the two dimensional Fourier transform of the target domain data D T is computed.
• step 303 the frequency spectrum X Ds are saved in the codebook. In a preferred embodiment only the amplitude spectra
• step 304 the amplitude spectra of the target domain data D T are replaced by the stored amplitude spectra of the codebook entry which satisfy
• the output of step 304 is a mixed amplitude spectrum ⁇ X
• step 305 a new image or style-transferred image X is computed via a two dimensional inverse discrete Fourier transform by using the unaltered phase spectrum f° t from step 302b and the mixed amplitude spectrum from step 304
• step 306 the new image or style-transferred image are used as training input for a machine learning algorithm.
• the source domain data image x° s can be used as training images together with
• Fig. 4 describes the function 400 which generates the mixed amplitude spectrum ⁇
• step 403 Given as input the spectral amplitude ⁇ X Dt ⁇ (step 401) and the codebook Ji° s (step 402) the codebook entry R°l is calculated in step 403. In step 404 a masking function is applied and is calculated
• Fig. 5 shows a vehicle 500.
• the vehicle comprises one or more sensors 501.
• the one or more sensors 501 can be one or more cameras.
• the sensor takes one or more images of the vehicle’s environment.
• the images constitute the target image dataset D T .
• the target image dataset is then sent to one or more processing devices 502.
• the processing devices 502 calculate the frequency spectra of the target image data according to the method explained above.
• the processing devices are connected one or more storage units 503.
• the one or more storage units 503 store the codebook 3?°* .
• the storage units 503 are located in an external server and the vehicle 500 is connected to them via a well-known vehicle to infrastructure technology.
• the one or more storage units 503 could also be also mounted directly in the vehicle and be connected with the processing devices 502 via cables.
• the processing devices 502 take the information from the codebook stored in the storage units 503 and modify the target source data according to the methods explained above in order to generate a new image X° T ,s .
• the new image is then used as input for a machine learning algorithm, previously trained with the source image dataset.
• the machine learning algorithm is preferably stored in the processing devices 502.
• the machine learning algorithm is an object classification algorithm. Given as input the new image X° T ,s ., the machine learning algorithm classifies as output the objects present in the input image according to classes. Examples of classes are vehicles, pedestrian, traffic signs, traffic lights and so on.
• the machine learning algorithm is an object detection algorithm. Given as input the new image X° T ,s ., the machine learning algorithm gives as output the object class and the object localization in the image (typically by drawing a bounding box around the object)
• the machine learning algorithm stored in the processing devices 502 is a semantic segmentation algorithm. Given as input the new image X° T >s ., the semantic segmentation algorithm labels each pixel of the image with a corresponding class. For example, the algorithm could label each pixel of the image according to the class “road”, “vehicle”, “pedestrian”.
• the results of the machine learning algorithm can be used for controlling the vehicle, for example by determining a trajectory to avoid collision with other vehicles and/or objects.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• General Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• Computational Linguistics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Molecular Biology (AREA)
• Computing Systems (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Health & Medical Sciences (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (1)

Publications (1)

Family

ID=77155741

Family Applications (1)

Country Status (2)

Families Citing this family (1)

• 2021
  • 2021-07-09 WO PCT/EP2021/069167 patent/WO2023280423A1/en not_active Ceased
  • 2021-07-09 EP EP21748803.0A patent/EP4367602A1/de active Pending

Also Published As

Similar Documents

Legal Events