EP4241237A1 - Dispositif et procédé pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique - Google Patents

Dispositif et procédé pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique

Info

Publication number
EP4241237A1
EP4241237A1 EP20807341.1A EP20807341A EP4241237A1 EP 4241237 A1 EP4241237 A1 EP 4241237A1 EP 20807341 A EP20807341 A EP 20807341A EP 4241237 A1 EP4241237 A1 EP 4241237A1
Authority
EP
European Patent Office
Prior art keywords
image
synthesized image
discriminator
neural network
mask
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP20807341.1A
Other languages
German (de)
English (en)
Inventor
Onay URFALIOGLU
Akhil GURRAM
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SHENZHEN YINWANG INTELLIGENTTECHNOLOGIES CO., LTD.
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of EP4241237A1 publication Critical patent/EP4241237A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • the present disclosure relates to the field of advanced driver assistance systems (ADAS), computer vision and machine learning (ML).
  • ADAS advanced driver assistance systems
  • ML machine learning
  • the present disclosure provides an ML based way to train a neural network based on a synthesized image (which is generated based on a real image or a virtual image) to improve determining of a depth map, a relative pose or a semantic segmentation (which can e.g. be used for training an autopilot of a self-driving vehicle).
  • the present disclosure relates to domain adaptation, i.e. to solving computer vision problems by training a neural network on virtual images and testing on real images.
  • Estimation of a relative pose, a depth map or a semantic segmentation based on sensor input is an important task for a robot, an ADAS or a self-driving system.
  • Sensor input used for said estimation e.g. comes from a visual sensor (e.g. an input image taken by a digital camera).
  • the information (depth map, semantic segmentation or relative pose), which is estimated based on the input image, can be used (together with the input image itself) for further training of the robot, the ADAS or the self-driving system).
  • an auto-pilot of a vehicle can be trained based on the input image and based on the corresponding relative pose, depth map or semantic segmentation.
  • estimation techniques such as stereo matching or self-supervised deep learning methods have made progress, but require a large quantity of real high-quality stereo images or real sequential images as input images for training of estimation.
  • real stereo images or real sequential images are easy to produce, it is difficult to create an accurate depth map, relative pose or semantic segmentation (which could be used as ground truths for further training) corresponding to said real images.
  • creating virtual images (which are not taken by a camera, but e.g. are generated by a computer) with corresponding accurate information regarding depth map, relative pose or semantic segmentation is feasible.
  • a large quantity of training data e.g. for training the robot, the ADAS or the self-driving system
  • a domain gap between virtual images and real images can be noticed e.g. because an image texture or a color intensity of a virtual image are not as good as in a real image.
  • This domain gap also decreases the quality of estimation of a depth map, a relative-pose estimation or a semantic segmentation, based on an input image, if the neural network which performs the estimation is only trained based on the virtual images.
  • an objective of embodiments of the present disclosure is to improve domain adaptation between different domains of images, such as virtual images and real images.
  • a first aspect of the present disclosure provides a device for determining a depth map, a relative pose, or a semantic segmentation, wherein the device comprises a neural network configured to, in an inference phase, determine the depth map, the relative pose, or the semantic segmentation based on an input image; and a generator configured to, in a training phase, generate a synthesized image based on a real image or a virtual image, and based on a loss function, and train the neural network based on the synthesized image; wherein the loss function comprises a semantic edge function.
  • the input image can be a real image or a virtual image, which is input to the device.
  • the real image or the virtual image is input to the device for generating training data for an autopilot of a vehicle.
  • the real image is a 2-dimensional picture (e.g. an RGB or a chrominanceluminance picture) acquired with a stereo camera in a real environment.
  • the real image is e.g. acquired based on at least one of: KITTI, CITYSCAPES, AEV datasets.
  • the virtual image is a 2-dimensional picture (e.g. an RGB picture or a chrominance-luminance picture) acquired with a stereo camera in a virtual environment such as Carla, or a photorealistic dataset.
  • a 2-dimensional picture e.g. an RGB picture or a chrominance-luminance picture
  • the synthetized image is a 2-dimensional picture (e.g. an RGB picture or a chrominance-luminance picture) generated by a generative adversarial network (GAN) based on a real image or a virtual image.
  • GAN generative adversarial network
  • a depth map is a 2D image or a matrix, in which each pixel or element depicts a depth of a corresponding 3D point in a scene with respect to a camera.
  • a scene is a predefined region of interest of a real world, captured by a camera.
  • a disparity map is a 2D image or matrix where each pixel or element depicts the disparity of that pixel or element.
  • a relative pose is a 6D vector comprising 3D location coordinates (e.g. x, y, z) and 3 angles for an orientation of a vehicle (e.g. yaw, pitch, roll).
  • a semantic segmentation comprises a classification for each pixel of the input image what kind of object it is depicting (e.g. at least one of a car a vegetation, a building, a sky, a road).
  • a semantic edge function considers at least one edge in the real image or the virtual image.
  • an edge comprises a border of an object in an image, wherein the border comprises a significant contrast change.
  • the semantic edge function is configured to maintain semantic gradient information and/or edge information in the synthesized image.
  • the device further comprises a first discriminator and a second discriminator, wherein the generator is further configured to, in the training phase, provide the synthesized image to the first discriminator or to the second discriminator, to train the neural network.
  • a texture is an area of an image which depicts content having significant variation in color intensities.
  • the generator is further configured to, in the training phase, train the neural network based on a determination result of the discriminator to which the synthesized image was provided.
  • the generator is further configured to, in the training phase, randomly provide the synthesized image to the first discriminator or to the second discriminator.
  • the first discriminator is further configured to, in the training phase, determine that a synthesized image that is generated by the generator based on a virtual image, is a fake, and determine that the virtual image is an original; and the second discriminator is further configured to, in the training phase, determine that a synthesized image that is generated by the generator based on a real image, is a fake, and determine that the real image is an original.
  • the device is further configured to train the neural network, based on the synthesized image, for determining at least one of a depth map, a relative pose, a semantic segmentation.
  • the device is further configured to, in the training phase, generate a learnable mask based on the synthesized image; and train the neural network based on the learnable mask.
  • the learnable mask is a region of interest in the synthesized image.
  • the learnable mask allows to determine if a pixel of the synthesized image can be used for view reconstruction or not.
  • the learnable mask is a semantic inlier mask.
  • the semantic inlier mask is a region of interest in the synthesized image.
  • the semantic inlier mask allows to determine if a pixel of the synthesized image can be used for view reconstruction or not.
  • the semantic inlier mask comprises semantic segmentation information.
  • the semantic segmentation information can be used for determining, if a pixel of the synthesized image can be used for view reconstruction or not.
  • the semantic inlier mask is generated based on semantic segmentation information by a neural network.
  • the device is further configured to train the neural network, based on the learnable mask, for determining at least one of a depth map, a relative pose, a semantic segmentation.
  • the neural network is trained, based on the semantic inlier mask, for determining a depth map.
  • the device is further configured to, in the training phase, determine segmentation information based on the synthesized image and generate the learnable mask based on the segmentation information.
  • the device is further configured to, in the training phase, determine pose information based on the synthesized image and generate the learnable mask based on the pose information.
  • the device is further configured to, in the training phase, determine an inlier mask based on the synthesized image and generate the learnable mask based on the inlier mask.
  • the inlier mask is a region of interest in the synthesized image.
  • the inlier mask allows to determine if a pixel of the synthesized image can be used for view reconstruction or not.
  • the device is further configured to, in the training phase, if the synthesized image is generated based on the real image, apply self-supervised training to the neural network based on the synthesized image.
  • the device is further configured to, in the training phase, if the synthesized image is generated based on the virtual image, apply supervised training to the neural network based on the synthesized image and/or ground truth labels corresponding to the synthesized image.
  • the device is the ground truth labels comprise a depth map, a relative pose, or a semantic segmentation.
  • the device further comprises a third discriminator and a fourth discriminator, wherein the device is further configured to train the neural network based on a determination result of the third discriminator, and/or based on a determination result of the fourth discriminator.
  • the device is configured to train the neural network for determining a depth map based on the third discriminator.
  • the device is configured to train the neural network for determining a semantic segmentation based on the fourth discriminator.
  • a second aspect of the present disclosure provides a method for determining a depth map, a relative pose, or a semantic segmentation, the method comprising the steps of: in an inference phase, determining, by a neural network of a device, the depth map, the relative pose, or the semantic segmentation based on an input image; and in a training phase, generating, by a generator of the device, a synthesized image based on a real image or a virtual image, and based on a loss function, and training, by the generator, the neural network based on the synthesized image; wherein the loss function comprises a semantic edge function.
  • the semantic edge function maintains semantic gradient information and/or edge information in the synthesized image.
  • the method further comprises, in the training phase, providing, by the generator, the synthesized image to a first discriminator of the device or to a second discriminator of the device, to train the neural network.
  • the method further comprises, in the training phase, training, by the generator, the neural network based on a determination result of the discriminator to which the synthesized image was provided.
  • the method further comprises, in the training phase, randomly providing, by the generator, the synthesized image to the first discriminator or to the second discriminator.
  • the method further comprises, in the training phase, determining, by the first discriminator, that a synthesized image that is generated by the generator based on a virtual image, is a fake, and determining, by the first discriminator, that the virtual image is an original; and, in the training phase, determining, by the second discriminator, that a synthesized image that is generated by the generator based on a real image, is a fake, and determine that the real image is an original.
  • the method further includes training, by the device, the neural network, based on the synthesized image, for determining at least one of: a depth map, a relative pose, a semantic segmentation.
  • the method further includes, in the training phase, generating, by the device, a learnable mask based on the synthesized image; and training, by the device, the neural network based on the learnable mask.
  • the learnable mask is a semantic inlier mask.
  • the method further includes training, by the device, the neural network, based on the learnable mask, for determining at least one of: a depth map, a relative pose, a semantic segmentation.
  • the method further includes, in the training phase, determining, by the device, segmentation information based on the synthesized image and generating, by the device, the learnable mask based on the segmentation information.
  • the method further includes, in the training phase, determining, by the device, pose information based on the synthesized image and generate the learnable mask based on the pose information.
  • the method further includes, in the training phase, determining, by the device, an inlier mask based on the synthesized image and generating, by the device, the learnable mask based on the inlier mask.
  • the second aspect and its implementation forms include the same advantages as the first aspect and its respective implementation forms.
  • a third aspect of the present disclosure provides a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method of the second aspect or any of its implementation forms.
  • the third aspect and its implementation forms include the same advantages as the second aspect and its respective implementation forms. It has to be noted that all devices, elements, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.
  • FIG. 1 shows a schematic view of a device according to an embodiment of the present disclosure
  • FIG. 2 shows a schematic view of a device according to an embodiment of the present disclosure in more detail
  • FIG. 3 shows a schematic view of an operating scenario according to the present disclosure
  • FIG. 4 shows a schematic view of a depth map according to the present disclosure
  • FIG. 5 shows a schematic view of an operating scenario according to the present disclosure
  • FIG. 6 shows a schematic view of an operating scenario according to the present disclosure
  • FIG. 7 shows a schematic view of an operating scenario according to the present disclosure
  • FIG. 8 shows a schematic view of an operating scenario according to the present disclosure
  • FIG. 9 shows a schematic view of a method according to an embodiment of the present disclosure.
  • FIG. 1 shows a schematic view of a device 100 according to an embodiment of the present disclosure.
  • the device 100 is for determining a depth map 101, a relative pose 102, or a semantic segmentation 103 based on an input image 105. Moreover, said determining is improved by an ML based approach.
  • the device 100 comprises a neural network 104 and a generator 106. The determining is in particular improved based on the neural network 104, which is trained based on a synthesized image 107.
  • the device 100 distinguishes between an inference phase and a training phase.
  • the trained neural network 104 is applied to an input image 105 to determine the depth map 101, the relative pose 102, or the semantic segmentation 103 based on the input image 105.
  • training data i.e. a real image 108 or a virtual image 109
  • the generator 106 is configured to generate a synthesized image 107 based on a real image 108 or a virtual image 109.
  • the generator also uses a loss function 110.
  • the loss function 110 in particular comprises a semantic edge function 111.
  • the semantic edge function 111 can maintain semantic gradient information and/or edge information in the synthesized image 107.
  • the device 100 can use virtual images 109 along with ground truth labels (e.g. a depth map, a semantic segmentation or a relative pose) which are specifically generated for the virtual images 109 to train the neural network 104 and e.g. test it on real images 108.
  • ground truth labels e.g. a depth map, a semantic segmentation or a relative pose
  • the device 100 may comprise a processor or processing circuitry (not shown) configured to perform, conduct or initiate the various operations of the device 100 described herein.
  • the processing circuitry may comprise hardware and/or the processing circuitry may be controlled by software.
  • the hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry.
  • the digital circuitry may comprise components such as applicationspecific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors.
  • the device 100 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor or by the processing circuitry, in particular under control of the software.
  • the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the various operations of the device 100 to be performed.
  • the processing circuitry comprises one or more processors and a non- transitory memory connected to the one or more processors.
  • the non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 100 to perform, conduct or initiate the operations or methods described herein.
  • FIG. 2 shows a schematic view of a device 100 according to an embodiment of the present disclosure in more detail.
  • the device 100 shown in FIG. 2 comprises all features and functionality of the device 100 of FIG. 1, as well as the following optional features:
  • the device 100 optionally comprises a first discriminator 201 and a second discriminator 202.
  • the generator 106 optionally can provide the synthesized image 107 to the first discriminator 201 or to the second discriminator 202, to train the neural network 104.
  • the first discriminator 201 and the second discriminator 202 support the generator 106 to create a synthesized image 107 with a texture similar to the domains using virtual image 109 and real image 108.
  • the generator 106 can provide the synthesized image 107 to the first discriminator 201 or to the second discriminator 202 on a random basis.
  • the generator 106 trains the neural network 104 based on a determination result of the discriminator 201, 202 to which the synthesized image 107 was provided.
  • the synthesized image 107 is produced by the generator 106 and by the first discriminator 201 and the second discriminator 202 based on an adversarial loss function (i.e. the loss function 110), which mainly considers universal common features, such as semantic edges and semantic information, patterns, scene structure, as well as artifacts such as texture, color filters, noise, lighting effects, shadows and reflections produced by a camera sensor for training the neural network 104.
  • an adversarial loss function i.e. the loss function 110
  • a semantic gradient or edge based reconstruction can be used (potentially in combination with a robust loss function). This supports the generator 106 to reconstruct the synthesized image 107 by keeping the scene structure and geometric properties and mix or create the texture, intensities, or quality between a virtual image 109 and a real image 108.
  • MSE mean square error
  • the loss function 110 is a semantic edge function 111, the purpose of which can be to maintain semantic gradient information and/or semantic edge information to generate a synthesized image 107. Additionally, by using the first discriminator 201 and the second discriminator 202, the generator 106 can produce an image texture similar to the real image domain, without losing geometry structure.
  • the synthesized images 107 which are produced by the generator 106, the first discriminator 201 and the second discriminator 202, can be used for further training of the neural network 104, e.g. for improving determining a depth map 101, a relative pose 102 or a semantic segmentation 103 based on the synthesized image 107.
  • a respective loss function 110 e.g. a Ll/view reconstruction function, a cross-entropy function, or an LI function.
  • the generator 106 and the discriminators 201, 202 generate a synthesized image 107 by considering that semantic edges in the synthesized image 107 should be same as in the original input (that is, in the real image 108 or the virtual image 109), wherein the synthesized image 107 has a higher variation of texture.
  • providing the synthesized image 107 to the first discriminator 201 or the second discriminator 202 on a random basis allows to generate, by the generator 106, the synthesized image 107, which has a higher variation of textures.
  • the first discriminator 201 determines that a synthesized image 107, which is generated by the generator 106 based on a virtual image 109, is a fake, and determines that the virtual image 109 is an original.
  • the second discriminator 202 determines that a synthesized image 107, which is generated by the generator 106 based on a real image 108, is a fake, and determines that the real image 108 is an original.
  • the device 100 in the training phase the device 100 optionally generates a learnable mask 203 based on the synthesized image 107.
  • the learnable mask 203 indicates pixels in the synthesized image 107, which are important for further training the neural network 104.
  • the device 100 optionally further trains the neural network 104 based on the learnable mask 203 and based on the synthesized image 107.
  • the learnable mask 203 can be used to train the neural network 104 to improve the determining of at least one of the depth map 101, the relative pose 102, the semantic segmentation 103.
  • the learnable mask 203 is a semantic inlier mask.
  • the semantic inlier mask indicates a region of interest in the synthesized image 107 by means of semantic segmentation information in the semantic inlier mask.
  • the device 100 in the training phase the device 100 optionally determines segmentation information 204 based on the synthesized image 107.
  • the segmentation information 204 allows to associate portions of the synthesized image 107 with labels, e.g. indicating that a predefined portion is at least one of a street, a wall, a tree, a traffic light, a sidewalk, a sky, a house.
  • the device 100 then generates the learnable mask 203 based on the segmentation information 204 and based on the synthesized image 107.
  • the device 100 in the training phase the device 100 optionally determines pose information 205 based on the synthesized image 107.
  • the pose information 205 e.g. comprises location coordinates and angles for an orientation of a vehicle.
  • the device 100 then generates the learnable mask 203 based on the pose information 205 and based on the synthesized image 107.
  • the device 100 in the training phase the device 100 optionally determines an inlier mask 206 based on the synthesized image 107.
  • estimated semantic segmentation information is incorporated within the inlier mask 206 for training a selfsupervised depth estimation model of the neural network 104.
  • the device 100 then generates the learnable mask 203 based on the inlier mask 206 and based on the synthesized image 107.
  • training the neural network 104 based on the inlier mask 206 and based on the segmentation information 204 enables to improve the determining of a depth map 101 based on an input image 105 by the device 100.
  • Two further discriminators (not shown in FIG. 2) can support to shift the real images domain and the corresponding distribution of estimated depth and semantic segmentation towards the virtual domain.
  • the device 100 enables to train the neural network 104 with the additional support of two discriminators 201, 202 to shift a domain from real to virtual based on estimated depth maps and semantic segmentation results of a synthesized image 107.
  • FIG. 1 and FIG. 2 can reduce the domain gap between virtual image 109 and real images 108 (or a virtual and a real dataset) based on a self-supervised approach to improve the determination of depth maps 101, semantic segmentation 103 and relative-pose estimation 102.
  • FIG. 3 shows a schematic view of a device 100 according to an embodiment of the present disclosure in even more detail.
  • the device 100 shown in FIG. 3 comprises all features and functionality of the device 100 of FIG. 1 and FIG. 2, as well as the following optional features:
  • the generator 106 (which can also be called domain adaptation module or generator block) is now going to be described in more detail.
  • the generator 106 can be configured to train a model (e.g. the neural network 104) to create novel domain images with common features (e.g. the synthesized images 107) based on the virtual image domain and the real image domains.
  • An adversarial loss function i.e. the loss function 110
  • the first and the second discriminator 201, 202 are used.
  • reconstruction loss is used as in the estimated common domain RGB image of the neural network 104.
  • the input images 105 can be from the virtual or the real domain in a random pattern.
  • An output of the generator 106 passes through a semantic gradient or edge based loss function (i.e. the loss function 110) and through one of the first or second discriminator 201, 202.
  • the selection of the discriminator is completely random.
  • the reason to choose only one discriminator is to compute the results by the generator 106 based on the judgement provided by the chosen discriminator, which helps to mix the texture information of the real image 108 and the virtual image 109 by keeping scene structure and edges safely.
  • the randomness of choosing discriminators or choosing the dataset helps the neural network 104 to not go into a local minima, instead the loss will be fluctuating as this might reach to global minima.
  • the loss function 110 (which can also be called image gradient based loss function) is now going to be described in more detail.
  • a robust loss function 110 is provided, which is completely based on image semantic edges.
  • Semantic edge based learning allows the generator 106 to produce images that maintain semantic gradient or edge information from the beginning of training and at the same time ensure a higher variation of texture on the images (i.e. the synthesized image 107) that are generated to train depth estimation (DE), semantic segmentation (SS) and relative pose estimation (RPE) models.
  • DE depth estimation
  • SS semantic segmentation
  • RPE relative pose estimation
  • computer vision application models are trained by considering the semantic edges as a common important feature being used to train the models (DE, SS, RPE. Convolutions of the DE, SS, and RPE model allows to learn computer vision applications independent of varying texture, shadows, lighting effects, weather conditions, or color filtering.
  • the first discriminator 201 and the second discriminator 202 (which can also be part of the domain adaptation module) are now going to be described in more detail.
  • a Wasserstein Discriminator can be used, which uses an Earth-Mover’s distance to minimize the discrepancy between the distributions for the virtual dataset (i.e. the virtual image 109) and the real dataset (i.e. the real images 108).
  • a gradient penalty can be applied by at least one of the discriminators 201, 202 to overcome the problem of vanishing or exploding gradients.
  • the purpose of the first discriminator 201 can be to learn if the input of the neural network 104 is from a virtual dataset or not.
  • virtual images 109 are considered as true/real while the output of the generator 106 is always considered false/fake.
  • the purpose of the second discriminator 202 is similar to the first discriminator 201 but operates exactly the other way round (i.e. oppositely). The second discriminator 202 always considers real images 108 as true/real and the output of the generator 106 as false/fake.
  • the output of the generator 106 is always considered as false/fake and randomly one of the discriminators 201, 202 is chosen to judge if the generated RGB (i.e. the synthesized image 107) is virtual or real.
  • the generator 106 will produce images with mixed texture and features.
  • a depth map 101 is a representation of a 3D scene structure projected as a pattern depending on camera sensor parameters and semantic information.
  • semantic information can be purely the pattern of 3D scene, edges, and shapes of objects.
  • the device 100 optionally may comprise a third discriminator 301 and a fourth discriminator 302.
  • the third discriminator 301 may operate on an estimated depth map 101, while the fourth discriminator 302 may operate on an estimated semantic segmentation.
  • the purpose of the third discriminator 301 and the fourth discriminator 302 is to further improve determining the depth map 101 and the semantic segmentation 103 based on input images 105.
  • FIG. 4 shows a portion of an input image 401 and two portions of depth maps 402, 403 which are determined based on the input image 401 by a device which only has been trained on virtual images 109.
  • the third and fourth discriminator 301, 302 help to improve the estimated depth map or semantic segmentation based on input texture (by training the device 100 both on a virtual or a real dataset).
  • the generator 106 is forced to produce better synthesized images 107 having common features of the virtual and real domain.
  • the neural network 104 is trained based on the output of generators for segmentation, depth net or pose and mask net, i.e. based on the segmentation information 204, depth information, the pose information 205 and the inlier mask 206.
  • a supervised approach for creating ground truths for improving the depth map 101, the relative pose 102 and the semantic segmentation 103 determination a self-supervised approach based on geometry and view reconstruction between images can be applied. These images e.g. can be from stereo or sequential images or images from mapping data.
  • Using the segmentation information 204 supports creating an efficient learnable mask 203.
  • outliers like occlusion, non-overlapping or around edges are tough regions for the neural network 104 when using a view reconstruction loss function so as to warp a network input into a stereo image, sequential image, or map-relative image.
  • the neural network 104 can be trained to learn further and improve the inlier mask 206 and the learnable mask 203 for producing better depth maps 101 with the help of view reconstruction loss.
  • FIG. 5 In view of FIG. 5, FIG. 6 and FIG. 7, various ways of training the neural network 104 are now described.
  • FIG. 5 illustrates using virtual images 109 (i.e. a virtual dataset) for training with depth maps 101 and semantic segmentation 103, e.g. while using real video (i.e. sequential images) to run a self- supervised approach (not illustrated in FIG. 5).
  • the main benefits of this approach is a continuous learning process. While the virtual dataset is always available for training, the real video is available on the fly to further improve the training. Thereby, depth or segmentation models can be enriched with precise virtual labels.
  • determination results of the device 100 are improved based on real images 108 taking learning results of virtual images as a base for using the third and fourth discriminators 301, 302.
  • FIG. 5 in particular shows the active components of the device 100 for training based on virtual images 109.
  • the virtual images 109 can produce labels such as depth maps 101 and semantic segmentation 103, they are used for training the neural network 104 with the loss function 110.
  • FIG. 6 illustrates a network architecture of the device 100 for training using sequential real images 108.
  • the active components for training real video (sequential real images 108) are shown.
  • a self- supervised approach is used (based on a view reconstruction loss function) to compute depth maps 101 and a relative-pose 102 (odometry) and additionally, the estimated semantic segmentation 103 is used (only forward propagation) to improve the inlier mask 206 and/or the learnable mask 203.
  • FIG. 7 illustrates a network architecture of the device 100 for training the neural network 104 based on a single real image 108 without any loss functions except discriminators to train a model for depth maps and semantic segmentation. More specifically, the network architecture is trained based on real images 108 (dataset) with no depth related loss function. In the illustrated example, the depth maps on real images adapt to produce better results based on the third and fourth discriminator 301, 302.
  • the shown approach allows to use stereo virtual images 109 and stereo real images 108 without any ground truth labels, to estimate a depth map 101 by using a self- supervised approach.
  • the shown approach also allows to use stereo virtual images 109 and sequential real images 108 without any ground truth labels, to estimate depth maps 101 by using a self-supervised approach.
  • FIG. 8 illustrates an inference phase (also called test phase) of the device 100.
  • inference phase also called test phase
  • depth maps 101 and semantic segmentation 103 for a given input image 105 from the real domain are determined.
  • the device 100 also allows to determine a relative pose 102 (which is not shown in FIG. 8).
  • FIG. 9 shows a schematic view of a method 900 according to an embodiment of the present disclosure.
  • the method 900 is for determining a depth map 101, a relative pose 102, or a semantic segmentation 103.
  • the method 900 comprises a step of, in an inference phase, determining 901, by a neural network 104, the depth map 101, the relative pose 102, or the semantic segmentation 103 based on an input image 105.
  • the method 900 further comprises a step of, in a training phase, generating 902, by a generator 106, a synthesized image 107 based on a real image 108 or a virtual image 109, and based on a loss function 110.
  • the method 900 further comprises a step of training 903, by the generator 106, the neural network 104 based on the synthesized image 107; wherein the loss function 110 comprises a semantic edge function 111.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

La présente divulgation se rapporte au domaine des systèmes avancés d'aide à la conduite (ADAS), de la vision artificielle et de l'apprentissage machine (ML). La présente divulgation concerne un moyen basé sur un ML pour entraîner un réseau neuronal sur la base d'une image synthétisée (qui est générée sur la base d'une image réelle ou d'une image virtuelle) pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique. La présente divulgation concerne donc un dispositif (100) pour déterminer une carte de profondeur (101), une pose relative (102) ou une segmentation sémantique (103). Le dispositif (100) comprend un réseau neuronal (104) configuré pour, dans une phase d'inférence, déterminer la carte de profondeur (101), la pose relative (102) ou la segmentation sémantique (103) sur la base d'une image d'entrée (105) ; et un générateur (106) configuré pour, dans une phase d'entraînement, générer une image synthétisée (107) sur la base d'une image réelle (108) ou d'une image virtuelle (109), et sur la base d'une fonction de perte (110), et entraîner le réseau neuronal (104) sur la base de l'image synthétisée (107) ; la fonction de perte (110) comprenant une fonction périphérique sémantique (111).
EP20807341.1A 2020-11-13 2020-11-13 Dispositif et procédé pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique Pending EP4241237A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2020/082042 WO2022100843A1 (fr) 2020-11-13 2020-11-13 Dispositif et procédé pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique

Publications (1)

Publication Number Publication Date
EP4241237A1 true EP4241237A1 (fr) 2023-09-13

Family

ID=73449050

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20807341.1A Pending EP4241237A1 (fr) 2020-11-13 2020-11-13 Dispositif et procédé pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique

Country Status (3)

Country Link
EP (1) EP4241237A1 (fr)
CN (1) CN114793457B (fr)
WO (1) WO2022100843A1 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116342800B (zh) * 2023-02-21 2023-10-24 中国航天员科研训练中心 一种多模态位姿优化的语义三维重建方法及系统
CN117077020B (zh) * 2023-03-21 2026-03-31 周涛 一种基于自监督掩码对比学习的电机故障诊断方法
CN117078524A (zh) * 2023-07-03 2023-11-17 中国银行股份有限公司 图像修复方法、装置、设备、介质和产品
CN117115786B (zh) * 2023-10-23 2024-01-26 青岛哈尔滨工程大学创新发展中心 一种联合分割跟踪的深度估计模型训练方法及使用方法
CN118735979B (zh) * 2024-07-06 2025-01-28 杭州遇渐文化艺术科技有限公司 一种用于艺术作品的虚拟图像生成方法及系统

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111007719B (zh) * 2019-11-12 2022-08-05 杭州电子科技大学 基于领域自适应神经网络的自动驾驶转向角度预测方法
CN110998663B (zh) * 2019-11-22 2023-12-01 驭势（上海）汽车科技有限公司 一种仿真场景的图像生成方法、电子设备和存储介质
CN111008979A (zh) * 2019-12-09 2020-04-14 杭州凌像科技有限公司 一种鲁棒的夜晚图像语义分割方法
CN111210435B (zh) * 2019-12-24 2022-10-18 重庆邮电大学 一种基于局部和全局特征增强模块的图像语义分割方法
CN111553397B (zh) * 2020-04-21 2022-04-29 东南大学 基于区域全卷积网络和自适应的跨域目标检测方法

Also Published As

Publication number Publication date
CN114793457B (zh) 2025-09-12
WO2022100843A1 (fr) 2022-05-19
CN114793457A (zh) 2022-07-26

Similar Documents

Publication Publication Date Title
Chen et al. Geosim: Realistic video simulation via geometry-aware composition for self-driving
EP4241237A1 (fr) Dispositif et procédé pour améliorer la détermination d'une carte de profondeur, d'une pose relative ou d'une segmentation sémantique
CN109003325B (zh) 一种三维重建的方法、介质、装置和计算设备
EP3590070B1 (fr) Apprentissage de la reconstruction de formes 3d en rendu de multiples vues 3d
US10019652B2 (en) Generating a virtual world to assess real-world video analysis performance
CN111127623B (zh) 模型的渲染方法、装置、存储介质及终端
JP2024012657A (ja) クロスリアリティシステムにおけるスケーラブル３次元オブジェクト認識
AU2022231680B2 (en) Techniques for re-aging faces in images and video frames
Wu et al. Emie-map: Large-scale road surface reconstruction based on explicit mesh and implicit encoding
CN110119679A (zh) 物体三维信息估计方法及装置、计算机设备、存储介质
Zhang et al. NDF: Neural deformable fields for dynamic human modelling
Liao et al. Self-supervised random mask attention GAN in tackling pose-invariant face recognition
Li et al. Genrc: Generative 3d room completion from sparse image collections
CN120318392B (zh) 一种基于vr的服装渲染方法及系统
Wegen et al. Non-photorealistic rendering of 3D point clouds for cartographic visualization
Gérard et al. Three dimensional model-based tracking using texture learning and matching
Yang et al. Mapping technology in visual slam: A review
CN119693582A (zh) 三维重建方法、装置、设备以及存储介质
CN113077547B (zh) 生成视线回归红外成像样本数据的方法、装置及存储介质
CN116958393A (zh) 一种增量式图像渲染方法及装置
Johnston et al. Single view 3D point cloud reconstruction using novel view synthesis and self-supervised depth estimation
CN121564161B (zh) 基于场景自适应轨迹与运镜控制的视频生成方法及系统
Tang et al. HR-2DGS: Hybrid regularization for sparse-view 3D reconstruction with 2D Gaussian splatting
Del Gallego Synthetic image generation and the use of virtual environments for image enhancement tasks
Rajendran Point Cloud Fusion with Diffusion Models: An Integrated Pipeline for High-Quality Surround View Rendering

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230606

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: SHENZHEN YINWANG INTELLIGENTTECHNOLOGIES CO., LTD.