WO2025217404A1

WO2025217404A1 - Large scale synthetic data generation for robotic perception applications

Info

Publication number: WO2025217404A1
Application number: PCT/US2025/024076
Authority: WO
Inventors: Ankur Handa; Ritvik Singh; Yu-Wei Chao; Jean-Francois Victor Lafleche; Karl Van Wyk; Jason Liu; Nathan Donald Ratliff
Original assignee: Nvidia Corp
Current assignee: Nvidia Corp
Priority date: 2024-04-12
Filing date: 2025-04-10
Publication date: 2025-10-16
Anticipated expiration: 2026-10-12

Abstract

In various examples, systems and methods are provided for large scale synthetic data generation. A scene may be generated within a simulation environment, and the scene may include one or more surfaces or one or more assets. A simulation may be executed to drop one or more objects using a physics simulator from random positions into the scene to generate a set of dropped objects at physically realistic ending positions relative to the random positions. A set of synthetic images that include at least a portion of the scene and at least a portion of the set of dropped objects may be generated using at least one virtual sensor. In some examples, at least some of one or more properties of the dropped objects or one or more properties of the at least one virtual sensor are different for each synthetic image of the set of synthetic images.

Description

LARGE SCALE SYNTHETIC DATA GENERATION FOR ROBOTIC PERCEPTION APPLICATIONS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63/633,188, filed on April 12, 2024, which is hereby incorporated by reference in its entirety.

BACKGROUND

[0002] Object detection is an important problem in robotics as it serves as the first step in most pose estimation methods. Object detection includes localizing objects of interest in an image with a two-dimensional (2D) bounding shape (e.g., box) and classifying the objects of interest. State estimation for robotics tasks often requires real-time speeds to be used in downstream control policies and should be robust to motion blur, occlusion, varying lighting conditions, and other imaging artifacts. Collecting and annotating real-world data for training state estimation networks may be prohibitively time consuming and expensive, especially for custom assets (e.g., industrial objects). This approach also does not scale well or adequately capture the wide-ranging distribution of possible images.

SUMMARY

[0003] Embodiments of the present disclosure relate to systems and methods for large scale synthetic data generation. Systems and methods are disclosed that may be used for, among other things, generating sets of high-quality synthetic images that may be used to train one or more models for a variety of tasks (e.g., object detection, semantic segmentation, depth estimation, pose estimation, and/or or optical flow estimation).

[0004] In contrast to conventional systems, such as those described above, the systems and methods presented in this disclosure may be used to generate large synthetic data sets using one or more virtual sensors and executing one or more simulations in a relatively short time. A scene of an area (e.g., a room having a floor, walls, and one or more assets) may be procedurally generated within a simulation environment or a scene may include a high dynamic range image (HDRI) background and a transparent enclosure overlaying the HDRI. The simulation(s) may be executed such that virtual objects may be dropped into the scene using a physics simulator from random positions, and synthetic images that include at least a portion of the dropped virtual objects in physically realistic ending positions relative to the random positions may be generated using one or more virtual sensors that capture image data for renderings of the scene from various positions and orientations. The synthetic images may also include at least a portion of the scene (e.g., virtual surface(s) and/or virtual asset(s)). One or more properties of the scene, the dropped virtual objects, and/or the virtual sensor(s) may be adapted (e.g., randomized) during the simulation to produce different synthetic images. The synthetic images may be generated with corresponding groundtruth annotations based at least on semantic label information corresponding to the dropped virtual objects. The synthetic images may also be generated with corresponding depth estimations based at least on positions of the dropped virtual objects, positions of the surface(s) (e.g., the floor and/or walls) and/or positions of the asset(s) (e.g., furniture) in the scene.

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present systems and methods for large scale synthetic data generation are described in detail below with reference to the attached drawing figures, wherein:

[0006] FIG. 1 is an illustration of an example flow diagram for a large scale synthetic data generation operating environment suitable for use in implementing some embodiments of the present disclosure;

[0007] FIG. 2 is an illustration of an example simulation environment suitable for use in implementing some embodiments of the present disclosure;

[0008] FIG. 3 is an example scene for a simulation suitable for use in implementing some embodiments of the present disclosure

[0009] FIG. 4 includes a collection example synthetic images generated using large scale synthetic data generation, in accordance with some embodiments of the present disclosure;

[0010] FIG. 5 includes an example synthetic image generated using large scale synthetic data generation, in accordance with some embodiments of the present disclosure;

[0011] FIG. 6 is an illustration of an example flow diagram for large scale synthetic data generation, in accordance with some embodiments of the present disclosure;

[0012] FIG. 7 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure; and

[0013] FIG. 8 is a block diagram of an example data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] Systems and methods are disclosed related to large scale synthetic data generation, which may be used to produce large amounts of synthetic data for training one or more models (e.g., object detection model(s)). Object detection may generally be broken down into category -level object detection and instance-level object detection. Category-level object detection involves regressing the bounding shape for object categories (e.g., “dog,” “cat,” etc.), and some examples of datasets used for training category-level object detection networks or models include ImageNet and Common Objects in Context (COCO) datasets. Instance-level object detection is more concerned with specific instances of categories, and some examples of datasets used fortraining instance-level object detection models include the Yale-CMU-Berkeley- Video (YCBV) and T-LESS datasets. These datasets are relatively small (on the order of 200K images) and consist of both data collected from the real world and simulation.

[0015] For instance-level object detection, a common benchmark test is the Benchmark for 6D Object Pose Estimation (BOP). The BOP leaderboard is dominated by object detectors that use You Only Look Once Version 8 (YOLOv8) or Fully Convolutional One-Stage (FCOS). These techniques employ single-stage, anchor-free detectors and train on both real images and simulated physically based rendering (PBR) images of the objects in order to achieve performance. However, these techniques use a relatively small number of synthetic images for the YCBV dataset and are only able to run inference at relatively low frequencies. Recently methods are shifting from using Residual Neural Network (ResNet) backbones to larger backbones (e.g., ConvNext), which comes with an increase in mean average precision (mAP) scores but a decrease in inference speed.

[0016] While at least some current object detection models use convolution-based neural networks, the growing popularity of transformers in sequence modelling tasks has led to the rise of transformer-based architectures as well. Detection transformers have recently emerged as promising architectures that can scale to larger dataset sizes. While these detection transformers also make use of a convolutional backbone, they treat object detection as a set prediction problem and are able to avoid using hand-designed modules such as non-maximum suppression or anchor generation. The Real-Time Detection Transformer (RT-DETR) builds upon the detection transformer architecture to make them capable of real-time inference, which is important for robotics settings. [0017] The recent rise of Visual Language Models (VLMs) has also brought about a new type of classification: Open Vocabulary Detection. This is the most general type of detection problem, which attempts to identify and localize objects in an image that were unseen at training time by making use of natural language text. Some examples include Generative Pretrained Transformer (GPT)-4 Vision (GPT-4V), Open-World Localization Vision Transformer (OWL-ViT), Large Language and Vision Assistant (LLaVA), and Geometric Laplacian Eigenmap Embedding (GLEE). While these techniques show ability to generalize to unseen objects in the training distribution, they are fundamentally limited in two ways for robotics usecases: inference speed and robustness for bespoke objects. These VLMs utilize some form of large transformer-like architecture to encode natural language prompts, and as a result, are unable to run at real-time speeds of more than 30Hz. Furthermore, for relatively rare objects that are not as common in internet-scale datasets, VLMs struggle to robustly detect them in videos.

[0018] There have also been techniques proposed for using diffusion models for synthetic data generation. Being able to query a generative model to produce images (e.g., from a text prompt) allows for a virtually unlimited amount of synthetic data, which has been used to great success in tasks of image classification. There has been work using generative models to produce more in-context data augmentations and for self-supervised representation learning methods. There have also been many studies analyzing the scaling laws of synthetic data as compared to real data. However, with all of these above examples, the vision problems do not involve dense annotations. For low-level vision tasks (e.g., detection, segmentation, optical flow, etc.), which are also of interest to roboticists, it is not straightforward to extract the dense annotations required to supervise a model when using diffusion models.

[0019] Much of the state estimation that is used for downstream robotics tasks comes in the form of estimating the pose of objects in the scene. For example, DeXtreme and Learning Dexterity generate millions of synthetic images to train their pose estimators. However, this work is limited to only generating images of a cube with very discernible textures to aid in the pose estimation process. Other works, such as ReorientBot, perform pose estimation on YCB objects to facilitate robust object re-posing. Existing pose estimation methods train on approximately fifty thousand images generated with BlenderProc alongside the real images for the YCBV dataset. Some other techniques generate images to train their pose estimators, but these techniques all rely on using off-the-shelf detectors to provide image crops of objects of interest in order to regress their pose. Moreover, it has been noticed that false or missing detections is what frequently bottlenecks the pose estimation performance. While the aforementioned work does scale synthetic data for object pose estimation, there has been limited work in applying it to object detectors. Additional techniques generate images using renderers to generate a large synthetic dataset for various vision tasks, but these methods have limited diversity of data.

[0020] In contrast to conventional systems, such as those described above, the systems and methods presented in this disclosure may be used to generate large amounts of high-quality synthetic data (e.g., for robotics tasks) in a relatively short time. In some embodiments, a scene of an area (e.g., a room having a floor, wall(s), and one or more assets) may be procedurally generated within a simulation environment. A simulation may be executed wherein virtual objects may be dropped into the scene using a physics simulator from random positions, and synthetic images that include at least a portion of the scene and at least a portion of the dropped virtual objects in physically realistic ending positions relative to the random positions may be generated using one or more virtual sensors that capture image data for renderings of the scene from various positions and orientations. In some embodiments, a scene with a high dynamic range image (HDRI) background and a transparent enclosure overlaying the HDRI background may be generated within a simulation environment. A simulation may be executed wherein virtual objects may be dropped into the transparent enclosure, and synthetic images that include at least a portion of the dropped virtual objects may be generated using one or more virtual sensors that capture image data from various positions and orientations. The synthetic images may be generated using a real-time ray tracing Tenderer (e.g., Omniverse Isaac Sim with Replicator). One or more parameters for the scene, the virtual objects, and/or the virtual sensor(s) may be different for each of the synthetic images, and the different parameters may be produced, for example, using defined ranges and randomization. The synthetic images may also be generated with corresponding groundtruth annotations based at least on semantic label information corresponding to the dropped virtual objects. The synthetic images may also be generated with corresponding depth estimations based at least on positions of the dropped virtual objects, positions of one or more of the surfaces (e.g., the floor and/or wall(s)) and/or positions of the one or more assets in the scene.

[0021] To procedurally generate a scene, the dimensions of a room (e.g., length and width defined by wall(s)) may be set. The width of the room may be set to be a random number in a selected range of values, and the length may be set to be a random scaling factor multiplied times the width. The parameters and range of values for the room may be selected in other ways and based on the particular application for the synthetic images. After the dimensions of the room are set, one or more assets may be placed in the room. For example, one or more pieces of furniture (e.g., table, chairs, couches, lamps, shelving, etc.) may be selected from an asset library (for example, the NVIDIA asset library for Isaac Sim) and placed around the room. A table may be placed in the middle of the room and other furniture may be placed along the wall(s). By including one or more assets in the room rather than just dropping objects in an empty room, the lighting of the objects may be more realistic, particularly when using real-time ray tracing, because the secondary rays produced using ray tracing may provide more realistic indirect lighting and reflections.

[0022] To increase realism of the procedurally generated scene, the asset(s) in the room may be rotated such that the front of the pieces of furniture faces a consistent direction (e.g., toward the center of the room). A transformation may be applied to each of the asset(s) along a particular wall such that a front axis of the asset faces the center of the room and may be parallel to a front axis of the particular wall. To simplify the process, the front axis of each asset may be aligned with an axis (e.g., the x-axis), and the wall(s) may be axis-aligned such that the front of each wall may be aligned with an axis (e.g., the x-axis or the y-axis). Then the assets may be rotated about a different axis (e.g., the z-axis) by 90 degrees, 180 degrees, or 270 degrees, if needed, to align the front axis of the asset with one of the wall(s) of the room in the scene.

[0023] After the scene is procedurally generated with the room and asset(s), a simulation may be executed where virtual objects may be dropped into the room (e.g., on a table) from random overhead positions toward the floor of the room. The simulation may be executed using a physics engine that simulates real physics and physical interactions between the virtual objects and between the virtual objects and the surfaces and/or assets in the room. The simulation may simulate rotation/spin of the dropped virtual objects and/or simulated collisions (collision detection, collision response, etc.) between the dropped virtual objects or between the dropped virtual objects and surfaces or assets of the room. The simulation may include configurable parameters for various forces (e.g., gravity, friction, etc.) that may be selected based on the desired operation of the simulation. The virtual objects that are dropped into the room may interact with each other, the surfaces, and/or the assets of the scene and may settle into a resting position during a portion of the executed simulation. By dropping the virtual objects into the room, the position and/or orientation of the virtual objects may be non-uniform and randomized, which may contribute to better performance of models that are trained using the synthetic images that include the dropped virtual objects. [0024] In some embodiments, rather than procedurally generating a scene of an area as discussed above, a high dynamic range image (HDRI) background and a transparent enclosure may be used. The HDRI background may be selected from a group of available HDRI images and essentially forms a 360-degree dome that shows the HDRI image. A transparent enclosure overlaying the HDRI may also be generated. The transparent enclosure may be invisible such that views of the HDRI background and the virtual objects may not be occluded by the transparent enclosure. The transparent enclosure may have configurable dimensions and may include a ground plane and one or more walls that funnel dropped objects to the ground plane. [0025] After the scene is generated with the HDRI background and transparent enclosure, a simulation may be executed where virtual objects may be dropped into the transparent enclosure from random overhead positions toward the ground plane. The simulation may include a physics engine that simulates real physics and physical interactions between the virtual objects and between the virtual objects and the ground plane and/or wall(s) of the transparent enclosure. The simulation may simulate rotation/spin of the dropped virtual objects and/or simulated collisions (collision detection, collision response, etc.) between the dropped virtual objects or between the dropped virtual objects and ground plane and/or wall(s) of the transparent enclosure. The simulation may include configurable parameters for various forces (e.g., gravity, friction, etc.) that can be selected based on the desired operation of the simulation. The objects that are dropped into the scene may interact with each other, the ground plane, and/or wall(s) and may settle into a resting position during a portion of the executed simulation. By dropping the virtual objects into the transparent enclosure, the position and/or orientation of the virtual objects may be non-uniform and randomized, which may contribute to better performance of the models that are trained using the synthetic images that include the dropped virtual objects.

[0026] The particular obj ects that are dropped in the scene (e.g., procedurally generated room or including an HDRI background) may be selected depending on the requirements of the system. In some embodiments, the objects dropped into the scene include both objects of interest (e.g., objects from a particular dataset desired for object detection) and distractor objectors (e.g., objects not included in the particular dataset desired for object detection), which may help avoid false-positive detections in the real world by adding more variety in the scene and increasing the amount of occlusions on the objects of interest. For example, for synthetic data used to train model(s) for the BOP, the objects of interest may be YCB objects, and the distractor objects may comprise objects from Objaverse, Google Scanned Objects, the NVIDIA asset library, or other objects from a different library or generated using other techniques. One technique for generating additional objects includes scanning an object using a 3D scanning application (e.g., AR Code Object Capture 3D scanning application) that utilizes a LiDAR sensor (e.g., in a smartphone).

[0027] Synthetic images that include at least some of the scene and at least some of the dropped objects may be generated using one or more virtual sensors (e.g., virtual red green blue (RGB) sensor(s), virtual Infrared (IR) sensor(s), virtual RGB-IR sensor(s), virtual depth sensor(s), virtual camera(s), and/or other virtual optical sensor(s)). The virtual sensor(s) may capture images of the scene at various times during the execution of the simulation when the objects are dropped and after the dropped objects have settled in the scene. The virtual sensor(s) may be located at various positions of the scene and have various orientations. In order to generate different synthetic images, one or more properties for the scene, the dropped objects, and/or the virtual sensor(s) may be different for each of the synthetic images. The properties of the scene may include, but are not limited to, lighting parameters and/or color parameters. The properties of the dropped objects may include, but are not limited to, albedo desaturation, albedo add, albedo brightness, diffuse tint, reflection roughness constant, metallic constant, specular level, and/or emissive color. The properties of the virtual sensor(s) may include, but are not limited to, position, orientation, and/or focal point. In some embodiments, the ranges for the one or more properties of the scene, the dropped objects, and/or the virtual sensor(s) may be defined and then the one or more properties of the scene, the dropped objects, and/or the virtual sensor(s) may be randomized for each of the different synthetic images that may be generated.

[0028] After a selected number of synthetic images (e.g., 3000 images) are generated using a particular scene (e.g., a particular room or HDRI background), a new scene may be generated, and additional synthetic images may be generated using the processes described above. The particular number of synthetic images generated using a particular scene and the number of scenes used for generating the synthetic images may be configurable based on the time and resources available and the requirements of the system.

[0029] In addition to modifying the one or more properties of scene, the dropped objects, and/or the virtual sensor(s) when generating the synthetic images as discussed herein, data augmentations may also be applied with varying probabilities (e.g., during training) to the synthetic images captured using the virtual sensor(s) in order to increase the variety of the set of synthetic images. The data augmentations applied may relate to, for example, color (e.g., contrast, brightness, and/or enhancement), background, blending, compression (e.g., JPEG), shot noise, specular highlights, reflectance, perspective, amplitude spectrum, and/or jittering. The data augmentations may be applied to mimic changes in lighting or other conditions without having to render new synthetic images.

[0030] The techniques described herein that include procedurally generating scenes or using HDRI backgrounds enable generating high-quality synthetic images and data more quickly compared to conventional techniques, which allows for better scaling of synthetic data generation. The synthetic images and data generated using the techniques described herein may be used to train one or more models for object detection, and object detection models trained using synthetic images generated using the techniques described herein have been shown to provide better performance and faster inference speed for benchmark testing compared to conventional techniques.

[0031] While embodiments presented in this disclosure may be applicable to generating synthetic data for training object detection models, it should be understood that embodiments may be applicable to generating synthetic data for training models for other types of tasks (e.g., semantic segmentation, depth estimation, pose estimation, and/or or optical flow estimation). Further, while the primary examples described herein utilize datasets that include household objects (e.g., YCBV dataset), it should be understood that embodiments may utilize different types of objects (e.g., industrial objects) depending on the requirements of the system.

[0032] With reference to FIG. 1, FIG. 1 is an example data flow diagram 100 illustrating the interconnection of components and flow of information or data for a synthetic data generation operating environment, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.

[0033] As shown in FIG. 1, the operating environment comprises a synthetic data generation system 110 that may be used to generate synthetic image(s) 116. The operating environment may also comprise a data augmenter 118 that may be used to generate augmented synthetic image(s) 120. The synthetic image(s) 116 output by the synthetic data generation system 1 10 and/or the augmented synthetic image(s) 120 output by the augmented synthetic image(s) 120 may be combined and provided as part of a set of synthetic images 122. The set of synthetic images 122 may be stored in a data store 124 for subsequent use by another process. For example, the set of synthetic images 122 may be retrieved from the data store 124 by a machine learning model training system 126, which may be used for training one or more models for a variety of different applications (object detection, pose estimation, etc.).

[0034] Input channels to the synthetic data generation system 110 may include a scene profile 102 and simulation parameters 108. The input channels may be provided to the synthetic data generation system 110, for example, by a user via one or more user interfaces (e.g., I/O component s) 714). The synthetic data generation system 110 may also receive the input channels from one or more algorithms (e.g., machine learning model(s) trained to generate scenes associated with one or more use cases (e.g., object detection)). The scene profile 102 and/or simulation parameters 108 for the input channels may be selected (e.g., by user(s) and/or algorithm(s)) based on the application for the set of synthetic images 122 to be generated using the synthetic data generation system 110.

[0035] The scene profile 102 may define characteristics for features of the scene to be generated using the synthetic data generation system 110. For example, the scene profile 102 may define dimensions of the scene (e.g., room), which may include a length, a width, and/or a height of the scene. These dimensions may correspond to one or more surfaces 103 (e.g., floor, ground plane, walls, ceiling, etc.) of the scene that may be defined by the scene profile 102. For example, the length and width of the scene may be defined as a distance between particular walls, and the height of the scene may be defined as a distance between a floor and a ceiling. In some embodiments, the width of the scene may be set to be a random number in a selected range of values, and the length and height may be set to be a random scaling factor times the width. The dimensions may also be set independently of each other. The parameters and range of values for the dimensions of the scene may be selected based on the particular application for the set of synthetic images 122 to be generated using the synthetic data generation system 110.

[0036] The scene profile 102 may also define characteristics of one or more assets 104 of the scene. The asset(s) 104 may include one or more pieces of furniture, one or more light sources (e.g., table lamps, floor lamps, ceiling lights, etc.), shelving, etc. For example, random furniture (e.g., table, chairs, couches, lamps, shelving, etc.) may be selected from an asset library (for example, the NVIDIA asset library for Isaac Sim) and placed around the room. In some embodiments, a table may be placed in the middle of the room and other furniture may be placed along the walls. In some embodiments, one or more of the asset(s) 104 may be rotated such that the front of the asset(s) 104 face a consistent direction (e.g., toward the center of the room). A transformation may be applied to each of the asset(s) 104 along a wall such that a front axis of the asset(s) 104 faces the center of the room and may be parallel to a front axis of the wall. To simplify this process, the front axis of the asset(s) 104 may be aligned with an axis (e.g., the x-axis) and the walls may be axis-aligned such that the front of each wall may be aligned with an axis (e.g., the x-axis or the y-axis). The asset(s) 104 may be rotated about a different axis (e.g., the z-axis) by 90 degrees, 180 degrees, or 270 degrees, if needed, to align the front axis of the asset(s) 104 with one of the walls of the room in the scene.

[0037] The scene profile 102 may also define characteristics of obj ects 106 to be dropped in the scene during execution of one or more simulations. The objects 106 may include objects of interest, which may comprise objects from one or more datasets desired for a particular application (e.g., object detection). In some embodiments, the objects 106 may also include one or more distractor objects, which may comprise objects not included in the particular dataset desired for a particular application (e.g., object detection). Where the set of synthetic images 122 may be used to train one or more models for BOP object pose estimation, the objects of interest could be YCB objects and the distractor objects could be from Objaverse, Google Scanned Objects, the NVIDIA asset library, or generated using other techniques. One technique for generating additional objects may include scanning an object using a 3D scanning application (e.g., AR Code Object Capture 3D scanning application) that utilizes a LiDAR sensor (e.g., in a smartphone).

[0038] The scene profile 102 may also define characteristics of one or more virtual sensors 107 that may capture images of the objects 106 in the scene during execution of one or more simulations. The characteristics of the sensor(s) 107 may include, but are not limited to, a sensor type, intrinsic and/or extrinsic parameters, initial position, and/or initial orientation. The type(s) of sensor(s) 107 may include without limitation RGB sensor(s), IR sensor(s), depth sensor(s), camera(s), and/or other optical sensor(s). The sensor(s) 107 may capture sensor data (e.g., image data) of the scene.

[0039] The scene profile 102 (including the characteristics for the surface(s) 103, the asset(s) 104, the objects 106, and/or the virtual sensor(s) 107) may be selected based on the application for the set of synthetic images 122 to be generated using the synthetic data generation system 110. The application for the set of synthetic images 122 to be generated using the synthetic data generation system 110 may include, for example, training an object detection model for a particular setting (e.g., household, industrial, etc.). The scene profile 102 selected may correspond to the features likely to be encountered in the particular setting.

[0040] The simulation parameters 108 provided to the synthetic data generation system 110 may include operating parameters relevant to structuring and performing the simulation itself, such as but not limited to, the total duration of the time to simulate during a simulation, the number of time-steps to include in the simulation, the duration of the simulation, a simulation iteration frequency (e.g., the number of simulation iterations per second) used during each time-step, and/or other parameters. The simulation parameters 108 may also include configurable parameters for various forces (e.g., gravity, friction, etc.) that may be simulated by the synthetic data generation system 110. In some embodiments, the simulation parameters 108 may also include defined ranges for properties of the surface(s) 103, the asset(s) 104, the objects 106, and/or the virtual sensor(s) 107 that may be adapted (e.g., randomized) for simulation. In some embodiments, the ranges for the properties may be defined by a minimum value, a maximum value, and/or a randomization distribution for the simulation.

[0041] FIG. 2 is a block diagram of an example simulation environment 200, in accordance with some embodiments of the present disclosure. The simulation environment 200 and components thereof may be generated using the synthetic data generation system 110 based on the scene profile 102, the asset(s) 104, the objects 106, and the simulation parameters 108. The simulation environment 200 may include one or more virtual surfaces 202, one or more virtual assets 204, one or more virtual objects 206, and one or more virtual sensors 212. The synthetic data generation system 110 may execute one or more simulations within the simulation environment 200 in order to generate the synthetic image(s) 116.

[0042] The virtual surface(s) 202 may be generated in the simulation environment 200 based on the characteristics of the surface(s) 103 in the scene profile 102 provided to the synthetic data generation system 110. Where the scenes are procedurally generated, the virtual surface(s) 202 may include, for example, a floor, one or more walls, and/or a ceiling of a room. Where an HDRI background is part of the scene, the virtual surface(s) 202 may include a transparent enclosure overlaying the HDRI background. For example, the transparent enclosure may have configurable dimensions and may include a ground plane and walls that funnel the virtual objects 206 to the ground plane during execution of the one or more simulations. [0043] The virtual asset(s) 204 may be generated in the simulation environment 200 based on the characteristics of the asset(s) 104 in the scene profile 102 provided to the synthetic data generation system 110. Where the scenes are procedurally generated, the virtual asset(s) 204 may include, for example, a piece of furniture (e.g., a table or sofa), a light source (e.g., a lamp), or other types of assets applicable to the application for the set of synthetic images 122. Where an HDRI background is part of the scene, the virtual asset(s) 204 may be omitted.

[0044] The virtual objects 206 may be generated in the simulation environment 200 based on the characteristics of the objects 106 in the scene profile 102 provided to the synthetic data generation system 110. In some embodiments, the virtual objects 206 may include the object(s) of interest 208 and the distractor object(s) 210. The object(s) of interest 208 may include objects from a particular dataset desired for a particular application, and the distractor obj ect(s) 210 may include obj ects not included in the particular dataset desired for the particular application. Duplicates of one or more virtual objects 206 may be generated and used for the simulation, if desired.

[0045] The virtual sensor(s) 212 may be generated in the simulation environment 200 based on the characteristics of the sensor(s) 107 in the scene profile 102 provided to the synthetic data generation system 110. In some embodiments, the virtual sensor(s) 212 may be generated in the simulation environment 200 based on a different input channel provided to the synthetic data generation system 110. The virtual sensor(s) 212 may include without limitation virtual RGB sensor(s), virtual IR sensor(s), virtual depth sensor(s), virtual camera(s), and/or other virtual optical sensor(s). The virtual sensor(s) 212 may capture virtual sensor data (e.g., image data) that include at least some of the virtual objects 206 and at least some of the scene from one or more positions and at one or more orientations.

[0046] The synthetic data generation system 110 may execute one or more simulations in the simulation environment 200. In some embodiments, the synthetic data generation system 110 may implement one or more light transport algorithms 112 and one or more physics engines 114 when rendering the scene (e.g., including the virtual surface(s) 202, the virtual asset(s) 204, the virtual objects 206, and/or the virtual sensor(s) 212) and/or executing the one or more simulations in the simulation environment 200. The synthetic data generation system 110 may utilize the light transport algorithm(s) 112 to render the scene based on ray-tracing and/or path-tracing. The synthetic data generation system 110 may utilize the physics engine(s) 114 to simulate real physics and physical interactions during the execution of the one or more simulations. [0047] The one or more simulations may include dropping the virtual objects 206 into the scene and simulating real physics and physical interactions (e.g., based on the simulation parameters 108). Each of the virtual objects 206 may start at a height and be dropped during execution of the one or more simulations. For example, each of the virtual objects 206 may be placed at an initial height and then dropped (e.g., according to the simulation parameters 108) during execution of the one or more simulations within the simulation environment 200. The simulated physics and interactions may include the virtual objects 206 falling at a particular rate toward at least one of the virtual surface(s) 202 according to a gravitational constant and other parameters included in the simulation parameters 108 provided to the synthetic data generation system 110. The simulated interactions may include collisions between two or more of the virtual objects 206 or collisions between at least one of the virtual objects 206 and the virtual surface(s) 202 and/or virtual asset(s) 204.

[0048] During the execution of the simulation, the synthetic data generation system 110 may generate the synthetic image(s) 116 using the virtual sensor(s) 212. The synthetic image(s) 116 include at least a portion of the scene generated in the simulation environment 200 (e.g., including at least a portion of the virtual surface(s) 202 and/or the virtual asset(s) 204) and at least a portion of the virtual objects 206 (e.g., the object(s) of interest 208 and/or the distractor object(s) 210) that are dropped into the scene generated in the simulation environment 200. The synthetic image(s) 116 may be generated using one or more virtual sensors 212 that may be included in the scene generated in the simulation environment 200. The synthetic image(s) 116 may be generated while the virtual objects 206 (e.g., the object(s) of interest 208 and/or the distractor object(s) 210) that are dropped into the scene generated in the simulation environment 200 are falling, interacting with each other, interacting with the virtual surface(s) 202 and/or the virtual asset(s) 204, and/or at rest after settling over the time-steps of the executed simulation.

[0049] One or more properties of the scene as a whole may be adapted during the executed simulation. In some embodiments, the adaptable properties of the scene may include, but are not limited to, lighting (e.g., ambient lighting) parameters and/or color parameters. The adaptable properties of the scene as a whole may be randomized within a defined range for one or more time-steps of the simulation.

[0050] One or more properties of the virtual surface(s) 202, the virtual asset(s) 204, and/or the virtual objects 206 may also be adapted during the simulation in addition to, or instead of, the one or more properties of the scene as a whole. In some embodiments, the adaptable properties of the virtual asset(s) 204 and the virtual objects 206 may include, but are not limited to, albedo desaturation, albedo add, albedo brightness, diffuse tint, reflection roughness constant, metallic constant, specular level, and/or emissive color. The adaptable properties of the virtual surface(s) 202, the virtual asset(s) 204, and/or the virtual objects 206 may be randomized within a defined range for one or more time-steps of the simulation. In some embodiments, the properties of the virtual surface(s) 202, the virtual asset(s) 204, and/or the virtual objects 206 may be adapted less frequently than the properties of the scene as a whole (e.g., lighting and/or coloring for the scene).

[0051] One or more properties of the 212 may also be adapted during the simulation. In some embodiments, the adaptable properties of the virtual sensor(s) 212 may include, but are not limited to, position, orientation, and/or focal point of the virtual sensor(s) 212. The adaptable properties of the virtual sensor(s) 212 may be randomized within a defined range every one or more time-steps of the simulation. In some embodiments, the focal point of the virtual sensor(s) 212 may be defined to be at one or more predefined locations within the scene, which may help ensure that at least some of the virtual objects 206 may be captured in the sensor data (e.g., image data) generated by the virtual sensor(s) 212.

[0052] In addition to, or instead of, adapting one or more properties of the scene, the virtual surface(s) 202, the virtual asset(s) 204, the virtual objects 206, and/or the virtual sensor(s) 212, post-processing effects may also be applied to the image data captured by the virtual sensor(s) 212. The post-processing effects may include, but are not limited to, TV noise, scan lines, vertical lines, random splotches, film grain (e.g., grain amount, size, and/or color), and/or vignetting.

[0053] When generating the set of synthetic images 122, the simulation executed by the synthetic data generation system 110 may involve multiple drops of the virtual objects 206 in the same scene (e.g., procedurally generated room or HDRI background). For example, the virtual surface(s) 202 and the virtual asset(s) 204 of the scene may be the same while multiple drops of the virtual objects 206 may be performed. In some embodiments, subsets of the virtual objects 206 may be dropped at distinct time-steps of the simulation such that only some of the virtual objects 206 may be dropped at a time. Multiple distinct simulations may also be executed by the synthetic data generation system 110 using the same scene (e.g., procedurally generated room or HDRI background). For example, the virtual surface(s) 202 and the virtual asset(s) 204 of the scene may be the same, but distinct simulations (e.g., including the drop(s) of the virtual objects 206 into the scene and capturing of images with the virtual sensor(s) 212) may be executed by the synthetic data generation system 110.

[0054] After a particular number of time-steps and/or a particular number of simulations are executed by the synthetic data generation system 110 using a particular scene, a different scene may be used for generating further synthetic image(s) 116. Where the scenes are procedurally generated, new room(s) having different characteristics for the virtual surface(s) 202 and/or virtual asset(s) 204 may be used during other simulation(s) that involve dropping the virtual objects 206. Where an HDRI background is part of the scene, new HDRI backgrounds may be used during other simulation(s) that involve dropping the virtual objects 206.

[0055] Referring back to FIG. 1, the synthetic data generation system 110 may output the generated synthetic image(s) 116. In some embodiments, no further modification of the synthetic image(s) 116 is made and the synthetic image(s) 116 generated and output by the synthetic data generation system 110 form at least part of a set of synthetic images 122, which may be used by other systems (e.g., as training data for training one or more machine learning models). The one or more of the synthetic image(s) 116 generated using the synthetic data generation system 110 may be provided to a data augmenter 118, which may apply one or more data augmentations to the one or more synthetic image(s) 116 to generate one or more augmented synthetic image(s) 120. In some embodiments, the data augmenter 118 may apply the data augmentations with varying probabilities (e.g., during training) to increase the variety of the training set. The data augmentations applied by the data augmenter 118 may relate to varying color contrast, color brightness, color enhancement, color intensity, background, blending, JPEG compression, shot noise, specular highlights, reflectance, perspective, amplitude spectrum, and/or jittering for the synthetic image(s) 116. Data augmentations related to other characteristics or features of the synthetic image(s) 116 may also be applied by the data augmenter 118.

[0056] The data augmentations applied by the data augmenter 118 may include, for example, changing the contrast (e.g., color contrast), brightness (e.g., color brightness), and/or color enhancement of the synthetic image(s) 116, which may mimic changes in lighting or other conditions without having to generate additional synthetic image(s) 116 using the synthetic data generation system 110. For the embodiments using an HDRI background, the data augmentations applied by the data augmenter 118 may include pasting a random background on areas of the synthetic image(s) 116 that are not occupied by objects (e.g., using ground truth segmentation), which may help avoid overfitting to the HDRI images. Random blend involves randomly blending the alpha (e.g., transparency) values of the synthetic image(s) 116 with one of the background images with alpha values chosen uniformly (e.g., between 0.05 and 0.12), which may help to simulate reflections on virtual objects 206. The data augmentations applied by the data augmenter 118 may also include adding shot noise and/or white splats (e.g., that resemble snow) to the synthetic image(s) 116, which may make the model (s) trained using the augmented synthetic image(s) 120 in the set of synthetic images 122 more robust to low quality images and specular highlights. The data augmentations applied by the data augmenter 118 may also include multiplying the synthetic image(s) 116 with a random reflectance map, equalizing the distribution of color intensities in the synthetic image(s) 116, applying a homography to the synthetic image(s) 116 to simulate different camera models, changing the variance in the high frequency components of the synthetic image(s) 116 to more closely match the variance of the amplitude spectra of real images, and/or resizing the synthetic image(s) 116.

[0057] The synthetic image(s) 116 output by the synthetic data generation system 110 and/or the augmented synthetic image(s) 120 output by the data augmenter 118 may be combined into a set of synthetic images 122. The set of synthetic images 122 may be stored (e.g., to a data store 124) and/or used, for example, for training one or more machine learning models (e.g., using machine learning model training system 126) or other purposes. The set of synthetic images 122 may be generated with corresponding groundtruth annotations based at least on semantic label information corresponding to the dropped virtual objects. In some embodiments, the semantic label information corresponding to the dropped virtual objects may be known by the synthetic data generation system 110, and the synthetic data generation system 110 automatically generates synthetic image(s) 116 to include groundtruth annotations based on this known information. For example, groundtruth annotations indicating color, shape, material, texture, brand, and/or other characteristics of one or more individual dropped virtual objects may be applied to the synthetic image(s) 116. The set of synthetic images 122 may also be generated with corresponding depth estimations based at least on positions of the dropped virtual objects 206, positions of the virtual surface(s) 202 (e.g., the floor and/or walls) and/or positions of the virtual asset(s) 204 (e.g., furniture) in the scene.

[0058] Now referring to FIG. 3, FIG. 3 is an example scene 300 that may be generated using a synthetic data generation system (e.g., synthetic data generation system 110). The scene 300 in FIG. 3 comprises a room having multiple surfaces and multiple assets. The surfaces in the scene 300 include a floor and multiple walls (of which two are visible in FIG. 3). The assets in the scene 300 include multiple tables, a sofa, pillows, and a lamp. It should be understood that the example scene 300 shown in FIG. 3 is shown from the perspective of a single sensor and there may be additional surface(s), asset(s), and/or virtual sensor(s) distributed throughout the room in the scene 300.

[0059] Now referring to FIG. 4, FIG. 4 depicts a collection 400 of various synthetic images that may be generated using a synthetic data generation system (e.g., synthetic data generation system 110). Each of the synthetic images in the collection 400 shown in FIG. 4 are generated based on the same scene. The position and/or orientation of the virtual sensor used to capture each synthetic image in the collection 400 may be different and the properties of the scene (e.g., ambient lighting and color) and the properties of virtual features (e.g., virtual surfaces, assets, and objects) of the scene may be different, for example, due to adaptation during the simulation where the synthetic images may be obtained using the virtual sensor(s). [0060] Now referring to FIG. 5, FIG. 5 is an example synthetic image 500 that may be generated using a synthetic data generation system (e.g., synthetic data generation system 110). The synthetic image 500 in FIG. 5 comprises multiple virtual objects (e.g., virtual objects 206) that have been dropped into a scene. The virtual objects in the synthetic image 500 include household objects (e.g., YCBV dataset) that have been dropped in a scene with an HDRI background and a transparent enclosure. It should be understood that the synthetic image 500 is shown from the perspective of a single virtual sensor at a particular position and orientation in the scene, but there may be additional objects and/or virtual sensor(s) distributed throughout the scene generated using the synthetic data generation system.

[0061] Now referring to FIG. 6, FIG. 6 is a flow diagram showing a method 600 for generating synthetic images, in accordance with some embodiments of the present disclosure. Each block of method 600, described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The method may also be embodied as computer-usable instructions stored on computer storage media. The method may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, method 600 is described, by way of example, with respect to the system of FIG. 1. However, this method may additionally or alternatively be executed by any one system, or any combination of systems, including, but not limited to, those described herein. The method 600 may be performed for multiple scenes in order to generate a large set of synthetic images. [0062] The method 600, at block B602, includes generating a scene in a simulation environment having one or more virtual surfaces and/or one or more virtual assets. The scene may be generated using a synthetic data generation system (e.g., synthetic data generation system 110) based on input channel(s) that include a scene profile (e.g., scene profile 102). The scene profile may include characteristics (e.g., type, size, position, orientation, etc.) of the virtual surface(s) and/or the virtual asset(s). The scene profile may also include characteristics (e.g., type, initial position, initial orientation, etc.) of virtual objects that may be dropped into the scene. The scene profile may also include characteristics (e.g., type, initial position, initial orientation, etc.) of one or more virtual sensors that may capture images of the scene.

[0063] The generated scene may include a household room (e.g., a living room). The virtual surface(s) for such a scene may include a floor and one or more walls and the virtual asset(s) may include, without limitation, table(s), chair(s), sofa(s), lamp(s), and/or other pieces of furniture or light sources. The virtual surface(s) and/or the virtual asset(s) may be different depending on the intended use case for the synthetic images generated. In some embodiments, the scene includes an HDRI background rather than a procedurally generated room. The virtual surface(s) and/or the virtual asset(s) for such a scene may include an enclosure that includes a ground plane and one or more walls, which may serve to funnel dropped virtual objects toward the ground plane of the enclosure and/or contain the dropped virtual objects. The enclosure may be transparent in order to enable images to be capture using the virtual sensor(s). Other types of scenes may also be generated depending on the use case for the synthetic images.

[0064] The method 600, at block B604, includes executing a simulation in the simulation environment that includes dropping virtual objects into the scene. The simulation may be executed using a synthetic data generation system (e.g., synthetic data generation system 110) based on input channel(s) that include simulation parameters (e.g., simulation parameters 108). The simulation parameters may include operating parameters relevant to structuring and performing the simulation within the simulation environment. For example, the simulation parameters may include, but are not limited to, duration of the time to simulate, the number of time-steps to include in the simulation, the duration of the simulation, a simulation iteration frequency (e.g., the number of simulation iterations per second) used during each time-step, and/or other parameters.

[0065] The virtual objects may be dropped from a height into the scene, and the simulation may be executed using a physics engine that simulates real physics and physical interactions between the virtual objects and between the virtual objects and the virtual surfaces and/or virtual assets in the scene. The simulation may simulate rotation/spin of the dropped virtual objects and/or simulated collisions (collision detection, collision response, etc.) between the dropped virtual objects or between the dropped virtual objects and the virtual surfaces and/or virtual assets in the scene. The simulation may include configurable parameters for various forces (e.g., gravity, friction, etc.) based at least on the simulation parameters provided to the synthetic data generation system.

[0066] The method 600, at block B606, includes capturing one or more images that include at least some of the dropped virtual objects using one or more virtual sensors. The one or more virtual sensors may include, but are not limited to, virtual RGB sensor(s), virtual IR sensor(s), virtual RGB-IR sensor(s), virtual depth sensor(s), virtual camera(s), and/or other virtual optical sensor(s). The virtual sensor(s) may capture images of the scene, for example, at each time-step during execution of the simulation from one or more positions and/or orientations. In some embodiments, the image(s) captured using the virtual sensor(s) during the execution of the simulation may also include at least some of the scene in addition to the at least some dropped virtual objects.

[0067] The method 600, at block B608, includes adapting one or more properties of the scene, one or more properties of the dropped virtual objects, and/or one or more properties of the virtual sensor(s). In some embodiments, the simulation parameters for the simulation may include defined ranges for properties of the surface(s), the asset(s), the objects, and/or the virtual sensor(s) that may be adapted for simulation. These ranges for the properties may be defined, for example, by a minimum value, a maximum value, and/or a randomization distribution for the simulation. The properties of the scene may include, but are not limited to, lighting parameters and/or color parameters. The properties of the surface(s), the asset(s), and the dropped virtual objects may include, but are not limited to, albedo desaturation, albedo add, albedo brightness, diffuse tint, reflection roughness constant, metallic constant, specular level, and/or emissive color. The properties of the virtual sensor(s) may include, but are not limited to, position, orientation, and/or focal point.

[0068] After adapting one or more properties of the scene, one or more properties of the dropped virtual obj ects, and/or one or more properties of the virtual sensor(s) at block B606, the method 600 may proceed to block B606 and additional image(s) may be captured using the virtual sensor(s) with the adapted properties of the scene, dropped objects, and/or virtual sensor(s). Blocks B606 and B608 may be repeated (e.g., every time-step of the executed simulation) to generate a variety of synthetic images using a particular scene. The blocks B606 and B608 may be repeated in order to generate a selected number of synthetic images (e.g., 3000 images) using a particular scene (e.g., a particular room or HDRI background) before a new scene is used. The particular number of synthetic images generated using a particular scene and the number of scenes used for generating the synthetic images may be configurable based on the time and resources available and the requirements of the system.

[0069] The systems and methods described herein may be performed within, or in conjunction with, a simulation environment (e.g., NVIDIA’ s Isaac Sim) using simulated data (e.g., simulated sensor data of simulated sensors of a virtual or simulated scene). For example, simulated movement and/or sensor data may be used to perform operations (e.g., simulate dropping objects) and to generate synthetic image data - e.g., image data including objects of interest or distractor objects from within the simulation. The synthetic image data (in addition to or alternatively from real -world data) may be used to train one or more models for object detection and/or other applications, for example. In any example, the simulation environment 200 and/or associated synthetic image data may be rendered or otherwise generated using one or more light transport algorithms - such as ray-tracing and/or path-tracing algorithms. In some embodiments, the virtual surface(s), the virtual asset(s), the virtual objects, and/or the virtual sensor(s) may be generated or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA’ s Omniverse) for industrial digitalization, generative physical artificial intelligence (Al), and/or other use cases, applications, or services. For example, the content collaboration platform or system may include a system for using or developing universal scene descriptor (USD) (e.g., OpenUSD) data for managing objects, features, scenes, etc., within a simulated environment, digital environment, etc. The platform may include real physics simulation, such as using NVIDIA’ s PhysX SDK, in order to simulate real physics and physical interactions with simulations hosted by the platform. The platform may integrate OpenUSD along with ray tracing/path tracing/light transport simulation (e.g., NVIDIA’s RTX rendering technologies) into software tools and simulation workflows for building, training, deploying, or testing Al systems - such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.), and/or other tasks related to automotive, robot, machine, or other applications.

[0070] In some examples, the machine learning model(s) (e.g., deep neural networks, language models, LLMs, VLMs, multi-modal language models, perception models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder- only models, encoder-decoder models, neural rendering field (NERF) models, etc.) described herein may be packaged as a microservice - such an inference microservice (e.g., NVIDIA NIMs) - which may include a container (e.g., an operating system (OS)-level virtualization package) that may include an application programming interface (API) layer, a server layer, a runtime layer, and/or at least one model “engine.” For example, the inference microservice may include the container itself and the model(s) (e.g., weights and biases). In some instances, such as where the machine learning model(s) is small enough (e.g., has a small enough number of parameters), the model(s) may be included within the container itself. In other examples - such as where the model(s) is large - the model(s) may be hosted/stored in the cloud (e.g., in a data center) and/or may be hosted on-premises and/or at the edge (e.g., on a local server or computing device, but outside of the container). In such embodiments, the model(s) may be accessible via one or more APIs - such as REST APIs. As such, and in some embodiments, the machine learning model(s) described herein may be deployed as an inference microservice to accelerate deployment of a model(s) on any cloud, data center, or edge computing system, while ensuring the data is secure. For example, the inference microservice may include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using a standardized Al model deployment an execution software, such as NVIDIA’s Triton Inference Server, and/or one or more APIs for high performance deep learning inference, which may include an inference runtime and model optimizations that deliver low latency and high throughput for production applications - such as NVIDIA’s TensorRT), and/or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and/or monitoring). The machine learning model(s) described herein may be included as part of the microservice along with an accelerated infrastructure with the ability to deploy with a single command and/or orchestrate and auto-scale with a container orchestration system on accelerated infrastructure (e.g., on a single device up to data center scale). As such, the inference microservice may include the machine learning model(s) (e.g., that has been optimized for high performance inference), an inference runtime software to execute the machine learning model(s) and provide outputs/responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software to provide health checks, identity, and/or other monitoring. In some embodiments, the inference microservice may include software to perform in-place replacement and/or updating to the machine learning model(s). When replacing or updating, the software that performs the replacement/updating may maintain user configurations of the inference runtime software and enterprise management software. [0071] The systems and methods described herein may be used for a variety of purposes, by way of example and without limitation, for machine control, machine locomotion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and/or digital twinning, data center processing, conversational Al, light transport simulation (e.g., ray-tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative Al, and/or any other suitable applications.

[0072] Disclosed embodiments may be comprised in a variety of different systems such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aerial systems, medial systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational Al operations, systems implementing one or more language models - such as one or more large language models (LLMs), systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and/or other types of systems.

[0073] EXAMPLE COMPUTING DEVICE

[0074] FIG. 7 is a block diagram of an example computing device(s) 700 suitable for use in implementing some embodiments of the present disclosure. In some embodiments, one or more functions of the synthetic data generation system 110 described herein may be performed using the computing device 700. Computing device 700 may include an interconnect system 702 that directly or indirectly couples the following devices: memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communication interface 710, input/output (I/O) ports 712, input/output components 714, a power supply 716, one or more presentation components 718 (e.g., display(s)), and one or more logic units 720. In at least one embodiment, the computing device(s) 700 may comprise one or more virtual machines (VMs), and/or any of the components thereof may comprise virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 708 may comprise one or more vGPUs, one or more of the CPUs 706 may comprise one or more vCPUs, and/or one or more of the logic units 720 may comprise one or more virtual logic units. As such, a computing device(s) 700 may include discrete components (e.g., a full GPU dedicated to the computing device 700), virtual components (e.g., a portion of a GPU dedicated to the computing device 700), or a combination thereof.

[0075] Although the various blocks of FIG. 7 are shown as connected via the interconnect system 702 with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 718, such as a display device, may be considered an VO component 714 (e.g., if the display is a touch screen). As another example, the CPUs 706 and/or GPUs 708 may include memory (e.g., the memory 704 may be representative of a storage device in addition to the memory of the GPUs 708, the CPUs 706, and/or other components). As such, the computing device of FIG. 7 is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device of FIG. 7.

[0076] The interconnect system 702 may represent one or more links or busses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 702 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 706 may be directly connected to the memory 704. Further, the CPU 706 may be directly connected to the GPU 708. Where there is direct, or point-to-point connection between components, the interconnect system 702 may include a PCIe link to carry out the connection. In these examples, a PCI bus need not be included in the computing device 700.

[0077] The memory 704 may include any of a variety of computer-readable media. The computer-readable media may be any available media that may be accessed by the computing device 700. The computer-readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, the computer-readable media may comprise computer- storage media and communication media.

[0078] The computer-storage media may include both volatile and nonvolatile media and/or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and/or other data types. For example, the memory 704 may store computer-readable instructions (e.g., that represent a program(s) and/or a program element(s), such as an operating system. Computer-storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information and which may be accessed by computing device 700. As used herein, computer storage media does not comprise signals per se.

[0079] The computer storage media may embody computer-readable instructions, data structures, program modules, and/or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the computer storage media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0080] The CPU(s) 706 may be configured to execute at least some of the computer- readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and/or processes described herein. The CPU(s) 706 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s) 706 may include any type of processor, and may include different types of processors depending on the type of computing device 700 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 700, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 700 may include one or more CPUs 706 in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

[0081] In addition to or alternatively from the CPU(s) 706, the GPU(s) 708 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and/or processes described herein. One or more of the GPU(s) 708 may be an integrated GPU (e.g., with one or more of the CPU(s) 706 and/or one or more of the GPU(s) 708 may be a discrete GPU. In embodiments, one or more of the GPU(s) 708 may be a coprocessor of one or more of the CPU(s) 706. The GPU(s) 708 may be used by the computing device 700 to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s) 708 may be used for General-Purpose computing on GPUs (GPGPU). The GPU(s) 708 may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s) 708 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s) 706 received via a host interface). The GPU(s) 708 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 704. The GPU(s) 708 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 708 may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs. In some embodiments, one or more functions of the synthetic data generation system 110 described herein may be executed, at least in part, by the CPU(s) 706 and/or GPU(s) 708.

[0082] In addition to or alternatively from the CPU(s) 706 and/or the GPU(s) 708, the logic unit(s) 720 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and/or processes described herein. In embodiments, the CPU(s) 706, the GPU(s) 708, and/or the logic unit(s) 720 may discretely or jointly perform any combination of the methods, processes and/or portions thereof. One or more of the logic units 720 may be part of and/or integrated in one or more of the CPU(s) 706 and/or the GPU(s) 708 and/or one or more of the logic units 720 may be discrete components or otherwise external to the CPU(s) 706 and/or the GPU(s) 708. In embodiments, one or more of the logic units 720 may be a coprocessor of one or more of the CPU(s) 706 and/or one or more of the GPU(s) 708. In some embodiments, one or more functions of the synthetic data generation system 110 described herein may be executed, at least in part, by the logic unit(s) 720.

[0083] Examples of the logic unit(s) 720 include one or more processing cores and/or components thereof, such as Data Processing Units (DPUs), Tensor Cores (TCs), Tensor Processing Units(TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Arithmetic-Logic Units (ALUs), Application- Specific Integrated Circuits (ASICs), Floating Point Units (FPUs), input/output (I/O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and/or the like.

[0084] The communication interface 710 may include one or more receivers, transmitters, and/or transceivers that allow the computing device 700 to communicate with other computing devices via an electronic communication network, included wired and/or wireless communications. The communication interface 710 may include components and functionality to allow communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet. In one or more embodiments, logic unit(s) 720 and/or communication interface 710 may include one or more data processing units (DPUs) to transmit data received over a network and/or through interconnect system 702 directly to (e.g., a memory of) one or more GPU(s) 708.

[0085] The I/O ports 712 may allow the computing device 700 to be logically coupled to other devices including the I/O components 714, the presentation component(s) 718, and/or other components, some of which may be built in to (e.g., integrated in) the computing device 700. Illustrative I/O components 714 include a microphone, mouse, keyboardjoystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components 714 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 700. The computing device 700 may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device 700 may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that allow detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device 700 to render immersive augmented reality or virtual reality. In some embodiments, the computing device 700 may include and/or be used in conjunction with the synthetic data generation system 110 described herein.

[0086] The power supply 716 may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 716 may provide power to the computing device 700 to allow the components of the computing device 700 to operate.

[0087] The presentation component(s) 718 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component s) 718 may receive data from other components (e.g., the GPU(s) 708, the CPU(s) 706, DPUs, etc.), and output the data (e.g., as an image, video, sound, etc.).

[0088] EXAMPLE DATA CENTER

[0089] FIG. 8 illustrates an example data center 800 that may be used in at least one embodiments of the present disclosure. The data center 800 may include a data center infrastructure layer 810, a framework layer 820, a software layer 830, and/or an application layer 840.

[0090] As shown in FIG. 8, the data center infrastructure layer 810 may include a resource orchestrator 812, grouped computing resources 814, and node computing resources (“node C.R.s”) 816(l)-816(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s 816(l)-816(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (NW I/O) devices, network switches, virtual machines (VMs), power modules, and/or cooling modules, etc. In some embodiments, one or more node C.R.s from among node C.R.s 816(1)-816(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node C.R.s 816(1)-8161(N) may include one or more virtual components, such as vGPUs, vCPUs, and/or the like, and/or one or more of the node C.R.s 816(1)-816(N) may correspond to a virtual machine (VM). In some embodiments, one or more functions of the synthetic data generation system 110 described herein may be implemented, at least in part, using one or more of the node C.R.s 816(1)-8161(N).

[0091] In at least one embodiment, grouped computing resources 814 may include separate groupings of node C.R.s 816 housed within one or more racks (not shown), or many racks housed in data centers at various geographical locations (also not shown). Separate groupings of node C.R.s 816 within grouped computing resources 814 may include grouped compute, network, memory or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 816 including CPUs, GPUs, DPUs, and/or other processors may be grouped within one or more racks to provide compute resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and/or network switches, in any combination. [0092] The resource orchestrator 812 may configure or otherwise control one or more node C.R.s 816(1)-816(N) and/or grouped computing resources 814. In at least one embodiment, resource orchestrator 812 may include a software design infrastructure (SDI) management entity for the data center 800. The resource orchestrator 812 may include hardware, software, or some combination thereof.

[0093] In at least one embodiment, as shown in FIG. 8, framework layer 820 may include a job scheduler 828, a configuration manager 834, a resource manager 836, and/or a distributed file system 838. The framework layer 820 may include a framework to support software 832 of software layer 830 and/or one or more application(s) 842 of application layer 840. The software 832 or application(s) 842 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. The framework layer 820 may be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that may use distributed file system 838 for large-scale data processing (e.g., "big data"). In at least one embodimentjob scheduler 828 may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center 800. The configuration manager 834 may be capable of configuring different layers such as software layer 830 and framework layer 820 including Spark and distributed file system 838 for supporting large-scale data processing. The resource manager 836 may be capable of managing clustered or grouped computing resources mapped to or allocated for support of distributed file system 838 and job scheduler 828. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource 814 at data center infrastructure layer 810. The resource manager 836 may coordinate with resource orchestrator 812 to manage these mapped or allocated computing resources.

[0094] In at least one embodiment, software 832 included in software layer 830 may include software used by at least portions of node C.R.s 816(1)-816(N), grouped computing resources 814, and/or distributed file system 838 of framework layer 820. One or more types of software may include, but are not limited to, Internet web page search software, e-mail virus scan software, database software, and streaming video content software.

[0095] In at least one embodiment, application(s) 842 included in application layer 840 may include one or more types of applications used by at least portions of node C.R.s 816(1)- 816(N), grouped computing resources 814, and/or distributed file system 838 of framework layer 820. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive compute, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or other machine learning applications used in conjunction with one or more embodiments.

[0096] In at least one embodiment, any of configuration manager 834, resource manager 836, and resource orchestrator 812 may implement any number and type of selfmodifying actions based on any amount and type of data acquired in any technically feasible fashion. Self-modifying actions may relieve a data center operator of data center 800 from making possibly bad configuration decisions and possibly avoiding underutilized and/or poor performing portions of a data center.

[0097] The data center 800 may include tools, services, software or other resources to train one or more machine learning models or predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model(s) may be trained by calculating weight parameters according to a neural network architecture using software and/or computing resources described above with respect to the data center 800. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to the data center 800 by using weight parameters calculated through one or more training techniques, such as but not limited to those described herein.

[0098] In at least one embodiment, the data center 800 may use CPUs, applicationspecific integrated circuits (ASICs), GPUs, FPGAs, and/or other hardware (or virtual compute resources corresponding thereto) to perform training and/or inferencing using above-described resources. Moreover, one or more software and/or hardware resources described above may be configured as a service to allow users to train or performing inferencing of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0099] EXAMPLE NETWORK ENVIRONMENTS

[0100] Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other device types. The client devices, servers, and/or other device types (e.g., each device) may be implemented on one or more instances of the computing device(s) 700 of FIG. 7 - e.g., each device may include similar components, features, and/or functionality of the computing device(s) 700. In addition, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center 800, an example of which is described in more detail herein with respect to FIG. 8.

[0101] Components of a network environment may communicate with each other via a network(s), which may be wired, wireless, or both. The network may include multiple networks, or a network of networks. By way of example, the network may include one or more Wide Area Networks (WANs), one or more Local Area Networks (LANs), one or more public networks such as the Internet and/or a public switched telephone network (PSTN), and/or one or more private networks. Where the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) may provide wireless connectivity.

[0102] Compatible network environments may include one or more peer-to-peer network environments - in which case a server may not be included in a network environment - and one or more client-server network environments - in which case one or more servers may be included in a network environment. In peer-to-peer network environments, functionality described herein with respect to a server(s) may be implemented on any number of client devices. [0103] In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of servers, which may include one or more core network servers and/or edge servers. A framework layer may include a framework to support software of a software layer and/or one or more application(s) of an application layer. The software or application(s) may respectively include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (e.g., by accessing the service software and/or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework such as that may use a distributed file system for large-scale data processing (e.g., "big data").

[0104] A cloud-based network environment may provide cloud computing and/or cloud storage that carries out any combination of computing and/or data storage functions described herein (or one or more portions thereof). Any of these various functions may be distributed over multiple locations from central or core servers (e.g., of one or more data centers that may be distributed across a state, a region, a country, the globe, etc.). If a connection to a user (e.g., a client device) is relatively close to an edge server(s), a core server(s) may designate at least a portion of the functionality to the edge server(s). A cloudbased network environment may be private (e.g., limited to a single organization), may be public (e.g., available to many organizations), and/or a combination thereof (e.g., a hybrid cloud environment).

[0105] The client device(s) may include at least some of the components, features, and functionality of the example computing device(s) 700 described herein with respect to FIG. 7. By way of example and not limitation, a client device may be embodied as a Personal Computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a Personal Digital Assistant (PDA), an MP3 player, a virtual reality headset, a Global Positioning System (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a flying vessel, a virtual machine, a drone, a robot, a handheld communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these delineated devices, or any other suitable device.

[0106] The disclosure may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types. The disclosure may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialty computing devices, etc. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network.

[0107] As used herein, a recitation of “and/or” with respect to two or more elements should be interpreted to mean only one element, or a combination of elements. For example, “element A, element B, and/or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0108] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and/or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

Claims

CLAIMS What is claimed is:

1. One or more processors comprising processing circuitry to: generate a first scene within a simulation environment, wherein the first scene comprises one or more surfaces or one or more assets; execute a simulation to drop one or more objects using a physics simulator from random positions into the first scene to generate a set of dropped objects in physically realistic ending positions relative to the random positions; and generate, using at least one virtual sensor, a set of synthetic images that include at least a portion of the first scene and at least a portion of the set of dropped objects.

2. The one or more processors of claim 1 , wherein the first scene comprises a room having a floor, walls, and one or more pieces of furniture, wherein the processing circuitry is to execute the simulation to drop the one or more objects from random positions into the room.

3. The one or more processors of claim 2, wherein the processing circuitry is further to generate depth estimation information based at least on positions of the set of dropped objects and positions of the floor, the walls, and the one or more pieces of furniture.

4. The one or more processors of claim 1, wherein the first scene comprises a high dynamic range image (HDRI) background and a transparent enclosure overlaying the HDRI background, wherein the processing circuitry is to execute the simulation to drop the one or more objects from random positions into the transparent enclosure.

5. The one or more processors of claim 1, wherein the processing circuitry is to generate the set of synthetic images by randomizing at least some of one or more properties of the first scene, one or more properties of dropped objects in the set of dropped objects, or one or more properties of at least one virtual sensor for each synthetic image in the set of synthetic images.

6. The one or more processors of claim 5, wherein the one or more properties of dropped objects in the set of dropped objects include one or more of albedo desaturation, albedo add, albedo brightness, diffuse tint, reflection roughness constant, metallic constant, specular level, or emissive color.

7. The one or more processors of claim 5, wherein the one or more properties of the at least one virtual sensor includes one or more of position, orientation, or focal point.

8. The one or more processors of claim 5, wherein the one or more properties of the first scene include one or more of lighting parameters or color parameters for the first scene.

9. The one or more processors of claim 1, wherein the processing circuitry is further to apply one or more data augmentations to the set of synthetic images to generate augmented synthetic images.

10. The one or more processors of claim 1, wherein the processing circuitry is to generate the first scene using a real-time ray-tracing Tenderer.

11. The one or more processors of claim 1, wherein each dropped object in the set of dropped objects has corresponding semantic label information, wherein the processing circuitry is further to generate groundtruth annotations for the set of synthetic images based at least on the corresponding semantic label information.

12. The one or more processors of claim 1, wherein the processing circuitry is further to: generate a second scene within the simulation environment; execute a second simulation to drop one or more second objects from random positions into the second scene to generate a second set of dropped objects; and generate, using at least one virtual sensor, a second set of synthetic images that include at least a portion of the second scene and at least a portion of the second set of dropped objects.

13. The one or more processors of claim 1, wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational Al operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system for generating synthetic data; a system for generating synthetic data using Al; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

14. A system comprising one or more processors to: execute a simulation to drop one or more virtual objects using a physics simulator from random positions into a scene in a simulation environment to generate a set of dropped virtual objects at physically realistic ending positions relative to the random positions; and generate, using at least one virtual sensor, a set of synthetic images that include at least a portion of the scene and at least a portion of the set of dropped virtual objects, wherein at least some of one or more properties for the scene, one or more properties for dropped virtual objects in the set of dropped virtual objects, or one or more properties for the at least one virtual sensor are different between each of two or more synthetic images in the set of synthetic images.

15. The system of claim 14, wherein the scene comprises a room having a floor, walls, and one or more pieces of furniture, wherein the one or more processors are to execute the simulation to drop the one or more virtual objects from random positions into the room.

16. The system of claim 14, wherein the scene comprises a high dynamic range image (HDRI) background and a transparent enclosure overlaying the HDRI background, wherein the one or more processors are to execute the simulation to drop the one or more virtual objects from random positions into the transparent enclosure.

17. The system of claim 14, wherein the simulation includes at least one of: rotation of the one or more virtual objects, spin of the one or more virtual objects, or simulated collisions based at least on interactions between the one or more virtual objects or between the one or more virtual objects and one or more surfaces or assets of the scene.

18. The system of claim 14, wherein each dropped virtual object in the set of dropped virtual objects has corresponding semantic label information, wherein the one or more processors are further to generate groundtruth annotations for the set of synthetic images based at least on the corresponding semantic label information.

19. The system of claim 14, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational Al operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system for generating synthetic data; a system for generating synthetic data using Al; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

20. A method comprising: generating a set of synthetic images using at least one virtual sensor, wherein the set of synthetic images include objects dropped from random positions into physically realistic ending positions in a generated scene using a physics simulator, the scene having one or more surfaces or one or more assets, wherein at least some of one or more properties of the dropped objects or one or more properties of the at least one virtual sensor are different for each synthetic image of the set of synthetic images.