JP2022553846A - Generating Arbitrary Views - Google Patents

Abstract

A machine learning based image processing and generation framework is disclosed. In some embodiments, depth values of objects or assets in received input images are determined at least in part using a machine learning-based framework that is constrained to a known predetermined environment. The determined depth value facilitates generating other views of the object or asset. [Selection drawing] Fig. 10

Description

CROSS- REFERENCES TO OTHER APPLICATIONS This application is a continuation-in-part of U.S. patent application Ser. No. 16/181,607, filed November 6, 2018, entitled "ARBITRARY VIEW GENERATION," which is a continuation-in-part of September 2017, entitled "ARBITRARY VIEW GENERATION." No. 15/721,426 (now U.S. Patent No. 10,163,250), filed May 29, 2017, entitled "FAST RENDERING OF ASSEMBLED SCENES." U.S. Patent Application No. 15/081,553, filed March 25, 2016, entitled "ARBITRARY VIEW GENERATION," claiming priority to U.S. Provisional Patent Application No. 62/541,607, filed Aug. 4; (now US Pat. No. 9,996,914), all of which are incorporated herein by reference for all purposes.

No. 62/933,258, filed November 8, 2019, entitled "FAST RENDERING OF IMAGE SEQUENCES FOR PRODUCT VISUALIZATION" and "SYSTEM AND METHOD FOR ACQUIRING IMAGES FOR PLANNING PLANNING". No. 62/933,261, filed November 8, 2019, entitled, U.S. Provisional Patent Application No. 62/933,261, both of which are incorporated herein by reference for all purposes.

Existing rendering techniques face trade-offs between the competing goals of quality and speed. High quality rendering requires significant processing resources and time. However, slow rendering techniques are unacceptable for many applications, such as interactive real-time applications. In general, lower quality but faster rendering techniques are preferred for such applications. For example, rasterization is commonly utilized by real-time graphics applications, sacrificing quality for relatively fast rendering. Therefore, there is a need for an improved technique that does not significantly impair quality or speed.

Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

1 is a high-level block diagram illustrating one embodiment of a system for generating arbitrary views of a scene; FIG.

The figure which shows an example of a database asset.

4 is a flowchart illustrating one embodiment of a process for generating arbitrary perspectives;

4 is a flowchart illustrating one embodiment of a process for generating a reference image or view of an asset from which arbitrary views of the asset may be generated.

4 is a flowchart illustrating one embodiment of a process for generating a reference image or view of an asset from which arbitrary views of the asset may be generated.

4 is a flow chart illustrating one embodiment of a process for providing a requested view of a scene.

1 is a high-level block diagram illustrating one embodiment of a machine learning-based image processing framework for learning attributes associated with image datasets; FIG.

4 is a flowchart illustrating one embodiment of a process for entering images associated with an asset into a database that can be used to generate other arbitrary views of the asset.

4 is a flow chart illustrating one embodiment of a process for generating an image or frame;

1 is a high-level flowchart illustrating one embodiment of a process for generating arbitrary or new views or perspectives of objects or assets.

The present invention may be a process, apparatus, system, composition of matter, computer program product embodied on a computer readable storage medium, and/or a processor (stored in and/or by a memory coupled to the processor). implemented in various forms, including a processor configured to execute the provided instructions). In this specification, these examples, or any other form that the invention may take, may be referred to as techniques. In general, the order of steps in disclosed processes may be altered within the scope of the invention. Unless otherwise stated, a component such as a processor or memory described as being configured to perform a task is a general component temporarily configured to perform the task at a time; , may be implemented as specific components manufactured to perform a task. As used herein, the term "processor" shall refer to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention includes many alternatives, modifications and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. These details are for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the sake of brevity, technical material that is well known in the technical fields related to the invention has not been described in detail so as not to unnecessarily obscure the invention.

Techniques are disclosed for generating arbitrary views of a scene. The examples described herein also provide high definition output with very low processing or computational overhead, effectively eliminating difficult tradeoffs between rendering speed and quality. The disclosed techniques are particularly useful for producing high quality output very quickly for interactive real-time graphics applications. Such applications rely on the substantially immediate presentation of desirable high quality output in response to and in accordance with user manipulation of a presented interactive view or scene.

FIG. 1 is a high-level block diagram illustrating one embodiment of a system 100 for generating arbitrary views of a scene. As shown, an arbitrary view generator 102 receives a request for an arbitrary view as input 104, generates the requested view based on existing database assets 106, responds to the input request, The generated view is provided as output 108 . In various embodiments, the arbitrary view generator 102 may comprise a processor such as a central processing unit (CPU) or graphics processing unit (GPU). The configuration of system 100 shown in FIG. 1 is presented for illustration. In general, system 100 may include any other suitable number and/or configuration of interconnected components that provide the described functionality. For example, in another embodiment, the arbitrary view generator 102 may comprise different configurations of internal components 110-116, and the arbitrary view generator 102 may comprise multiple parallel physical and/or virtual processors. Often, database 106 may comprise multiple network databases or a cloud of assets, and so on.

Arbitrary view requests 104 include requests for arbitrary perspectives of the scene. In some embodiments, other perspectives of the scene, ie, the requested perspective of the scene containing the viewpoint, do not yet exist in the asset database 106 . In various embodiments, the arbitrary view request 104 may be received from a process or user. For example, input 104 may be received from a user interface in response to user manipulation of a presented scene or portion thereof (such as user manipulation of a camera view of the presented scene). In another example, the arbitrary view request 104 may be received in response to specifying a path of movement or movement within the virtual environment, such as a flythrough of a scene. In some embodiments, the possible arbitrary views of the scene that can be requested are at least partially constrained. For example, a user may not be able to steer the camera viewpoint of a presented interactive scene to arbitrary random positions, and is constrained to a particular position or perspective of the scene.

Database 106 stores multiple views of each stored asset. In a given context, an asset is an individual scene whose specifications are stored in the database 106 as multiple views. In various embodiments, a scene may include a single object, multiple objects, or a rich virtual environment. Specifically, database 106 stores multiple images corresponding to different perspectives or viewpoints of each asset. The images stored in database 106 include high quality photographs or photorealistic renderings. Such high definition or high resolution images that are input to database 106 may be captured or rendered during offline processing, or obtained from external sources. In some embodiments, corresponding camera characteristics are stored with each image stored in database 106 . That is, camera attributes such as relative position or location, orientation, rotation, depth information, focal length, aperture, zoom level are stored with each image. In addition, camera optical information such as shutter speed and exposure may be stored with each image stored in database 106 .

Any number of different perspectives of an asset may be stored in database 106 in various embodiments. FIG. 2 shows an example of a database asset. In the example given, 73 views corresponding to different angles around the chair object are captured or rendered and stored in database 106 . The view may be captured, for example, by rotating the camera around the chair or rotating the chair in front of the camera. Relative object and camera position and orientation information is stored with each image generated. FIG. 2 illustrates a view of a scene containing one object. Database 106 may also store specifications for scenes containing multiple objects or rich virtual environments. In such cases, multiple views corresponding to different positions or locations in the scene or three-dimensional space are captured or rendered and stored in database 106 along with corresponding camera information. In general, the images stored in database 106 may include two or three dimensions and may include stills or frames of animations or video sequences.

In response to a request 104 for an arbitrary view of a scene that does not already exist in database 106 , arbitrary view generator 102 generates the requested arbitrary view from multiple other existing views of the scene stored in database 106 . In the example configuration of FIG. 1, the asset management engine 110 of the arbitrary view generator 102 manages the database 106 . For example, asset management engine 110 may facilitate storing and retrieving data in database 106 . In response to a request for an arbitrary view of scene 104 , asset management engine 110 identifies and retrieves multiple other existing views of the scene from database 106 . In some embodiments, asset management engine 110 retrieves all existing views of the scene from database 106 . Alternatively, asset management engine 110 may select a portion of an existing view (eg, the view closest to any requested view) to retrieve. In such cases, the asset management engine 110 is configured to intelligently select some existing view from which pixels may be collected to generate the requested arbitrary view. In various embodiments, multiple existing views may be retrieved together by the asset management engine 110 or by other components of the arbitrary view generator 102 as needed.

Each existing view perspective retrieved by the asset management engine 110 is transformed into the requested arbitrary view perspective by the perspective transformation engine 112 of the arbitrary view generator 102 . As noted above, the exact camera information is known and stored with each image stored in database 106 . Perspective change from an existing view to any desired view therefore involves a simple geometric mapping or transformation. In various embodiments, perspective transformation engine 112 may use any one or more suitable mathematical techniques to transform the perspective of an existing view into the perspective of an arbitrary view. If the requested view includes an arbitrary view that is not identical to any existing view, the conversion of the existing view to the perspective of the arbitrary view will result in at least some unmapped or missing pixels, i.e. It will contain pixels at angles or positions introduced into arbitrary views that do not exist.

Pixel information from a single perspective-transformed existing view cannot fill all the pixels of another view. However, in many cases, most, if not all, of the pixels of any requested view can be collected from multiple perspective-transformed existing views. A merge engine 114 of the arbitrary view generator 102 combines pixels from multiple perspective-transformed existing views to generate the requested arbitrary view. Ideally, all pixels that make up an arbitrary view are collected from existing views. This is possible, for example, if a sufficiently diverse set of existing views or perspectives are available for the considered asset and/or if the requested perspective is not significantly different from the existing perspectives. can be

Any suitable technique may be used to combine or merge pixels from multiple perspective-transformed existing views to generate the desired arbitrary view. In one embodiment, the first existing view closest to the requested arbitrary view is selected and retrieved from the database 106 and transformed into the perspective of the requested arbitrary view. Pixels are then collected from this perspective-transformed first existing view and used to fill the corresponding pixels in the requested arbitrary view. To fill in the pixels of the requested arbitrary view that could not be obtained from the first existing view, a second existing view containing at least some of these remaining pixels is selected and retrieved from the database 106 to provide the perspective of the requested arbitrary view. is converted to Pixels that could not be obtained from the first existing view are then collected from this perspective transformed second existing view and used to fill the corresponding pixels in the requested arbitrary view. This process continues until all pixels of the requested arbitrary view are filled and/or until all existing views are exhausted or a predetermined threshold number of existing views are utilized. may be repeated for

In some embodiments, the requested arbitrary view may contain some pixels that could not be obtained from any existing view. In such cases, interpolation engine 116 is configured to fill in all remaining pixels of the requested arbitrary view. In various embodiments, any one or more suitable interpolation techniques may be used by interpolation engine 116 to generate these unfilled pixels in the requested arbitrary view. Examples of interpolation techniques that can be used include, for example, linear interpolation, nearest neighbor interpolation, and the like. Pixel interpolation introduces averaging or smoothing. Overall image quality is not significantly affected by some interpolation, but excessive interpolation can introduce unacceptable blurriness. Therefore, it may be desirable to use interpolation sparingly. As mentioned above, interpolation is avoided entirely if all pixels of the requested arbitrary view can be obtained from existing views. However, if the requested arbitrary view contains some pixels that cannot be obtained from any view, interpolation is introduced. In general, the amount of interpolation required depends on the number of existing views available, the diversity of perspectives of the existing views, and/or how the perspective of any view differs with respect to the perspective of the existing views.

For the example shown in FIG. 2, 73 views around the chair object are stored as existing views of the chair. An arbitrary view around the chair object that is different or unique from any of the stored views can be generated using a plurality of these existing views, preferably with minimal, if any, interpolation. However, it may not be efficient or desirable to generate and store such a comprehensive set of existing views. In some cases, a significantly smaller number of existing views covering a sufficiently diverse set of perspectives may instead be generated and stored. For example, 73 views of a chair object may be reduced to a small set of a few views around the chair object.

As noted above, in some embodiments, the possible arbitrary views that can be requested may be at least partially constrained. For example, a user may be restricted from moving a virtual camera associated with an interactive scene to certain positions. For the example given in Figure 2, the possible arbitrary views that can be requested are limited to arbitrary positions around the chair object, e.g. It cannot include any position under the chair object. Such constraints on allowed arbitrary views ensure that the requested arbitrary view can be generated by the arbitrary view generator 102 from existing data.

Arbitrary view generator 102 generates and outputs requested arbitrary view 108 in response to input arbitrary view request 104 . The resolution or quality of the generated arbitrary view 108 is the same as the quality of the existing view used to generate it, since pixels from the existing view are used to generate the arbitrary view, or are equivalent. Therefore, using high definition existing views in most cases yields high definition output. In some embodiments, the generated arbitrary view 108 is stored in the database 106 along with other existing views of the relevant scene, and later, in response to future requests for arbitrary views, other arbitrary views of that scene are generated. may be used to generate If the input 104 contains a request for an existing view in the database 106, then the requested view need not be generated from other views as described above; retrieved and presented directly as output 108 .

Arbitrary view generator 102 may also be configured to generate arbitrary ensemble views using the described techniques. That is, input 104 may include a request to combine multiple objects into a single custom view. In such cases, the techniques described above are performed on each of the multiple objects and combined to produce a single unified or ensemble view containing the multiple objects. Specifically, existing views of each of the plurality of objects are selected and retrieved from the database 106 by the asset management engine 110, and those existing views are transformed by the perspective transformation engine 112 into the perspective of the requested view. , the pixels from the perspective-transformed existing views are used by the merge engine 114 to fill the corresponding pixels of the requested ensemble view, and any remaining unfilled pixels in the ensemble view are used by the interpolation engine 116. In some embodiments, the requested ensemble view may include perspectives that already exist for one or more of the objects that make up the ensemble. In such cases, the existing view of the object asset corresponding to the requested perspective fills the pixels corresponding to the object in the ensemble view directly instead of first generating the requested perspective from other existing views of the object. used for

As an example of an arbitrary ensemble view containing multiple objects, consider the chair object and separately photographed or rendered table object in FIG. Chair objects and table objects may be combined using the disclosed techniques to produce a single ensemble view of both objects. Thus, using the disclosed techniques, separately captured or rendered images or views of each of multiple objects can be consistently combined to produce a scene containing multiple objects and having a desired perspective. . As mentioned above, the depth information for each existing view is known. The perspective transformation of each existing view, including the depth transformation, allows multiple objects to be properly positioned relative to each other within the ensemble view.

Generating arbitrary ensemble views is not limited to combining multiple single objects into custom views. Rather, multiple scenes with multiple objects or multiple rich virtual environments may be similarly combined into a custom ensemble view. For example, multiple separately and independently generated virtual environments (perhaps from different content generation sources and possibly with different pre-existing individual perspectives) are combined into an ensemble view with a desired perspective. good. Thus, in general, the arbitrary view generator 102 may be configured to consistently combine or match multiple independent assets, including possibly different existing views, into an ensemble view with a desired, possibly arbitrary perspective. Since all combined assets are normalized to the same perspective, a perfectly matched resulting ensemble view is produced. The possible arbitrary perspectives of the ensemble view can be constrained based on existing views of individual assets available to generate the ensemble view.

FIG. 3 is a flow diagram illustrating one embodiment of a process for generating arbitrary perspectives. Process 300 may be used, for example, by arbitrary view generator 102 of FIG. In various embodiments, process 300 may be used to generate arbitrary or arbitrary ensemble views of a given asset.

Process 300 begins at step 302 where a request for an arbitrary perspective is received. In some embodiments, the request received at step 302 may include a request for an arbitrary perspective of a given scene that differs from any existing available perspective of the scene. In such cases, for example, an arbitrary perspective request may be received in response to a requested change in perspective of the presented view of the scene. Such changes in perspective may be prompted by changes or manipulations of the virtual camera associated with the scene, such as panning the camera, changing the focal length, changing the zoom level, and the like. Alternatively, in some embodiments, the request received at step 302 may include a request for arbitrary ensemble views. As an example, such an arbitrary ensemble view request may be received for an application that allows selection of multiple independent objects and provides an integrated perspective-modified ensemble view of the selected objects.

At step 304, a plurality of existing images from which to generate at least a portion of the requested arbitrary perspective are retrieved from one or more related asset databases. Multiple retrieved images may be associated with a given asset if the request received at step 302 includes a request for an arbitrary perspective of the given asset, and the request received at step 302 may be any If it contains an ensemble view request, it may relate to multiple assets.

At step 306 , each of the plurality of existing images retrieved at step 304 having different perspectives is converted to the arbitrary perspective requested at step 302 . Each of the existing images retrieved in step 304 includes associated perspective information. The perspective of each image is defined by the camera properties associated with generating that image, such as relative position, orientation, rotation, angle, depth, focal length, aperture, zoom level, and lighting information. Since complete camera information is known for each image, the perspective transformation of step 306 involves simple mathematical operations. In some embodiments, step 306 optionally further includes an optical transformation such that all images are consistently normalized to the same desired lighting conditions.

At step 308, at least a portion of the image having the arbitrary perspective requested in step 302 is filled with pixels collected from an existing image that has been perspective transformed. That is, pixels from multiple perspective-corrected existing images are used to generate an image with a desired arbitrary perspective.

At step 310, it is determined whether the generated image with the requested arbitrary perspective is complete. If it is determined at step 310 that the generated image with the requested arbitrary perspective is not complete, then additional existing images are available to obtain any remaining unfilled pixels of the generated image. It is determined in step 312 whether . If additional existing images are available as determined at step 312 , one or more additional existing images are retrieved at step 314 and process 300 proceeds to step 306 .

If it was determined at step 310 that the generated image with the requested arbitrary perspective was not complete and it was determined at step 312 that no more existing images were available, all remaining of the generated image Unfilled pixels are interpolated in step 316 . Any one or more suitable interpolation techniques may be used at step 316 .

If it is determined in step 310 that the generated image with the requested arbitrary perspective is complete, or after interpolating all remaining unfilled pixels in step 316, generate with the requested arbitrary perspective. The finished image is output at step 318 . Thereafter, process 300 ends.

As noted above, the disclosed techniques may be used to generate arbitrary perspectives based on other existing perspectives. Since camera information is stored with each existing perspective, it is possible to normalize different existing perspectives to a common desired perspective. A resulting image with a desired perspective can be constructed by taking pixels from an existing image that has been perspective transformed. Not only is the processing associated with generating arbitrary perspectives using the disclosed technique fast and near-instantaneous, but it also produces high-quality output, so the disclosed technique is useful for interactive real-time graphics. It has become a particularly powerful technology for applications.

The techniques described above comprise a uniquely efficient paradigm for generating any desired arbitrary view or perspective of a scene using an existing reference view or image having a different perspective than the desired perspective. More specifically, the disclosed techniques generate a high-definition image having any desired arbitrary perspective from one or more existing reference images in which most, if not all, pixels of the desired arbitrary perspective are collected. facilitates the rapid generation of As noted above, the existing reference images may include high-quality photographs or photorealistic renderings, captured or rendered during offline processing, or obtained from external sources. Furthermore, the (virtual) camera properties are stored as metadata with each reference image and may be used later to facilitate perspective transformation of the images. Various techniques for generating reference images, such as the images or views stored in the asset database 106 of FIG. 1, and further details regarding their associated metadata are now described.

FIG. 4 is a flowchart illustrating one embodiment of a process for generating reference images or views of assets from which arbitrary views or perspectives of assets may be generated. In some embodiments, process 400 is used to generate reference images or views of assets stored in database 106 of FIG. Process 400 may include offline processing.

Process 400 begins at step 402 where an asset is imaged and/or scanned. Multiple views or perspectives of the asset are captured at step 402, for example, by rotating an imaging or scanning device around the asset or rotating the asset in front of such a device. In some cases, an imaging device such as a camera may be used to capture a high quality picture of the asset at step 402 . In some cases, a scanning device such as a 3D scanner may be used to collect point cloud data associated with the asset at step 402 . Step 402 also includes collecting applicable metadata (camera attributes, relative location or position, depth information, lighting information, surface normal vectors, etc.) along with the image data and/or scan data. Some of these metadata parameters may be estimated. For example, normal data may be estimated from depth data. In some embodiments, at least a predetermined set of perspectives of the asset covering most, if not all, of the target area or surface of the asset is captured in step 402 . Additionally, different imaging or scanning devices with different characteristics or attributes may be used in step 402 for different perspectives of a given asset and/or for different assets stored in database 106. .

At step 404 , a three-dimensional polygonal mesh model of the asset is generated from the image data and/or scan data captured at step 402 . That is, a fully tuned three-dimensional mesh model is generated based on the photograph and/or point cloud data captured in step 402 and associated metadata. In some embodiments, enough asset data is captured at step 402 to ensure that a complete mesh model can be built at step 404 . Portions of the generated mesh model not sufficiently captured in step 402 may be interpolated. In some cases, step 404 is not fully automated and involves at least some human intervention to ensure that the generated three-dimensional mesh model is orderly.

At step 406 , multiple reference images or views of the asset are rendered from the three-dimensional mesh model generated at step 404 . Any suitable rendering technique may be used in step 406 depending on available resources. For example, simpler rendering techniques (such as scanline rendering or rasterization) may be used when there are constraints on computational resources and/or rendering time, at the expense of rendering quality. In some cases, more complex rendering techniques (such as ray tracing) that consume more resources but produce high quality photorealistic images may be used. Each reference image rendered in step 406 has associated metadata determined from the 3D mesh model, including (virtual) camera attributes, relative location or position, depth information, lighting information, surface normal vectors, etc. may contain parameters for

In some embodiments, any source image captured in step 402 includes a very small portion of the reference image or view of the asset stored in database 106 . Rather, most of the images or views of the asset stored in database 106 are rendered using the 3D mesh model of the asset generated at step 404 . In some embodiments, the reference image or view of the asset includes one or more orthographic views of the asset. Such orthographic views of multiple different assets may be combined to produce an orthographic view of a composite asset constructed from or by combining multiple separately captured or rendered individual assets ( stacked together, or placed next to each other, like building blocks, etc.), then the orthographic view of the composite asset transforms the orthographic view of each of the individual assets into any desired perspective. collectively into any arbitrary camera view.

3D mesh model-based rendering of process 400 of FIG. 4 is computationally intensive and time consuming. Therefore, in most cases, process 400 includes offline processing. Additionally, although there may be a 3D mesh model of the asset, rendering high quality arbitrary perspectives directly from such a model is an efficient accomplishment for many applications, including most real-time or on-demand applications. I can't. Rather, despite the existence of an underlying 3D mesh model that can render any desired arbitrary perspective of the asset, more efficient techniques need to be used to meet speed constraints. For example, the arbitrary view generation techniques described above with respect to the descriptions of FIGS. 1-3 very quickly generate a desired arbitrary view or perspective based on an existing reference view or image of an asset while matching the quality of the reference view. May be used to maintain quality. However, in some embodiments, the inefficiencies associated with building a 3D mesh model and rendering a reference view from the model can be avoided despite having the option of performing these steps offline. may be undesirable or unacceptable. In some such cases, the steps of building the mesh model and utilizing complex rendering techniques to generate the reference view may be omitted, as described further below.

FIG. 5 is a flowchart illustrating one embodiment of a process for generating reference images or views of assets from which arbitrary views or perspectives of assets may be generated. In some embodiments, process 500 is used to generate reference images or views of assets stored in database 106 of FIG. Process 500 may include offline processing.

Process 500 begins at step 502 where an asset is imaged and/or scanned. Multiple views or perspectives of the asset are captured at step 502, for example, by rotating an imaging or scanning device around the asset or rotating the asset in front of such a device. The views captured in step 502 may include, at least in part, orthographic views of the asset. In some embodiments, the image/scan captured in step 502 has a field of view that overlaps with at least one other image/scan captured in step 502, and the relative (camera/scan) between the two. Scanner) Attitude is known and stored. In some cases, an imaging device such as a DSLR (digital single lens reflex) camera may be used to capture a high quality picture of the asset at step 502 . For example, a camera with a long focus lens may be used to simulate an orthographic view. In some cases, a scanning device, such as a 3D scanner, may be used to collect point cloud data associated with the asset at step 502 . Step 502 also includes applying any applicable metadata, such as camera attributes, relative location or position, lighting information, surface normal vectors, relative poses between images/scans with overlapping fields of view, to images and/or Storing with scan data. Some of these metadata parameters may be estimated. For example, normal data may be estimated from depth data. In some embodiments, at least a predetermined set of perspectives of the asset that sufficiently covers most, if not all, of the target area or surface of the asset is captured in step 502 . Additionally, different imaging or scanning devices with different characteristics or attributes may be used in step 502 for different perspectives of a given asset and/or for different assets stored in database 106. .

At step 504 , multiple reference images or views of the asset are generated based on the data captured at step 502 . A reference view is generated at step 504 simply from the images/scans captured at step 502 and associated metadata. That is, with the appropriate metadata and overlapping perspectives captured in step 502, any arbitrary view or perspective of the asset may be generated. In some embodiments, a comprehensive set of reference views of assets stored in database 106 is generated from the images/scans captured in step 502 and their associated metadata. The data captured in step 502 may be sufficient to form fragments of the mesh model, but it is not necessary that an integral, fully-tuned mesh model is generated. Therefore, no full 3D mesh model of the asset is generated, nor are complex rendering techniques such as ray tracing used to render the reference image from the mesh model. Process 500 improves efficiency by eliminating the steps of process 400 that consume the most processing resources and time.

The reference images generated in step 504 may facilitate faster generation of arbitrary views or perspectives using the techniques described above with respect to the descriptions of FIGS. 1-3. However, in some embodiments, a repository of reference images need not be created at step 504 . Rather, the views captured in step 502 and their associated metadata are sufficient to generate any desired arbitrary view of the asset using the techniques described above with respect to the description of FIGS. i.e. simply any desired arbitrary view or Perspectives can be generated. The processing associated with generating the desired arbitrary view from only the source images captured in step 502 is sufficiently fast for many on-demand real-time applications. However, if further efficiencies in speed are desired, a repository of reference views may be generated, such as at step 504 of process 500 .

As noted above, each image or view of an asset in database 106 may be stored with corresponding metadata. Metadata may be used when rendering a view from a model, imaging or scanning an asset (in which case depth and/or surface normal data may be estimated), or a combination of both. It may be generated from a mesh model.

A given view or image of an asset includes pixel intensity values (eg, RGB values) for each pixel in the image and various metadata parameters associated with each pixel. In some embodiments, one or more of a pixel's red, green, and blue (RGB) channels or values may be used to encode pixel metadata. Pixel metadata may include, for example, information about the relative location or position (eg, x, y, and z coordinate values) of the point in three-dimensional space that is projected onto that pixel. Additionally, the pixel metadata may include information about the surface normal vector at that location (eg, angles with the x, y, and z axes). Pixel metadata may also include texture mapping coordinates (eg, u and v coordinate values). In such cases, the actual pixel value at the point is determined by reading the RGB values of the corresponding coordinates in the texture image.

Surface normal vectors facilitate modifying or changing the lighting of any generated view or scene. More specifically, scene lighting changes determine how well a pixel's surface normal vector matches the direction of newly added, removed, or otherwise changed light sources (e.g., light source direction and the pixel's normal vector, which can be quantified at least in part by the dot product of the pixel's normal vector. Using texture mapping coordinates to define pixel values facilitates modifying or altering the texture of any generated view or scene or portion thereof. More specifically, the texture can be modified by simply exchanging or replacing the referenced texture image with another texture image having the same dimensions.

As noted above, a reference image or view of an asset may be generated with or without an underlying mesh model of the asset. In the most efficient embodiment, only a small set of source images/scans capturing various (overlapping) views around the asset, and the associated metadata associated with them, are shown in FIGS. 3, to generate any desired arbitrary view of the asset and/or a set of reference views from which the desired arbitrary view can be generated. In such embodiments, the most resource intensive process of modeling and rendering based path tracing is eliminated. Images or views generated using the disclosed arbitrary view generation techniques may include static or dynamic scenes and may include still images or frames of animations or video sequences. For motion capture, a set of images or views of one or more assets may be generated for each time slice. The disclosed technique is particularly useful in applications requiring fast generation of high quality arbitrary views, such as gaming applications, virtual/alternative reality applications, CGI (Computer Generated Image) applications, and the like.

Existing 3D content frameworks based on rendering from 3D models are typically developed and optimized for specific uses and lack scalability for different platforms and applications. As a result, substantial effort and resources must be invested and repeated in generating the same 3D content for different use cases. Furthermore, the requirement of 3D content is faced with moving objects over time. Therefore, 3D content must be manually regenerated as requirements change. As such, the proliferation of 3D content has been hampered as a result of the difficulty in standardizing 3D content formats across different platforms, devices, applications, use cases, and generally varying quality requirements. Accordingly, there is a need for a more extensible format for representing three-dimensional content that can be used to achieve any desired level of quality as disclosed herein.

The disclosed technology provides a fundamentally new framework for representing 3D content as 2D content, while retaining the attributes of conventional 3D frameworks, as well as various other features and advantages. provide everything. As described above, three-dimensional content and corresponding information can be divided into multiple images from which any desired arbitrary view can be generated without requiring an underlying three-dimensional model of the associated asset. encoded. That is, the techniques described above effectively involve the conversion of three-dimensional source content into two-dimensional content (ie, images). More specifically, it provides a two-dimensional platform containing a set of images associated with assets that effectively replaces traditional three-dimensional platforms containing three-dimensional models. As noted above, the images that make up the 2D platform may be generated from a 3D model and/or from a small set of source images or scans. Associated metadata is stored for each view of the asset, in some cases encoded as pixel values. The image-based view and metadata of a given 2D architecture facilitates using 2D content as a 3D source. Thus, the disclosed technique completely replaces conventional 3D architectures that rely on rendering with an underlying 3D polygonal mesh model. 3D source content (such as physical assets or 3D mesh models of assets) has traditionally only been available using 3D frameworks, including the ability to generate multiple different views or perspectives of the asset. is encoded or converted into a two-dimensional format containing a set of views and metadata that are used in turn to represent and present the features that have been used. In addition to providing all the features of conventional three-dimensional frameworks, the disclosed two-dimensional representation offers various additional unique features, such as being suitable for conventional image processing techniques.

In the disclosed two-dimensional framework for representing three-dimensional content, information about assets is encoded as image data. The image comprises an array having a height, a width and a third dimension containing pixel values. Images associated with an asset may comprise various reference views or perspectives of the asset and/or corresponding metadata encoded as pixel values (eg, RGB channel values). Such metadata includes, for example, camera characteristics, textures, uv coordinate values, xyz coordinate values, surface normal vectors, lighting information (global illumination values, or values associated with a given lighting model, etc.), etc. may contain. In various embodiments, the image containing the reference view or perspective of the asset may be a (high quality) photograph or a (photorealistic) rendering.

For example, the ability to render any desired arbitrary view or perspective of assets with arbitrary camera characteristics (such as camera position and lens type), arbitrary asset ensembles or combinations, arbitrary lighting, arbitrary texture variations, etc. Features are supported by the disclosed two-dimensional framework. Since complete camera information is known for and stored with the reference view of an asset, other new views of the asset, including any camera characteristics, can be generated from multiple perspective-transformed reference views of the asset. you can More specifically, a given arbitrary view or perspective of a single object or scene may be generated from multiple pre-existing reference images associated with the object or scene, while a given arbitrary ensemble view of the object Or it may be generated by normalizing and consistently combining multiple objects or scenes from a set of reference images associated with the scene into an integrated view. A reference view of an asset may have lighting modeled by one or more lighting models (such as a global illumination model). A surface normal vector known for the reference view facilitates arbitrary lighting control, such as the ability to change the lighting of an image or scene according to any desired lighting model. A reference view of an asset has a texture specified in texture mapping (uv) coordinates, which can be used to replace any desired texture by simply changing the referenced texture image Facilitates texture control.

As mentioned above, the disclosed two-dimensional framework is based on image datasets and is therefore suitable for image processing techniques. Thus, the disclosed image-based two-dimensional framework for representing three-dimensional content is inherently seamlessly scalable and resource-adaptive both up and down the computation and bandwidth spectrum. Existing techniques for scaling images (such as image compression techniques) may be advantageously used to scale the image-based 3D content of the disclosed framework. Images, including the disclosed two-dimensional framework, can be easily scaled in terms of quality or resolution to appropriately comply with the requirements of different channels, platforms, devices, applications, and/or use cases. Image quality or resolution requirements may vary across different platforms (e.g. mobile vs. desktop), different models of devices for a given platform, different applications (e.g. online viewers vs. native applications running locally on a machine), over time, and different It can vary significantly over network bandwidth, etc. Therefore, a need exists for an architecture (such as the disclosed two-dimensional framework) that comprehensively meets the requirements of different use cases and is immune to changes in requirements over time.

In general, the disclosed two-dimensional framework supports resource-adaptive rendering. Additionally, time-varying quality/resolution adaptations may be provided based on current or real-time availability of computational resources and/or network bandwidth. Scaling (i.e., the ability to smoothly and seamlessly raise and lower image quality levels is in most cases fully automated. For example, the disclosed two-dimensional framework allows the reference Assets (i.e., including assets) across one or more features such as views or perspectives as well as images that encode metadata (e.g., textures, surface normal vectors, xyz coordinates, uv coordinates, lighting values, etc.) provides the ability to automatically downsample one or more images), in some such cases the scaling of the asset may not be uniform across all features of the asset and the associated It may vary depending on the type of information that contains or is encoded within the image, for example, the actual image pixel values of the asset's reference view or perspective may be irreversibly compressed, but the specific images that encode metadata (such as depth (i.e., xyz values) and normal values) may not be compressed in the same way, or in some cases such loss of information is acceptable at rendering time. Since it cannot be compressed, it does not have to be compressed at all.

In some embodiments, a master asset (ie, a set of images comprising the master asset) having the highest available quality or resolution is generated and stored, eg, in database 106 of FIG. In some such cases, one or more lower quality/resolution versions of the asset are automatically generated from the master asset to generate the requested perspective by the server, the requesting client, and/or one or more are stored so that the appropriate version can be selected to generate the requested perspective or view, based on the (current) capabilities of the associated communication network. Alternatively, in some cases, a single version of the asset (i.e., the master asset) is stored, and the disclosed framework is used by the server, requesting client, and/or Supports streaming or progressive delivery of quality or resolution up to the quality or resolution of the master asset, based on the (current) capabilities of one or more associated communication networks.

FIG. 6 is a flow diagram illustrating one embodiment of a process for providing a requested view of a scene. Process 600 may be used, for example, by arbitrary view generator 102 of FIG. In some embodiments, process 300 of FIG. 3 is part of process 600 . In various embodiments, process 600 may be used to generate arbitrary views of a scene including one or more assets (ie, a given asset or arbitrary ensemble of assets).

Process 600 begins at step 602 where a request is received for a desired arbitrary view of a scene that does not yet exist that is different from any other existing available views of the scene. In general, arbitrary views may include any desired view of a scene or asset whose specifications are not known in advance before it is requested. The arbitrary view request of step 602 is received from the client and may include specifications such as predetermined camera characteristics (eg, lens type and pose/perspective), lighting, textures, asset ensemble, and the like.

At step 604, the arbitrary view of the scene requested at step 602 is generated or rendered based on the available resources. For example, the requested arbitrary view generated in step 604 may include the client requesting the arbitrary view, the computing or processing power of the server generating the requested arbitrary view, and/or one or more relationships between the client and the server. It can be scaled appropriately based on the bandwidth availability of the communication network. More specifically, step 604 implements resource adaptive rendering by trading off image quality for responsiveness by scaling or adjusting along one or more associated axes as described next. make it easier.

The quality of the images, including the demand views, generated or rendered in step 604 using the disclosed techniques depends, at least in part, on the number of existing perspective-transformed reference images used to generate the demand views. may be based on Using more reference images often leads to higher quality and using fewer reference images leads to lower quality. Accordingly, the number of reference images with different perspectives used to generate a request view may be adapted or optimized for various platforms, devices, applications, or use cases, further reducing real-time resource consumption. May be adapted based on availability and constraints. As some examples, a relatively large number of reference images (e.g., 60 images) are used to generate request views containing still images or for native applications on desktops with high-speed Internet connections. while a relatively small number of reference images (e.g., 12 images) are used to generate a desired view containing frames of a video or augmented reality sequence or for a web application for a mobile device. you can

The quality of the image containing the requested view generated or rendered in step 604 using the disclosed techniques is at least partially dependent on the image containing one or more assets used to generate the requested view (i.e. , an image including the reference perspective and associated metadata of one or more assets) resolution (ie, pixel density). A higher resolution version of the image containing the asset leads to higher quality, while a lower resolution version of the image containing the asset leads to lower quality. Accordingly, the resolution or pixel density of the images, including the different perspectives and associated metadata used to generate the requested view, may be adapted or optimized for various platforms, devices, applications or use cases, Further, it may be adapted based on real-time resource availability and constraints. As some examples, a relatively high resolution (e.g., 2K x 2K) of images associated with one or more assets to generate the desired view for native applications on desktops with high-speed internet connections may be used, while a relatively low resolution of images associated with one or more assets (e.g., 512×512 ) may be used.

The quality of the image containing the requested view generated or rendered in step 604 using the disclosed techniques is at least partially dependent on the image containing one or more assets used to generate the requested view (i.e. , images including the reference perspective and associated metadata of one or more assets) bit depth (ie, bits per pixel). A higher bit-depth version of the image containing the asset leads to higher quality, while a lower bit-depth version of the image containing the asset leads to lower quality. Accordingly, the pixel accuracy of the images, including the different perspectives and associated metadata used to generate the requested view, may be adapted or optimized for various platforms, devices, applications, or use cases, and , may be adapted based on real-time resource availability and constraints. As a few examples, a higher precision version of the image associated with one or more assets (e.g., 64bpp for texture values, xyz coordinates and normal vectors) to produce a higher quality request view. float for ) may be used, while a lower-precision version of the image associated with one or more assets (e.g., 24 bpp for texture values, 48 bpp) may be used for xyz coordinates and normal vectors.

The disclosed technique for resource-adaptive rendering can be any one or more of the three axes of the image (number, resolution, and bit depth) used to generate or render the desired arbitrary view of the scene. Supports discrete and/or continuous scaling along . The image quality of the requested view may be changed by appropriately scaling and/or selecting different combinations or versions of the image containing the reference view and metadata used to generate or render the requested view. The output image quality of the requested view may be selected at step 604 based on one or more (real-time) considerations and/or constraints. For example, the image quality selected for the requesting view may depend on the requesting client's platform or device type (e.g., mobile vs. desktop and/or models thereof), the predetermined viewport size and/or fill factor (e.g., 512 x512 windows vs. 4K windows), application type (e.g. still images vs. video, games, or frames in virtual/augmented reality sequences), network connection type (e.g. mobile vs. broadband) etc. Quality may thus be selected based on a given use case and the client's capabilities for the given use case.

In some embodiments, the disclosed technology further supports streaming or progressive delivery from low quality to high quality below the highest quality available or achievable on the client device. In many cases, the selection of scaling or number of reference images used to generate a request view depends at least in part on the latency requirements of the associated application. For example, a relatively large number of reference images may be used to generate a still image, while a relatively small number of reference images are used to generate frames for applications where the view changes rapidly. be taken. In various embodiments, the scaling may be the same across one or more of the above axes available for scaling and/or depending on the type of information being encoded by the various images. can be different. For example, the resolution and bit depth of the images used to generate the request views may be scaled uniformly or independently in direct proportion. As an example, the resolution may be downsampled, but the bit depth is not scaled down at all to preserve high dynamic range and color depth in applications where preservation of tonal quality (lighting, color, contrast) is important. good. Additionally, the resolution and bit depth of the images used to generate the request view may be lossy for some types of data, such as the actual pixel values of the reference view, but the depth (xyz coordinates) and plane Other types of data, including metadata, such as normal vectors, are unacceptable and may be scaled differently depending on the type of information encoded in the image.

At step 606 , the requested view generated or rendered at step 604 is provided to, for example, a requesting client to satisfy the incoming request of step 602 . Thereafter, process 600 ends.

As described above, the two-dimensional framework described above for generating or rendering any desired arbitrary view of a scene containing an asset or an ensemble of assets includes reference views having different perspectives and associated with each reference view or perspective. It is based on images that contain metadata. As some examples, the metadata associated with each reference view or perspective associates each pixel of the reference view or perspective with its position in three-dimensional space (xyz coordinate values) and the surface normal vector at that position. good. For images generated with physically-based rendering techniques using a three-dimensional model, relevant metadata may be captured or generated from the corresponding three-dimensional model and associated with the image. For images in which one or more types of metadata are unknown (eg, photographs/scans or other renderings), such metadata values may be determined using machine learning-based techniques. For example, a neural network may be used to determine the mapping from image space to metadata space, as described further below.

FIG. 7 is a high-level block diagram illustrating one embodiment of a machine learning-based image processing framework 700 for learning attributes associated with image datasets. Available three-dimensional (polygon mesh) models of assets and a pre-defined modeled environment 702 are used to render a wide image dataset 704 using, for example, physically-based rendering techniques. In some embodiments, the modeled environment closely matches or substantially simulates the actual physical environment in which the physical asset is imaged or photographed. The rendered image dataset 704 may include photorealistic renderings and may include multiple views or perspectives of the asset and textures. Additionally, the rendered image dataset 704 is appropriately labeled or tagged or otherwise associated with relevant metadata determined and/or captured during rendering.

A broad tagged dataset 704 is perfectly suited for artificial intelligence-based learning. For example, training 706 on data set 704 using any combination of one or more suitable machine learning techniques (such as deep neural networks and convolutional neural networks) may result in data set 704 including associated metadata values. A set of one or more characteristics or attributes 708 associated with is learned. Such learned attributes may be derived or inferred from labels, tags, or metadata associated with dataset 704 . The image processing framework 700 may be trained on multiple different training data sets associated with various assets and combinations of assets. However, in some embodiments, at least a portion of the training data set is constrained to a predetermined modeled environment. After training on multiple datasets to learn different attributes or attribute types, the image processing framework 700 then renders other renderings of the assets rendered with respect to the same or similar model environment as the training data, as well as To detect or derive similar attributes or combinations thereof in other images where such attributes are unknown, such as photographs captured in a real physical environment that matches or resembles the environment modeled by the model environment of the training data; may be used. As an example, a machine-learning-based framework trained on a dataset in which physical xyz position coordinates are tagged with image pixels and surface normal vectors are tagged with image pixels, such metadata values are known. It may be used to predict the position (ie, depth or xyz distance from the camera) and surface normal vector of the unexposed image.

The disclosed framework is particularly useful when known controlled or constrained physical environments that can be simulated or modeled are used to image or film individual assets or combinations thereof. In one application, for example, the devices described above (eg, camera rigs) for imaging or photographing objects or items can be used in a retailer's product warehouse. In such applications, accurate information about the actual physical environment in which the object is imaged or filmed is known, for example, in some cases from the viewpoint or perspective of the imaged object from within the imaging device. Known information about the actual physical environment may include, for example, the structure and shape of the imager; the number, type, and pose of the cameras employed; the location and intensity of light sources and ambient lighting; Such known information about the actual physical environment is machine learning-based so that the modeled environment is identical to, or at least substantially reproduces or simulates, the actual physical environment. It is used to define a modeled environment for rendering training datasets in an image processing framework. In some embodiments, for example, the modeled environment includes a three-dimensional model of the imaging device and the same camera configuration and lighting as the actual physical environment. To be able to utilize the disclosed machine learning-based framework for detecting or predicting metadata values in images where such metadata values are not known (such as photographs captured in real-world physical environments); Metadata values are learned from a training dataset tagged with known metadata values. Constrain certain attributes of the environment (e.g. shape, camera, lighting) to known values to facilitate learning and predict other attributes (e.g. depth/position, surface normal vector) become.

As mentioned above, a machine learning-based image processing framework, for which the metadata values are known, is used to learn metadata from available 3D models and renders generated from a given modeled environment. Machine learning-based image processing frameworks may then be used to identify metadata values in images where such metadata values are unknown. Although described in terms of given physical environments and corresponding modeled environments in some of the examples given, the disclosed techniques are generally applicable to different types of assets, modeled environments, and/or combinations thereof. may be utilized and adapted to learn and predict different types of image metadata. For example, the described machine-learning-based framework is capable of predicting images of any asset rendered or captured in any environment, assuming that the training dataset spans a sufficiently comprehensive and diverse set of assets and environments. It may be trained to determine unknown metadata values.

FIG. 8 is a flowchart illustrating one embodiment of a process for entering images associated with an asset or scene into a database that can be used to generate other arbitrary views of the asset or scene. For example, process 800 of FIG. 8 may be used to populate asset database 106 of FIG. Process 800 utilizes a machine learning-based framework (such as framework 700 in FIG. 7). In some embodiments, the images of process 800 are constrained to a given physical environment and a corresponding modeled environment. More generally, however, process 800 may be used with respect to any physical or modeled environment.

Process 800 begins at step 802 where metadata associated with the training data set is learned using machine learning based techniques. In some embodiments, the image dataset used for training is a known three-dimensional model of an asset or scene in a simulated or modeled environment defined by predetermined specifications, e.g. geometry, camera, lighting, etc. Contains an extensive collection of images of assets or scenes rendered from. The learned metadata may include different types of image metadata values. The training data set of step 802 may cover different assets within a given model environment or, more generally, may comprehensively cover different assets within different environments.

At step 804, an image with one or more unknown or incomplete Gaso metadata values is received. The received images may include renderings or photographs or scans. In some embodiments, the received images were generated or captured with respect to a modeled or physical environment that is the same or similar to the rendering environment used for at least a portion of the training image dataset of step 802. is.

At step 806 , unknown or incomplete metadata values for the received image are determined or predicted using the machine learning-based framework of process 800 . At step 808, the received images and associated metadata are stored, for example, in asset database 106 of FIG. Thereafter, process 800 ends.

By determining and associating relevant metadata with an image (i.e., the image received at step 804 and stored at step 808), the process 800 converts the image into an associated asset or other arbitrary view of the scene. It effectively facilitates conversion into a reference image or view of the relevant asset or scene that can be used later for rendering. In various embodiments, when storing an image as a reference image, the image is appropriately tagged with corresponding metadata and/or associated with one or more images encoding associated metadata values. good. Process 800 generally uses machine learning-based techniques such that an image becomes a reference image from which related assets or other views of a scene can be generated, e.g., with arbitrary camera characteristics, textures, lighting, etc. It may be used to transform any image into a reference image by determining the unknown image metadata values needed to do so. Additionally, process 800 is particularly useful for determining or predicting types of metadata where accuracy is important (such as depth values and surface normal vector values).

As noted above, most of the disclosed techniques are based on making available and utilizing extensive datasets of existing reference images or views and corresponding metadata. Therefore, often a sequence of images or views with different camera perspectives around one or more objects or assets is rendered or generated and stored in a database or repository. For example, a 360 degree rotation may be rendered or generated, including angles covering 360 degrees around an object or asset. Such datasets may be built offline, but strictly physically-based rendering techniques are costly operations in terms of resource consumption, requiring significant processing power and time. Several techniques have already been described for more efficiently generating or rendering images or views of objects or assets. Additional techniques for rendering or generating images or views of objects or assets more efficiently are described in detail below.

Substantial redundancy exists for certain types of data or data sets. For example, there is a lot of redundancy among a set of images or views that include one rotation around an object or asset, especially between nearby images or views that differ by a small camera angle or rotation. Similarly, redundancy exists among the frames of an animation or video sequence, especially between adjacent or nearby frames. As another example, there is a lot of redundancy in images or views that contain the same texture. More generally, therefore, in a particular feature space, many images exhibit the same or very similar features and share significant feature space correlations. For example, in the example above, substantially similar texture features may be shared by many images or views. Given the availability of large existing object or asset datasets, when rendering or generating new images or views (e.g., of different perspectives or of different object or asset types or shapes), Redundancy with respect to such existing images can be exploited. In addition, inherent redundancies in sequences of slowly changing images or frames may be exploited as well. Machine learning is particularly well suited for learning and detecting features in large datasets containing relatively well-defined and constrained feature spaces. Thus, in some embodiments, machine learning frameworks (such as neural networks) generate new images or views more efficiently based on exploiting feature redundancy with respect to other (existing) images or views. used to render or generate In general, any suitable neural network configuration may be utilized with the disclosed techniques.

FIG. 9 is a high-level flowchart illustrating one embodiment of a process for generating an image or frame. In some embodiments, process 900 includes super-resolution processing to upscale the input image. As further described below, the process 900 provides substantially less than exact physically-based rendering and other existing techniques, especially when producing photorealistic high-quality or high-definition (HD) images. It may be used to generate output images more efficiently, resulting in less resource consumption.

Process 900 begins at step 902 where a feature space is identified or defined. The feature space identified in step 902 may include one or more features (such as predetermined texture features). In some embodiments, the feature space is identified at step 902 using a neural network-based machine learning framework. In some such cases, for example, a neural network determines or detects one or more features specific to a given set of images comprising a constrained feature space that is known and well-defined for that set of images. used to That is, the set of images acts as a prior distribution for defining the feature space. The set of images may be, for example, strictly rendered or generated images (e.g., high or full resolution or definition images) and/or previously rendered or generated existing images or portions thereof (e.g., existing image patches).

At step 904, features are detected in the input image. More specifically, an input image is processed by a neural network to determine feature space data values for the input image. The input images of step 904 include images of low quality or resolution or small size rendered or generated using techniques with low computational complexity or cost compared to the image set of step 902 . That is, the input images of step 904 are noisier (e.g., due to not using enough samples for convergence) and/or of inferior quality compared to the images that make up the image set of step 902. Contains images (eg, of lower resolution and/or size).

At step 906, the output image is obtained by replacing the features detected in the input image at step 904 with corresponding (e.g., closest or closest matching) features in the feature space identified at step 902. generated. More specifically, with respect to the feature space identified in step 902, a nearest neighbor search is performed on the features detected in the input image in step 904, and the features detected in the input image in step 904 are identified in step 902 is replaced with the corresponding closest matching feature from the feature space identified in . The feature detection, nearest neighbor search, and feature replacement described above are performed in the feature space. Accordingly, in some embodiments, step 906 includes decoding or transforming from feature space back to image space to produce a resulting output image. Manipulations in feature space lead to consistent corresponding pixel-level transformations in image space.

In general, process 900 provides an efficient framework for image restoration or upscaling or modification by exploiting redundancy and information available from other existing images. For example, process 900 may be used to clean an input image (ie, to transform a noisy input image into a relatively noise-free output image). Similarly, the process 900 may be used to improve the quality of the input image (i.e., to convert a relatively low quality input image into a high quality output image, e.g., in terms of resolution, size, bit depth, etc.). ) may be used. More specifically, process 900 facilitates applying image set features to inferior or degraded input images that share redundancy in feature space with the image set. Since the process 900 essentially involves a low-computation-cost lookup operation combined with some other relatively simple distance calculation, such as a nearest-neighbor search, image rendering is much easier, especially compared to strict physically-based rendering techniques. Provides substantial efficiency in space. Process 900 is therefore particularly useful in image rendering or generation pipelines for generating images or frames with higher speed and efficiency. For example, physically-based rendering or other techniques with low computational complexity may be used to render or generate images of low quality or resolution or small size, after which process 900 converts the images to high quality. Or may be used to convert to full resolution or larger size versions. Additionally, process 900 similarly restores or uploads input images, including photographs, captured in a given physical environment based on a training data set constrained to a simulated or modeled version of the given physical environment. May be used to scale or otherwise modify. That is, process 900 may be used with the machine learning-based architectures described in detail above with respect to FIGS.

Process 900 may be utilized for and adapted to many specific use cases. In some embodiments, process 900 is used to generate a sequence of images, such as a reference view containing (360°) rotations around an object or asset or frame of a video or animation sequence. In such cases, substantial redundancy exists between neighboring images or frames of the sequence, and the redundancy can be exploited by process 900 . In one example, some images of a sequence are classified as independent frames (I-frames) and rendered at high or full definition or resolution or size. All images in the sequence that are not classified as I-frames are rendered with lower quality or resolution or smaller size because they depend on other frames (i.e., I-frames) for upscaling and are classified as dependent frames (D-frames). be done. With respect to process 900 , an I-frame corresponds to the set of images of step 902 and each D-frame corresponds to an input image of step 904 . In this example, the number of I-frames selected for the sequence may depend on the desired trade-off between speed and quality, with more I-frames selected for better image quality. be done. In some cases, fixed rules that define predetermined intervals may be used to designate I-frames (e.g., every other or every third image in a sequence is an I-frame), Alternatively, a specific threshold may be set to identify and select I-frames within the sequence. Alternatively, adaptive techniques may be used to select new I-frames as the correlation between D-frames and existing I-frames weakens. In some embodiments, process 900 is used to generate an image containing a given texture. For process 900 , the low quality or low resolution or small size version of the image comprises the input image of step 904 and the set of images or patches of the given texture comprises the set of images of step 902 . More specifically, in this case the given texture is known and well defined from existing images containing the same texture. In this embodiment, texture patches are generated from one or more existing renders or assets with a given texture, and the generated patches are processed in an appropriate manner (e.g., finding patches with more diverse feature content). (by performing clustering in the feature space to select . The output image of step 906 from either of the two above examples includes a high quality or high resolution or large size or denoised version of the input image of step 904 . Although several specific examples are described, process 900 may generally be adapted to any applicable application where sufficient redundancy exists.

In some embodiments, one or more machine learning-based techniques may be used to generate arbitrary or novel views or perspectives of objects or assets. In some such cases, associated machine learning-based frameworks are constrained to a known and well-defined feature space. For example, images processed by such machine learning-based frameworks may be constrained to a given environment and/or one or more known textures. As detailed with respect to FIGS. 7 and 8, for example, the training data set is constrained to a predetermined model environment that simulates the actual physical environment in which the input images (i.e., photographs) of physical assets are captured. good. In such cases, the input image itself may not be associated with any image metadata values, or at least very precise image metadata values. However, one or more neural networks are utilized to learn metadata values in simulation from a synthetic training data set containing accurate metadata values, and then predict or determine relevant metadata values; and/or or captured with a real camera to generate a corresponding reference image or view that can later be used to generate other views or images as described in the disclosed Arbitrary View Generation Framework. may be applied to an input image (i.e., a photograph).

FIG. 10 is a high-level flowchart illustrating one embodiment of a process for generating arbitrary or new views or perspectives of objects or assets. As detailed further below, process 1000 may be used to transform an image or photograph of an object or asset captured in a known physical environment into an arbitrary view or perspective of that object or asset. Many of the steps of process 1000 are facilitated by machine learning-based frameworks (e.g., one or more associated neural networks learning from a training data set constrained to a given model environment simulating the physical environment). be. The feature space may be further constrained to known textures for which there are extensive training datasets.

Process 1000 begins at step 1002 where an input image of an object or asset is received. In some embodiments, the input image comprises a photograph of the object or asset captured in a known physical environment, such as a predetermined imaging device (eg, camera rig) for photographing the object or asset. In some embodiments, the input image includes multiple images (eg, images from different cameras or camera angles). For example, the input images may include a stereo pair including left and right images captured by left and right cameras of a given imaging device or camera rig.

At step 1004, the background of the input image received at step 1002 is removed such that only the subject (ie, object or asset) of the input image remains. In general, any one or more suitable image processing techniques for background removal may be used in step 1004 . In some embodiments, background removal is facilitated by image segmentation. In some such cases, neural networks may be used to facilitate image segmentation. For example, during training, a convolutional neural network or other suitable neural network learns image features (edges, corners, shapes, sizes, etc.) at a lower resolution, e.g., 128x128 or 256x256, Those learned features may be combined to create an upscaled segmentation mask.

At step 1006, depth values for objects or assets in the input image are determined. A depth value is determined at step 1006 for each pixel. Step 1006 may include determining a depth estimate and/or fine-tuning the determined depth estimate. For example, depth estimates may be determined from the left and right stereo pairs that make up the input image and/or predicted using a neural network. The determined depth estimate may be later cleaned or fine-tuned using, for example, neural networks and/or other techniques.

At step 1008 , an output image containing a predetermined arbitrary perspective of the object or asset that differs from the perspective of the input image is generated by performing a perspective transformation based on the depth values determined at step 1006 . In general, any given perspective may include any desired or requested camera view of an object or asset. For example, a given arbitrary perspective may include an orthographic (eg, top-down or bird's-eye) view of an object or asset. Step 1008 may include determining a perspective transformation estimate and/or fine-tuning the determined perspective transformation estimate. For example, perspective transformation estimates may be determined directly from mathematical transformations and/or indirectly predicted using a neural network (such as a generative adversarial network (GAN)). The determined perspective transformation estimate may then be cleaned or fine-tuned using, for example, a neural network (such as a reconstruction network as described with respect to FIG. 9, or a GAN). In some cases, the determined perspective transform estimate may alternatively or additionally be fine-tuned using traditional techniques such as denoising, inpainting, and the like.

Thereafter, process 1000 ends. As described with respect to process 1000, multiple stages and/or layers of neural network-based techniques may be used to generate arbitrary views or perspectives of objects or assets.

Although the above embodiments are described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative, not limiting.

Claims (20)

a method,
remove the background of the input image of the object or asset,
determining a depth value for the object or asset in the input image;
generating an output image that includes a predetermined perspective of the object or asset that differs from the perspective of the input image by performing a perspective transformation based on the determined depth values;
with
A method, wherein a machine learning-based framework is utilized for one or more steps of said method, said machine learning-based framework being constrained to a known predetermined environment. 2. The method of claim 1, wherein the known predetermined environment comprises a physical environment in which the input image was taken and a simulated physical environment for training data sets of the machine learning-based framework. a model environment for and a method. 2. The method of claim 1, wherein the machine learning-based framework is constrained to one or more known textures. 2. The method of Claim 1, wherein the input image comprises a photograph captured by a camera of the object or asset. 2. The method of claim 1, wherein the input image comprises multiple images from different cameras or camera angles. 2. The method of claim 1, wherein removing the background is based at least in part on neural network-based image segmentation. 2. The method of claim 1, wherein depth values are determined on a pixel-by-pixel basis. 2. The method of claim 1, wherein determining the depth value comprises one or both of determining a depth estimate and fine-tuning the determined depth estimate. 9. The method of claim 8, wherein the depth estimate is determined from left and right stereo pairs that make up the input image. 9. The method of Claim 8, wherein the depth estimate is predicted using a neural network. 9. The method of claim 8, wherein the determined depth estimate is fine-tuned using a neural network. 2. The method of claim 1, wherein the predetermined perspective includes an orthographic view. 2. The method of claim 1, wherein performing the perspective transform comprises one or both of determining a perspective transform estimate and fine-tuning the determined perspective transform estimate. including, method. 14. The method of claim 13, wherein the perspective transformation estimate is determined from a mathematical transformation. 14. The method of claim 13, wherein the perspective transform estimate is predicted using a neural network. 14. The method of claim 13, wherein the perspective transformation estimate is fine-tuned using a neural network. 2. The method of claim 1, wherein the machine learning-based framework comprises one or more neural networks. 2. The method of claim 1, wherein the machine learning-based framework comprises a generative adversarial network (GAN). a system,
a processor,
remove the background of the input image of the object or asset,
determining a depth value for the object or asset in the input image;
a processor configured to generate an output image that includes a predetermined perspective of the object or asset that differs from the perspective of the input image by performing a perspective transformation based on the determined depth values;
a memory coupled to the processor and configured to provide instructions to the processor;
with
A system, wherein a machine learning-based framework is utilized for one or more steps of said processor, said machine learning-based framework being constrained to a known predetermined environment. A computer program product embodied in a persistent computer-readable storage medium,
computer instructions for removing the background of an input image of an object or asset;
computer instructions for determining a depth value of the object or asset in the input image;
computer instructions for generating an output image that includes a predetermined perspective of the object or asset that differs from the perspective of the input image by performing a perspective transformation based on the determined depth values;
with
A computer program product wherein a machine learning based framework is utilized for one or more steps of said computer program product, said machine learning based framework being constrained to a known predetermined environment.

Images