JP2022553844A - Generating Arbitrary Views - Google Patents

Abstract

A technique is disclosed for generating arbitrary views or perspectives of an ensemble scene. In some embodiments, in response to receiving a request for a given perspective of an ensemble scene including multiple assets, an output image of the ensemble scene for the requested given perspective is generated for each of at least a portion of the multiple assets. is generated based at least in part on combining at least a portion of the existing images of the . [Selection drawing] Fig. 5

Description

CROSS REFERENCE TO OTHER APPLICATIONS This application is a continuation-in-part of U.S. Patent Application Serial No. 16/ 171,221 , filed October 25, 2018, entitled "ARBITRARY VIEW GENERATION", the latter of which is entitled "ARBITRARY VIEW GENERATION No. 15/721,421 (now U.S. Pat. No. 10,163,249), filed September 29, 2017, entitled "ARBITRARY VIEW GENERATION." is a continuation-in-part of U.S. patent application Ser. incorporated herein by. U.S. patent application Ser. 541,607, the latter of which is incorporated herein by reference for all purposes.

This application claims priority from U.S. Provisional Patent Application No. 62/933,254, filed November 8, 2019, entitled "QUANTIZED PERSPECTIVE CAMERA VIEWS," which is incorporated herein by reference for all purposes. claim.

Existing rendering techniques face trade-offs between the competing goals of quality and speed. High quality rendering requires significant processing resources and time. However, slow rendering techniques are unacceptable for many applications, such as interactive real-time applications. Generally, lower quality but faster rendering techniques are preferred for such applications. For example, rasterization is commonly utilized by real-time graphics applications, sacrificing quality for relatively fast rendering. Therefore, there is a need for an improved technique that does not significantly impair quality or speed.

Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

1 is a high-level block diagram illustrating one embodiment of a system for generating arbitrary views of a scene; FIG.

The figure which shows an example of a database asset.

4 is a flowchart illustrating one embodiment of a process for generating arbitrary perspectives;

FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object. FIG. 4 illustrates an example embodiment of an application in which independent objects are combined to produce an ensemble or composite object.

4 is a flowchart illustrating one embodiment of a process for generating arbitrary ensemble views.

The present invention may be a process, apparatus, system, composition of matter, computer program product embodied on a computer readable storage medium, and/or a processor (stored in and/or by a memory coupled to the processor). implemented in various forms, including a processor configured to execute the provided instructions). In this specification, these examples, or any other form that the invention may take, may be referred to as techniques. In general, the order of steps in disclosed processes may be altered within the scope of the invention. Unless otherwise stated, a component such as a processor or memory described as being configured to perform a task is a generic component temporarily configured to perform the task at a time; , may be implemented as specific components manufactured to perform a task. As used herein, the term "processor" shall refer to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention includes many alternatives, modifications and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. These details are for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the sake of brevity, technical material that is well known in the technical fields related to the invention has not been described in detail so as not to unnecessarily obscure the invention.

Techniques are disclosed for generating arbitrary views of a scene. The examples described herein also provide high definition output with very low processing or computational overhead, effectively eliminating difficult tradeoffs between rendering speed and quality. The disclosed technique is particularly effective for producing high quality output very quickly for interactive real-time graphics applications. Such applications rely on the substantially immediate presentation of desirable high quality output in response to and in accordance with user manipulation of a presented interactive view or scene.

FIG. 1 is a high-level block diagram illustrating one embodiment of a system 100 for generating arbitrary views of a scene. As shown, an arbitrary view generator 102 receives a request for an arbitrary view as input 104, generates the requested view based on existing database assets 106, responds to the input request, The generated view is provided as output 108 . In various embodiments, the arbitrary view generator 102 may comprise a processor such as a central processing unit (CPU) or graphics processing unit (GPU). The configuration of system 100 shown in FIG. 1 is presented for illustration. In general, system 100 may include any other suitable number and/or configuration of interconnected components that provide the described functionality. For example, in another embodiment, the arbitrary view generator 102 may comprise different configurations of internal components 110-116, and the arbitrary view generator 102 may comprise multiple parallel physical and/or virtual processors. Often, the database 106 may comprise multiple network databases or a cloud of assets, and so on.

Arbitrary view requests 104 include requests for arbitrary perspectives of the scene. In some embodiments, other perspectives of the scene, ie, the requested perspective of the scene containing the viewpoint, do not yet exist in the asset database 106 . In various embodiments, the arbitrary view request 104 may be received from a process or user. For example, input 104 may be received from a user interface in response to user manipulation of a presented scene or portion thereof (such as user manipulation of a camera view of the presented scene). In another example, the arbitrary view request 104 may be received in response to specifying a path of movement or movement within the virtual environment, such as a flythrough of a scene. In some embodiments, the possible arbitrary views of the scene that can be requested are at least partially constrained. For example, a user may not be able to steer the camera viewpoint of a presented interactive scene to arbitrary random positions and is constrained to a particular position or perspective of the scene.

Database 106 stores multiple views of each stored asset. In a given context, an asset is an individual scene whose specifications are stored in the database 106 as multiple views. In various embodiments, a scene may include a single object, multiple objects, or a rich virtual environment. Specifically, database 106 stores multiple images corresponding to different perspectives or viewpoints of each asset. The images stored in database 106 include high quality photographs or photorealistic renderings. Such high definition or high resolution images that are input to database 106 may be captured or rendered during offline processing, or obtained from external sources. In some embodiments, corresponding camera characteristics are stored with each image stored in database 106 . That is, camera attributes such as relative position or location, orientation, rotation, depth information, focal length, aperture, zoom level are stored with each image. In addition, camera optical information such as shutter speed and exposure may be stored with each image stored in database 106 .

Any number of different perspectives of an asset may be stored in database 106 in various embodiments. FIG. 2 shows an example of a database asset. In the example given, 73 views corresponding to different angles around the chair object are captured or rendered and stored in database 106 . The view may be captured, for example, by rotating the camera around the chair or rotating the chair in front of the camera. Relative object and camera position and orientation information is stored with each image generated. FIG. 2 illustrates a view of a scene containing one object. Database 106 may also store specifications for scenes containing multiple objects or rich virtual environments. In such cases, multiple views corresponding to different positions or locations in the scene or three-dimensional space are captured or rendered and stored in database 106 along with corresponding camera information. In general, the images stored in database 106 may include two or three dimensions and may include stills or frames of animations or video sequences.

In response to a request 104 for an arbitrary view of a scene that does not already exist in database 106 , arbitrary view generator 102 generates the requested arbitrary view from multiple other existing views of the scene stored in database 106 . In the example configuration of FIG. 1, the asset management engine 110 of the arbitrary view generator 102 manages the database 106 . For example, asset management engine 110 may facilitate storing and retrieving data in database 106 . In response to a request for an arbitrary view of scene 104 , asset management engine 110 identifies and retrieves multiple other existing views of the scene from database 106 . In some embodiments, asset management engine 110 retrieves all existing views of the scene from database 106 . Alternatively, asset management engine 110 may select a portion of an existing view (eg, the view closest to any requested view) to retrieve. In such cases, the asset management engine 110 is configured to intelligently select some existing view from which pixels may be collected to generate the requested arbitrary view. In various embodiments, multiple existing views may be retrieved together by the asset management engine 110 or by other components of the arbitrary view generator 102 as needed.

Each existing view perspective retrieved by the asset management engine 110 is transformed into the requested arbitrary view perspective by the perspective transformation engine 112 of the arbitrary view generator 102 . As noted above, the exact camera information is known and stored with each image stored in database 106 . Perspective changes to any existing view-requested view therefore involve simple geometric mappings or transformations. In various embodiments, perspective transformation engine 112 may use any one or more suitable mathematical techniques to transform the perspective of an existing view into the perspective of an arbitrary view. If the requested view includes an arbitrary view that is not identical to any existing view, the conversion of the existing view to the perspective of the arbitrary view will result in at least some unmapped or missing pixels, i.e. It will contain pixels at angles or positions introduced into arbitrary views that do not exist.

Pixel information from a single perspective-transformed existing view cannot fill all the pixels of another view. However, in many cases, most, if not all, of the pixels of any requested view can be collected from multiple perspective-transformed existing views. A merge engine 114 of the arbitrary view generator 102 combines pixels from multiple perspective-transformed existing views to generate the requested arbitrary view. Ideally, all pixels that make up an arbitrary view are collected from existing views. This is possible, for example, if a sufficiently diverse set of existing views or perspectives are available for the considered asset and/or if the requested perspective is not significantly different from the existing perspectives. can be

Any suitable technique may be used to combine or merge pixels from multiple perspective-transformed existing views to generate the desired arbitrary view. In one embodiment, the first existing view closest to the requested arbitrary view is selected and retrieved from the database 106 and transformed into the perspective of the requested arbitrary view. Pixels are then collected from this perspective-transformed first existing view and used to fill the corresponding pixels in the requested arbitrary view. To fill in the pixels of the requested arbitrary view that could not be obtained from the first existing view, a second existing view containing at least some of these remaining pixels is selected and retrieved from the database 106 to provide the perspective of the requested arbitrary view. is converted to Pixels that could not be obtained from the first existing view are then collected from this perspective transformed second existing view and used to fill the corresponding pixels in the requested arbitrary view. This process continues until all pixels of the requested arbitrary view are filled and/or until all existing views are exhausted or a predetermined threshold number of existing views are utilized. may be repeated for

In some embodiments, the requested arbitrary view may contain some pixels that could not be obtained from any existing view. In such cases, interpolation engine 116 is configured to fill in all remaining pixels of the requested arbitrary view. In various embodiments, any one or more suitable interpolation techniques may be used by interpolation engine 116 to generate these unfilled pixels in the requested arbitrary view. Examples of interpolation techniques that can be used include, for example, linear interpolation, nearest neighbor interpolation, and the like. Pixel interpolation introduces averaging or smoothing. Overall image quality is not significantly affected by some interpolation, but excessive interpolation can introduce unacceptable blurriness. Therefore, it may be desirable to use interpolation sparingly. As noted above, interpolation is avoided entirely if all pixels of the requested arbitrary view can be obtained from existing views. However, if the requested arbitrary view contains some pixels that cannot be obtained from any view, interpolation is introduced. In general, the amount of interpolation required depends on the number of existing views available, the diversity of perspectives of the existing views, and/or how the perspective of any given view differs with respect to the perspective of the existing views.

For the example shown in FIG. 2, 73 views around the chair object are stored as existing views of the chair. An arbitrary view around the chair object that is different or unique from any of the stored views can be generated using a plurality of these existing views, preferably with minimal, if any, interpolation. However, it may not be efficient or desirable to generate and store such a comprehensive set of existing views. In some cases, a sufficiently small number of existing views may instead be generated and stored that cover a sufficiently diverse set of perspectives. For example, 73 views of a chair object may be reduced to a small set of a few views around the chair object.

As noted above, in some embodiments the possible arbitrary views that can be requested may be at least partially constrained. For example, a user may be restricted from moving a virtual camera associated with an interactive scene to certain positions. For the example given in Figure 2, the possible arbitrary views that can be requested are limited to arbitrary positions around the chair object, e.g. It cannot include any position under the chair object. Such constraints on allowed arbitrary views ensure that the requested arbitrary view can be generated by the arbitrary view generator 102 from existing data.

Arbitrary view generator 102 generates and outputs requested arbitrary view 108 in response to input arbitrary view request 104 . The resolution or quality of the generated arbitrary view 108 is the same as the quality of the existing view used to generate it, since pixels from the existing view are used to generate the arbitrary view, or are equivalent. Therefore, using high definition existing views in most cases yields high definition output. In some embodiments, the generated arbitrary view 108 is stored in the database 106 along with other existing views of the relevant scene, and later, in response to future requests for arbitrary views, other arbitrary views of the scene are generated. may be used to generate If the input 104 contains a request for an existing view in the database 106, then the request view need not be generated from other views as described above; instead the request view can be created using a simple database lookup. retrieved and presented directly as output 108 .

Arbitrary view generator 102 may also be configured to generate arbitrary ensemble views using the described techniques. That is, input 104 may include a request to combine multiple objects into a single custom view. In such cases, the techniques described above are performed on each of the multiple objects and combined to produce a single unified or ensemble view containing the multiple objects. Specifically, existing views of each of the plurality of objects are selected and retrieved from the database 106 by the asset management engine 110, and those existing views are transformed by the perspective transformation engine 112 into the perspective of the requested view. , the pixels from the perspective-transformed existing views are used by the merge engine 114 to fill the corresponding pixels of the requested ensemble view, and any remaining unfilled pixels in the ensemble view are used by the interpolation engine 116. In some embodiments, the requested ensemble view may include perspectives that already exist for one or more of the objects that make up the ensemble. In such cases, an existing view of the object asset corresponding to the requested perspective is used to directly fill the pixels corresponding to the object in the ensemble view, instead of first generating the requested perspective from other existing views of the object. Used.

As an example of an arbitrary ensemble view containing multiple objects, consider the chair object and separately photographed or rendered table object in FIG. A chair object and a table object may be combined using the disclosed techniques to produce a single ensemble view of both objects. Thus, using the disclosed techniques, separately captured or rendered images or views of each of multiple objects can be consistently combined to produce a scene containing multiple objects and having a desired perspective. . As mentioned above, the depth information for each existing view is known. The perspective transformation of each existing view, including the depth transformation, allows multiple objects to be properly positioned relative to each other within the ensemble view.

Generating arbitrary ensemble views is not limited to combining multiple single objects into custom views. Rather, multiple scenes with multiple objects or multiple rich virtual environments may be similarly combined into a custom ensemble view. For example, multiple separately and independently generated virtual environments (perhaps from different content generation sources and possibly with different pre-existing individual perspectives) are combined into an ensemble view with a desired perspective. good. Thus, in general, the arbitrary view generator 102 may be configured to consistently combine or match multiple independent assets, including possibly different existing views, into an ensemble view with a desired, possibly arbitrary perspective. Since all combined assets are normalized to the same perspective, a perfectly matched resulting ensemble view is produced. The possible arbitrary perspectives of the ensemble view can be constrained based on existing views of individual assets available to generate the ensemble view.

FIG. 3 is a flow diagram illustrating one embodiment of a process for generating arbitrary perspectives. Process 300 may be used, for example, by arbitrary view generator 102 of FIG. In various embodiments, process 300 may be used to generate arbitrary or arbitrary ensemble views of a given asset.

Process 300 begins at step 302 where a request for an arbitrary perspective is received. In some embodiments, the request received at step 302 may include a request for an arbitrary perspective of a given scene that differs from any existing available perspective of the scene. In such cases, for example, an arbitrary perspective request may be received in response to a requested change in perspective of the presented view of the scene. Such changes in perspective may be prompted by changes or manipulations of the virtual camera relative to the scene, such as camera pans, focal length changes, zoom level changes, and the like. Alternatively, in some embodiments, the request received at step 302 may include a request for arbitrary ensemble views. As an example, such an arbitrary ensemble view request may be received for an application that allows selection of multiple independent objects and provides an integrated perspective-modified ensemble view of the selected objects.

At step 304, a plurality of existing images from which to generate at least a portion of the requested arbitrary perspective are retrieved from one or more related asset databases. Multiple retrieved images may be associated with a given asset if the request received at step 302 includes a request for an arbitrary perspective of the given asset, and the request received at step 302 may be any If it contains an ensemble view request, it may relate to multiple assets.

At step 306 , each of the plurality of existing images retrieved at step 304 having different perspectives is converted to the arbitrary perspective requested at step 302 . Each of the existing images retrieved in step 304 includes associated perspective information. The perspective of each image is defined by the camera properties associated with generating that image, such as relative position, orientation, rotation, angle, depth, focal length, aperture, zoom level, and lighting information. Since complete camera information is known for each image, the perspective transformation of step 306 involves simple mathematical operations. In some embodiments, step 306 optionally further includes an optical transformation such that all images are consistently normalized to the same desired lighting conditions.

At step 308, at least a portion of the image having the arbitrary perspective requested in step 302 is filled with pixels collected from an existing image that has been perspective transformed. That is, pixels from multiple perspective-corrected existing images are used to generate an image with a desired arbitrary perspective.

At step 310, it is determined whether the generated image with the requested arbitrary perspective is complete. If it is determined in step 310 that the generated image with the requested arbitrary perspective is not complete, then additional existing images are available to obtain any remaining unfilled pixels of the generated image. It is determined in step 312 whether . If additional existing images are available as determined at step 312 , one or more additional existing images are retrieved at step 314 and process 300 proceeds to step 306 .

If it is determined in step 310 that the generated image with the requested arbitrary perspective is not complete, and if it is determined in step 312 that no existing image is available, all remaining padding of the generated image is performed. Pixels that are not interpolated are interpolated in step 316 . Any one or more suitable interpolation techniques may be used at step 316 .

If it is determined in step 310 that the generated image with the requested arbitrary perspective is complete, or after interpolating all remaining unfilled pixels in step 316, generate with the requested arbitrary perspective. The finished image is output at step 318 . Thereafter, process 300 ends.

As noted above, the disclosed techniques may be used to generate arbitrary perspectives based on other existing perspectives. Since camera information is stored with each existing perspective, it is possible to normalize different existing perspectives to a common desired perspective. A resulting image with a desired perspective can be constructed by taking pixels from an existing image that has been perspective transformed. The disclosed technique is well suited for interactive real-time graphics applications because the processing associated with generating arbitrary perspectives using the disclosed technique is not only fast and near-instantaneous, but also produces high-quality output. It is a particularly powerful technology.

The disclosed technique further describes generating an arbitrary ensemble view containing multiple objects using available images or views of each of the multiple objects. As described above, perspective transformation and/or normalization allows pixels comprising separately captured or rendered images or views of multiple objects to be consistently combined into any desired ensemble view. to

In some embodiments, it may be desirable to first construct or assemble the scene or ensemble view by selecting and arranging the content that is desired to be included in the scene or ensemble view. In some such cases, multiple objects may be stacked or combined like building blocks to create a composite object comprising a scene or ensemble view. As an example, consider an interactive application in which multiple independent objects are selected and appropriately positioned, eg, on a canvas, to create a scene or ensemble view. Interactive applications may include, for example, visualization or modeling applications. In such applications, arbitrary views of objects cannot be used to construct scene or ensemble views due to perspective distortions due to the focal lengths involved. Rather, a predetermined object view with substantially no perspective distortion is utilized as described below.

Orthographic views of objects are used in some embodiments to model or define a scene or ensemble view containing multiple independent objects. An orthographic view is placed at a far distance from the object for its size, such that the rays or projection lines are substantially parallel, approximated by a (virtual) camera with a relatively long focal length. Including projection. An orthographic view has no depth, or has a fixed depth, so there is little or no perspective distortion. Orthographic views of objects may thus be used like building blocks when specifying ensemble scenes or composite objects. After an ensemble scene containing any combination of objects has been specified or defined using such orthographic views, the scene or its objects can be generated using arbitrary view generation techniques described above with respect to the description of FIGS. may be converted to the desired camera perspective of

In some embodiments, the multiple views of an asset stored in database 106 of system 100 of FIG. 1 include one or more orthographic views of the asset. Such orthographic views may be captured (eg, photographed or scanned) or rendered from the 3D polygonal mesh model. Alternatively, orthographic views may be generated from other views of assets available in database 106 according to the arbitrary view generation techniques described above with respect to the description of FIGS. 1-3.

Figures 4A-4N illustrate example embodiments of applications in which independent objects are combined to produce an ensemble or composite object or scene. Specifically, Figures 4A-4N illustrate an example of a furniture construction application in which various individual sofa components are combined to create different unit sofa configurations.

FIG. 4A is an example of a perspective view showing three separate sofa components (ie, a single seat with left arm, a love seat without arm, and a chaise longue with right arm). The perspective views in the example of FIG. 4A each have a focal length of 25 mm. As can be seen in the figure, the resulting perspective distortion causes the stacking of the components next to each other (i.e., the side-by-side placement of the components), which may be desired when assembling a unit sofa construction containing the components. there is).

FIG. 4B shows an example of an orthographic view of the same three components as in FIG. 4A. As shown, an orthographic view of an object is modular or blocky and suitable for being stacked or placed next to each other. However, depth information is substantially lost in orthographic views. As can be seen, in FIG. 4A there is a difference in depth, especially for the chaise longue, but in the orthographic view all three appear to have the same depth.

FIG. 4C shows an example of combining orthographic views of the three components of FIG. 4B to define a composite object. That is, FIG. 4C shows the generation of an orthographic view of a unit sofa by placing the orthographic views of the three components of FIG. 4B next to each other. As shown in FIG. 4C, the orthographic view bounding boxes of the three sofa components fit adjacent to each other to create the unit sofa orthographic view. That is, an orthographic view of a component facilitates user-friendly manipulation and precise placement of the component within the scene.

4D and 4E each show an example of transforming the orthographic view of the composite object of FIG. 4C to an arbitrary camera perspective using the arbitrary view generation techniques described above with respect to the description of FIGS. 1-3. That is, in each of the examples of Figures 4D and 4E, the orthographic view of the composite object has been transformed into a normal camera perspective that accurately depicts depth. As shown, the relative depth of the chaiselongue to the singles and loveseats, which was lost in the orthographic view, is now visible in the perspective views of Figures 4D and 4E.

Figures 4F, 4G, and 4H show examples of multiple orthographic views of a one-seater with left arm, a two-seater without arm, and a chaise longue with right arm, respectively. As noted above, any number of different views or perspectives of an asset may be stored in database 106 of system 100 of FIG. The set of FIGS. 4F-4H includes 25 orthographic views corresponding to different angles around each asset separately captured or rendered and stored in database 106, from which the object Any arbitrary view of any combination of . In furniture construction applications, for example, the top view may be useful for floor placement, while the front view may be useful for wall placement. In some embodiments, to maintain a more compact reference data set, only a predetermined number of orthographic views are stored for an asset in database 106, from which any arbitrary view of the asset is generated. may be

Figures 4I-4N show various examples of generating arbitrary views or perspectives of arbitrary combinations of objects. Specifically, each of Figures 4I-4N illustrates the generation of an arbitrary perspective or view of a unit sofa containing multiple separate sofa objects or components. Each arbitrary view represents one or more orthographic (or other) views of the objects that make up the ensemble view or composite object, using, for example, the arbitrary view generation techniques described above with respect to the description of FIGS. , taking pixels to fill any view and possibly interpolating any remaining missing pixels.

As described above, each image or view of an asset in database 106 may be stored with corresponding metadata (such as relative object and camera position and orientation information and lighting information). Metadata may be used when rendering a view from a 3D polygonal mesh model of an asset, imaging or scanning the asset (in which case depth and/or surface normal data may be estimated), or both. may be produced when combined.

A given view or image of an asset includes pixel intensity values (eg, RGB values) for each pixel in the image and various metadata parameters associated with each pixel. In some embodiments, one or more of a pixel's red, green, and blue (RGB) channels or values may be used to encode pixel metadata. Pixel metadata may include, for example, information about the relative location or position (eg, x, y, and z coordinate values) of a point in three-dimensional space that is projected onto that pixel. Additionally, the pixel metadata may include information about the surface normal vector at that location (eg, angles with the x, y, and z axes). Pixel metadata may also include texture mapping coordinates (eg, u and v coordinate values). In such cases, the actual pixel value at the point is determined by reading the RGB values of the corresponding coordinates in the texture image.

Surface normal vectors facilitate modifying or changing the lighting of any generated view or scene. More specifically, scene lighting changes determine how well a pixel's surface normal vector matches the direction of newly added, removed, or otherwise changed light sources (e.g., light source direction and the pixel's normal vector, which can be quantified at least in part by the dot product of the pixel's normal vector. Using texture mapping coordinates to define pixel values facilitates modifying or altering the texture of any generated view or scene or portion thereof. More specifically, the texture can be modified by simply exchanging or replacing the referenced texture image with another texture image having the same dimensions.

The disclosed arbitrary view generation techniques are effectively based on relatively low computational cost perspective transformation and/or lookup operations. Arbitrary (ensemble) views may be generated by simply selecting the correct pixels and appropriately filling the generated arbitrary view with those pixels. In some cases, pixel values may optionally be scaled, for example, when lighting is being adjusted. The low storage and processing overhead of the disclosed technique enables fast, real-time, or on-demand generation of arbitrary views of complex scenes with quality comparable to the high-definition reference views from which they are generated. make it easier.

As noted above, assembling an ensemble or composite object or scene in some embodiments involves using an orthographic view to specify a plurality of object assets that make up the ensemble. Orthographic views facilitate accurate placement and alignment of multiple objects or assets in an ensemble scene. The orthographic view of the ensemble scene may then be transformed to any arbitrary camera perspective, eg, to generate any desired or required perspective. Transforming the ensemble view to the predetermined camera perspective may include individually transforming each of the multiple objects or assets that make up the ensemble scene to the predetermined perspective using the techniques described above. While the above-described techniques for generating arbitrary ensemble views are relatively efficient, even more efficient is the ability to generate almost instantaneous or at least very fast, with almost imperceptible latency penalties for end-users. , may be desirable in certain applications where it is advantageous to generate output (eg, applications that provide interactive, real-time experiences to users).

In some embodiments, further improvements in efficiency relate to transforming a majority of the objects or assets that make up the ensemble scene (e.g., an orthographic or other view thereof) to a predetermined arbitrary perspective. Elimination of processing may at least partially facilitate. Instead, the output ensemble view or image representing the given arbitrary perspective is the available existing view of the object or asset that is closest or most similar to the given arbitrary perspective for a given position and orientation of the object or asset in the ensemble scene. Used for an object or asset when creating a . In most cases, the resulting output ensemble view is not perfectly perspective-accurate, but is acceptable for many applications, with significantly lower latency than producing fully perspective-accurate output. Provide a good approximation to be generated. We now describe in more detail the generation of such an approximation of an arbitrary ensemble view for an arbitrary camera pose using a maximally quantized subset of pre-existing reference views of one or more of the objects or assets that make up the ensemble. do.

FIG. 5 is a high-level flowchart illustrating one embodiment of a process for generating arbitrary ensemble views. In some embodiments, process 500 suitably combines or synthesizes a single best match existing view of at least one or more objects or assets (if not most or all) that make up the ensemble scene. Based at least in part on the ensemble scene output image, it is used to efficiently generate an output image of the ensemble scene.

Process 500 begins at step 502 where a request for a given perspective of an ensemble scene is received. The requested predetermined perspective of the ensemble scene includes the camera perspective selected or otherwise specified for the ensemble scene, and may generally include any arbitrary view. Arbitrary views in a given context include any desired view or perspective of a scene for which the specifications or camera poses are not known in advance prior to the request. An ensemble scene contains a composite view of multiple independent objects or assets. In general, an independent object or asset specification includes a set of existing reference images or views of individual objects or assets with different camera perspectives and corresponding metadata, one or more of which are may be used to generate or specify the portion of the ensemble scene associated with the . In some embodiments, the request of step 502 facilitates manipulation of camera angles or poses in ensemble scene space and/or placement of multiple objects or assets to create a composite or ensemble scene. is received from an interactive mobile or web-based application that For example, the request may be received from a visualization application or a modeling application or an augmented reality (AR) application. In some embodiments, the orthographic view facilitates easier manipulation, placement, and alignment of multiple objects or assets that make up the ensemble scene, so the request of step 502 may be an orthographic view of the ensemble scene. Received for a view.

At step 504, the closest or most similar matching existing reference image or view is selected for each of at least some of the one or more objects or assets that make up the ensemble scene. Step 504 may be performed sequentially and/or in parallel on individual or independent objects or assets that make up the ensemble scene. In some embodiments, for a given pose of an object or asset in the ensemble scene space, only one or single existing reference image or view that best matches the desired predetermined perspective is attached to the object or asset. selected against. The ensemble scene space comprises an ensemble scene coordinate system with a given origin (such as the ensemble scene center (eg, centroid)) defined in a suitable manner. In step 504, the positions and orientations or poses of the objects or assets with respect to the ensemble scene coordinate system are determined to select the closest matching existing reference image or view for the objects or assets that make up the ensemble scene. or transformed or otherwise correlated to equivalent poses in individual coordinate systems associated with existing reference images or views of the asset. Therefore, simple camera metric calculations with relatively low computational complexity can be used to determine the perspective and relative objects or assets required in the ensemble scene so that the closest matching existing reference image or view can be selected in step 504. is performed based on the posture of

One or more criteria and/or thresholds may be defined for determining or identifying the closest matching existing reference image or view for an object or asset. In some cases, an existing reference image or view is selected at step 504 only if one or more such thresholds are met. In the ideal case, a perfect match is found and selected in step 504 . However, in some cases, if the available existing reference image dataset is too incomplete (such as when the available existing reference image or view of the object or asset is significantly different from the requested perspective), Alternatively, one or more selection criteria and/or thresholds may not be met if there is no reference image or view available for the object or asset. In some such cases, the closest matching placeholder or ghost image or view of the object or asset is selected in step 504 instead. Such placeholder images or views represent the shape of the object or asset, but lack other attributes (such as texture and optical properties). In some embodiments, a set of placeholder images spanning a sufficiently dense set of possible views around the object or asset (e.g., including angles covering 360° around the object or asset) It is generated and stored for each unique object shape using rendering techniques of relatively low computational complexity. Placeholders are used when a fully rendered version of an object or asset is not available or exhibits an unacceptable deviation from the requested perspective.

At step 506, an output image of the ensemble scene is generated, at least in part, by appropriately combining or synthesizing the closest matching existing reference images or views of the objects or assets that make up the ensemble scene selected at step 504. , is generated for the given perspective requested. At step 506, appropriately scaling or resizing the selected closest matching existing reference image or view for the object or asset and/or the selected closest matching existing reference image or view for the object or asset. determining a place or position within the ensemble view to paste or composite the view. In most cases, the generated output image of the ensemble scene closely approximates the desired predetermined perspective. Since most objects or assets that make up an ensemble scene are represented in the output image with the closest or most similar existing pose available to them, these objects or assets are not strictly rendered or generated. , the perspective is not entirely accurate. That is, in most cases these objects or assets will not have the required predetermined perspective in the output image unless an exact match is found among the existing images or views available. Although not all vanishing points of such objects or assets will point to the same point in the output image, objects or assets will most likely be viewed by the human eye so as to perceive the output image as the correct perspective for the most part. It is offset or tilted by an amount small enough (eg, a few degrees) to create an illusion of the system.

Consistency in the output image of an ensemble scene is further facilitated by generating at least some portions of the ensemble scene in a globally consistent or similar manner, which further facilitates the ability of humans to reproduce the output image substantially. make it easier to interpret as visually accurate. For example, one or more objects or assets that make up the ensemble scene, and/or surfaces (planar or otherwise), structural elements, global features, etc. That is, it may be rendered or generated exactly to have the desired predetermined perspective rather than an approximation of the requested perspective. For example, if the ensemble scene includes spaces such as rooms, walls, ceilings, floors, rugs, wall hangings, etc. can be generated with the required perspective camera pose, so the ensemble scene generated in step 506 can be accurately represented in the output image of In addition, the output image of the ensemble scene is a global lighting position that affects all parts of the scene in a similar and consistent manner when changing lighting, e.g. with available metadata (surface normal vectors, etc.) may be provided. Therefore, by generating some part of the ensemble view in a global way or in a way that modifies the perspective, and representing the most independent objects that make up the ensemble view as a best approximation, often completely perspective Produces output that is almost indistinguishable from the exact version. In some cases some slant may be seen, but not perfect, such as mood board applications or space/room planning applications where the designer or user would benefit from seeing an ensemble of objects or assets regardless of the accuracy of perspective. It may still be acceptable for certain applications that do not require an accurate view of . That said, as the repository or database of existing images or views available grows over time, the disclosed techniques will continue to produce output that more and more accurately represents the desired given perspective. In the best case, an exact match is found for all objects or assets and used to produce an output image that actually has the desired given perspective rather than an approximation.

Although the above embodiments are described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative, not limiting.

Claims (23)

a method,
receiving a request for a given perspective of an ensemble scene containing multiple assets;
generating an output image of the ensemble scene that approximates the requested predetermined perspective based at least in part on combining a single existing image of each of at least a portion of the plurality of assets;
A method. 2. The method of claim 1, wherein the request is received for an orthographic view of the ensemble scene. 3. The method of claim 2, wherein the orthographic view of the ensemble scene comprises a combined orthographic view of the plurality of assets. 2. The method of claim 1, further comprising selecting the single existing image of each of the at least a portion of the plurality of assets. 5. The method of claim 4, wherein said selecting comprises selecting an exact match with said requested predetermined perspective. 5. The method of claim 4, wherein the selecting comprises selecting an available match that is closest or most similar to the requested predetermined perspective. 5. The method of claim 4, wherein the selecting comprises selecting based on poses of related assets within the ensemble scene. 5. The method of claim 4, wherein the selecting comprises selecting a rotated existing image of a related asset. 5. The method of claim 4, wherein the selecting includes the closest or most similar available match to the requested predetermined perspective based on poses of associated assets within the ensemble scene. A method comprising selecting. 2. The method of claim 1, wherein generating the output image of the ensemble scene comprises scaling the single existing image of one or more assets within the portion of assets. ,Method. 2. The method of claim 1, wherein generating the output image of the ensemble scene comprises resizing the single existing image of one or more assets within the portion of assets. ,Method. 2. The method of claim 1, wherein generating the output image of the ensemble scene includes determining where to include the single existing image of each of the at least some of the assets in the ensemble scene. method, including 2. The method of claim 1, wherein said combining comprises synthesizing. 2. The method of claim 1, wherein generating the output image of the ensemble scene comprises generating a view of at least one of the plurality of assets having the requested predetermined perspective. A method, including 15. The method of claim 14, wherein the views are generated using multiple existing images of the at least one asset. 2. The method of claim 1, wherein generating the output image of the ensemble scene comprises generating at least one portion of the ensemble scene to have the requested predetermined perspective. Method. 17. The method of Claim 16, wherein said at least one portion comprises a surface of said ensemble scene. 17. The method of claim 16, wherein said at least one portion comprises structural elements of said ensemble scene. 17. The method of claim 16, wherein said at least one portion includes global features of said ensemble scene. 2. The method of claim 1, further comprising globally lighting the generated output image of the ensemble scene. 2. The method of claim 1, wherein the output images comprise frames of a video sequence. a system,
a processor,
receiving a request for a given perspective of an ensemble scene containing multiple assets;
configured to generate an output image of the ensemble scene approximating the requested predetermined perspective based at least in part on combining a single existing image of each of at least a portion of the plurality of assets. there is a processor and
a memory coupled to the processor and configured to provide instructions to the processor;
A system comprising: A computer program product embodied in a persistent computer-readable storage medium,
computer instructions for receiving a request for a predetermined perspective of an ensemble scene including multiple assets;
Computer instructions for generating an output image of the ensemble scene that approximates the requested predetermined perspective based at least in part on combining a single existing image of each of at least a portion of the plurality of assets. When,
A computer program product comprising:

Images