KR20220076514A - arbitrary view creation - Google Patents

Abstract

Techniques for creating an arbitrary view or perspective of an ensemble scene are disclosed. In some embodiments, in response to a received request for a prescribed perspective of an ensemble scene comprising a plurality of assets, an output image of the ensemble scene for the requested prescribed perspective is an existing image of each of at least a subset of the plurality of assets. generated based at least in part on a get that combines at least a portion of the image.

Description

arbitrary view creation

Cross-references to other applications

This application is a continuation-in-part of U.S. Patent Application No. 16/171,221, titled ARBITRARY VIEW GENERATION, filed on October 25, 2018, which is filed on September 29, 2017 ( U.S. Patent Application No. 15/721,421 entitled ARBITRARY VIEW GENERATION, which is currently a continuation of U.S. Patent No. 10,163,249, which is a U.S. patent application titled ARBITRARY VIEW GENERATION, filed March 25, 2016 No. 15/081,553 is now a continuation-in-part of U.S. Patent No. 9,996,914, all of which are incorporated herein by reference for all purposes. U.S. Patent Application Serial No. 15/721,421, now U.S. Patent No. 10,163,249, is in U.S. Provisional Patent Application Serial No. 62/541,607, entitled FAST RENDERING OF ASSEMBLED SCENES, filed on August 4, 2017. priority, which is incorporated herein by reference for all purposes.

This application claims priority to U.S. Provisional Patent Application No. 62/933,254, QUANTIZED PERSPECTIVE CAMERA VIEWS, filed on November 8, 2019, which is incorporated herein by reference for all purposes. .

The present invention relates to arbitrary view creation.

Existing rendering technologies must strike a balance between competing goals of quality and speed. High-quality rendering requires significant processing resources and time. However, slow rendering techniques are not acceptable in many applications, such as interactive real-time applications. For these applications, generally lower quality but faster rendering techniques are preferred. For example, rasterization is commonly used by real-time graphics applications for relatively fast rendering, but at a lower quality.

Accordingly, there is a need for improved techniques that do not significantly compromise quality or speed.

A method is provided. The method includes: receiving a request for a prescribed perspective of an ensemble scene comprising a plurality of assets; and generating an output image of the ensemble scene that approximates the requested prescribed perspective based at least in part on combining a single existing image of each of at least a subset of the plurality of assets. In the method, the request is received for an orthographic view of an ensemble scene. In the method, the orthogonal view of the ensemble scene comprises combined orthogonal views of a plurality of assets. The method further includes selecting a single existing image of each of at least a subset of the plurality of assets. In the method, said selecting comprises selecting an exact match with a requested defined perspective. In the method, the selecting comprises selecting the closest available match or closest to the requested defined perspective. In the method, said selecting comprises selecting based on a pose of an associated asset in the ensemble scene. In the method, said selecting comprises selecting an existing rotated image of an associated asset. In the method, the selecting comprises selecting the closest available match or closest to the requested prescribed perspective based on the pose of the associated asset in the ensemble scene. In the method, generating an output image of the ensemble scene comprises scaling a single existing image of one or more of the subset of assets. In the method, generating an output image of the ensemble scene comprises resizing a single existing image of one or more of the subset of assets. In the method, generating an output image of the ensemble scene comprises determining a position in the ensemble scene to contain a single existing image of each of at least a subset of assets. In the method, said binding comprises synthesizing. In the method, generating an output image of the ensemble scene comprises generating a view of at least one of a plurality of assets having a requested defined perspective. In the method, the view is created using a plurality of existing images for the at least one asset. In the method, generating an output image of the ensemble scene comprises generating at least a portion of the ensemble scene to have a requested prescribed perspective. In the method, said at least one portion comprises a surface of an ensemble scene. Said at least one portion comprises a structural element of the ensemble scene. In the method, said at least one portion comprises a global feature of an ensemble scene. The method further includes globally re-illuminating the generated output image of the ensemble scene. In the method, the output image comprises a frame of a video sequence.

A system is also disclosed. The system may be configured to: receive a request for a defined perspective of an ensemble scene including a plurality of assets; the processor configured to generate an output image of the ensemble scene that approximates a requested prescribed perspective based at least in part on combining a single existing image of each at least a subset of a plurality of assets; and a memory coupled to the processor and configured to provide instructions to the processor.

A computer program product embodied on a non-transitory computer-readable medium comprising computer instructions is disclosed, the computer instructions being configured to: receive a request for a defined perspective of an ensemble scene comprising a plurality of assets; to generate an output image of the ensemble scene approximating the requested prescribed perspective based at least in part on combining a single existing image of each of at least a subset of the plurality of assets.

Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings.
1 is a high-level block diagram illustrating an embodiment of a system for generating an arbitrary view of a scene.
2 shows an example of a database asset.
3 is a flow diagram illustrating an embodiment of a process for generating an arbitrary perspective.
4A-4N illustrate examples of an embodiment of an application in which independent objects are combined to create an ensemble or composite object.
5 is a flow diagram illustrating an embodiment of a process for generating an arbitrary ensemble view.

The present invention is a process; Device; system; composition of things; a computer program product contained in a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored in and/or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the invention may take may be referred to as techniques. In general, the order of steps in the disclosed processes may be varied within the scope of the present invention. Unless otherwise specified, a component such as a processor or memory that is described as being configured to perform a task may be implemented as a generic component temporarily configured to perform a task at a given time, or as a specific component built to perform a task. can As used herein, the term 'processor' means one or more devices, circuits and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below in conjunction with the accompanying drawings, which illustrate the principles of the invention. Although the present invention is described in connection with these embodiments, the present invention is not limited to any embodiments. The scope of the invention is limited only by the claims, and the invention includes numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. These details are provided for purposes of illustration, and the invention may be practiced in accordance with the claims without some or all of these specific details. For clarity, technical data known in the technical fields related to the present invention have not been described in detail, so as not to unnecessarily obscure the present invention.

Techniques for creating an arbitrary view of a scene are disclosed. The paradigm described here effectively eliminates the difficult trade-off between rendering speed and quality, which entails very low processing or computational overhead while still providing high-definition output. The disclosed techniques are particularly useful for generating high quality output very quickly in connection with interactive real-time graphics applications. Such applications rely on providing a desirable high-quality output substantially immediately upon or in response to user manipulations of a presented interactive view or scene.

1 is a high-level block diagram illustrating an embodiment of a system 100 for generating an arbitrary view of a scene. As shown, an arbitrary view generator 102 receives a request for an arbitrary view as an input 104 and generates the requested view based on existing database assets 106 . and provides as output 108 the generated view in response to the input request. In various embodiments, optional view generator 102 may include a processor, such as a central processing unit (CPU) or graphics processing unit (GPU). The configuration of the system 100 shown in FIG. 1 is provided for illustration. In general, system 100 may include any other suitable number and/or configuration of interconnected components that provide the described functionality. For example, in other embodiments, the arbitrary view generator 102 may include different configurations of internal components 110 - 116 , and the arbitrary view generator 102 may include a plurality of parallel physical and/or virtual It may include processors, and the database 106 may include a plurality of networked databases or a cloud of assets, and the like.

Arbitrary view request 104 includes a request for an arbitrary perspective of a scene. In some embodiments, the requested perspective of the scene does not previously exist in the asset database 106 containing another perspective or viewpoint of the scene. In various embodiments, an arbitrary view request 104 may be received from a process or a user. For example, the input 104 may be received from the user interface in response to a user manipulation of the presented scene or a portion thereof, such as a user manipulation of a camera viewpoint of the presented scene. As another example, the arbitrary view request 104 may be received in response to designation of a movement or movement path within a virtual environment, such as a fly-through of a scene. In some embodiments, possible arbitrary views of a scene that may be requested are at least partially limited. For example, the user cannot manipulate the camera viewpoint of the presented interactive scene to any random position, but rather may be limited to specific positions or perspectives of the scene.

The database 106 stores a plurality of views of each stored asset. In a given context, an asset refers to a particular scene whose specification is stored as a plurality of views in the database 106 . In various embodiments, a scene may include a single object, multiple objects, or a rich virtual environment. Specifically, the database 106 stores a plurality of images corresponding to different perspectives or viewpoints of each asset. The images stored in database 106 include high quality photos or photorealistic renderings. These high-definition, high-resolution images that populate the database 106 may be captured or rendered during an offline process or obtained from external sources. In some embodiments, corresponding camera characteristics are stored with each image stored in database 106 . That is, camera properties such as relative position or position, orientation, rotation, depth information, focal length, aperture, zoom level, etc. are stored with each image. In addition, camera lighting information such as shutter speed and exposure may also be stored with each image stored in database 106 .

In various embodiments, any number of different perspectives of an asset may be stored in database 106 . 2 shows an example of a database asset. In the given example, 73 views corresponding to different angles around the chair object are captured or rendered and stored in database 106 . For example, views may be captured by rotating the camera around the chair or rotating the chair in front of the camera. Relative object and camera position and orientation information is stored with each image generated. 2 specifically illustrates views of a scene comprising a single object. Database 106 may also store a specification of a scene including a plurality of objects or a rich virtual environment. In such a case, multiple views corresponding to different locations or positions in the scene or three-dimensional space are captured or rendered and stored together with the camera information corresponding to the database 106 . In general, the images stored in database 106 may include two or three dimensions and may include stills or frames of animation or video sequences.

In response to a request for an arbitrary view of the scene 104 that does not previously exist in the database 106 , the arbitrary view generator 102 generates the requested arbitrary view from a plurality of other existing views of the scene stored in the database 106 . create In the exemplary configuration of FIG. 1 , the asset management engine 110 of the arbitrary view generator 102 manages the database 106 . For example, the asset management engine 110 may facilitate storage and retrieval of data in the database 106 . In response to a request for an arbitrary view of the scene 104 , the asset management engine 110 identifies and obtains a plurality of other existing views of the scene from the database 106 . In some embodiments, the asset management engine 110 retrieves all existing views of the scene from the database 106 . Alternatively, the asset management engine 110 may select and retrieve a subset of existing views that are closest to, for example, any requested view. In such a case, the asset management engine 110 is configured to intelligently select a subset of existing views from which pixels may be collected to generate the requested arbitrary view. In various embodiments, multiple existing views may be retrieved together by the asset management engine 110 or as needed by other components of the arbitrary view generator 102 .

The perspective of each existing view retrieved by the asset management engine 110 is transformed into the perspective of the requested arbitrary view by the perspective transformation engine 112 of the arbitrary view generator 102 . As previously described, accurate camera information is known and stored with each image stored in database 106 . Thus, changing the perspective from an existing view to the requested arbitrary view involves a simple geometric mapping or transformation. In various embodiments, perspective transformation engine 112 may use any one or more suitable mathematical techniques to transform the perspective of an existing view into the perspective of an arbitrary view. If the requested view contains an arbitrary view that does not match the existing view, transforming the existing view into the arbitrary view's perspective will result in at least some unmapped or missing pixels (i.e., angles introduced into the arbitrary view that are not in the existing view). in the fields or positions).

Pixel information from a single perspective transformed existing view cannot fill all the pixels of a different view. However, in many cases, most if not all of the pixels constituting the requested arbitrary view may be collected from a plurality of perspective transformed existing views. The merging engine 114 of the arbitrary view generator 102 combines pixels from a plurality of perspective transformed existing views to generate the requested arbitrary view. Ideally, all pixels that make up an arbitrary view are collected from existing views. This may be possible, for example, if a sufficiently diverse set of existing views or perspectives of the asset under consideration is available and/or if the requested perspective is not too different from the existing perspectives.

Any suitable techniques may be used to combine or merge pixels from a plurality of perspective transformed existing views to generate the requested arbitrary view. In one embodiment, the first existing view closest to the requested arbitrary view is selected and retrieved from the database 106 and converted into the perspective of the requested arbitrary view. Pixels are then collected from this perspective transformed first existing view and used to populate the corresponding pixels in the requested arbitrary view. In order to populate the pixels of the requested arbitrary view that were not available from the first existing view, a second existing view comprising at least some of these remaining pixels is selected and retrieved from the database 106 and into the perspective of the requested arbitrary view. is converted Pixels that were not available from the first existing view are collected from this perspective transformed second existing view and used to fill the corresponding pixels in the requested arbitrary view. This process repeats for any number of additional existing views until all pixels of any requested view are filled and/or until all existing views are exhausted or the threshold of a prescribed number of existing views is fully used. can be

In some embodiments, the optional view requested may include some pixels that are not available from any existing views. In such a case, the interpolation engine 116 is configured to fill in any remaining pixels of the requested arbitrary view. In various embodiments, any one or more suitable interpolation techniques may be used by interpolation engine 116 to generate such unfilled pixels in the requested random view. Examples of interpolation techniques that may be used include, for example, linear interpolation, nearest neighbor interpolation, and the like. Interpolation of pixels introduces averaging or smoothing. The overall image quality may not be significantly affected by some interpolation, but excessive interpolation can cause unacceptable blurriness. Therefore, it may be desirable to use interpolation sparingly. As explained earlier, interpolation can be completely avoided if all pixels of the requested arbitrary view can be obtained from existing views. However, if the arbitrary view requested contains some pixels not available from existing views, interpolation is introduced. In general, the amount of interpolation required depends on the number of available existing views, the diversity of perspectives of the existing views, and/or how different the perspective of an arbitrary view is with respect to the perspectives of the existing views.

With respect to the example shown in FIG. 2 , 73 views around a chair object are stored as existing views of the chair. An arbitrary view around a chair object that is different or unique to any of the stored views may be created using a plurality of such existing views, preferably using minimal interpolation if present. However, creating and storing a complete set of these existing views may not be efficient or desirable. In some cases, a much smaller number of existing views covering a sufficiently diverse set of perspectives may be created and stored instead. For example, 73 views of a chair object may be decimated to a small set of minority views around the chair object.

As mentioned previously, in some embodiments the possible arbitrary views that may be requested may be limited, at least in part. For example, a user may be restricted from moving a virtual camera associated with an interactive scene to certain positions. With regard to the given example of FIG. 2 , the possible arbitrary views that may be requested may be limited to arbitrary positions around the chair object, but below the chair object, for example because there is not enough pixel data for the floor of the chair object. may not include arbitrary positions of This constraint on allowed arbitrary views ensures that the requested arbitrary view can be generated by arbitrary view generator 102 from existing data.

The arbitrary view generator 102 generates and outputs the requested arbitrary view 108 in response to the input arbitrary view request 104 . The resolution or quality of the generated random view 108 is the same as or similar to the qualities of the existing views used to generate it, since pixels from these views are used to generate the random view. Thus, in most cases, using a high-definition legacy view produces a high-definition output. In some embodiments, the generated random view 108 is stored in the database 106 along with other existing views of the associated scene, which then create other arbitrary views of the scene in response to future requests for random views. can be used to If the input 104 includes a request for an existing view of the database 106, the requested view need not be created from other views as described; Instead, the requested view is retrieved through a simple database query and provided directly as output 108 .

Arbitrary view generator 102 may also be configured to generate an arbitrary ensemble view using the described techniques. That is, input 104 may include a request to combine a plurality of objects into a single custom view. In such a case, the techniques described above are performed on each of the plurality of objects and combined to create a single unified or ensemble view comprising the plurality of objects. Specifically, the existing view of each of the plurality of objects is selected and retrieved from the database 106 by the asset management engine 110 , and the existing views are converted into the perspective of the requested view by the perspective transformation engine 112 , The pixels of the perspective-transformed existing views are used by the merging engine 114 to fill the corresponding pixels of the requested ensemble view, and the remaining unfilled pixels in the ensemble view are interpolated by the interpolation engine 116 . In some embodiments, the requested ensemble view may include a perspective that already exists for one or more objects constituting the ensemble. In such a case, the existing view of the object asset corresponding to the requested perspective is used to directly populate the pixels corresponding to the object in the ensemble view instead of first generating the requested perspective from other existing views of the object.

As an example of an arbitrary ensemble view including a plurality of objects, consider a table object photographed or rendered independently of the chair object of FIG. 2 . The chair object and table object can be combined using the disclosed techniques to create a single ensemble view of both objects. Thus, using the disclosed techniques, independently captured or rendered images or views of each of a plurality of objects may be consistently combined to create a scene comprising the plurality of objects and having a desired perspective. As described above, the depth information of each existing view is known. The perspective transformation of each existing view includes a depth transformation so that multiple objects can be properly positioned relative to each other in the ensemble view.

Creating an arbitrary ensemble view is not limited to combining a plurality of single objects into a custom view. Rather, a plurality of scenes with a plurality of objects or a plurality of rich virtual environments may similarly be combined into a custom ensemble view. For example, a plurality of individually and independently created virtual environments, possibly from different content creation sources and possibly with different existing individual perspectives, may be combined into an ensemble view with a desired perspective. Thus, in general, the arbitrary view generator 102 may be configured to coherently combine or harmonize a plurality of independent assets, possibly comprising different existing views, into an ensemble view, possibly having a desired arbitrary perspective. All combined assets are normalized to the same perspective, resulting in a perfectly harmonious ensemble view. A possible arbitrary perspective of the ensemble view may be limited based on existing views of the individual assets available to create the ensemble view.

3 is a flow diagram illustrating an embodiment of a process for generating an arbitrary perspective. Process 300 may be used, for example, by arbitrary view generator 102 of FIG. 1 . In various embodiments, process 300 may be used to generate an arbitrary view or any ensemble view of a defined asset.

Process 300 begins at step 302 where a request for an optional perspective is received. In some embodiments, the request received at step 302 may include a request for an arbitrary perspective of the defined scene that is different from any existing available perspectives of the scene. In such a case, for example, an optional perspective request may be received in response to the requested change in perspective of a presented view of the scene. This change of perspective may be enabled by changing or manipulating the virtual camera associated with the scene, such as panning the camera, changing the focal length, changing the zoom level, and the like. Alternatively, in some embodiments, the request received at step 302 may include a request for an arbitrary ensemble view. As an example, such an arbitrary ensemble view request may be received in connection with an application that allows a plurality of independent objects to be selected and provides a unified perspective-corrected ensemble view of the selected objects.

In step 304, a plurality of existing images for generating at least a portion of the requested arbitrary perspective are retrieved from one or more associated asset databases. The plurality of retrieved images may be associated with the defined asset if the request received in step 302 includes a request for an arbitrary perspective of the defined asset, or the request received in step 302 may be in an arbitrary ensemble view. In the case of including a request for , it may be associated with a plurality of assets.

In step 306 , each of the plurality of existing images retrieved in step 304 having a different perspective is transformed into an arbitrary perspective requested in step 302 . Each of the existing images retrieved in step 304 includes associated perspective information. The perspective of each image is defined by camera characteristics related to image creation, such as relative position, orientation, rotation, angle, depth, focal length, aperture, zoom level, and lighting information. Since complete camera information is known for each image, the perspective transformation of step 306 involves simple mathematical operations. In some embodiments, step 306 also optionally includes a lighting transformation such that all images are consistently normalized to the same desired lighting condition.

In step 308, at least a portion of the image with the arbitrary perspective requested in step 302 is filled by pixels collected from the perspective-transformed existing image. That is, pixels from a plurality of perspective-corrected existing images are used to create an image with the requested arbitrary perspective.

At step 310, it is determined whether the generated image with the requested arbitrary perspective is complete. If it is determined in step 310 that the generated image with the requested arbitrary perspective is incomplete, then in step 312 there are more existing images from which any remaining unfilled pixels of the generated image may be mined. It is decided whether or not If it is determined at step 312 that more existing images are available, then one or more additional existing images are retrieved at step 314 and the process 300 continues at step 306 .

If it is determined in step 310 that the generated image with the requested arbitrary perspective is incomplete, and there are no more existing images available in step 312, then any remaining unfilled pixels of the generated image are It is interpolated in step 316 . Any one or more suitable interpolation techniques may be used in step 316 .

After it is determined in step 310 that the generated image with the requested arbitrary perspective is complete or after interpolating any remaining unfilled pixels in step 316, the generated image with the requested arbitrary perspective is determined in step 318 is output from Process 300 is then terminated.

As described, the disclosed techniques can be used to create an arbitrary perspective based on other existing perspectives. Since camera information is preserved with each existing perspective, it is possible to normalize different existing perspectives to a common desired perspective. The resulting image with the desired perspective can be constructed by mining pixels from the perspective-transformed existing images. The process associated with generating an arbitrary perspective using the disclosed techniques is fast and almost instantaneous, as well as results in high quality output, making the disclosed techniques particularly powerful for interactive real-time graphics applications.

The disclosed techniques also describe the creation of an arbitrary ensemble view comprising a plurality of objects by using available images or views of each of the plurality of objects. As described, perspective transformation and/or normalization allows pixels comprising independently captured or rendered images or views of a plurality of objects to be consistently combined into a desired arbitrary ensemble view.

In some embodiments, it may be desirable to first build or assemble the scene or ensemble view by selecting and locating the content desired to be included in the scene or ensemble view. In some such cases, a plurality of objects may be stacked or combined like building blocks to create a composite object that constitutes a scene or ensemble view. Consider, for example, an interactive application in which a plurality of independent objects are selected and appropriately placed on, for example, a canvas to create a scene or ensemble view. For example, an interactive application may include a visualization or modeling application. In this application, arbitrary views of objects cannot be used to construct a scene or ensemble view due to perspective distortions that occur at the associated focal length. Rather, defined object views that are substantially free of perspective distortion are utilized as described below.

Orthographic views of objects are used in some embodiments to model or define a scene or ensemble view that includes a plurality of independent objects. An orthographic view includes a parallel projection approximated by a (virtual) camera that has a relatively long focal length such that the rays or projection lines are substantially parallel, and is positioned at a large distance relative to the object of interest. . Orthographic views have little or no perspective distortion as they have no depth or are fixed. As such, orthogonal views of objects can be used similarly to building blocks when specifying an ensemble scene or a composite object. After an ensemble scene comprising arbitrary combinations of objects has been specified or defined using these orthogonal views, the scene or its objects can be defined using arbitrary view creation techniques previously described in connection with the description of FIGS. 1-3 . It can be converted to the desired camera perspective.

In some embodiments, the plurality of views of an asset stored in database 106 of system 100 of FIG. 1 includes one or more orthogonal views of the asset. These orthogonal views may be captured (eg, photographed or scanned) or rendered from a three-dimensional polygonal mesh model. Alternatively, the orthographic view may be created from other views of the asset available in the database 106 according to the arbitrary view creation techniques described in connection with the description of FIGS. 1-3 .

4A-4N illustrate examples of an embodiment of an application in which independent objects are combined to create an ensemble or composite object or scene. Specifically, FIGS. 4A-4N illustrate a furniture building application in which various independent seating components are combined to create different sectional configurations.

4A shows an example of perspective views of three independent seating components: a left-arm chair, an armless loveseat, and a right-arm chaise. In the example of FIG. 4A each of the perspective views has a focal length of 25 mm. As can be seen, the resulting perspective distortions are caused by stacking components next to each other, i.e. placing components side by side, which may be desirable when building a prefab construction comprising the components. interfere

FIG. 4B shows an example of orthographic views of the same three components of FIG. 4A . As shown, orthogonal views of objects are modular or block-like and can be stacked or placed side by side. However, depth information is significantly lost in orthogonal views. As can be seen, all three components appear to have the same depth in orthogonal views, despite the actual difference in depth seen in FIG. 4A especially with regard to the chaise.

Fig. 4c shows an example of combining orthogonal views of the three components of Fig. 4b to specify a composite object; That is, FIG. 4C shows the creation of an orthogonal view of a sectional view through side-by-side placement of orthogonal views of the three components of FIG. 4B . As shown in FIG. 4C , the bounding boxes of the orthogonal views of the three seating components fit perfectly next to each other to create an orthogonal view of the prefabricated furniture. That is, orthogonal views of components facilitate user-friendly manipulations of components in the scene as well as precise placement.

4D and 4E each show an example of converting an orthogonal view of the composite object of FIG. 4C into an arbitrary camera perspective using the arbitrary view generation techniques previously described in connection with the description of FIGS. 1-3 . That is, the orthographic view of the composite object is transformed into a general camera perspective that accurately represents the depth in each example of FIGS. 4D and 4E . As shown, the relative depth of the long chair to the love seat and chair lost in the orthogonal views can be seen in the perspective views of FIGS. 4D and 4E .

4F, 4G, and 4H respectively show examples of a plurality of orthogonal views of a left arm chair, a love seat without an armrest, and a right arm bench. As previously described, any number of different views or perspectives of an asset may be stored in the database 106 of the system 100 of FIG. 1 . The sets of Figures 4F-4H are independently captured or rendered and stored in the database 106 and have 25 orthogonal views corresponding to different angles around each asset from which any arbitrary view of any combination of objects can be created. include those For example, in a furniture building application, top views may be useful for ground placement and front views may be useful for wall placement. In some embodiments, in order to maintain a more compact set of reference data, only a prescribed number of orthogonal views are stored for an asset in database 106 from which any arbitrary view of the asset can be created.

4I-4N show various examples of creating arbitrary views or perspectives of arbitrary combinations of objects. Specifically, each of FIGS. 4I-4N illustrates creating an arbitrary perspective or view of a prefab furniture comprising a plurality of independent seating objects or components. Each arbitrary view is an arbitrary view of one or more orthogonal (or other) views of objects comprising the ensemble view or composite object, for example, using the arbitrary view creation techniques previously described in connection with the description of FIGS. 1-3 . It can be created by transforming into a view and collecting pixels to fill an arbitrary view and interpolating any remaining missing pixels.

As previously described, each image or view of an asset in database 106 may be stored along with lighting information as well as corresponding metadata such as relative object and camera position and orientation information. Metadata can be generated when rendering a view from a three-dimensional polygonal mesh model of an asset, imaging or scanning the asset (where depth and/or surface normal data can be estimated), or a combination of the two. .

A defined view or image of an asset includes pixel intensity values (eg, RGB values) for each pixel that makes up the image and various metadata parameters associated with each pixel. In some embodiments, one or more red, green, and blue (RGB) channels or values of a pixel may be used to encode pixel metadata. For example, the pixel metadata may include information on a relative position or position (eg, x, y, and z coordinate values) of a point in a 3D space projected onto a corresponding pixel. Also, the pixel metadata may include information about surface normal vectors (eg, angles made up of x, y, and z axes) at the corresponding position. Moreover, the pixel metadata may include texture mapping coordinates (eg, u and v coordinate values). In this case, the actual pixel value at a point is determined by reading the RGB values at the corresponding coordinates of the texture image.

Surface normal vectors facilitate modifying or changing the lighting of an arbitrary view or scene created. More specifically, re-lighting a scene involves scaling pixel values according to how well the pixel's surface normal vectors match the direction of a newly added, removed, or changed light source, which for example It can be quantified, for example, at least in part by the dot product of the direction of light and the normal vectors of the pixels. Specifying pixel values via texture mapping coordinates facilitates modifying or changing the texture of an arbitrary view or scene or part thereof that is created. More specifically, the texture can be changed by simply replacing or replacing the referenced texture image with another texture image having the same dimensions.

The disclosed arbitrary view creation techniques are effectively based on relatively low computational cost perspective transformation and/or query operations. An arbitrary (ensemble) view can be created simply by selecting the correct pixels and properly filling the resulting arbitrary view with those pixels. In some cases, pixel values may be selectively scaled (eg, when lighting is adjusted). The low storage and processing overhead of the disclosed techniques facilitates fast real-time or on-demand generation of arbitrary views of complex scenes with a quality comparable to the high-definition reference view being generated.

As described, in some embodiments assembling an ensemble or composite object or scene includes specifying a plurality of objects or assets comprising the ensemble using orthographic views. Orthographic views facilitate precise placements and alignments of a plurality of objects or assets in an ensemble scene. An orthogonal view of an ensemble scene may be transformed into any arbitrary camera perspective, for example to generate any desired or requested perspective. Transforming the ensemble view into a prescribed camera perspective may include individually transforming each of a plurality of objects or assets comprising the ensemble scene into a prescribed perspective using techniques previously described. have. Although the techniques previously described for generating an arbitrary ensemble view are relatively efficient, they suffer a little detectable latency penalty to the end user, such as, for example, in applications that provide interactive real-time experiences to the user. Even higher efficiencies may be desirable in certain applications where it is advantageous to generate an output with , almost immediately, or at least very quickly.

In some embodiments, a further improvement in efficiency is at least in part by eliminating the processing associated with transforming a majority (eg, orthogonal or other view) of a plurality of objects or assets comprising an ensemble scene into a defined arbitrary perspective. can be promoted by Instead, an output ensemble view or image in which an available existing view of an object or asset closest to or closest to a specified arbitrary perspective with respect to the specified position and orientation of that object or asset in the ensemble scene represents any specified perspective. It is used for the object or asset when creating it. In most cases, the resulting output ensemble view is not a perfectly accurate perspective, but provides an acceptable approximation for many applications and has much lower latency than producing a perfectly accurate perspective output. Creating an approximation of an arbitrary ensemble view to an arbitrary camera pose by using a maximally quantized subset of pre-existing reference views of one or more objects or assets that make up the ensemble is described in more detail below.

5 is a high-level flow diagram illustrating an embodiment of a process for generating an arbitrary ensemble view. In some embodiments, process 500 is at least partially involved in suitably combining or compositing at least one or more single best-matching existing views, if not most, if not all, of the objects or assets that make up the ensemble scene. It is used to efficiently generate an output image of the ensemble scene based on

Process 500 begins at step 502 where a request for a defined perspective of an ensemble scene is received. The requested defined perspective of the ensemble scene includes a camera perspective or pose selected or specified for the ensemble scene and may generally include any arbitrary view. An arbitrary view in a given context includes any desired view or perspective of a scene that is not known in advance before a specification or camera pose is requested. An ensemble scene includes a combined view of a plurality of independent objects or assets. In general, the specification of an independent object or asset includes a set of existing reference images and views of the individual object or asset with different camera perspectives and corresponding metadata, one or more of which is that object. Alternatively, it can be used to create or designate a part of an ensemble scene associated with an asset. In some embodiments, the request of step 502 is an interactive mobile or facilitating manipulation of a camera angle or pose in ensemble scene space and/or placement of a plurality of objects or assets to create a composite or ensemble scene. Received from a web-based application. For example, the request may be received from a visualization or modeling application or an augmented reality (AR) application. In some embodiments, the request of step 502 is an orthographic view of the ensemble scene because orthographic views facilitate easier manipulation, placement, and alignment of the plurality of objects or assets that make up the ensemble scene. Received for a view.

In step 504 , the closest or closest matching existing reference image or view is selected for each of at least a subset of the one or more objects or assets constituting the ensemble scene. Step 504 may be performed serially and/or in parallel on individual or independent objects or assets that make up the ensemble scene. In some embodiments, only one or a single existing reference image or view is selected for the object or asset that best matches the requested prescribed perspective for a given pose of that object or asset in ensemble scene space. The ensemble scene space includes an ensemble scene coordinate system having a defined origin defined in an appropriate manner, such as the center (eg, center of mass) of the ensemble scene. In step 504, the position and orientation or pose of the object or asset relative to the ensemble scene coordinate system are determined to select an existing reference image or view that most closely matches the object or asset constituting the ensemble scene. , translated, transformed or correlated to an equivalent pose in its respective coordinate system associated with the existing reference images or views of that object or asset. Thus, a simple camera metric calculation with relatively low computational complexity is performed based on the requested perspective and relative object or asset pose in the ensemble scene so that in step 504 the closest matching existing reference image or view can be selected. do.

One or more criteria and/or thresholds may be defined to determine or identify a closest matching existing reference image or view for an object or asset. In some cases, an existing reference image or view is selected at step 504 only if one or more such thresholds are met. In the ideal case, an exact match is found and selected at step 504 . However, in some cases, the available existing reference image dataset is too incomplete, such as when the available existing reference images or views of the object or asset are too different from the requested perspective, or the reference image available for the object or asset. If there are no fields or views, one or more selection criteria and/or thresholds may not be met. In some such cases, the closest matching placeholder or ghost image or view of the object or asset is selected instead at step 504 . These placeholder images or views represent the shape of an object or asset, but lack other properties such as texture and optical properties. In some embodiments, a relatively low set of placeholder images spanning a sufficiently dense set of possible views around an object or asset (eg, including an angle covering 360 degrees around the object or asset) It is created and stored for each unique object shape using computational complexity rendering techniques. Placeholders are then used when fully rendered versions of an object or asset are not available or exhibit unacceptable deviations from the requested perspective.

In step 506, the output image of the ensemble scene is obtained, at least in part, by suitably combining or synthesizing existing closest matching existing reference images or views of the objects or assets constituting the ensemble scene selected in step 504, as appropriate. Generated for the requested prescribed perspective. Step 506 comprises appropriately scaling or resizing an existing closest matching reference image or view selected for the object or asset and/or the closest matching existing reference image or view selected for the object or asset. This may include pasting the reference image or view or determining the position or position in the ensemble view to be composited. In most cases, the resulting output image of the ensemble scene is very close to the requested prescribed perspective. Since most of the objects or assets that make up the ensemble scene are represented in an output image with their closest or closest available existing poses, these objects or assets are not in a perfectly accurate perspective because they are strictly ) because it is neither rendered nor created. That is, in most cases these objects or assets do not have the specified perspective requested in the output image unless an exact match is found in the existing images or views available. Although the vanishing points of these objects or assets do not all converge to the same point in the output image, the objects or assets are in most cases the human vision to perceive the output image in an accurate perspective for the most part. It is offset or distorted by a corresponding amount (eg, a few degrees) enough to deceive the system.

Consistency of the output image of the ensemble scene is also facilitated by generating at least some portions of the ensemble scene in a generally consistent or similar manner, which also enables human interpretation of the output image as substantially visually correct. For example, one or more objects or assets that make up the ensemble scene and/or the (flat or other) surfaces, structural elements, global features, etc. that make up the ensemble scene are accurate in perspective; That is, it may be rendered or generated strictly to have the requested specified perspective rather than an approximation of the requested perspective. For example, if the ensemble scene includes a space such as a room, walls, ceilings, floors, rugs, wall hangings, etc. may be created using the camera poses of the requested perspective, thus, the ensemble created in step 506 . It can be accurately represented in the output image of the scene. Moreover, the output image of the ensemble scene may contain a global illumination location that affects all parts of the scene in a similar and consistent manner when re-illuminated using available metadata such as, for example, surface normal vectors. Thus, by creating some parts of an ensemble view in a global or perspective corrective manner and representing most of the independent objects that make up the ensemble view with best approximations, in many cases a perfectly correct perspective version An output that is almost indistinguishable from the past is produced. In some cases, some distortion may be seen, but with mood board applications or space/room planning applications where a designer or user benefits from viewing an ensemble of objects or assets together regardless of the correct perspective. This may still be acceptable for certain applications that do not require a perfectly accurate view, such as Nevertheless, the repository or database of existing images or views available will grow over time, and thus the disclosed techniques will continue to produce outputs that more and more accurately represent the requested prescribed perspective. In the best case, exact matches are found for all objects or assets and this is used to produce an output image with the specified perspective requested in practice, rather than an approximation.

Although the foregoing embodiments have been described in some detail for clarity of understanding, the present invention is not limited to the details provided. There are many alternative ways of implementing the present invention. The disclosed embodiments are illustrative and not restrictive.

Claims (23)

A method comprising:
receiving a request for a prescribed perspective of an ensemble scene comprising a plurality of assets; and
generating an output image of the ensemble scene approximating the requested prescribed perspective based at least in part on combining a single existing image of each at least a subset of the plurality of assets. The method of claim 1 , wherein the request is received for an orthographic view of an ensemble scene. The method of claim 2 , wherein the orthogonal view of the ensemble scene comprises combined orthogonal views of a plurality of assets. The method of claim 1 , further comprising selecting a single existing image of each of at least a subset of the plurality of assets. 5. The method of claim 4, wherein the selecting comprises selecting an exact match with a requested defined perspective. 5. The method of claim 4, wherein the selecting comprises selecting the closest available match or closest to the requested prescribed perspective. 5. The method of claim 4, wherein selecting comprises selecting based on poses of associated assets in the ensemble scene. 5. The method of claim 4, wherein selecting comprises selecting an existing rotated image of an associated asset. 5. The method of claim 4, wherein the selecting comprises selecting the closest or closest available match to the requested prescribed perspective based on the pose of the associated asset in the ensemble scene. The method of claim 1 , wherein generating an output image of the ensemble scene comprises scaling a single existing image of one or more of the subset of assets. The method of claim 1 , wherein generating an output image of the ensemble scene comprises resizing a single existing image of one or more of the subset of assets. The method of claim 1 , wherein generating an output image of the ensemble scene comprises determining a position in the ensemble scene to contain a single existing image of each of at least a subset of assets. The method of claim 1 , wherein said binding comprises synthesizing. The method of claim 1 , wherein generating an output image of the ensemble scene comprises generating a view of at least one of a plurality of assets having a requested defined perspective. The method of claim 14 , wherein the view is created using a plurality of existing images for the at least one asset. The method of claim 1 , wherein generating an output image of the ensemble scene comprises generating at least a portion of the ensemble scene to have a requested prescribed perspective. 17. The method of claim 16, wherein the at least one portion comprises a surface of an ensemble scene. 17. The method of claim 16, wherein the at least one portion comprises structural elements of an ensemble scene. 17. The method of claim 16, wherein the at least one portion comprises a global feature of an ensemble scene. The method of claim 1 , further comprising globally re-illuminating the generated output image of the ensemble scene. The method of claim 1 , wherein the output image comprises a frame of a video sequence. In the system,
As a processor:
receive a request for a defined perspective of an ensemble scene including a plurality of assets;
the processor configured to generate an output image of the ensemble scene that approximates a requested prescribed perspective based at least in part on combining a single existing image of each of at least a subset of a plurality of assets; and
and a memory coupled to the processor and configured to provide instructions to the processor. A computer program product embodied on a non-transitory computer-readable medium comprising computer instructions, the computer instructions comprising:
receive a request for a defined perspective of an ensemble scene including a plurality of assets;
and generating an output image of the ensemble scene approximating a requested prescribed perspective based at least in part on combining a single existing image of each of at least a subset of the plurality of assets.

Images