JP2014519132A - Controlling objects in a virtual environment - Google Patents

Abstract

  In this document, it is discussed to digitize objects in a photograph. The user presents the object to the camera, and the camera captures an image containing color data and depth data for the front and back surfaces of the object. For both the front and back images, the closest point to the camera is determined by analyzing the depth data. From the nearest point, the edge of the object is found by noting the large difference in depth data. Depth data is also used to construct point cloud configurations for the front and back of the object. Various techniques are applied to extrapolate edges, remove seams, intelligently spread colors, filter noise, apply skeleton structures to objects, and further optimize digitization. Ultimately, the digital representation is presented to the user and potentially used in various applications (eg, games, web, etc.).

Description

  An embodiment of the present invention relates to control of an object in a virtual environment, for example.

  [0001] Recent game and Internet technologies interact with users in far more personal applications than these technologies have in the past. Rather than simply pressing a button on the controller connected to the game console, today's gaming systems operate the player standing in front of the camera or the action the player performs with the wireless controller (for example, a baseball bat Shake the controller). This personal dialogue expands the whole new territory of the game.

  An embodiment of the present invention relates to control of an object in a virtual environment, for example.

  [0002] This "Summary of the Invention" is provided to introduce in a simplified form the selection of concepts further described below in "DETAILED DESCRIPTION OF THE INVENTION". This Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter. Nor is this Summary of the Invention intended to be used as an aid in determining the scope of the claimed subject matter.

  [0003] One aspect is directed to creating a digital representation ("digitization") of an object in an image. The user presents the object to the camera, and the camera captures an image that includes color data and depth data for the front and back surfaces of the object. For both the front and back images, the closest point to the camera is determined by analyzing the depth data. From the nearest point, the edge of the object is found by noting the large difference in depth data. Depth data is also used to build point cloud configurations for the front and back of the object. Various techniques are applied to extrapolate edges, remove seams, intelligently spread colors, filter noise, apply skeleton structures to objects, and further optimize digitization. Ultimately, the digital representation is presented to the user and potentially used in various applications (eg, games, web, etc.).

  [0004] Exemplary embodiments of the invention are described in detail below with reference to the accompanying drawings.

[0005] FIG. 1 is a block diagram of an exemplary computing environment suitable for implementing the embodiments discussed herein. [0006] FIG. 4 is a diagram of a user presenting an object for digitization, according to one embodiment. [0007] FIG. 2 is a flow diagram of a workflow for digitizing an object, according to one embodiment. [0008] FIG. 4A is a camera view image of a user presenting an object for digitization according to one embodiment.

FIG. 4B is a camera view image of a user presenting an object for digitization according to one embodiment.
[0009] FIG. 3 is a segmented depth image that can be used to digitize an object, according to one embodiment. [0010] FIG. 4 is a diagram of depth and color offsets according to one embodiment. [0011] FIG. 3 is a source color image that can be used to digitize an object, according to one embodiment. [0012] FIG. 4 is a diagram of color segmentation of captured objects, according to one embodiment. [0013] FIG. 4 is a user interface (UI) that provides instructions for holding an object to be digitized, according to one embodiment. FIG. 6 is a user interface (UI) that provides instructions for holding objects to be digitized, according to one embodiment. [0014] FIG. 3 is a diagram of a three-dimensional (3D) point cloud configuration of an object, according to one embodiment. [0015] FIG. 5 is a diagram of two views of an aligned point sheet, according to one embodiment. [0016] FIG. 4 is a diagram of a final point cloud configuration, according to one embodiment. [0017] FIG. 4 is a UI for displaying a confirmation image of a digitized object displayed to a user, according to one embodiment. [0018] FIG. 6 is a diagram of mesh output of a captured image, according to one embodiment. [0019] FIG. 6 is a smoothed and processed image of an object, according to one embodiment. [0020] FIG. 6 is an image having UV coordinates, according to one embodiment. [0021] FIG. 6 is a diagram of forward-facing triangle edges drawn into a section of the final texture map, according to one embodiment. [0022] FIG. 19A is a diagram illustrating the weighting applied to the bone of the generated skeleton structure, according to one embodiment. FIG. 19B is a diagram illustrating weighting applied to the bone of the generated skeleton structure, according to one embodiment. FIG. 19C is a diagram illustrating the weighting applied to the bone of the generated skeleton structure, according to one embodiment. FIG. 19D illustrates the weighting applied to the bone of the generated skeleton structure, according to one embodiment. FIG. 19E illustrates weighting applied to the bone of the generated skeleton structure, according to one embodiment. [0023] FIG. 20A is a diagram of an image before luminance / saturation processing, according to one embodiment. FIG. 20B is an image after luminance / saturation processing according to one embodiment. [0024] FIG. 21A is a diagram of a source image, according to one embodiment. FIG. 21B is an output image after the edges have been filtered, according to one embodiment. [0025] FIG. 22A is an illustration of an image when an edge repair filter finds a color that becomes a background color, according to one embodiment. FIG. 22B is an image when an edge repair filter finds a color of interest according to one embodiment. [0026] FIG. 23A is an image showing the distance from an edge to a problematic area, according to one embodiment. FIG. 23B is an image showing calculated background probability values according to one embodiment. [0027] FIG. 6 is a diagram of a final composite texture map, according to one embodiment. [0028] FIG. 25A is a diagram of masked values, according to one embodiment. FIG. 25B is a diagram of severely blurred vertex colors, according to one embodiment. [0029] FIG. 26A is a diagram of a mesh having only a texture, according to one embodiment. FIG. 26B is a diagram of a mesh having a texture with vertex colors intermixed by mask values, according to one embodiment. [0030] FIG. 4 is a final rendering of a digitized object, according to one embodiment. [0031] FIG. 6 is a flowchart detailing a workflow for digitizing an object, according to one embodiment. [0032] FIG. 7 is a flow diagram detailing a workflow for digitizing an object, according to one embodiment.

  [0033] The subject matter of embodiments of the present invention is described herein with specificity to meet legal requirements. However, the description itself is not necessarily intended to limit the scope of the claims. Rather, the claimed subject matter may be embodied in other ways to cooperate with other current or future technologies to provide different steps or combinations of similar steps than those described herein. May be included. These terms are to be interpreted as implying any particular order between the various steps disclosed herein unless and unless otherwise stated to the order of the individual steps. Should not be done.

  [0034] The embodiments described herein generally relate to creating a digital representation of an object captured by a camera. In one embodiment, the user holds the object in front of the camera, the camera captures an image of the object, and the device converts the captured image into a 3D display that can be displayed digitally, for example, as an entity in a video game. And digitize. To illustrate, consider the following example. The user holds a toy octopus over a game device equipped with a camera. Using the camera, the gaming device takes pictures of the front and back of the object and captures both color data and depth data for each side. Based on the depth data, a 3D representation of the octopus is constructed and the color data is then added to the 3D representation to create a digital representation of the octopus (referred to herein as “digitizing”). Digitization can then be used in games or any other software or web application where octopus display is useful.

  [0035] At least one embodiment is directed to digitizing an object. A user presents an object to a camera on a computing device (such as a game console). The device can direct the user to determine the location of the object for display, for example by placing an outline on a screen that reflects the image seen by the camera, and the user places the object into the outline. Optimize the captured image by indicating that it should move. Ultimately, the device captures one or more images of the object. The user may then be instructed to present the back side of the object to the camera for capture. The device can then capture one or more images of the back side of the object. The captured front and back images are processed to form a 3D digitization of the object.

  [0036] In one embodiment, the process uses image depth data captured by the camera. Depth data describes the proximity of an object captured in an image on a pixel-by-pixel basis or other spatial representation. The depth data is used to locate the closest point of the object in the image. This embodiment assumes that the closest object in the image is the object that the user is looking to capture, eg, a user holding an octopus with respect to the camera, Probably means that.

  [0037] Exemplary operating environments in which various aspects of the invention, briefly described in the summary of the invention may be implemented, are now described. Referring generally to the drawings and in particular first to FIG. 1, an exemplary operating environment for implementing embodiments of the present invention is shown and designated generally as computing device 100. The computing device 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing device 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

  [0038] Embodiments of the present invention include computer code or machine usable instructions, including computer executable instructions, such as program modules, executed by a computer or other machine such as a personal data assistant or other handheld device. Can be described in the general context of Generally, a program module that includes routines, programs, objects, components, data structures, etc. refers to code that performs a particular task or implements a particular abstract data type. Embodiments of the present invention can be implemented in various system configurations including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, and the like. Embodiments of the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

  [0039] Continuing to refer to FIG. 1, computing device 100 includes the following devices: memory 102, one or more processors 103, one or more presentation components 104, input / output (I / O) ports. 105, an I / O component 106, and an exemplary power source 107, including a bus 101 that connects directly or indirectly. Bus 101 represents one or more buses (such as an address bus, a data bus, or combinations thereof) whatever. Although the various blocks in FIG. 1 are shown with lines for clarity, in practice, the various components depicted are less clear, for example, the lines are more accurate. It should be gray and ambiguous. For example, a presentation component such as a display device that becomes an I / O component can be considered. In addition, many processors have memory. The inventors of the present invention recognize that such is the essence of the technology, and that the diagram of FIG. 1 is an exemplary computing device that can be used with one or more embodiments of the present invention. I repeat that it is just an example. All of the following are considered within the scope of reference to FIG. 1 and “computing device”, so “workstation”, “server”, “laptop”, “game console”, “handheld device”, No distinction is made between categories such as.

  [0040] Computing device 100 typically includes a variety of computer-readable media. Computer readable media can be any available media that can be accessed by computing device 100 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media can be implemented in any method or technique for storage of information such as computer readable instructions, data structures, program modules or other data, both volatile and non-volatile media, removable and non-removable media. Includes media. Computer storage media include random access memory (RAM), read only memory (ROM), electrically erasable writable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) Or any other holographic memory, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any that can be used to encode the desired information and is accessible by the computing device 100 Including, but not limited to, other media.

  [0041] The memory 102 includes computer storage media in the form of volatile and / or nonvolatile memory. Memory 102 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical disk drives, and the like. Computing device 100 includes one or more processors that read data from various entities such as memory 102 or I / O component 106. The presentation component (s) 104 presents a data display to the user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, and the like.

  [0042] The I / O component 106 may comprise a camera capable of taking still images or videos. In one embodiment, the camera captures color data (eg, red, green, blue) and depth data when taking a picture. Depth data indicates the proximity of objects captured by the camera itself, in one embodiment, on a pixel-by-pixel basis. Depth data may be captured in many ways, such as using an infrared (IR) camera to read the projected IR light, reading the projected laser light, and the like. Depth data may be stored per centimeter, per meter, or other spatial representation. For example, IR dots may be projected and read by an IR camera, producing an output file detailing the depth of the image in the region immediately in front of the camera, and measured in meter-by-meter orientation Sometimes. In addition, depth data can also indicate the orientation of a particular portion of the captured object by recording the pixels of the screen area where the depth is measured. Because the color camera and the depth camera may be installed separately from each other, the conversion may be performed to map the acquired color data to the corresponding depth data.

  [0043] The I / O port 118 allows the computing device 100 to be logically coupled to other devices that include the I / O component 120, some of which may be built-in. . Exemplary I / O components 120 include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, and the like.

  [0044] As previously indicated, some embodiments are directed to creating a digital representation of an object in a virtual environment. FIG. 2 is a diagram of an environment 200 for a user 204 for creating a digital representation of an object 206, according to one embodiment. It should be understood that this and other arrangements described herein are shown by way of example only. Even though other arrangements and elements (eg, machines, interfaces, functions, order, and function groupings, etc.) may be used in addition to or instead of those shown, some elements may be omitted entirely. May be. Further, many of the elements described herein are functional entities that may be implemented as independent or distributed components, or with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

  [0045] Turning to FIG. 2, environment 200 shows a user 204 presenting an object 206 shown as an octopus doll to computing device 202, which includes two cameras, a color camera 208 and a depth. The camera 210 is mounted. In environment 200, computing device 202 is a game console, such as Microsoft Kinect ™, made by Microsoft Corporation ™. A camera on the computing device 202 captures one or more images that include the object 206. The color camera 208 captures image color data, and the depth camera 210 captures depth data. In an alternative embodiment, computing device 202 may only have one camera that captures both color data and depth data.

  [0046] Even though shown as a stand-alone device, the computing device 202 may be integrated or communicatively connected to other computing devices (eg, game consoles, servers, etc.). The components of the computing system 200 can communicate with each other over a network, which includes, without limitation, one or more local area networks (LANs) and / or wide area networks (WANs). Can do. Such network environments are commonplace in offices, company-wide computer networks, intranets, and the Internet. It should be understood that some embodiments can include additional computing devices 202. Each device may include a single device / interface or multiple devices / interfaces cooperating in a distributed environment.

  [0047] In some embodiments, one or more of the digitization techniques described herein may be implemented by a stand-alone application. Alternatively, one or more of the digitization techniques may be implemented by disparate computing devices across a network, such as the Internet, or by modules within a gaming system. Those skilled in the art will appreciate that the components / modules shown in FIG. 2 are exemplary in nature and number and should not be construed as limiting. Any number of components / modules may be employed to achieve the desired functionality within the scope of embodiments of the present invention. Further, the components / modules can be installed on any number of servers or client computing devices.

  [0048] Although shown in FIG. 2 as the user 204 presents the front side of the object 206 to the computing device 202, the user 204 can present the back side of the object 206 to the computing device 202. Therefore, the back side image of the object 206 can be captured. The back side image can then be combined with the front side image of the object 206 and a 3D display of the object 206 can be generated. Each captured image can include color data and depth data, both of which allow the computing device 202 to accurately create a 3D representation of the object 106.

  [0049] Additional image fields of the object 106 may also be used to assist in digitization in different embodiments. The object 106 may be photographed from any different angle or may be filmed. For example, several images may be taken from the right, left, bottom, and top of the image 106 in addition to or instead of the front and back views to produce a more robust 3D digitization. is there. For example, several side views may be used to digitize specific sides of the object 106. In at least an embodiment, the more field of view of the object 106 used, the more complete or accurate the 3D display.

  [0050] FIG. 3 is a diagram of a workflow 300 for digitizing an object, according to one embodiment. Initially, as shown at 302, a user presents an object to a camera on a computing device so that an image can be taken. The computing device, in some embodiments, can instruct the user to move the object to a particular area to capture an optimal image of the image. For example, inquire to form an outline on the display, show real time images of the user and object, and then instruct the user to move the object to the outline. Once the initial image has been taken, the computing device can instruct the user to present the back side of the object for capture, as shown at 304. The instructions for capturing the back side may be performed by the computing device as well. For each captured image, color data and depth data are stored and used to digitize the displayed object. In addition, multiple images may be captured for the front and back images of the object. For example, the computing device may take 10 front images and 10 back images, and possibly merge the front 10 together and the back 10 together, or digitize the image It can be configured to use all 20 sheets to do. Although 10 images are shown to be the ideal number of images for digitizing an object, other embodiments can use a different number of captured images.

  [0051] Once the front and back images of the object are captured by the camera, one embodiment uses the image depth data to find the closest point in the image up to the camera, as shown at 306. Begin to digitize objects by searching for them. The user probably holds the object to digitize in front of the user, so the object should be closer to the camera than the others. Returning to FIG. 2, the user 204 may notice that he holds the object 206 in front of him and is therefore closer to the computing device 202. Finding the location of the closest object in the image may be achieved using depth data related to the image, and some embodiments perform processing on both the front and back images Then, the closest object among them is specified.

  [0052] As indicated at 308, the closest object identified in the image is then searched for an edge to identify where the object edge is. The depth data is again used to find the position of the edge of the object in the image. The edge search is started outward from the closest point and looks for significant differences in the depth of the points. For example, the edge of the octopus in FIG. 2 may have a point that is approximately 0.5 meters closer than the adjacent point that represents the shoulder of the user 204. Such significant differences represent a readable signal that adjacent points are not part of the object and therefore should not be included in further digitization steps. Finding the location of all edges of an object in this way allows the computing device to identify the object in the image.

  [0053] Once an object is determined, one embodiment switches off color data related to the rest of the image (ie, the portion of the image that is not identified as an object). In some embodiments, it may be necessary to capture multiple images (eg, 10 images on the front side and 10 images on the back side of the object), so smoothing as shown at 310. Techniques may be required to blend found edges between frames. For example, the object may move between frame 1 and frame 4, so smoothing of the edges between frames may be required to obtain an accurate representation of the object. In addition, noise, low resolution, and imperfections in depth and color alignment may also require additional smoothing and / or filtering of the edges.

  [0054] In one embodiment, as shown at 312, the resulting smoothed and / or filtered object is presented to the user for confirmation. The user can then accept or reject the resulting object. If accepted, additional processing may continue to digitize the object. If rejected, the embodiment can ask the user whether to start the entire process by taking a new picture of the object, or simply re-smooth or re-filter the object.

  [0055] Finally, the front and back images are used to generate a point cloud composition of the object in 3D. The “point cloud configuration” shown in detail in FIG. 11 is a mapping of the front and / or back image of an object to 3D space using the depth of each point or pixel of the identified object. A point cloud configuration is used for further digitization of objects. Nonetheless, alternative embodiments can use different representations or spatial collections of depth data and color data to create object constructs or other types of representations from various images. it can.

  [0056] FIGS. 4 through 26 show images of various steps in the digitization process and are discussed in further detail below to illustrate the processes used by various embodiments. Specifically, FIGS. 4A and 4B are camera view images of a user presenting an object for digitization according to one embodiment. In the illustrated embodiment, two views of the object are captured. The color camera is zoomed in on the center of the frame to obtain a 640 × 480 color window around the object of interest, and the corner of the color window is then (the corner is in front of the object of interest (Assuming) converted to depth frame coordinates. A matching 160 × 120 window is then captured from the depth frame. Without this pre-frame window adjustment (depending on the distance of the object of interest to the camera), the depth window and the color window may not overlap as well as possible. Moreover, raw color and raw depth can be captured without performing depth-to-color alignment or color-to-depth alignment. Since various other resolutions can be used instead, this resolution number and window are provided for illustrative purposes only.

  [0057] In one embodiment, the depth image is segmented into objects of interest. To do so, the depth pixel closest to the camera is searched and found assuming such a point is on the object of interest. This embodiment then fills outward from the closest point found until it hits a depth edge (ie, if the depth is too far from the front of the object or there is no depth data). In addition, points that are around large gradient regions and have very few adjacent points may be excluded. The result is a mask of depth pixels (referred to herein as a “segmented depth image”) over the object of interest, as shown in FIG. The segmented depth image is stored in the depth buffer (BAB / GOE shipped with a ring buffer size of 10) in the ring buffer, rewriting the oldest depth frame and averaging all frames together Get the final depth image. In one embodiment, only the segmented depth pixels contribute to the final average. As a result, the noise is smoothed, resulting in a more stable object edge, improving the scenario where multiple parts of the object blink and go out of segmentation due to noise or poor IR reflection material.

  [0058] FIG. 6 is a diagram of depth and color offsets according to one embodiment. As shown, one embodiment creates a depth and color offset table with green (shown in the upper right corner), red (shown in the lower left corner), and two mixed colors in between. The offset between the depth and color space coordinates of each pixel is stored in a table for quick lookup during color segmentation and meshing, as well as depending on the specific camera calibration settings. Rather, it helps to completely reproduce the output mesh using only the two captured color images. The area of the table outside the object segmentation may be filled by copying the offset at the outward segmentation edge. The copied offset at the edge can later be used to handle the case where the vertices in the output mesh projected onto the depth image are outside the boundary of the depth segmentation.

  [0059] FIG. 7 is a source color image and FIG. 8 is a color segmentation diagram of a captured object, according to one embodiment. Starting with segmentation in depth space, one embodiment uses a star distribution pattern to distribute each segmented depth pixel into a 320 × 240 color segmentation buffer. The resulting pattern is then “upsampled” to 640 × 480 and a “distance from ideal” value describing how far the source depth pixel is from the “ideal” distance, It is then calculated for each segmented color pixel. The ideal distance is how far the user should keep the object of interest close to the camera to get as much color / depth data as possible, without crossing the depth camera's front clip plane. Represent. These values can be displayed as feedback to the user during the capture process. Pixels that are further from the ideal may be more seriously blurred and smudged than pixels that are closer to the ideal. The distance value from the ideal is eventually copied to the alpha channel of the color image used for real-time preview.

  [0060] FIGS. 9 and 10 are diagrams of a user interface (UI) that provides instructions for holding an object to be digitized, according to one embodiment. FIG. 9 illustrates how the illustrated embodiment analyzes the number of segmented pixels, the distance to the camera, the distance from the center of the camera field of view, the pixel stability, the object size, and how to best position the object. Shows giving visual feedback and text feedback about placing. The feedback may be in the form of an outline on the screen. FIG. 10 shows the color data and depth data of the back image of the object of interest using the same process as described above. One embodiment uses the outline of the segmented frontal captured image to direct the user to correctly point the object. The user does not have to match the outline exactly because the front and back captured images can be automatically aligned later.

  [0061] FIG. 11 illustrates a point cloud configuration, according to one embodiment. At this point, the two color data images and the depth data image have been segmented into objects of interest. Using these images, a point cloud composition of points on the surface of the object of interest can be created and later used to reconstruct the triangular mesh. The segmented pixels in the front depth image are converted into 3D points “sheets”. In one embodiment, the position is not projected from depth image space to model space using depth data, and the origin is the back center of the sheet. The edge of the sheet protrudes back by adding additional points to form the sides of the object. In order to infer how “deep” an object is, BAB / GOE can use a constant value for the protruding distance.

  [0062] Similarly, a 3D point sheet from a back depth image is created using the back center of the front captured image as the origin. FIG. 12 is two views of a aligned point sheet, according to one embodiment. To align the sheet, the initial transformation is calculated by rotating the sheet 180 degrees around the upward axis, thereby forming the back of the point cloud. In one embodiment, another transform is calculated that aligns the edges of the front and back sheets as close as possible. The alignment process may move the back sheet so that the center of gravity of the back sheet matches the center of gravity of the front sheet. The brute force iteration is then used over the entire range of translation and rotation to minimize the “alignment error” value calculated as the sum of the distance from each front edge point to its nearest back edge point. Is done. Iterations may be made in multiple passes (at each pass, try to calculate the best value for each translation and rotation axis one at a time), and searching across each axis is for efficiency This is done using a two-stage hierarchical approach. Nearest neighbor search is accelerated using 3D cell space partitioning. One embodiment also implements an iterative closest point (“ICP”) algorithm for fast fine-grained alignment, or alternatively, the need for better control dictates the use of only brute force iterations There is.

  [0063] Points from the front sheet that do not have corresponding points in the back sheet may be extracted to search backward from each front point to find the nearest back point. Similarly, points from the back sheet that do not have corresponding points in the front sheet may be extracted. This is the case when the user's hand is in the captured image but changes position between captured images or when the object changes shape between the front captured image and the back captured image. The portion of the sheet that does not match between the captured image and the back captured image is removed.

  [0064] In one embodiment, the remaining points are merged together into the final point cloud, and the normals for the points are calculated using the plane formed by each point and its right and bottom neighbors. Is done. FIG. 13 illustrates a final point cloud configuration according to one embodiment.

  [0065] The confirmation image may then be presented to the user as shown in FIG. The confirmation image incorporates the results of sheet alignment and point excerpts to allow the user to detect cases where capture, alignment, or excerpts failed badly, and to recapture without going through the rest of the configuration process To do. The image projects and distributes the points in the final point cloud to the alpha channel of the front and back color images, rotates the back image based on the alignment transformation, and cleans up some additional images. Made by doing.

  [0066] The surface reconstruction step takes a final point cloud and generates a triangular mesh. FIG. 15 shows a diagram of mesh output using surface reconstruction. One embodiment was developed by Minmin Gong in the Xin Tong group of MRS-Beijing, “Poisson Surface Reconstruction” by Kazhdan, Bolitho, and Hoppe, and by Zhou, Gong, Ghou, Gong, A hybrid CPU / GPU implementation of the Poisson Surface Reconstruction algorithm described in detail in "Highly Parallel Surface Reconstruction" is used. This may be the computationally most intense part of digitization in both memory and time, and in some embodiments 10 to 10 points for typical point cloud data of approximately 20,000 points. It takes 20 seconds. The amount of hole filling can be limited during reconfiguration to keep the memory usage under control, but if there are large holes in the point cloud, such a limit can be seen as a leaky mesh. Can result in a bad state.

  [0067] FIG. 16 is a smoothed and processed image of an object, according to one embodiment. A vertex adjacency list is created, and surface normals and vertex normals are calculated. Next, one embodiment uses a Laplace algorithm to smooth some constraints. As a result, the sides of the object are rounded, noise is removed, and areas where the point sheets are not perfectly aligned are cleaned up.

  [0068] Depending on the quality of the point cloud, surface reconstruction can create small “islands” of shape instead of a single large mesh. One embodiment uses connected component labels to find islands, calculate their volume, and remove islands that are significantly smaller than the largest island.

  [0069] For each vertex, one embodiment considers the dot product between the normal of that vertex and the front and back capture field directions. The front view direction may be along the negative Z axis of the model space, and the back view direction may depend on the result of the sheet alignment process and may not be along the positive Z axis. As a result, some vertices may be seen in both the front capture field and the back capture field, and some vertices may not be seen in either field. Some vertices may be classified as “front” if their normals are facing the front rather than the back, and vice versa. This again allows the location of “seam” vertices (ie vertices that span the front and back views of the object).

  [0070] To create a texture map to apply onto the final mesh, one embodiment places the color image from the front captured image on top of the texture and the color image from the back captured image to the front captured image. Just below. The texels from the top part of the texture are then mapped onto mainly forward triangles, and so on for the back triangles. Vertices may initially be shared between the front and back triangles just along the front-back seam, and later, these shared vertices may be duplicated, resulting in Map different parts of the texture to the back triangle as opposed to the front triangle.

  [0071] According to one embodiment, FIG. 17 shows an image with UV coordinates, and FIG. 18 shows a diagram of forward-facing triangular edges drawn into a portion of the final texture map. To calculate the UV coordinates, the forward triangle is mapped to the top portion of the texture where the front captured color image is placed, and so on for the bottom. The vertex position is in the depth camera space, but the color image is in the color camera space, so after projecting the vertex onto the front / back depth image, one embodiment Converts coordinates to color camera space using a depth and color offset table.

  [0072] In one embodiment, the mesh is repositioned, mirrored about the upward axis, and scaled to preserve the maximum width / height aspect ratio. The captured color and depth images are mirrored for comparison with actual physical objects, so another mirroring is used to invert it. Skeletons may be optimized and animations may be added to tall objects rather than wide objects, so width / height aspect ratio restrictions may be some sort of skeleton Draw a border on the artifact that results from animating a wide object that doesn't match.

  [0073] In one embodiment, one skeleton is used for all animations of the skeleton. Skeletons have a good range of movement (walking, jumping, hopping, dancing, looking left and right, etc.) without requiring the object in question to have much more shapes. You can have bones to give.

  [0074] In order to skin the digitized image, the mesh is rescaled and positioned so that the skeleton is a percentage of the top bone from the top of the object (eg, approximately 90%). (The top bone is placed roughly inside the object's "head") and the bottom bone is placed in the bottom area of the object, and it fits snugly inside. The bone index can then be calculated, weighting the skeleton by finding the closest bone along the upward axis to each vertex, and weighting the bone using a falloff curve . 19A-19E illustrate weighting applied to various bones of the generated skeleton structure, according to one embodiment.

  [0075] Color and / or depth images are processed to reduce noise and improve quality. Processing is performed independently for the front and back images, in one embodiment, and the results are merged into the final texture map, which may require additional processing. After several experiments and feedback from the artists, the following steps were found to be optimal. Convert sRGB colors to linear space, apply “gray world” automatic white balance, repair edge artifacts, calculate luminance and saturation values, bilateral filtering on luminance, histogram equalization, and sharpening Apply ning, apply median filtering to saturation, convert back to sRGB, and finally stretch the color edges outward to the unsegmented region of the image. Other steps may be added, and some of the above may be deleted in other embodiments.

  [0076] FIGS. 20A and 20B show images before and after luminance / saturation processing, according to one embodiment. Independently processing the luminance / saturation allows the saturation to be filtered much more strongly while preserving details in the luminance image, which is effective in denoising the image. Histogram equalization can be applied very lightly to compensate for poorly exposed images.

  [0077] FIGS. 21A and 21B illustrate the source image and the output image after the edges have been filtered, according to one embodiment. In one embodiment, an “edge repair filter” attempts to replace the color at the edge of the object of interest that is actually from the background and not from the object itself. Bad colors can be confused due to the relatively low resolution and large noise of depth images and imperfect depth and color alignment. The edge repair filter operates on the “discussion area” of the pixels immediately surrounding the object edge. Using the assumption that the pixels inside the discussion area are clearly part of the object of interest and the more external pixels are part of the background, the "background probability" value is Calculated and used to blend high probability background pixels against the inner color.

  [0078] FIGS. 22A and 22B illustrate images in which an edge repair filter finds background and colors of interest, according to one embodiment. The target color is extrapolated from the outside to the area of discussion.

  [0079] FIGS. 23A and 23B are images showing the distance from the edge to the area of discussion and the calculated background probability value, according to one embodiment. Furthermore, FIG. 24 illustrates a final composite texture map of an image that has been textured onto an image that has not been final processed, according to one embodiment.

  [0080] Seams resulting from placing the front and back images together may need to be repaired. The slight remainder of the meshing is used to improve the appearance of objects in areas near the front and back seams and invisible to the color camera during capture. First, a per-vertex mask value is calculated that represents how “bad” the texture color is at the vertices. This value is due to the distance to the seam (where the front and back images touch but are generally not well aligned), and due to the surface facing away from the camera field of view, and again from insufficient texel density. The texture color is decomposed) is the product of how far the vertices are facing away from any of the captured images. These values can be stored in the vertex color alpha channel. Next, a blurred version of the surface color is calculated and stored in the vertex color RGB channel. These colors are of fairly good quality (although low in detail). Negative artifacts that require repair are relatively localized, high frequency, and blurring gives a more overall, low frequency color.

  [0081] FIGS. 25A and 25B show masked values and severely blurred vertex colors, according to one embodiment. At run time, the mask value is used, in one embodiment, to blend the source texture and the color of the blurred vertex. FIGS. 26A and 26B show a mesh with texture only (FIG. 26A) and a mesh with vertex colors mixed by mask values (FIG. 26B), according to one embodiment.

  [0082] FIG. 27 illustrates the final rendering of a digitized object, according to one embodiment. In one embodiment, once the final mesh and texture are complete, a Unreal Engine 3 mesh is created and rendered with the environment and rim lighting, self-occlusion, and animation. The GOE app also allows objects to be avatared by mapping Nui skeletons onto skeletons.

  [0083] The above steps balance usability, CPU / GPU / memory constraints, output quality, artistic considerations, sensor accuracy, and development time. Tradeoffs are made and this may not be unique to each scenario. As such, various steps may be added or some of the above may be deleted to improve the speed or quality of the final digitization.

  [0084] FIG. 28 illustrates a workflow 2800 for digitizing an object, according to one embodiment. As indicated at 2802, color data and depth data for an image is received. Analyzing the depth data and finding the object of interest by identifying the closest point in the image to the camera, based on the assumption that the user is most likely to present the object to the capturing camera It is done. Alternative methods for determining the object of interest may alternatively or additionally be used. Since embodiments are not limited to any particular type of means for positioning an object in the image, various image recognition techniques or algorithmic matching techniques are used to position the object in the image. Sometimes. Embodiments can also use image color data in addition to or instead of depth data to position objects. For example, a Coca-Cola can may contain a red trademark color, which is particularly relevant when trying to position the can in a photograph. In this way, the target object can be found in many different ways.

  [0085] Once the target object is positioned, the edges of the object are identified, as shown at 2806. Such a determination may be made by analyzing differences in color, depth, or contrast in the image around the object. Once the edge is positioned, the object's point cloud configuration may be performed using the color and depth data of the image, as shown at 2808. To digitize an object in 3D, multiple point cloud configurations for various aspects of the object are based on color and depth data for multiple images (eg, back, front, top, bottom, etc.). May be configured. Once created, multiple point cloud configurations can be aggregated to create the final digitization of the object, as shown at 2810.

  [0086] FIG. 29 illustrates a workflow 2900 for digitizing an object, according to one embodiment. Once the image of the object is received, the closest points of the image are identified as shown at 2904, as shown at 2902. The side of the object (eg, left, right, north, south, top, bottom, etc.) is identified as shown at 2906. Multiple point cloud configurations of multiple images are created as shown at 2908 and merged into a single representation as shown at 2910. The resulting representation can then be saved as shown at 2912 and presented on the display device.

  [0087] Many different arrangements of the various components shown, as well as those not shown, are possible without departing from the scope of the appended claims. Embodiments of our technology have been described for purposes of illustration rather than limitation. Alternative embodiments will become apparent after reading this disclosure and for that reason. Alternative means of implementing the foregoing can be completed without departing from the scope of the appended claims. Certain features and subcombinations are useful and may be utilized without reference to other features and subcombinations and are considered to be within the scope of the claims.

Claims (15)

One or more computer storage media incorporating computer-executable instructions that, when executed, digitize objects captured by the camera, the method comprising:
Receiving color data and depth data related to the image;
Identifying the closest point in the image to the camera;
Identifying an edge of an object in the image from the closest point;
Generating a point cloud construction of the object using the depth data;
Generating one or more digitizations of the object using the point cloud configuration. Identifying the closest point in the image to the camera,
A sub-step of calculating distances between the camera aperture and a plurality of points in the image;
The one or more computer storage media of claim 1, further comprising a substep of selecting a shortest distance among the distances. A sub-step for identifying a connected feature to the object at the shortest distance;
The one or more computer storage media of claim 2, further comprising a substep of storing an indication of the shape that represents the closest portion of the object to the camera. Receiving color data and depth data for a second image of the object;
Determining a second closest point in the second image;
Identifying the object at the second closest point;
Generating a second point cloud configuration of the object to be oriented in the second image (as oriented in);
The one or more computer storage media of claim 1, further comprising: Identifying a seam between the point cloud configuration and the second point cloud configuration;
Determining a filler color to fill a portion of the seam;
Painting the portion of the seam with the fill color to create the 3D rendition having a seamless edge between the point cloud configuration and the second point cloud configuration;
Storing the 3D display;
The one or more computer storage media of claim 4, further comprising: Identifying the object comprises:
Performing an image analysis within a spatial region surrounding the nearest point in the image;
Determining a color difference between two regions in the spatial region based on the image analysis;
A sub-step of designating one of the regions to be associated with the object;
The one or more computer storage media of claim 1, further comprising a sub-step of removing another region having a different color from the one of the regions.   Removing one or more points of the image around a region associated with the object and having a threshold number less than the threshold number of adjacent points, and masking the depth pixels of the object The one or more computer storage media of claim 1, further comprising:   Overwrites at least one depth frame and averages multiple frames together to produce a final depth image in the depth frame ring buffer The one or more computer storage media of claim 7, further comprising storing the mask of the depth pixels at a time. A method for displaying a digital representation of an object comprising:
Receiving a plurality of images incorporating the object from different views;
Using each depth data to identify a plurality of closest points in the image up to one or more cameras in two separate images;
Identifying at least two different sides of the object from the plurality of closest points;
Creating constructions incorporating at least two different aspects of the object;
Merging the constructs together into a representation of the object;
Storing the representation of the object. Determining a plurality of points of one of the components to connect to another component;
The method of claim 9, further comprising aligning the component at the plurality of points.   The method of claim 10, wherein at least one of the images is received from a server. For each image, identifying a boundary of the object;
The method of claim 9, further comprising: filling at least one color gap between two boundaries to alleviate a portion of a seam between the constructs when merging the constructs. The method described.   The method of claim 9, wherein the image includes color data and depth data. A camera capable of capturing or receiving images each containing color data and depth data;
One or more computer storage media storing the color data and the depth data of at least one image;
One or more processors,
(1) identifying an object in the at least one image;
(2) creating a digital representation of the object in the at least one image;
(3) one or more processors configured to create a 3D representation of the object by combining the digital representation with a second digital representation of the object created from a second image;
A computing device comprising: a display device configured to display the 3D display.   The one or more computer storage media of claim 14, wherein the one or more processors move the object on the display device according to a set of rules for moving the object's limbs.