DE112016003134T5 - Display objects based on a variety of models - Google Patents

Abstract

Provided are a system and method for displaying surfaces of an object (240) from a viewing angle (230) that is different from the viewing angle from which a pictorial representation of the object has been acquired (210, 220). In some aspects, the pictorial representation (710) may be created for display by combining visual characteristics from multiple source images (215, 225) and applying greater weight to the visual characteristics of some of the source images relative to the other source images. The weighting may be based on the orientation of the surface (310) relative to the location from which the image was acquired (320) and the location from which the object will be displayed (430).

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Patent Application No. 14 / 877,368, filed October 7, 2015, the disclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

Certain panoramas of objects are associated with information relating to the geographic location and an orientation from where the image was acquired. For example, each pixel of the image may be associated with data that identifies the angle from the geographic location from which the image was acquired to the partial area of the surface of an object (if any), its appearance by the visual characteristics of the object Pixels is shown. Each pixel may also be associated with depth data that identifies the distance from the detection location to the portion of the surface represented by the pixel.

Three-dimensional models of the locations of surfaces appearing in the images can be generated based on the depth data. The models contain polygons whose vertices correspond to surface locations. The polygons may be intersected by projecting visual characteristics of the panoramic image onto the model using ray tracing. A user can select a viewing angle from which the models can be displayed to the user.

BRIEF SUMMARY OF THE INVENTION

Aspects of the disclosure provide a system that includes one or more processors, a memory that stores a model of the orientation and visual characteristics of the surface of an object relative to viewing angles, and instructions that are executable by the one or more processors , The visual characteristics may include a first group of visual characteristics representing the appearance of the surface from a first viewing angle and a second group of visual characteristics representing the appearance of the surface from a second viewing angle. The instructions may include: receiving a request for an image of the object from a requested viewing angle that is different from the first viewing angle and the second viewing angle; Identifying a first visual characteristic from the first group of visual characteristics and a second visual characteristic from the second group of visual characteristics; Determining a first weighting value for the first visual characteristic based on the orientation of the surface relative to the requested viewing angle and the first viewing angle; Determining a second weighting value for the second visual characteristic based on the orientation of the surface relative to the requested viewing angle and the second viewing angle; Determining a visual characteristic of the requested image based on the first and second visual characteristics and the first and second weighting values; and delivering the requested image.

Aspects of the disclosure also provide a method of providing an image for display. The method may include: receiving a request for an image of an object from a requested viewpoint; Accessing a model of orientation and the visual Characteristics of the surface of the object relative to viewing angles, wherein the visual characteristics include a first group of visual characteristics representing the appearance of the surface from a first viewing angle, and a second group of visual characteristics representing the appearance of the surface from a second viewing angle from, wherein the first and the second angle are different from the requested angle; Identifying a first visual characteristic from the first group of visual characteristics and a second visual characteristic from the second group of visual characteristics; Determining a first weighting value for the first visual characteristic based on the orientation of the surface relative to the requested viewing angle and the first viewing angle; Determining a second weighting value for the second visual characteristic based on the orientation of the surface relative to the requested viewing angle and the second viewing angle; Determining a visual characteristic of the requested image based on the first and second visual characteristics and the first and second weighting values; and delivering the requested image to the display.

Aspects of the disclosure further provide a non-transitory computer device readable storage medium having computer program readable instructions of a program stored thereon. The instructions, when executed by one or more computing devices, may cause the one or more computing devices to perform a method that includes: receiving a request for an image of an object from a requested viewing angle; Accessing a model of the orientation and visual characteristics of the surface of the object relative to viewing angles, the visual characteristics comprising a first group of visual characteristics representing the appearance of the surface from a first viewing angle, and a second group of visual characteristics; representing the appearance of the surface from a second viewing angle, and wherein the first and second viewing angles are different from the requested viewing angle; Identifying a first visual characteristic from the first group of visual characteristics and a second visual characteristic from the second group of visual characteristics; Determining a first weighting value for the first visual characteristic based on the orientation of the surface relative to the requested viewing angle and the first viewing angle; Determining a second weighting value for the second visual characteristic based on the orientation of the surface relative to the requested viewing angle and the second viewing angle; Determining a visual characteristic of the requested image based on the first and second visual characteristics and the first and second weighting values; and delivering the requested image to the display.

BRIEF DESCRIPTION OF THE DRAWINGS

1 FIG. 10 is a functional diagram of a system in accordance with aspects of the disclosure. FIG.

2 is a graphical representation of an object relative to viewing angles from which the object can be captured and displayed.

3 Figure 12 is a graphical representation of an ellipse created based on a viewing angle and the orientation of a surface.

4 is a graphical representation of a texel ellipse and a pixel ellipse.

5 is a graphical representation of a hidden surface relative to viewing angles from which the object can be captured and displayed.

6 Figure 12 is a graphical representation of a surface from the outside relative to viewing angles from which the object can be captured and displayed.

7 is an example of an image that can be displayed to a user.

8th FIG. 10 is an exemplary flowchart in accordance with aspects of the disclosure. FIG.

DETAILED DESCRIPTION

Overview

The technology relates to displaying an object from a viewing angle different from the viewing angle from which a pictorial representation of the object has been acquired. For example, two or more panoramic images may capture an object from two different angles of view, and a user may request an image of the object from one location between the two detection points. The system may generate the user requested image by interspersing corresponding fragments of the images in proportion to the likelihood that the fragment is a visually accurate representation of the corresponding surface of the object. By way of example, when the user-requested image is generated, the system may calculate a quality value based on the relationship of the detection location and the user-requested point to the orientation of the surface of the object to be displayed. If the fragments are mixed, greater emphasis can be placed on fragments that have better quality values than the other fragments.

Based on a presentation shows 2 two different angles from which two source images of a car were captured. In this example, the angle from which the front of the car is detected is relatively orthogonal in the first source image and relatively acute in the second source image. In contrast, the angle from which the side of the car is detected is relatively acute in the first image and relatively orthogonal in the second image. The figure also shows a viewing angle selected by a user to view the car.

To display the object from the user-selected viewing angle, the system can generate a three-dimensional (3D) model of all the surfaces captured at each source image. For example, a laser range finder has been used to prepare a depth map, which in turn has been used to prepare a source model containing a grid of polygons whose vertices correspond to the locations of points along the surfaces of the object.

The source model associated with each source image may also identify the location of the detection point relative to the model, and the system may use that location to project the visual information captured in the source image onto the model. Essentially, the 3D model associated with a source image can may be identical to the 3D model associated with another source image in terms of surface locations, but the visual characteristics of textures projected onto the models may vary depending on the angle from which the surfaces were acquired ,

If the visual characteristics of the pixels of the image to be displayed to the user are determined, the system may use ray tracing and the location of the user requested viewing angle to identify the location at which a ray extending through each pixel displayed will be the one Textures of the model cuts, eg a Texel. The system may merge the texels from the different source images to determine the visual characteristics of the displayed pixel (e.g., hue, saturation, and brightness).

When the texels are merged from the models of the source images, a larger weighting on the texel from one source image over a texel from another source image can be applied. The weighting may be based on a quality score that takes into account the likelihood that the texel is an accurate representation of the visual characteristics of an object to be displayed.

In at least one aspect, the quality value may depend on the resolution of the displayed pixels relative to the resolution of the texels. For example, it can be defined that optimal quality occurs when there is a single texel for each displayed pixel. By contrast, it can be defined that low quality occurs when there are many texels associated with a single pixel (which can occur when a texture of a surface is detected straight ahead but viewed at a grazing angle), or many Can give pixels associated with a single texel (which can occur if the texture of a surface was detected at a grazing angle but viewed in straight line).

The quality value of a texel may be calculated based on the location of the user-defined and acquisition viewing angles relative to the orientation of the surface to be displayed. Based on a presentation shows 3 the location of two points: a point of view and a point on the surface of the model to be displayed to the user. The figure also shows an ellipse representing the interface of a cone and a plane. The plane reflects the orientation of the surface relative to the viewing angle, eg a plane defined by the vertices of the source model polygon containing the texel. The cone is centered on a line that extends from the viewpoint to the surface point (the "gaze / surface line"). The extent to which the ellipse is stretched is related to the orientation of the surface and the angle from which the surface is viewed from the viewing angle. If the gaze / surface line is perfectly orthogonal to the orientation of the surface, the ellipse will be a circle. As the fixed angle of the gaze / surface line becomes more accurate relative to the orientation of the surface, the ellipse will be stretched more and the ratio of the major axis of the ellipse relative to the minor axis will increase.

The quality value of a texel may be based on the differences between an ellipse associated with the detection point (the "texel ellipse") and an ellipse associated with the user-requested angle of view (the "pixel ellipse"). be determined. 4 provides an example of a texel ellipse and a pixel ellipse. The quality value may be calculated from the ratio of the radius of the pixel ellipse relative to the radius of the texel ellipse at an angle that gives the largest difference in length between the two radii.

Once the quality value has been calculated for each texel, the quality value can be applied as weights during a merge. For example, if three source images were used to identify three texels T ₁ , T _2, and T ₃ , the output can be written as (w ₁ T ₁ + w ₂ T ₂ + w ₃ T ₃ ) / (w ₁ + w ₂ + w ₃ ), where w _{n is} equal to the quality value of the texel. The weights can also be applied in other ways.

The system may also use weighting values to address occlusion. As it is in 5 2, a beam used to determine the characteristics of a displayed pixel may extend through surfaces of the object that were captured in the source images but would be biased at viewing at the user requested viewing angle. The system can render both forward and rearward surfaces so that the weighting for each rearwardly facing surface is set to zero. The system may also generate the image so that the surface closest to the viewer is selected for display within each source image, thereby allowing the system to blend textures from both models without computing a depth.

Film errors can also be addressed with weight values. For example and how it works in 6 Discontinuities in depth data can cause gaps in surfaces incorrectly modeled as the surface of an object. Rather than removing non-existent surfaces from a model, the system can determine a quality value for a texel of the nonexistent surface. If the angle of the beam is relatively orthogonal to the orientation of the non-existent surface, the quality value of the texel may be very low compared to the quality value of a texel on another surface detected under a different angle. However, if the angle at which the non-existent surface is viewed is relatively parallel to the orientation of the nonexistent surface, the quality value of the texel on that surface may be relatively high.

The system can be used to display the object to the user from the viewpoint requested by the user. In this regard and as it is in 7 As shown, a user may be able to view the object at angles other than the angles at which source imaging was detected.

Exemplary systems

1 represents a possible system 100 in which the aspects disclosed herein may be implemented. In this example, the system can 100 computer devices 110 and 120 contain. The computer device 110 can be one or more processors 112 , a store 114 and other components typically present in general computing devices. Even though 1 functionally each of the processor 112 and the memory 114 as a single block within the device 110 , which is also shown as a single block, may include the system and may include the methods described herein: multiple processors, memory, and devices that may or may not be stored within the same physical housing. For example, various methods, which are described below as including a single component (eg, the processor 112 ) contain a variety of components (such as multiple processors in a load balanced server farm). Likewise, various methods, which are described below to include different components (such as the device 110 and the device 120 ) contain a single component (eg, the device 120 rather than that the device 120 performs a determination described below, the relevant data for the device 110 send for processing and receive the results of the determination for further processing or display).

The memory 114 the computer device 110 can store information by the processor 112 accessible, including instructions 116 that can be executed by the processor. The memory 114 can also data 118 included by the processor 112 can be read out, manipulated or stored. The memory 114 can be any type of memory that can store information accessible by the relevant processor, such as media that can store nonvolatile data. By way of example, the memory 114 a hard disk drive, a solid state drive, a memory card, a RAM, a DVD, a writable memory, or a read-only memory. In addition, the memory may include a distributed memory system in which data, such as data 150 are stored on a variety of different storage devices that may be physically located at the same or different geographical locations.

The instructions 116 can be any group of instructions to get through the processor 112 or another computing device to be executed. In this regard, the terms "instructions", "application", "steps" and "programs" may be used interchangeably herein. The instructions may be stored in an object code format for intermediate processing by a processor, or in any other computer device language, including scripts or collections of independent source code modules that are interpreted upon request or compiled in advance. Functions, procedures and routines of the instructions are explained in more detail below. The processor 112 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a particular component, such as an ASIC or other hardware-based processor.

The data 118 can through the computer device 110 according to the instructions 116 be read, saved or modified. For example, although the subject matter described herein is not limited by any particular data structure, the data may be stored in computer registers, in a relational database as a table with many different fields and records, or as XML documents. The data may also be formatted in any computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, the data may include any information sufficient to identify the relevant information, such as numbers, descriptive text, ownership codes, pointers, references to data stored in other memories, such as other network sites, or information used by a function to compute the relevant data.

The computer device 110 can be at a node of a network 160 be and be able to directly and indirectly connect to other nodes of the network 160 to communicate. Although in 1 While only a few computing devices are shown, a typical system may include a large number of connected computing devices, each different computing device at a different node of the network 160 is. The network 160 and intervening nodes described herein may be interconnected using various protocols and systems such that the network may be part of the Internet, the Word Wide Web, specific intranets, wide area networks, or local area networks. The network may use standard communication protocols, such as Ethernet, Wi-Fi, and HTTP, protocols owned by one or more companies, and various combinations of the foregoing. As an example, the computing device 110 a web server working with the computer device 120 over the network 160 can communicate. The computer device 120 may be a client computing device and the server 110 can get information by using the network 160 Show information to a user 135 the device 120 via an ad 122 to transmit and present. Although certain advantages are obtained when transmitting or receiving information, as stated above, other aspects of the subject matter described herein are not limited to any particular type of sending of information.

The computer device 120 can be the same or similar to the server 110 be configured with a processor, memory and instructions as described above. The computer device 120 may be a personal computer device intended for use by a user and having all the components normally used in connection with a personal computer device, such as a central processing unit (CPU), a memory storing data and instructions, a display , such as the ad 122 (eg, a monitor with a screen, a touch screen, a projector, a TV or other device that is operable to display information), a user input device 162 (eg, a mouse, a keyboard, a touch screen, a microphone, etc.), and a camera 163 ,

The computer device 120 may also be a mobile computing device that can wirelessly exchange data with a server over a network, such as the Internet. By way of example only, the device 120 a mobile phone or device such as a wireless PDA, a tablet PC, a portable computing device, or a netbook that can receive information over the Internet. The device may be configured to work with an operating system, such as Google's Android operating system, Microsoft's Windows, or Apple's iOS. In this regard, some of the instructions that are executed during the operations described herein may be provided by the operating system, while other instructions may be provided by an application installed on the device. Computer devices according to the systems and methods described herein may include other devices that can process instructions and send and / or transmit data to and from humans, and / or other computers, including network computers lacking local storage capability and digital receivers for televisions ,

The computer device 120 can be a component 130 such as circuits to determine the geographic location and orientation of the device. For example, the client device 120 a GPS receiver 131 to determine the latitude, longitude, and elevation of the device. The component may also include software for determining the position of the device based on other signals present at the client device 120 are received, such as signals that are received at an antenna of a cell phone from one or more cell phone towers when the client device is a cellular phone. You can also use a magnetic compass 132 , an accelerometer 133 and a gyroscope 134 to determine the direction in which the device is oriented. By way of example only, the apparatus may determine its pitch, yaw, or roll (or changes thereto) relative to the direction of gravity or plane perpendicular thereto. The component 130 may also include a laser range finder or similar device for determining the distance between the device and the surface of an object.

The server 110 may store card-related information, from which at least a portion may be transmitted to a client device. The map information is not limited to any particular format. For example, the map data may be bitmap images of geographic Locations, such as photographs captured by a satellite or aircraft.

The server 110 can also store a pictorial representation, such as by way of example only, a flat photograph, a photosphere, or a video of a scene. The pictorial representation may be captured and uploaded by end users for the purpose of making the photo available for later access, or for anyone looking for information regarding the feature. In addition to the image data captured by a camera, individual elements of a pictorial representation may be associated with additional data, such as a date of capture, time or day of capture, geographic orientation (eg, camera angle or direction), and location of capture ( eg latitude, longitude and altitude).

Portions of the pictorial representation may be associated with additional information, including a model of the geographic location of features that appear within the pictorial representation. For example, the model may identify the location of surfaces of objects captured in a panoramic image. The location of the surfaces can be stored in memory in different ways, e.g. a constellation of points whose location is defined as a fixed angle and distance from a fixed location (eg, orientations and distances from the point from which the imagery was captured), or geographically located polygons whose vertices are latitude / longitude / Altitude coordinates are expressed. The system and method may further implement locations from one reference system to another, such as generating a model of geographically located polygons from a constellation of points whose locations have been directly acquired, using a laser rangefinder, or generating images by using a stereographic triangulation. Locations can be expressed in other ways as well and depend on the type of application and a required precision. By way of example only, geographical locations may be identified by a street address, xy coordinates relative to edges of a map (such as a pixel position relative to the edge of a road map), or other reference systems that can identify geographic locations (eg, land and block numbers on survey maps). , be identified. A location may also be described by a range, e.g. For example, a geographic location may be described by a range or discrete series of latitude / longitude / elevation coordinates.

Exemplary methods

Operations in accordance with a variety of aspects of the invention will now be described. It should be understood that the following operations need not be performed in the precise order described below. Rather, different steps may be handled in a different order or simultaneously.

A geographic object can be captured by a pictorial representation from multiple angles. Using an example shows 2 a car 240 that in two separate pictures 215 and 225 each from two different locations 210 and 220 was recorded from. From the perspective 210 is the camera angle 211 to a level 202 , which is generally defined by the front of the car, is relatively orthogonal and is the camera angle 212 to a level 201 , which is generally defined by the side of the car, relatively pointed. In contrast to this is from the perspective 220 the point of view 221 to a front level 202 relatively pointed and is the point of view 222 to the page level 201 relatively orthogonal. (Reference 201 and 202 are used interchangeably to lean on the front and side surfaces of the car 240 and the levels generally defined by these surfaces.) In this regard, those of the sites 210 and 220 From detected images, different surfaces of the car at different angles, one surface is detected relatively straight ahead and the other is detected at an acute angle.

A model of the location of the object surfaces captured in the pictorial representation may be generated and associated with the pictorial representation. By way of example, the pictorial representation of the source may include a panoramic image. At the time a source image was captured, a laser range finder may be used 135 or another depth finding technique has been used to associate each pixel of the panoramic image with the distance from the detection location to the portion of a surface whose visual characteristics are represented by the pixel. Based on such information and the location from which the image was acquired (eg, latitude / longitude / elevation information provided by a GPS receiver 131 supplied), the system can generate a source model containing a grid of polygons (eg triangles) whose vertices are the location of points correspond (eg latitude / longitude / altitude), along the surface of the object.

The visual characteristics captured in the source image can be used to texture the model associated with that source image. For example, each pixel may be pictorial 215 and 216 also associated with a camera angle, eg, data defining a beam extending from the camera to the portion of the surface associated with the pixel. The camera angle data may be based on information provided by a geographic component 130 is delivered at the time the image was captured, such as the compass direction 132 is delivered, and orientation data by the gyroscope 134 are delivered. A ray tracing can be used to visualize the visual characteristics of each pixel of the image 215 on the polygons of the model 216 so that the visual characteristics of the polygon at the interface with the beam match the visual characteristics of the associated pixel.

The visual information of the polygons can be stored as textures made of texels. By way of example only, the texels of a texture may be arranged in a manner similar to the manner in which pixels of an image may be placed, e.g. a grid or other collection of individual units that define one or more visual characteristics. Portions of the following description may, for convenience of explanation, refer only to a single visual characteristic of a pixel or texel, such as color. However, pixels and texels may be associated with data that defines many different visual characteristics, including hue, saturation, and brightness.

Even if two different source models contain identical representations of a location of a surface, the visual characteristics of the surface may differ greatly in each model. For example, because a pictorial representation 215 the page 201 a car 240 at an acute angle 212 captures the entire length of the side of the car in just a few pixels of a picture 215 appear horizontally. As a result, when the pixel information is projected onto polygons that represent the side of the car, the information may extend from a single pixel over many texels in the horizontal direction. If the model 216 from the perspective 230 Thus, the side of the car may appear to have long horizontal color stripes as a result. However, if the model 216 from the perspective 210 (the same location from which the textures were projected) could display the page 201 of the car 240 with some or no movie errors.

The system can combine visual information captured at multiple angles to display a geographic object from a different angle. For example, a user can take a picture 235 of the car 240 from a single point of view 230 Requests. If the visual characteristics of the pixels of the image to be displayed to the user are determined ("displayed pixels"), the system may associate each displayed pixel with a ray extending from the viewing angle and through an image plane containing the pixel. The system may determine the point and a texel in which each beam associated with a pixel intersects a surface defined by the model. For example, if the visual characteristics of a pixel in the image 235 determine which texels (T ₁ ) determine where the ray of the pixel is a polygon of a model 216 and the texel (T ₂ ), where the ray of the pixel is a polygon of the model 226 cuts. The system can determine the color of the pixel by alpha blending the color of T ₁ and the color of T ₂ .

When the visual characteristics of the cut texels are mixed together, a larger weighting can be applied to the texel derived from one source image versus a texel derived from another source image. The weighting may be based on the likelihood that the texel is an accurate representation of the visual characteristics of the object to be displayed.

In at least one aspect, the quality value may depend on the resolution of the displayed pixels relative to the resolution of the texels; it can be defined that optimal quality occurs when there is a single texel for each displayed pixel. In contrast, it can be defined that low quality occurs when there are many texels associated with a single pixel, or there are many pixels associated with a single texel. By way of example, a first polygon may be generated in which there is a single texel for each pixel projected on it. If the pixels were projected onto the first polygon at a relatively straight-line angle, the texture may contain many close-packed texels. If this polygon is displayed straight ahead in one direction, there may be roughly a texel for each displayed pixel, and thus the texture would be considered to have a relatively high quality. However, if the polygon were displayed at an acute angle, there could be many texels for each pixel displayed and would be the Texture can be considered such that it has a relatively low quality despite the relatively high resolution of the texture.

By way of further example, a second polygon may be generated, with a single texel for each pixel projected thereon. However, if the pixels were projected onto the second polygon at a relatively acute angle, the texture may contain long and thin texels. If this polygon is then displayed facing straight, many displayed pixels would derive their characteristics from only a single texel, in which case the texture would be considered to have a relatively low quality. However, if the polygon is displayed at a relatively acute angle, there may be only one texel for each displayed pixel, in which case the texture would be considered to have a relatively high quality despite the relatively low resolution of the texture.

The quality value of a texel may be calculated based on the location of the user-defined and detection viewing angles relative to the orientation of the surface to be displayed. Based on a representation, a point 320 of the 3 the geographic location from which a geographically located polygon of a model will be viewed. The point 330 is the point where the camera angle associated with a displayed pixel intersects the polygon. A line "s" extends from the viewpoint 320 to the interface 330 , A cone 350 extends from the point 320 at an acute fixed angle ("α") to the outside. The measure of α can be arbitrarily selected. The measure of α can also be chosen so that the cone provides a similar fixed angle as the texel, as seen from the location from which the texture was detected, eg, the higher the texture resolution, the smaller the cone. An ellipse 310 represents the interface of the cone 350 and a plane (not shown). The plane reflects the orientation of the surface at the interface, eg, a plane defined by the corner points of the polygon that defines the interface 330 contains.

The extent to which the ellipse is stretched refers to the orientation of the surface and the viewing angle. For example, if the line "s" is perfectly orthogonal to the orientation of the plane (which would be associated with looking at the surface in a straight direction), the ellipse would 310 to be a perfect circle. When the fixed angle of the line "s" becomes sharpened relative to the orientation of the plane, the ellipse becomes 310 more stretched and will increase the ratio of the major axis of the ellipse ("b") relative to the minor axis ("a"), where "n" is the surface normal. The axes "a" and "b" can be determined from the equation a = αs × n and b = (n · s / | s |) (a × n).

The quality value of a texel may be based on the differences between the ellipse associated with a cone extending from the detection site to the interface (the "texel ellipse") and the ellipse associated with a cone that is different from the cone Location selected by the user extends to the interface (the "pixel ellipse"). In 4 is the Texel ellipse 425 with the detection site 420 and is the pixel ellipse 435 with a user-requested angle 430 associated. The quality value of the texels at the interface 450 can be calculated from the ratio of the radius of the pixel ellipse relative to the radius of the texel ellipse at a certain angle θ, eg quality (θ) = radius _t (θ) / radius _p (θ). In one aspect, the angle θ is the angle or an estimate of the angle that yields the minimum ratio. For example, the quality value may be calculated at different values of θ and the quality value of the texel may be equal to the lowest of the calculated values, eg quality _min = quality (argmin _θ {quality (θ)}). The minimum can be determined by expressing both ellipses in matrix form and multiplying the pixel ellipse by the inverse of the texel ellipse, mapping the pixel ellipse into a coordinate system where the texel ellipse would be a unit circle. Within this coordinate system, the ratio is equal to the newly mapped radius of the pixel ellipse, and the minimum ratio is equal to the smaller axis length of the newly mapped pixel ellipse.

The quality value may be estimated by projecting each of the texel ellipse axes onto each of the pixel ellipse axes, and by selecting the pixel-ellipse axis that gives the largest ratio. By way of example, a shader may calculate the minimum quality value by calculating values according to the equation quality _min -1 / max ((a _t * a _p ) / (a _p * a _p ), (b _t * a _p ) / (a _p · A _p ), (a _t · b _p ) / (b _p · b _p ), (b _t · b _p ) / (b _p · b _p )), which samples four possible directions and selects the angle that associated with the lowest quality score. Other methods can also be used to calculate the quality.

Once the quality of a texel has been calculated, the quality can be used to determine how similar the displayed color of a pixel will be to the color of a texel. For example, if three source images were used to identify three texels T ₁ , T _2, and T ₃ , the blending weight for each texel in a fragment shader for each input texel and output (w ₁ T ₁ + w ₂ T ₂ + w ₃ T ₃ ) / (w ₁ + w ₂ + w ₃ ), where w _{n is} equal to the quality value of the texel. The weights may also be applied in other ways, such as by increasing the quality values to a power. For example, by increasing the weights by a large exponent, the surface with the largest weight may dominate the other weights, thereby reducing ghosting or stenciling by reducing the impact of the lower weighted texels.

The system may also use weighting values to address obstruction, as illustrated by an example in FIG 5 is shown. surfaces 502 and 503 were in a picture 520 captured from an angle A and surfaces 501 and 503 were in a picture 530 detected from a perspective B. A model 521 and 531 can each for each picture 520 and 531 be generated. When the model is created, the model may associate the texture with a specific side of the polygon. For example, the surface 502 in a model 521 be represented by a single triangle, with one side directed towards the angle of view A and the other side is directed away from the angle of view A. When the image data is projected onto the model, the model can indicate if the texture is on the side of the triangle that is directed toward the viewing angle.

When the requested image is created, a hidden surface may be hidden by determining if its associated texture is directed toward or away from the requested viewing angle and determining if the texture is closer to the viewing angle than other textures in the same model. For example, the system may, if the color of a displayed pixel under a viewing angle 540 is determined to identify all of the points at which the beam associated with the pixel 550 every surface cuts, namely the points 511 - 513 , The system may also determine for each model the cut texel closest to the viewpoint and whether the texture of the cut texture on the side of the polygon is toward ("forward") or away from ("backward"). ) is the point of view. Thus, the system may determine that T _{511B is} the next texel in the model 531 and is directed to the front (where "T _pppm" refers to the texel is at the point of interface ppp and stored in the view angle with the associated m model). The system can also determine that T _{512A is} the next texel in the model 521 is and is directed backwards. Because T _{512A is} facing backwards, the system can automatically set its weight to zero. As a result, the color of the pixel associated with the ray 550 are determined by (w _511B T _511B + w _512A T _512A ) / (w _511B + w _512A ), where w _{511B is} the quality _value determined for T _511B and w _{512A is set} to zero. As a result, the color of the displayed pixel would be the same as T _511B .

Alternatively, rather than ignoring certain textures or setting their weights to zero, their relative weights may be reduced. For example, rather than setting w _512A to zero and ignoring T _513A and T _513B, all of them, the back-facing texels and other texels may also be used for blending, but with reduced weights.

Film errors can also be addressed with weighting values, including film defects caused by discontinuities in depth data. For example and how it works in 6 shown is a model 621 be associated with the image from a standpoint 620 is captured, and can be a model 631 be associated with a picture from a point of view 630 is detected. The gap 603 between a surface 601 and 602 can in the model 621 be shown exactly. However, an inaccuracy arising from the depth data read out at the time of detection or any other error occurring during generation of the model 631 occurred, resulting in that the model 631 incorrectly indicates that a surface 635 over the gap 603 extends. If the model 631 from the perspective 640 could be viewed, an outer surface 635 have the appearance of a rubber sheet stretched from one edge of the gap to the other. For some aspects, the system can check for external polygons and remove them by comparing the model 621 with the model 631 ,

In other aspects, the system may use the outer surfaces when creating an image for display. For example, a user may request an image asking the surfaces from a viewpoint 640 displays. If the contribution of the model 621 is determined to the color of the pixel, with the beam 641 The system may have a point A on the surface 601 identify as the first statement point. The system may also determine that the texel has a relatively high quality value at point A because the texel is viewed at an angle equal to the angle at which it was detected (location 620 ). If the contribution of the model 631 is determined to the color of the pixel, the system may have a point B on an outer surface 635 identify as the first interface. The system may determine that the texel has a relatively low quality value at point B because the texel is viewed at an angle that is relatively orthogonal to the angle which it was captured (location 630 ). As a result, when the texels are intermingled at points A and B, a relatively small weighting will be applied to the texel at point B; the color of the pixel becomes almost entirely on the color of point A on the surface 601 based.

At certain angles, the outer surfaces may have a significant contribution to the visual characteristics of the image to be displayed. For example, a user may view an image from a viewpoint 650 which is relatively close to the detection location 630 of the model 631 is. If the contribution of the model 631 the color of the pixel is determined with the beam 641 the system may have a point C on the outer surface 635 identify as the first interface and can further determine that the texel is at point C of the model 631 has a relatively high quality value. If the contribution of the model 621 is determined to the color of the same pixel, the system can not identify an interface when the surface 602 not represented by the model. Alternatively, if the surface 602 represented by the model, the quality value of the texel at the interface in the model 621 be relatively low, because the angle from the interface to the viewing angle 650 relatively orthogonal to the angle from the interface to the acquisition location 620 is. In any case, and as a result, the color of the displayed pixel may be substantially the same as the texel of the outer surface. This can be particularly beneficial if the model 621 no representation of the surface 602 has as an indication of the outer surface 635 displaying nothing (eg, a color indicating the absence of a surface).

Users can use the system and interact with the models. Just by way of example and with reference to 1 the user can 135 a camera 163 and a geographical component 130 the client device 120 use to capture image data and depth data from multiple angles. The user 135 can the image and depth data to the server 110 Upload. The processor 112 may generate a textured model for each image based on the data provided by the user and the data in memory 114 to save. The server 110 can then receive a request for an image of the object from a specific viewpoint. For example, a user 135 (or another user using the client device 121 ) a panoramic image at a specific geographical location by selecting the geographical location with a user input 162 and by sending a request to the server 110 Request. Upon receiving the request, the server may 110 Read two or more models based on the requested geographic location, such as by selecting all or a limited number of models having locations or surface locations within a threshold distance from the requested geographic location. The server can then use the models over a network 160 to the client device 120 send. Upon receipt of the models, the processor of the client device may 120 generate a panoramic image based on the models and using the requested geographic location as the viewing angle. If the requested geographic location did not include elevation, the elevation of the elevation may be based on the altitudes of the detection locations of the models.

The panoramic picture may be on the display 122 are displayed. For example and with reference to the 2 and 7 can then if the models 216 and 226 a car 240 sent to the client device, an image 710 of the car 240 based on a viewpoint 230 generated and displayed to the user. The user may select different viewpoints other than the detection sites by providing commands to the client device user interface, eg, pressing buttons to pan the image.

8th FIG. 10 is a flowchart according to some of the aspects described above. FIG. At a block 801 a model of the surface of an object is accessed, comprising: the orientation of the surface of an object relative to a first viewing angle and a second viewing angle; a first group of visual characteristics representing the appearance of the surface at a first viewing angle; and a second set of visual characteristics that represent the appearance of the surface under a second viewing angle. At a block 802 For example, a request for an image of the object is received at a requested viewing angle different from the first viewing angle and the second viewing angle. At a block 803 For example, a first visual characteristic from the first group of visual characteristics is identified and a second visual characteristic from the second group of visual characteristics is identified. At a block 804 For example, a first weighting value for the first visual characteristic is determined based on the orientation of the surface relative to the requested viewing angle and the first viewing angle. At a block 805 For example, a second weighting value for the second visual characteristic is determined based on the orientation of the surface relative to the requested viewing angle and the second viewing angle. At a block 806 For example, a visual characteristic of the requested image is determined based on the first and second visual characteristics and the first and second weighting values. At a block 807 the requested image is delivered to the user for display.

Since these and other variations and combinations of the above-discussed features may be utilized without departing from the invention as defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than limitation of the invention, such as: it is defined by the claims. It will also be understood that the provision of examples of the invention (as well as halfphrases which are formulated as "such as," "for example," "including," and the like) should not be interpreted as limiting the invention to the specific examples; rather, the examples are intended to depict only a few or many possible aspects.

Claims (20)

System comprising: one or more processors, a memory storing a model of the orientation and visual characteristics of the surface of an object relative to viewing angles, the visual characteristics comprising a first group of visual characteristics representing the appearance of the surface at a first viewing angle and a second group of visual characteristics representing the appearance of the surface under a second viewing angle, and Instructions executable by the one or more processors, comprising: Receiving a request for an image of the object at a requested viewing angle that is different from the first viewing angle and the second viewing angle; Identifying a first visual characteristic from the first group of visual characteristics and a second visual characteristic from the second group of visual characteristics; Determining a first weighting value for the first visual characteristic based on the orientation of the surface relative to the requested viewing angle and the first viewing angle, Determining a second weighting value for the second visual characteristic based on the orientation of the surface relative to the requested viewing angle and the second viewing angle, Determining a visual characteristic of the requested image based on the first and second visual characteristics and the first and second weighting values; and Deliver the requested image. The system of claim 1, wherein the request is received by a user and providing the requested image comprises displaying the requested image to the user. The system of claim 1, wherein determining a visual characteristic of the requested image comprises determining a visual characteristic more similar to the first characteristic than the second characteristic, and the measure of similarity is based on the first weighting value relative to the second weighting value. The system of claim 3, wherein: the model stored in memory comprises a first model and a second model; the first group of visual characteristics are texels generated on the first model by projecting image data captured from the first viewpoint; and the second group of visual characteristics are texels generated on the second model by projecting image data captured from the second viewpoint. The system of claim 1, wherein the visual characteristics are associated with colors and wherein determining a visual characteristic of the requested image comprises determining a color by alpha blending proportional to the first weighting value relative to the second weighting value, a color associated with the first visual Characteristics are associated, and a color that is associated with the second visual characteristic includes. The system of claim 1, wherein the orientation of the surface defines a plane, and: the instructions further comprise selecting a point on the level; determining a first weighting value determining a value based on the relationship of the fixed angle between the plane and a line extending from the first viewing angle to the selected point and the fixed angle between the plane and a line extending from the requested viewing angle to extends selected point includes; and determining a second weighting value, determining a value based on the relationship of the fixed angle between the plane and a line extending from the second angle of view to the selected point, and the fixed angle between the plane and a line extending from the requested angle of view extends selected point includes. The system of claim 6, wherein the instructions further include selecting a pixel of the requested image and determining a direction with the selected pixel, and wherein: selecting a point on the plane comprises selecting the point at which the direction associated with the pixel intersects the plane; and determining a visual characteristic of the requested image comprises determining a visual characteristic of the selected pixel. The system of claim 6, wherein determining a first weighting value comprises determining a value based on a radius of a first ellipse and a corresponding radius of the second ellipse, wherein the first ellipse is defined by the intersection of the plane and a cone surrounding the line centered, which extends from the first viewing angle to the selected point, and wherein the second ellipse is defined by the interface of the plane and a cone centered around the line extending from the requested viewing angle to the selected point. The system of claim 8, wherein determining a first weighting value further comprises determining a ratio based on an estimate of the radius of the first ellipse relative to an estimate of the radius of the second ellipse at a selected angle. A method of providing an image for display, comprising: Receiving a request for an image of an object from a requested viewpoint; Accessing a model of the orientation and visual characteristics of the surface of the object relative to viewing angles, wherein the visual characteristics comprise a first group of visual characteristics representing the appearance of the surface from a first viewing angle, and a second group of visual characteristics depict the appearance of the surface from a second viewing angle, and wherein the first and second viewing angles are different from the requested viewing angle; Identifying a first visual characteristic from the first group of visual characteristics and a second visual characteristic from the second group of visual characteristics; Determining a first weighting value for the first visual characteristic based on the orientation of the surface relative to the requested viewing angle and the first viewing angle, Determining a second weighting value for the second visual characteristic based on the orientation of the surface relative to the requested viewing angle and the second viewing angle, Determining a visual characteristic of the requested image based on the first and second visual characteristics and on the first and second weighting values; and Deliver the requested image to the ad. The method of claim 10, wherein the request is received from a user, and providing the requested image comprises displaying the requested image to the user. The method of claim 10, wherein determining a visual characteristic of the requested image comprises determining a visual characteristic that is more similar to the first characteristic than the second characteristic, and the degree of similarity is based on the first weighting value relative to the second weighting value. The method of claim 12, wherein: the model comprises a first model and a second model; the first group of visual characteristics are texels generated on the first model by projecting image data captured from the first viewpoint; and the second group of visual characteristics are texels generated on the second model by projecting image data captured from a second viewing angle. The method of claim 10, wherein the visual characteristics are associated with colors and wherein determining a visual characteristic of the requested image comprises determining a color by alpha blending proportional to the first weighting value relative to the second weighting value, a color corresponding to the first visual characteristic and a color associated with the second visual characteristic. The method of claim 10, wherein the orientation of the surface defines a plane, and further comprising: Selecting a point on the plane; determining a first weighting value determining a value based on the relationship of the fixed angle between the plane and a line extending from the first viewing angle to the selected point and the fixed angle between the plane and a line extending from the requested viewing angle to extends selected point includes; and determining a second weighting value, determining a value based on the relationship of the fixed angle between the plane and a line extending from the second angle of view to the selected point, and the fixed angle between the plane and a line extending from the requested angle of view extends selected point includes. The method of claim 15, further comprising selecting a pixel of the requested image and determining a direction associated with the selected pixel, and wherein: selecting a point on the plane comprises selecting the point at which the direction associated with the pixel intersecting the plane; and determining a visual characteristic of the requested image comprises determining a visual characteristic of the selected pixel. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions of a program, the instructions, when executed by one or more computing devices, causing the one or more computing devices to perform a method, the method comprising: Receiving a request for an image of an object from a requested viewpoint; Accessing a model of the orientation and visual characteristics of the surface of the object relative to viewing angles, wherein the visual characteristics comprise a first group of visual characteristics representing the appearance of the surface from a first viewing angle, and a second group of visual characteristics depict the appearance of the surface from a second viewing angle, and wherein the first and second viewing angles are different from the requested viewing angle; Identifying a first visual characteristic from the first group of visual characteristics and a second visual characteristic from the second group of visual characteristics; Determining a first weighting value for the first visual characteristic based on the orientation of the surface relative to the requested viewing angle and the first viewing angle, Determining a second weighting value for the second visual characteristic based on the orientation of the surface relative to the requested viewing angle and the second viewing angle, Determining a visual characteristic of the requested image based on the first and second visual characteristics and the first and second weighting values; and Deliver the requested image to the ad. The storage medium of claim 17, wherein the request is received from a user, and providing the requested image comprises displaying the requested image to the user. The storage medium of claim 17, wherein determining a visual characteristic of the requested image comprises determining a visual characteristic that is more similar to the first characteristic than the second characteristic, and the similarity measure is based on the first weighting value relative to the second weighting value. A storage medium according to claim 19, wherein: the model comprises a first model and a second model; the first group of visual characteristics are texels generated on the first model by projecting image data captured from the first viewpoint; and the second group of visual characteristics are texels generated on the second model by projecting image data captured from a second viewing angle.

Images