US20130101175A1 - Reimaging Based on Depthmap Information - Google Patents
Reimaging Based on Depthmap Information Download PDFInfo
- Publication number
- US20130101175A1 US20130101175A1 US13/278,499 US201113278499A US2013101175A1 US 20130101175 A1 US20130101175 A1 US 20130101175A1 US 201113278499 A US201113278499 A US 201113278499A US 2013101175 A1 US2013101175 A1 US 2013101175A1
- Authority
- US
- United States
- Prior art keywords
- points
- image
- panoramic image
- location
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/36—Input/output arrangements for on-board computers
- G01C21/3626—Details of the output of route guidance instructions
- G01C21/3635—Guidance using 3D or perspective road maps
- G01C21/3638—Guidance using 3D or perspective road maps including 3D objects and buildings
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C3/00—Measuring distances in line of sight; Optical rangefinders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30236—Traffic on road, railway or crossing
Abstract
Description
- The present patent application is related to the copending patent applications filed on the same date, Ser. No. ______, entitled “DEPTH CURSOR AND DEPTH MEASUREMENT IN IMAGES,” Attorney Docket No. N0406US and Ser. No. ______, entitled “THREE DIMENSIONAL ROUTING,” Attorney Docket No. N0407US, the entire disclosures of which are incorporated by reference herein.
- The present invention relates to maps and/or navigation and, more particularly, to method(s) and system(s) for reimaging images, such as panoramic, street level view, or high-altitude view images, in a map application or a navigation application.
- Navigation applications may be installed in vehicles to provide images to passengers or in portable (e.g., handheld) devices such as phones, personal digital assistants, or personal navigation devices. The navigation application may provide images that aid the user in determining current position and guiding the user to a destination. In addition the images provide information such as points of interest to the user. Alternatively, map applications, which may be stationary or portable, provide users with similar services from a website or other service without regard to the current position of the user.
- In some situations, the user may select an object or specific area of the images to choose a route or request additional information about the object or location. Because navigation or map systems typically involve a two-dimensional (2D) representation of a three-dimensional (3D) image, when an object is selected, distinction between whether the object is a foreground image or a background image is often difficult.
- According to one aspect, one or more systems, devices, and/or methods for emphasizing objects in an image, such as a panoramic image, are disclosed. For example, a method includes receiving a depthmap generated from an optical distancing system, wherein the depthmap includes depth data for each of a plurality of points, and overlaying the depthmap on the panoramic image according to the plurality of points. Data indicative of a location on the panoramic image and indicative of a first point of the plurality of points is received. The depth data of the first point is compared to depth data of surrounding points to identify an area on the panoramic image corresponding to a subset of the surrounding points. The panoramic image is altered with a graphical effect at one or more of the surrounding points.
- The present invention is defined by the following claims, and nothing in this section should be taken as a limitation on those claims. Further aspects and advantages of the invention are discussed below in conjunction with the preferred embodiments.
-
FIG. 1 illustrates an example embodiment of a navigation system or map system. -
FIG. 2A illustrates an image for use with the navigation system or map system ofFIG. 1 . -
FIG. 2B illustrates a predetermined camera angle that the image was collected. -
FIG. 3A illustrates the generation of optical distance data using the optical distancing system. -
FIG. 3B illustratesdepthmap data 303 as generated by the optical distancing system. -
FIG. 3C illustrates depthmap data represented in scan lines. -
FIG. 4 illustrates alignment of the depthmap data. -
FIGS. 5A and 5B illustrate two example depthmap grids -
FIGS. 6A and 6B illustrate the calculation of depth values. -
FIG. 7 illustrates the selection of points for reimaging. -
FIG. 8A illustrates an embodiment where the graphical effect is blurring. -
FIG. 8B illustrates an embodiment where the graphical effect is highlighting. -
FIGS. 9A , 9B and 9C illustrate another embodiment where the graphical effect is highlighting. -
FIG. 10 illustrates an embodiment in which the graphical effect is a simulated spotlight. -
FIG. 11 illustrates a detailed view of the server ofFIG. 1 . -
FIG. 12 illustrates a detailed view of the navigation device ofFIG. 1 . -
FIG. 13 illustrates a flowchart for reimaging a panoramic image. - Map systems or navigation systems provide users with panoramic images, street level view images, or high-altitude images (e.g., satellite, airplane) for navigation related features or map related features. Navigation related features include determining an optimum route and providing directions (e.g., exit cues or turn cues) to travel along a path from an original location to a destination location. The path may be a road network, a pedestrian pathway network, or an indoor walkway network. Navigation related features may be provided by mobile devices. Map related features, or features without a navigational or guidance component, may be provided by mobile devices or stationary devices. Map related features include locating nearby points of interest or directions between two positions without regard to current location. Each of the embodiments described herein may be configured to provide navigation related features, map related features, or both.
- Map related features and navigation related features may involve reimaging the image. Reimaging includes emphasizing, de-emphasizing, isolating, highlighting, accentuating, or focusing a region or object in the image. For example, the destination location may be emphasized to illustrate the destination location to the user. In addition, the user may select an object or location in the image through touching, clicking, or hovering (e.g., mousing) above the image. The selection or attempted selection is illustrated to the user.
- In a 2D image, the map or navigation system may indicate the exact location to the user. However, the user may not be able to distinguish between the object that was desired to be selected or indicated and another object nearby in the 2D image. Often, the objects are substantially apart geographically but appear very close in the 2D representation. Other methods that distinguish between objects in 2D images may be possible, but utilize image processing techniques such as edge detection, feature extraction, and complex filters that require high processing resources. The system for emphasizing objects in a panoramic image utilizes depthmap information to illustrate the selected object to the user based on the principle that any pixel and the surrounding pixels having similar depths make up the same object.
- Depthmaps (depth maps) may be constructed from light detection and ranging (LIDAR) data. LIDAR, which may also be referred to as three-dimensional laser scanning or an optical distancing system, employs one or more lasers or “cameras” to collect data points representing an area, such as an area about a road or walkway. Software generates the depthmap based on the measured distance, the locations of the optical distancing system, which may be on a moving platform such as a car, and the angle of the laser. Other optical distancing systems include a stereoscopic camera, a time of flight infrared camera, and a structured light device. A LIDAR device collects and gathers data points in a point cloud in which each data point corresponds to a local coordinate, such as (x, y, z), and is either on or off. Alternatively, the LIDAR data may be a grayscale point cloud including an intensity (indicating reflectivity) for each data point, in a predetermined range of values (e.g., 0 to 255, 0 to 65536) with black at one end of the range and white at the other. The point cloud may be stored in ASCII or LIDAR exchange format (e.g., the American Society for Photogrammetry and Remote Sensing (ASPRS) .LAS file format). The one or more lasers may be in a near infrared spectrum (such as about 700 nm to about 5000 nm or about 800 nm to about 2500 nm) or other light spectrum.
-
FIG. 1 illustrates asystem 150 for emphasizing objects in an image, such as a panoramic image. Thesystem 150 includes anavigation device 100, anetwork 110, aserver 120, and adatabase 130. Thedatabase 130 may be maintained by a map developer, such as NAVTEQ North America, LLC located in Chicago, Ill. The map developer may collect geographic data to generate and enhance thedatabase 130. Thenavigation device 100 may be a cellular telephone (smart phone), a personal digital assistant (“PDA”), a tablet computer, a laptop, a personal navigation device (“PND”), an in-vehicle navigation system, a personal computer or another computing device. - The
system 150 for emphasizing objects in or reimaging a panoramic image receives a depthmap generated from optical distance data of an optical distancing system. The optical distancing system may be a LIDAR device, a stereoscopic camera, or a structured light device. The depthmap is generated from the collected optical distance data. The depthmap may be created using any arbitrary viewpoint or perspective. The optical distance data is 3D data it is easily manipulate to any viewpoint or perspective, which is in contrast to a 2D image which is valid from a fixed viewpoint. Depthmaps may be generated for each panoramic image viewpoint and stored in thedatabase 130. The depthmap includes position data (X-Y coordinates) and depth data (Z coordinate) for each of plurality of points. - The
system 150 overlays the depthmap on a panoramic image according to the position data. Assuming the optical distance data and the panoramic image were collected from the same perspective and/or location, the overlay may involve combining the depth data with the image data of the panoramic image. If the depthmap and the panoramic image were collected from different angles and/or positions, additional geometric calculations are necessary to properly align the depthmap and the panoramic image. The associations between the pixels of the panoramic image and the depth map values may be stored indatabase 130. In other words, the depthmap and the panoramic image may be pre-aligned and the overlaid or combined data is stored in thedatabase 130. - The user may select an object or a location on the panoramic image or a navigation or map application may select a map object shown on the panoramic image, which is received by
system 150. The map object may be a destination, a point of interest, a turn cue, a street, an address, a route, a sign, a building, etc. Thesystem 150 compares depth data of the first point to depth data of surrounding points and identifies an area on the panoramic image corresponding to the surrounding points having depth data within a predetermined range of the depth data of the first point. Within the panoramic image, a graphical effect is added that indicates at least one of the surrounding points. - The user may select the location on the panoramic by touch or mouse click. Alternatively, when the navigation or map application selects a map object, the selection may be made without any user input. For example, a navigation application may emphasize a map object by reimaging the panoramic image to illustrate the destination, the current path or route, the next turn, or any other navigation related map object. Further, points of interest are map objects that may be illustrated by reimaging the panoramic image. Points of interest may include buildings, addresses, location names, or store fronts.
- The system and method for reimaging color images for any of these implementations utilizes depth information. The depth information provides each pixel with a 3D location that combined with the stored image geo-position and orientation provides the real-world position of each pixel as a 3D value. The 3D value is compared to other real-
world 2D and 3D spatial map data, points, regions, paths, volumes, etc. to enhance the original 2D camera image based on real-world 3D objects rather than simple alterations in ‘flat’ 2D screen space. - For example, consider a navigation route that ends at a particular city building. The
map database 130 is queried to find the geo-location and 3D shape/volume of the building, an image near the building, and an image depthmap. To indicate which building is important, the 3D depth values determine which 2D pixels overlap with the building volume. Since the process uses real depth values, objects in front of the building (e.g., a tree) would not have depth values that overlap thebuilding 3D volume. Therefore, only pixels that represent the building are altered and tree pixels are left unaltered. - Now that pixels have been classified to distinguish between pixels within the building volume and those outside of the building volume, the image is reimaged to recolor either group of pixels using one of many effects. For example, the building classified pixels may be brightened and the non-building classified pixels may be blurred. Alternatively, saturation may be added to the building pixels and removed color from the non-building pixels. Dark edge lines may be drawn around the building by darkening pixels which have two neighbors that are classified as both a building pixel and a non-building pixel.
- The image may also be reimaged to illustrate map queries. For example, a map application may be used to lookup a street address, business name, resident name, city name, or other point of interest. The map query identifies 2D or 3D point location. The distance of each other pixel, as measured using the 3D depth position, to the point of interest is used to emphasize only the image pixels that are within a specified 3D radius of the point of interest. In addition, for a 2D point of interest, the vertical component may be ignored so as to emphasize any pixel that is within a cylinder above the 2D point of interest.
- The computing resources necessary for emphasizing objects in a panoramic image may be divided by the
server 120 and thenavigation device 100. In some embodiments, theserver 120 performs a majority of the processing (“server-based embodiments”). Thedatabase 130 stores a depthmap generated from an optical distancing system, and theserver 120 receives the depthmap from thedatabase 130. Theserver 120 overlays the depthmap on a panoramic image according to the position data. Theserver 120 receives data indicative of an object in the panoramic image on thenavigation device 100. Theserver 120 compares depth data of the first point to depth data of surrounding points and identifies an area on the panoramic image corresponding to the surrounding points having depth data within a predetermined range of the depth data of the first point. Theserver 120 transmits the panoramic image including a graphical effect that indicates at least one of the surrounding points to thenavigation device 100. - In other embodiments, the
navigation device 100 performs a majority of the processing (“user device-based embodiments”). Thenavigation device 100 receives a depthmap from theserver 120. Alternatively, thenavigation device 100 may receive the combined depthmap and panoramic image from theserver 120. The image or panoramic image may be, for example, pixilated raster data in a bitmap, JPEG, GIF, and/or other image format. The panoramic image is displayed to a user. The user may have selected an object on the panoramic image or thenavigation device 100 may have selected the object according to a route or a point of interest. Thenavigation device 100 compares depth data of the first point to depth data of surrounding points and identifies an area on the panoramic image corresponding to the surrounding points having depth data within a predetermined range of the depth data of the first point. Thenavigation device 100 alters the panoramic image by adding a graphical effect is added that indicates at least one of the surrounding points. - In a hybrid embodiment, the processing is divided between the
server 120 and thenavigation device 100. In one example, the server overlays the depthmap on the panoramic image and sends the combined depthmap and panoramic image to thenavigation device 100, and thenavigation device 100 adds the graphical effect. The overlaid depthmap and panoramic image may be pre-aligned and stored in thedatabase 130. - For navigation related applications, the
navigation device 100 may generate a location according to the geographic location of thenavigation device 100. The location may be generated using positioning circuitry including one or more of a global navigation satellite system based on a satellite signal, a triangulation system that utilizes one or more terrestrial communication signals, a inertial position system based on sensors such as gyroscopes or accelerometers, and/or a or dead reckoning system based on a previously known position. The positioning circuitry may also determine an orientation using any of the above systems and/or a magnetic sensor such as a compass. The orientation and the location allow the appropriate depthmap and panoramic image to be selected based on the perspective of the user at the current geographic location of thenavigation device 100. - The
network 110 may include any combination of a cellular network, the Internet, or a local computer network. For example, thenavigation device 100 may communicate with thenetwork 110 wirelessly though protocols known as Wi-Fi, the protocols defined by the IEEE 802.11 standards, the protocols defined by the Bluetooth standards, or other protocols. Alternatively or in addition, thenavigation device 100 may communicate with thenetwork 110 wirelessly as a cellular network such as a Global System for Mobile Communication (GSM) or other third generation (3G) or fourth generation (4G) networks. -
FIG. 2A illustrates animage 200 for use with a navigation system or a map system. Theimage 200 may depict outdoor environments, as shown, indoor environments (e.g., shopping mall, convention center, etc.) or hybrid outdoor and indoor environments (e.g., fairgrounds, amusement park, etc.). Alternatively, theimage 200 may be displayed in an augmented reality system. In the augmented reality system, theimage 200 is received in real time by a camera on thenavigation device 100. - The
image 200 is reimaged bysystem 150 to emphasize a portion of theimage 200. Theimage 200 may be a panoramic image, a street level view image, or a satellite view image. Theimage 200 includes objects at a plurality of depths. In the example shown inFIG. 2A , four depths are shown. Asign 207 is nearest to the viewer, which is followed by awall 205, and atree 203. Ahouse 201 is the farthest from the view. Ahypothetical user selection 209 is in an area in theimage 200 that is very near thehouse 201, thetree 203, and thesign 207 in theimage 200 even though in the physical world, these objects are relatively far apart. -
FIG. 2B illustrates a predetermined camera angle that theimage 200 was collected by acamera 250. The predetermined camera angle may have one, two, or three components (e.g., roll, pitch, yaw) measured from any of three coordinate axes. The predetermined camera angle may be used to align the depth data to the panoramic image. Theimage 200 may be a planar image additionally defined by a field of view. Alternatively, thecamera 250 may be an unknown position and/or orientation at the time theimage 200 was collected. Feature matching may match features in theimage 200 to known images in thedatabase 130 in order to determine the position or orientation of the image. Thecamera 250 may be included in theuser device 100. -
FIG. 3A illustrates the generation of optical distance data using theoptical distancing system 350. Theoptical distancing system 350 may be a LIDAR device, a stereoscopic camera, a time of flight camera, a structured light device, or another device. In the example of a LIDAR device, one or more (e.g., 1, 2, 10) lasers rotate about an axis. Theoptical distancing system 350 may be coupled with an inertial measurement unit (IMU) and/or an inertial navigation system (INS) in order to provide a geographic reference to the optical distance data.FIG. 3A illustrates one such plane includinglaser paths 301 and depth points 303. Theoptical distance system 350 measures the distance from theoptical distancing system 350 to the various objects. In another example, the structured light device emits light onto objects and a camera captures images the structured light to determine distance based on the shape of the light in the captured images. The structured light may be infrared light or another light spectrum not visible to humans -
FIG. 3B illustratesdepthmap data 303 as generated by theoptical distancing system 350. Thedepthmap data 303 may be arranged approximately linearly because the laser scans in lines.FIG. 3B is not to scale and typically many more points are used than shown and may be in any arrangement. Objects that are closer to theoptical distancing system 350 may be associated with a higher density of points and objects farther away from theoptical distancing system 350 are associated with a lower density of points.FIG. 3C illustrates depthmap data represented inscan lines 401 of theoptical distancing system 350. Other patterns may be used. -
FIG. 4 illustrates the depthmap data aligned withcamera 250 at the angle ofFIG. 2B . It should be noted that the optical distancing data may be received from any angle. However, at other angles, the optical distancing data may be manipulated to form depthmap data. The collected optical distance data may be true 3D data points. Because of the 3D nature of the points (as opposed to 2D image pixels), the data may be viewed from any angle. Therefore, a ‘virtual’ camera viewpoint of the 3D data may be simulated. In contrast, the camera capture of theimage 200 is fixed based on 2D image data and a camera view direction. For example, the virtual camera viewpoint is simulated by creating a view of the 3D data from the same viewpoint as that of the 2D color image by projecting the 3D points onto the camera viewplane. The distance of the 3D point from the camera is saved to generate a depthmap image such that each RGB color pixel of the original image has a direct mapping to a value in the depthmap that represents the distance of the pixel from the camera. -
FIGS. 5A and 5B illustrate twoexample depthmap grids 500. Eachcell 501 in thedepthmap grid 500 spans multiple pixel locations. The depth data is displayed as a grid for demonstration purposes. However, each pixel may specify a unique depth value determined from the 3D sensor data points. Likewise, depth figures may be demonstrated by discrete depth values while in reality, these values may include a nearly continuous range of depth distances. - Generally, a depthmap is described as an image containing pixels that store depth values rather than RGB values. However, there are alternative mechanisms for storing and retrieving depth information. One mechanism for associating an image pixel with a depth value includes storing raw 3D points. An alternative mechanism includes pre-generating depth values for each image pixel and storing the depthmap as an image alongside the original color image. Another alternative mechanism includes converting the depthmap image pixels to a vector, polygon, or mesh based mapping to save space. The image may be converted to one of these vector/polygon/mesh representation by detecting approximate edges and gradients in the depth image. This storage model may be more efficient for storage and delivery due to smaller memory requirements.
FIGS. 5A and 5B illustrate a mesh type depth map storage model. -
FIGS. 6A and 6B illustrate the calculation of image mapped depth values from a pre-generated depthmap.FIG. 6A illustrates a first cross section A-A of theimage 200, which corresponds to a second cross section B-B in thedepthmap grid 500. The C-C cross section plane or top downview 600 inFIG. 6A shows the depth values are defined to correspond to thesign 207,house 201, andtree 203. The depth map cross section C-C indicates that thehouse 201 is the furthest from the camera while thesign 207 is the closest.FIG. 6B illustrates a first cross section A-A of theimage 200, which corresponds to a second cross section B-B in thedepthmap grid 500. The C-C cross section plane or top downview 600 inFIG. 6B shows the depth values are defined to correspond to thesign 207 and thewall 205. -
FIG. 7 illustrates the selection of points for reimaging. Reimaging includes emphasizing, de-emphasizing, isolating, highlighting, accentuating, or focusing a region or object in the image. Each pixel, for example pixel P1 in theimage 200 aligns with a particular depth value D1 in thedepthmap 500. The top-down view 600 illustrates the magnitude of each of the depth values that are associated with thesign 207,wall 205,house 201, andtree 203. Specifically, the depth value D1 of thesign 207 is shown measured along the depth axis d. - A
pixel color function 700 calculates the modified pixel color for reimaging. Thepixel color function 700 may be linear, quadratic, cubic, logarithmic, or another type of function of depth. As shown inFIG. 7 , thepixel color function 700 may be a continuous function or a series of piecewise linear portions to approximate the function. Depth values are input to thepixel color function 700, and a re-target weight e is output from the pixel color function 700 (e.g., f(D1)=e1). The depth values input to thepixel color function 700 may be a reference depth value of the pixel P1 (e.g., depth value D1 as measured along the depth axis d) or a calculated depth value measured from a selected pixel. The re-target weight e may be multiplied by the original pixel value color value C1 to arrive at the modified pixel color value Cmod. - Depending on the
pixel color function 700, many graphical effects are possible. Thepixel color function 700 may produce an outline of a physical object. For example, thepixel color function 700 may define e as at or near 1.0 for all values near the depth of D1 and significantly increase or decrease e for a group of surrounding pixel on the edge of a change in depth values. Pixel values at or near the selected pixel depth treated differently than pixels where the depth changes significantly. Essentially, e is calculated as a flood fill using depth values and depth gradients. The flood fill continues from the user selected depth until a large change in depth is detected. For edge highlighting, the edge values of the resulting flood fill may be assigned low e values (dark), while all other pixels are defined with e=1 (no color change). Thus the outline of the selected object is displayed. - The
pixel color function 700 may use the re-target weight e as a blur factor rather than a darkening factor. The result is that pixels with specific depths will have a specific amount of blur. For example, a function that defines that e (in this case a blur radius scale factor) as decreasing with depth will produce an image that appears to get more blurry as objects get further from the camera. In other words, rather than an object in the panoramic image, the location that determines the graphical effect in the panoramic image may be a perspective used to select the panoramic image. Thepixel color function 700 may invert portions of theimage 200. Thepixel color function 700 may fade portions of the panoramic image by setting the re-target weight e to a fraction for certain depth values. Further, thepixel color function 700 may hide portions of the panoramic image by setting the re-target weight e to 0 for a specific range of depth values, which results in cutting pixels from portions of theimage 200 or removing portions of theimage 200. - Another application is to retarget the entire image based on the depth of each pixel depth from the camera (rather than a user-defined location or a map/navigation application defined location) to simulate a camera depth of field blur or simulate a night view where light falls off rapidly as objects recede from the camera. The re-target weight e is defined for each color pixel based on the associated depth value. The user may specify a focal distance such that pixels at this distance are not altered, but pixels closer or further blurred more as the pixels are further from the focal distance.
-
FIG. 8A illustrates an embodiment where the graphical effect is blurring. The top-down view 600 again illustrates thesign 207,house 201,tree 203, andwall 205. In this example shown inFIG. 8A , a point on or near thetree 203 has been selected. The point on or near thetree 203 corresponds to a depth value D2. A depth function may define a predetermined range of depth values that will be blurred and a blur amount. The predetermined range of depth values may be centered around D2 or off center. For example, dfar may be defined as D2 plus a predetermined amount and dnear may be defined as D2 minus the predetermined amount. In the example shown inFIG. 8A , thetree 203 is blurred at 100% and all other objects in theimage 200 are not blurred or reimaged. -
FIG. 8B illustrates an embodiment where the graphical effect is highlighting. The top-down view 600 again illustrates thesign 207,house 201,tree 203, andwall 205. Each object is associated with a footprint region in thedatabase 130. In this example shown inFIG. 8B , a pre-defined region from a map/building database is used to determine which 3D region or footprint region is of interest. For example, a user may input a building name or address that is associated with the 3D region in the map/building database. A subset of pixels surrounding the selected point that fall within the same 3D region is modified to attract user attention to thehouse 201. Likewise, the 3D region may be defined as the region above some 2D footprint (regardless of height). These calculations are possible because the depth of each pixel is included in the depthmap, and the 3D position of each pixel may be determined based on the depth, image position, and direction in 3D space. For example, as indicated inFIG. 8B , thehouse 201 may be shown brighter than other portions of theimage 200 because the 2D image pixels (combined with depth) lie within the defined 3D building volume. In another example, the 2D footprint of an object may be reimaged to highlight a characteristic of the object. For example, the 2D area footprint of a park may be tinted green or the 2D area of a body of water may be tinted blue. -
FIG. 9A illustrates another embodiment where the graphical effect is highlighting. The top-down view 600 again illustrates thesign 207,house 201,tree 203, andwall 205. In this example shown inFIG. 9A , a point on or near thetree 203 has been selected. The point on or near thetree 203 corresponds to a depth value D3. A depth function may define a predetermined range of depth values that will be brightened and a brightness amount. While 100% brightness is shown, any amount could be used. Further, the brightness could vary for depths moving away from D3. -
FIG. 9B illustrates a variation of the embodiment ofFIG. 9A . InFIG. 9B , the depth function pixels fade out from the predetermined range between dfar and dnear. This is one example of an intensity fall off model to gradually change the fade amount. The pixels fade out as the distance in the depth direction from the selected pixel increases.FIG. 9C illustrates another example of an intensity fall offmodel 901 where pixels fade out as the depth increases. -
FIG. 10 illustrates an embodiment in which the graphical effect is a simulated spotlight. The depth function for the simulated spotlight utilizes both the depth data and the 2D position inimage 200 to determine whether each pixel falls within or outside of the light cone 1001. Those pixels outside the light cone 1001 are reduced in intensity or brightness. Within the light cone 1001, the intensity or brightness may be faded based on a distance from the center of the cone 1001. In other words, an intensity of each of point surrounding a selected point is a function of a distance to the selected portion. A user selection may specify the spot light direction such that the spotlight follows the user's mouse and simulates 3D interaction on a 2D image. - The depthmap combined with the
image 200 builds a true 3D model. Accordingly, the spot light may appear behind some objects and in front of others. For the example inFIG. 10 , the depth information reveals that the house pixels are lit with the spotlight, while the tree is not. From the 2D view, the cone of the spot light intersects the tree. However, the depth information indicates that the spot light is actually behind the tree. This would not be possible with the depthmap information. - Further, a day time photo can be simulated as a night view with a localized light source. A light source location is defined. The depth of each pixel in the
panoramic image 200 is used to determine a 3D location and distance from the light source location. Pixels of thepanoramic image 200 are darkened based on some function of distance from the light source. The function may be based on square of distance. -
FIG. 11 illustrates a detailed view of theserver 120 ofFIG. 1 . Theserver 120 includes aserver controller 500, amemory 501, and acommunication interface 505. Thedatabase 130 may be external or internal to theserver 120. In the server-based embodiments, theserver 120 is apparatus for emphasizing objects in a panoramic image.FIG. 12 illustrates a detailed view ofnavigation device 100 ofFIG. 1 . In the user device-based embodiments, thenavigation device 100 is an apparatus for emphasizing objects in a panoramic image. Thenavigation device 100 includes anavigation controller 600, amemory 601, acommunication interface 605, andposition circuitry 607. Thenavigation device 100 may also include auser input device 603, amedia input device 609, and adisplay 611. - As discussed above, the apparatus for emphasizing objects in the panoramic image may operate with or without position data for the current location of the
navigation device 100. When the position data is used in the server-based embodiments, theposition circuitry 607 determines a current geographic position of thenavigation device 100 and thecommunication interface 605 sends the current geographic position to theserver 120. When the position data is used in the user-device based embodiments, theposition circuitry 607 determines location data including the position and/or the orientation of thenavigation device 100. The location data may be generated using one or more of a global navigation satellite system based on a satellite signal (such as Global Positioning System (GPS), the Russian GLONASS or European Galileo), a triangulation system that utilizes one or more terrestrial communication signals, a inertial position system based on relative position sensors such as gyroscopes, accelerometers, and altimeters, and/or a or dead reckoning system based on a previously known position. The orientation may be determined using any of the above systems and/or a magnetic sensor such as a compass or a three dimensional magnetic sensor array. Magnetic sensors determine the direction and or strength of a magnetic field and can be used to determine heading or orientation. Inertial sensors such as accelerometers and gyroscopes measure acceleration, which can be used to calculate position, orientation, and velocity (direction and speed of movement) of thenavigation device 100. The location and/or orientation data may be used to select thedepthmap 500 and thepanoramic image 200 from thedatabase 130. - In the server-based embodiments the
memory 501 stores the depthmap generated from the optical distancing system and thepanoramic image 200. Theserver controller 500 is a processor configured to receive a user selected location on thepanoramic image 200 and overlay the depthmap on the panoramic image according to position data to identify a focus area on the panoramic image corresponding to the user selected location. The focus area is defined according to a function of the depth data of the user selected location and the depth data of points surrounding the user selected location. The function may be any of the color pixel functions 700 or depth functions discussed above or any shown inFIGS. 7-10 . Based on the output of the function various reimaging techniques may be used to illustrate the focus area to the user. Such graphical effects include, but are not limited to an outline of a physical object, a highlighting a 3D region such as a building from a map database, a blurring of the focus area, a color inversion of the focus area, a fading of the focus area, a flood fill of the focus area, cutting pixels from portions of the panoramic image, or removes portions of the panoramic image. Similar effects may be applied to the areas outside of the focus area. - In the user device-based embodiments, the
memory 601 stores the depthmap generated from theoptical distancing system 350 and thepanoramic image 200, which are downloaded from theserver 120. Theuser input device 603 receives data indicative of a user selected location on thepanoramic image 200. Thenavigation controller 600 is a processor configured to receive the user selected location and overlay the depthmap on thepanoramic image 200 according to position data to identify a focus area on the panoramic image corresponding to the user selected location. The focus area is defined by examining surrounding points according to a function. The analysis may occur pixel by pixel. The function receives inputs of the depth of the user selected location and the depth of surrounding pixels. - The surrounding pixels may be defined several ways. For example, the surrounding pixels may be the pixels that border the user selected location, the surrounding pixels may be within a predetermined pixel distance (e.g., 50, 100, or 1000 pixels away) from the user selected location, or the surrounding pixels may be a share of the image 200 (e.g., the closest 10%, 25% or 50%). Regardless of the definition, each of the surrounding pixels is examined by the
server controller 500 ornavigation controller 600. While the analysis may take many forms, two types of analysis are discussed in detail below. - In a first type of analysis, the
server controller 500 ornavigation controller 600 compares the depth of each of the surrounding pixels to the depth of the user selected location to calculate a gradient quantity. If the difference between the depth of a surrounding pixel and the depth of the user selected location (gradient quantity) is below a predetermined threshold, the pixel is declared to be on the same object in theimage 200 as the user selected location. In other words, the surrounding pixel is in the focus area and may be subject to the graphical effect. The first type of analysis is repeated for subsequent surrounding pixels to identify points for which the predetermined value exceeds the gradient quantity. - In a second type of analysis, the
server controller 500 ornavigation controller 600 divides the depth data of surrounding pixels into a first level of surrounding points and a second level of surrounding points. The first level of surrounding points is closer to the user selected location in theimage 200 than the second level of surrounding points. The specific division between the first level and the second level may vary. For example, the first level of surrounding points may include a certain number of pixels (e.g., 100, 1000) closest to the user selected points, and the second level of surrounding points may include the remaining surrounding points. - The
server controller 500 ornavigation controller 600 calculates a first gradient based on the rate of change of the depths of first level of surrounding points and a second gradient based on the rate of change of the depths of the second level of surrounding points. - The first gradient is compared to the second gradient to determine the size of the focus area. For example, the focus area may include the first level of surrounding points or both the first level of surrounding points and the second level of surrounding points. A gradient threshold may be stored in
memory 501 ormemory 601. The gradient threshold defines the rate of change of depth values necessary to distinguish between object in theimage 200. When the difference between the second gradient and the first gradient is more than the gradient threshold only the first level of surrounding pixels receive the graphical effect. When the difference between the second gradient and the first gradient is less than the gradient threshold, both the first level of surrounding pixels and the second level of surrounding pixels receive the graphical effect. - The
user input device 603 includes one or more buttons, keypad, keyboard, mouse, stylist pen, trackball, rocker switch, touch pad, voice recognition circuit, or other device or component for inputting data to thenavigation device 100. Theuser input device 603 and thedisplay 611 may be combined as a touch screen, which may be capacitive or resistive. Thedisplay 611 may be a liquid crystal display (LCD) panel, light emitting diode (LED) screen, thin film transistor screen, or another type of display. - The
navigation controller 600 orserver controller 500 may include a general processor, digital signal processor, an application specific integrated circuit (ASIC), field programmable gate array, analog circuit, digital circuit, combinations thereof, or other now known or later developed processor. Thenavigation controller 600 orserver controller 500 may be a single device or combinations of devices, such as associated with a network, distributed processing, or cloud computing. - The
memories memory memory navigation device 100, such as a secure digital (SD) memory card. - The communication interfaces 505, 605 may include any operable connection. An operable connection may be one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a physical interface, an electrical interface, and/or a data interface. The communication interfaces 505, 605 provides for wireless and/or wired communications in any now known or later developed format.
- In an alternative embodiment, the
navigation device 100 may omit theposition circuitry 607 or use of theposition circuitry 607. In this alternative embodiment, thenavigation device 100 may be a personal computer, which encompasses laptops and other mobile processing platforms. -
FIG. 13 illustrates a flowchart for an algorithm for reimaging a panoramic image and emphasizing objects in the panoramic image. As discussed above, the algorithm may be performed with or without using the user's current location and may be performed on thenavigation device 100, theserver 120, or a combination of the two. A processor (e.g.,server controller 500 or navigation controller 600) generally performs the steps in the algorithm. Fewer, more, or different steps or acts may be provided, and a combination of steps may be provided. Also, the steps or acts may be performed in the order as shown or in a different order. The method is implemented by the system and/or devices described herein or by different devices or systems. - At S101, the processor receives a depthmap generated from the
optical distancing system 350. The depthmap was generated from collected optical distance data in the physical world and describes distances to objects in the physical world from theoptical distancing system 350 and the time of collection. The distances may be measured by time of flight (laser, infrared), structured light, or a stereoscopic camera. - At S103, the processor overlays the depth map on a panoramic image. The panoramic image may be part of a map database including one or more of street level images or high-altitude level maps. The panoramic image may be several photographs stitched together or may include a single photograph. If the panoramic image and the optical distance data were taken from similar locations and orientation, the overlay may involve simply combining the depth values of the depth map to the two-component data of the panoramic image. If the panoramic image and the optical distance may were collected from difference locations and/or orientations geometric calculations are used to align the depthmap and the panoramic image.
- At S105, the processor receives data indicative of a location on the panoramic image. The data may be data received from the
user input device 603, which may be a touch screen or a display. The data may be received from a map application or a navigation application. Alternatively, the data may be based on a user input. The location corresponds to an object in the panoramic image. The object may be a destination, such as a road, a name on a sign, or a building or another point of interest. The selected location is associated with pixels in the panoramic image and accordingly, depth data in the depthmap overlaid on the panoramic image. - At S107, the processor compares the depth data that is associated with the location to depth data of surrounding points to define a focus area. In addition, the comparison may utilize both the depth data and the position data. For example, the comparison may define the focus area to be all points within a predetermined distance measured in all directions from the location. The focus area is defined by a function. The function may be a discrete comparison, which means that the focus area includes points surrounding the user selected location that have depth data within a predetermined distance from the depth data of the user selected location. Alternatively, the function may be dependent on the rate of change of the depth data of the surrounding pixels. In other words, the function examines a rate of change of the depth data in a direction moving away from the user selected location to define the focus area where the rate of change is less than a predetermined rate.
- At S109, the processor alters the panoramic image with a graphical effect that indicates at least one of the surrounding points. The graphical effect may adjust pixel values in the focus area to change one or more of the brightness, intensity, or color of the focus area. The graphical effect may be an outline around the focus area or a blur effect. The graphical effect may be flood fill, which converts all of the pixels in the focus area to the same color (e.g., all black, all white). The graphical effect may remove pixels in the focus area. Alternatively, the graphical effect may change the appearance of pixels in the panoramic image outside of the focus area and still indicate at least one of the surrounding points. For example, a map navigation route path can be used to highlight a path and nearby objects in the camera color image. Pixels may be classified based on depth. Pixels that are above the path or within a specified radius of the path may be emphasized.
- The embodiments described above may be combined with systems and methods for illustrating routes in panoramic or other images as described in copending application “THREE DIMENSIONAL ROUTING” by James D. Lynch, filed Oct. 21, 2011 (Attorney Docket No. N0407US), which is incorporated by reference in its entirety. The embodiments described above may also be combined with the systems and method for illustrating depth in a panoramic image or other images as described in copending application “DEPTH CURSOR AND DEPTH MEASUREMENT IN IMAGES” By James D. Lynch, filed Oct. 21, 2011 (Attorney Docket No. N0406US), which in incorporated by reference in its entirety.
- The embodiments described above may be implemented using computer executable instructions stored in the
memory 501 and/or thememory 601, which are non-transitory. The processors may execute computer executable instructions. The computer executable instructions may be written in any computer language, such as C++, C#, Java, Pascal, Visual Basic, Perl, HyperText Markup Language (HTML), JavaScript, assembly language, extensible markup language (XML), graphics languages such as WebGL and OpenGL, and shading languages such as OpenGL Shading Language (GLSL) and High Level Shading Language (HLSL), and any combination thereof. - The computer executable instructions may be logic encoded in one or more tangible media or one or more non-transitory tangible media for execution by the processors. Logic encoded in one or more tangible media for execution may be defined as instructions that are executable by the processors and that are provided on the computer-readable storage media, memories, or a combination thereof. Instructions for instructing a network device may be stored on any logic. As used herein, “logic”, includes but is not limited to hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another logic, method, and/or system. Logic may include, for example, a software controlled microprocessor, an ASIC, an analog circuit, a digital circuit, a programmed logic device, and a memory device containing instructions.
- The computer readable instructions may be stored on any non-transitory computer readable medium. A non-transitory computer readable medium may include, but are not limited to, a floppy disk, a hard disk, an ASIC, a compact disk, other optical medium, a random access memory (RAM), a read only memory (ROM), a memory chip or card, a memory stick, and other media from which a computer, a processor or other electronic device can read.
- As used herein, the phrases “in communication” and “couple” are defined to mean directly connected to or indirectly connected through one or more intermediate components. Such intermediate components may include both hardware and software based components.
- Various embodiments described herein can be used alone or in combination with one another. The foregoing detailed description has described only a few of the many possible implementations of the present invention. It is intended that the foregoing detailed description be regarded as illustrative rather than limiting and that it is understood that the following claims including all equivalents are intended to define the scope of the invention.
Claims (20)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/278,499 US8553942B2 (en) | 2011-10-21 | 2011-10-21 | Reimaging based on depthmap information |
PCT/EP2012/070526 WO2013057127A1 (en) | 2011-10-21 | 2012-10-17 | Reimaging based on depthmap information |
EP12773321.0A EP2769181B1 (en) | 2011-10-21 | 2012-10-17 | Reimaging based on depthmap information |
US14/025,366 US9641755B2 (en) | 2011-10-21 | 2013-09-12 | Reimaging based on depthmap information |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/278,499 US8553942B2 (en) | 2011-10-21 | 2011-10-21 | Reimaging based on depthmap information |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/025,366 Continuation US9641755B2 (en) | 2011-10-21 | 2013-09-12 | Reimaging based on depthmap information |
Publications (2)
Publication Number | Publication Date |
---|---|
US20130101175A1 true US20130101175A1 (en) | 2013-04-25 |
US8553942B2 US8553942B2 (en) | 2013-10-08 |
Family
ID=47040729
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/278,499 Active 2031-12-20 US8553942B2 (en) | 2011-10-21 | 2011-10-21 | Reimaging based on depthmap information |
US14/025,366 Active 2032-08-28 US9641755B2 (en) | 2011-10-21 | 2013-09-12 | Reimaging based on depthmap information |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/025,366 Active 2032-08-28 US9641755B2 (en) | 2011-10-21 | 2013-09-12 | Reimaging based on depthmap information |
Country Status (3)
Country | Link |
---|---|
US (2) | US8553942B2 (en) |
EP (1) | EP2769181B1 (en) |
WO (1) | WO2013057127A1 (en) |
Cited By (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130275920A1 (en) * | 2012-01-06 | 2013-10-17 | Tourwrist, Inc. | Systems and methods for re-orientation of panoramic images in an immersive viewing environment |
US20130342568A1 (en) * | 2012-06-20 | 2013-12-26 | Tony Ambrus | Low light scene augmentation |
US20140157349A1 (en) * | 2012-11-30 | 2014-06-05 | Scott H. Robinson | Verified Sensor Data Processing |
US20150009295A1 (en) * | 2013-07-03 | 2015-01-08 | Electronics And Telecommunications Research Institute | Three-dimensional image acquisition apparatus and image processing method using the same |
US20150249794A1 (en) * | 2010-10-05 | 2015-09-03 | Sony Computer Entertainment Inc. | Apparatus and method for displaying images |
US9373190B2 (en) * | 2014-07-09 | 2016-06-21 | Google Inc. | High-quality stereo reconstruction featuring depth map alignment and outlier identification |
CN106101743A (en) * | 2016-08-23 | 2016-11-09 | 广东欧珀移动通信有限公司 | Panoramic video recognition methods and device |
CN106908043A (en) * | 2017-02-23 | 2017-06-30 | 周良辰 | The three-dimensional amount measuring method of geographic position of target object and height based on Streetscape picture |
WO2017173578A1 (en) * | 2016-04-05 | 2017-10-12 | 华为技术有限公司 | Image enhancement method and device |
CN107633554A (en) * | 2017-03-16 | 2018-01-26 | 柳州桂通科技股份有限公司 | Mobile surface targets and its methods of exhibiting with the three-dimensional scenic of present position |
CN107850445A (en) * | 2015-08-03 | 2018-03-27 | 通腾全球信息公司 | Method and system for generating and using locating reference datum |
US20190069117A1 (en) * | 2017-08-23 | 2019-02-28 | Harman International Industries, Incorporated | System and method for headphones for monitoring an environment outside of a user's field of view |
CN109509255A (en) * | 2018-07-26 | 2019-03-22 | 京东方科技集团股份有限公司 | A kind of labeling map structuring and space map updating method and device |
US10289911B2 (en) * | 2014-06-08 | 2019-05-14 | Uber Technologies, Inc. | Entrance detection from street-level imagery |
CN109785261A (en) * | 2019-01-11 | 2019-05-21 | 辽宁工程技术大学 | A kind of airborne LIDAR three-dimensional filtering method based on gray scale volume element model |
US20190206029A1 (en) * | 2017-12-28 | 2019-07-04 | Panasonic Intellectual Property Management Co., Ltd. | Image processing device, display device, navigation system, and image processing method |
US10453114B2 (en) | 2013-06-23 | 2019-10-22 | Intel Corporation | Selective sharing of user information based on contextual relationship information, such as to crowd-source gifts of interest to a recipient |
US10769840B2 (en) | 2018-02-27 | 2020-09-08 | Nvidia Corporation | Analysis of point cloud data using polar depth maps and planarization techniques |
US10776992B2 (en) * | 2017-07-05 | 2020-09-15 | Qualcomm Incorporated | Asynchronous time warp with depth data |
CN112241930A (en) * | 2019-07-18 | 2021-01-19 | 杭州海康威视数字技术股份有限公司 | Vehicle-mounted all-round looking method, device and system and vehicle |
CN112313714A (en) * | 2018-06-22 | 2021-02-02 | 三星电子株式会社 | Method for displaying content and electronic device |
CN113014754A (en) * | 2017-02-23 | 2021-06-22 | 钰立微电子股份有限公司 | Image device for generating panoramic depth image and related image device |
US20210350629A1 (en) * | 2012-09-21 | 2021-11-11 | Navvis Gmbh | Visual localisation |
US20220036604A1 (en) * | 2018-09-28 | 2022-02-03 | Align Technology, Inc. | Generation of images with tooth color determined using depth information |
US11961202B1 (en) * | 2023-02-23 | 2024-04-16 | Illuscio, Inc. | Systems and methods for editing three-dimensional image data with realistic camera and lens effects |
Families Citing this family (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5254297B2 (en) * | 2010-09-27 | 2013-08-07 | 株式会社東芝 | Image processing device |
US9047688B2 (en) | 2011-10-21 | 2015-06-02 | Here Global B.V. | Depth cursor and depth measurement in images |
EP2669707B1 (en) * | 2012-05-29 | 2019-07-24 | Leica Geosystems AG | Method and hand-held distance measuring device for indirect distance measurement by means of image-based angle determination function |
US9275302B1 (en) * | 2012-08-24 | 2016-03-01 | Amazon Technologies, Inc. | Object detection and identification |
KR101370718B1 (en) * | 2012-10-26 | 2014-03-06 | 한국과학기술원 | Method and apparatus for 2d to 3d conversion using panorama image |
US9215448B2 (en) | 2013-01-31 | 2015-12-15 | Here Global B.V. | Stereo panoramic images |
US10013507B2 (en) | 2013-07-01 | 2018-07-03 | Here Global B.V. | Learning synthetic models for roof style classification using point clouds |
CN103941264B (en) * | 2014-03-26 | 2017-04-19 | 南京航空航天大学 | Positioning method using laser radar in indoor unknown environment |
USD780777S1 (en) | 2014-04-22 | 2017-03-07 | Google Inc. | Display screen with graphical user interface or portion thereof |
US9934222B2 (en) | 2014-04-22 | 2018-04-03 | Google Llc | Providing a thumbnail image that follows a main image |
USD781317S1 (en) | 2014-04-22 | 2017-03-14 | Google Inc. | Display screen with graphical user interface or portion thereof |
US9972121B2 (en) * | 2014-04-22 | 2018-05-15 | Google Llc | Selecting time-distributed panoramic images for display |
US9986154B2 (en) * | 2014-05-21 | 2018-05-29 | Here Global B.V. | Developing a panoramic image |
US10547825B2 (en) | 2014-09-22 | 2020-01-28 | Samsung Electronics Company, Ltd. | Transmission of three-dimensional video |
US11205305B2 (en) | 2014-09-22 | 2021-12-21 | Samsung Electronics Company, Ltd. | Presentation of three-dimensional video |
DE102014116882B4 (en) * | 2014-11-18 | 2018-05-24 | B. Strautmann & Söhne GmbH u. Co. KG | Method for taking feed from silos |
CN106791351B (en) * | 2015-11-24 | 2018-11-09 | 腾讯科技(深圳)有限公司 | Panoramic picture treating method and apparatus |
US20170213885A1 (en) * | 2016-01-21 | 2017-07-27 | Micron Technology, Inc. | Semiconductor structure and fabricating method thereof |
AU2017221222B2 (en) * | 2016-02-15 | 2022-04-21 | Pictometry International Corp. | Automated system and methodology for feature extraction |
CN107193820B (en) * | 2016-03-14 | 2020-11-06 | 腾讯科技(深圳)有限公司 | Position information acquisition method, device and equipment |
US10325390B2 (en) * | 2016-03-31 | 2019-06-18 | Qualcomm Incorporated | Geometric work scheduling with dynamic and probabilistic work trimming |
US10346950B2 (en) | 2016-10-05 | 2019-07-09 | Hidden Path Entertainment, Inc. | System and method of capturing and rendering a stereoscopic panorama using a depth buffer |
CN107886477B (en) * | 2017-09-20 | 2020-07-14 | 武汉环宇智行科技有限公司 | Fusion correction method for three-dimensional vision and low-beam laser radar in unmanned driving |
JP6977667B2 (en) * | 2018-06-01 | 2021-12-08 | 日本電信電話株式会社 | Object-likeness estimation device, method, and program |
TWI677413B (en) * | 2018-11-20 | 2019-11-21 | 財團法人工業技術研究院 | Calibration method and device for robotic arm system |
US11303877B2 (en) | 2019-08-13 | 2022-04-12 | Avigilon Corporation | Method and system for enhancing use of two-dimensional video analytics by using depth data |
US11195065B2 (en) | 2019-11-22 | 2021-12-07 | Samsung Electronics Co., Ltd. | System and method for joint image and lidar annotation and calibration |
CN111031278B (en) * | 2019-11-25 | 2021-02-05 | 广州恒龙信息技术有限公司 | Monitoring method and system based on structured light and TOF |
US11548156B2 (en) * | 2020-09-17 | 2023-01-10 | Techman Robot Inc. | Device and method for calibrating coordinate system of 3D camera and robotic arm |
CN112686197B (en) * | 2021-01-07 | 2022-08-19 | 腾讯科技(深圳)有限公司 | Data processing method and related device |
CN114780762B (en) * | 2022-06-23 | 2022-09-06 | 济南信通达电气科技有限公司 | Point cloud ranging automatic labeling method and system for night vision image of power transmission line |
Family Cites Families (68)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5070401A (en) | 1990-04-09 | 1991-12-03 | Welch Allyn, Inc. | Video measurement system with automatic calibration and distortion correction |
US5790086A (en) | 1995-01-04 | 1998-08-04 | Visualabs Inc. | 3-D imaging system |
US6552744B2 (en) * | 1997-09-26 | 2003-04-22 | Roxio, Inc. | Virtual reality camera |
US20020176619A1 (en) | 1998-06-29 | 2002-11-28 | Love Patrick B. | Systems and methods for analyzing two-dimensional images |
US6208934B1 (en) | 1999-01-19 | 2001-03-27 | Navigation Technologies Corp. | Method and system for providing walking instructions with route guidance in a navigation program |
US6417850B1 (en) | 1999-01-27 | 2002-07-09 | Compaq Information Technologies Group, L.P. | Depth painting for 3-D rendering applications |
US6507665B1 (en) * | 1999-08-25 | 2003-01-14 | Eastman Kodak Company | Method for creating environment map containing information extracted from stereo image pairs |
US6999620B1 (en) | 2001-12-10 | 2006-02-14 | Hewlett-Packard Development Company, L.P. | Segmenting video input using high-level feedback |
US7003136B1 (en) | 2002-04-26 | 2006-02-21 | Hewlett-Packard Development Company, L.P. | Plan-view projections of depth image data for object tracking |
JP4065507B2 (en) | 2002-07-31 | 2008-03-26 | キヤノン株式会社 | Information presentation apparatus and information processing method |
US7725258B2 (en) | 2002-09-20 | 2010-05-25 | M7 Visual Intelligence, L.P. | Vehicle based data collection and processing system and imaging sensor system and methods thereof |
US7567274B2 (en) * | 2002-12-09 | 2009-07-28 | Frank Edughom Ekpar | Method and apparatus for creating interactive virtual tours |
JP2005149409A (en) * | 2003-11-19 | 2005-06-09 | Canon Inc | Image reproduction method and apparatus |
JP2005321370A (en) | 2004-04-05 | 2005-11-17 | Sony Corp | Navigation system, data processing method and computer program |
US8751156B2 (en) * | 2004-06-30 | 2014-06-10 | HERE North America LLC | Method of operating a navigation system using images |
US20060002590A1 (en) | 2004-06-30 | 2006-01-05 | Borak Jason M | Method of collecting information for a geographic database for use with a navigation system |
US7460953B2 (en) | 2004-06-30 | 2008-12-02 | Navteq North America, Llc | Method of operating a navigation system using images |
US20060072848A1 (en) * | 2004-09-29 | 2006-04-06 | Razzano Michael R | Image charting system and method |
KR101114722B1 (en) | 2005-02-11 | 2012-02-29 | 삼성전자주식회사 | Apparatus and method of guiding rout based on step |
US7840032B2 (en) | 2005-10-04 | 2010-11-23 | Microsoft Corporation | Street-side maps and paths |
US20070237424A1 (en) | 2006-03-29 | 2007-10-11 | Matsushita Electric Industrial Co., Ltd. | Method and apparatus instrumenting a camera to: measure, model and insert target objects at their right size into pictures |
JP4935145B2 (en) | 2006-03-29 | 2012-05-23 | 株式会社デンソー | Car navigation system |
US20090027418A1 (en) | 2007-07-24 | 2009-01-29 | Maru Nimit H | Map-based interfaces for storing and locating information about geographical areas |
US7978191B2 (en) | 2007-09-24 | 2011-07-12 | Dolphin Imaging Systems, Llc | System and method for locating anatomies of interest in a 3D volume |
WO2009094064A1 (en) | 2008-01-22 | 2009-07-30 | Exxonmobil Upstream Research Company | Dynamic connectivity analysis |
US8502821B2 (en) | 2008-02-04 | 2013-08-06 | C Speed, Llc | System for three-dimensional rendering of electrical test and measurement signals |
KR100889469B1 (en) | 2008-04-23 | 2009-03-19 | 팅크웨어(주) | System and method for displaying three-dimensional map based on road information |
US9041915B2 (en) | 2008-05-09 | 2015-05-26 | Ball Aerospace & Technologies Corp. | Systems and methods of scene and action capture using imaging system incorporating 3D LIDAR |
US8738291B2 (en) | 2008-05-21 | 2014-05-27 | Navteq B.V. | Method and system for representing pedestrian crosswalks in a geographic database used by a navigation system |
US8224097B2 (en) * | 2008-06-12 | 2012-07-17 | Sri International | Building segmentation for densely built urban regions using aerial LIDAR data |
US8099237B2 (en) | 2008-07-25 | 2012-01-17 | Navteq North America, Llc | Open area maps |
US8417446B2 (en) | 2008-07-25 | 2013-04-09 | Navteq B.V. | Link-node maps based on open area maps |
CA2725800A1 (en) | 2008-07-31 | 2010-02-04 | Tele Atlas B.V. | Method of displaying navigation data in 3d |
WO2010024212A1 (en) | 2008-08-29 | 2010-03-04 | 三菱電機株式会社 | Bird's-eye image forming device, bird's-eye image forming method, and bird's-eye image forming program |
US8745536B1 (en) | 2008-11-25 | 2014-06-03 | Perceptive Pixel Inc. | Volumetric data exploration using multi-point input controls |
US20100204964A1 (en) * | 2009-02-09 | 2010-08-12 | Utah State University | Lidar-assisted multi-image matching for 3-d model and sensor pose refinement |
US8473681B2 (en) | 2009-02-17 | 2013-06-25 | Rambus Inc. | Atomic-operation coalescing technique in multi-chip systems |
US8164617B2 (en) * | 2009-03-25 | 2012-04-24 | Cisco Technology, Inc. | Combining views of a plurality of cameras for a video conferencing endpoint with a display wall |
US8473187B2 (en) | 2009-06-01 | 2013-06-25 | Robert Bosch Gmbh | Method and apparatus for combining three-dimensional position and two-dimensional intensity mapping for localization |
WO2010140613A1 (en) * | 2009-06-03 | 2010-12-09 | 学校法人中部大学 | Object detection device |
EP2309463A3 (en) | 2009-10-07 | 2011-07-27 | Thomson Licensing | Method of displaying a 3D video with insertion of a graphic item and terminal for implementing the method |
US20110099507A1 (en) | 2009-10-28 | 2011-04-28 | Google Inc. | Displaying a collection of interactive elements that trigger actions directed to an item |
US20110109617A1 (en) | 2009-11-12 | 2011-05-12 | Microsoft Corporation | Visualizing Depth |
US9766089B2 (en) | 2009-12-14 | 2017-09-19 | Nokia Technologies Oy | Method and apparatus for correlating and navigating between a live image and a prerecorded panoramic image |
JP5618569B2 (en) | 2010-02-25 | 2014-11-05 | キヤノン株式会社 | Position and orientation estimation apparatus and method |
US8660316B2 (en) | 2010-03-04 | 2014-02-25 | Navteq B.V. | Navigating on images |
US9217648B2 (en) | 2010-03-30 | 2015-12-22 | Here Global B.V. | Method of operating a navigation system to provide a pedestrian route |
KR101717787B1 (en) | 2010-04-29 | 2017-03-17 | 엘지전자 주식회사 | Display device and method for outputting of audio signal |
JP5136671B2 (en) | 2010-05-13 | 2013-02-06 | 株式会社デンソー | Map display device |
US9582166B2 (en) | 2010-05-16 | 2017-02-28 | Nokia Technologies Oy | Method and apparatus for rendering user interface for location-based service having main view portion and preview portion |
US8625897B2 (en) * | 2010-05-28 | 2014-01-07 | Microsoft Corporation | Foreground and background image segmentation |
EP2606472A2 (en) | 2010-06-11 | 2013-06-26 | Estill, James A. | System and method for manipulating data having spatial coordinates |
US8762041B2 (en) | 2010-06-21 | 2014-06-24 | Blackberry Limited | Method, device and system for presenting navigational information |
US8649592B2 (en) | 2010-08-30 | 2014-02-11 | University Of Illinois At Urbana-Champaign | System for background subtraction with 3D camera |
KR20120049636A (en) | 2010-11-09 | 2012-05-17 | 삼성전자주식회사 | Image processing apparatus and method |
US20120133639A1 (en) | 2010-11-30 | 2012-05-31 | Microsoft Corporation | Strip panorama |
US9013469B2 (en) | 2011-03-04 | 2015-04-21 | General Electric Company | Method and device for displaying a three-dimensional view of the surface of a viewed object |
US8718922B2 (en) * | 2011-07-28 | 2014-05-06 | Navteq B.V. | Variable density depthmap |
US8784301B2 (en) | 2011-08-12 | 2014-07-22 | Intuitive Surgical Operations, Inc. | Image capture unit and method with an extended depth of field |
US9116011B2 (en) | 2011-10-21 | 2015-08-25 | Here Global B.V. | Three dimensional routing |
US9047688B2 (en) | 2011-10-21 | 2015-06-02 | Here Global B.V. | Depth cursor and depth measurement in images |
US10008021B2 (en) | 2011-12-14 | 2018-06-26 | Microsoft Technology Licensing, Llc | Parallax compensation |
US20130162665A1 (en) | 2011-12-21 | 2013-06-27 | James D. Lynch | Image view in mapping |
US9024970B2 (en) | 2011-12-30 | 2015-05-05 | Here Global B.V. | Path side image on map overlay |
US9404764B2 (en) | 2011-12-30 | 2016-08-02 | Here Global B.V. | Path side imagery |
US10091489B2 (en) | 2012-03-29 | 2018-10-02 | Sharp Kabushiki Kaisha | Image capturing device, image processing method, and recording medium |
US20130308013A1 (en) | 2012-05-18 | 2013-11-21 | Honeywell International Inc. d/b/a Honeywell Scanning and Mobility | Untouched 3d measurement with range imaging |
WO2014020663A1 (en) | 2012-07-30 | 2014-02-06 | 三菱電機株式会社 | Map display device |
-
2011
- 2011-10-21 US US13/278,499 patent/US8553942B2/en active Active
-
2012
- 2012-10-17 WO PCT/EP2012/070526 patent/WO2013057127A1/en active Application Filing
- 2012-10-17 EP EP12773321.0A patent/EP2769181B1/en active Active
-
2013
- 2013-09-12 US US14/025,366 patent/US9641755B2/en active Active
Cited By (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9497391B2 (en) * | 2010-10-05 | 2016-11-15 | Sony Corporation | Apparatus and method for displaying images |
US20150249794A1 (en) * | 2010-10-05 | 2015-09-03 | Sony Computer Entertainment Inc. | Apparatus and method for displaying images |
US20130275920A1 (en) * | 2012-01-06 | 2013-10-17 | Tourwrist, Inc. | Systems and methods for re-orientation of panoramic images in an immersive viewing environment |
US20130342568A1 (en) * | 2012-06-20 | 2013-12-26 | Tony Ambrus | Low light scene augmentation |
US20210350629A1 (en) * | 2012-09-21 | 2021-11-11 | Navvis Gmbh | Visual localisation |
US11887247B2 (en) * | 2012-09-21 | 2024-01-30 | Navvis Gmbh | Visual localization |
US9124635B2 (en) * | 2012-11-30 | 2015-09-01 | Intel Corporation | Verified sensor data processing |
US20140157349A1 (en) * | 2012-11-30 | 2014-06-05 | Scott H. Robinson | Verified Sensor Data Processing |
US10453114B2 (en) | 2013-06-23 | 2019-10-22 | Intel Corporation | Selective sharing of user information based on contextual relationship information, such as to crowd-source gifts of interest to a recipient |
US20150009295A1 (en) * | 2013-07-03 | 2015-01-08 | Electronics And Telecommunications Research Institute | Three-dimensional image acquisition apparatus and image processing method using the same |
US10872239B2 (en) | 2014-06-08 | 2020-12-22 | Uber Technologies, Inc. | Entrance detection from street-level imagery |
US10289911B2 (en) * | 2014-06-08 | 2019-05-14 | Uber Technologies, Inc. | Entrance detection from street-level imagery |
US9373190B2 (en) * | 2014-07-09 | 2016-06-21 | Google Inc. | High-quality stereo reconstruction featuring depth map alignment and outlier identification |
CN107850445A (en) * | 2015-08-03 | 2018-03-27 | 通腾全球信息公司 | Method and system for generating and using locating reference datum |
WO2017173578A1 (en) * | 2016-04-05 | 2017-10-12 | 华为技术有限公司 | Image enhancement method and device |
CN106101743A (en) * | 2016-08-23 | 2016-11-09 | 广东欧珀移动通信有限公司 | Panoramic video recognition methods and device |
CN106908043A (en) * | 2017-02-23 | 2017-06-30 | 周良辰 | The three-dimensional amount measuring method of geographic position of target object and height based on Streetscape picture |
CN113014754A (en) * | 2017-02-23 | 2021-06-22 | 钰立微电子股份有限公司 | Image device for generating panoramic depth image and related image device |
CN107633554A (en) * | 2017-03-16 | 2018-01-26 | 柳州桂通科技股份有限公司 | Mobile surface targets and its methods of exhibiting with the three-dimensional scenic of present position |
US10776992B2 (en) * | 2017-07-05 | 2020-09-15 | Qualcomm Incorporated | Asynchronous time warp with depth data |
US10567904B2 (en) * | 2017-08-23 | 2020-02-18 | Harman International Industries, Incorporated | System and method for headphones for monitoring an environment outside of a user's field of view |
US20190069117A1 (en) * | 2017-08-23 | 2019-02-28 | Harman International Industries, Incorporated | System and method for headphones for monitoring an environment outside of a user's field of view |
US20190206029A1 (en) * | 2017-12-28 | 2019-07-04 | Panasonic Intellectual Property Management Co., Ltd. | Image processing device, display device, navigation system, and image processing method |
US10796415B2 (en) * | 2017-12-28 | 2020-10-06 | Panasonic Intellectual Property Management Co., Ltd. | Image processing device, display device, navigation system, and image processing method |
US10769840B2 (en) | 2018-02-27 | 2020-09-08 | Nvidia Corporation | Analysis of point cloud data using polar depth maps and planarization techniques |
US10776983B2 (en) | 2018-02-27 | 2020-09-15 | Nvidia Corporation | Analysis of point cloud data using depth and texture maps |
US11908203B2 (en) | 2018-02-27 | 2024-02-20 | Nvidia Corporation | Analysis of point cloud data using depth maps |
US11301697B2 (en) | 2018-02-27 | 2022-04-12 | Nvidia Corporation | Analysis of point cloud data using depth maps |
CN112313714A (en) * | 2018-06-22 | 2021-02-02 | 三星电子株式会社 | Method for displaying content and electronic device |
US11341729B2 (en) * | 2018-06-22 | 2022-05-24 | Samsung Electronics Co., Ltd. | Method and electronic device for correcting external reality pixels and virtual content pixels within an augmented reality environment |
CN109509255A (en) * | 2018-07-26 | 2019-03-22 | 京东方科技集团股份有限公司 | A kind of labeling map structuring and space map updating method and device |
US20220036604A1 (en) * | 2018-09-28 | 2022-02-03 | Align Technology, Inc. | Generation of images with tooth color determined using depth information |
US11727607B2 (en) * | 2018-09-28 | 2023-08-15 | Align Technology, Inc. | Generation of images with tooth color determined using depth information |
CN109785261A (en) * | 2019-01-11 | 2019-05-21 | 辽宁工程技术大学 | A kind of airborne LIDAR three-dimensional filtering method based on gray scale volume element model |
CN112241930A (en) * | 2019-07-18 | 2021-01-19 | 杭州海康威视数字技术股份有限公司 | Vehicle-mounted all-round looking method, device and system and vehicle |
US11961202B1 (en) * | 2023-02-23 | 2024-04-16 | Illuscio, Inc. | Systems and methods for editing three-dimensional image data with realistic camera and lens effects |
Also Published As
Publication number | Publication date |
---|---|
US9641755B2 (en) | 2017-05-02 |
US20140015919A1 (en) | 2014-01-16 |
WO2013057127A1 (en) | 2013-04-25 |
US8553942B2 (en) | 2013-10-08 |
EP2769181B1 (en) | 2019-11-20 |
EP2769181A1 (en) | 2014-08-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9641755B2 (en) | Reimaging based on depthmap information | |
US9390519B2 (en) | Depth cursor and depth management in images | |
US9116011B2 (en) | Three dimensional routing | |
US10235787B2 (en) | Path side image in map overlay | |
US9404764B2 (en) | Path side imagery | |
EP2737279B1 (en) | Variable density depthmap | |
US9256983B2 (en) | On demand image overlay | |
US20130162665A1 (en) | Image view in mapping | |
US9147285B2 (en) | System for visualizing three dimensional objects or terrain | |
US10444954B2 (en) | Distinguishable geographic area presentation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: NAVTEQ NORTH AMERICA, LLC, ILLINOIS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LYNCH, JAMES D.;REEL/FRAME:027122/0413 Effective date: 20111021 |
|
AS | Assignment |
Owner name: NAVTEQ B.V., NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NAVTEQ NORTH AMERICA, LLC;REEL/FRAME:027588/0051 Effective date: 20111229 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
AS | Assignment |
Owner name: HERE GLOBAL B.V., NETHERLANDS Free format text: CHANGE OF NAME;ASSIGNOR:NAVTEQ B.V.;REEL/FRAME:033830/0681 Effective date: 20130423 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |