US20180059247A1 - Method and arrangement for developing a three dimensional model of an environment - Google Patents
Method and arrangement for developing a three dimensional model of an environment Download PDFInfo
- Publication number
- US20180059247A1 US20180059247A1 US15/791,501 US201715791501A US2018059247A1 US 20180059247 A1 US20180059247 A1 US 20180059247A1 US 201715791501 A US201715791501 A US 201715791501A US 2018059247 A1 US2018059247 A1 US 2018059247A1
- Authority
- US
- United States
- Prior art keywords
- model
- images
- distance
- information
- distance information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000005259 measurement Methods 0.000 claims abstract description 69
- 238000012804 iterative process Methods 0.000 claims abstract description 7
- 238000011161 development Methods 0.000 claims description 11
- 238000003384 imaging method Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 238000011156 evaluation Methods 0.000 claims description 3
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 claims description 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 33
- 230000003287 optical effect Effects 0.000 description 10
- 230000018109 developmental process Effects 0.000 description 9
- 230000000875 corresponding effect Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 4
- 238000012935 Averaging Methods 0.000 description 2
- 230000002596 correlated effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000007423 decrease Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000001454 recorded image Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/06—Interpretation of pictures by comparison of two or more pictures of the same area
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3804—Creation or updating of map data
- G01C21/3833—Creation or updating of map data characterised by the source of data
- G01C21/3848—Data obtained from both position sensors and additional sensors
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3804—Creation or updating of map data
- G01C21/3833—Creation or updating of map data characterised by the source of data
- G01C21/3852—Data derived from aerial or satellite images
-
- G01S17/023—
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/42—Simultaneous measurement of distance and other co-ordinates
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G06K9/6215—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- H04N13/0207—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
Definitions
- the present invention relates to a method and arrangement for developing a 3D model of an environment.
- the geographical information can comprise digital maps having superposed information layers such as infrastructure, terrain type and different types of objects.
- This way of providing digital maps is time consuming and comprises forming two dimensional maps comprising capturing images of the terrain from an aircraft and post-processing of the captured images. It is an even more time consuming process to form three dimensional maps from captured images or range data sets of the terrain/infrastructure.
- WO 2009/003529 relates to another type of geographical information system. It relates to an arrangement and a method for providing a three dimensional map representation or a model of an area.
- the arrangement comprises a processing unit arranged to, for a plurality of time recorded, overlapping images of the area to be stereo processed, associate navigation states so that each pixel of each time recorded image is correlated to a corresponding navigation state and to perform the stereo processing based on the associated navigation states so that all pixels in the map representation or 3D model are specified in three geographical dimensions.
- One problem with the stereo processing of overlapping images is that a sharp spatial change tends to disappear or partly disappear during the stereo processing since only a part of the overlapping images catches the sharp spatial change.
- One way to overcome this problem is to introduce a laser rangefinder or a LIDAR device in the vicinity of the imaging device. Laser range measurements are conducted on a particular object or area during the period when the overlapping images are taken of the particular object or area. Each laser range measurement is very accurate at a single point and can be used to improve the accuracy of the 3D model.
- US2010/0204974 discloses a LIDAR and one or more electro-optical (EO) imaging device which may asynchronously acquire LIDAR shots and EO images.
- Navigation and timing data may be used to associate a particular LIDAR shot and/or EO image with navigation data.
- the navigation data may be used to cross correlate a LIDAR shot to a selected plurality of overlapping EO images.
- Ranging model information may be determined from EO image sequences using a stereo imaging technique.
- the stereo imaging technique may be seeded using the LIDAR shot data.
- One object of the present invention is to further improve modelling.
- the method comprises the steps of providing a plurality of overlapping images of the environment, each image being associated to navigation data; providing distance information, said distance information comprising a distance value and navigation data from a plurality of distance measurements; and developing the 3D model based on the plurality of overlapping images and the distance information.
- the step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images and updating the 3D model with the distance information using an iterative process.
- the step of providing of a plurality of overlapping images of the environment comprises capturing a plurality of overlapping images of the environment using an imaging device, providing navigation data related to the images; and associating the plurality of overlapping images with respective navigation data.
- the step of providing the distance information comprises transmitting a plurality of pulses from a distance measuring device for reflection in the environment, receiving pulses reflected from the environment; providing navigation data related to the transmitted and/or received pulses; determining information related to a relation between time of transmission and time of reception of each received pulse; and associating positioning data to each information related the relation between time of transmission and time of reception with respective navigation data;
- the distance information may be provided by means of LIDAR, wherein the pulses are laser pulses.
- the distance information may be provided by means of radar.
- the distance information may be provided by means of sonar.
- the navigation data comprises information regarding position, orientation and timing.
- the method further comprises a step of determining a weight relation between the distance measurement and the provided 3D model, wherein the updating of the 3D model with the distance information is based on the determined weight.
- the weight relation may be determined based on a percentage of a difference between the value from the distance measurement and the provided model.
- the weight relation may be determined based on an uncertainty in the provided 3D model.
- the weight relation may be determined based on an uncertainty in the distance measurement.
- the step of developing the 3D model comprises the steps of:
- Step e) of evaluating the updated model against the provided model may comprise re-projecting the stereo images based on the updated 3D model.
- Step f) of updating the selection of overlapping images may comprise selecting only those images and/or sub-images which show parts of the environment relevant for the most accurate model.
- Step g) of repeating the step of developing the 3D model may comprise the steps of determining a difference between the model generated based on stereo distances and a model generated based on stereo distances in a previous step, wherein the development of the 3D model is exited if the if the difference is below a selected level.
- the 3D model may be represented as a mesh.
- the 3D model may be represented as a surface representation.
- the 3D model may be represented as a voxel representation
- the invention also relates to a computer program comprising a program code for developing a 3D model of an environment, comprising the step of providing a plurality of overlapping images of the environment, each image associated of navigation data, providing distance information, said LIDAR information comprising a distance value and navigation data from a plurality of distance measurements; and developing the 3D model based on the plurality of overlapping images and the distance information.
- the step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images; and updating the 3D model with the distance information using an iterative process.
- the invention also relates to a computer program product comprising a program code stored on a computer readable media for developing a 3D model of an environment, comprising the step of providing a plurality of overlapping images of the environment, each image associated of navigation data, providing distance information, said LIDAR information comprising a distance value and navigation data from a plurality of distance measurements; and developing the 3D model based on the plurality of overlapping images and the distance information.
- the step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images; and updating the 3D model with the distance information using an iterative process.
- the present invention comprises an arrangement for developing a 3D model of an environment, said arrangement comprising a memory arranged to store a plurality of overlapping images of the environment, each image associated to navigation data and to store distance information comprising a distance value and navigation data from a plurality of distance measurements; a processing unit arranged to develop the 3D model based on the plurality of overlapping images and the distance information.
- the processing unit is arranged to determine the 3D model based on the plurality of overlapping images;
- FIG. 1 illustrates an arrangement for developing a 3D model of an environment at least partly mounted on an aircraft.
- FIG. 2 illustrates schematically the function of a camera in the arrangement in FIG. 1 .
- FIG. 3 illustrates schematically the function of a camera and a LIDAR device in the arrangement in FIG. 1 .
- FIGS. 4 a and 4 b illustrate schematically an example of a scanning pattern provided by the LIDAR device in the arrangement in FIG. 1 .
- FIG. 5 is a block scheme illustrating an example of the arrangement of FIG. 1 .
- FIG. 6 is a schematic illustration of a first part of modelling a scene between buildings.
- FIG. 7 is a schematic illustration of a second part of modelling a scene between buildings.
- FIG. 8 is a schematic illustration of a scene wherein a laser pulse is transmitted.
- FIG. 9 is a schematic illustration of modelling a scene with a forest.
- FIG. 10 shows a flow chart illustrating an example of a method for developing a 3D model of an environment.
- FIG. 11 shows a flow chart illustrating an example of a method for developing a 3D model of an environment.
- an arrangement 101 for developing a three dimensional model of an environment 103 is mounted on a movable carrier 102 .
- the carrier is a in an airborne vehicle.
- the airborne vehicle is for example a manned or unmanned fighter or civilian aircraft.
- the movable carrier is in an alternative example (not shown) a satellite, a land vehicle, or a watercraft, for example a lorry, ship or submarine.
- the arrangement 101 can also be hand held or mounted on a person. In one example, only parts of the arrangement are mounted in the movable carrier while other parts, for example processing parts, are at a remote location.
- the three dimensional model provided by the arrangement 101 is related to geographical coordinate system.
- the 3D model is provided from stereo image processing a plurality of overlapping geo-referenced images of the environment.
- the geo-referenced images are provided from at least one camera.
- the 3D model is updated with information from a distance measuring device.
- one or a plurality of cameras can be used comprising a camera for visual light, an IR camera, and/or a video camera.
- the distance measuring device can be any type of distance measuring device arranged to determine a distance with spatial resolution.
- the distance measuring device comprises a transmitter arranged to continuously transmit pulses a receiver arranged to receive pulses transmitted from the transmitter and reflected in the environment.
- the distance measuring device is arranged to determine a distance to the reflection point based on the time difference between transmittal and reception at a certain pulse.
- the distance measuring device is in one example operating based on optical light, ultrasonic and/or radar based.
- the optical distance measuring device is in one example LIDAR based.
- the optical distance measuring device comprises a laser transmitter and a detector arranged to detect laser radiation.
- the transmitter is a projector transmitting a light pattern and the associated receiver is a camera.
- the distance measurements will be described in relation to LIDAR.
- the arrangement for developing the three dimensional model comprises at least one camera and a LIDAR device supported by a movable carrier 102 .
- the LIDAR device is configured to obtain ranging information of the environment by transmitting laser energy towards the environment and detecting laser energy reflected and/or emitted there from.
- the arrangement 101 for developing the three dimensional model is then arranged to first provide the 3D model based on the plurality of overlapping images and then to update the 3D model with the LIDAR information.
- a difference between the provided 3D model and the LIDAR information is in one example determined at each location where LIDAR information is available.
- the weight of the LIDAR measurement in the updated model is determined based on a predetermined scheme. For example, the weight of the LIDAR information is determined as a percentage of the of a difference between the 3D model and the LIDAR information. In one example, the percentage is 100% of the difference, i.e. the weight of the LIDAR information is 100%. In an alternative example, the weight of the LIDAR information is 40-60%.
- an uncertainty in the 3D model is determined at each location where LIDAR information is available.
- the weight of the LIDAR information is also or instead determined based on the uncertainty in the 3D model.
- uncertainties in the 3D model will be described in relation to FIG. 5 .
- an uncertainty can also be determined related to the LIDAR measurement.
- the weight of the LIDAR measurement in the updated model is then also or instead determined based on the uncertainty in the LIDAR measurement.
- At least one camera 204 is illustrated supported by a not shown movable carrier. At a first time, the at least one camera is pointing in a first direction to a first field of view 205 a for a first image 206 a captured by the camera 204 . At a second time, the at least one camera 204 is directed in a second direction to a second field of view 205 b for a second image 206 b captured by the camera 204 .
- the at least one camera 204 is arranged to provide a plurality of at least partly overlapping images 206 a , 206 b each covering at least a part of the environment.
- the model can be made better the more images from different camera positions are available in modelling an object or a surface in the environment. In one example, when images are captured from an airborne vehicle, some surfaces visible from many positions in the air are captured in 20 or more different images while other surfaces are visible in fewer different images.
- a distance measuring device (not shown) is arranged on a moving carrier together with above described camera(s) 304 .
- the ranging information provided from the LIDAR measurements is used to assist developing a 3D model.
- the LIDAR device is directed to the ground to scan with pulses 307 the same parts as covered by images 306 a , 306 b taken by the camera(s) 304 .
- the area of a LIDAR pulse is much smaller than the area of an image taken from the same height. Therefore in one example, a plurality of LIDAR pulses are provided within the area of each image.
- FIGS. 4 a and 4 b examples of different scanning patterns 408 , 409 on the ground provided by a distance measuring device arranged on a moving carrier are illustrated.
- the direction of the movable carrier (not shown) is illustrated with an arrow.
- the laser is arranged to perform a movement back and forward in a direction substantially perpendicular to the direction of movement of the movable carrier (not shown) to scan the environment in a zigzag pattern 408 .
- the LIDAR device is arranged so that the laser beam directed to the environment rotates in a circle, thereby providing LIDAR pulses 231 forming a helical pattern 409 as the carrier moves in the direction of the arrow so as to scan the environment.
- the LIDAR device is arranged to perform the linear and/or circular movement.
- the LIDAR device is fixedly arranged and a mirror arrangement in the beam path from the LIDAR device is arranged to perform a pivoting movement so as to provide the linear and/or circular scanning.
- the camera images are taken and the distance measures are performed at different times. In an alternative example, the camera images are taken and the distance measures are performed at the same time.
- a 3D model is provided using an arrangement 501 for developing a three dimensional model of an environment.
- the arrangement 501 comprises at least one camera 504 arranged to generate images.
- the at least one camera 504 is arranged to provide a plurality overlapping images covering the environment for which the model is built.
- the camera is for example a camera for visual light or an IR camera.
- the arrangement 501 comprises further a distance measuring device 510 .
- the distance measuring device 510 can be any type of distance measuring device arranged to determine a distance with spatial resolution.
- ladar, sonar, distance measurement using structured light and/or radar can be used in addition to measurements based on camera images.
- the distance measurements will be described in relation to LIDAR.
- the arrangement 501 comprises in accordance with this example further a positioning system 511 or a receiver of a positioning system arranged to provide positioning and direction information related to the at least one camera and related to the LIDAR device.
- the direction information relates to the optical direction of the camera/LIDAR device.
- the images are associated to this positioning and direction information.
- the distance measurements are associated to this positioning and direction information.
- the images and/or LIDAR measured distances may be associated to timing information.
- the timing information is provided with accuracy sufficient for the application.
- the positioning system 511 comprises in one example a receiver of a satellite based positioning system, such as GPS.
- the positioning system may also comprise an inertial navigation system.
- the timing information may be provided from the receiver in the positioning system, said receiver being arranged to receive and process signals of a satellite based positioning system, such as GPS.
- the arrangement 501 comprises a processing unit 512 arranged to, based on the position and direction information related to the at least one camera 504 , stereo image process an arbitrary number of at least partly overlapping image sets generated by the at least one camera so as to provide the three dimensional model.
- the processing unit 512 is in one example arranged to find corresponding points in the at least partly overlapping images and to find disparity estimations based on the corresponding points so as to provide the stereo image processing.
- the processing unit 512 is arranged to, for each image to be stereo image processed, associate the position and direction information so that each pixel of each image is correlated to corresponding position and direction information. The stereo image processing is then performed based on the associated position and direction information so that all pixels in the 3D model are specified in three geographical dimensions.
- the processing unit 512 is arranged to divide the environment into a plurality of areas or points, providing for each area or point a plurality of geo-referenced image sets, wherein each image comprises the area or point, performing for each area or point image stereo processing on each image set so as to provide a plurality of 3D sub models for that area or point and providing the 3D model for each area or point based on the plurality of 3D sub-models.
- the 3D model for each area or point is provided by averaging the point or area provided from the different sub-models.
- the images or image sets are associated to a weigh factor dependent on the quality of the image. The averaging can then be weighted.
- the processing unit 512 is arranged to compose the 3D model based on the 3D models related to the different areas or points.
- the processing unit 512 may be arranged to perform bundle adjustment.
- the processing unit 512 is further arranged to provide the 3D model also based on information from the distance measurement device 510 .
- the processing unit 512 is in one example arranged to develop the 3D model based on the plurality of overlapping images and to update the model with the distance information from the distance measuring device, where appropriate.
- the 3D model covering a given area or environment is developed using the plurality of overlapping images.
- the 3D model is then compared with information from the distance measuring device. In those parts of the environment where the 3D model substantially coincides with the information provided from the distance measurements, the provided 3D model is regarded as finalized.
- the 3D model is updated in those parts, based on the distance information.
- the weight of the distance information in the updated model may be determined as described above and will be exemplified more in detail below.
- the processing unit 512 is arranged to verify that the 3D model parts updated based on the distance information better describes the reality as presented in the images than the corresponding 3D model not updated with the distance information. This is in one example performed by re-projecting one image to another image in those parts where the model has been updated so as to determine if the updated model gives a better re-projected estimated image than the not updated model.
- the not updated model and the updated model can be determined based on comparing an image I 2 taken from one certain location with different estimated images Î 2 determined for the same certain location.
- the estimated images are determined based on another image I 1 taken from another location and projected in the not updated 3D model respectively the updated 3D model to the position of the location of the image I 2 .
- M represents the not updated 3D model respectively the updated model.
- the images are compared or matched in small windows of the images.
- the images are compared on a sub-image by sub-image basis.
- the matching is performed based on a correlation technique.
- the matching is performed based on a phase based algorithm.
- the matching is performed based on a segmentation based algorithm.
- the model performed based only on the overlapping images provides the best image estimate for the image I 2 , then it is assumed that the distance information should not be used in the model or at least to a smaller extent than in the updated model. If on the other hand, the model developed based also on the distance information gives the best image estimate for the image I 2 , then it is assumed that the distance information enhances the 3D model.
- the processing unit 512 may be arranged to verify in other ways known to the person skilled in the art that the 3D model updated based on the distance information better describes the reality as presented in the images than the 3D model not updated with the distance information.
- the processing unit 512 is arranged to repeat the development of a model based on only overlapping images. This time, the developing of the model is based on a selection of the overlapping images in which the coordinates of the updated model are visible. The processing unit 512 is then arranged to compare the model with the distance information. If there is a difference between the model and the distance information at some point, then the 3D model is updated in this point as described above. The processing unit is then arranged to verify that the 3D model updated based on the distance information better describes the reality as presented in the images than the 3D model not updated with the distance information, as described above.
- the processing unit 512 is arranged to repeat the development of the model based on a selection of the overlapping images and/or based on a selection of overlapping sub-images. If it is not verified that the updated model better describes the environment, the updating of the model may be finalized or repeated again using a decreased influence from the distance measurement. If for example, the distance information is wrong, such as if it has been measured against a flying bird, this distance information will not improve the model and will thus be rejected in the verification of the updated model.
- the processing unit 512 is in one example arranged to determine a difference between the developed model and a model developed in the previous step and to exit the development of the 3D model if the difference decreases a predetermined value. As is understood from above, the processing unit 512 may also be arranged to exit the development of the model if the distance information is determined not to improve the model.
- the processing unit 512 is in one example arranged to determine an uncertainty at some locations or at each location where LIDAR information is available. The processing unit is then arranged to determine the weight of the LIDAR measurement in the updated model based on the uncertainty in the 3D model at that specific location. In one example, the uncertainty in the model is determined based on the angle between the optical axis of the camera images used for modelling and a plane of the surface of the model at that specific location. For example, for surfaces of the 3D model which are perpendicular to an optical axis of the camera at that specific location, the uncertainty in the 3D model is lower than for surfaces which are substantially parallel to the optical axis of the camera.
- the processing unit 512 may further be arranged to determine an uncertainty related to the LIDAR measurement.
- the weight of the LIDAR measurement in the updated model is then determined based on the uncertainty in the 3D model and/or the uncertainty in the LIDAR measurement.
- the 3D model is represented as a mesh. In an alternative example, the 3D model is represented as a surface representation. In an alternative example, the 3D model is represented as a voxel representation.
- the processing unit 512 comprises in one example a computer program comprising a program code for developing a 3D model as discussed above. Further, a computer program product comprises a program code stored on a computer readable media for developing a 3D model of an environment.
- the arrangement 501 also comprises a memory 513 for storing data related to the three dimensional model calculated by the processing unit 512 .
- the memory is also arranged to store information related to the overlapping images and the distance information.
- the arrangement 501 further comprises a display or presentation unit 514 arranged to provide information related to the 3D model.
- the display unit may be arranged to present a selected part of the 3D model.
- the arrangement 501 may also comprise input means (not shown) for selecting a part of the 3D model and the display is arranged to present information related to the selected part.
- the arrangement 501 may also comprise a transmitter (not shown) arranged to transmit the information related to the 3D model to a receiver in a remote location.
- the transmitter at least partly substitutes the memory 513 and/or the display unit 514 .
- the transmitter is provided in addition to the memory 513 and/or the display unit 514 .
- the arrangement does not comprise the camera(s) 504 and the distance measuring device 510 .
- the processing unit 512 is then arranged to develop the model based on image and distance information stored in the memory 513 .
- FIG. 6 a first part of a procedure for developing a 3D model of an environment is illustrated.
- a 3D model 620 has been determined based on images taken from a plurality of camera locations 604 a , 604 b , 604 c , 604 d .
- a field of view 605 a , 605 b , 605 c , 605 d is shown for the respective camera 604 a , 604 b , 604 c , 604 d at the time of taking the images.
- the field of view illustrates the cover of the images captured.
- the 3D model 620 is compared to LIDAR measurement points 621 a , 621 b , 621 c at corresponding locations.
- the 3D model and the LIDAR measurements are substantially coinciding at a first and third measurement location 621 a , 621 c .
- the LIDAR measurement point 621 b differs from a corresponding point in the model 620 .
- the second measurement point 621 b of the LIDAR measurement is formed at the ground of an alley between two buildings 622 a , 622 b .
- the 3D model and the LIDAR measurement differs, the 3D model is locally drawn in a direction towards the LIDAR measurement point.
- the 3D model 620 is drawn to at least one updated point between the model and the second measurement point 621 b as determined by a weight of the LIDAR measurement.
- the at least one updated point is selected such that the 3D model 620 is drawn to the second measurement point 621 b .
- the at least one updated point is determined based on the weight of the LIDAR information. The weight of the LIDAR information is determined based on a predetermined scheme, as earlier discussed.
- the weight of the LIDAR measurement is also or instead determined based on an uncertainty in the 3D model.
- the uncertainty in the model is based on the angular relationship between the surface of the model 620 at the location of the second measurement point 621 b and the optical axes of the cameras when taking the available images. If the surface and the optical axes are close to parallel, then uncertainty is greater than if the relation between the surface and the optical axes is close to perpendicular.
- the uncertainty in the model is instead or in addition thereto determined by from how many or for how high percentage of the camera images the LIDAR measurement points are visible.
- the uncertainty is in one example then higher than if the LIDAR point is visible from a high percentage or substantially all the images used in the development of the 3D model.
- the 3D model point is determined to be moved towards the corresponding LIDAR measurement location to an extent determined based on the 3D model uncertainty.
- FIG. 7 a second part of a procedure for developing a 3D model of an environment is shown. An updated model 722 has been determined.
- the updated model is based on the model developed only using overlapping image pairs and updated with at least one updated point 723 determined based on the distance information. As discussed above, the weight of the distance information has been determined according to a predetermined scheme
- the updated model 722 is evaluated. This was also described in relation to FIG. 5 . If the updated model 722 is considered to be better than the not updated model, then the model will be re-developed using only those images which shows the updated point(s) 723 of the 3D model provided based on the second measurement point ( 621 b in FIG. 6 ) and based on the weight of the LIDAR measurement.
- a cone 724 illustrates which images and/or sub-images may be used for the re-developed of the model.
- a 3D model 620 will be determined based on the images taken from a plurality of camera locations 704 b , 704 c within the cone 724 .
- Sub-images may also be used, In the illustrated example, a field of view 705 b , 705 c is shown for the respective camera 704 b , 704 c at the time of taking the images. Thus the field of view illustrates the cover of the images captured.
- weight of the LIDAR measurement is determined based on an uncertainty in the LIDAR measurement.
- the pulse from a LIDAR device 810 used has a beam lobe 825 having a certain size in cross section.
- one pulse for determining a distance generates two different distance results due to the extent of the lobe 825 in its cross section.
- the weight of the LIDAR information may be less than in areas where a high spatial resolution is not required in a plane perpendicular to the beam.
- a model of a forest is developed.
- the model produced by the images and the LIDAR measurements will provide different results.
- a model 920 provided based on overlapping images will be close to the tree tops and potentially the tree tops will not be modelled to their full extent.
- the LIDAR measurements will provide spread measurement points. Some measurement points will be in the tree tops and some measurement points will be close to ground.
- the LIDAR measurements are evaluated and it can be determined from the pattern of points that these points cannot be used in the modelling. Then those LIDAR measurement points which are not desirable to use can be removed. In one additional or alternative example, the LIDAR measurement points are used.
- LIDAR measurement points which are close to the ground will be determined not to improve the model and will thus be rejected.
- Those LIDAR measurement points which are close to the tree tops can be used to model the tree tops more close to their full extension.
- the pattern of the LIDAR measurement points are used for determining an uncertainty in the LIDAR information and thus can be used in modelling to an extent determined by the uncertainty in the LIDAR measurement.
- FIG. 10 an example of a method 1000 for developing a 3D model based on overlapping images and distance information is illustrated.
- the method comprises providing 1010 , overlapping images associated to navigation data. Further, distance information is also provided 1020 comprising a distance value and associated to navigation data. Thereafter, the 3D model is developed 1030 based on the plurality of overlapping images and the distance information.
- the 3D model is provided 1031 based on the plurality of overlapping images, the provided 1031 model is compared 1032 with the distance information and the provided model is updated 1034 with the distance information using an iterative process, where appropriate.
- a weight relation between the distance measurement and the provided 3D model is determined 1033 .
- the 3D model can then be updated 1034 with the distance information on the determined 1033 weight.
- the weight relation may for example be determined based on a percentage of a difference between the value from the distance measurement and the provided model.
- the weight relation can also or instead be determined based on an uncertainty in the provided 3D model.
- the weight relation may instead or in addition thereto be determined based on an uncertainty in the distance measurement.
- the stereo distances are determined 1015 from the provided overlapping images.
- the model can then be provided based on the determined stereo distances.
- the stereo distances are determined from the overlapping images based on correlation.
- the stereo distances are determining from the overlapping images based on a phase based algorithm.
- the stereo distances are determined from the overlapping images is based on segmentation based algorithm. In one example, any other stereo method known to the person skilled in the art is used.
- the updated mode is then evaluated 1035 against the provided, not updated model to determine which of the models is most accurate. If the not updated model is determined to be most accurate, the development of the model may be finalized. If it is determined that the updated model is most accurate, or at least it is determined that the distance information does improve the model, the process is repeated.
- the evaluation 1035 of the updated model against the provided model comprises in one example re-projecting the stereo images based on the updated and the not updated 3D models and comparing the results.
- the selection of images is then updated 1036 based on the coordinates to the updated model.
- the updating 1036 of the selection of overlapping images comprises selecting only those images and/or sub-images which show parts of the environment relevant for the most accurate model (updated or not updated).
- the decision of repeating 1037 development of the model comprises the steps of determining a difference between the model generated based on the overlapping images and a model generated based on the selection of overlapping in a previous step.
- the development of the 3D model can then be exited if the difference is below a selected level.
- FIG. 11 a flow chart illustrating an example of a method 1100 for developing a 3D model of an environment is shown.
- a step of providing 1110 image data comprises the steps of providing 1111 a plurality of overlapping images of the environment, providing navigation data 1112 and associating 1113 the navigation data to the respective images.
- the navigation data may comprise position information and a pointing direction of a camera capturing the images at each instant of capturing.
- navigation data may comprise information regarding position and orientation. It may also comprise information related to timing.
- step of providing 1120 distance information comprises the steps of transmitting 1121 a plurality of pulses such as laser from a distance measuring device for reflection in the environment, receiving 1122 pulses reflected from the environment, providing 1123 navigation data related to pulses and determining 1124 information related to a relation between time of transmission and time of reception of each received pulse.
- the distance between the distance measuring device and the reflection point associated to each pulse can then be determined based on the determined relation.
- navigation data is associated 1125 to each determined distance.
- the navigation data may comprise position information and the direction of the transmitted pulses.
- navigation data may comprise information regarding position and orientation. It may also comprise information related to timing.
- the distance information is provided by means of LIDAR. In one example, the distance information is provided by means of radar. In one example, the distance information is provided by means of sonar.
- the 3D model is developed 1130 based on the provided image data and based on the provided distance information.
- the image data and the distance measurements are provided using the same platform. It may then not be necessary to determine navigation data specifically to both the camera(s) and the distance measuring unit. It may then be enough to know the relation between the camera(s) and the distance measuring device.
- the 3D model may be represented as a surface representation and/or a voxel and/or a mesh.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- General Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Multimedia (AREA)
- Automation & Control Theory (AREA)
- Computer Graphics (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Architecture (AREA)
- Computer Hardware Design (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Optical Radar Systems And Details Thereof (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Processing (AREA)
Abstract
Description
- This application is a continuation of U.S. patent application Ser. No. 14/762,174, filed Jul. 20, 2015, published as US 2015/0362595, Dec. 17, 2015, which is a U.S. National Stage of PCT Application No. PCT/SE2013/050044, filed Jan. 21, 2013, published as WO2014/112911, Jul. 24, 2014, all of which are hereby incorporated by reference in their entireties.
- The present invention relates to a method and arrangement for developing a 3D model of an environment.
- A fast growing market both in civilian and military business is geographical information systems. Knowledge about geographical conditions forms a fundamental decision support to companies, authorities and in the military. The geographical information can comprise digital maps having superposed information layers such as infrastructure, terrain type and different types of objects. This way of providing digital maps is time consuming and comprises forming two dimensional maps comprising capturing images of the terrain from an aircraft and post-processing of the captured images. It is an even more time consuming process to form three dimensional maps from captured images or range data sets of the terrain/infrastructure.
- WO 2009/003529 relates to another type of geographical information system. It relates to an arrangement and a method for providing a three dimensional map representation or a model of an area. The arrangement comprises a processing unit arranged to, for a plurality of time recorded, overlapping images of the area to be stereo processed, associate navigation states so that each pixel of each time recorded image is correlated to a corresponding navigation state and to perform the stereo processing based on the associated navigation states so that all pixels in the map representation or 3D model are specified in three geographical dimensions.
- One problem with the stereo processing of overlapping images is that a sharp spatial change tends to disappear or partly disappear during the stereo processing since only a part of the overlapping images catches the sharp spatial change. One way to overcome this problem is to introduce a laser rangefinder or a LIDAR device in the vicinity of the imaging device. Laser range measurements are conducted on a particular object or area during the period when the overlapping images are taken of the particular object or area. Each laser range measurement is very accurate at a single point and can be used to improve the accuracy of the 3D model.
- US2010/0204974 discloses a LIDAR and one or more electro-optical (EO) imaging device which may asynchronously acquire LIDAR shots and EO images. Navigation and timing data may be used to associate a particular LIDAR shot and/or EO image with navigation data. The navigation data may be used to cross correlate a LIDAR shot to a selected plurality of overlapping EO images. Ranging model information may be determined from EO image sequences using a stereo imaging technique. The stereo imaging technique may be seeded using the LIDAR shot data.
- One object of the present invention is to further improve modelling.
- This has in one example been solved by means of a method for developing a 3D model of an environment. The method comprises the steps of providing a plurality of overlapping images of the environment, each image being associated to navigation data; providing distance information, said distance information comprising a distance value and navigation data from a plurality of distance measurements; and developing the 3D model based on the plurality of overlapping images and the distance information. The step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images and updating the 3D model with the distance information using an iterative process.
- In one option, the step of providing of a plurality of overlapping images of the environment comprises capturing a plurality of overlapping images of the environment using an imaging device, providing navigation data related to the images; and associating the plurality of overlapping images with respective navigation data.
- In one option, the step of providing the distance information comprises transmitting a plurality of pulses from a distance measuring device for reflection in the environment, receiving pulses reflected from the environment; providing navigation data related to the transmitted and/or received pulses; determining information related to a relation between time of transmission and time of reception of each received pulse; and associating positioning data to each information related the relation between time of transmission and time of reception with respective navigation data;
- The distance information may be provided by means of LIDAR, wherein the pulses are laser pulses. The distance information may be provided by means of radar. The distance information may be provided by means of sonar.
- The navigation data comprises information regarding position, orientation and timing.
- In one option, the method further comprises a step of determining a weight relation between the distance measurement and the provided 3D model, wherein the updating of the 3D model with the distance information is based on the determined weight. The weight relation may be determined based on a percentage of a difference between the value from the distance measurement and the provided model. The weight relation may be determined based on an uncertainty in the provided 3D model. The weight relation may be determined based on an uncertainty in the distance measurement.
- In one option, the step of developing the 3D model comprises the steps of:
- a) determining stereo distances from the overlapping images;
b) generating the 3D model based on the stereo distances from a selection of the overlapping images;
c) determining the difference between the 3D model and the distance information;
d) updating the 3D model based on the difference between the 3D model and the distance information and based on the determined weight relation,
e) evaluate the updated model against the provided model to determine which of the models is most accurate
f) updating the selection of the overlapping images based on the evaluation,
g) repeating step b) to f). - Step e) of evaluating the updated model against the provided model may comprise re-projecting the stereo images based on the updated 3D model.
- Step f) of updating the selection of overlapping images may comprise selecting only those images and/or sub-images which show parts of the environment relevant for the most accurate model.
- Step g) of repeating the step of developing the 3D model may comprise the steps of determining a difference between the model generated based on stereo distances and a model generated based on stereo distances in a previous step, wherein the development of the 3D model is exited if the if the difference is below a selected level.
- The 3D model may be represented as a mesh. The 3D model may be represented as a surface representation. The 3D model may be represented as a voxel representation
- The invention also relates to a computer program comprising a program code for developing a 3D model of an environment, comprising the step of providing a plurality of overlapping images of the environment, each image associated of navigation data, providing distance information, said LIDAR information comprising a distance value and navigation data from a plurality of distance measurements; and developing the 3D model based on the plurality of overlapping images and the distance information. The step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images; and updating the 3D model with the distance information using an iterative process.
- The invention also relates to a computer program product comprising a program code stored on a computer readable media for developing a 3D model of an environment, comprising the step of providing a plurality of overlapping images of the environment, each image associated of navigation data, providing distance information, said LIDAR information comprising a distance value and navigation data from a plurality of distance measurements; and developing the 3D model based on the plurality of overlapping images and the distance information. The step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images; and updating the 3D model with the distance information using an iterative process.
- In one embodiment, the present invention comprises an arrangement for developing a 3D model of an environment, said arrangement comprising a memory arranged to store a plurality of overlapping images of the environment, each image associated to navigation data and to store distance information comprising a distance value and navigation data from a plurality of distance measurements; a processing unit arranged to develop the 3D model based on the plurality of overlapping images and the distance information. The processing unit is arranged to determine the 3D model based on the plurality of overlapping images;
- and to update the 3D model with the distance information using an iterative process.
- The invention will be further described with reference to the accompanying drawings.
-
FIG. 1 illustrates an arrangement for developing a 3D model of an environment at least partly mounted on an aircraft. -
FIG. 2 illustrates schematically the function of a camera in the arrangement inFIG. 1 . -
FIG. 3 illustrates schematically the function of a camera and a LIDAR device in the arrangement inFIG. 1 . -
FIGS. 4a and 4b illustrate schematically an example of a scanning pattern provided by the LIDAR device in the arrangement inFIG. 1 . -
FIG. 5 is a block scheme illustrating an example of the arrangement ofFIG. 1 . -
FIG. 6 is a schematic illustration of a first part of modelling a scene between buildings. -
FIG. 7 is a schematic illustration of a second part of modelling a scene between buildings. -
FIG. 8 is a schematic illustration of a scene wherein a laser pulse is transmitted. -
FIG. 9 is a schematic illustration of modelling a scene with a forest. -
FIG. 10 shows a flow chart illustrating an example of a method for developing a 3D model of an environment. -
FIG. 11 shows a flow chart illustrating an example of a method for developing a 3D model of an environment. - In
FIG. 1 , anarrangement 101 for developing a three dimensional model of anenvironment 103 is mounted on amovable carrier 102. In the illustrated example, the carrier is a in an airborne vehicle. The airborne vehicle is for example a manned or unmanned fighter or civilian aircraft. The movable carrier is in an alternative example (not shown) a satellite, a land vehicle, or a watercraft, for example a lorry, ship or submarine. Thearrangement 101 can also be hand held or mounted on a person. In one example, only parts of the arrangement are mounted in the movable carrier while other parts, for example processing parts, are at a remote location. - The three dimensional model provided by the
arrangement 101 is related to geographical coordinate system. The 3D model is provided from stereo image processing a plurality of overlapping geo-referenced images of the environment. The geo-referenced images are provided from at least one camera. In addition thereto, the 3D model is updated with information from a distance measuring device. - In one example, one or a plurality of cameras can be used comprising a camera for visual light, an IR camera, and/or a video camera. The distance measuring device can be any type of distance measuring device arranged to determine a distance with spatial resolution. The distance measuring device comprises a transmitter arranged to continuously transmit pulses a receiver arranged to receive pulses transmitted from the transmitter and reflected in the environment. The distance measuring device is arranged to determine a distance to the reflection point based on the time difference between transmittal and reception at a certain pulse. The distance measuring device is in one example operating based on optical light, ultrasonic and/or radar based. The optical distance measuring device is in one example LIDAR based. In one example, the optical distance measuring device comprises a laser transmitter and a detector arranged to detect laser radiation. In one example, the transmitter is a projector transmitting a light pattern and the associated receiver is a camera. In the following description, the distance measurements will be described in relation to LIDAR.
- Thus, the arrangement for developing the three dimensional model comprises at least one camera and a LIDAR device supported by a
movable carrier 102. The LIDAR device is configured to obtain ranging information of the environment by transmitting laser energy towards the environment and detecting laser energy reflected and/or emitted there from. - The
arrangement 101 for developing the three dimensional model is then arranged to first provide the 3D model based on the plurality of overlapping images and then to update the 3D model with the LIDAR information. A difference between the provided 3D model and the LIDAR information is in one example determined at each location where LIDAR information is available. The weight of the LIDAR measurement in the updated model is determined based on a predetermined scheme. For example, the weight of the LIDAR information is determined as a percentage of the of a difference between the 3D model and the LIDAR information. In one example, the percentage is 100% of the difference, i.e. the weight of the LIDAR information is 100%. In an alternative example, the weight of the LIDAR information is 40-60%. In one example, an uncertainty in the 3D model is determined at each location where LIDAR information is available. In this example, the weight of the LIDAR information is also or instead determined based on the uncertainty in the 3D model. Detailed examples of uncertainties in the 3D model will be described in relation toFIG. 5 . Further, an uncertainty can also be determined related to the LIDAR measurement. The weight of the LIDAR measurement in the updated model is then also or instead determined based on the uncertainty in the LIDAR measurement. - In
FIG. 2 , at least onecamera 204 is illustrated supported by a not shown movable carrier. At a first time, the at least one camera is pointing in a first direction to a first field ofview 205 a for afirst image 206 a captured by thecamera 204. At a second time, the at least onecamera 204 is directed in a second direction to a second field of view 205 b for a second image 206 b captured by thecamera 204. The at least onecamera 204 is arranged to provide a plurality of at least partly overlappingimages 206 a, 206 b each covering at least a part of the environment. The model can be made better the more images from different camera positions are available in modelling an object or a surface in the environment. In one example, when images are captured from an airborne vehicle, some surfaces visible from many positions in the air are captured in 20 or more different images while other surfaces are visible in fewer different images. - In
FIG. 3 , a distance measuring device (not shown) is arranged on a moving carrier together with above described camera(s) 304. We herein describe the distance measurement device in relation to LIDAR. The ranging information provided from the LIDAR measurements is used to assist developing a 3D model. The LIDAR device is directed to the ground to scan withpulses 307 the same parts as covered byimages - In
FIGS. 4a and 4b , examples ofdifferent scanning patterns zigzag pattern 408. InFIG. 4b , the LIDAR device is arranged so that the laser beam directed to the environment rotates in a circle, thereby providing LIDAR pulses 231 forming ahelical pattern 409 as the carrier moves in the direction of the arrow so as to scan the environment. In one example, the LIDAR device is arranged to perform the linear and/or circular movement. In an alternative example, the LIDAR device is fixedly arranged and a mirror arrangement in the beam path from the LIDAR device is arranged to perform a pivoting movement so as to provide the linear and/or circular scanning. An advantage with LIDAR pulses forming a helical pattern is that high resolution in all direction are achieved since the difference between two LIDAR pulses in a certain direction is small at least between some pairs of LIDAR pulses along the plurality of LIDAR pulses forming a helical pattern. It should be mentioned that the plurality of LIDAR pulses can form any other pattern, such as sine formed pattern or any irregular pattern. - In one example, the camera images are taken and the distance measures are performed at different times. In an alternative example, the camera images are taken and the distance measures are performed at the same time.
- In the example of
FIG. 5 , a 3D model is provided using anarrangement 501 for developing a three dimensional model of an environment. Thearrangement 501 comprises at least onecamera 504 arranged to generate images. The at least onecamera 504 is arranged to provide a plurality overlapping images covering the environment for which the model is built. The camera is for example a camera for visual light or an IR camera. - The
arrangement 501 comprises further adistance measuring device 510. As described in relation toFIG. 1 , thedistance measuring device 510 can be any type of distance measuring device arranged to determine a distance with spatial resolution. For example, ladar, sonar, distance measurement using structured light and/or radar can be used in addition to measurements based on camera images. As stated above, in the following description, the distance measurements will be described in relation to LIDAR. - The
arrangement 501 comprises in accordance with this example further apositioning system 511 or a receiver of a positioning system arranged to provide positioning and direction information related to the at least one camera and related to the LIDAR device. The direction information relates to the optical direction of the camera/LIDAR device. The images are associated to this positioning and direction information. Further, the distance measurements are associated to this positioning and direction information. Further, the images and/or LIDAR measured distances may be associated to timing information. The timing information is provided with accuracy sufficient for the application. - The
positioning system 511 comprises in one example a receiver of a satellite based positioning system, such as GPS. The positioning system may also comprise an inertial navigation system. The timing information may be provided from the receiver in the positioning system, said receiver being arranged to receive and process signals of a satellite based positioning system, such as GPS. - Further, the
arrangement 501 comprises aprocessing unit 512 arranged to, based on the position and direction information related to the at least onecamera 504, stereo image process an arbitrary number of at least partly overlapping image sets generated by the at least one camera so as to provide the three dimensional model. In detail, theprocessing unit 512 is in one example arranged to find corresponding points in the at least partly overlapping images and to find disparity estimations based on the corresponding points so as to provide the stereo image processing. In one example, theprocessing unit 512 is arranged to, for each image to be stereo image processed, associate the position and direction information so that each pixel of each image is correlated to corresponding position and direction information. The stereo image processing is then performed based on the associated position and direction information so that all pixels in the 3D model are specified in three geographical dimensions. - In one example, the
processing unit 512 is arranged to divide the environment into a plurality of areas or points, providing for each area or point a plurality of geo-referenced image sets, wherein each image comprises the area or point, performing for each area or point image stereo processing on each image set so as to provide a plurality of 3D sub models for that area or point and providing the 3D model for each area or point based on the plurality of 3D sub-models. For example, the 3D model for each area or point is provided by averaging the point or area provided from the different sub-models. In one example, the images or image sets are associated to a weigh factor dependent on the quality of the image. The averaging can then be weighted. Finally, theprocessing unit 512 is arranged to compose the 3D model based on the 3D models related to the different areas or points. - The
processing unit 512 may be arranged to perform bundle adjustment. - The
processing unit 512 is further arranged to provide the 3D model also based on information from thedistance measurement device 510. Theprocessing unit 512 is in one example arranged to develop the 3D model based on the plurality of overlapping images and to update the model with the distance information from the distance measuring device, where appropriate. In detail, the 3D model covering a given area or environment is developed using the plurality of overlapping images. The 3D model is then compared with information from the distance measuring device. In those parts of the environment where the 3D model substantially coincides with the information provided from the distance measurements, the provided 3D model is regarded as finalized. However, in those parts of the 3D model where there is a discrepancy between the 3D model and the information from the distance measurements, the 3D model is updated in those parts, based on the distance information. The weight of the distance information in the updated model may be determined as described above and will be exemplified more in detail below. - The
processing unit 512 is arranged to verify that the 3D model parts updated based on the distance information better describes the reality as presented in the images than the corresponding 3D model not updated with the distance information. This is in one example performed by re-projecting one image to another image in those parts where the model has been updated so as to determine if the updated model gives a better re-projected estimated image than the not updated model. In detail, the not updated model and the updated model can be determined based on comparing an image I2 taken from one certain location with different estimated images Î2 determined for the same certain location. The estimated images are determined based on another image I1 taken from another location and projected in the not updated 3D model respectively the updated 3D model to the position of the location of the image I2. Thus the estimated image is determined as Î2=f(I1 M), wherein M represents the not updated 3D model respectively the updated model. In comparing the image I2 taken from the certain location with the estimated images Î2 that estimated image Î2 which is most similar to the original image I2 is associated to the best model. In one example, the images are compared or matched in small windows of the images. Thus, the images are compared on a sub-image by sub-image basis. In one example, the matching is performed based on a correlation technique. In one example, the matching is performed based on a phase based algorithm. In one example, the matching is performed based on a segmentation based algorithm. - Thus, if the model performed based only on the overlapping images provides the best image estimate for the image I2, then it is assumed that the distance information should not be used in the model or at least to a smaller extent than in the updated model. If on the other hand, the model developed based also on the distance information gives the best image estimate for the image I2, then it is assumed that the distance information enhances the 3D model.
- The
processing unit 512 may be arranged to verify in other ways known to the person skilled in the art that the 3D model updated based on the distance information better describes the reality as presented in the images than the 3D model not updated with the distance information. - In those parts of the 3D model where it has been determined that the distance information enhances the 3D model, the
processing unit 512 is arranged to repeat the development of a model based on only overlapping images. This time, the developing of the model is based on a selection of the overlapping images in which the coordinates of the updated model are visible. Theprocessing unit 512 is then arranged to compare the model with the distance information. If there is a difference between the model and the distance information at some point, then the 3D model is updated in this point as described above. The processing unit is then arranged to verify that the 3D model updated based on the distance information better describes the reality as presented in the images than the 3D model not updated with the distance information, as described above. If it is verified that the updated model better describes the environment, theprocessing unit 512 is arranged to repeat the development of the model based on a selection of the overlapping images and/or based on a selection of overlapping sub-images. If it is not verified that the updated model better describes the environment, the updating of the model may be finalized or repeated again using a decreased influence from the distance measurement. If for example, the distance information is wrong, such as if it has been measured against a flying bird, this distance information will not improve the model and will thus be rejected in the verification of the updated model. - The
processing unit 512 is in one example arranged to determine a difference between the developed model and a model developed in the previous step and to exit the development of the 3D model if the difference decreases a predetermined value. As is understood from above, theprocessing unit 512 may also be arranged to exit the development of the model if the distance information is determined not to improve the model. - The
processing unit 512 is in one example arranged to determine an uncertainty at some locations or at each location where LIDAR information is available. The processing unit is then arranged to determine the weight of the LIDAR measurement in the updated model based on the uncertainty in the 3D model at that specific location. In one example, the uncertainty in the model is determined based on the angle between the optical axis of the camera images used for modelling and a plane of the surface of the model at that specific location. For example, for surfaces of the 3D model which are perpendicular to an optical axis of the camera at that specific location, the uncertainty in the 3D model is lower than for surfaces which are substantially parallel to the optical axis of the camera. Further, points measured by the LIDAR device, which are not visible for any of the cameras since there are modelled objects which obscure this location, the uncertainty of the model may be regarded as high. Theprocessing unit 512 may further be arranged to determine an uncertainty related to the LIDAR measurement. The weight of the LIDAR measurement in the updated model is then determined based on the uncertainty in the 3D model and/or the uncertainty in the LIDAR measurement. - In one example, the 3D model is represented as a mesh. In an alternative example, the 3D model is represented as a surface representation. In an alternative example, the 3D model is represented as a voxel representation.
- The
processing unit 512 comprises in one example a computer program comprising a program code for developing a 3D model as discussed above. Further, a computer program product comprises a program code stored on a computer readable media for developing a 3D model of an environment. - In the shown example, the
arrangement 501 also comprises amemory 513 for storing data related to the three dimensional model calculated by theprocessing unit 512. The memory is also arranged to store information related to the overlapping images and the distance information. Thearrangement 501 further comprises a display orpresentation unit 514 arranged to provide information related to the 3D model. The display unit may be arranged to present a selected part of the 3D model. Thearrangement 501 may also comprise input means (not shown) for selecting a part of the 3D model and the display is arranged to present information related to the selected part. - The
arrangement 501 may also comprise a transmitter (not shown) arranged to transmit the information related to the 3D model to a receiver in a remote location. In one example, the transmitter at least partly substitutes thememory 513 and/or thedisplay unit 514. In an alternative example, the transmitter is provided in addition to thememory 513 and/or thedisplay unit 514. In one example, the arrangement does not comprise the camera(s) 504 and thedistance measuring device 510. Theprocessing unit 512 is then arranged to develop the model based on image and distance information stored in thememory 513. - In
FIG. 6 , a first part of a procedure for developing a 3D model of an environment is illustrated. A3D model 620 has been determined based on images taken from a plurality ofcamera locations view respective camera - The
3D model 620 is compared to LIDAR measurement points 621 a, 621 b, 621 c at corresponding locations. In the illustrated example, the 3D model and the LIDAR measurements are substantially coinciding at a first andthird measurement location LIDAR measurement point 621 b differs from a corresponding point in themodel 620. In the illustrated example, thesecond measurement point 621 b of the LIDAR measurement is formed at the ground of an alley between twobuildings 3D model 620 is drawn to at least one updated point between the model and thesecond measurement point 621 b as determined by a weight of the LIDAR measurement. In one example, the at least one updated point is selected such that the3D model 620 is drawn to thesecond measurement point 621 b. In one example, the at least one updated point is determined based on the weight of the LIDAR information. The weight of the LIDAR information is determined based on a predetermined scheme, as earlier discussed. - In one example the weight of the LIDAR measurement is also or instead determined based on an uncertainty in the 3D model. In one example, the uncertainty in the model is based on the angular relationship between the surface of the
model 620 at the location of thesecond measurement point 621 b and the optical axes of the cameras when taking the available images. If the surface and the optical axes are close to parallel, then uncertainty is greater than if the relation between the surface and the optical axes is close to perpendicular. - Further, in one example, the uncertainty in the model is instead or in addition thereto determined by from how many or for how high percentage of the camera images the LIDAR measurement points are visible. In this illustrated example, only a few of the camera images are taken from such location and in such direction that the LIDAR measurement point would be visible. The uncertainty is in one example then higher than if the LIDAR point is visible from a high percentage or substantially all the images used in the development of the 3D model. The 3D model point is determined to be moved towards the corresponding LIDAR measurement location to an extent determined based on the 3D model uncertainty. In
FIG. 7 , a second part of a procedure for developing a 3D model of an environment is shown. An updatedmodel 722 has been determined. The updated model is based on the model developed only using overlapping image pairs and updated with at least one updatedpoint 723 determined based on the distance information. As discussed above, the weight of the distance information has been determined according to a predetermined scheme The updatedmodel 722 is evaluated. This was also described in relation toFIG. 5 . If the updatedmodel 722 is considered to be better than the not updated model, then the model will be re-developed using only those images which shows the updated point(s) 723 of the 3D model provided based on the second measurement point (621 b inFIG. 6 ) and based on the weight of the LIDAR measurement. InFIG. 7 , acone 724 illustrates which images and/or sub-images may be used for the re-developed of the model. The procedure as described in relation toFIG. 6 is repeated using the images in the determined cone. Thus, a3D model 620 will be determined based on the images taken from a plurality ofcamera locations cone 724. Sub-images may also be used, In the illustrated example, a field ofview respective camera - In
FIG. 8 , it is illustrated an example wherein it would be useful that weight of the LIDAR measurement is determined based on an uncertainty in the LIDAR measurement. The pulse from aLIDAR device 810 used has abeam lobe 825 having a certain size in cross section. In the illustrated example, one pulse for determining a distance generates two different distance results due to the extent of thelobe 825 in its cross section. Thus, there is an uncertainty in the distance measurements at sharp edges or boarders such as buildings. In this case, the weight of the LIDAR information may be less than in areas where a high spatial resolution is not required in a plane perpendicular to the beam. - In
FIG. 9 , a model of a forest is developed. In this situation, the model produced by the images and the LIDAR measurements will provide different results. Amodel 920 provided based on overlapping images will be close to the tree tops and potentially the tree tops will not be modelled to their full extent. The LIDAR measurements will provide spread measurement points. Some measurement points will be in the tree tops and some measurement points will be close to ground. Thus situation can be handled in a plurality of ways. In one example, the LIDAR measurements are evaluated and it can be determined from the pattern of points that these points cannot be used in the modelling. Then those LIDAR measurement points which are not desirable to use can be removed. In one additional or alternative example, the LIDAR measurement points are used. Those which are close to the ground will be determined not to improve the model and will thus be rejected. Those LIDAR measurement points which are close to the tree tops can be used to model the tree tops more close to their full extension. In a third example, the pattern of the LIDAR measurement points are used for determining an uncertainty in the LIDAR information and thus can be used in modelling to an extent determined by the uncertainty in the LIDAR measurement. These different examples may be combined. - In
FIG. 10 an example of amethod 1000 for developing a 3D model based on overlapping images and distance information is illustrated. The method comprises providing 1010, overlapping images associated to navigation data. Further, distance information is also provided 1020 comprising a distance value and associated to navigation data. Thereafter, the 3D model is developed 1030 based on the plurality of overlapping images and the distance information. In developing the 3D model, the 3D model is provided 1031 based on the plurality of overlapping images, the provided 1031 model is compared 1032 with the distance information and the provided model is updated 1034 with the distance information using an iterative process, where appropriate. In one example, a weight relation between the distance measurement and the provided 3D model is determined 1033. The 3D model can then be updated 1034 with the distance information on the determined 1033 weight. The weight relation may for example be determined based on a percentage of a difference between the value from the distance measurement and the provided model. The weight relation can also or instead be determined based on an uncertainty in the provided 3D model. The weight relation may instead or in addition thereto be determined based on an uncertainty in the distance measurement. - In one example, the stereo distances are determined 1015 from the provided overlapping images. The model can then be provided based on the determined stereo distances. In one example, the stereo distances are determined from the overlapping images based on correlation. In one example, the stereo distances are determining from the overlapping images based on a phase based algorithm. In one example, the stereo distances are determined from the overlapping images is based on segmentation based algorithm. In one example, any other stereo method known to the person skilled in the art is used.
- The updated mode is then evaluated 1035 against the provided, not updated model to determine which of the models is most accurate. If the not updated model is determined to be most accurate, the development of the model may be finalized. If it is determined that the updated model is most accurate, or at least it is determined that the distance information does improve the model, the process is repeated. The
evaluation 1035 of the updated model against the provided model comprises in one example re-projecting the stereo images based on the updated and the not updated 3D models and comparing the results. - The selection of images is then updated 1036 based on the coordinates to the updated model. In one example, the updating 1036 of the selection of overlapping images comprises selecting only those images and/or sub-images which show parts of the environment relevant for the most accurate model (updated or not updated).
- In one example the decision of repeating 1037 development of the model comprises the steps of determining a difference between the model generated based on the overlapping images and a model generated based on the selection of overlapping in a previous step. The development of the 3D model can then be exited if the difference is below a selected level. In
FIG. 11 , a flow chart illustrating an example of amethod 1100 for developing a 3D model of an environment is shown. - In the illustrated example, a step of providing 1110 image data comprises the steps of providing 1111 a plurality of overlapping images of the environment, providing
navigation data 1112 and associating 1113 the navigation data to the respective images. The navigation data may comprise position information and a pointing direction of a camera capturing the images at each instant of capturing. Thus, navigation data may comprise information regarding position and orientation. It may also comprise information related to timing. - Further, as step of providing 1120 distance information comprises the steps of transmitting 1121 a plurality of pulses such as laser from a distance measuring device for reflection in the environment, receiving 1122 pulses reflected from the environment, providing 1123 navigation data related to pulses and determining 1124 information related to a relation between time of transmission and time of reception of each received pulse. The distance between the distance measuring device and the reflection point associated to each pulse can then be determined based on the determined relation. Then navigation data is associated 1125 to each determined distance. The navigation data may comprise position information and the direction of the transmitted pulses. Thus, navigation data may comprise information regarding position and orientation. It may also comprise information related to timing.
- In one example, the distance information is provided by means of LIDAR. In one example, the distance information is provided by means of radar. In one example, the distance information is provided by means of sonar.
- Thereafter the 3D model is developed 1130 based on the provided image data and based on the provided distance information. In one example, the image data and the distance measurements are provided using the same platform. It may then not be necessary to determine navigation data specifically to both the camera(s) and the distance measuring unit. It may then be enough to know the relation between the camera(s) and the distance measuring device. The 3D model may be represented as a surface representation and/or a voxel and/or a mesh.
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/791,501 US20180059247A1 (en) | 2013-01-21 | 2017-10-24 | Method and arrangement for developing a three dimensional model of an environment |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/SE2013/050044 WO2014112911A1 (en) | 2013-01-21 | 2013-01-21 | Method and arrangement for developing a three dimensional model of an environment |
US201514762174A | 2015-07-20 | 2015-07-20 | |
US15/791,501 US20180059247A1 (en) | 2013-01-21 | 2017-10-24 | Method and arrangement for developing a three dimensional model of an environment |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/SE2013/050044 Continuation WO2014112911A1 (en) | 2013-01-21 | 2013-01-21 | Method and arrangement for developing a three dimensional model of an environment |
US14/762,174 Continuation US9891321B2 (en) | 2013-01-21 | 2013-01-21 | Method and arrangement for developing a three dimensional model of an environment |
Publications (1)
Publication Number | Publication Date |
---|---|
US20180059247A1 true US20180059247A1 (en) | 2018-03-01 |
Family
ID=51209895
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/762,174 Active 2033-07-23 US9891321B2 (en) | 2013-01-21 | 2013-01-21 | Method and arrangement for developing a three dimensional model of an environment |
US15/791,501 Abandoned US20180059247A1 (en) | 2013-01-21 | 2017-10-24 | Method and arrangement for developing a three dimensional model of an environment |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/762,174 Active 2033-07-23 US9891321B2 (en) | 2013-01-21 | 2013-01-21 | Method and arrangement for developing a three dimensional model of an environment |
Country Status (4)
Country | Link |
---|---|
US (2) | US9891321B2 (en) |
EP (1) | EP2946365B1 (en) |
ES (1) | ES2693785T3 (en) |
WO (1) | WO2014112911A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2582554A (en) * | 2019-03-21 | 2020-09-30 | Rovco Ltd | Subsea surveying system |
Families Citing this family (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9547935B2 (en) * | 2014-02-24 | 2017-01-17 | Vricon Systems Ab | Method and a system for building a three-dimensional model from satellite images |
GB201412060D0 (en) | 2014-07-07 | 2014-08-20 | Vito Nv | Method and system for photogrammetric processing of images |
US9880263B2 (en) * | 2015-04-06 | 2018-01-30 | Waymo Llc | Long range steerable LIDAR system |
CN106289157B (en) * | 2015-06-12 | 2019-10-29 | 联想(北京)有限公司 | A kind of information processing method and electronic equipment |
EP3317852B1 (en) | 2015-06-30 | 2021-05-12 | Mapillary AB | Method in constructing a model of a scenery and device therefor |
US11846733B2 (en) * | 2015-10-30 | 2023-12-19 | Coda Octopus Group Inc. | Method of stabilizing sonar images |
EP3182157B1 (en) | 2015-12-14 | 2020-04-15 | Leica Geosystems AG | Method for creating a spatial model with a hand-held distance measuring device |
US20180364741A1 (en) * | 2016-03-11 | 2018-12-20 | Raytheon Bbn Technologies Corp. | Human indication of target drone for interception |
US20170350968A1 (en) * | 2016-06-06 | 2017-12-07 | Goodrich Corporation | Single pulse lidar correction to stereo imaging |
US20180100927A1 (en) * | 2016-10-12 | 2018-04-12 | Faro Technologies, Inc. | Two-dimensional mapping system and method of operation |
SE543568C2 (en) * | 2017-02-27 | 2021-03-30 | Katam Tech Ab | Improved forest surveying apparatus and method using video sequences to generate 3D models and for height determination |
SE541287C2 (en) * | 2017-02-27 | 2019-06-11 | Katam Tech Ab | Forest surveying apparatus and method using video sequences to generate 3D models |
JP6854195B2 (en) * | 2017-06-09 | 2021-04-07 | 株式会社トプコン | Image processing device, image processing method and program for image processing |
JP6939195B2 (en) * | 2017-07-27 | 2021-09-22 | 株式会社大林組 | Inspection processing system, inspection processing method and inspection processing program |
DE102017117049A1 (en) * | 2017-07-27 | 2019-01-31 | Testo SE & Co. KGaA | Method for creating a 3D model of an object |
KR20210104033A (en) * | 2018-11-21 | 2021-08-24 | 광저우 엑스에어크래프트 테크놀로지 씨오 엘티디 | Planning method, apparatus, control terminal and storage medium of survey and mapping sample points |
ES2894046T3 (en) | 2019-09-17 | 2022-02-11 | Maxar Int Sweden Ab | Enhance resolution of aerial images or satellite images |
CN111133477B (en) * | 2019-12-20 | 2023-06-23 | 驭势科技(浙江)有限公司 | Three-dimensional reconstruction method, device, system and storage medium |
GB2595246A (en) * | 2020-05-19 | 2021-11-24 | Airbus Defence & Space Ltd | Photogrammetry |
EP4102467A1 (en) | 2021-06-08 | 2022-12-14 | Maxar International Sweden AB | A method for 3d reconstruction from satellite imagery |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6757445B1 (en) * | 2000-10-04 | 2004-06-29 | Pixxures, Inc. | Method and apparatus for producing digital orthophotos using sparse stereo configurations and external models |
US8994822B2 (en) * | 2002-08-28 | 2015-03-31 | Visual Intelligence Lp | Infrastructure mapping system and method |
US7424133B2 (en) * | 2002-11-08 | 2008-09-09 | Pictometry International Corporation | Method and apparatus for capturing, geolocating and measuring oblique images |
WO2006083297A2 (en) | 2004-06-10 | 2006-08-10 | Sarnoff Corporation | Method and apparatus for aligning video to three-dimensional point clouds |
US7697748B2 (en) | 2004-07-06 | 2010-04-13 | Dimsdale Engineering, Llc | Method and apparatus for high resolution 3D imaging as a function of camera position, camera trajectory and range |
WO2009003529A1 (en) | 2007-07-04 | 2009-01-08 | Saab Ab | Arrangement and method for providing a three dimensional map representation of an area |
US8284190B2 (en) * | 2008-06-25 | 2012-10-09 | Microsoft Corporation | Registration of street-level imagery to 3D building models |
US9185361B2 (en) * | 2008-07-29 | 2015-11-10 | Gerald Curry | Camera-based tracking and position determination for sporting events using event information and intelligence data extracted in real-time from position information |
US8427505B2 (en) * | 2008-11-11 | 2013-04-23 | Harris Corporation | Geospatial modeling system for images and related methods |
US20100204974A1 (en) | 2009-02-09 | 2010-08-12 | Utah State University | Lidar-Assisted Stero Imager |
US20100204964A1 (en) * | 2009-02-09 | 2010-08-12 | Utah State University | Lidar-assisted multi-image matching for 3-d model and sensor pose refinement |
GB0908200D0 (en) | 2009-05-13 | 2009-06-24 | Red Cloud Media Ltd | Method of simulation of a real physical environment |
US9208612B2 (en) * | 2010-02-12 | 2015-12-08 | The University Of North Carolina At Chapel Hill | Systems and methods that generate height map models for efficient three dimensional reconstruction from depth information |
US8494687B2 (en) * | 2010-03-12 | 2013-07-23 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | Method for enhancing a three dimensional image from a plurality of frames of flash LIDAR data |
US8718393B2 (en) * | 2010-07-28 | 2014-05-06 | Shenzhen Institutes Of Advanced Technology Chinese Academy Of Sciences | Method for reconstruction of urban scenes |
US9229106B2 (en) | 2010-08-13 | 2016-01-05 | Ryan Dotson | Enhancement of range measurement resolution using imagery |
US8736818B2 (en) * | 2010-08-16 | 2014-05-27 | Ball Aerospace & Technologies Corp. | Electronically steered flash LIDAR |
-
2013
- 2013-01-21 EP EP13871359.9A patent/EP2946365B1/en not_active Not-in-force
- 2013-01-21 ES ES13871359.9T patent/ES2693785T3/en active Active
- 2013-01-21 WO PCT/SE2013/050044 patent/WO2014112911A1/en active Application Filing
- 2013-01-21 US US14/762,174 patent/US9891321B2/en active Active
-
2017
- 2017-10-24 US US15/791,501 patent/US20180059247A1/en not_active Abandoned
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2582554A (en) * | 2019-03-21 | 2020-09-30 | Rovco Ltd | Subsea surveying system |
GB2582554B (en) * | 2019-03-21 | 2021-04-21 | Rovco Ltd | Subsea surveying system |
Also Published As
Publication number | Publication date |
---|---|
US9891321B2 (en) | 2018-02-13 |
EP2946365B1 (en) | 2018-08-08 |
EP2946365A4 (en) | 2016-09-21 |
WO2014112911A1 (en) | 2014-07-24 |
US20150362595A1 (en) | 2015-12-17 |
EP2946365A1 (en) | 2015-11-25 |
ES2693785T3 (en) | 2018-12-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20180059247A1 (en) | Method and arrangement for developing a three dimensional model of an environment | |
US7616817B2 (en) | Three dimensional shape correlator | |
AU2014247986B2 (en) | Underwater platform with lidar and related methods | |
US8655513B2 (en) | Methods of real time image enhancement of flash LIDAR data and navigating a vehicle using flash LIDAR data | |
US11335059B2 (en) | Method and arrangement for providing a 3D model | |
CN104380133B (en) | 3-D scanning beam and imaging system | |
US9709673B2 (en) | Method and system for rendering a synthetic aperture radar image | |
US20120155744A1 (en) | Image generation method | |
US20120033196A1 (en) | Method for Enhancing a Three Dimensional Image from a Plurality of Frames of Flash Lidar Data | |
RU2591875C1 (en) | Method of constructing map of exogenous geological processes of area along route of main oil line | |
JP5762131B2 (en) | CALIBRATION DEVICE, CALIBRATION DEVICE CALIBRATION METHOD, AND CALIBRATION PROGRAM | |
US20090008554A1 (en) | Method for infrared imaging of living or non-living objects including terrains that are either natural or manmade | |
Wagner et al. | Long-range geo-monitoring using image assisted total stations | |
US20220049956A1 (en) | Method for water level measurement and method for obtaining 3d water surface spatial information using unmanned aerial vehicle and virtual water control points | |
CN112461204B (en) | Method for satellite to dynamic flying target multi-view imaging combined calculation of navigation height | |
EP2948791B1 (en) | Improved laser range finding | |
US11747141B2 (en) | System and method for providing improved geocoded reference data to a 3D map representation | |
US20230045287A1 (en) | A method and system for generating a colored tridimensional map | |
US20230141795A1 (en) | Method for georeferencing of optical images | |
Elshorbagy | A Crosscutting Three-Modes-Of-Operation Unique LiDAR-Based 3D Mapping System Generic Framework Architecture, Uncertainty Predictive Model And SfM Augmentation | |
Eboigbe et al. | Assessment of the precision of a smart-phone pole photogrammetry for a second-order cliff surface deformation studies | |
KR101782299B1 (en) | Method for inspecting gas facilities | |
Ravi | Interactive Environment for the Calibration and Visualization of Multi-Sensor Mobile Mapping Systems | |
Sardemann | Conception and realisation of a photogrammetric multisensor system for an uncrewed water vehicle | |
JP2024021131A (en) | Posture angle correction device and posture angle correction method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAAB VRICON SYSTEMS AB, SWEDEN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ISAKSSON, FOLKE;ANDERSSON, INGMAR;BEJERYD, JOHAN;AND OTHERS;REEL/FRAME:043931/0569 Effective date: 20150810 Owner name: VRICON SYSTEMS AKTIEBOLAG, SWEDEN Free format text: CHANGE OF NAME;ASSIGNOR:SAAB VRICON SYSTEMS AB;REEL/FRAME:045372/0247 Effective date: 20150601 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |