WO2013142819A1 - Systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces - Google Patents

Systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces Download PDF

Info

Publication number
WO2013142819A1
WO2013142819A1 PCT/US2013/033559 US2013033559W WO2013142819A1 WO 2013142819 A1 WO2013142819 A1 WO 2013142819A1 US 2013033559 W US2013033559 W US 2013033559W WO 2013142819 A1 WO2013142819 A1 WO 2013142819A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
dimensional
image
point
points
Prior art date
Application number
PCT/US2013/033559
Other languages
French (fr)
Inventor
Christopher Richard Sweet
James Christopher SWEET
Original Assignee
University Of Notre Dame Du Lac
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University Of Notre Dame Du Lac filed Critical University Of Notre Dame Du Lac
Publication of WO2013142819A1 publication Critical patent/WO2013142819A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/04Texture mapping
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/08Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00Indexing scheme for image generation or computer graphics
    • G06T2210/56Particle system, point based geometry or rendering

Definitions

  • the present disclosure relates generally to computer visualization and more particularly to systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces.
  • Texturing three-dimensional (3D) surfaces with two-dimensional (2D) graphics is commonplace in the field of computer visualization.
  • visualization is a rapidly advancing field which provides immersive environments that can be enhanced by employing 3D viewing hardware.
  • the use of such environments is now commonplace in technical and scientific fields where the correct interpretation of data is vital in both diagnostics and extending the frontiers of knowledge.
  • Such fields generally require geometrically correct rendering of data in contrast to many areas of the entertainment industry, and hence specialist techniques are required.
  • a point-cloud is a set of vertices in a three-dimensional coordinate system. These vertices are usually defined by X, Y, and Z coordinates, and typically are intended to be representative of the internal and external surface of an object. Point-clouds are most often created by 3D scanners. These devices measure in an automatic way a large number of points on the surface of an object, and often output a point-cloud as a data file. The point- cloud represents the set of points that the device has measured.
  • point-clouds are used for many purposes, including to create 3D CAD models for manufactured parts, metrology/quality inspection, post-acquisition analysis (e.g., engineers and/or scholars) and a multitude of visualization, animation, rendering and mass customization applications.
  • US Patent No. 5,412,765 describes a method for visualizing vector fields uses texture mapping for the vectors to create an animated vector field.
  • the described methods utilize time lapse as a component.
  • vectors are textured with one dimensional texture maps composed of alternating visible and invisible segments.
  • Successively applied texture maps differ from each other in order to create a moving line effect on the vectors being visualized.
  • a fading effect is further provided by varying the intensity of the visible segments.
  • US Patent No. 7,689,019 describes a method for registering a 2D projection image of an object relative to a 3D image data record of the same object, in which, from just a few 2D projection images, a 3D feature contained in an object, which is also identifiable in the 3D images, is symbolically reconstructed.
  • the 3D feature obtained in this way is then registered by 3D-3D registration with the 3D image data record.
  • the methods described identify three features on the target structure, form, surface, and centerline.
  • FIG. 1 is a block diagram illustrating components of an example network system in which the disclosed systems and methods may be employed.
  • FIG. 2A shows an example architectural 3D point-cloud.
  • FIG. 2B shows an example 2D image corresponding to the example 3D point-cloud of FIG. 2A.
  • FIG. 3 illustrates an example process diagram for mapping an example 2D image to an example 3D point-cloud.
  • FIG. 4 illustrates an example block diagram of sample calculations utilized in the process of FIG. 3.
  • FIG. 5 illustrates an example process which allows for the blending of image date and the removal of visual artifacts.
  • FIG. 6 illustrates an example of the determination of surface 3D point-cloud data from general 3D data.
  • FIG. 7 A illustrates an example image before any normalization and cleaning process.
  • FIG. 7B illustrates the example image of FIG. 7A after an example normalization and cleaning process.
  • FIG. 8A illustrates an example image before compression.
  • FIG. 8B illustrates the example image of FIG. 8A after an example compression process.
  • FIG. 9 illustrates an example image showing the partial rendering using an example internal camera from a scanner.
  • FIG. 10 is an example plot illustrating the remaining points after reduction using a center comparison process.
  • FIG. 1 1 is an example plot illustrating the remaining points after reduction using a two norm comparison process.
  • FIG. 12A illustrates an example 3D point-cloud of an architectural object.
  • FIG. 12B illustrates the resultant 3D point-cloud of FIG. 12A rendered as a surface utilizing the example methods of the present invention.
  • FIG. 13A illustrates an example 3D point-cloud of a CT Scan of an object.
  • FIG. 13B illustrates the resultant 3D point-cloud of FIG. 13A rendered as a surface utilizing the example methods of the present invention.
  • the present disclosure provides methods of determining the geometrically correct registration between a 2D image, or images, and the underlying 3D surface associated with them to allow automatic alignment of, for example, many images to many point-clouds.
  • the requirement arises from technical and scientific fields, such as architecture and/or historical preservation, where, for example, a measurement from the underlying 3D data needs to be determined by observation of surface data.
  • the present disclosure facilitates the combination of photographic data from multiple viewpoints with 3D data from laser scanners or Computer Aided Design (CAD) computer programs, to generate immersive environments where the user can query the underlying data by visual interpretation.
  • CAD Computer Aided Design
  • Another example allows the combination of surface data, including internal surfaces, for biological specimens with CT scan data for diagnostic or educational purposes.
  • the present disclosure allows for mapping of multiple images acquired from robotic tripods, such as a GigaPan tripod with a camera attached, to scanned data from scanners such as the Leica CIO.
  • the present disclosure maps images from camera enabled scanners to point-clouds.
  • 3D data such as point-cloud data
  • 2D surface data which may include internal surfaces, such that the registration between the data sets, and hence the validity of observations and measurements based on them, is determined geometrically and maintained over the surface of interest.
  • the optimal set can be determined for each rendered pixel set.
  • visual artifacts generally present with adjacent multiple images can be reduced.
  • the surface images do not have to be produced from fixed viewpoints in relation to the 3D representation, or map directly to individual 3D features of the object, allowing the acquisition of data from the architectural to nano scales. This is particularly important from complex surfaces, for example those generated from biological specimens.
  • 3D points are culled from individual point-clouds, where a number of point-clouds are used, to reduce or eliminate overlap for optimal texturing.
  • 3D points are culled from individual point-clouds to remove artifacts where the distance between adjacent points is large, indicating a real spatial separation.
  • 3D points are culled to reduce their number, and hence the number of triangles, for efficiency, such that important features (such as edges) are preserved.
  • 3D data is combined with 2D surface images acquired from the same device, mapping images from camera enabled scanners to point- clouds.
  • mapping images from camera enabled scanners to point- clouds In another example of the present disclosure there is described a process of mapping multiple images acquired from robotic tripods, such as a GigaPan robotic tripod, to 3D scan data.
  • the systems and methods disclosure herein also facilitates the combination of photographic data with 3D data from laser scanners to generate immersive environments where the user can query the underlying data by visual interpretation.
  • the examples disclosed have found particular use in, but are not restricted to, architectural and biological diagnostic fields.
  • a processing device 20 illustrated in the exemplary form of a computer system, is provided with executable instructions to, for example, provide a means for an operator, i.e., a user, to access a remote processing device, e.g., a server system 68, via the network to, among other things, distribute processing power to map 2D images to 3D surfaces.
  • a remote processing device e.g., a server system 68
  • the computer executable instructions disclosed herein reside in program modules which may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Accordingly, those of ordinary skill in the art will appreciate that the processing device 20 may be embodied in any device having the ability to execute instructions such as, by way of example, a personal computer, mainframe computer, personal-digital assistant ("PDA"), cellular or smart telephone, tablet computer, or the like.
  • PDA personal-digital assistant
  • the processing device 20 preferably includes a processing unit 22 and a system memory 24 which may be linked via a bus 26.
  • the bus 26 may be a memory bus, a peripheral bus, and/or a local bus using any of a variety of bus architectures.
  • the system memory 24 may include read only memory (ROM) 28 and/or random access memory (RAM) 30. Additional memory devices may also be made accessible to the processing device 20 by means of, for example, a hard disk drive interface 32, a magnetic disk drive interface 34, and/or an optical disk drive interface 36.
  • these devices which would be linked to the system bus 26, respectively allow for reading from and writing to a hard disk 38, reading from or writing to a removable magnetic disk 40, and reading from or writing to a removable optical disk 42, such as a CD/DVD ROM or other optical media.
  • the drive interfaces and their associated non-transient, computer- readable media allow for the nonvolatile storage of computer readable instructions, data structures, program modules, and other data for the processing device 20.
  • Those of ordinary skill in the art will further appreciate that other types of non-transient, computer readable media that can store data may be used for this same purpose. Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories, nano-drives, memory sticks, and other read/write and/or read-only memories.
  • a number of program modules may be stored in one or more of the memory/media devices.
  • a basic input/output system (BIOS) 44 containing the basic routines that help to transfer information between elements within the processing device 20, such as during start-up, may be stored in ROM 28.
  • the RAM 30, hard drive 38, and/or peripheral memory devices may be used to store computer executable instructions comprising an operating system 46, one or more applications programs 48, other program modules 50, and/or program data 52.
  • computer-executable instructions may be downloaded to one or more of the computing devices as needed, for example, via a network connection.
  • the operator may enter commands and information into the processing device 20, e.g., a textual search query, a selection input, etc., through input devices such as a keyboard 54 and/or a pointing device 56. While not illustrated, other input devices may include a microphone, a joystick, a game pad, a scanner, a camera, etc. These and other input devices would typically be connected to the processing unit 22 by means of an interface 58 which, in turn, would be coupled to the bus 26. Input devices may be connected to the processor 22 using interfaces such as, for example, a parallel port, game port, Firewire, a universal serial bus (USB), etc.
  • USB universal serial bus
  • a monitor 60 or other type of display device may also be connected to the bus 26 via an interface, such as a video adapter 62.
  • the processing device 20 may also include other peripheral output devices, not shown, such as speakers, printers, 3D rendering hardware such as 3D glasses to separate right-left vision and autosteroscopic screens, etc.
  • the processing device 20 may also utilize logical connections to one or more remote processing devices, such as the server system 68 having one or more associated data repositories 68A, e.g., a database.
  • the server system 68 has been illustrated in the exemplary form of a computer, it will be appreciated that the server system 68 may, like processing device 20, be any type of device having processing capabilities. Again, it will be appreciated that the server system 68 need not be implemented as a single device but may be implemented in a manner such that the tasks performed by the server system 68 are distributed to a plurality of processing devices linked through a communication network. Additionally, the server system 68 may have logical connections to other third party server systems via the network 12 and, via such connections, will be associated with data repositories and/or functionalities that are associated with such other third party server systems.
  • the server system 68 may include many or all of the elements described above relative to the processing device 20.
  • the server system 68 includes executable instructions stored on a non-transient memory device for, among other things, handling mapping requests, etc. Communications between the processing device 20 and the server system 68 may be exchanged via a further processing device, such as a network router, that is responsible for network routing.
  • Communications with the network router may be performed via a network interface component 73.
  • a networked environment e.g., the Internet, World Wide Web, LAN, or other like type of wired or wireless network
  • program modules depicted relative to the processing device 20, or portions thereof may be stored in the memory storage device(s) of the server system 68.
  • FIGS. 2A and 2B An example of an architectural 3D point-cloud 10 and corresponding 2D image 12 is illustrated in FIGS. 2A and 2B, respectively.
  • a set of processes describe a method for geometrically determining the correct registration between a 2D image, or images, and the underlying 3D surface associated with the subject image(s).
  • a process for mapping of a single image to a 3D surface point-cloud as well as extending to multiple images from different viewpoints.
  • there are described processes for extracting surface 3D point-clouds from general 3D data such as for example, point-clouds, CT scans, or the like.
  • the following disclosure assumes that the user has a 3D point-cloud of surface points i.e., a collection of points in 3 dimensional space representing a surface which may be internal to the object, and a single 2D image which overlaps with the 3D data.
  • the presently described systems and methods also assume the position of the camera is known in relation to the point-cloud. It will be appreciated, however, that the position of the camera need not be known in relation to the point-cloud and may be calculated and/or otherwise determined utilizing any suitable calculation method.
  • Use of the present disclosure with multiple images and the generation of surface point-clouds from data, such as general point-clouds, CT scans, etc., will be discussed as well.
  • the 3D point-cloud data may be obtained from any suitable source, including for instance, a scanner such as a Leica scanner available from Leica Geosystems AG, St. Gallen, Switzerland (for architectural use).
  • the 2D image may be selected from any suitable imaging source, such as an image from a set obtained using a GigaPan robotic tripod in association with an SLR camera, a high-resolution imaging system.
  • FIGS. 3 and 4 in one example of the present disclosure, a process 300 for mapping a 2D image to a 3D point-cloud is illustrated.
  • processing starts at a block 302, wherein two coincident points between the 2D image and the 3D point-cloud are determined.
  • a 3D point-cloud 400 in a 3D point-cloud 400, two points, A and B in the 3D data 400, and the 2D image are determined. While not necessarily limiting, for his example, we will assume point A is closer to the center of the image 400. As illustrated, the 'noses' of the two statues are used.
  • the process 300 will continue with a block 304, wherein the process 300 finds the viewpoint.
  • the process 304 finds the vector from the camera origin O (see reference 402, FIG. 4) to point A, denoted as vector C. Without loss of generality the process 300 assumes that point O is at the system origin.
  • the example plane P (406) is constructed, at a distance 1 from the camera origin
  • the process 300 finds the Y-axis in the viewing plane at a block 310.
  • the process defines a "vertical" plane VP containing the vector C, and finds the line Y where this plane intersects the plane P.
  • the process 300 also finds the X-axis in the viewing plane at a block 312.
  • each of the points PPi are projected onto the x and y axes to determine their 'x-y' coordinates PPXi and PPYi. This may be determined through the equation:
  • the process 300 finds the scale factors and offsets for the 2D image. For example, the process 300 calculates a scale factor S and the 'x-y' offsets X 0ff and Y 0ff for the 2D image. Given that points A and B on the 3D point-cloud 400 correspond to points A and B in the plane P, and assuming these points on the 2D image have 'x-y' coordinates (a x , a y ) and (b x , b y ), respectively, the process 300 can determine that
  • the process calculates the actual 'x-y' coordinates for all points at a block 318.
  • the process 300 may utilize the equation:
  • the process 300 may then create a 3D surface and texture using the 2D coordinates corresponding to each 3D point at a block 322.
  • the surface texturing may be accomplished through any suitable procedure, including, for example, the procedure disclosure in Table 1.
  • the process 300 may draw a single OpenGL Quad (see Table 1) textured by one quadrilateral from the image (consisting of four adjacent points in the point-cloud). It will be understood that while the present example is provided in openGL, any suitable process may be utilized as desired.
  • the top left point on the Quads 3D point-cloud is P2, the bottom left P 10, the bottom right PI 1 and the top right P3, with corresponding actual 'x-y' coordinates of the image (ppx_2, ppy_2), (ppx lO, ppy_10), (ppx lO, ppy_10) and (ppx_3, ppy_3) (for ease of representation, the nomenclature has changed from ppxi to ppx i in the example of Table 1).
  • the vector "normal” represents a normal vector to the Quad for the calculation of lighting effects and can be found as:
  • the process 500 first calculates the areas for each corresponding image section, generally a triangle or quadrilateral, for the adjacent 3D points of interest at a block 502.
  • the area A can be calculated for a triangle with vertices V0, VI, V2 as
  • the process 500 may also select the image section with the largest area after adding a random perturbation to the area. In this instance, the process 500 allows 'blending' of the images to prevent artifacts such as image interface lines.
  • the process 500 uses the section from image else the process uses the section from Ii, for random variable R, which may preferably be determined with zero mean and known distribution.
  • the process 500 may select the image section with the largest area up to some error E, and if the areas are within some chosen error E, merge the image data during texturing. This process allows the 'blending' of the images to prevent artifacts such as image exposure level/color differences.
  • the process 500 uses the section from I 2
  • Ai > A 2 + E the process 500 uses the section from Ii.
  • the process 500 blends the sections from Ii and I 2 . Blending can be accomplished utilizing any suitable blending procedure, including, for example, an example OpenGL as shown in Table 2.
  • Some data may contain general 3D coordinates not restricted to surface points of the object.
  • CT scan computed tomography
  • the 3D surface point-cloud can be extracted from the data utilizing a process 600 as illustrated in FIG. 6. It will be appreciated that the process 600 assumes the camera position and viewpoint is known, but it will be understood that in various examples, the camera position and/or viewpoint can be determined as needed by one of ordinary skill in the art.
  • the viewpoint plane is first defined at a block 602.
  • a plane P is constructed with the camera viewpoint (e.g, a vector describing the viewing direction of the camera) vector A as a normal plane.
  • the camera viewpoint e.g, a vector describing the viewing direction of the camera
  • A is also unit normal vector to the plane.
  • the example process 600 defines the Y-axis at a block 604.
  • the process 600 performs a step of determining the x-axis at a block 606.
  • the x-axis may be determined through the rotation of the line y through 90 degrees clockwise in the plane P around the point where the vector A intersects the plane P.
  • This line X is the x-axis.
  • the process 600 continues with the construction of an array of 'x-y' points on the plane at a block 608.
  • the array is constructed with points on the plane given by the equation:
  • the process 600 may then generate an array of lines from the viewpoint through each array point in the plane at a block 610. In this step, for each Gij, the process 600 generates a line Ly consisting of points given by the equation:
  • the process 600 can find the surface of the 3D point-cloud at a block 612.
  • the process 600 finds the first intersection of each line Ly with the model data (such as non-zero CT scan data) and/or finds the points ,t in the 3D point-cloud set which are within some distance E of each line Ly but closest to the origin O.
  • This data represents the surface point-cloud with respect to the viewpoint (e.g., the camera view).
  • the distance d from a point x to a line Ly is given by the equation:
  • process 600 is described as being well suited to find surface point-cloud date, the process 600 can also be utilized and extended to finding surface features within the object for such object as CT scans by considering the density with the scan itself.
  • scanning complex structures can generate artifacts where adjacent points in the scanner data have significant difference in distance from the scanner. In the case of multiple scans this can lead to important surface data being hidden behind 'stretched' data from overlapping scans. For correct texture registration may be vital that these artifacts be removed.
  • FIGS. 7A and 7B there are illustrated examples of artifacts generated by near and far objects appearing adjacent to each other in the point-cloud.
  • an image 700 appears before any normalization and cleaning process.
  • an image 710 appears after the triangle areas are calculated and normalized and removed if they are larger than a given threshold.
  • point-clouds acquired by scanning can oftentimes be extremely large.
  • To distribute merged data to computers 20 and mobile devices 20' via the network 14 typically requires that the data be compressed.
  • the present disclosure includes example processes that can reduce the number of triangles (or points) in an "intelligent" manner such that important structural features, such as edges or relief features, are retained.
  • FIGS. 8A and 8B an image 800 shows an non-compressed image, while an image 810 shows an example of compression by removal of points while retaining important structural features such as edges.
  • the described methods are efficient with a computationally linear time complexity.
  • mapping the image data acquired from camera enabled scanners requires the solution to a geometric mapping and scaling algorithm.
  • the present disclosure provided herein includes processes to accomplish this automatically. This contrasts with current methods where each image is manually aligned with the point-cloud. As such, the manual approach found in prior art is impractical for multiple scans each with many images, such as for example, a Leica Scanstation II, which can oftentimes produce more than 100 images for a full 360 degree scan.
  • An example image 900 showing the partial rendering of the Arch of Septimius Severus at the Roman Forum is illustrated in FIG. 9.
  • the current disclosure assumes that the points are ordered in vertical (Z axis) lines with the zero vector representing points that are not included in the set (i.e. those removed by clipping). If this is not the case then the points must be ordered prior to analysis.
  • Table 3 describes one example process used to find triangles that make up the surface to be textured. The example of Table 3 loops over all of the points and, for each point, tests the three other points adjacent to it to make a square (two triangles). If one point is missing we can form a single triangle and if more than one point is missing then no triangles are added.
  • the present examples calculate the average compensated size of a triangle and then prune all triangles that are greater than a threshold.
  • This threshold is, in one example, based off of the average compensated size and can be tuned to an individual model if required to achieve the greatest quality end result.
  • the compensated area is calculated as the area divided by the distance from the scanner. In some situations it may be desirable to use the reduction algorithm without using compensation. In one instance, given points A, B, C representing the vertices of the triangle it can be determined that the area by forming vectors AB and AC, then the magnitude of the cross product AB x AC is twice the area of the triangle. To compensate for distance it may be divided by the magnitude of A.
  • a process can be utilized to remove redundant points from the data set whilst keeping important features, such as edges.
  • This process has time complexity of 0(N).
  • the process can test four squares of increasing sizes to see if all of the points within the four squares face the same direction. If the points do face the same way within a threshold, the four squares can be reduced into a single square. To test if the points in the four squares face the same way, the process calculates the average normal vector for each point and then compares the center point to each point via a comparison metric. [0080] To test if a set of points is reducible, two comparison metrics are produced. The first comparison metric compares the angle between the center point and all eight surrounding points.
  • This method provides the most accurate final representation, as it does not remove any sets of points that do not all pass the test.
  • This method provides reduction in triangles as shown in a plot 1000 illustrated in FIG. 10, which illustrates the remaining points after reduction where a center comparison is applied.
  • the second comparison metric compares the square root of the sum of the squares of all of the angles between the center point and all eight surrounding points. If this value is less than the threshold then the set can be reduced. This provides a less accurate
  • FIG. 1 shows the remaining points after reduction where two norm comparisons are applied.
  • step 2 whilestep ⁇ max(Horizontal,Vertical):
  • a system and method of automatically mapping images generated by camera enabled scanners, to their associated point-clouds incorporate a camera to provide pseudo images by coloring the point-cloud.
  • the present disclosure may be used to map these images to the point-cloud to allow a more seamless rendering than can be obtained by colored points.
  • An additional benefit is provided by allowing an intelligent point reduction algorithm to be used without degrading the rendered scene, which could not be used with the colored point representation without serious visual degradation.
  • a "focal sphere” that represents a sphere which contains the points at which the center of each image will reside (i.e. with the plane of the image tangental to the sphere), and where the image scaling is coincident with the scaling of the point-cloud data from the point of view of the camera.
  • the "focal sphere” is related to the camera FOV and is typically a fixed value.
  • fs is a scalar that scales the vector in the preceding terms.
  • the example process finds x and y orthogonal axes in the plane tangential to the "focal sphere" at the location of the image center.
  • V p VR zyx [ J ) Eq. 30 where the bracketed term is a scalar that scales the vector in the preceding terms.
  • the process finds the orthogonal axes in the plane tangential to the Focal Sphere. For example, the process finds x and y orthogonal axes as:
  • the example process then associates triangles with an image and calculate the corresponding texture coordinates for a surface, in particular, for a given point Pi on a triangle, and camera offset C 0 , the process can calculate a texture coordinate Ti as:
  • the example process then tests if either of the x, y components of Ti are less than 0 or greater than 1. In this event the point is outside the image and is not included in the surface. The test continues for all points in the triangle, if all points pass then they are stored with their texture coordinates in the surface and removed from the available points/triangle list. This process is then repeated for n images to produce n surfaces as described, for example, in the example listing in Table 8.
  • Offset Vector3(TextureData[3], TextureData[4], TextureData[5])
  • x_angle -Angle. toRadians(TextureData[0])
  • triangleOK False for i in range(len(Meshlndex)):
  • texlnt point * (sqrtaa / IbDotN)
  • a number of the currently available laser scanners for example the Leica CIO, are fitted with an internal camera with which the points in the point-cloud can be colored to produce a pseudo-photographic effect.
  • the resulting "image" is limited to the resolution of the scan rather than the image, and hence the user cannot observe the detail provided by traditional photographic sources.
  • the scanner would be required to scan sixteen million points in the area covered by a picture.
  • this equates to a resolution of around 0.2mm at a distance of 20m, orders of magnitude better than current scanner technology.
  • Images from internal cameras are typically acquired through the mirror system used to launch the laser beam.
  • Post processing software such as Leica Cyclone, can output not only the images, but also the metadata that describes the orientation and offset of the camera. For the Leica CIO camera the attitude is arranged in a gimbal format i.e.
  • the effective scaling can be determined from either the field of view (FOV) of the camera, or by matching some non-central image point to the point-cloud.
  • FOV field of view
  • the invention described herein we find the "focal sphere" where the center point of each image must touch (and the image plane is then tangential to) in order to be projected onto the points in the point-cloud with the correct scaling. For the example Leica Scanstation II described herein, this was found to be a sphere of diameter 2.2858996m.
  • the process may be utilized to map multiple images acquired from robotic tripods, such as a GigaPan, to scan data.
  • this process may be achieved in three steps; finding the 'focal sphere' where a tangential image has unity dimension, calculating the image center and normal vectors, and calculating the spherical coordinates of each image.
  • the example process first finds the "focal sphere" radius, fp, representing the distance from the point of acquisition of the images to a sphere where the image size would be unity. For far objects this would be equal to the lens focal length F divided by the image sensor vertical size d.
  • fp the "focal sphere" radius
  • F can be found from the image EXIF information, e.g. 300mm for a typical telephoto lens. For a full frame DSLR camera d will be around 24mm. SI is found by taking the norm of one of the object 3D points.
  • the process defines the vector to the center of the image as PC with length fp, then
  • the center and normal vectors are calculated given two arbitrary points on the image.
  • the process can find the center vector by solving the equation set
  • the center and normal vectors are calculated given one vector and the focal sphere.
  • the process first finds the vector in the image plane P AP coinciding with the known vector PA
  • A is the coordinate length in the image plane for PA.
  • the approximate relative orientation is determined with one vector, given the focal sphere.
  • the focal sphere fp the elevation of one of the points PA or PB and the y coordinate of that point on the image the process can determine the z and ratio x/y of the center point on the image.
  • PC Z sin(e PA - e DIFF ) fp Eq. 62
  • IM y is the y coordinate of the vector PA on the image.
  • the process can then calculate the image sphere coordinates.
  • the process can calculate the spherical coordinates of the main image normal vector.
  • the azimuth ( ⁇ ) and elevation ( ⁇ ) can be determined by:
  • the process can take account of the increase in horizontal overlap with elevation angle. Because the overlap distance is proportional to the radius of the circle that represents the sphere cross section at the given elevation, fp cos(9) for sphere radius fp, it can be determined that:
  • Vrt 2 tan -1 (—— -— ) .
  • Eq. 71 [00112] Given the spherical coordinates for the center of each image, relative to the scan data, and the focal sphere diameter allows the process to use the previous methods described herein.
  • surfaces are generated and textured with the images to generate a query-able merged data set that can be viewed and manipulated in a virtual 3D-space or printed using emerging 3D color printers.
  • 3D data is combined with 2D surface images acquired from the same device.
  • FIGS 12A and 12B an example scan of a statue, namely a statute at Bond Hall at the University of Notre Dame has been textured with a photograph of the same statue, taken from the same point as the scanner.
  • a resultant surface 1210 generated from the point-cloud data can be seen in FIG. 12A and a resulting textured surface 1220 can be seen in FIG. 12B.
  • FIGS. 13A and 13B an example CT scan of a beetle was obtained and a 3D surface point-cloud 1310 was generated from the CT scan data as described hereinabove.
  • a photographic image was used to texture the resulting surface using the processes described herein, and an example resulting image 1320 can be seen in FIG. 13B.

Abstract

A method of mapping a two-dimensional image to a three-dimensional surface includes capturing data for a two-dimensional image and a three-dimensional structure. A process determines coincident points between the 2D image and the 3D structure and maps points on the 2D image to the 3D structure by assigning relative two-coordinate points from the two dimensional image to relative three-coordinate points of the three-dimensional structure. The mapping creates a 3D surface and texturing and removes superfluous data from the created three-dimensional surface to clean the mapped resultant.

Description

SYSTEMS AND METHODS FOR GEOMETRICALLY MAPPING TWO- DIMENSIONAL IMAGES TO THREE-DIMENSIONAL SURFACES
Cross Reference to Related Application
[0001] This application is a non-provisional application claiming priority from U.S.
Provisional Application Serial No. 61/685,678, filed March 22, 2012, entitled "Method of mapping a 2D image to a 3D surface", U.S. Provisional Application Serial No. 61/732, 164, filed August 3, 2012, entitled "System and Method of Mapping a 2D Image to a 3D
Structure", and U.S. Provisional Application Serial No. 61/797,143, filed November 30, 2012, entitled "Method of mapping a 2D image to a 3D surface", each of which are incorporated herein by reference in their entirety.
Field of the Disclosure
[0002] The present disclosure relates generally to computer visualization and more particularly to systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces.
Background of Related Art
[0003] Texturing three-dimensional (3D) surfaces with two-dimensional (2D) graphics is commonplace in the field of computer visualization. For example, visualization is a rapidly advancing field which provides immersive environments that can be enhanced by employing 3D viewing hardware. The use of such environments is now commonplace in technical and scientific fields where the correct interpretation of data is vital in both diagnostics and extending the frontiers of knowledge. Such fields generally require geometrically correct rendering of data in contrast to many areas of the entertainment industry, and hence specialist techniques are required.
[0004] A point-cloud is a set of vertices in a three-dimensional coordinate system. These vertices are usually defined by X, Y, and Z coordinates, and typically are intended to be representative of the internal and external surface of an object. Point-clouds are most often created by 3D scanners. These devices measure in an automatic way a large number of points on the surface of an object, and often output a point-cloud as a data file. The point- cloud represents the set of points that the device has measured.
[0005] As the result of a 3D scanning process point-clouds are used for many purposes, including to create 3D CAD models for manufactured parts, metrology/quality inspection, post-acquisition analysis (e.g., engineers and/or scholars) and a multitude of visualization, animation, rendering and mass customization applications.
[0006] In one example, US Patent No. 5,412,765 describes a method for visualizing vector fields uses texture mapping for the vectors to create an animated vector field. The described methods utilize time lapse as a component. Specifically, vectors are textured with one dimensional texture maps composed of alternating visible and invisible segments.
Successively applied texture maps differ from each other in order to create a moving line effect on the vectors being visualized. A fading effect is further provided by varying the intensity of the visible segments.
[0007] Meanwhile, US Patent No. 7,689,019 describes a method for registering a 2D projection image of an object relative to a 3D image data record of the same object, in which, from just a few 2D projection images, a 3D feature contained in an object, which is also identifiable in the 3D images, is symbolically reconstructed. The 3D feature obtained in this way is then registered by 3D-3D registration with the 3D image data record. The methods described identify three features on the target structure, form, surface, and centerline.
[0008] While the known systems and methods may work for their intended purposes, there is a known need in the art to provide improvements thereto.
Brief Description of the Drawings
[0009] These and other features, aspects and advantages of the present disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings.
[0010] FIG. 1 is a block diagram illustrating components of an example network system in which the disclosed systems and methods may be employed.
[0011] FIG. 2A shows an example architectural 3D point-cloud.
[0012] FIG. 2B shows an example 2D image corresponding to the example 3D point-cloud of FIG. 2A.
[0013] FIG. 3 illustrates an example process diagram for mapping an example 2D image to an example 3D point-cloud.
[0014] FIG. 4 illustrates an example block diagram of sample calculations utilized in the process of FIG. 3.
[0015] FIG. 5 illustrates an example process which allows for the blending of image date and the removal of visual artifacts. [0016] FIG. 6 illustrates an example of the determination of surface 3D point-cloud data from general 3D data.
[0017] FIG. 7 A illustrates an example image before any normalization and cleaning process.
[0018] FIG. 7B illustrates the example image of FIG. 7A after an example normalization and cleaning process.
[0019] FIG. 8A illustrates an example image before compression.
[0020] FIG. 8B illustrates the example image of FIG. 8A after an example compression process.
[0021] FIG. 9 illustrates an example image showing the partial rendering using an example internal camera from a scanner.
[0022] FIG. 10 is an example plot illustrating the remaining points after reduction using a center comparison process.
[0023] FIG. 1 1 is an example plot illustrating the remaining points after reduction using a two norm comparison process.
[0024] FIG. 12A illustrates an example 3D point-cloud of an architectural object.
[0025] FIG. 12B illustrates the resultant 3D point-cloud of FIG. 12A rendered as a surface utilizing the example methods of the present invention.
[0026] FIG. 13A illustrates an example 3D point-cloud of a CT Scan of an object.
[0027] FIG. 13B illustrates the resultant 3D point-cloud of FIG. 13A rendered as a surface utilizing the example methods of the present invention.
Detailed Description
[0028] The following descriptions of various examples are not intended to limit the scope of the descriptions to the precise form or forms detailed herein. Instead the following descriptions are intended to be illustrative so that others may follow their teachings.
[0029] The present disclosure provides methods of determining the geometrically correct registration between a 2D image, or images, and the underlying 3D surface associated with them to allow automatic alignment of, for example, many images to many point-clouds. The requirement arises from technical and scientific fields, such as architecture and/or historical preservation, where, for example, a measurement from the underlying 3D data needs to be determined by observation of surface data. In this example, the present disclosure facilitates the combination of photographic data from multiple viewpoints with 3D data from laser scanners or Computer Aided Design (CAD) computer programs, to generate immersive environments where the user can query the underlying data by visual interpretation. Another example allows the combination of surface data, including internal surfaces, for biological specimens with CT scan data for diagnostic or educational purposes. In yet another example, the present disclosure allows for mapping of multiple images acquired from robotic tripods, such as a GigaPan tripod with a camera attached, to scanned data from scanners such as the Leica CIO. In still another example, the present disclosure maps images from camera enabled scanners to point-clouds.
[0030] Acquisition of 3D data and the associated 2D surface data can be accomplished in a number of ways, depending on the problem. Where it is possible to use a single scan to obtain both 3D coordinates and surface images simultaneously, registration of the sets of data is guaranteed but with known technology, the object has to be of the order of 1cm to lm in size. In many situations this is not possible, for example large architectural structures, landscapes, large area scenes, nano or biological-scale objects and data from devices such as CT scanners, etc.
[0031] In such cases, previous methods have manually adjusted the registration between the data sets visually, which does not guarantee the fidelity of registration over the full extents of the data sets and, in general, cannot correctly align the data sets. There also exist computer programs to approximately align the data to produce single point visualizations. In general this visualization cannot be rotated through all viewpoints and it is not possible to measure surface features based on the underlying 3D data.
[0032] In the present disclosure, 3D data, such as point-cloud data, is automatically combined with 2D surface data, which may include internal surfaces, such that the registration between the data sets, and hence the validity of observations and measurements based on them, is determined geometrically and maintained over the surface of interest. Where multiple sets of 2D data are available for the same surface area, the optimal set can be determined for each rendered pixel set. In addition, visual artifacts generally present with adjacent multiple images can be reduced. The surface images do not have to be produced from fixed viewpoints in relation to the 3D representation, or map directly to individual 3D features of the object, allowing the acquisition of data from the architectural to nano scales. This is particularly important from complex surfaces, for example those generated from biological specimens.
[0033] Still further, 3D points are culled from individual point-clouds, where a number of point-clouds are used, to reduce or eliminate overlap for optimal texturing. 3D points are culled from individual point-clouds to remove artifacts where the distance between adjacent points is large, indicating a real spatial separation. 3D points are culled to reduce their number, and hence the number of triangles, for efficiency, such that important features (such as edges) are preserved.
[0034] In one of the present disclosure, 3D data is combined with 2D surface images acquired from the same device, mapping images from camera enabled scanners to point- clouds. In another example of the present disclosure there is described a process of mapping multiple images acquired from robotic tripods, such as a GigaPan robotic tripod, to 3D scan data. The systems and methods disclosure herein also facilitates the combination of photographic data with 3D data from laser scanners to generate immersive environments where the user can query the underlying data by visual interpretation. As such, the examples disclosed have found particular use in, but are not restricted to, architectural and biological diagnostic fields.
[0035] With reference to the figures, systems and methods for mapping 2D images to 3D surfaces is now described. While not intended to be limiting, the systems and methods will be described in the context of a plurality of processing devices linked via a network, such as a local area network or a wide area network, as illustrated in FIG. 1. In this regard, a processing device 20, illustrated in the exemplary form of a computer system, is provided with executable instructions to, for example, provide a means for an operator, i.e., a user, to access a remote processing device, e.g., a server system 68, via the network to, among other things, distribute processing power to map 2D images to 3D surfaces. Generally, the computer executable instructions disclosed herein reside in program modules which may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Accordingly, those of ordinary skill in the art will appreciate that the processing device 20 may be embodied in any device having the ability to execute instructions such as, by way of example, a personal computer, mainframe computer, personal-digital assistant ("PDA"), cellular or smart telephone, tablet computer, or the like. Furthermore, while described and illustrated in the context of a single processing device 20, those of ordinary skill in the art will also appreciate that the various tasks described hereinafter may be practiced in a distributed environment having multiple processing devices linked via a local or wide-area network whereby the executable instructions may be associated with and/or executed by one or more of multiple processing devices, e.g., in a cloud computing environment.
[0036] For performing the various tasks in accordance with the executable instructions, the processing device 20 preferably includes a processing unit 22 and a system memory 24 which may be linked via a bus 26. Without limitation, the bus 26 may be a memory bus, a peripheral bus, and/or a local bus using any of a variety of bus architectures. As needed for any particular purpose, the system memory 24 may include read only memory (ROM) 28 and/or random access memory (RAM) 30. Additional memory devices may also be made accessible to the processing device 20 by means of, for example, a hard disk drive interface 32, a magnetic disk drive interface 34, and/or an optical disk drive interface 36. As will be understood, these devices, which would be linked to the system bus 26, respectively allow for reading from and writing to a hard disk 38, reading from or writing to a removable magnetic disk 40, and reading from or writing to a removable optical disk 42, such as a CD/DVD ROM or other optical media. The drive interfaces and their associated non-transient, computer- readable media allow for the nonvolatile storage of computer readable instructions, data structures, program modules, and other data for the processing device 20. Those of ordinary skill in the art will further appreciate that other types of non-transient, computer readable media that can store data may be used for this same purpose. Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories, nano-drives, memory sticks, and other read/write and/or read-only memories.
[0037] A number of program modules may be stored in one or more of the memory/media devices. For example, a basic input/output system (BIOS) 44, containing the basic routines that help to transfer information between elements within the processing device 20, such as during start-up, may be stored in ROM 28. Similarly, the RAM 30, hard drive 38, and/or peripheral memory devices may be used to store computer executable instructions comprising an operating system 46, one or more applications programs 48, other program modules 50, and/or program data 52. Still further, computer-executable instructions may be downloaded to one or more of the computing devices as needed, for example, via a network connection.
[0038] The operator may enter commands and information into the processing device 20, e.g., a textual search query, a selection input, etc., through input devices such as a keyboard 54 and/or a pointing device 56. While not illustrated, other input devices may include a microphone, a joystick, a game pad, a scanner, a camera, etc. These and other input devices would typically be connected to the processing unit 22 by means of an interface 58 which, in turn, would be coupled to the bus 26. Input devices may be connected to the processor 22 using interfaces such as, for example, a parallel port, game port, Firewire, a universal serial bus (USB), etc. To view information from the processing device 20, a monitor 60 or other type of display device may also be connected to the bus 26 via an interface, such as a video adapter 62. In addition to the monitor 60, the processing device 20 may also include other peripheral output devices, not shown, such as speakers, printers, 3D rendering hardware such as 3D glasses to separate right-left vision and autosteroscopic screens, etc.
[0039] The processing device 20 may also utilize logical connections to one or more remote processing devices, such as the server system 68 having one or more associated data repositories 68A, e.g., a database. In this regard, while the server system 68 has been illustrated in the exemplary form of a computer, it will be appreciated that the server system 68 may, like processing device 20, be any type of device having processing capabilities. Again, it will be appreciated that the server system 68 need not be implemented as a single device but may be implemented in a manner such that the tasks performed by the server system 68 are distributed to a plurality of processing devices linked through a communication network. Additionally, the server system 68 may have logical connections to other third party server systems via the network 12 and, via such connections, will be associated with data repositories and/or functionalities that are associated with such other third party server systems.
[0040] For performing tasks as needed, the server system 68 may include many or all of the elements described above relative to the processing device 20. By way of further example, the server system 68 includes executable instructions stored on a non-transient memory device for, among other things, handling mapping requests, etc. Communications between the processing device 20 and the server system 68 may be exchanged via a further processing device, such as a network router, that is responsible for network routing.
Communications with the network router may be performed via a network interface component 73. Thus, within such a networked environment, e.g., the Internet, World Wide Web, LAN, or other like type of wired or wireless network, it will be appreciated that program modules depicted relative to the processing device 20, or portions thereof, may be stored in the memory storage device(s) of the server system 68.
[0041] An example of an architectural 3D point-cloud 10 and corresponding 2D image 12 is illustrated in FIGS. 2A and 2B, respectively. In the present disclosure, a set of processes describe a method for geometrically determining the correct registration between a 2D image, or images, and the underlying 3D surface associated with the subject image(s). Still further, there is disclosed a process for mapping of a single image to a 3D surface point-cloud, as well as extending to multiple images from different viewpoints. Finally, there are described processes for extracting surface 3D point-clouds from general 3D data, such as for example, point-clouds, CT scans, or the like. [0042] For simplicity of understanding, the following disclosure assumes that the user has a 3D point-cloud of surface points i.e., a collection of points in 3 dimensional space representing a surface which may be internal to the object, and a single 2D image which overlaps with the 3D data. Furthermore, the presently described systems and methods also assume the position of the camera is known in relation to the point-cloud. It will be appreciated, however, that the position of the camera need not be known in relation to the point-cloud and may be calculated and/or otherwise determined utilizing any suitable calculation method. Use of the present disclosure with multiple images and the generation of surface point-clouds from data, such as general point-clouds, CT scans, etc., will be discussed as well.
[0043] In the present example, the 3D point-cloud data may be obtained from any suitable source, including for instance, a scanner such as a Leica scanner available from Leica Geosystems AG, St. Gallen, Switzerland (for architectural use). Additionally, in this example, the 2D image may be selected from any suitable imaging source, such as an image from a set obtained using a GigaPan robotic tripod in association with an SLR camera, a high-resolution imaging system.
Mapping 2D image to 3D point-cloud
[0044] Turning to FIGS. 3 and 4, in one example of the present disclosure, a process 300 for mapping a 2D image to a 3D point-cloud is illustrated. In this example, processing starts at a block 302, wherein two coincident points between the 2D image and the 3D point-cloud are determined. Referring to FIG. 4, in a 3D point-cloud 400, two points, A and B in the 3D data 400, and the 2D image are determined. While not necessarily limiting, for his example, we will assume point A is closer to the center of the image 400. As illustrated, the 'noses' of the two statues are used.
[0045] Once the two coincident points are determined, the process 300 will continue with a block 304, wherein the process 300 finds the viewpoint. In other words, the process 304 finds the vector from the camera origin O (see reference 402, FIG. 4) to point A, denoted as vector C. Without loss of generality the process 300 assumes that point O is at the system origin.
O = (0,0,0) Eq. 1
[0046] Upon determination of the viewpoint, the process 300 continues with a
determination of the viewing plane at a block 306. In this instance, the process 300
constructs a plane P (reference 406, FIG. 4) with a normal vector C without loss of generality. The example plane P (406) is constructed, at a distance 1 from the camera origin
O (402). As such, using Eq. (1), a point on the plane is A = A/||A|| and the vector from the origin O to points, A- O, is a unit normal vector to the plane. Then for point p on plane P, using Eqn. (1),
p.(A - 0)- Ά. (Ά - O) = p. A - 1 = 0 Eq. 2
[0047] Processing then continues with a block 608, wherein the process 300 finds the points in the viewing plane. Specifically, for each point Pi on the 3D surface, the process draws a line Li from the camera origin O to the point Pi and finds the point of intersection with the plane P, PP;4. For point 1 on the line Li and parameter t we have:
1 = 0 + t(Pi - O) Eq. 3
which intersects the plane P when:
-1 - O. (A - O)
t = ^ Eq. 4
(Pi - O). (A - O)
Using Eqs. 1-4, this reduces to
Pi
PPi =— ^ Eq. 5
PiA
[0048] The process 300 then finds the Y-axis in the viewing plane at a block 310. In this instance, the process defines a "vertical" plane VP containing the vector C, and finds the line Y where this plane intersects the plane P. This axis will be the "y-axis" for the 2D image. In one example, this can be accomplished by adding a vertical offset V0ffs to point A to give a point V = A + Voffs, and a new point on the plane V can be calculated using the technique in Eq. 5.
_ V
V = — Eq. 6
VA
And the process 300 requires a unit vector Y in the "y" direction from the plane origin, hence:
V - A
Υ = τρ Eq. 7
\\V - A\\
[0049] The process 300 also finds the X-axis in the viewing plane at a block 312. In this instance, the line Y is rotated through 90 degrees clockwise in the plane P around the point where the vector C intersects the plane P. It will, however, be appreciated that the order and/or methodology for determining the Y and X axis may vary as desired. In this example, given 0 = ( , b, c), V = (x, y, z) Eq.8
The process 300 finds a point H rotated around A in the plane P, (including u=a, v=b, w=c) dotp = u*x+v*y+w*z,
H = (a*(v*v+w *w) + u * (—b * v - c * w + dotp)
+(— c *v + b*w-w*y + v*z),
b*(u*u+w*w) + v* (-a * u - c * w + dotp)
+(c *t/-a*w+w*x-t/*z), c*(t/*t/ + v*v) + w* (-a * u - b* v + dotp)
+(-£> *t/ + a*v-v*x+t/*y)).
Requiring a unit vector in the 'x' direction from the origin A hence
H -A
II" -41
[0050] Once the X and Y axis are determined, the process 300 continues at a block 314, wherein the planar points are projected onto the "x-y" axes. In this instance, each of the points PPi are projected onto the x and y axes to determine their 'x-y' coordinates PPXi and PPYi. This may be determined through the equation:
Figure imgf000011_0001
PPYi = pp.- ).Y Eq.10
[0051] At a block 316, the process finds the scale factors and offsets for the 2D image. For example, the process 300 calculates a scale factor S and the 'x-y' offsets X0ff and Y0ff for the 2D image. Given that points A and B on the 3D point-cloud 400 correspond to points A and B in the plane P, and assuming these points on the 2D image have 'x-y' coordinates (ax, ay) and (bx, by), respectively, the process 300 can determine that
\\B-A\\
S = τ Eq.11
{(bx - ax)(bx - ax) + by - ay){by - ay))2
xoff = ~ax E - 12
Yoff = ~ay Eq.13 [0052] Once the scale factor S and the 'x-y' offsets are determined, the process calculates the actual 'x-y' coordinates for all points at a block 318. In this example, the process 300 may utilize the equation:
ppx{ = PPXi * S - xoff
ppyt = PPYt * S - yQff Eq. 14
[0053] The process 300 may then create a 3D surface and texture using the 2D coordinates corresponding to each 3D point at a block 322. The surface texturing may be accomplished through any suitable procedure, including, for example, the procedure disclosure in Table 1. As an example the process 300 may draw a single OpenGL Quad (see Table 1) textured by one quadrilateral from the image (consisting of four adjacent points in the point-cloud). It will be understood that while the present example is provided in openGL, any suitable process may be utilized as desired. In the example disclosure herein, it is assumed for illustration purposes only that the top left point on the Quads 3D point-cloud is P2, the bottom left P 10, the bottom right PI 1 and the top right P3, with corresponding actual 'x-y' coordinates of the image (ppx_2, ppy_2), (ppx lO, ppy_10), (ppx lO, ppy_10) and (ppx_3, ppy_3) (for ease of representation, the nomenclature has changed from ppxi to ppx i in the example of Table 1).
// Draw a S ingle Quad
glBegin(GL QUADS);
//set normal
glNormal3f(normal.x(), normal. y(), normal. z());
glTexCoord2f (ppx_2, ppy_2);
gl Vertex 3 f (top le ft_p o s iti on . x() ,
top_left_position.y(),
top_left_position.z()); //
Top Left glNormal3f(normal.x(), normal. y(),
normal. z()); glTexCoord2f (ppx lO, ppy_10));
glVertex3f(bottom_left_position.x(), bottom_left_position.y(),
bottom_left_position.z()); //
Bottom Left glNormal3f(normal.x(), normal.y(), normal. z());
glTexCoord2f (ppx l l , ppy_H));
glVertex3f(bottom_right_position.x(),
bottom_right_position.y(),
bottom_right_position.z()); //
Bottom Right glNormal3f(normal.x(), normal .y(), normal. z());
glTexCoord2f (ppx_3, ppy_3));
glVertex3f(top_right_position.x(),
top_right_position.yQ,
top_right_position.z() ); // Top Right
glEndO;
Figure imgf000013_0001
a e
[0054] In the example Quad textured process, The vector "normal" represents a normal vector to the Quad for the calculation of lighting effects and can be found as:
normal = (P10 - P2) x (P3 - P2) Eq. 15 wherein X is the cross product.
Mapping multiple overlapping images
[0055] The processes described herein can be extended to multiple images by considering each image separately. Where images overlap, however, the process 300 needs to choose the optimal image section to render. Specifically, image overlap can be advantageous during data capture to ensure continuity of data over the entire surface of interest, i.e. all 3D points are associated with image data. Overlapping images also allow fill-in of areas that would be hidden by other features of the object being rendered.
[0056] Referring then to FIG. 5, there is presented a process 500 which allows for the blending of image data and the removal of visual artifacts. In this example, the process 500 first calculates the areas for each corresponding image section, generally a triangle or quadrilateral, for the adjacent 3D points of interest at a block 502. The area A can be calculated for a triangle with vertices V0, VI, V2 as
A = ^ \\ (V1 - V0) x (V2 - V1) \\ Eq. ie for 2D coordinates with V; = (x;, y
1
A = - l - ½) (y2 - y0) - C¾ - *ο)0¾— yo) l Eq. n
For example, given images Ii and ¾ with areas Ai and A2, if A2 > Ai the process 500 will use the section from image I2, else the process will use the section from Ii.
[0057] The process 500 may also select the image section with the largest area after adding a random perturbation to the area. In this instance, the process 500 allows 'blending' of the images to prevent artifacts such as image interface lines. Using the nomenclature from Equations 16 and 17, if A2 + R > Ai, the process 500 uses the section from image else the process uses the section from Ii, for random variable R, which may preferably be determined with zero mean and known distribution.
[0058] Still further, the process 500 may select the image section with the largest area up to some error E, and if the areas are within some chosen error E, merge the image data during texturing. This process allows the 'blending' of the images to prevent artifacts such as image exposure level/color differences. In this example, if A2 > Ai + E, the process 500 uses the section from I2, and if Ai > A2 + E, the process 500 uses the section from Ii. Else, the process 500 blends the sections from Ii and I2. Blending can be accomplished utilizing any suitable blending procedure, including, for example, an example OpenGL as shown in Table 2.
glBindTexture(GL TEXTURE 2D, texture[0]); // Select Texture 1 (0) glBegin(GL QUADS); // Begin Drawing the Quad glEnd();
glBlendFunc(GL_SRC_ALPHA, CL ONE); // Set Blending Mode To Mix Based On SRC Alpha glEnable(GL ALPHA);
glBindTexture(GL_TEXTURE_2D, /*your alpha channel texture*/ texture[2]); // Select Texture 2 (0) glBegin(GL QUADS); // Begin Drawing the Quad glEnd();
Table 2
[0059] The example process illustrated in Table 2 effectively finds the image that is most closely aligned with the area defined in the 3D point-cloud, but also allows blending of image data to reduce artifacts.
Determination of the surface 3D point-cloud from general 3D data
[0060] Some data, for example computed tomography (CT scan), may contain general 3D coordinates not restricted to surface points of the object. In such instance, the 3D surface point-cloud can be extracted from the data utilizing a process 600 as illustrated in FIG. 6. It will be appreciated that the process 600 assumes the camera position and viewpoint is known, but it will be understood that in various examples, the camera position and/or viewpoint can be determined as needed by one of ordinary skill in the art.
[0061] In the example process 600, the viewpoint plane is first defined at a block 602. In this process, a plane P is constructed with the camera viewpoint (e.g, a vector describing the viewing direction of the camera) vector A as a normal plane. Assuming without loss of generality that the camera is positioned at the origin and the plane P is at a distance 1 from the camera. Using these assumptions, a point on the plane is A = A/ \\A || and A is also unit normal vector to the plane. Then for a point p on plane P
p. A - l = 0 Eq. 18
[0062] Once the viewpoint plane is defined, the example process 600 defines the Y-axis at a block 604. In this example, the process 600 defines a camera 'vertical' plane VP containing a vector A, and finds the line Y, where this plane intersects the plane P. This intersect will be considered the 2D image 'y' axis. As previously noted, this can be accomplished by adding a vertical offset V0ffs to point A to give point V = A + V0ffs, and a new point on the plane V can be calculated using the technique noted in Eq.5.
_ V
V =— ~ Eq.19
V.A
For a point y and parameter t on the line y, the process concludes
y = A + t(V-A) Eq.20
[0063] Once the y-axis is determined, the process 600 performs a step of determining the x-axis at a block 606. As with the previous example, the x-axis may be determined through the rotation of the line y through 90 degrees clockwise in the plane P around the point where the vector A intersects the plane P. This line X is the x-axis. Given that:
O = (a,b,c),V = (x,y,z) Eq.21
The process 600 can then find point H rotated around O in the plane P (utilizing u=a, v=b, w=c)
dotp = u*x+v*y+w*z,
H = ( a*(v*v+w *w) + u * (-b * v - c * w + dotp)
+(— c *v + b*w-w*y + v*z),
b*(u*u+w*w) + v* (-a * u - c * w + dotp) +(c *u-a*w+w*x-u*z),
c*(u*u + v*v) + w* (-a * u - b* v + dotp) +(-b *u + a*v-v*x+u*y)).
For a point x and a parameter t on the line x, it can be determined that
x = A + t(H-A) Eq.22
[0064] Once the x and y axes are defined, the process 600 continues with the construction of an array of 'x-y' points on the plane at a block 608. In this step, the array is constructed with points on the plane given by the equation:
Gij =A + iAt(H -A)+ jAt (v-(A + iAt(H - A)^j Eq.23 for grid size parameter At and the range of i, j determined by the model size. [0065] Upon construction of the array, the process 600 may then generate an array of lines from the viewpoint through each array point in the plane at a block 610. In this step, for each Gij, the process 600 generates a line Ly consisting of points given by the equation:
Lij = tGij Eq. 24
for a parameter t.
[0066] After generating the array of lines, the process 600 can find the surface of the 3D point-cloud at a block 612. In this example, to perform the step, the process 600 finds the first intersection of each line Ly with the model data (such as non-zero CT scan data) and/or finds the points ,t in the 3D point-cloud set which are within some distance E of each line Ly but closest to the origin O. This data represents the surface point-cloud with respect to the viewpoint (e.g., the camera view). In this example, the distance d from a point x to a line Ly is given by the equation:
Figure imgf000016_0001
The solution for the index of the point-cloud data can be determined as: argmin = rr—— Eci- 26
k II l
[0067] It will be appreciated by one of ordinary skill in the art that while the process 600 is described as being well suited to find surface point-cloud date, the process 600 can also be utilized and extended to finding surface features within the object for such object as CT scans by considering the density with the scan itself.
Culling 3D Point-Clouds
[0068] As will be appreciated, scanning complex structures can generate artifacts where adjacent points in the scanner data have significant difference in distance from the scanner. In the case of multiple scans this can lead to important surface data being hidden behind 'stretched' data from overlapping scans. For correct texture registration may be vital that these artifacts be removed.
[0069] The collection of increasingly high resolution scanning data, and constraints due to current rendering hardware (especially the current choice of output device such as mobile devices), typically requires data reduction for the visualization of complex objects. The present disclosure provides intelligent processes for efficient point reduction for both of these cases while retaining important features such as object edges and relief features. [0070] For example, for full coverage, objects oftentimes need to be scanned from multiple viewpoints and each scan will generally overlap due to a desire to cover the entire object and restrictions on scanning locations. To merge multiple scans it is desirable to retain the optimal scans (those that have the highest resolution for a given common area) and remove artifacts connected by near and far points that appear adjacent in the ordering of the point- cloud. The examples provides herein provide a parameterized method of removing "poor" triangles (points) from the list.
[0071] For example, referring to FIGS. 7A and 7B, there are illustrated examples of artifacts generated by near and far objects appearing adjacent to each other in the point-cloud. In FIG. 7A, an image 700 appears before any normalization and cleaning process. In the example of FIG. 7B, an image 710 appears after the triangle areas are calculated and normalized and removed if they are larger than a given threshold.
[0072] Still further, point-clouds acquired by scanning can oftentimes be extremely large. To distribute merged data to computers 20 and mobile devices 20' via the network 14 typically requires that the data be compressed. The present disclosure includes example processes that can reduce the number of triangles (or points) in an "intelligent" manner such that important structural features, such as edges or relief features, are retained. For example, as illustrated in FIGS. 8A and 8B, an image 800 shows an non-compressed image, while an image 810 shows an example of compression by removal of points while retaining important structural features such as edges. In addition, the described methods are efficient with a computationally linear time complexity.
[0073] Yet further, mapping the image data acquired from camera enabled scanners requires the solution to a geometric mapping and scaling algorithm. The present disclosure provided herein includes processes to accomplish this automatically. This contrasts with current methods where each image is manually aligned with the point-cloud. As such, the manual approach found in prior art is impractical for multiple scans each with many images, such as for example, a Leica Scanstation II, which can oftentimes produce more than 100 images for a full 360 degree scan. An example image 900 showing the partial rendering of the Arch of Septimius Severus at the Roman Forum is illustrated in FIG. 9.
Convert the point-cloud to a surface
[0074] Without loss of generality the current disclosure assumes that the points are ordered in vertical (Z axis) lines with the zero vector representing points that are not included in the set (i.e. those removed by clipping). If this is not the case then the points must be ordered prior to analysis. Table 3 describes one example process used to find triangles that make up the surface to be textured. The example of Table 3 loops over all of the points and, for each point, tests the three other points adjacent to it to make a square (two triangles). If one point is missing we can form a single triangle and if more than one point is missing then no triangles are added.
Def CalculateMesh():
index=[]
For i in range(Horizontal-l):
For j in range(Vertical-l):
lineA=(i* Vertical)+j ;
lineB=((i+ 1 )*Vertical)+j ; count=4;
if Vertex[lineA]==Zero :
count— if Vertex[lineB]==Zero:
count— if Vertex[lineB+ 1 ]==Zero :
count— if Vertex [line A+ 1 ] ==Zero :
count— if count<3):
continue if Vertex[lineA]==Zero :
index.insert(Polygon(lineB,lineB+ 1 ,lineA+ 1 ))
continue if Vertex[lineB]==Zero:
index.insert(Polygon(lineA,lineB+ 1 ,lineA+ 1 ))
continue if Vertex[lineB+ 1 ]==Zero :
index. insert(Polygon(lineA,lineB,lineA+ 1 ))
continue if Vertex [line A+ 1 ] ==Zero :
index. insert(Polygon(lineA,lmeB,lmeB+l))
continue index.insert(Polygon(lineA,lineB,lineB+ 1 ))
index.insert(Polygon(lineB+ 1 ,lineA+ 1 ,lineA)) return index;
Table 3
Mesh pruning to remove "poor" triangles and overlap
[0075] Referring to FIGS 7A and 7B, when multiple scan locations are combined together and meshed, there are regions of the final model that have overlap from multiple scans. If these are mapped, the resolution of the scan (and hence the size of the triangles calculated) would be poor where the scanner is not directly aligned with the artifact. It would also be noticed artifacts in the form of very large (stretched) triangles where the angle that the scanner scanned from was not direct towards the model or where adjacent scan points have a large difference in distance from the scanner. These triangles are classified in the present disclosure as "poor" because to attain the highest resolution model, the present processes want as many small triangles as possible.
[0076] To overcome this issue, the present examples calculate the average compensated size of a triangle and then prune all triangles that are greater than a threshold. This threshold is, in one example, based off of the average compensated size and can be tuned to an individual model if required to achieve the greatest quality end result. The compensated area is calculated as the area divided by the distance from the scanner. In some situations it may be desirable to use the reduction algorithm without using compensation. In one instance, given points A, B, C representing the vertices of the triangle it can be determined that the area by forming vectors AB and AC, then the magnitude of the cross product AB x AC is twice the area of the triangle. To compensate for distance it may be divided by the magnitude of A.
[0077] In the examples illustrated in Tables 4 and 5 herein below, the disclosed processes compute the average triangle size (Table 4) and the "prune" any triangle above a
predetermined threshold (Table 5). def FindAverage(Verticies,Polygons) :
average=0
for poly in Polygons:
bDiff=Verticies[poly.b]-Verticies[poly.a]
cDiff=Verticies[poly.c]-Verticies[poly.a]
size=CrossProduct(bDiff,cDiff).Magnitude()*0.5
average+=retVal/V erticies [poly .a] .Magnitude
return Absolute(average/len(Polygons))
Table 5
def Prune(Threshold,Verticies,Polygons):
remaining=[]
for poly in Polygons:
bDiff=Verticies[poly.b]-Verticies[poly.a]
cDiff=Verticies[poly.c]-Verticies[poly.a]
size=CrossProduct(bDiff,cDiff).Magnitude()*0.5
area=size/V erticies [poly .a] .Magnitude if area<Threshold:
remaining.append(poly)
return remaining
Table 6
Point Reduction in O(N)
[0078] Referring to FIGS. 8A and 8B, to improve the speed of rendering mapped models, a process can be utilized to remove redundant points from the data set whilst keeping important features, such as edges. This process has time complexity of 0(N).
[0079] Due to the structure of the data set (a grid of points where holes are symbolized by a predefined "empty" point) the process can test four squares of increasing sizes to see if all of the points within the four squares face the same direction. If the points do face the same way within a threshold, the four squares can be reduced into a single square. To test if the points in the four squares face the same way, the process calculates the average normal vector for each point and then compares the center point to each point via a comparison metric. [0080] To test if a set of points is reducible, two comparison metrics are produced. The first comparison metric compares the angle between the center point and all eight surrounding points. If any of the surrounding points are greater than the threshold angle, then the process knows that the current set cannot be reduced and processing moves on. This method provides the most accurate final representation, as it does not remove any sets of points that do not all pass the test. This method provides reduction in triangles as shown in a plot 1000 illustrated in FIG. 10, which illustrates the remaining points after reduction where a center comparison is applied.
[0081] The second comparison metric compares the square root of the sum of the squares of all of the angles between the center point and all eight surrounding points. If this value is less than the threshold then the set can be reduced. This provides a less accurate
representation due to the way that the two norm calculation works, however it allows for a much greater reduction of points as shown a plot 1 100 illustrated in FIG. 1 1, which shows the remaining points after reduction where two norm comparisons are applied.
[0082] Examples of may be seen in Table 7.
defPrimePoints(Threshold,Vertex,Polygons Siormal,V^ result=Polygons[:]pointUses=PointUses[:]
step=2 whilestep<max(Horizontal,Vertical):
hStep= step/2
pUses=[0]*len(Vertex) for i in range(hStep,Horizontal,step):
if i + hStep>Horizontal:
continue for j in range(hStep,Vertical,step):
if j + hStep> Vertical:
continue lineA=((i-hStep)*Vertical)+j
lineB =(i * Vertic al)+j
lineC=((i+hStep)*Vertical)+j if point Uses[lineB]==4: normals=[
Normal [line A+h Step] ,Normal [lineB+h Step] ,Normal [line C+h Step] , Normal[lineA],Normal[lineB],normal[lineC],
Normal[lineA-hStep],Normal[lineB-hStep],Normal[lineC-hStep] ] removable=true if Criteria == Center: for k in range(lO): top=dot(normals[4],normals[k])
bottom=normals[4].Magnitude()*normals[k].Magnitude() angle=acos(top/bottom)*( 180.0/3.14159) if angle >= Threshold:
removable=falsebreak
else:
angle=0.0 for k in range(lO):
ifk!=4:
top=dot(normals[4],normals[k])
bottom=normals[4] .Magnitude()*normals[k] .Magnitude()
angle+=pow(acos(top/bottom)*(180.0/3.14159),2)angle=sqrt(angle)/8 ifangle>=Threshold:
removable=falsebreak if removable:
//RemoveBottomLeft
result-=Polygon(lineA-hStep,lineB-hStep,lineB)
result-=Polygon(lineB,lineA,lineA-hStep)
//RemoveTopLeft
result-=Polygon(lineA,lineB,lineB+hStep)
result-=Polygon(lineB+hStep,lineA+hStep,lineA) //RemoveBottomRight
result-=Polygon(lineB-hStep,lineC-hStep,lineC)
result-=Polygon(lineC,lineB,lineB-hStep)
//RemoveTopRight
result-=Polygon(lineB,lineC,lineC+hStep)
result-=Polygon(lineC+hStep,lineB+hStep,lineB)
// CreateReplacementPoints
result-=Polygon(lineA-hStep,lineC-hStep,lineC+hStep)
result+=Polygon(lineC+hStep,lineA+hStep,lmeA-hStep)
//UpdateUsagesforNextSteppUses[lineA-hStep]++
pUses[lineA+hStep]++pUses[lineC-hStep]++pUses[lmeC+hStep]++
pointUse s=pUses [ : ] step+= step return result
Table 7
[0083] It will be appreciated that the algorithm has an 0(n) time complexitybecause every iteration works on a progressively smaller fraction of points, n/2 , n/4 , . . . , n/n , which is a geometric series that always increases but never actually reaches n.
Camera Enabled Scanners
[0084] In this example of the present disclosure, there is provided a system and method of automatically mapping images generated by camera enabled scanners, to their associated point-clouds. Specifically, a number of the latest Laser Scanners incorporate a camera to provide pseudo images by coloring the point-cloud. The present disclosure may be used to map these images to the point-cloud to allow a more seamless rendering than can be obtained by colored points. An additional benefit is provided by allowing an intelligent point reduction algorithm to be used without degrading the rendered scene, which could not be used with the colored point representation without serious visual degradation.
[0085] In the following example there is described the existence of a "focal sphere" that represents a sphere which contains the points at which the center of each image will reside (i.e. with the plane of the image tangental to the sphere), and where the image scaling is coincident with the scaling of the point-cloud data from the point of view of the camera. In short, this means that a vector from the camera location a through a given image pixel will intersect the point-cloud where the pixel light originated. The "focal sphere" is related to the camera FOV and is typically a fixed value.
[0086] To find the location of an image center on the "focal sphere", given rotation angles ψ, θ, φ for axes z, y, x, rotation matrices Rz (ψ), Ry (Θ), Rx((p) are combines to give
Rzyx = ffzO/ ffy(0)ff*(0)
cos Θ cos xp — cos Θ sin xp + sin φ sin Θ cos xp sin φ sin xp + cos φ sin Θ cos xp
cos Θ sin ) ; cos Θ cos xp + sin φ sin Θ sin xp - sin φ cos i/> + cos φ sin 0 sin xp Eq. 27 — sin S sin φ cos Θ cos COS Θ
In the camera coordinates, the view is a vector in the z direction so without loss of generality a camera vector of C = (0, 0, 1)T is chosen (where T denotes the transpose). The
corresponding point on the "focal sphere" can then be found as
CfS = CRzyxfs Eq. 28
where fs is a scalar that scales the vector in the preceding terms.
[0087] The normalized vector can then be found for future calculations as
Figure imgf000024_0001
Where |.| is the vector norm.
[0088] To calculate texture coordinates, the example process finds x and y orthogonal axes in the plane tangential to the "focal sphere" at the location of the image center. In the camera coordinates the y-axis is vertical so without loss of generality the process forms a vertical vector V= (0, 1, 1)T and finds the corresponding point on the plane as:
fs
Vp = VRzyx [ J ) Eq. 30 where the bracketed term is a scalar that scales the vector in the preceding terms.
[0089] The process then finds the orthogonal axes in the plane tangential to the Focal Sphere. For example, the process finds x and y orthogonal axes as:
Vp - Cfs „ 1
y = ¥^ \\ Eq- 31
where |.| is the vector norm.
[0090] To find the x the process rotates the vertical vector point Vp around the camera axis normal Cn by 90 degrees to give:
where, given substitutions Vp = (x *, y =, z i)tT, C=n^ = (a,\ b, c E)Tq,'32 a{b2 + c2) + a(—b2— c2 + ax + by + cz) + {—cb + cy + bz)
H b(a2 + c2) + v(—a2— c2 + ax + by + cz) + (ca— ac + cx— az) Eq. 33 Lc( 2 + b2) + c(—a2— b2 + ax + by + cz) + —ba + a2— bx + ay)\
[0091] The example process then associates triangles with an image and calculate the corresponding texture coordinates for a surface, in particular, for a given point Pi on a triangle, and camera offset C0, the process can calculate a texture coordinate Ti as:
PTt.x +
Eq. 34
PTi.y +
Where vector PTi
Figure imgf000025_0001
[0092] The example process then tests if either of the x, y components of Ti are less than 0 or greater than 1. In this event the point is outside the image and is not included in the surface. The test continues for all points in the triangle, if all points pass then they are stored with their texture coordinates in the surface and removed from the available points/triangle list. This process is then repeated for n images to produce n surfaces as described, for example, in the example listing in Table 8.
def CreateRotationMatrix( xAngle, yAngle, zAngle ]
matrix = [0.0] * 9 sin( x_angle )
cos( x_angle )
sin( y_angle )
cos( y_angle )
sin( z_angle )
cos( z_angle ) matrix[0]
matrix[l]
matrix[2] matrix[3]
matrix[4]
matrix[5] matrix[6] = -( a * d * e ) + b * f matrix[7] = b * e + a * d * f
matrix[8] = a * c return matrix def CalcTextureMapping( TextureData, Meshlndex ):
if len(TextureData) < 6:
print "Insufficient data for texture, " + len(TextureData) + return
Offset = Vector3(TextureData[3], TextureData[4], TextureData[5]) x_angle = -Angle. toRadians(TextureData[0])
y_angle = -Angle. toRadians(TextureData[l])
z_angle = -Angle. toRadians(TextureData[2]) matrix = CreateRotationMatrix( x_angle, y_angle, z_angle ) normal = Vector3( matrix[2], matrix[5], matrix[8] ). Normalised)) sqrtaa = 2.2858996
centerPointOnPlane = normal * sqrtaa verticalPoint = Vector3(
matrix[l] + matrix[2],
matrix[4] + matrix[5],
matrix[7] + matrix[8]
(.Normalised))
IbDotNv = Vector3.Dot( verticalPoint, normal ) if IbDotNv == 0.0:
return verticalPointOnPlane = verticalPoint * ( sqrtaa / I bDotNv ) x = verticalPointOnPlane. x()
y = verticalPointOnPlane. y()
z = verticalPointOnPlane. z() a = normal. x()
b = normal. y()
c = normal. z()
u = a
v= b
w = c dotpl = u*x + v*y + w*z xd = a * (v * v + w * w)
xd += u * (-b * v - c * w + dotpl)
xd += (-c * v + b * w - w * y + v * z) yd = b * (u * u + w * w)
yd += v * (-a * u - c * w + dotpl)
yd+=(c*u-a*w + w*x-u*z) zd = c * (u * u + v * v)
zd += w * (-a * u - b * v + dotpl)
zd += (-b * u + a * v - v * x + u * y) horizontalPointOnPlane = Vector2( xd, yd, zd ) yVector = (verticalPointOnPlane - centerPointOnPlane). Normalised)) xVector = (centerPointOnPlane - horizontalPointOnPlane). Normalised))
Index = [] temptex = []
triangleOK = False for i in range(len(Meshlndex)):
if i%3 ==0:
triangleOK = True point = Vertex[Meshlndex[i]] - Offset
IbDotN = Vector3.Dot(point, normal) if IbDotN != 0.0:
texlnt = point * (sqrtaa / IbDotN) pVector = texlnt - centerPointOnPlane xCoord = Vector3.Dot(pVector, xVector) + 0.5
yCoord = Vector3.Dot(pVector, yVector) + 0.5 temptex[i % 3] = Vector2(xCoord, yCoord) if xCoord < O.Of 1 1 xCoord > l.Of 1 1 yCoord < O.Of 1 1 yCoord > l.Of:
triangleOK = False
else:
triangleOK = True if i % 3 == 2 and triangleOK:
ii = i / 3
for j in range(3):
Index. push_back( Meshlndex[ii * 3 + j] )
Table 8
[0093] In one example, a number of the currently available laser scanners, for example the Leica CIO, are fitted with an internal camera with which the points in the point-cloud can be colored to produce a pseudo-photographic effect. The resulting "image" is limited to the resolution of the scan rather than the image, and hence the user cannot observe the detail provided by traditional photographic sources. For example, to match a sixteen megapixel camera, the scanner would be required to scan sixteen million points in the area covered by a picture. For a 300mm telephoto zoom lens this equates to a resolution of around 0.2mm at a distance of 20m, orders of magnitude better than current scanner technology. It is also an improvement to map the internal camera data to the point-cloud in order to build a surface on which to render the full resolution of the pictures. In addition, this allows the removal of redundant points, as described below, without degrading the rendered scene. This allows for fast and efficient rendering, even on mobile or internet connected devices. Unlike textured surfaces, removal of points is not possible in the "colored points" method, which limits the usability of data across a broad platform of devices. [0094] Images from internal cameras are typically acquired through the mirror system used to launch the laser beam. Post processing software, such as Leica Cyclone, can output not only the images, but also the metadata that describes the orientation and offset of the camera. For the Leica CIO camera the attitude is arranged in a gimbal format i.e. z,y,x rotation order in relation to the coordinate system of the scanner. The effective scaling can be determined from either the field of view (FOV) of the camera, or by matching some non-central image point to the point-cloud. In the invention described herein we find the "focal sphere" where the center point of each image must touch (and the image plane is then tangential to) in order to be projected onto the points in the point-cloud with the correct scaling. For the example Leica Scanstation II described herein, this was found to be a sphere of diameter 2.2858996m. Robotic Tripods
[0095] In this yet another example use of the present invention, the process may be utilized to map multiple images acquired from robotic tripods, such as a GigaPan, to scan data. In one example, this process may be achieved in three steps; finding the 'focal sphere' where a tangential image has unity dimension, calculating the image center and normal vectors, and calculating the spherical coordinates of each image.
[0096] The example process first finds the "focal sphere" radius, fp, representing the distance from the point of acquisition of the images to a sphere where the image size would be unity. For far objects this would be equal to the lens focal length F divided by the image sensor vertical size d. In practice, for nearer objects S I, we have:
1 1 1
τ; = 7Γ + 7Γ Eq. 35
H
Figure imgf000029_0001
Here F can be found from the image EXIF information, e.g. 300mm for a typical telephoto lens. For a full frame DSLR camera d will be around 24mm. SI is found by taking the norm of one of the object 3D points.
[0097] For the angle of view the process can then determine that:
1
fP = a Eci- 37
2 tan -2
for a the angle of view.
[0098] All mapping calculations require the process to find fp. If the camera parameters are known the process can calculate the focal sphere fp directly. Otherwise the process needs to use known coincident points in both 3D-space and the 2D images and calculate fp. Two example methods are described herein.
[0099] In the first example method, if fp is not known the process can calculate it by choosing a non-central point on the image, PP, that is coincident with a point-cloud point DP. Given a point at the center of the image CP = (0.5, 0.5)T the process can use the sine rule to find
Figure imgf000030_0001
where
DP. C
[00100] In the second example method, given two 3D points PA and PB with
corresponding points on the image with vectors A and B with respect to the image center (normalized to picture dimensions), with length a and b respectively. The angle between the vectors satisfies the equation
Figure imgf000030_0002
[00101] As such, the process defines the vector to the center of the image as PC with length fp, then
which yields
Figure imgf000030_0003
then
(cos2ip - 1)// + (cos2ip(a2 + b2) - 2A. B)f2 + cos2ipa2b2 - (A. B)2 = 0 Eq.46 which can be solved using standard quadratic techniques known to one of ordinary skill in the art. [00102] To calculate the image center and normal vectors, the process requires the vector to the center of at least one image be known to align the image data to the scan data. In this example, at least one of three methods may be used to determine this.
[00103] In a first example, the center and normal vectors are calculated given two arbitrary points on the image. To find the normal vector, the process can find the center vector by solving the equation set
PC. PA = -= jL= \\PA \\fp Eq. M
Figure imgf000031_0001
\\PC \\ = fp Eq. 49
then the normal vector N is
1
N = PC - Eq. 50
\\PC \\ H
It will be noted that this result does not guarantee (or require) that the image is vertical.
[00104] In a second example, the center and normal vectors are calculated given one vector and the focal sphere. In this example, the process first finds the vector in the image plane P AP coinciding with the known vector PA
Figure imgf000031_0002
[00105] Then, the process finds a vertical vector in the image plane PVP coinciding with the y-coordinate of the known vector, noting that it shares a plane in the z-axis with this vector. This yields equations
PVP. P AP = fp2 + A2 Eq. 52
\\PVP X PAP \\ = Ax (fp2 + Ay2 ) Eq. 53
PVPZ = PAPZ Eq. 54
where A. is the coordinate length in the image plane for PA. These equations can be solved simultaneously for PVP. As will be noted, this yields two results, the correct one can be selected using the identity
\\PVP \\ = fp2 + A2, . Eq. 55
Resulting in three equations in PC
PC. P AP = fp2 Eq. 56
PC. P VP = fp2 Eq. 57
\\PC \\ = fp2 Eq. 58 which can be solved simultaneously for PC. Again, this yields two results. The correct result can be determined by considering the value of the z component in relation to the z component of PA utilizing the process of Table 9, assuming z(l) is the first result for the z- coordinate and z(2) the second.
//coords reversed as from TLHC, so A(2) < 0 means PAP a bove center!
if(A(2)<0){
//if zccord positive then PAP or PVP z coord > PC z coord
if(PAP(3) > z(l)){
PVP_z = z( l);
}else{
PVP(3) = z(2);
}
}else{
//if zccord negative then PAP or PVP z coord < PC z coord
if(PAP(3) < z(l)){
PVP(3) = z(l);
}else{
PVP(3) = z(2);
}
}
Table 9
[00106] As previously noted, the no N is then
Figure imgf000032_0001
In this case the image will be orientated vertically.
[00107] In the third example method, the approximate relative orientation is determined with one vector, given the focal sphere. In this example given the focal sphere fp the elevation of one of the points PA or PB and the y coordinate of that point on the image the process can determine the z and ratio x/y of the center point on the image.
Figure imgf000032_0002
PCZ = sin(ePA - eDIFF) fp Eq. 62
where P Cz is the center point z element, IMy is the y coordinate of the vector PA on the image. Similarly,
0PA = tan"1 ^) Eq. 63 0DIFF = tan"1 ^) Eq. 64
PC
-^- = tan(0PA - 0DIFF) cos(6PA - 6DIFF) . Eq. 65 [00108] The process can then find the x coordinate
Figure imgf000033_0001
and y coordinate
PAxPCx-≠ +PAzPCz
PC,, Eq. 67
y
It will be noted that this result does not guarantee that the image is vertical.
[00109] Once the focal sphere and the image center and normal vectors are calculated, the process can then calculate the image sphere coordinates. In this instance, the process can calculate the spherical coordinates of the main image normal vector. In particular, Given the normal vector n the azimuth (φ) and elevation (Θ) can be determined by:
Θ = sin_1(nz) Eq. 6S
0 = tan"1 (^j Eq. 69 where the x, y, z suffix represents the individual components of the 3 space vector.
[00110] The process can also calculate the elevation and Azimuth steps for overlapping images. For instance, Given that the overlap of two adjacent images is o then the associated angle step γβ in elevation is: γθ = 2 tan_1 (^^) ^ 70
[00111] For the associated angle step γφ in azimuth the process can take account of the increase in horizontal overlap with elevation angle. Because the overlap distance is proportional to the radius of the circle that represents the sphere cross section at the given elevation, fp cos(9) for sphere radius fp, it can be determined that:
Vrt = 2 tan-1 (—— -— ) . Eq. 71 [00112] Given the spherical coordinates for the center of each image, relative to the scan data, and the focal sphere diameter allows the process to use the previous methods described herein. Here surfaces are generated and textured with the images to generate a query-able merged data set that can be viewed and manipulated in a virtual 3D-space or printed using emerging 3D color printers. In one embodiment of the invention 3D data is combined with 2D surface images acquired from the same device.
[00113] Referring to FIGS 12A and 12B, an example scan of a statue, namely a statute at Bond Hall at the University of Notre Dame has been textured with a photograph of the same statue, taken from the same point as the scanner. A resultant surface 1210 generated from the point-cloud data can be seen in FIG. 12A and a resulting textured surface 1220 can be seen in FIG. 12B.
[00114] Still further, referring to FIGS. 13A and 13B, an example CT scan of a beetle was obtained and a 3D surface point-cloud 1310 was generated from the CT scan data as described hereinabove. A photographic image was used to texture the resulting surface using the processes described herein, and an example resulting image 1320 can be seen in FIG. 13B.
[00115] Although certain example methods and apparatus have been described herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the appended claims either literally or under the doctrine of equivalents.

Claims

We claim;
1. A method of mapping a two-dimensional image to a three-dimensional surface comprising:
capturing data for a two-dimensional image;
capturing data for a three-dimensional structure having a surface;
determining coincident points between the data associated with the two-dimensional image and the data associated with the three-dimensional image;
mapping points on the two-dimensional image to the three dimensional structure by assigning relative two-coordinate points from the two dimensional image to relative three- coordinate points of the three-dimensional structure;
creating a three-dimensional surface and texturing the three-dimensional surface; and removing superfluous data from the created three-dimensional surface.
2. A method as recited in claim 1, wherein the three-dimensional structure is a point- cloud.
3. A method as recited in claim 1, wherein the determination of coincident points comprises the input of three known coincident points.
4. A method as recited in claim 1, wherein the determination of coincident points comprises the input of two known coincident points and the calculation of the relative angular orientation of the device capturing the data for the two-dimensional image and the device capturing the data for the three-dimensional structure.
5. A method as recited in claim 1, wherein the determination of coincident points comprises the input of one known coincident points and the calculation of the relative angular orientation of the device capturing the data for the two-dimensional image and the device capturing the data for the three-dimensional structure, the focal length of the device capturing the data for the two-dimensional image, and the relative size of the sensor capturing the data for the two-dimensional image.
6. A method as recited in claim 1, comprising capturing data for a plurality of two- dimensional images, and repeating the mapping method for each of the plurality of two- dimensional images.
7. A method as recited in claim 6, further comprising measuring at least one of the vertical or horizontal overlap of the plurality of two-dimensional images.
8. A method as recited in claim 1, wherein removing superfluous data comprises calculating the average compensated size of a triangle corresponding the three coincident points and removing all triangles that are greater than a predetermined threshold.
9. A method as recited in claim 8, wherein the predetermined threshold is the modeled after the average compensated size of the triangle.
10. A method as recited in claim 1, wherein the data captured for the two- dimensional image and the data captured for the three-dimensional structure are obtained from the same location.
11. A method as recited in claim 1, wherein the created three-dimensional surface is compressed.
12. A method as defined in claim 11, wherein compression of the created three- dimensional surface comprises identification of a plurality of triangles formed on the created three-dimensional surface and combining similarly formed triangles.
13. A method as recited in claim 1, further comprising a plurality of captured data for the three-dimensional structure, and repeating the mapping method for each of the plurality of captured data for the three-dimensional structure.
PCT/US2013/033559 2012-03-22 2013-03-22 Systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces WO2013142819A1 (en)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201261685678P 2012-03-22 2012-03-22
US61/685,678 2012-03-22
US201261742164P 2012-08-03 2012-08-03
US61/742,164 2012-08-03
US201261797143P 2012-11-30 2012-11-30
US61/797,143 2012-11-30

Publications (1)

Publication Number Publication Date
WO2013142819A1 true WO2013142819A1 (en) 2013-09-26

Family

ID=49223377

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2013/033559 WO2013142819A1 (en) 2012-03-22 2013-03-22 Systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces

Country Status (1)

Country Link
WO (1) WO2013142819A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016019576A1 (en) * 2014-08-08 2016-02-11 Carestream Health, Inc. Facial texture mapping to volume image
RU2679964C1 (en) * 2013-12-13 2019-02-14 Авева Солюшнз Лимитед Image rendering of laser scan data
CN110782516A (en) * 2019-10-25 2020-02-11 四川视慧智图空间信息技术有限公司 Texture merging method and related device for three-dimensional model data

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001059708A1 (en) * 2000-02-11 2001-08-16 Btg International Limited Method of 3d/2d registration of object views to a surface model
US20010016063A1 (en) * 1996-12-15 2001-08-23 Cognitens, Ltd. Apparatus and method for 3-dimensional surface geometry reconstruction
EP1160731A2 (en) * 2000-05-30 2001-12-05 Point Cloud, Inc. Three-dimensional image compression method
US20080310757A1 (en) * 2007-06-15 2008-12-18 George Wolberg System and related methods for automatically aligning 2D images of a scene to a 3D model of the scene
US20100266220A1 (en) * 2007-12-18 2010-10-21 Koninklijke Philips Electronics N.V. Features-based 2d-3d image registration
US20100328308A1 (en) * 2008-07-10 2010-12-30 C-True Ltd. Three Dimensional Mesh Modeling
US20110116718A1 (en) * 2009-11-17 2011-05-19 Chen ke-ting System and method for establishing association for a plurality of images and recording medium thereof
US20110298800A1 (en) * 2009-02-24 2011-12-08 Schlichte David R System and Method for Mapping Two-Dimensional Image Data to a Three-Dimensional Faceted Model

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010016063A1 (en) * 1996-12-15 2001-08-23 Cognitens, Ltd. Apparatus and method for 3-dimensional surface geometry reconstruction
WO2001059708A1 (en) * 2000-02-11 2001-08-16 Btg International Limited Method of 3d/2d registration of object views to a surface model
EP1160731A2 (en) * 2000-05-30 2001-12-05 Point Cloud, Inc. Three-dimensional image compression method
US20080310757A1 (en) * 2007-06-15 2008-12-18 George Wolberg System and related methods for automatically aligning 2D images of a scene to a 3D model of the scene
US20100266220A1 (en) * 2007-12-18 2010-10-21 Koninklijke Philips Electronics N.V. Features-based 2d-3d image registration
US20100328308A1 (en) * 2008-07-10 2010-12-30 C-True Ltd. Three Dimensional Mesh Modeling
US20110298800A1 (en) * 2009-02-24 2011-12-08 Schlichte David R System and Method for Mapping Two-Dimensional Image Data to a Three-Dimensional Faceted Model
US20110116718A1 (en) * 2009-11-17 2011-05-19 Chen ke-ting System and method for establishing association for a plurality of images and recording medium thereof

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
RU2679964C1 (en) * 2013-12-13 2019-02-14 Авева Солюшнз Лимитед Image rendering of laser scan data
US10467805B2 (en) 2013-12-13 2019-11-05 Aveva Solutions Limited Image rendering of laser scan data
WO2016019576A1 (en) * 2014-08-08 2016-02-11 Carestream Health, Inc. Facial texture mapping to volume image
CN110782516A (en) * 2019-10-25 2020-02-11 四川视慧智图空间信息技术有限公司 Texture merging method and related device for three-dimensional model data
CN110782516B (en) * 2019-10-25 2023-09-05 四川视慧智图空间信息技术有限公司 Texture merging method and related device for three-dimensional model data

Similar Documents

Publication Publication Date Title
US9972120B2 (en) Systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces
CA3103844C (en) Method for reconstructing three-dimensional space scene based on photographing
JP6057298B2 (en) Rapid 3D modeling
Zhang et al. A UAV-based panoramic oblique photogrammetry (POP) approach using spherical projection
WO2014024579A1 (en) Optical data processing device, optical data processing system, optical data processing method, and optical data processing-use program
US20030091227A1 (en) 3-D reconstruction engine
Mousavi et al. The performance evaluation of multi-image 3D reconstruction software with different sensors
EP3427227A1 (en) Methods and computer program products for calibrating stereo imaging systems by using a planar mirror
JP2013539147A5 (en)
Fangi et al. Improving spherical photogrammetry using 360 omni-cameras: Use cases and new applications
Winnemöller et al. Light waving: Estimating light positions from photographs alone
JP2000268179A (en) Three-dimensional shape information obtaining method and device, two-dimensional picture obtaining method and device and record medium
Grussenmeyer et al. 4.1 ARCHITECTURAL PHOTOGRAMMETRY
Hafeez et al. Image based 3D reconstruction of texture-less objects for VR contents
WO2008034942A1 (en) Method and apparatus for stereopanoramic imaging
WO2013142819A1 (en) Systems and methods for geometrically mapping two-dimensional images to three-dimensional surfaces
JPH05135155A (en) Three-dimensional model constitution device using successive silhouette image
Beraldin et al. Exploring a Byzantine crypt through a high-resolution texture mapped 3D model: combining range data and photogrammetry
Rahman et al. Calibration of an underwater stereoscopic vision system
Wong et al. 3D object model reconstruction from image sequence based on photometric consistency in volume space
Remondino Accurate and detailed image-based 3D documentation of large sites and complex objects
Doan et al. A low-cost digital 3d insect scanner
Barazzetti Planar metric rectification via parallelograms
JP2001084362A (en) Method and device for estimating light source direction and three-dimensional shape and recording medium
JP6073121B2 (en) 3D display device and 3D display system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13764001

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13764001

Country of ref document: EP

Kind code of ref document: A1