JP5185777B2 - Method and apparatus for aligning 3D range data - Google Patents

Description

  The present invention relates to inspection of 3D shapes, and in particular to application fields such as entertainment, education and design that require early inspection of displayed 3D models.

  The problem of displaying a complete 3D model as soon as it is scanned is still not solved. Current switching methods required to build a complete digital model of an object include either user intervention to manually select corresponding points or automatic 3D view registration. Accompany. In automatic alignment, the display occurs after all views are aligned. In the former, it is possible (and indispensable) to visualize the intermediate view, but if the number of scans is large, it will be complicated to perform the alignment process. Combining the visualization part with automatic alignment can create a sharper sense of real-time display.

  Patent Document 1 describes a method for transferring and displaying 3D shape data in a communication network with data packet / cell loss. The system includes a transmission unit for transmitting 3D shape data composed of a plurality of elements representing the shape, and a receiving side display unit for receiving the 3D shape data and displaying the 3D shape data as an image.

  In this system, a transmission unit reads 3D data from an external storage device, and rearranges 3D shape elements according to region attributes. It then sends 3D data progressively element by element (starting with an element having an area attribute representing a large area and then processing elements having an area attribute representing a smaller area). The receiving side display unit receives the elements transmitted from the transmitting unit and displays them gradually in the order of reception.

  In Non-Patent Document 1, a brightness image stored together with range data is used, and a completely automatic alignment technique is executed. Find the corresponding point in the 3D view using 2D-image features with inherent scale information. Thereafter, the feature points themselves are first aligned, followed by an alignment step that takes into account the surface elements, so that the two ranges of images are precisely aligned. Finally, the overall alignment error is minimized using graph relaxation techniques.

  The software described in Non-Patent Document 2 uses either surface geometry or volume information to align a large collection of points (cloud) without a target or other alignment tool.

US Pat. No. 5,850,226 3D shape data transfer and display method, December 15, 1998, inventor: Michio Nagasawa, Daisuke Nishioka

5th International Symposium 2004 on G.3D Range Data Image Using Feature Surface Elements, Virtual Reality, Archeology and Cultural Heritage H. Vendels, P.A. DeJenner, R.D. Waal, M.W. Cotgen, R.A. Klein (Image-Based Registration of 3D-Range Data Using Feature Surface Elements, The 5th International Symposium on Virtual Reality, Archeology and Cultural Heritage 2004, G. H. Bendels, P. Degener, R. Wahl, M. Kortgen, R. Klein) Point Cloud Alignment Technology from Rapidform (http://www.rapidform.com/)

  Patent document 1 discloses the gradual transmission and display of image data, but does not touch the visualization of the 3D model with automatic alignment combined with a sharp sense of real-time display.

  Non-Patent Document 1 deals with automatic alignment of 3D models. However, visualization is slow and real time display is not possible.

  Non-Patent Document 2 deals with automatic alignment of many point clouds. However, this is independent of the real-time visualization of the 3D model.

  Accordingly, one object of the present invention is to provide a method and apparatus for automatically aligning and merging acquired 3D models in a manner that gives the viewer a sense of real-time display.

  Another object of the present invention is to provide a method and apparatus for automatically aligning and merging acquired 3D models in a manner that gives the viewer a sense of real-time display by showing the registration process. Is to provide.

  Yet another object of the present invention is to automatically align and merge acquired 3D models in a way that gives the viewer a sense of real-time display by gradually revealing the actual shape from multiple directions. It is to provide a method and apparatus for doing so.

  A computer-implemented method for aligning a 3D range data set of an object according to a first aspect of the present invention includes a 3D range data and a set of 2D image data corresponding to the 3D range data. A step of connecting to a storage medium for storing, a step of displaying a point representing the set of 3D range data on a monitor, and a set of 3D range data, each including a set of consecutive 3D range data Dividing into a predetermined number of groups, and in each of the groups, using the set of 2D image data, aligning a predetermined number of consecutive range data pairs and updating corresponding 3D range data And including. The predetermined number is smaller than the number of consecutive range data pairs in each group. The step further comprises: updating points displayed on the monitor according to the matching step; repeating the matching step and the updating step until all 3D range data is matched in each group; Aligning all sets of 3D range data aligned and updated during the iteration step.

  The dividing step may include a step of dividing the set of 3D range data into four groups each including continuous 3D range data.

  Preferably, the four groups may include the same number of consecutive 3D range data.

  More preferably, the step of matching includes a step of matching a pair of continuous range data and updating corresponding 3D range data in each group.

  When the computer program according to the second aspect of the present invention is executed on a computer, the computer program causes the computer to execute all of the steps described in any of the methods described above.

  An apparatus for aligning a 3D range data set of an object according to a third aspect of the present invention stores means for storing 3D range data and a set of 2D image data corresponding to the 3D range data. And means for displaying on the monitor a point representing the 3D range data set; and dividing the 3D range data set into a predetermined number of groups each including a continuous 3D range data set. And first means for matching a predetermined number of consecutive range data pairs and updating corresponding 3D range data using each set of 2D image data in each of the groups And including. The predetermined number is smaller than the number of consecutive range data pairs in each group. The apparatus further includes means for updating the points displayed on the monitor according to the alignment by the first means, and alignment of the 3D range data until all 3D range data is aligned in each group. Aligned and updated between iterations based on control of the first means and the means for updating and control of the means for controlling, so that updates are repeated Second means for matching all sets of 3D range data.

FIG. 1 is a diagram showing the arrangement of 3D views according to the first embodiment of the present invention. 1 is a plan view of a 3D model registration system 50 according to a first embodiment of the present invention. FIG. FIG. 3 is a block diagram of the system 50 shown in FIG. 2. It is a flowchart which shows the flow of control of the program which implement | achieves the method of this embodiment. It is a detailed flowchart of step 120 shown in FIG. FIG. 5 is a detailed flowchart of a routine performed in Step 128 of FIG. 4. FIG. It is a flowchart of the alignment process performed by step 174 about each division. It is a figure which illustrates 2D coincidence point.

[First Embodiment]
-Overview-
In the embodiment described below, a method for automatically constructing and displaying a digitized 3D model of an object in real life from a sequence of 3D range data and corresponding 2D image data will be introduced. The goal is a system that gives the viewer a sense of real-time display by scanning the actual object and displaying its 3D model without interruption. Instead of displaying the expanded 3D data after recommending successive scans to be aligned, the present invention takes a different approach.

  First, a complete set of scans is displayed at a time on a 3D display device. Thereafter, as shown in FIG. 1, the complete view of the object 20 is divided into four compartments 30A-30D, within each of which the successive views are aligned. In all four sections, after the first pair of consecutive views are independently aligned, they are displayed on the 3D display accordingly. For example, two consecutive views 32A and 34A of partition 30A are first aligned, and then pairs of views 32B and 34B, 32C and 34C, 32D and 34D of partitions 30B, 30C and 30D are aligned, respectively. The display will be updated with a set of partially aligned views. Subsequently, processing is performed on successive views in each partition to match them with the preceding view. They are then displayed according to the result. In this way, the cloud of initial points gradually reveals the true shape and color of the actual object simultaneously from four directions. As a final step, overall alignment is performed. Here, as in the prior art, all views are matched to a unique world coordinate system.

  Here, two views are said to be “aligned” when the position and orientation of their viewpoints are accurately determined using feature points with reference to a predetermined world coordinate system.

  In this embodiment, a complete digital model of an object is generated using 3D range data and corresponding texture data as inputs. This input data can be provided by some kind of range sensor (laser range finder, stereo camera, structured light). On the other hand, the results (3D digital model) can be visualized on some kind of 3D display device that has been enhanced by interaction with the user or can be manipulated for examination.

  In this embodiment, in situations where it is necessary to inspect the displayed model early, and as a result, the displayed object is rotated or moved to obtain an overall view. It can be used in situations where there is a means for interacting with things. Keep in mind that a digital copy of a real-world object, such as a cultural heritage, is scanned and displayed using a 3D handheld display device or a conventional 3D display monitor that can interact with the user. This is an example. Another example is collaborative work where users share the same object as a whole while having different views of their digital models.

-System configuration-
FIG. 2 is a plan view of a 3D model registration system 50 that implements the 3D model registration method of this embodiment, and FIG. 3 is a block diagram of the system 50.

  2 and 3, the system 50 includes a computer 60 to which a monitor 62, a keyboard 66 and a mouse 68 are connected, a laser range scanner 92 having a camera 96 connected to the computer 60, and a computer 60. A rotary table 94 to be controlled. The computer 60 includes an I / F 72 for mounting the semiconductor memory 86 and a DVD (Digital Versatile Disc) drive 80 for driving the DVD medium 82. The rotary table 94 rotates by a predetermined amount under the control of the computer 60. When the object is placed on the turntable 94, the turntable 94 rotates, and the range scanner 92 and the camera 96 capture 3D range data and 2D image data at each of the rotation positions of the object.

  With particular reference to FIG. 3, the system 50 further includes a processor 70, a memory / bus controller 76 connected to the processor 70, a random access memory (RAM) 78 and a cache memory (connected to the memory / bus controller 76). (Not shown), a graphics controller 90 connected to the processor 70 and the monitor 62, a bus 98 connected to the memory / bus controller 76, and an input / output (I / O) controller 72 connected to the bus 98. And a graphic card 88 connected to the I / O controller 72 and a hard disk drive (HDD) connected to the bus 98. The DVD drive 80 and the graphic card 88 are connected to the I / O controller 88 internally. The monitor 62 is connected to the graphic controller 90. The rotary table 94 is connected to the I / O controller 72. Range sensor 92 and camera 96 are connected to graphic card 88. The keyboard 66 and the mouse 68 are connected to the I / F 84.

  A program to be described later is stored in the hard disk. When executing the program, the program is read from the hard disk by the HDD 74 and stored in the RAM 78. The processor 70 has an internal register called a program counter. The processor 70 fetches an instruction from the address of the RAM 78 designated by the program counter, decodes the instruction, fetches data used for execution of the instruction, executes the instruction, and then executes data at the address designated by the instruction. Remember.

  The program may be stored in the DVD medium 82 and installed on the hard disk.

  FIG. 4 is a flowchart showing the flow of control of a program for realizing the method of this embodiment. Referring to FIG. 4, the program begins at step 120, where after initializing the associated memory space, the computer 60 digitizes the actual object placed on the turntable 94. In step 120, the computer 60 rotates the rotary table 94 and stops it at a specific angle. At each angle, the range scanner 92 captures 3D range data of the object, and the camera 96 captures the corresponding 2D image.

  The program further includes a step 122 for displaying the complete set of 3D range data captured in step 120 on the monitor 62, and a step 124 following the step 122 for roughly placing the 3D range data in four sections; Following step 128 is the step of registering the 3D view captured by range scanner 92 in step 120 and incrementally updating the model view each time it repeats the alignment of successive view pairs in each of the four sections. Step 130 for registering a complete set of 3D range views, each including four sections aligned in step 128, and step 132 for generating the digitized object surface aligned in step 130. And following step 134 to the hard disk Includes a step 134 for storing view data elephant product, the.

  FIG. 5 shows details of step 120 of FIG. Referring to FIG. 5, the routine of step 120 extracts 3D range data and corresponding 2D image data from range scanner 92 and camera 96 at each angle of turntable 94, and stores the data at each location. And step 152 for determining whether or not the scan is completed and branching the control flow to step 150 or step 122 in FIG. 4 according to the result of the determination.

  FIG. 6 is a detailed flowchart of the routine executed in step 128 of FIG. Referring to FIG. 6, this routine starts at step 170, where the iteration control variable i is initialized to 0, and processing 172 is repeated as long as the value of i is smaller than a constant M-1, and is repeated once. The value of i is incremented. Here, M is the number of views in each partition. When the variable i is equal to M−1, the iteration ends. At each repetition, processing 172 shown in FIG. 6 is executed.

  Process 172 includes, for each partition, step 174 where the i th and (i + 1) th views are aligned and the 3D range data is updated accordingly, and a complete set of updated 3D range data is displayed 178. And including.

  For example, when i = 0, views 32A and 34A are aligned in partition 30A, views 32B and 34B are aligned in partition 30B, views 32C and 34C are aligned in partition 30C, and view 32D And 34D are aligned in section 30D. The 3D range data is updated according to the alignment result, and then the complete set of updated 3D range data is displayed on the monitor 62. I is then incremented to 1, views 34A and 36A are aligned in section 30A, views 34B and 36B are aligned in section 30B, and so on. The 3D range data is updated accordingly and the complete set of 3D range data is displayed.

  Thus, the 3D model view on the monitor 62 is gradually updated during the execution of the processing shown in FIG. In each section, consecutive 3D range view pairs are aligned and the model view is updated. This process will be repeated for all successive pairs, giving a sense of real time display.

  FIG. 7 is a flowchart of the alignment process performed in step 174 for each section. Referring to FIG. 7, the routine includes a step 200 of analyzing the corresponding 2D image and extracting matching 2D point pairs for each of the 3D views to be registered. This step can be realized by publicly available software such as a SIFT keypoint detector (http://www.cs.ubc.ca/~lowe_keypoints/).

  A 2D coincidence point is illustrated in FIG. Assume that the image of the Japanese doll 220 is the Nth 2D view of the first partition 30A and the image 222 is the (N + 1) th 2D view of the same doll. When each of these views is analyzed using the software described above, feature points can be found in each image.

  Following the step 200, the routine further includes a step 202 of eliminating the false match points extracted in the step 200. This can be done using publicly available software such as the OpenCV library (http://opencvlibrary.sourceforge.net/). Referring again to FIG. 8, the feature points of images 220 and 222 are further analyzed, false match points are eliminated, and only true match point pairs remain.

  Following step 202 is a step 204 of extracting corresponding 3D point pairs based on the 2D match. This step can be achieved using a one-to-one correspondence between the pixels of the 2D image and the depth values of the 3D view.

  The routine further includes, following step 204, calculating 206 an absolute orientation of the 3D view based on the 3D coincidence points extracted in step 204. This step can be implemented with publicly available software at http://isiswiki.georgetown.edu/zivy/ by Zib Yaniv. This software is based on the Horn method (Berthold KP Horn. Closed-form solution of absolute orientation using unit quarternions). Journal of the Optical Society of America, 1987). Alignment of two 3D range data requires at least three pairs of corresponding points. Further, the software by Jib can be realized by using a set of C ++ libraries (http: //vxl.sourceforge.net#download) called VXL. After step 206, control exits from this routine.

-Operation-
The 3D model registration system 50 operates as follows. 2 and 3, the model is placed on the turntable 94 and the initial position of the turntable 94 is determined. The range scanner 92 captures the model range data. A camera 96 captures a 2D image of the model. The range data and the 2D image are transmitted to the computer 60 via the graphic card 88, the I / O controller 72, and the bus 98, and stored in the address specified by the hard disk processor 70 by the HDD 74.

  Thereafter, the processor 70 rotates the rotary table 94 by a predetermined angle. Range scanner 92 and camera 96 capture range data and 2D images at this angle, respectively. Both the range data and the 2D image are stored in the hard disk. Thus, the range data and the 2D image are captured and stored at each predetermined rotation position of the turntable 94. This task is performed in step 120 shown in FIG.

  Once the complete set of range data and corresponding 2D images has been captured and stored, control proceeds to step 122 of FIG. Here, the complete set of 3D range data is read from the hard disk and displayed on the monitor 62. Since the view is not aligned, the model displayed is a cloud of range data points.

  At step 124, the 3D range data is roughly arranged in four sections. At step 128, for each partition, a pair of selected consecutive 3D range views is aligned. After alignment in each section, the range data display on the monitor 62 is updated accordingly. Next, the next pair of consecutive 3D range views in each section is selected and aligned and the display is updated. This process is repeated until all of the successive view pairs in each partition are aligned (step 128).

  When step 128 is complete, the complete set of 3D range views is re-aligned at step 130 using conventional alignment methods. A surface is generated for the registered object at step 132 and the model data is stored on the hard disk at step 134.

  As described above, this embodiment aligns selected pairs of 3D views of range data in each section and then updates the display of range data with each iteration. During the alignment process, the model view is gradually updated, giving the user a sharp sense of real-time alignment.

  In the above-described embodiment, the range data is arranged in four sections. However, the present invention is not limited to this embodiment, and the range data may be divided into any number of groups greater than one. However, this number should not be too large to give a gradual real-time display sensation. The range data is preferably divided into 4, 8 or 16 groups. The range data may be roughly grouped, and the number of range data in the group may be different from each other.

  In the above-described embodiment, the range scanner 92 is used to capture the range data. However, the present invention is not limited to such an embodiment. A stereo camera or structured light may be used in place of the range scanner 92 as long as the distance to an arbitrarily selected point on the surface of the object can be measured.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

50 3D Model Alignment System 60 Computer 62 Monitor 70 Processor 72 I / O Controller 74 Hard Disk Drive 76 Memory / Bus Controller 78 Random Access Memory 80 DVD Drive 84 Interface 88 Graphic Card 90 Graphic Controller 92 Range Scanner 94 Rotary Table 96 Camera

Claims (6)

A computer-implemented method for registering a 3D range data set of an object, comprising:
A step wherein the computer is to connect a set of 3D range data, the storage medium for storing a set of 2D image data corresponding to the set of the 3D range data,
The computer reading the set of 3D range data from the storage medium and displaying on the monitor the points represented by each of the 3D range data;
The computer includes a dividing step of dividing the set of 3D range data, the number of groups to a predetermined comprising each a plurality of 3D Renjide data,
The computer, in each of the groups, said one set of 2D image data, by using the corresponding image feature points in the 2D image data corresponding to a predetermined number of 3D range data included in the group, the predetermined as the feature points represented by the number of 3D range data matches, the matching step is aligning the 3D Renjide data of the predetermined number, stored in the storage medium, and updates the 3 D range data of the predetermined number and,
The computer reads the 3D range data updated at the integer Gosu step from the storage medium, the points displayed on the monitor, and updating step of updating in accordance with 3D range data the reading,
And repeating the computer until all of the 3D range data is aligned in said each group, and said further Synth step and the integer Gosu step while changing a combination of 3D range data of the predetermined number,
Said computer comprises the steps of aligning all the sets of 3D range data updated aligned between said repeating step method. The method of claim 1, wherein the dividing step includes the computer dividing the set of 3D range data into four groups, each containing consecutive 3D range data.   The method of claim 2, wherein the four groups include the same number of consecutive 3D range data. The integer Gosu step, the computer, in each of the groups, of the set of the 2D image data, corresponding image feature points in the 2D image data corresponding to a pair of 3D range data included in the group The pair of 3D range data is matched and stored in the storage medium so that the feature points represented by the pair of 3D range data match, and stored in the storage medium. The method of claim 1, comprising updating the range data.   A computer program that, when executed on a computer, causes the computer to execute all of the steps according to any one of claims 1 to 4. An apparatus for aligning a 3D range data set of an object,
A set of 3D range data, storage means for storing a set of 2D image data corresponding to the set of the 3D range data,
Means for reading the set of 3D range data from the storage means and displaying the points represented by each of the 3D range data on a monitor;
The set of 3D range data, and dividing means for each divided into the number of groups that have been predetermined that includes a plurality of 3D Renjide data,
In each of the groups, said one set of 2D image data, by using the corresponding image feature points in the 2D image data corresponding to a predetermined number of 3D range data included in the group, the predetermined number of 3D range as the feature point matches represented by the data, first means for aligned the 3D Renjide data of the predetermined number, stored in the storage unit, updates the 3D range data of the predetermined number ,
Reading the 3D range data thus updated to the matching by the prior SL first means from the storage means, the point to be displayed on the monitor, and updating means for updating in accordance with 3D range data the reading,
Until all of the 3D range data is aligned in the respective groups, so that the the alignment of 3D range data updates and are repeated while changing the combination of 3D range data of the predetermined number, further wherein said first means and means for controlling the newcomer stage,
And a second means for aligning all sets of 3D range data aligned and updated during the iteration based on the control of the means for controlling.

Images