WO2015024361A1

WO2015024361A1 - Three-dimensional reconstruction method and device, and mobile terminal

Info

Publication number: WO2015024361A1
Application number: PCT/CN2014/070135
Authority: WO
Inventors: 刘兆祥; 廉士国
Original assignee: 华为技术有限公司
Priority date: 2013-08-20
Filing date: 2014-01-06
Publication date: 2015-02-26
Also published as: CN104424630A

Abstract

Disclosed are a three-dimensional reconstruction method and device, and a mobile terminal. The three-dimensional reconstruction method comprises: projecting linear laser to an object; continuously collecting image information about the object irradiated by the linear laser from at least two angles, and continuously collecting motion information about a camera; according to the image information, obtaining the three-dimensional coordinates of the object under a camera coordinate system corresponding to each collection time at each collection time; and according to the motion information, obtaining a position relationship of the camera coordinate system relative to a global three-dimensional coordinate system at each collection time, and conducting three-dimensional reconstruction of the object according to the three-dimensional coordinates and the position relationship. By means of the three-dimensional reconstruction method and device, and the mobile terminal in the embodiments of the present invention, image information about an object can be continuously collected from different angles and motion information about a camera can be continuously collected, and quick and omnidirectional three-dimensional scanning and reconstruction can be conducted on the object according to the collected information.

Description

Three-dimensional reconstruction method and device, mobile terminal

This application claims to be filed on August 20, 2013 in the Chinese Patent Office, Application No. 201310364666.0, entitled "Three-Dimensional Reconstruction Method and Apparatus, Mobile Terminal" Chinese Patent Application Priority, the entire contents of which are incorporated herein by reference. In the application.

The present invention relates to the field of three-dimensional information technology, and in particular, to a three-dimensional reconstruction method and apparatus, and a mobile terminal.

BACKGROUND OF THE INVENTION Three-dimensional scanning and reconstruction is a high-tech integrating light, machine, electricity and computer technology. It is mainly used to scan the external structure and color of an object to obtain the spatial coordinates of the object. Its important significance is that it can convert the stereo information of an object into a digital signal that can be directly processed by a computer, which provides a convenient and quick means for digital digitization. 3D scanning and reconstruction technology is widely used in more than 4 fields, such as industrial reverse engineering calculations, medical surface inspection, and product quality control in production.

In the prior art, the following two devices are generally used to realize three-dimensional scanning and reconstruction of an object. One is a hand-held three-dimensional scanning device consisting of a line laser projector, a camera and an external auxiliary positioning device, which performs laser tracking by external auxiliary positioning device or wireless positioning indoors to realize three-dimensional scanning and reconstruction. The main disadvantage of the device is that the device is bulky and thus has poor portability, is easily limited by the spatial extent, and the three-dimensional reconstruction result does not realize the collection of color information, and therefore, the color information is seriously missing. The second is a mobile phone with a rear camera and a micro-projector that uses three projected structured lights to achieve three-dimensional scanning and reconstruction. The main disadvantage of this device is that it is relatively expensive and can only measure one surface. Since the acquired image information is not correlated, the omnidirectional three-dimensional reconstruction of the object is not achieved, and the color information of the object is not obtained.

Summary of the invention

3⁄4 problem

In view of this, the technical problem to be solved by the present invention is how to implement a three-dimensional reconstruction method and apparatus for performing fast and omnidirectional three-dimensional scanning and reconstruction of an object.

solution In order to solve the above problems, in a first aspect, the present invention provides a three-dimensional reconstruction method, comprising: projecting a linear laser to an object; continuously acquiring image information of an object illuminated by the linear laser from at least an angle, and continuously acquiring the image of the camera According to the image information, obtaining three-dimensional coordinates of the object at each acquisition time in a camera coordinate system corresponding to each acquisition time; according to the motion information, obtaining a camera coordinate system at each acquisition time is relatively global a positional relationship of the three-dimensional coordinate system; and three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship.

With reference to the first aspect, in a possible implementation, the method further includes, according to the image information, obtaining the three-dimensional coordinates of the object in the camera coordinate system at the time of the collection at each acquisition time, the method further includes: And calibrating the internal parameters and the external parameters of the camera; and obtaining, according to the image information, the three-dimensional coordinates of the object at each acquisition time in a camera coordinate system corresponding to each acquisition time, including: collecting according to each collection time The image information and the inner and outer parameters are converted to three-dimensional coordinates of the object at a camera coordinate system corresponding to each acquisition time.

With reference to the first aspect, in a possible implementation manner, the motion information includes acceleration, angular velocity, and heading, and according to the motion information, obtaining a positional relationship between a camera coordinate system and a global three-dimensional coordinate system at each acquisition time The method includes: according to the acceleration, the angular velocity, and the heading, using a dead reckoning algorithm to calculate a positional relationship of the camera coordinate system of each of the acquisition moments relative to the global three-dimensional coordinate system.

With reference to the first aspect, in a possible implementation manner, the performing three-dimensional reconstruction on the object according to the three-dimensional coordinates and the positional relationship comprises: comparing a global coordinate coordinate according to a camera coordinate system of each acquisition time The positional relationship of the system converts the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time into three-dimensional coordinates in the global three-dimensional coordinate system.

With reference to the first aspect, in a possible implementation, after the three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship, the method further includes: calculating a camera coordinate system at different acquisition times A relative relationship between the mappings between the image information of the objects at different acquisition times is established according to the relative relationship.

With reference to the first aspect, in a possible implementation, the mapping relationship between the image information of the object at different acquisition moments according to the relative relationship includes:: scanning the object scan line at the i-1th acquisition time The three-dimensional coordinates of the upper point in the camera coordinate system are mapped to the camera coordinate system at the ith acquisition time, and the image sitting at the ith collection time is calculated. And obtaining a pixel value of the point according to image coordinates of the point at the ith acquisition time, where i is an arbitrary integer greater than 1.

With reference to the first aspect, in a possible implementation manner, after the mapping relationship between the image information of the objects at different acquisition times is established, the method further includes: combining the result of the three-dimensional reconstruction with the result of establishing the mapping relationship.

In a second aspect, the present invention provides a three-dimensional reconstruction apparatus, comprising: a line laser projector for projecting a linear laser to an object; and a camera for continuously collecting the light irradiated by the linear laser from different angles Image information of the object; a sensor for continuously acquiring motion information of the camera; and a processor coupled to the camera, the linear laser projector, and a sensor, the processor comprising: an image information processing module, configured to: Obtaining, according to the image information, three-dimensional coordinates of the object at each acquisition time in a camera coordinate system corresponding to each acquisition time; a motion information processing module, configured to obtain a camera for each acquisition time according to the motion information a positional relationship of the coordinate system with respect to the global three-dimensional coordinate system; and a three-dimensional reconstruction module for performing three-dimensional reconstruction on the object according to the three-dimensional coordinates and the positional relationship.

With reference to the second aspect, in a possible implementation manner, the processor further includes a calibration module, configured to calibrate internal parameters and external parameters of the camera; and then the image information processing module is specifically configured to collect according to each The image information collected at the moment and the inner and outer parameters are calculated, and the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time are calculated.

With reference to the second aspect, in a possible implementation, the motion information includes an acceleration, an angular velocity, and a heading, and the motion information processing module is specifically configured to calculate a dead reckoning algorithm according to the acceleration, the angular velocity, and the heading. The positional relationship of the camera coordinate system at each acquisition time relative to the global three-dimensional coordinate system.

With reference to the second aspect, in a possible implementation, the three-dimensional reconstruction module is specifically configured to: according to the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time, the object is corresponding to each acquisition The three-dimensional coordinates in the camera coordinate system at the moment are converted into three-dimensional coordinates in the global three-dimensional coordinate system.

With reference to the second aspect, in a possible implementation manner, the processor further includes a color map restoration module, configured to calculate a relative relationship between each camera coordinate system at different acquisition times, and establish according to the relative relationship A mapping relationship between image information of the objects at different acquisition times. With reference to the second aspect, in a possible implementation, the color map restoration module is specifically configured to map the three-dimensional coordinates of the point on the object scan line of the object at the il acquisition time to the i-th acquisition time. In the lower camera coordinate system, the image coordinates of the point at the ith acquisition time are calculated, and the pixel value of the point is obtained according to the image coordinates of the point at the ith acquisition time, where i is an arbitrary value greater than 1. Integer.

With reference to the second aspect, in a possible implementation manner, the processor further includes a fusion module, configured to fuse the result of the three-dimensional reconstruction with the result of establishing a mapping relationship.

In a third aspect, the present invention provides a mobile terminal comprising the above-described three-dimensional reconstruction apparatus.

Beneficial effect

The three-dimensional reconstruction method and device and the mobile terminal of the embodiment of the invention can continuously acquire the image information of the object from different angles and continuously acquire the motion information of the camera, and can realize fast and comprehensive three-dimensional scanning on the object according to the collected information. And reconstruction.

Further features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in FIG.

1 is a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention; FIG. 2 is a flow chart of a three-dimensional reconstruction method according to another embodiment of the present invention; FIG. 3 shows still another embodiment of the present invention. A flowchart of a three-dimensional reconstruction method provided; FIG. 4 is a schematic diagram showing a color map restoration method provided by still another embodiment of the present invention;

FIG. 5 is a block diagram showing the structure of a three-dimensional reconstruction apparatus according to an embodiment of the present invention; FIG. 6 is a schematic diagram showing the working principle of the line laser of the line laser projector of FIG. 4. FIG. 7 is a view showing another embodiment of the present invention. Schematic diagram of the three-dimensional scanning principle of the mobile terminal;

FIG. 8 is a structural block diagram of a three-dimensional reconstruction apparatus according to still another embodiment of the present invention; FIG. 9 is a structural block diagram of a three-dimensional reconstruction apparatus according to still another embodiment of the present invention. FIG. 10 is a structural block diagram of a three-dimensional reconstruction apparatus according to still another embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the drawings. The same reference numerals in the drawings denote the same or similar elements. The various aspects of the embodiments are shown in the drawings, and the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustrative." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous.

Further, in order to better illustrate the invention, numerous specific details are set forth in the Detailed Description. Those skilled in the art will appreciate that the present invention may be practiced without these specific details. In other instances, well-known methods, means, components, and circuits have not been described in detail in order to the present invention.

For ease of understanding, the basic principle of line structure light vision detection based on the present invention, the relationship between several coordinate systems and coordinates of each coordinate system according to the present invention will be first introduced.

First, the basic principle of line structure light vision detection

When performing three-dimensional scanning, the laser projector first projects a linear laser to the object, and the projected linear laser forms a laser projection plane. When the laser plane projection intersects the surface of the object, a bright scanning line is formed on the surface of the object, that is, light. article. Since the light bar contains all the surface points where the laser projection plane intersects the object, the three-dimensional coordinates (U _W ) of the corresponding surface points of the object can be obtained according to the coordinates of the light bar. The three-dimensional coordinates are mapped onto the laser projection plane, and a two-dimensional image of the light strip is obtained. The point on the two-dimensional image of the light strip is marked as ( ^M , , and can be calculated according to the point coordinates ( ^M , ^v ) of the two-dimensional image The three-dimensional coordinates of the surface point of the corresponding object ( uJ , which is the basic principle of line structure light vision detection. The above process of calculating the three-dimensional coordinates < , uj is shown in Equation 1.

< , 3^, ^ζ ) = /( ^Μ , ^ν ) Equation 1 2. Common coordinate systems and their relationship

(1) Image coordinate system

The digital image captured by the camera can be stored as an array in the computer. The value of each element (pixel, pixel) in the array is the brightness (grayscale) of the image point, and the Cartesian coordinates are defined on the image. The system u, v is used as the image coordinate system, and the pixels (u, v) on the image coordinate system are the number of columns and the number of rows of the pixel in the array, respectively, so (u, v) is the image coordinate system in units of pixels. coordinate of. Since the image coordinate system only indicates the number of columns and rows of pixels in the digital image, and the physical position of the pixel in the image is not represented by physical units, it is necessary to establish an image coordinate system expressed in physical units (for example, centimeters). The coordinates of the image coordinate system measured in physical units are represented by (X, y). In this coordinate system, the origin 0 is defined at the intersection of the camera's optical axis and the image plane, called the main point of the image, the origin is generally at the center of the image, and the X and y axes are usually associated with the u-axis of the Cartesian coordinate system, respectively. The V axis is parallel. If the coordinates of 0 in the u, v coordinate system are ( ^M. , ^v .), the physical size of each pixel in the X-axis and y-axis directions is dx, dy, respectively, then any pixel in the image is in two coordinates. The relationship of the coordinates under the system is as shown in Equation 2.

I I dx 0

0 1 / dy

0 0 1

Equation 2

( 2 ) camera coordinate system

The coordinates of the camera coordinate system are represented by (AA, ^z . The coordinate system is based on the optical center of the camera. The A axis and the axis are parallel to the X and y axes of the image coordinate system respectively. The ^z axis is the optical axis of the camera, and the image plane. Vertical The intersection of the optical axis and the image plane is the origin of the image coordinate system. The Cartesian coordinate system consisting of the camera optical center and the AAA axis is called the camera coordinate system. c ⁰ is the camera focal length f. The camera coordinate system and the image coordinate system The relationship can be expressed by Equations 3 and 4:

X - -

Z type 3

Equation 4 Equations 3 and 4 can be expressed as homogeneous equations and matrices as Equation 5:

Y _c

Z

z _c

1 formula 5

(3) Global three-dimensional coordinate system

Since the camera can be mounted in any position, you also need to select a reference coordinate system. Describe the position of the camera and describe the position of any object in the environment. In this application, the world coordinate system is selected as the reference coordinate system. The world coordinate system is also called the global three-dimensional coordinate system O _w -X _w Y _w Z _w ^ The coordinate system can be arbitrarily specified. FIG. 1 is a flow chart showing a three-dimensional reconstruction method provided by an embodiment of the present invention. As shown in FIG. 1, the three-dimensional reconstruction method mainly includes:

Step S100, projecting a linear laser to the object;

Step S110: continuously acquiring image information of an object illuminated by the linear laser from at least two angles, and continuously acquiring motion information of the camera;

Step S120: Obtain, according to the collected image information, a three-dimensional coordinate of the object in a camera coordinate system corresponding to each acquisition time at each acquisition time;

Step S130: Obtain a positional relationship between the camera coordinate system and the global three-dimensional coordinate system at each acquisition time according to the collected motion information;

Step S140: Perform three-dimensional reconstruction on the object according to the three-dimensional coordinates and the positional relationship. Specifically, the camera may be built in a three-dimensional reconstruction device, which may be a mobile terminal such as a smart phone, on which a line laser projector can be installed, and the line laser projector can be connected to an external interface of the mobile terminal, such as audio. The interface is connected. When the object needs to be scanned and reconstructed in three dimensions, the line laser projector controls the linear light source to emit a linear laser to the object, and the linear laser refers to the object that is projected onto the object through a laser projection plane. Form a bright line. Therefore, when a linear laser is emitted to an object, a laser projection plane is formed. The laser projection plane intersects the object to form a bright scan line, that is, a light strip. Then, the camera built in the three-dimensional reconstruction device continuously acquires image information of the scanning line of the reflected object for a predetermined length of time, for example, 0.5 seconds, and passes through the sensor built in the three-dimensional reconstruction device within the above-mentioned length of time. The motion information of the camera is continuously acquired, and the motion information may mainly include acceleration, angular velocity, heading, etc., and the three-dimensional reconstruction device may use the motion information to determine the position and posture of the camera in space. Then, the three-dimensional reconstruction device performs corresponding processing on the image information and the motion signal collected at each acquisition time, specifically, according to the image information collected at each acquisition time, obtaining an object at each acquisition time corresponding to each The three-dimensional coordinates of the camera coordinate system at the time of acquisition. For example, according to the image information collected at the time of T1 acquisition, the three-dimensional coordinates of the object in the camera coordinate system at the time of T1 acquisition can be obtained; according to the image information collected at the time of T2 acquisition, the camera of the object at the time of T2 acquisition is obtained. The three-dimensional coordinates in the coordinate system, and so on, you can get each The three-dimensional coordinates of the object at the acquisition time are in the camera coordinate system corresponding to each acquisition time. According to the motion information collected at each acquisition time, the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time is obtained. For example, according to the motion information collected at the time of T1 acquisition, the positional relationship of the camera coordinate system at the T1 acquisition time relative to the global three-dimensional coordinate system can be obtained; the T2 acquisition time can be obtained according to the motion information collected at the T2 acquisition time. The positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system, and so on, obtains the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at each acquisition time. In a possible implementation manner, the camera coordinate system when starting three-dimensional scanning and reconstruction may be set as a global three-dimensional coordinate system. Finally, according to the obtained three-dimensional coordinates of the object at each acquisition time in the camera coordinate system corresponding to each acquisition time and the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time, the global three-dimensional coordinates of the object are obtained. , thereby three-dimensional reconstruction of the object.

Preferably, the three-dimensional reconstruction device can be controlled to continuously acquire image information of the object around the object, thereby realizing continuous and omnidirectional image information of the collected object.

Further, in order to improve the speed and efficiency of the scan reconstruction, the acquisition of image information and the acquisition of motion information can be performed simultaneously.

In the three-dimensional reconstruction method of the embodiment of the present invention, the three-dimensional reconstruction device can continuously acquire image information of an object continuously collected from different angles and continuously acquire motion information of the camera, and can realize fast and comprehensive three-dimensional scanning and reconstruction of the object according to the collected information. .

FIG. 2 is a flow chart showing a three-dimensional reconstruction method provided by another embodiment of the present invention. Figure

2 The same steps as those in FIG. 1 have the same meanings. As shown in FIG. 2, the main difference between this embodiment and the previous embodiment is that, according to the collected image information, the object is corresponding to each acquisition time. Before the three-dimensional coordinates in the camera coordinate system of each acquisition time, the method further includes:

Step S200: Calibrate the internal parameters and the external parameters of the camera.

Correspondingly, according to the collected image information, obtaining the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time at each acquisition time may specifically include:

Step S210: Calculate, according to the image information collected at each acquisition time and the internal parameters and the external parameters, the three-dimensional coordinates of the object under the camera coordinate system corresponding to each acquisition time.

For the above step S200, before determining the three-dimensional coordinates of the object in the camera coordinate system, It is necessary to first calibrate the internal and external parameters of the camera. Among them, the internal parameter reflects the intrinsic property of the camera itself, and the external parameter represents the positional relationship between the module coordinate system and the camera coordinate system. The 0 _L coordinate plane of the module coordinate system is a laser projection plane orthogonal to the laser projection plane. The outer parameter can usually be represented by the rotation matrix R and the translation vector T, where R represents the rotation of the module coordinate system relative to the camera coordinate system, and R is a 3 x 3 orthogonal unit recorded as

Equation 6, where l, r ₄ , r ₇ ), ^ ₅ , and ( „ ₉ ) respectively represent the unit vector of the upper coordinate, axis, and axis of the module coordinate system <3⁄4 _ ^. T represents the camera coordinate system relative to the module coordinates. The flat of the system is a 3-dimensional translation vector, recorded as

Equation 7, where, and are the coordinates of the module coordinate system origin in the camera coordinate system. In the above step S210, in the process of continuously scanning the object by the three-dimensional reconstruction device, the collected image information is corrected according to the internal number, and the center of the light bar is calculated row by row to obtain an array of coordinates of the center of the light bar, that is, a group is obtained. The image coordinates of the center of the strip, according to the image coordinates of the center of each strip and the internal and external parameters of the calibration, calculate the coordinates mapped in the module coordinate system, and then calculate the camera corresponding to each acquisition time. The three-dimensional coordinates in the coordinate system. According to the image coordinates of the center of a group of light strips, the three-dimensional coordinates of the entire scan line of the object in the camera coordinate system can be calculated, and the object can be scanned continuously. The above method can be used to calculate the three-dimensional coordinates of the whole object in the camera coordinate system. The following method is described by taking a light bar center as an example. It is assumed that the coordinates of the center of a light bar calculated at time T1 are (x, y), and the coordinates of the module coordinate system at the time of T1 acquisition are ( = ⁰ ), and the coordinates of the camera coordinate system at the time of T1 acquisition are , , ), according to the formula

8 can calculate the coordinates (3⁄4, Λ, ) mapped in the module coordinate system of the T1 acquisition time by the coordinates (x, y) of the center of the light bar;

According to Equation 9, the coordinates ( , , converted in the module coordinate system can be converted to the coordinates of the camera coordinate system at the T1 acquisition time, ^z c).

By repeating the above calculation method, the three-dimensional coordinates of the entire object in the camera coordinate system at each acquisition time can be calculated.

In a possible implementation, the motion information includes an acceleration, an angular velocity, and a heading. The positional relationship between the camera coordinate system and the global three-dimensional coordinate system at each acquisition time may be obtained according to the collected motion information.

Step S220: According to the acceleration, the angular velocity, and the heading, the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time is calculated by using a dead reckoning algorithm.

Specifically, in the process of continuously scanning the object by the three-dimensional reconstruction device, the motion information of the camera at each acquisition time can be obtained by the sensor. The sensor can mainly include a three-axis acceleration sensor, a three-axis gyroscope, and a three-axis electronic compass. The three-axis accelerometer is mainly used to measure the three-axis acceleration of the camera. The three-axis gyroscope is mainly used to measure the three-axis angular velocity of the camera. The three-axis electronic compass is mainly used to measure the three-axis heading of the camera. For example, at the kth acquisition time, the triaxial acceleration sensor can measure the three-axis acceleration of the camera, the three-axis gyroscope can measure the three-axis angular velocity of the camera, and the three-axis electronic compass can measure the three-axis heading of the camera, and then utilize The dead reckoning algorithm can derive the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at the kth acquisition time. Specifically, at the kth acquisition time, the triaxial acceleration sensor can measure the three-axis acceleration of the camera respectively. ^ By performing double integration on the three-axis acceleration 'A, respectively, three-axis displacements x, y, _Z can be obtained _; By the difference, the three-axis displacement A, Ay, Az and the three-axis rotation angle ^Δ ^, ^{Δ of the} three-dimensional reconstruction device of two adjacent measurement moments can be obtained. The specific calculation steps are as shown in Equation 10-12.

v _x (k) = v _x (k-\) + ^*a _x (k-\)

v _y (k) = v _y (k-\) + At*a _y (k-\)

_{_{v z (k) = v z}} (ki) + At * a z (ki) of formula ₁₀

Ax = x(k) - x(k -\) = At*v _x (k-1)

Ay = y(k)-y(k-\) = At*v _y (k-\)

Az^z(k)-z(kl)^At^v _z (kl) Αφ = φ(Κ) - φ(}ί - ί) = At * - 1)

Αθ = 0(k) - 0(k _ 1) = At * ί3⁄4 (A _ 1)

Αφ = φ{Κ) - φ{]ί -\) = ^* k- 1) where 12 is the sampling interval, which is the interval between the kth acquisition time and the k-1th acquisition time.

The relative positional relationship of the camera coordinate system of two adjacent measurement moments can be expressed by rotation and translation, which is specifically expressed by the following formula 14.

Equation 13 r = cosA^cosA6*

r = cosA^sinA^sinA^-sinA^

r ₁₃ = cos Δ ^ sin Δ ^ cos Αφ - sin Αφ sin

r ₂₁ = sinA^cosA^

r ₂₂ = sin Αφ sin sin Αφ + cosA^cosA^

r ₂₃ = sin Αφ sin cos Αφ + cos sin

-sinA^ Equation 14

Indicates the coordinates of the object in the camera coordinate system at the kl acquisition time and the kth acquisition time.

Through multiple iterations, the relative positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at the current measurement time can be obtained. Similarly, according to the measured acceleration and angular velocity of the camera at each acquisition time, the positional relationship of the camera coordinate system corresponding to the global three-dimensional coordinate system corresponding to each acquisition time can also be calculated by using the dead reckoning algorithm. In this way, the relative positional relationship of the camera coordinate system at different acquisition times can be obtained, thereby obtaining a continuous sequence of camera coordinate systems. Among them, the three-axis rotation angle can be calculated from the three-axis angular velocity, or the three-axis rotation angle can be derived from the three-axis heading.

In a possible specific implementation, the foregoing three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship includes:

Step S230: Convert the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time into the three-dimensional coordinates in the global three-dimensional coordinate system according to the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time. For the above step S230, in step S210, the three-dimensional coordinates of the object at each acquisition time in the camera coordinate system corresponding to each acquisition time are obtained, and in step S220, the positional relationship between the camera coordinate system and the global three-dimensional coordinate system of each acquisition time is obtained. Then, the three-dimensional coordinates of the object in the global three-dimensional coordinate system at each acquisition time can be converted, thereby performing three-dimensional reconstruction of the object.

In the three-dimensional reconstruction method of the embodiment of the present invention, the three-dimensional reconstruction device can continuously collect image information of the object from different angles, and obtain the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time and the continuous acquisition camera at each acquisition time. The motion information obtains the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at each acquisition time, thereby realizing fast and comprehensive three-dimensional scanning and reconstruction of the object.

FIG. 3 is a flow chart showing a three-dimensional reconstruction method provided by still another embodiment of the present invention. 3 is the same as the steps of FIG. 1 and FIG. 2 having the same meaning. As shown in FIG. 3, the main difference between this embodiment and the previous embodiment is that the pair is based on the three-dimensional coordinates and the positional relationship. After the object is three-dimensionally reconstructed, the method further includes: Step S240: Calculate a relative relationship between camera coordinate systems at different acquisition times, and establish a mapping relationship between image information of objects at different acquisition times according to the relative relationship.

Specifically, as shown in FIG. 4, according to the above step S220, a continuous sequence of camera coordinate systems can be obtained, that is, the relative positional relationship of the camera coordinate system at any two acquisition times can be obtained, for example, two acquisition times T0 can be obtained. , T1, the relative positional relationship between the camera coordinate systems of the two acquisition moments can use the rotation matrix and the translation matrix ^G. To represent, where, is a 3 * 3 rotation matrix, ^G . A 3* 1 translation matrix. At the time of TO acquisition, a certain measured point on the scanning line on the object is covered by the linear laser, and its three-dimensional coordinate in the camera coordinate system is ^p . 0. , z. At the time of T1 acquisition, the measured point shows its true color, ^^ sets the three-dimensional coordinates of the measured point in the camera coordinate system at the time of T1 acquisition as ( ^χ ι, , ^ζ ι), the measured point is The mapping relationship in the camera coordinate system at the two acquisition moments is as shown in Equation 15.

According to Equation 10, it can be obtained that if the image coordinate of the measured point at time T1 is ^SX M , then ( ^ can be substituted into Equation 3 and Equation 4 to calculate ¹ ^(Α, , finally, according to Equation 2 Obtaining the pixel value of the coordinate, thereby obtaining the true color information of the measured point, that is, realizing the color restoration of the measured point, and traversing the measured point on the scanning line to realize the whole scanning The color reproduction of the line.

Similarly, record the coordinates of the camera coordinate system of a measured point on the scan line of the object at the time of T1 acquisition. If the relative positional relationship between the two camera coordinate systems at the time of T1 and Τ2 acquisition, use the rotation matrix and the translation matrix ^G. When i is expressed, then, ^G i replaces ^, ^G respectively. , Replace a 0 of the camera coordinate system coordinates of a measured point on the scan line of the object at the T1 acquisition time. , y. ,z. According to Equation 10, the coordinates of a measured point at the T1 acquisition time in the camera coordinate system at the time of T2 acquisition can also be obtained, thereby obtaining the image coordinates of the measured point at the time of T2 acquisition, and further can be obtained according to Equation 2. The pixel value of the image coordinates, thereby obtaining the true color information of the measured point, that is, the color restoration of the measured point can be realized, and the color of the entire scan line can be realized by traversing the point on the scan line. And so on, the above method can be used to achieve the color reproduction of the entire object.

Preferably, in a possible specific implementation, the two acquisition moments selected in step S240 may be adjacent acquisition moments. Thereby improving the color reproduction accuracy of the measured point.

In a possible implementation manner, after the mapping relationship between the image information of the objects at different acquisition moments is established, the method further includes:

Step S250: merging the result of the three-dimensional reconstruction with the result of establishing a mapping relationship. Specifically, the result of the three-dimensional reconstruction, that is, the position corresponding to each measured point on the object and the result of establishing a mapping relationship, that is, the color of each measured point in the color restoration, thereby realizing three-dimensional reconstruction and color restoration of the object .

The three-dimensional reconstruction method provided in this embodiment can realize the restoration of the color texture of the object by establishing the mapping relationship of the image information of the adjacent acquisition time.

FIG. 5 is a structural block diagram of a three-dimensional reconstruction apparatus according to an embodiment of the present invention. As shown in FIG. 5, the three-dimensional reconstruction apparatus 100 mainly includes a camera 41, a sensor 42, a processor 43, and a line laser projector 44. The above three-dimensional reconstruction device can be a mobile terminal. The line laser projector 44 is mainly used to project a linear laser to the object; the camera 41 is mainly used for continuously collecting image information of the object illuminated by the linear laser from at least two angles; the sensor 42 is mainly used for continuously acquiring the motion information of the camera 41; The processor 43 is connected to the camera 41 and the sensor 42 and the line laser projector 44, and may include the following modules: an image information processing module 431, a motion information processing module 432, and a three-dimensional reconstruction module 433. The image information processing module 431 is mainly configured to obtain, according to the image information collected by the camera 41, three-dimensional coordinates of the object at each acquisition time in a camera coordinate system corresponding to each acquisition time; The module 432 is mainly configured to obtain a positional relationship between the camera coordinate system and the global three-dimensional coordinate system at each acquisition time according to the motion information; the three-dimensional reconstruction module 433 is mainly configured to perform three-dimensionality on the object according to the three-dimensional coordinates and the positional relationship. reconstruction.

Specifically, the camera 41, the sensor 42 and the processor 43 can all be built in the three-dimensional reconstruction device 100, and the three-dimensional reconstruction device 100 can be mounted with a line laser projector 44 for a mobile terminal such as a smartphone. The line laser projector 44 can be connected to an external interface of the mobile terminal such as an audio interface. A three-dimensional scanning control switch can also be provided to control the three-dimensional reconstruction device for three-dimensional scanning. When three-dimensional scanning and reconstruction of the object is required, the three-dimensional scanning is started by the three-dimensional scanning control switch, and the linear laser projector 44 controls the linear light source to the object emission line. The laser forms a laser projection plane.

Fig. 6 is a schematic view showing the working principle of the line laser of the line laser projector 44. As shown in Fig. 6, the line laser projector 44 mounted on the smartphone is taken as an example to describe the working principle of the line laser. The line laser projector 44 is mounted on the smart phone and connected to an external interface of the smart phone, such as an audio interface. When three-dimensional scanning and reconstruction of the object is required, the three-dimensional scanning control switch is used to start the three-dimensional scanning, the left channel or the right of the smart phone. The channel generates a square wave of a certain frequency, and the micro-transformer 441 and the rectifier 442 built in the line laser projector 44 supply power to the laser diode 443, and then the cylindrical mirror 444 converts the laser light emitted from the laser diode 443 into a linear laser, and the laser projection The plane intersects the object to form a bright scan line, that is, the light strip, and the three-dimensional scan can be started. The method includes: the camera 41 collects image information of the reflected object, and the sensor 42 collects motion information of the camera.

Preferably, the three-dimensional reconstruction device can be controlled to wrap around the object, so that the camera 41 can continuously acquire image information of the object, thereby realizing continuous and omnidirectional image information of the collected object.

Further, to improve the speed and efficiency of the scan reconstruction, the image information acquired by the camera 41 and the sensor 42 can be simultaneously acquired.

FIG. 7 is a schematic diagram of the principle of three-dimensional scanning and reconstruction of a mobile terminal. As shown in Figure 7, the mobile terminal can mainly include a camera, a line laser projector, an audio interface, and a sensor. The above line laser projector can be externally connected to the audio interface. The mobile terminal can be used to realize the full, continuous and fast image information of the collected object and the motion information of the camera, thereby calculating the coordinates of the object in the global three-dimensional coordinate system, and finally realizing the three-dimensional scanning and reconstruction of the object.

The three-dimensional reconstruction device of the embodiment of the invention can continuously collect image information of an object collected from different angles and continuously acquire motion information of the camera, and can be based on the collected information. Fast and omni-directional 3D scanning and reconstruction.

FIG. 8 is a structural block diagram of a three-dimensional reconstruction apparatus according to still another embodiment of the present invention. 8 and FIG. 5 have the same functions, and as shown in FIG. 8, the main difference between this embodiment and the previous embodiment is that the processor 43 of the three-dimensional reconstruction apparatus 200 of the present embodiment may further include a calibration module. 434, the calibration module 434 is mainly used to calibrate the internal parameters and the external parameters of the camera 41. Correspondingly, the image information processing module 431 can be specifically configured to convert the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time according to the image information collected by each camera camera 41 and the internal parameters and the external parameters. .

Specifically, the image information processing module 431 first needs to calibrate the internal parameters and the external parameters of the camera 41 by using the calibration module 434 before determining the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time. The definitions of the internal parameters and the external parameters can be referred to the related description of the above embodiments. After that, the image information processing module 431 can correct the collected image information by internal parameters, identify and calculate the image coordinates of the center of the light bar, use the image coordinates of the center of the light bar, and calibrate the internal parameters and external parameters according to the calibration module 434. Calculate the coordinates of the object under the laser projection plane and the three-dimensional coordinates of the object in the camera coordinate system at the corresponding acquisition time.

In a possible implementation, the motion information includes acceleration, angular velocity, and heading. The motion information processing module 432 may be specifically configured to calculate each acquisition time by using a dead reckoning algorithm according to the acceleration, angular velocity, and heading. The positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system.

Specifically, in the process of continuously scanning the object by the three-dimensional reconstruction device, the three-dimensional reconstruction device obtains the motion information of the camera 41 at each acquisition time by the sensor 42. The sensor 42 body may include a three-axis acceleration sensor 421, a three-axis gyroscope 422, and a three-axis electronic compass 423. The three-axis acceleration sensor 421 is mainly used to measure the acceleration of the camera 41, the three-axis gyroscope 422 is mainly used to measure the angular velocity of the camera 41, and the three-axis electronic compass 423 is used to measure the heading of the camera 41. Then, based on the measured acceleration, angular velocity and heading of the camera 41, the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time is calculated by using the dead reckoning algorithm, so that the camera coordinate system at different acquisition times can be obtained. The relative positional relationship, resulting in a continuous sequence of camera coordinate systems.

In a possible implementation manner, the three-dimensional reconstruction module 433 may be specifically configured to: according to the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system, each of the collection time objects is in a camera coordinate system corresponding to each acquisition time. Convert three-dimensional coordinates into global three-dimensional coordinates The three-dimensional coordinates under the standard are used to perform three-dimensional reconstruction of the object.

The three-dimensional reconstruction device of the embodiment of the invention can obtain the image information of the object continuously collected from different angles, obtain the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition time, and the motion of the continuous acquisition camera at each acquisition time. The information obtains the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at each acquisition time, thereby realizing fast and comprehensive three-dimensional scanning and reconstruction of the object.

FIG. 9 is a structural block diagram of a three-dimensional reconstruction apparatus according to still another embodiment of the present invention. 9 is the same as the components of FIG. 8 and FIG. 5. The main difference between the present embodiment and the previous embodiment is that the processor 43 of the three-dimensional reconstruction apparatus 300 of the present embodiment further includes The color map restoration module 435 is mainly configured to calculate a relative relationship between each camera coordinate system at different acquisition times, and establish a mapping relationship of image information of the objects at different acquisition times according to the relative relationship.

Specifically, according to the processing result of the motion information processing module 432, a continuous camera coordinate system sequence can be obtained, and the camera coordinate system of the two acquisition moments, the relative relationship between the two camera coordinate systems, and the realization of the measured point can be obtained. For the color reproduction method, reference may be made to the related description in the above embodiments.

Preferably, in a possible specific implementation, the two acquisition moments selected by the color map restoration module 435 may be adjacent moments, thereby implementing color reduction of the measured point, thereby being restored by the color of the entire scan line. In turn, the color reproduction of the entire object is achieved.

In a possible specific implementation, the processor 43 may further include a fusion module 436, and the fusion module 436 is mainly used to combine the result of the three-dimensional reconstruction with the result of establishing a mapping relationship.

The three-dimensional reconstruction apparatus provided in this embodiment can realize the restoration of the color texture of the object by establishing the mapping relationship of the image information of the adjacent acquisition time.

It should be noted that the camera, the sensor and the processor may be built in the three-dimensional reconstruction device, and the line laser projector may be installed on the three-dimensional reconstruction device, and the three-dimensional reconstruction device may be a mobile terminal, such as a smart phone, a PAD or the like. Thus, the three-dimensional reconstruction device of the present application has the advantage of portability.

FIG. 10 is a structural block diagram of a three-dimensional reconstruction apparatus according to still another embodiment of the present invention. The three-dimensional reconstruction device 700 may be a host server having a computing capability, a personal computer PC, or a portable computer or terminal that can be carried. The specific embodiment of the present invention is not The specific implementation of the arithmetic node is limited.

The three-dimensional reconstruction apparatus 700 includes a processor 710, a communications interface 720, a memory array 730, and a bus 740. The processor 710, the communication interface 720, and the memory 730 complete communication with each other through the bus 740.

The communication interface 720 is for communicating with a network element, wherein the network element includes, for example, a virtual machine management center, shared storage, and the like.

The processor 710 is for executing a program. The processor 710 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

The memory 730 is used to store files. Memory 730 may include high speed RAM memory and may also include non-volatile memory, such as at least one disk memory. Memory 730 can also be a memory array. Memory 730 may also be partitioned, and the blocks may be combined into virtual volumes according to certain rules.

In one possible implementation, the above program may be a program code that includes computer operating instructions. This program can be used to perform the following steps:

Projecting a linear laser to the object;

Collecting image information of the object illuminated by the linear laser from at least an angle, and continuously acquiring motion information of the camera;

Obtaining, according to the image information, three-dimensional coordinates of the object at each acquisition time in a camera coordinate system corresponding to each acquisition time;

Obtaining, according to the motion information, a positional relationship of a camera coordinate system at each acquisition time with respect to a global three-dimensional coordinate system;

The object is three-dimensionally reconstructed according to the three-dimensional coordinates and the positional relationship.

In a possible implementation manner, before the acquiring, according to the image information, the three-dimensional coordinates of the object in the camera coordinate system at the respective acquisition time, the program further includes: calibrating the inside of the camera The parameter and the external parameter; then, according to the image information, obtaining the three-dimensional coordinates of the object at each acquisition time in the camera coordinate system corresponding to each acquisition time comprises: the image information collected according to each collection time And the inner parameter and the outer parameter, and the three-dimensional coordinates of the object under the camera coordinate system corresponding to each acquisition time are converted. In a possible implementation manner, the motion information includes an acceleration, an angular velocity, and a heading, and according to the motion information, obtaining a positional relationship of a camera coordinate system at each acquisition time relative to a global three-dimensional coordinate system, including: according to the acceleration, The angular velocity and the heading are calculated by using the dead reckoning algorithm to calculate the positional relationship of the camera coordinate system with respect to the global three-dimensional coordinate system at each acquisition time.

In a possible implementation, the three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship includes: according to a positional relationship of a camera coordinate system of each acquisition time relative to a global three-dimensional coordinate system, The three-dimensional coordinates of the object under the camera coordinate system corresponding to each acquisition time are converted into three-dimensional coordinates under the global three-dimensional coordinate system.

In a possible implementation manner, after the three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship, the method further includes: calculating a relative relationship between camera coordinate systems at different acquisition times, according to the relative relationship The relationship establishes a mapping relationship between image information of the objects at different acquisition times.

In a possible implementation, the mapping relationship between the image information of the object at different acquisition times according to the relative relationship includes: setting a point on the scan line of the object at the i-1th acquisition time at the camera The three-dimensional coordinates in the coordinate system are mapped to the camera coordinate system at the ith acquisition time, and the image coordinates of the point at the ith acquisition time are calculated, and the points are obtained according to the image coordinates of the point at the ith acquisition time. Pixel value, where i is any integer greater than one.

In a possible implementation manner, after the mapping relationship between the image information of the objects at different acquisition times is established, the method further includes: combining the result of the three-dimensional reconstruction with the result of establishing the mapping relationship.

Those of ordinary skill in the art will appreciate that the various exemplary elements and algorithm steps in the embodiments described herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can select different methods for implementing the described functions for a particular application, but such implementation should not be considered to be outside the scope of the present invention.

If the function is implemented in the form of computer software and sold or used as a stand-alone product, it is considered to some extent that all or part of the technical solution of the present invention (for example, a part contributing to the prior art) is It is embodied in the form of computer software products. The computer software product is typically stored in a computer readable storage medium, including a number of instructions In order to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The foregoing storage medium includes various media that can store program codes, such as a USB flash drive, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the claims.

Claims

Rights request

1. A three-dimensional reconstruction method, characterized by including:

Project a linear laser onto an object;

Continuously collect image information of the object illuminated by the linear laser from at least two angles, and continuously collect motion information of the camera;

According to the image information, the three-dimensional coordinates of the object in the camera coordinate system corresponding to each collection moment are obtained;

According to the motion information, the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at each acquisition moment is obtained; and

Three-dimensional reconstruction of the object is performed based on the three-dimensional coordinates and the positional relationship.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, according to the image information, before obtaining the three-dimensional coordinates of the object in the camera coordinate system corresponding to each collection time at each collection time, The method also includes:

Calibrate the internal parameters and external parameters of the camera;

Then, obtaining the three-dimensional coordinates of the object at each collection time in the camera coordinate system corresponding to each collection time based on the image information includes:

According to the image information collected at each collection moment and the internal parameters and external parameters, the three-dimensional coordinates of the object in the camera coordinate system corresponding to each collection moment are calculated.

3. The three-dimensional reconstruction method according to claim 2, characterized in that the motion information includes acceleration, angular velocity and heading, and based on the motion information, the camera coordinate system at each acquisition time relative to the global three-dimensional coordinate system is obtained The positional relationships include:

Based on the acceleration, angular velocity and heading, the dead reckoning method is used to calculate the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at each acquisition moment.

4. The three-dimensional reconstruction method according to claim 3, wherein the three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship includes:

According to the positional relationship between the camera coordinate system at each acquisition moment and the global three-dimensional coordinate system, the three-dimensional coordinates of the object in the camera coordinate system corresponding to each acquisition moment are converted into three-dimensional coordinates in the global three-dimensional coordinate system.

5. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that, during the three-dimensional reconstruction of the object according to the three-dimensional coordinates and the positional relationship, Finally, the method also includes:

Calculate the relative relationship between the camera coordinate systems at different collection times, and establish a mapping relationship between the image information of the object at different collection times based on the relative relationship.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: establishing a mapping relationship of image information of the object at different acquisition times according to the relative relationship includes: scanning the object at the i-th acquisition time The three-dimensional coordinates of the points on the line in the camera coordinate system are mapped to the camera coordinate system at the i-th collection time, and the image coordinates of the points at the i-th collection time are calculated, based on the coordinates of the points at the i-th collection time. The image coordinates obtain the pixel value of the point, where i is any integer greater than 1.

7. A three-dimensional reconstruction device, characterized by including:

Line laser projector, used to project linear laser to objects;

A camera for continuously collecting image information of the object illuminated by the linear laser from at least two angles;

A sensor for continuously collecting movement information of the camera; and

Processor, connected to the camera, linear laser projector and sensor, the processor includes:

An image information processing module, configured to obtain the three-dimensional coordinates of the object at each collection moment in the camera coordinate system for each collection moment based on the image information;

A motion information processing module, configured to obtain the positional relationship of the camera coordinate system relative to the global three-dimensional coordinate system at each collection moment based on the motion information; and

A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction of the object based on the three-dimensional coordinates and the position relationship.

8. The three-dimensional reconstruction device according to claim 7, wherein the processor further includes a calibration module for calibrating the internal parameters and external parameters of the camera;

The image information processing module is specifically configured to calculate the position of the object in the camera coordinate system corresponding to each collection time based on the image information collected at each collection time and the internal parameters and external parameters. three-dimensional coordinates.

9. The three-dimensional reconstruction device according to claim 8, wherein the motion information includes acceleration, angular velocity and heading, and the motion information processing module is specifically configured to use dead reckoning based on the acceleration, angular velocity and heading. method to calculate each acquisition time The positional relationship between the engraved camera coordinate system and the global three-dimensional coordinate system.

10. The three-dimensional reconstruction device according to claim 9, wherein the three-dimensional reconstruction module is specifically configured to reconstruct the object corresponding to the position of the camera coordinate system at each acquisition time relative to the global three-dimensional coordinate system. The three-dimensional coordinates in the camera coordinate system at each acquisition moment are converted into three-dimensional coordinates in the global three-dimensional coordinate system.

1 1. The three-dimensional reconstruction device according to any one of claims 7 to 9, characterized in that the processor further includes a color mapping restoration module for calculating the color difference between the camera coordinate systems at different acquisition times. Relative relationship: establishing a mapping relationship between the image information of the object at different collection moments based on the relative relationship.

12. The three-dimensional reconstruction device according to claim 11, wherein the color mapping restoration module is specifically used to map the three-dimensional coordinates of the points on the object scanning line at the i-1th acquisition time in the camera coordinate system. Go to the camera coordinate system at the i-th collection time, calculate the image coordinates of the point at the i-th collection time, and obtain the pixel value of the point based on the image coordinates of the point at the i-th collection time, where, i is any integer greater than 1.

13. A mobile terminal, characterized in that, the mobile terminal includes: as claimed in claim

The three-dimensional reconstruction device described in any one of 7-12.