TWI692968B - 3d model establishing device and calibration method applying to the same - Google Patents

Abstract

A 3D model constructing device, includes a camera and a wearable display. The camera is configured to obtain a plurality of first frames, a second frame and a depth information. The wearable display is coupled to the camera, and includes a display unit, a processing unit, a storage unit and a projection unit. The storage unit is configured to store a first module and a second module. When the first module is performed by the processing unit, cause the processing unit to calculate a first pose of the wearable display. When the second module is performed by the processing unit, cause the processing unit to calculate a 3D model according to the first frames, the depth information, the first pose and a plurality of calibration parameters, and to update the 3D model according to the second frame. The projection unit is configured to project the 3D model and the second frame onto the display unit according to the first pose for being displayed with a real image on the display unit.

Description

Three-dimensional modeling device and calibration method applied thereto

The invention relates to a three-dimensional modeling device and method

Three-dimensional scene modeling is a popular research topic in the field of computer vision. Generally, multiple frames and depth information are used to construct a complete three-dimensional scene model. The pros and cons of the 3D scene model depends on whether the correct pose of the frame can be estimated. However, in fact, pose poses are prone to pose drift due to error accumulation or insignificant feature points. Most of the existing technologies use optimization algorithms to reduce these errors, but the results are limited.

The object of the present invention is to provide a three-dimensional modeling device and a calibration method applied thereto, which can establish a three-dimensional model of an environmental scene.

One aspect of the present invention discloses a three-dimensional modeling device, including a camera and a wearable display. The camera is used to obtain a plurality of first image frames, a second image frame and a depth information. The wearable display is coupled to the camera. The wearable display includes a display unit, a processing unit, a storage unit, and a projection unit. The storage unit is coupled to the processing unit and is used to store a first module and a second module. When the first module is executed by the processing unit, the processing unit is caused to calculate a first pose of the wearable display. When the second module When executed by the processing unit, the processing unit is caused to calculate a three-dimensional model based on the first image frames, depth information, first pose and a plurality of calibration parameters, and update the three-dimensional model based on the second image frame. The projection unit is coupled to the processing unit, and is used to project the three-dimensional model and the second image frame to the display unit according to the first posture to display together with an actual image on the display unit.

Another aspect of the present invention discloses a calibration method applied to a three-dimensional modeling device, including: making the three-dimensional modeling device move linearly in a plurality of directions, and respectively obtaining a plurality of wearable displays of the three-dimensional modeling device corresponding to a plurality of directions A first moving speed vector and a camera of the three-dimensional modeling device correspond to a plurality of second moving speed vectors in various directions; a rotation parameter is calculated according to the plural first moving speed vectors and the second moving speed vectors; Model device, and record a plurality of pieces of rotation amount information, wherein the plurality of pieces of rotation amount information respectively record the rotation amount of the wearable display at the time point when the angular velocity of the 3D modeling device is greater than a first threshold, The displacement amount of the time point relative to the previous time point, the displacement amount of the camera relative to the previous time point at that time point, and the rotation axis unit vector at that time point is calculated according to the rotation amount of the time point; and based on the plurality of rotations The displacement information is calculated based on the quantity information and the unit vector of the rotation axis.

In order to have a better understanding of the above and other aspects of the present invention, the following examples are specifically described in conjunction with the accompanying drawings as follows:

10: 3D modeling device

102: Camera

104: Wearable display

1041: Processing unit

1043: Storage unit

1045: Projection unit

1047: Input unit

1049: display unit

M1: the first module

M2: second module

S301~S307: Steps

FIG. 1 is a block diagram of a three-dimensional modeling device according to an embodiment of the invention.

FIG. 2 is a schematic diagram of a three-dimensional modeling device according to an embodiment of the invention.

FIG. 3 is a flowchart of a calibration parameter generation method according to an embodiment of the invention.

Please refer to FIG. 1, which illustrates a block diagram of a three-dimensional modeling device according to an embodiment of the invention. Please also refer to FIG. 2, which is a schematic diagram of a three-dimensional modeling device according to an embodiment of the invention. The three-dimensional modeling device 10 includes a camera 102 and a wearable display 104. The three-dimensional modeling device 10 is used to create a three-dimensional model of an environmental scene. In one embodiment, the three-dimensional modeling device 10 is an Augmented Reality (AR) device. In other embodiments, the three-dimensional modeling device 10 may be a virtual reality VR (Virtual Reality, VR) device, a substitutional reality (Substitutional Reality) device, or a mixed reality (MR) device.

The camera 102 may be an RGB-D camera (RGB-D camera). In addition to capturing the color information of the environment, the camera 102 can also capture the depth information of the environment with its own depth sensor. The camera 102 is used to obtain a plurality of first picture frames, a second picture frame and a depth information, wherein the second picture frame is the latest picture frame currently obtained, and the first picture frame is before the second picture frame Get multiple picture frames. As shown in FIG. 2, the camera 102 is detachably or fixedly installed on the wearable display 104.

The wearable display 104 is coupled to the camera 102 so that signals and data can be transmitted between the two. The wearable display 104 includes a display unit 1049, a processing unit 1041, a storage unit 1043, a projection unit 1045, and an input unit 1047. The configuration of the display unit 1049 is shown in FIG. 2. The display unit 1049 may be a head-mounted display (HMD). The processing unit 1041 may include one or Multiple dedicated integrated circuit chips or general-purpose processors are used to perform 3D modeling operations. The storage unit 1043 may be a non-volatile memory, such as a NAND flash memory, for storing applications and data. The storage unit 1043 includes a first module M1 and a second module M2. The first module M1 and the second module M2 are computer readable programs and include a plurality of computer readable instructions. When the first module M1 is executed by the processing unit 1041, it causes the processing unit 1041 to calculate a first pose of the wearable display 104. The processing unit 1041 may calculate the first posture according to one or more sensors (not shown) included in the wearable display 104, and the calculation method may use a method commonly used in the art, so details are not described here. When the second module M2 is executed by the processing unit 1041, it will cause the processing unit 1041 to calculate a three-dimensional model based on the plurality of first frames, depth information, first pose and the plurality of calibration parameters, and update according to the second frame The details of the calibration parameters of the three-dimensional model will be described below. In short, the first module M1 is used to calculate the current pose of the wearable display 104, and the second module M2 is used to create a three-dimensional model of the environment scene.

The projection unit 1045 is coupled to the processing unit 1041. The projection unit 1045 is used to project the three-dimensional model and the second image frame created by the processing unit 1041 to the display unit 1049. In the application of mixed reality, the user can see an actual image through the display unit 1049, that is, the actual environment scene. The three-dimensional model and the second frame projected onto the display unit 1049 can be presented on the display unit 1049 together with the actual image.

Since the camera 102 and the wearable display 104 have their own coordinate systems, the wearable display 104 must first convert the coordinate system (or calibration) when using the first frame, second frame, and depth information from the camera 102 Standard) so that the 3D model does not deviate significantly from the actual scene. The calibration parameters mentioned above are the basis for the coordinate system conversion.

The input unit 1047 is coupled to the processing unit 1041. The input unit 1047 may be an input controller, which may be connected to the display unit 1049 to receive input commands from the user and transmit the input commands to the processing unit 1041. The processing unit 1041 can adjust the second image frame and/or the three-dimensional model in response to the user's input command. Details about this part will be explained below.

Please refer to FIG. 3, which is a flowchart of a method for generating a calibration parameter according to an embodiment of the present invention. The calibration parameters include a rotation parameter R and a displacement parameter T, where R and T may be in the form of a matrix.

In step S301, the three-dimensional modeling device 10 is linearly moved in a plurality of directions, and respectively obtains a plurality of first movement speed vectors corresponding to each direction of the wearable display 104, and a plurality of second movement speed vectors corresponding to each direction of the camera 102 . For example, a three-axis platform (X-axis, Y-axis, and Z-axis, the three axes are perpendicular to each other) may be prepared, and the modeling device 10 is fixed on a carrying unit of the three-axis platform. First, operate the carrying unit to move the modeling device 10 in the X-axis direction to obtain the first movement speed vector v _{1 of the} wearable display 104 corresponding to the X-axis direction and the second movement speed vector of the camera 102 corresponding to the X-axis direction c ₁ , get v ₁ = Rc ₁ , where R is the rotation parameter. Next, the carrying unit is operated to move the modeling device 10 in the Y-axis direction to obtain the first movement speed vector v _{2 of the} wearable display 104 corresponding to the Y-axis direction and the second movement speed vector of the camera 102 corresponding to the X-axis direction c ₂ , get v ₂ = Rc ₂ . Then, the carrying unit is operated to move the modeling device 10 along the Z-axis direction to obtain the first moving speed vector v ₃ corresponding to the Z-axis direction of the wearable display 104 and the second moving speed corresponding to the X-axis direction of the camera 102 Vector c ₃ gives v ₃ = Rc ₃ .

When acquiring the first moving speed vector, it can be measured by a sensor provided in the wearable display 104. On the other hand, when acquiring the second moving speed vector, it can be measured by a sensor provided in the camera 102. The obtained first moving speed vector and second moving speed vector may be provided to the processing unit 1041 of the wearable display 104 or an external computer (not shown) for subsequent calculation. The so-called "external computer" refers to a computer that is not included in the three-dimensional modeling device 10 but is externally coupled to the three-dimensional modeling device 10, such as a backpack computer.

Step S303: Calculate the rotation parameter according to the first moving speed vector and the second moving speed vector. In the above example, R=[ v ₁ v ₂ v ₃ ][ c ₁ c ₂ c ₃ ] ^-1 .

It is worth mentioning that the direction of movement is not limited to three, but can be expanded to N (N>3) to improve accuracy. In the case of N moving directions, the relationship between the obtained first moving speed vector and the second moving speed vector can be organized as:

By least square method (least square) can solve the above equation, get R as follows:

After the calculation of the rotation parameter R is completed, the displacement parameter T is then calculated.

Step S305: Move the three-dimensional modeling device 10 arbitrarily and record the information of the rotation amount of the plural pens, wherein the information of the rotation amount of the plural pens respectively record the rotation amount of the wearable display 104 at the time point t when the angular velocity is greater than a first threshold R _t , the displacement T _{vt of the} wearable display 104 relative to the previous time at this time t, the displacement T _{ct of the} camera 102 relative to the previous time at this time t, and according to the time t The rotation amount R _t calculates the rotation axis unit vector ω _{t at} the time point _t .

In step S307, the displacement parameter T is calculated according to the rotation amount information and the rotation axis unit vector. For example, suppose there are n pieces of rotation information collected, where n is an integer greater than or equal to 2. When n is an odd number, the relationship between the rotation amount information, the rotation axis unit vector and the displacement parameter can be expressed by the following formula:

When n is an even number, the relationship between the rotation amount information, the rotation axis unit vector and the displacement parameter can be expressed by the following formula:

Where I is the identity matrix, △T _t = T _ct - T _vt , [ ω _t ] _X is a custom operator, defined as follows:

Where ω _{x t} is the component of the unit rotation axis vector ω _t on the X axis, ω _{y t} is the component of the unit rotation axis vector ω _t on the Y axis, and ω _{z t} is the unit rotation axis vector ω _t on the Z axis Weight.

It should be noted that the rotation amount R _t here is different from the rotation parameter R calculated in step S303.

By Psudo Inverse solving the above relationship, the displacement parameter T can be obtained.

In one embodiment, the processing unit 1041 can update the rotation parameter R and the displacement parameter T in real time by managing a first buffer and a second buffer, such as a first in first out (FIFO) buffer . For example, the first buffer is used to store related information for calculating the rotation parameter R, and the second buffer is used for storing related information for calculating the displacement parameter T. When the first buffer is full (for example, n pieces of related information have been accumulated), the processing unit 1041 calculates the rotation parameter R according to the information stored in the first buffer. The calculation method may be the method described above. Each time a new piece of information is stored in the first buffer, the oldest piece of information is discarded, and the processing unit 1041 can calculate the rotation parameter R again to update the rotation parameter R. Similarly, the displacement parameter T can be updated in a similar manner in real time.

After calculating the calibration parameters, they can be recorded in the second module M2. When the second module M2 is executed by the processing unit 1041, the processing unit 1041 can provide the first image provided by the camera 102 according to these calibration parameters. Frames, second picture frames and depth information to wear The coordinate system of the display 104 indicates that the three-dimensional model is created/updated based on the first pose and the converted first image frame, second image frame, and depth information.

In one embodiment, the user determines whether the projected second frame matches the actual image (or the three-dimensional model) by comparing the second frame projected on the display unit 1049 with the actual image. If the user believes that the second image frame deviates from the actual image, the user can issue a frame adjustment instruction through the input unit 1047. The frame adjustment instruction may include movement in the X-axis, Y-axis, and Z-axis directions and/or rotation centered on the X-axis, Y-axis, and Z-axis. In the case where the second module M2 is executed by the processing unit 1041, after receiving the frame adjustment instruction, the processing unit 1041 determines that the calibration parameters may need to be adjusted, so the processing unit 1041 generates a plurality of first according to the frame adjustment instruction Correct the parameters. Next, the processing unit 1041 may update the calibration parameters according to the first correction parameters.

In an embodiment, the user can know whether the three-dimensional model created by the processing unit 1041 matches the actual image by comparing the three-dimensional model projected on the display unit 1049 with the actual image. If the user thinks that the three-dimensional model deviates from the actual image or thinks that the position of the three-dimensional model needs to be adjusted, the user can issue a model adjustment command through the input unit 1047. The model adjustment instruction may include movement in the X-axis, Y-axis, and Z-axis directions and/or rotation about the X-axis, Y-axis, and Z-axis. In the case where the second module M2 is executed by the processing unit 1041, after receiving the model adjustment instruction, the processing unit 1041 determines that the first posture needs to be adjusted, so the processing unit 1041 generates a plurality of second according to the model adjustment instruction Correct the parameters. When the first mode When the group M1 is executed by the processing unit 1041, the processing unit 1041 updates the first pose according to the second correction parameter.

In an embodiment, the first correction parameter and the second correction parameter may be generated according to the displacement and/or rotation adjusted by the frame adjustment instruction and the model adjustment instruction, respectively.

In one embodiment, when the second module M2 is executed by the processing unit 1041, it causes the processing unit 1041 to calculate a similarity based on the second image frame and the three-dimensional model, generate an adjustment prompt based on the similarity, and cause the projection unit 1045 The adjustment prompt is projected to the display unit 1049. For example, the processing unit 1041 will first calculate an overlap area A=Overlap(F,pf,M,pm), and the voxelhulls method may be used to find the overlap area. Then, calculate the difference square and the average value of the corresponding points in the overlapping area as the similarity, which includes the two parts of the RGB information map and the depth information map, and multiply each by a positive real weight ( ω _r , ω _d ) to form the final The similarity S is as follows:

Where △ rgb is the amount of difference in RGB values in the RGB information map, △ depth is the amount of difference in depth in the depth information map, and N is the total number of points in the overlapping area.

In another embodiment, the three-dimensional modeling device is a virtual reality device. In this embodiment, the three-dimensional modeling device further includes a first camera, which is used to photograph the environmental scene and display it on the display unit. The processing unit can cause the projection unit to project the built 3D model and the second image frame on the display unit, or use the calculation function to project the 3D model and the second image The frame is displayed on the display unit. The calibration method described above, adjusting the second frame/three-dimensional model, and calculating the similarity can be applied to this embodiment.

With the three-dimensional modeling device provided by the present invention, the camera and the wearable display can be integrated, and the calibration parameters generated by the calibration method enable the wearable display to convert the frame and depth information obtained from the camera to its own location according to the calibration parameters Coordinate system and used to build a three-dimensional model. In addition, by comparing the current frame provided by the camera with the actual image, the user can adjust the position and/or angle of the frame or three-dimensional model in real time, and the wearable display can target its own position and/or position according to the user's adjustment. Or the calibration parameters are modified to make the updated 3D model closer to the actual environment scene, and to achieve the effect of interactive adjustment of the frame and scene model. Not only that, the wearable display can also calculate the similarity between the current image frame and the three-dimensional model, and generate an adjustment prompt according to the similarity, for the user to use as a reference for adjustment.

In summary, although the present invention has been disclosed as above with examples, it is not intended to limit the present invention. Those with ordinary knowledge in the technical field to which the present invention belongs can make various modifications and retouching without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be deemed as defined by the scope of the attached patent application.

10: 3D modeling device

102: Camera

104: Wearable display

1041: Processing unit

1043: Storage unit

1045: Projection unit

1047: Input unit

1049: display unit

M1: the first module

M2: second module

Claims (10)

A three-dimensional modeling device includes: a camera for acquiring a plurality of first image frames, a second image frame and a depth information; and a wearable display coupled to the camera, the wearable display includes: a Display unit; a processing unit; a storage unit, coupled to the processing unit, and used to store a first module and a second module, when the first module is executed by the processing unit, causing the processing The unit calculates a first pose of the wearable display. When the second module is executed by the processing unit, the processing unit is caused to be based on the first frames, the depth information, the first pose and the plurality of poses The calibration parameters calculate a three-dimensional model and update the three-dimensional model according to the second image frame; and a projection unit, coupled to the processing unit, and used to apply the three-dimensional model and the second image frame to the first pose Projected onto the display unit to be displayed on the display unit together with an actual image, wherein the calibration parameters are the basis for conversion between a camera system of the camera and a system of the wearable display. The three-dimensional modeling device as described in item 1 of the patent application scope, wherein the wearable display further includes an input unit coupled to the processing unit and used to receive a frame adjustment command from a user when the The second module consists of the processing unit During execution, the processing unit is caused to generate a plurality of first correction parameters according to the frame adjustment command, and change at least one of the calibration parameters according to the first correction parameters. The three-dimensional modeling device as described in item 1 of the patent application scope further includes an input unit coupled to the processing unit and used to receive a model adjustment command from a user when the second module is When the processing unit is executed, the processing unit is caused to generate a plurality of second correction parameters according to the model adjustment command. When the first module is executed by the processing unit, the processing unit is caused to change the second correction parameter according to the second correction parameters. The first posture. The three-dimensional modeling method as described in item 1 of the patent application scope, wherein when the second module is executed by the processing unit, the processing unit is further caused to calculate a similarity based on the second frame and the three-dimensional model, and An adjustment prompt is generated based on the similarity. The three-dimensional modeling device as described in item 1 of the patent application, wherein the calibration parameters include a rotation parameter, which is generated by: making the three-dimensional modeling device move linearly in a plurality of directions and obtain separately The wearable display corresponds to a plurality of first movement speed vectors in each direction, and the camera corresponds to a plurality of second movement speed vectors in each direction; and The rotation parameter is calculated according to the first moving speed vectors and the second moving speed vectors. The three-dimensional modeling device as described in item 1 of the patent application, wherein the calibration parameters include a displacement parameter, which is generated by: moving the three-dimensional modeling device arbitrarily and recording a plurality of pieces of rotation information, The rotation amount information records the rotation amount of the wearable display at the time point each time the angular velocity of the three-dimensional modeling device is greater than a first threshold, and the wearable display relative to the previous time point at the time point The displacement, the displacement of the camera relative to the previous time at the time, and the rotation axis unit vector at the time is calculated according to the rotation at the time; and according to the rotation information and the rotation axis units The vector calculates the displacement parameter. A calibration method applied to a three-dimensional modeling device includes: linearly moving the three-dimensional modeling device along a plurality of directions, and obtaining a plurality of first movements corresponding to each direction of a wearable display of the three-dimensional modeling device respectively A speed vector, a camera of the three-dimensional modeling device corresponding to a plurality of second moving speed vectors in each direction; calculating a rotation parameter based on the first moving speed vectors and the second moving speed vectors; arbitrarily moving the three-dimensional Modeling device, and record a plurality of pieces of rotation information, wherein the rotation information records the angular velocity of the three-dimensional modeling device each time is greater than one The amount of rotation of the wearable display at the time point at the first threshold, the amount of displacement of the wearable display at the time point relative to the previous time point, and the amount of displacement of the camera at the time point relative to the previous time point And calculate a unit of rotation axis unit vector at the time point according to the rotation amount at the time point; and calculate a displacement parameter according to the rotation amount information and the rotation axis unit vectors. The calibration method as described in item 7 of the patent application scope, wherein a processing unit of the wearable display is based on the rotation parameter, the displacement parameter, a plurality of first image frames provided by the camera, a depth information, a first The pose and the plurality of calibration parameters calculate a three-dimensional model, and update the three-dimensional model according to the second frame provided by the camera. The calibration method as described in item 8 of the patent application, wherein the processing unit generates a plurality of first correction parameters according to a user's frame adjustment command, and changes the rotation parameter and the displacement according to the first correction parameters At least one of the parameters. The calibration method as described in item 8 of the patent application scope, wherein the processing unit generates a plurality of second correction parameters according to a model adjustment from a user, and the processing unit changes the first pose according to the second correction parameters .

Images