WO2020237492A1

WO2020237492A1 - Three-dimensional reconstruction method, device, apparatus, and storage medium

Info

Publication number: WO2020237492A1
Application number: PCT/CN2019/088698
Authority: WO
Inventors: 毛一杰; 吴勇辉; 范文文
Original assignee: 深圳市汇顶科技股份有限公司
Priority date: 2019-05-28
Filing date: 2019-05-28
Publication date: 2020-12-03
Also published as: CN110337674A; CN110337674B

Abstract

Provided are a three-dimensional reconstruction method, a device, an apparatus, and a storage medium. The method comprises: acquiring sets of first position coordinates corresponding to respective structured light patterns on an measurement image; determining, according to the measurement image, a first calibration pattern and a second calibration pattern, a calibration interval for the respective structured light patterns on the measurement image, wherein the first calibration pattern is a structured light calibration pattern calibrated on a first calibration plane, the second calibration pattern is a structured light calibration pattern calibrated on a second calibration plane, and the first calibration plane and the second calibration plane are parallel to a camera plane; performing calculation according to the calibration interval for the respective structured light patterns on the measurement image to obtain a depth value of each set of first position coordinates; and performing reconstruction of the measurement image according to the depth value of each set of first position coordinates and a calibrated reconstruction parameter. The invention enhances the efficiency of three-dimensional reconstruction.

Description

Three-dimensional reconstruction method, device, equipment and storage medium

Technical field

The present invention relates to the field of computer vision technology, in particular to a three-dimensional reconstruction method, device, equipment and storage medium.

Background technique

Three-dimensional reconstruction is an important research field of computer vision. It refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects, which is used to restore the surface shape of the object or the distance between the camera and the object in the scene, and quickly restore the scene and the object. The three-dimensional shape improves the efficiency of measurement and modeling. It is widely used in ancient architecture digitization, cultural relic digitization, robot positioning and navigation, reverse engineering and other scenes. Three-dimensional reconstruction is also a key technology for establishing virtual reality expressing the objective world in a computer. On the one hand, compared with the traditional two-dimensional model, the three-dimensional model has a larger space for differentiation, and the acquisition of three-dimensional information is necessary for its development.

In the prior art, three-dimensional reconstruction methods are roughly divided into stereo vision methods (binocular, trinocular, and multi-eye vision), structured light reconstruction methods, structure from motion (Structure from Motion, SFM), etc. Among them, the structured light reconstruction method occupies a certain advantage in application. The structured light reconstruction method uses a structured light projector to actively project a controllable light spot, light strip or light surface with a known coding structure to the object or scene, and the camera captures The image of the object or scene with the projected pattern is processed by computer vision technology, the coded pattern is analyzed, the three-dimensional information of the object or scene is obtained, and the three-dimensional model of the object or scene is constructed. The key to structured light reconstruction technology is system calibration and stereo matching.

However, in the structured light reconstruction method in the prior art, the reconstruction process is complicated and the three-dimensional correction accuracy is high, resulting in low efficiency of the three-dimensional reconstruction.

Summary of the invention

The present invention provides a three-dimensional reconstruction method, device, equipment and storage medium to realize the simplification of the three-dimensional reconstruction method and improve the efficiency of the three-dimensional reconstruction.

In the first aspect, an embodiment of the present invention provides a three-dimensional reconstruction method, including:

Obtain the first position coordinates corresponding to each structured light pattern on the measurement image.

According to the measurement image, the first calibration diagram and the second calibration diagram, determine the calibration interval of each structured light pattern on the measurement image. The first calibration diagram is the structure cursor calibration diagram calibrated on the first calibration surface, and the second calibration diagram is The structure cursor is calibrated on the second calibration surface, and the first calibration surface and the second calibration surface are parallel to the camera plane.

According to the calibration interval of each structured light pattern on the measurement image, the depth value of each first position coordinate is calculated.

According to the depth value of each first position coordinate and the calibration reconstruction parameters, the measurement image is reconstructed in three dimensions.

In this solution, the calibration interval of each structured light pattern on the measurement graph is determined according to the measurement image, the first calibration graph and the second calibration graph, and then the calibration interval of each structured light pattern on the measurement image is calculated. The depth value of a position coordinate, and finally according to the depth value of each first position coordinate and the calibration reconstruction parameters, the measurement image is reconstructed in three dimensions, which not only realizes the three-dimensional reconstruction of the measurement image, but also directly passes through each structured light on the measurement image The calibration interval of the graph calculates the depth value of each first position coordinate, which avoids the step of stereo correction and improves the efficiency of three-dimensional reconstruction.

Optionally, before performing the three-dimensional reconstruction of the measured image according to the depth value of each first position coordinate and the calibration reconstruction parameter, the method further includes:

Acquire a third calibration map and a fourth calibration map, the third calibration map is an unstructured cursor calibration map calibrated on the first calibration surface, and the fourth calibration map is an unstructured cursor calibration map calibrated on the second calibration surface;

According to the third calibration map, the fourth calibration map and the camera imaging model, the calibration reconstruction parameters are determined.

In this solution, the calibration of the depth value is achieved through the first calibration map and the second calibration map, and then the third calibration map, the fourth calibration map and the camera imaging model determine the calibration reconstruction parameters, and the depth value calibration and calibration reconstruction parameters are realized The separation reduces the accuracy requirements of calibration, and thus reduces the difficulty of calibration.

Optionally, the calibration reconstruction parameters are determined according to the third calibration map, the fourth calibration map and the camera imaging model, including:

In the third calibration map and the fourth calibration map, multiple sets of pixel coordinates and coordinates on the calibration board corresponding to the multiple sets of pixel coordinates are acquired.

For multiple sets of pixel coordinates, each set of pixel coordinates and the coordinates on the calibration board corresponding to each set of pixel coordinates are input into the camera imaging model to determine the homography matrix of any plane parallel to the camera plane in the preset space.

Correspondingly, according to the depth value of each first position coordinate and the calibrated reconstruction parameters, the measurement image is reconstructed in three dimensions, including:

According to the depth value of each first position coordinate, the homography matrix of any plane parallel to the camera plane in the preset space, and the camera imaging model, the measurement image is reconstructed in three dimensions.

In this solution, the homography matrix of any plane parallel to the camera plane in the preset space is determined through the pixel coordinates in the third calibration map and the fourth calibration map and the coordinates on the calibration board corresponding to the pixel coordinates. It is assumed that the three-dimensional coordinates of any plane parallel to the camera plane in the space are determined, and then the three-dimensional reconstruction of the measured image is realized.

Optionally, determining the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration image, and the second calibration image includes:

Match the structured light patterns in the measurement image, the first calibration map and the second calibration map.

Determine the first position coordinates of each structured light pattern in the measurement image, the second position coordinates in the first calibration map, and the third position coordinates in the second calibration map.

According to the first position coordinate, the second position coordinate and the third position coordinate of each structured light pattern, the calibration interval of each structured light pattern on the measurement image is determined.

Matching the structured light patterns of the first calibration map and the second calibration map, and determine the coordinate corresponding relationship of each structured light pattern of the first calibration map and the second calibration map.

Matching the structured light patterns of the measurement image and the first calibration map, and determine the first position coordinates of each structured light pattern in the measurement image and the second position coordinates in the first calibration map.

According to the first position coordinates, the second position coordinates, and the coordinate correspondence between the first calibration map and each structured light pattern in the second calibration map, the calibration interval of each structured light pattern on the measurement image is determined.

In this solution, by determining the coordinate corresponding relationship of each structured light pattern in the first calibration map and the second calibration map, the calibration data is compressed and a large amount of storage space is saved.

Optionally, calculating the depth value of each first position coordinate according to the calibration interval of each structured light pattern of the measurement image includes:

Determine the first parallax between the measured image and the first calibration image for each structured light pattern.

In the calibration interval of each structured light pattern, the interval except the first parallax is determined to be the second parallax, and the second parallax is the parallax between the measured image and the second calibration image of the structured light pattern.

According to the first disparity and the second disparity, the depth value of each first position coordinate is calculated.

The following describes the devices, equipment, storage media, and computer program products provided by the embodiments of the present application. For the content and effects, please refer to the first aspect and the three-dimensional reconstruction methods provided in the first aspect alternatively, and will not be repeated.

In the second aspect, an embodiment of the present application provides a three-dimensional reconstruction device, including:

The first acquisition module is used to acquire the first position coordinates corresponding to each structured light pattern on the measurement image.

The first determination module is used to determine the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration map and the second calibration map. The first calibration map is the structure cursor calibration on the first calibration surface. Figure, the second calibration diagram is a structural cursor calibration diagram calibrated on the second calibration surface, the first calibration surface and the second calibration surface are parallel to the camera plane.

The second determining module is used to calculate the depth value of each first position coordinate according to the calibration interval of each structured light pattern on the measurement image.

The reconstruction module is used for three-dimensional reconstruction of the measured image according to the depth value of each first position coordinate and the calibration reconstruction parameters.

Optionally, the three-dimensional reconstruction device provided by the embodiment of the present application further includes:

The second acquisition module is used to acquire the third calibration map and the fourth calibration map, the third calibration map is the unstructured cursor calibration map calibrated on the first calibration surface, and the fourth calibration map is calibrated on the second calibration surface Unstructured cursor to set the map.

The third determining module is used to determine the calibration reconstruction parameters according to the third calibration map, the fourth calibration map and the camera imaging model.

Optionally, the third determining module is specifically used for:

Correspondingly, the reconstruction module is specifically used for:

Optionally, the first determining module is specifically used for:

According to the first position coordinates, the second position coordinates, and the coordinate correspondence between each structured light pattern in the first calibration map and the second calibration map, the calibration interval of each structured light pattern on the measurement image is determined.

Optionally, the second determining module is specifically used for:

In a third aspect, an embodiment of the present application provides a device, including:

A processor; a memory; and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor, and the computer program includes a method for executing the three-dimensional reconstruction method as in the first aspect and an optional manner of the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program causes the server to execute the three-dimensional reconstruction method provided in the first aspect and the optional methods of the first aspect.

In a fifth aspect, an embodiment of the present invention provides a computer program product, including: executable instructions, which are used to implement the three-dimensional reconstruction method as in the first aspect or an optional manner in the first aspect.

The three-dimensional reconstruction method, device, equipment and storage medium provided by the present invention obtain the first position coordinates corresponding to each structured light pattern on the measurement image, and then determine the measurement according to the measurement image, the first calibration map and the second calibration map The calibration interval of each structured light pattern on the image. The first calibration diagram is the structure cursor calibration diagram calibrated on the first calibration surface, the second calibration diagram is the structure cursor calibration diagram calibrated on the second calibration surface, the first calibration The surface and the second calibration surface are parallel to the camera plane. According to the calibration interval of each structured light pattern on the measured image, the depth value of each first position coordinate is calculated, and finally the depth value of each first position coordinate and the calibration reconstruction parameter are calculated , Perform three-dimensional reconstruction of the measured image. It not only realizes the three-dimensional reconstruction of the measurement image, but also directly calculates the depth value of each first position coordinate through the calibration interval of each structured light pattern on the measurement image, avoids the step of stereo correction, and improves the efficiency of the three-dimensional reconstruction.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG. 1 is a schematic flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of depth value measurement provided by an embodiment of the present application;

3 is a schematic flowchart of a three-dimensional reconstruction method provided by another embodiment of the present application;

4 is a schematic flowchart of a three-dimensional reconstruction method provided by another embodiment of the present application;

FIG. 5 is a schematic flowchart of a three-dimensional reconstruction method provided by still another embodiment of the present application;

Fig. 6 is a schematic structural diagram of a three-dimensional reconstruction device provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a three-dimensional reconstruction device provided by another embodiment of the present application;

FIG. 8 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of this application.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, without having to use To describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein, for example, can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to the clearly listed Those steps or units may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or equipment.

Three-dimensional reconstruction is an important research field of computer vision, and it is also a key technology for establishing virtual reality that expresses the objective world in a computer. In terms of biometrics, compared with traditional two-dimensional models, three-dimensional models have a larger space for differentiation. The information acquisition technology has its development necessity. In the prior art, the structured light reconstruction method occupies a certain advantage in application. However, the structured light reconstruction method in the prior art has a complicated reconstruction process and high requirements for three-dimensional correction accuracy, resulting in low efficiency of three-dimensional reconstruction. In order to solve the above problems, The embodiments of the present application provide a three-dimensional reconstruction method, device, equipment, and storage medium.

Hereinafter, an exemplary application scenario of the embodiment of the present application will be introduced.

Compared with two-dimensional images, three-dimensional images have a larger difference space. Three-dimensional reconstruction technology is used to reconstruct two-dimensional images taken by cameras into three-dimensional images. With the development of computer vision technology, three-dimensional reconstruction technology is widely used in military and civilian applications. For example, investigation and monitoring, military situation display, environmental monitoring, geological and landform surveying and mapping, road monitoring, traffic guidance control, etc. Based on this, the embodiments of the present application provide a three-dimensional reconstruction method, device, equipment, and storage medium.

Figure 1 is a schematic flow chart of a three-dimensional reconstruction method provided by an embodiment of the present application. The method can be executed by a three-dimensional reconstruction device, which can be implemented by software and/or hardware. For example, the device can be part of a terminal device or All, the terminal device can be a personal computer, a smart phone, a user terminal, a tablet computer, etc. The following describes the three-dimensional reconstruction method with the terminal device as the execution subject. As shown in FIG. 1, the method in the embodiment of the present invention may include:

Step S101: Acquire the first position coordinates corresponding to each structured light pattern on the measurement image.

The measurement image is a structured light image to be three-dimensionally reconstructed, and the structured light is the active structure information projected onto the surface of the object to be measured through the projector. The embodiment of the present application does not limit the structured light pattern. For example, the structured light pattern may be laser stripes, Gray code, sine fringe, light spot, etc. After the projector projects structured light onto the surface of the object to be measured, a single or multiple cameras are used to photograph the surface to be measured to obtain a structured light image, that is, a measurement image.

There are multiple structured light patterns on the measurement image, and the first position coordinates corresponding to each structured light pattern on the measurement image are obtained. Among them, the first position coordinates corresponding to each structured light pattern can be one of the structured light patterns. The fixed position is the pixel coordinates of the measurement image, and the embodiment of the present application does not limit the fixed position, for example, it may be the center point of the structured light pattern. The structured light pattern may be a partial image in the measurement image formed by the output signal of the one or more pixels after a single spot in the structured light is reflected by the face and projected onto one or more pixels of the sensor.

Step S102: Determine the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration image and the second calibration image.

The first calibration diagram is a structural cursor calibration diagram calibrated on the first calibration surface, the second calibration diagram is a structural cursor calibration diagram calibrated on the second calibration surface, and the first calibration surface and the second calibration surface are parallel to the camera plane. The positions of the first calibration surface and the second calibration surface can be selected according to the quality of the image taken by the camera and user needs. The embodiment of this application does not limit the specific position and selection method of the calibration surface, for example: for a face recognition system , The working distance of the system is 20-100 cm, the first calibration surface and the second calibration surface can choose any plane 20-100 cm away from the camera plane, for example, the first calibration surface is 40 cm away from the camera plane, that is The first calibration distance is 40 cm, the second calibration surface is 80 cm from the camera plane, that is, the second calibration distance is 80 cm, etc. The above is only an exemplary introduction to the selection of the calibration surface, and the calibration surface of the embodiment of the application is selected Not limited to this.

After determining the first calibration surface and the second calibration surface, a calibration board is placed on the first calibration surface. The embodiment of the present application does not limit the type and size of the calibration board, for example, the calibration board pattern is a checkerboard pattern; Then shoot a calibration map with structured light to obtain the first calibration map. The embodiment of this application does not limit the acquisition method of the first calibration map. Similarly, the calibration board can be moved to the second calibration surface along the optical axis of the camera. , Take another calibration map with structured light to obtain a second calibration map, which is not limited in the embodiment of the application. The structured light used in the measurement image, the first calibration map and the second calibration map are the same structured light, for example, the same structured light projected by the same lattice projector.

After the first calibration map and the second calibration map are obtained, the calibration interval of each structured light pattern on the measurement image is determined according to the measurement image, the first calibration map and the second calibration map. The embodiment of the present application does not limit the specific implementation of how to determine the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration image, and the second calibration image.

In order to facilitate the introduction of the calibration interval of each structured light image, FIG. 2 is a schematic diagram of depth value measurement provided by an embodiment of the present application. As shown in FIG. 2, the projector projects the structured light to the camera shooting space, where Z represents the object plane The distance to the camera plane, Z1 represents the first calibration distance, Z2 represents the second calibration distance. Because the first calibration map, the second calibration map and the object plane are different from the camera plane, the position of the structured light pattern will exist Parallax. And for the same structured light pattern, there are corresponding structured light patterns and position coordinates of the structured light patterns in the measurement image, the first calibration map, and the second calibration map. For example, the structured light pattern at position F in the first calibration image, the structured light pattern at position C on the object plane, and the structured light pattern at position A on the second calibration image match each other. In the measurement image, Position F corresponds to F'in the measurement image, position C corresponds to C'in the measurement image, and position A corresponds to A'in the measurement image. The line segment formed by connecting F'and A'is the structured light pattern , And the coordinate position of the structured light pattern is in its corresponding calibration interval. By calculating the calibration interval of each structured light pattern on the measurement image, the structured light pattern can be searched for within the calibration interval corresponding to each structured light pattern. The graphics avoids the stereo correction steps in the prior art and can improve the efficiency of three-dimensional reconstruction.

Step S103: Calculate the depth value of each first position coordinate according to the calibration interval of each structured light pattern on the measurement image.

According to the calibration interval of each structured light pattern on the measurement image, the depth value of each first position coordinate can be calculated. The embodiment of the present application is concerned with how to calculate the depth value of each first position coordinate according to the calibration interval on the measurement image. The specific implementation is not limited.

In a possible implementation, as shown in FIG. 2, since the first calibration surface and the second calibration surface are parallel to the camera plane, in order to calculate the depth value of each position coordinate, that is, the distance Z between the object plane and the camera plane, According to the triangle similarity principle, we know:

Therefore, you can get:

(Z2-Z1)*(C′A′)*Z=Z1*(F′A′)*(Z2-Z) (2)

Finally get the depth value calculation formula:

Therefore, the depth value of each first position coordinate can be calculated by the above calculation formula (3).

In a possible implementation manner, calculating the depth value of each first position coordinate according to the calibration interval of each structured light pattern of the measurement image includes:

Determine the first parallax between the measured image and the first calibration image for each structured light pattern. In the calibration interval of each structured light pattern, the interval except the first parallax is determined to be the second parallax, and the second parallax is the parallax between the measured image and the second calibration image of the structured light pattern. According to the first disparity and the second disparity, the depth value of each first position coordinate is calculated.

Determine the first parallax between the measured image and the first calibration image for each structured light pattern, such as C'A' in Figure 2. Since the calibration interval F'A' of the structured light image is known, the structured light can be calculated The graph is measuring the second parallax F'C' between the image and the second calibration graph. Since the first calibration distance Z1 and the second calibration distance Z2 are known, the first position of the structured light graph can be calculated according to the depth value calculation formula The depth value of the coordinate.

Similarly, by determining the second parallax between the measured image and the second calibration image for each structured light pattern, and then according to the calibration interval of each structured light, the difference between the measurement image and the first calibration image for each structured light pattern can be obtained. The first disparity, and then according to the depth value calculation formula, obtain the depth value of the first position coordinate of the structured light pattern.

Step S104: Perform three-dimensional reconstruction on the measured image according to the depth value of each first position coordinate and the calibration reconstruction parameter.

After the depth value of each first position coordinate is determined, the three-dimensional coordinates of the first position coordinate are calculated according to the calibration reconstruction parameters to realize the three-dimensional reconstruction of the measurement image. The embodiment of the present application does not limit the specific implementation of the three-dimensional reconstruction. At the same time, there are no restrictions on specific calibration reconstruction parameters.

The three-dimensional reconstruction method provided by the embodiment of the application determines the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration image and the second calibration image, and then determines the calibration interval of each structured light image on the measurement image. Interval, calculate the depth value of each first position coordinate, and finally perform three-dimensional reconstruction of the measured image according to the depth value of each first position coordinate and the calibration reconstruction parameters, which not only realizes the three-dimensional reconstruction of the measured image, but also directly through the measurement For the calibration interval of each structured light pattern on the image, the depth value of each first position coordinate is calculated, which avoids the step of stereo correction and improves the efficiency of three-dimensional reconstruction.

Optionally, FIG. 3 is a schematic flowchart of a three-dimensional reconstruction method provided by another embodiment of the present application. The method may be executed by a three-dimensional reconstruction device, which may be implemented by software and/or hardware, for example: the device may be Part or all of the terminal device, the terminal device can be a personal computer, a smart phone, a user terminal, a tablet computer, etc. The following describes the 3D reconstruction method with the terminal device as the execution subject, as shown in Fig. 3, before step S103, this The three-dimensional reconstruction method provided by the application embodiment may further include:

Step S201: Obtain a third calibration map and a fourth calibration map. The third calibration map is an unstructured cursor calibration map calibrated on the first calibration surface, and the fourth calibration map is an unstructured cursor calibration calibrated on the second calibration surface. Figure.

The third calibration map can be obtained by placing a calibration board on the first calibration surface. The embodiment of the present application does not impose restrictions on the type and size of the calibration board. For example, the calibration board pattern is a checkerboard pattern; then a calibration map is taken. , The third calibration map is obtained. The embodiment of the present application does not limit the acquisition method of the third calibration map. Similarly, you can move the calibration plate along the optical axis of the camera to the second calibration surface and take another calibration map to obtain the first calibration map. Four calibration diagrams, which are not limited in the embodiment of this application.

Step S202: Determine calibration reconstruction parameters according to the third calibration map, the fourth calibration map and the camera imaging model.

This application does not limit the specific implementation of how to determine the calibration reconstruction parameters according to the third calibration map, the fourth calibration map and the camera imaging model. Exemplarily, a camera imaging model is introduced below, and the camera imaging model is:

Among them, u represents the abscissa in the pixel plane coordinate system, v represents the ordinate in the pixel plane coordinate system, K represents the camera internal parameter matrix, R represents the rotation matrix, T represents the translation vector, and x _w represents X in the world coordinate system. The axis coordinate value, y _w represents the Y axis coordinate value in the world coordinate system, and z _w represents the Z axis coordinate value in the world coordinate system. Therefore, in a possible implementation manner, the calibration reconstruction parameters can be determined by calculating K[R|T].

In another possible implementation manner, the above formula can be further derived, where,

get:

Among them, fx, fy are parameters related to the focal length of the camera and the size of the pixel, cx, cy are the coordinates of the camera's optical center on the pixel plane, r1 to r9 are the rotation matrix parameters, Tx, Ty, and Tz are X, Y, respectively, The translation parameters in the three directions of Z. Homography matrix H to Z = 0 plane homography, ΔH increments by Z = z _w homography matrix when the matrix, (H + z _w ΔH) z _w represent Z = single plane should matrix. Therefore, the calibration reconstruction parameters can be determined by calculating the values of H and ΔH.

Based on the above formula, in a possible implementation manner, in order to determine the calibration reconstruction parameters, the calibration reconstruction parameters are determined according to the third calibration map, the fourth calibration map and the camera imaging model, including:

In the third calibration map and the fourth calibration map, obtain multiple sets of pixel coordinates and the coordinates on the calibration board corresponding to multiple sets of pixel coordinates; for multiple sets of pixel coordinates, respectively calibrate each set of pixel coordinates and each set of pixel coordinates The coordinates on the board are input into the camera imaging model to determine the homography matrix of any plane parallel to the camera plane in the preset space.

In the third calibration image and the fourth calibration image, obtain multiple sets of pixel coordinates (u _i, v _i ) and the coordinates (xi, yi) on the calibration board corresponding to each of the multiple sets of pixel coordinates, where i is greater than zero An integer, (u _i, v _i ) represents the coordinates of the i-th group of pixels. The embodiment of the present application does not limit the value of i. In a possible implementation, i is an integer greater than or equal to 8. The pixel coordinates (u _i, v _i ) and the coordinates (xi, yi) on the calibration board corresponding to the pixel coordinates are respectively brought into formula (5) to determine the homography matrix H and the homography matrix ΔH.

According to the depth value Z = z _{w of the} first position coordinate, the homography matrix H, the homography matrix ΔH, the first position coordinate (u, v) and formula (5), calculate (x _w , y _w ), and finally determine each (X _w , y _w , z _w ) of the first position coordinates to realize the three-dimensional reconstruction of the measurement image.

Since the pixel coordinates in the third calibration map and the fourth calibration map and the coordinates on the calibration board corresponding to the pixel coordinates are used to determine the homography matrix of any plane parallel to the camera plane in the preset space, the The determination of the three-dimensional coordinates of any plane parallel to the camera plane realizes the three-dimensional reconstruction of the measured image. Moreover, the depth value is calibrated through the first calibration map and the second calibration map, and then the calibration reconstruction parameters are determined through the third calibration map, the fourth calibration map and the camera imaging model, which realizes the separation of the depth value calibration and the calibration reconstruction parameters. This reduces the accuracy requirements for calibration, thereby reducing the difficulty of calibration.

In order to realize the determination of the calibration interval of each structured light image on the measurement image, in a possible implementation manner, FIG. 4 is a schematic flowchart of a three-dimensional reconstruction method provided by another embodiment of the present application. The device can be implemented by software and/or hardware. For example, the device can be part or all of a terminal device. The terminal device can be a personal computer, a smart phone, a user terminal, a tablet computer, etc. To perform the subject's description of the three-dimensional reconstruction method, as shown in FIG. 4, step S102 may include:

Step S301: Match the structured light patterns in the measurement image, the first calibration image and the second calibration image.

Match the structured light patterns in the measurement image, the first calibration image and the second calibration image, and determine the corresponding structured light images in the measurement image, the first calibration image, and the second calibration image. The specific implementation of the structured light pattern in the first calibration map and the second calibration map is not limited.

Step S302: Determine the first position coordinate in the measurement image, the second position coordinate in the first calibration map, and the third position coordinate in the second calibration map of each structured light pattern.

This application does not limit the specific implementation of this step, as long as it can determine the first position coordinates of each structured light pattern in the measurement image, the second position coordinates in the first calibration image, and the first position coordinates in the second calibration image. Three position coordinates are sufficient.

Step S303: Determine the calibration interval of each structured light pattern on the measurement image according to the first position coordinate, the second position coordinate and the third position coordinate of each structured light pattern.

As shown in Figure 2, according to the first position coordinates, second position coordinates and third position coordinates of each structured light, the calibration interval of each structured light pattern on the measurement image can be determined. According to the principle of triangle similarity, the F'A 'For calculation, the embodiment of this application does not limit this.

In order to realize the determination of the calibration interval of each structured light image on the measurement image, in another possible implementation manner, FIG. 5 is a schematic flowchart of a three-dimensional reconstruction method provided by still another embodiment of the present application. The reconstruction device is executed. The device can be implemented by software and/or hardware. For example, the device can be part or all of a terminal device. The terminal device can be a personal computer, a smart phone, a user terminal, a tablet computer, etc. The device is the execution subject to explain the three-dimensional reconstruction method. As shown in FIG. 5, step S102 may include:

Step S401: Match the structured light patterns of the first calibration map and the second calibration map, and determine the coordinate correspondence between each structured light pattern of the first calibration map and the second calibration map.

Match the structured light patterns of the first calibration diagram and the second calibration diagram, and determine the coordinate corresponding relationship of each structured light diagram in the first calibration diagram and the second calibration diagram, where each structure in the first calibration diagram and the second calibration diagram The coordinate correspondence of the light pattern can be expressed in the form of a homography matrix.

Step S402: Match the structured light patterns of the measured image and the first calibration image, and determine the first position coordinates of each structured light pattern in the measured image and the second position coordinates of the first calibration image.

Match the structured light pattern of the measurement image and the first calibration image, and determine the first position coordinates of each structured light pattern in the measurement image and the second position coordinates in the first calibration image. The embodiment of this application specifically matches this step The way is not limited.

Step S403: Determine the calibration interval of each structured light pattern on the measurement image according to the first position coordinates, the second position coordinates, and the coordinate correspondence between each structured light pattern in the first calibration map and the second calibration map.

In this step, the third position coordinates can be determined according to the corresponding relationship between the second position coordinates and the coordinates of each structured light pattern in the first calibration map and the second calibration map, and then according to the first position coordinates, The second position coordinates and the third position coordinates are used to determine the calibration interval of each structured light pattern on the measurement image. For details, please refer to step 303, which will not be repeated.

It should be noted that in this embodiment, there is no limitation on the calibration diagrams represented by the first calibration diagram and the second calibration diagram, and the first calibration diagram in step S402 may also be represented as the second calibration diagram. Since by determining the coordinate corresponding relationship of each structured light pattern in the first calibration diagram and the second calibration diagram, it is only necessary to determine the position coordinates of the structured light diagram row in any calibration diagram to realize the corresponding structure in another calibration diagram. The determination of the position coordinates of the light pattern realizes the compression of the calibration data and saves a lot of storage space.

The following embodiments of the apparatus, equipment, storage medium, and computer program product provided by the embodiments of the present application may be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiment of this application, please refer to the method embodiment of this application.

FIG. 6 is a schematic structural diagram of a three-dimensional reconstruction device provided by an embodiment of the present application. The device can be implemented by software and/or hardware. For example, the device can be part or all of a terminal device, and the terminal device can be a personal computer, Smart phones, user terminals, tablet computers, etc., as shown in FIG. 6, the three-dimensional reconstruction apparatus provided by the embodiment of the present application may include:

The first acquisition module 61 is configured to acquire the first position coordinates corresponding to each structured light pattern on the measurement image.

The first determination module 62 is configured to determine the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration map and the second calibration map, the first calibration map being a structure cursor calibrated on the first calibration surface Calibration diagram, the second calibration diagram is a structure cursor calibration diagram calibrated on the second calibration surface, the first calibration surface and the second calibration surface are parallel to the camera plane.

Optionally, the first determining module 62 is specifically configured to:

Matching the structured light patterns of the measurement image and the first calibration image, and determine the first position coordinates of each structured light image in the measurement image and the second position coordinates of the first calibration image.

The second determining module 63 is configured to calculate the depth value of each first position coordinate according to the calibration interval of each structured light pattern on the measurement image.

Optionally, the second determining module 63 is specifically used for:

Within the calibration interval of each structured light pattern, the interval except the first parallax is determined to be the second parallax, and the second parallax is the parallax between the structured light pattern and the second calibration image.

The reconstruction module 64 is configured to perform three-dimensional reconstruction of the measured image according to the depth value of each first position coordinate and the calibration reconstruction parameters.

Optionally, FIG. 7 is a schematic structural diagram of a three-dimensional reconstruction device provided by another embodiment of the present application. The device can be implemented by software and/or hardware. For example, the device can be part or all of a terminal device. It may be a personal computer, a smart phone, a user terminal, a tablet computer, etc., as shown in FIG. 7, the three-dimensional reconstruction apparatus provided by the embodiment of the present application may further include:

The second acquisition module 71 is used to acquire the third calibration map and the fourth calibration map, the third calibration map is the unstructured cursor calibration map calibrated on the first calibration surface, and the fourth calibration map is the calibration map on the second calibration surface The non-structural cursor fixed map.

The third determining module 72 is configured to determine the calibration reconstruction parameters according to the third calibration map, the fourth calibration map and the camera imaging model.

Optionally, the third determining module 72 is specifically configured to:

Correspondingly, the reconstruction module 64 is specifically used for:

According to the depth value of each first position coordinate, the homography matrix of any plane parallel to the camera plane in the preset space and the camera imaging model, the measurement image is reconstructed in three dimensions.

An embodiment of the present application provides a terminal device. FIG. 8 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 8, the terminal device includes:

A processor 81, a memory 82, a transceiver 83, and a computer program; among them, the transceiver 83 implements data transmission between the car radio and other devices. The computer program is stored in the memory 82 and is configured to be executed by the processor 81, The computer program includes instructions for executing the above-mentioned three-dimensional reconstruction method. For the content and effect, please refer to the method embodiment.

In addition, the embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions. When at least one processor of the user equipment executes the computer-executable instructions, the user equipment executes the aforementioned various possibilities. Methods.

Among them, the computer-readable medium includes a computer storage medium and a communication medium, where the communication medium includes any medium that facilitates the transfer of a computer program from one place to another. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in the ASIC. In addition, the ASIC may be located in the user equipment. Of course, the processor and the storage medium may also exist as discrete components in the communication device.

A person of ordinary skill in the art can understand that all or part of the steps in the foregoing method embodiments can be implemented by a program instructing relevant hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps including the foregoing method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: It is still possible to modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention range.

Claims

A three-dimensional reconstruction method, characterized in that it comprises:

Acquiring a first position coordinate corresponding to each structured light pattern on the measurement image, the measurement image is a structured light image, and the structured light pattern is a spot pattern of any structured light in the structured light image;

Determine the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration image, and the second calibration image. The first calibration image is taken by placing a calibration board on the first calibration surface The structure cursor calibration diagram, the second calibration diagram is a structure cursor calibration diagram taken by placing the calibration board on the second calibration surface, the first calibration surface and the second calibration surface are parallel to the camera plane, so The camera plane is a plane perpendicular to the optical axis of the camera;

Calculating the depth value of each first position coordinate according to the calibration interval of each structured light pattern on the measurement image;

According to the depth value of each of the first position coordinates and the calibration reconstruction parameters, the measurement image is reconstructed in three dimensions.
The method according to claim 1, characterized in that, before performing the three-dimensional reconstruction of the measurement image according to the depth value of each of the first position coordinates and the calibration reconstruction parameter, the method further comprises:

Acquire a third calibration map and a fourth calibration map, where the third calibration map is an unstructured cursor calibration map calibrated on the first calibration surface, and the fourth calibration map is a calibration map on the second calibration surface The unstructured cursor positioning map;

The calibration reconstruction parameters are determined according to the third calibration map, the fourth calibration map and the camera imaging model.
The method according to claim 2, wherein the determining calibration reconstruction parameters according to the third calibration map, the fourth calibration map, and the camera imaging model comprises:

In the third calibration map and the fourth calibration map, acquiring multiple sets of pixel coordinates and coordinates on the calibration board corresponding to the multiple sets of pixel coordinates;

For the multiple sets of pixel coordinates, each set of pixel coordinates and the coordinates on the calibration board corresponding to each set of pixel coordinates are input into the camera imaging model to determine a single plane parallel to the camera plane in the preset space. Should matrix

Correspondingly, the three-dimensional reconstruction of the measurement image according to the depth value of each of the first position coordinates and the calibration reconstruction parameter includes:

According to the depth value of each of the first position coordinates, the homography matrix of any plane parallel to the camera plane in the preset space, and the camera imaging model, perform three-dimensional reconstruction on the measurement image.
The method according to claim 3, wherein determining the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration map, and the second calibration map comprises:

Matching the structured light patterns in the measurement image, the first calibration map and the second calibration map;

Determine the first position coordinates of each structured light pattern in the measurement image, the second position coordinates in the first calibration map, and the third position coordinates in the second calibration map;

According to the first position coordinate, the second position coordinate and the third position coordinate of each structured light pattern, the calibration interval of each structured light pattern on the measurement image is determined.
The method according to claim 3, wherein the determining the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration map and the second calibration map comprises:

Matching the structured light patterns of the first calibration map and the second calibration map, and determine the coordinate correspondence between each structured light pattern of the first calibration map and the second calibration map;

Matching the structured light patterns of the measurement image and the first calibration image, and determine the first position coordinates of each structured light image in the measurement image and the second position coordinates in the first calibration image;

Determine the calibration of each structured light pattern on the measurement image according to the first position coordinates, the second position coordinates, and the coordinate correspondence between the first calibration map and each structured light pattern in the second calibration map Interval.
The method according to any one of claims 1 to 5, wherein the calculating the depth value of each of the first position coordinates according to the calibration interval of each structured light pattern of the measurement image comprises:

Determining the first parallax of each structured light pattern between the measurement image and the first calibration image;

Within the calibration interval of each structured light pattern, determine that an interval other than the first parallax is a second parallax, where the second parallax is the parallax of the structured light pattern between the measurement image and the second calibration image;

According to the first disparity and the second disparity, a depth value of each of the first position coordinates is calculated.
The method according to any one of claims 1-5, characterized in that,

The first calibration surface and the second calibration surface are respectively located on two sides of the object plane, or the first calibration surface and the second calibration surface are located on the same side of the object plane.
A three-dimensional reconstruction device is characterized in that it comprises:

The first acquisition module is configured to acquire the first position coordinates corresponding to each structured light pattern on the measurement image, the measurement image is a structured light image, and the structured light pattern is any structured light in the structured light image. The spot pattern;

The first determination module is used to determine the calibration interval of each structured light pattern on the measurement image according to the measurement image, the first calibration map, and the second calibration map. A structure cursor calibration map shot by a calibration board is placed on the surface, the second calibration diagram is a structure cursor calibration map shot by placing the calibration board on a second calibration surface, the first calibration surface, the second calibration surface The surface is parallel to the camera plane, and the camera plane is a plane perpendicular to the optical axis of the camera;

The second determining module is configured to calculate the depth value of each first position coordinate according to the calibration interval of each structured light pattern on the measurement image;

The reconstruction module is configured to perform three-dimensional reconstruction on the measurement image according to the depth value of each of the first position coordinates and the calibration reconstruction parameters.
The device according to claim 8, further comprising:

The second acquisition module is used to acquire a third calibration map and a fourth calibration map, where the third calibration map is an unstructured cursor calibration map calibrated on the first calibration surface, and the fourth calibration map is Describe the unstructured cursor calibration map on the second calibration surface;

The third determining module is configured to determine the calibration reconstruction parameters according to the third calibration map, the fourth calibration map, and the camera imaging model.
The device according to claim 9, wherein the third determining module is specifically configured to:

In the third calibration map and the fourth calibration map, acquiring multiple sets of pixel coordinates and coordinates on the calibration board corresponding to the multiple sets of pixel coordinates;

For the multiple sets of pixel coordinates, each set of pixel coordinates and the coordinates on the calibration board corresponding to each set of pixel coordinates are input into the camera imaging model to determine a single plane parallel to the camera plane in the preset space. Should matrix

Correspondingly, the reconstruction module is specifically used for:

According to the depth value of each of the first position coordinates, the homography matrix of any plane parallel to the camera plane in the preset space, and the camera imaging model, perform three-dimensional reconstruction on the measurement image.
The device according to claim 10, wherein the first determining module is specifically configured to:

Matching the structured light patterns in the measurement image, the first calibration map and the second calibration map;

Determine the first position coordinates of each structured light pattern in the measurement image, the second position coordinates in the first calibration map, and the third position coordinates in the second calibration map;

According to the first position coordinate, the second position coordinate and the third position coordinate of each structured light pattern, the calibration interval of each structured light pattern on the measurement image is determined.
The device according to claim 10, wherein the first determining module is specifically configured to:

Matching the structured light patterns of the first calibration map and the second calibration map, and determine the coordinate correspondence between each structured light pattern of the first calibration map and the second calibration map;

Matching the structured light patterns of the measurement image and the first calibration image, and determine the first position coordinates of each structured light image in the measurement image and the second position coordinates in the first calibration image;

Determine the calibration of each structured light pattern on the measurement image according to the first position coordinates, the second position coordinates, and the coordinate correspondence between the first calibration map and each structured light pattern in the second calibration map Interval.
The device according to any one of claims 8-12, wherein the second determining module is specifically configured to:

Determining the first parallax of each structured light pattern between the measurement image and the first calibration image;

Within the calibration interval of each structured light pattern, determine that an interval other than the first parallax is a second parallax, where the second parallax is the parallax of the structured light pattern between the measurement image and the second calibration image;

According to the first disparity and the second disparity, a depth value of each of the first position coordinates is calculated.
The device according to any one of claims 8-12, wherein:

The first calibration surface and the second calibration surface are located on both sides of the object plane, or the first calibration surface and the second calibration surface are located on one side of the object plane.
A device, characterized in that it comprises:

processor;

Memory; and

Computer program;

Wherein, the computer program is stored in the memory and configured to be executed by the processor, and the computer program includes instructions for executing the method according to any one of claims 1-7.
A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program that enables a server to execute the method according to any one of claims 1-7.