WO2021142843A1

WO2021142843A1 - Image scanning method and device, apparatus, and storage medium

Info

Publication number: WO2021142843A1
Application number: PCT/CN2020/073038
Authority: WO
Inventors: 张洪伟
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-01-19
Filing date: 2020-01-19
Publication date: 2021-07-22
Also published as: CN114981845A

Abstract

An image scanning method and device, an apparatus, and a storage medium. The method comprises: acquiring point cloud data of a scanning scenario and an initial scanned image of the scanning scenario (101); performing curved surface detection on the point cloud data, so as to obtain initial curved surface data (102); performing optimization on a curved surface represented by the initial curved surface data, so as to obtain target curved surface data (103); and correcting, according to three-dimensional coordinates in the target curved surface data, the pixel coordinates of pixels in the initial scanned image, so as to obtain a target scanned image (104).

Description

Image scanning method, device, equipment and storage medium

Technical field

The embodiments of the present application relate to image processing methods, in particular to image scanning methods, devices, equipment, and storage media.

Background technique

Document scanning technology based on image information can be integrated in mobile terminals such as mobile phones, which is convenient to carry and easy to use. Document scanning technology based on image information requires information such as texture and boundary to calculate the transformation matrix, so this technology cannot be applied to document scanning with no borders and less texture.

Time of flight (Time of flight, TOF) sensors are not affected by light changes and object textures, and can reduce costs on the premise of meeting accuracy requirements. With the help of TOF three-dimensional data to assist in obtaining the three-dimensional information of the target object and calculating the transformation matrix, the document scanning can be made independent of the picture information, which can greatly improve the scope of application of the document scanning.

However, the current image scanning technology based on TOF data cannot cope with the situation where the target object in the scanned scene is curved.

Summary of the invention

In view of this, the embodiments of the present application provide an image scanning method, device, device, and storage medium.

The technical solutions of the embodiments of the present application are implemented as follows:

In a first aspect, an embodiment of the present application provides an image scanning method, the method includes: acquiring point cloud data of a scanned scene and an initial scanned image of the scanned scene; performing curved surface detection on the point cloud data to obtain an initial curved surface Data; optimizing the curved surface represented by the initial curved surface data to obtain target curved surface data; rectify the pixel coordinates of the pixels in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain the target scanned image.

In a second aspect, an embodiment of the present application provides an image scanning device, including: a data acquisition module for acquiring point cloud data of a scanned scene and an initial scanned image of the scanned scene; a curved surface detection module for detecting the point Cloud data performs curved surface detection to obtain initial curved surface data; a curved surface optimization module for optimizing the curved surface represented by the initial curved surface data to obtain target curved surface data; an image correction module for obtaining target curved surface data according to the three-dimensional coordinates in the target curved surface data Rectify the pixel coordinates of the pixel points in the initial scanned image to obtain the target scanned image.

In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory stores a computer program that can be run on the processor, and when the processor executes the program, any of the embodiments of the present application is implemented. A step in the image scanning method.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in any of the image scanning methods described in the embodiments of the present application are implemented.

In the embodiment of the present application, after obtaining the point cloud data of the scanned scene and the initial scanned image of the scanned scene, the electronic device performs curved surface detection on the point cloud data, and optimizes the detected initial curved surface data to obtain the target curved surface Data; on the one hand, the electronic device can correct the pixel coordinates in the initial scanned image according to the three-dimensional coordinates in the target surface data, so as to obtain a more accurate and detailed scanning result, that is, the target scanned image; on the other hand, , Perform surface detection on point cloud data. Compared with plane detection, it can be applied to more scanning scenes. For example, it can scan the surface of cylinders and nearly cylinders.

Description of the drawings

FIG. 1 is a schematic diagram of an implementation process of an image scanning method according to an embodiment of the application;

2 is a schematic diagram of comparison between an initial scanned image and a target scanned image in an embodiment of the application;

FIG. 3 is a schematic diagram of column features detected by an embodiment of this application;

FIG. 4 is a schematic diagram of a grid division result according to an embodiment of the application;

Figure 5 is a schematic diagram of the reference plane of the embodiment of the application;

6 is a schematic diagram of radial optimization based on a cube bounding box according to an embodiment of the application;

FIG. 7 is a schematic diagram of radial optimization based on Poisson Surface Reconstruction (PSR) according to an embodiment of the application;

FIG. 8 is a schematic diagram of radial optimization based on dual-camera data fusion according to an embodiment of the application;

FIG. 9 is a schematic diagram of the implementation process of another image scanning method according to an embodiment of the application;

10 is a schematic structural diagram of an image scanning device according to an embodiment of the application;

FIG. 11 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the application.

Detailed ways

In order to make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be described in further detail below in conjunction with the drawings in the embodiments of the present application. The following examples are used to illustrate the application, but are not used to limit the scope of the application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of this application. The terminology used herein is only for the purpose of describing the embodiments of the application, and is not intended to limit the application.

In the following description, “some embodiments” are referred to, which describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and Can be combined with each other without conflict.

It should be pointed out that the term "first\second\third" referred to in the embodiments of this application only distinguishes similar objects, and does not represent a specific order for the objects. Understandably, "first\second\third" "When permitted, the specific order or sequence can be interchanged, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein.

The embodiment of the present application provides an image scanning method, which can be applied to electronic devices, which can be devices with information processing capabilities such as mobile phones, tablet computers, notebook computers, desktop computers, robots, and drones. The functions implemented by the image scanning method can be implemented by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

FIG. 1 is a schematic diagram of the implementation process of the image scanning method according to the embodiment of the application. As shown in FIG. 1, the method at least includes the following steps 101 to 104:

Step 101: Obtain an initial scanned image of a scanned scene and point cloud data of the scanned scene.

The target object scanned by the electronic device, such as a label on the surface of a bottle, a poster on a cylinder, a curved book surface, etc., it is understandable that the above-mentioned bottle surface or the cylindrical surface itself is a curved surface.

In implementation, the electronic device can collect data from the scanned scene through the TOF sensor, and then preliminarily filter the sensor data output by the TOF sensor through the processor, and transform the filtered sensor data to three-dimensional coordinates in the camera coordinate system. To obtain the point cloud data.

In some embodiments, the preliminary filtering includes denoising the characteristics of the sensor data output by the TOF sensor. Denoising methods, for example, point cloud removal based on a position threshold, that is, removing points in the sensor data whose location is greater than a threshold (for example, 7 meters); or, point cloud removal based on mutual distance, that is, removing the point cloud from the surroundings A point whose average distance is greater than the surrounding average of other points.

For the initial scanned image, the electronic device may photograph the scanned scene through a Red Green Blue (RGB) sensor, so as to obtain the initial scanned image.

In a word, the electronic device at least includes a TOF sensor and an RGB sensor, where the TOF sensor is used to obtain point cloud data of the scanned scene, and the RGB sensor is used to photograph the scanned scene to obtain the initial scanned image.

Step 102: Perform surface detection on the point cloud data to obtain initial surface data.

In some embodiments, the electronic device may use a nearly cylinder as a fitting model, and determine the rough shape in the point cloud data through cylinder detection, so as to obtain the initial curved surface data. The so-called near-cylindrical body includes a cylinder whose upper and lower bottom surfaces are equal and a truncated cone whose upper and lower bottom surfaces are not equal.

Step 103: Optimize the curved surface represented by the initial curved surface data to obtain target curved surface data.

The curved surface represented by the initial curved surface data is usually a relatively rough curved surface. In order to obtain a more refined curved surface, in the embodiment of the present application, the electronic device optimizes the curved surface represented by the initial curved surface data, for example, when the curved surface is the side of a cylinder. In this case, the surface is meshed to obtain a mesh surface, and then the vertices of each mesh on the mesh surface are radially optimized to obtain the target surface data.

Step 104: Correct the pixel coordinates of the pixels in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain the target scanned image.

Understandably, in most cases, the initial scanned image scanned by the electronic device is an image that does not meet the application conditions. For example, as shown in FIG. 2, the initial scanned image, that is, the image before correction, is the image 20, in which the target object 201 is inclined. After correcting it, the front-view scan result of the target object 201 can be obtained, that is, the target scan image 202 shown in FIG. 2.

In some embodiments, the electronic device may implement step 104 through step 307 and step 308 in the following embodiments to obtain the target scanned image.

The embodiment of the present application further provides an image scanning method, the method at least includes the following steps 201 to 205:

Step 201: Obtain point cloud data of a scanned scene and an initial scanned image of the scanned scene;

Step 202: Perform cylindrical shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object.

In some embodiments, the target object may be a cylinder. For example, as shown in FIG. 3, the characteristic parameter values of the cylinder 30 include: the center position p _{0 of the} bottom surface of the cylinder, the axial direction n ₀ , the radius r, the tangential range (θ ₁ , θ ₂ ) and the axial range (h) ; Among them, θ ₁ is the angle between the tangential line 31 and the reference line, θ ₂ is the angle between the tangential line 32 and the reference line; the axial range h is the height of the cylinder. In some embodiments, the electronic device may use, for example, a random sampling consensus (RANdom SAmple Consensus, RANSAC) cylinder detection algorithm to obtain these parameter values.

Step 203: Determine the initial curved surface data from the point cloud data according to the multiple characteristic parameter values;

Step 204, optimizing the curved surface represented by the initial curved surface data to obtain target curved surface data;

Step 205: Correct the pixel coordinates of the pixels in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain the target scanned image.

In the embodiment of the present application, the electronic device detects the shape of a cylinder on the point cloud data to obtain a plurality of characteristic parameter values of the cylinder; and according to the plurality of characteristic parameter values, determine the initial Curved surface data; in this way, compared to the complex fitting model, using a nearly cylinder as the fitting model can greatly simplify the algorithm, reduce the amount of calculation, and reduce the cost of the algorithm while losing less precision; compared to The plane detection method can cover more target objects and adapt to more user scenarios.

The embodiment of the present application further provides an image scanning method. The method at least includes the following steps 301 to 308:

Step 301: Obtain point cloud data of a scanned scene and an initial scanned image of the scanned scene;

Step 302: Perform cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object; wherein the shape of the target object is a cylinder, and the multiple characteristic parameter values of the target object are as shown in FIG. As shown in 3, it includes the center position p _{0 of the} bottom surface of the cylinder, the axial direction n ₀ , the radius r, the tangential range (θ ₁ , θ ₂ ), and the axial range (h).

Step 303: Determine initial curved surface data from the point cloud data according to the multiple characteristic parameter values.

Taking the image 20 shown in FIG. 2 as an example, the initial curved surface data is the point cloud data of the target object 201 in the image 20.

Step 304, according to the axial direction, the axial range, the tangential direction, the tangential range and the specific meshing interval, mesh the curved surface represented by the initial curved surface data to obtain N meshes. Grid, N is an integer greater than 0.

The specific grid division interval includes the grid division interval in the axial direction and the grid division interval in the tangential direction. The grid division interval in these two directions may be the same or different. The result of the mesh division, for example, the mesh surface 401 shown in FIG. 4, the mesh surface 401 has N meshes 402. Understandably, the size of the mesh division interval determines the mesh density of the mesh surface to a certain extent, that is, the value of N. If the grid division interval is larger, the less grids are obtained, the coarser the optimization result is; the smaller the grid division interval is, the more grids are obtained, the finer the optimization result will be, but the algorithm complexity will be higher.

Step 305: Perform an optimal solution search for each vertex of each grid according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex.

In some embodiments, each vertex is searched for the optimal solution along its radial direction to obtain the optimal three-dimensional coordinates of the corresponding vertex. In the embodiment of the present application, three optimal solution search methods are provided. For example, using steps 405 to 407 in the following embodiment to search for the optimal solution in the search space where the radial vector of the vertex is located; for another example, using steps 505 to step 508 in the following embodiment, that is, The optimal solution is obtained by using a radial optimization method based on PSR; for another example, steps 605 to 608 in the following embodiment are adopted, that is, the optimal solution is obtained by using a radial optimization method based on at least dual-camera data fusion.

Step 306: Determine the optimal three-dimensional coordinates of each vertex as the target curved surface data.

Understandably, what is obtained through step 306 is an optimized mesh surface, where the coordinates of each vertex of each mesh are the optimal three-dimensional coordinates. That is, the target surface data includes the optimal three-dimensional coordinates of each vertex of each mesh.

Step 307: Determine a transformation relationship between the target surface data and the reference surface data according to the optimal three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data.

In fact, through step 307, the electronic device determines the position transformation relationship between the optimized mesh surface and the reference surface. The reference surface data can be pre-configured. For example, the reference surface data represents a frontal plane, such as the reference surface 50 shown in FIG. 5, where the reference surface data includes the three-dimensional coordinates of each vertex of each grid in the reference surface 50.

In the embodiment of the present application, the transformation relationship may be characterized by a transformation matrix, a transformation matrix group, or a free mapping relationship.

Step 308: Correct the pixel coordinates of the pixels in the initial scanned image according to the transformation relationship to obtain the target scanned image.

In the embodiment of the present application, the surface represented by the initial surface data is meshed, and only the vertices of the mesh are searched for the optimal solution, which can greatly reduce the amount of calculation for surface optimization.

The embodiment of the present application further provides an image scanning method, the method at least includes the following steps 401 to 410:

Step 401: Obtain point cloud data of a scanned scene and an initial scanned image of the scanned scene;

Step 402: Perform cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object; wherein the shape of the target object may be a cylinder, and the multiple characteristic parameter values of the target object, Including the center position p _{0 of the} bottom surface of the cylinder, the axial direction n ₀ , the radius r, the tangential range (θ ₁ , θ ₂ ), and the axial range (h).

Step 403: Determine the initial curved surface data from the point cloud data according to the multiple characteristic parameter values;

In step 404, the curved surface is meshed according to the axial direction, the axial range, the tangential direction, the tangential range, and the specific meshing interval to obtain N meshes, where N is greater than An integer of 0;

Step 405: Determine the radial vector where the j-th vertex is located according to the position of the curved surface where the j-th vertex of the i-th grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder; i is an integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4;

Step 406: Determine a search space from the initial surface data according to the radial vector.

In some embodiments, the search space may adopt the cube bounding box shown in formula (1):

L＜[R T]p＜U (1);

In formula (1), L and U are the lower boundary point and the upper boundary point of the bounding box coordinate system, p is the three-dimensional coordinate of the point in the initial surface data, and [R T] is the augmentation from the world coordinate to the bounding box coordinate Transformation matrix.

When the electronic device implements step 406, the j-th vertex may be the center point of the cube bounding box, and according to the length of the axial direction, the grid division interval in the tangential direction, and the size of the radial vector, The cube bounding box is determined from the initial curved surface data.

For example, as shown in Figure 6, the center point of the bounding box 60 is the j-th vertex, the length of the bounding box is the radius r of the cylinder, the axial length (that is, the height) is r/2, and the width is the tangential mesh. The grid is divided into intervals.

Step 407: Determine the optimal three-dimensional coordinates of the j-th vertex according to the three-dimensional coordinates of the sampling points in the search space.

In some embodiments, the electronic device may determine the three-dimensional coordinates of the center of gravity of the search space according to the three-dimensional coordinates of the sampling points in the search space; project the three-dimensional coordinates of the center of gravity onto the radial vector to obtain The optimal three-dimensional coordinate of the j-th vertex.

For example, (2) the search space is calculated {p _{_i} | p _i ∈BOX} according to the formula of the three-dimensional coordinates of the center of gravity p _g:

In formula (2), p _i is the three-dimensional coordinates of a point in the search space, and K is a constant.

Then, according to the following formula (3), the calculated _{three-dimensional coordinates of the center of gravity p g} are projected to the radial vector where the j-th vertex is located

, So as to obtain the optimal three-dimensional coordinate p ^{* of} the j-th vertex.

In the formula, p ₀ represents the three-dimensional coordinates of the j-th vertex described before optimization.

It should be noted that the determination of the optimal three-dimensional coordinates of each vertex can be achieved through the above steps 405 to 407.

Step 408: Determine the optimal three-dimensional coordinates of each vertex as the target curved surface data;

Step 409: Determine a transformation relationship between the target surface data and the reference surface data according to the three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

Step 410: Correct the pixel coordinates of the pixels in the initial scanned image according to the transformation relationship to obtain a target scanned image.

The embodiment of the present application further provides an image scanning method. The method at least includes the following steps 501 to 511:

Step 501: Obtain point cloud data of a scanned scene and an initial scanned image of the scanned scene;

Step 502: Perform cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object; wherein the shape of the target object may be a cylinder, and the multiple characteristic parameter values of the target object include _{The center position p 0 of the} bottom surface of the cylinder, the axial direction n ₀ , the radius r, the tangential range (θ ₁ , θ ₂ ), and the axial range (h).

Step 503: Determine the initial curved surface data from the point cloud data according to the multiple characteristic parameter values;

Step 504, mesh the curved surface according to the axial direction, the axial range, the tangential direction, the tangential range, and the specific meshing interval to obtain N meshes, where N is greater than An integer of 0;

Step 505: Perform Poisson surface reconstruction according to the three-dimensional coordinates of the sampling points in the initial curved surface data to obtain an equivalent surface.

In some embodiments, the electronic device may use the PSR algorithm to obtain the isosurface. Based on the assumption that the surface of the object in the real world is continuous, the surface of the object is estimated. The PSR algorithm can eliminate the influence of point cloud measurement error on the result to a certain extent, and restore the surface of the real object.

Step 506: Determine the radial vector where the j-th vertex is located according to the position of the curved surface where the j-th vertex of the i-th grid is located, the radius of the cylinder, and the center position of the bottom surface of the cylinder; , I is an integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4;

Step 507: Determine the intersection point of the radial vector and the isosurface;

Step 508: Determine the three-dimensional coordinates of the intersection on the isosurface as the optimal three-dimensional coordinates of the j-th vertex.

As shown in Figure 7, before the radial optimization, the electronic device first uses the PSR algorithm to process the initial surface data to generate the corresponding mesh (Mesh), and then detect the radial direction where the j-th vertex is located in the generated Mesh. For the intersection point of the lines, the three-dimensional coordinates of the Mesh corresponding to the intersection point are used as the optimization result, that is, the optimal three-dimensional coordinates of the j-th vertex.

The PSR algorithm has the characteristics of being able to recover the surface of real-world objects from the point cloud with a lot of noise. PSR is an algorithm derived from the fact that the surface of the real object is smooth and continuous, so it is in line with the scanning characteristics of real objects. The resulting surface is closer to the true value. Therefore, the PSR algorithm can be used here to remove the influence of noise or misdetection, thereby reducing the influence of the deviation value in the point cloud.

It should be noted that the input of the PSR algorithm is the initial surface data, and the output is Mesh. The implementation steps of the PSR algorithm can include the following steps S1 to S4: S1, perform point cloud normal estimation on the initial surface data; S2, perform spatial meshing on the initial surface data, such as the eighth tree method; S3, find The optimal surface conforms to the continuity of the surface and the estimated normal; S4, the optimized optimal surface is output to form a Mesh grid.

It should also be noted that the determination of the optimal three-dimensional coordinates of each vertex is achieved through the above steps 506 to 508.

Step 509: Determine the optimal three-dimensional coordinates of each of the vertices as the target surface data;

Step 510: Determine a transformation relationship between the target surface data and the reference surface data according to the three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

Step 511: Correct the pixel coordinates of the pixels in the initial scanned image according to the transformation relationship to obtain a target scanned image.

The embodiment of the present application further provides an image scanning method. The method at least includes the following steps 601 to 611:

Step 601: Obtain point cloud data of a scanned scene and an initial scanned image of the scanned scene;

Step 602: Perform cylinder shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object; wherein the shape of the target object is a cylinder, and the multiple characteristic parameter values of the target object include: _{The center position p 0 of the} bottom surface of the body, the axial direction n ₀ , the radius r, the tangential range (θ ₁ , θ ₂ ) and the axial range (h).

Step 603: Determine the initial curved surface data from the point cloud data according to the multiple characteristic parameter values;

Step 604: Perform meshing on the curved surface according to the axial direction, the axial range, the tangential direction, the tangential range and the specific meshing interval, to obtain N meshes, where N is an integer greater than 0;

Step 605: Back-project the three-dimensional coordinates of the k-th sampling point on the radial vector where the j-th vertex of the i-th grid is located on the imaging plane of each camera to obtain the corresponding pixel coordinates; where i Is an integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4, and k is an integer greater than 0;

Step 606: Determine the area block of each pixel coordinate on the image collected by the corresponding camera according to the specific sampling window;

Step 607: Determine the degree of correlation between each of the regional blocks;

Step 608: In the case that the degree of correlation does not meet a specific condition, backproject the three-dimensional coordinates of the next sampling point of the radial vector onto the imaging plane of each camera, until the determined degree of correlation satisfies the Up to the specified conditions, the three-dimensional coordinates of the corresponding sampling points are determined as the optimal three-dimensional coordinates of the j-th vertex.

Steps 605 to 608 provide another optimization method, that is, a radial optimization method based on at least dual-camera data fusion, which can further improve the robustness.

In some embodiments, the j-th vertex may be used as a starting point to perform cost function optimization, so as to obtain the optimal three-dimensional coordinates of the j-th vertex. Taking the radial optimization method of dual-camera data fusion as an example, as shown in Figure 8, the three-dimensional coordinate p _j before the optimization of the j-th vertex is used as the starting point, the gradient descent method or the LM algorithm is used, and the formula (4 ) The cost function shown in the optimization:

p ^* = argmin _p (R(pp _j )+α*C(W(π ₀ (p _j )),W(π ₁ (p _j )))) (4);

In the formula, p ^* is the optimal solution, that is, the optimal three-dimensional coordinates of the j-th vertex, p is the position of the search point in the initial surface data, that is, the three-dimensional coordinates of the k-th sampling point, and R is a regular function, for example L ₂ regular function is used, π ₀ and π ₁ are the back projection function of the dual-camera space coordinate back-projected to the pixel coordinate, W is the sampling window function, for example, the sampling of 3*3 square pixels or 7*7 of the neighboring back-projection point For the sampling of square pixels, C is the cross-correlation function of the neighboring area blocks of the projection point in the dual-camera image. The cross-correlation function can use functions such as NCC or ZNCC, and α is the scale coefficient to adjust the dependence on TOF data.

It should be noted that for the case of three shots or more than three shots, as shown in formula (5), the form of the cost function can be kept consistent, and the difference is the calculation of the cross-correlation function:

p ^* = argmin _p (R(pp _j )+α*C(W(π ₀ (p _j )),W(π ₁ (p _j )),W(π ₂ (p _j )...)) ( 5);

That is, the projection items of the third and fourth cameras are added to the equation (4), the input of the function C for calculating the cross-correlation becomes multiple, and the calculation method of the cross-correlation function can adopt the result of the pairwise camera correlation. Sum, or average.

Step 609: Determine the optimal three-dimensional coordinates of each vertex as the target surface data;

Step 610: Determine a transformation relationship between the target surface data and the reference surface data according to the three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

Step 611: Correct the pixel coordinates of the pixels in the initial scanned image according to the transformation relationship to obtain a target scanned image.

An exemplary application of the embodiment of the present application in an actual application scenario will be described below.

In the shape detection algorithm based on TOF data in the embodiment of this application, for the case of a target object with a curved surface, a nearly cylindrical body is used as a fitting model, and the rough shape in the point cloud data is determined by cylinder detection, and then the diameter of the cylinder is determined. Optimal processing is performed to obtain a more refined surface result.

Compared with complex model fitting, it can greatly simplify the algorithm, reduce the amount of calculation, and reduce the cost of the algorithm while losing less precision. Compared with the plane detection method, it can cover more target objects and adapt to More user scenarios.

In some embodiments, the document scanning system includes a TOF sensor, an RGB sensor, and a conversion parameter generation module and an image conversion module. The functions of the conversion parameter generation module and the image conversion module can be performed by the processor; wherein,

The transformation parameter generation module is used to calculate the sensor data output by the TOF sensor to generate transformation parameters required by the image transformation module. The transformation parameter may be one of the following: transformation matrix, transformation matrix group, and free mapping relationship.

The image transformation module is used to transform the RGB data (that is, an example of the initial scan image) obtained from the RGB sensor according to the transformation parameters generated by the transformation parameter generation module, so as to obtain the transformed orthographic scan result, namely The target scan image.

In some embodiments, the transformation parameter generation module includes a cylinder detection unit, a mesh division unit, a cylindrical radial optimization unit, a free-form surface generation unit, and a transformation parameter generation unit; wherein,

The cylinder detection unit can use the RANSAC cylinder detection algorithm;

The mesh division unit can be used to divide the target area of the cylindrical surface at regular intervals according to the tangential and axial directions;

Cylindrical radial optimization unit can be used to search for the optimal solution for each vertex of the divided mesh in the radial direction of the cylinder;

Among them, the search for the optimal solution can use one of the following methods:

(1) The method of finding the most concentrated point in the tangential range in the radial direction as the optimal solution;

(2) The method of calculating the intersection point between the radial line and the grid reconstructed from the surface of the overall point cloud in the radial direction as the optimal solution;

(3) Use the dual-camera data fusion method to calculate the optimization of the back-projection cost function.

The free-form surface generating unit is used to take all the optimal solutions in the radial direction as vertices, and connect topologically according to the topological structure of the mesh division to form a free-form surface mesh.

The transformation parameter generation unit is used to generate transformation parameters according to the free-form surface mesh and the correction target parameters. According to the spatial position of the free-form surface mesh and the set spatial position of the correction result (that is, the three-dimensional coordinates of the vertices in the reference surface data), transformation parameters such as transformation matrix or transformation matrix group or free mapping relationship are generated.

The embodiment of the present application further provides an image scanning method. FIG. 9 is a schematic diagram of the implementation process of the image scanning method according to the embodiment of the present application. As shown in FIG. 9, it may at least include the following steps 901 to 907:

Step 901: Perform preliminary filtering on the sensor data output by the TOF, and then transform the preliminary filtered sensor data to three-dimensional coordinates in the camera coordinate system to obtain three-dimensional point cloud data;

In some embodiments, the preliminary filtering includes denoising the characteristics of the sensor data output by the TOF, and denoising methods, for example, point cloud removal based on position threshold, or point cloud removal based on mutual distance;

Step 902: Perform cylinder detection on the three-dimensional point cloud data to obtain cylinder fitting parameters, that is, multiple characteristic parameter values of the cylinder, where the parameters include: the center position p _{0 of the} cylinder bottom surface and the axis direction n ₀ , Radius r, tangential range (θ ₁ ,θ ₂ ) and axial range (h). In some embodiments, the RANSAC cylinder detection algorithm may be used to obtain these parameters.

In step 903, the target area on the cylindrical surface is divided into regular grids at regular intervals according to the tangential and axial directions;

Step 904, searching for an optimal solution for each vertex of the divided grid along the radial direction of the cylinder;

Step 905: According to the original mesh topology, update the coordinate position to the optimal position obtained by the radial optimization to form an optimized free-form mesh surface;

Step 906: Generate corresponding transformation parameters according to the corresponding relationship of the coordinates;

Transformation parameters, for example, when the transformation is relatively uniform, a single homography matrix (Homography) can be used; when the transformation is relatively complex, the coordinate pair relationship can be directly established to generate an interpolation function for subsequent image transformation.

Step 907: Transform the input image (ie, the initial scanned image) with the generated transformation parameters to obtain the corrected front view result, that is, the target scanned image.

Based on the foregoing embodiment, the embodiment of the present application provides an image scanning device, which includes each module included and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; In the process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

FIG. 10 is a schematic diagram of the composition structure of an image scanning device according to an embodiment of the application. As shown in FIG. 10, the device 100 includes a data acquisition module 101, a curved surface detection module 102, a curved surface optimization module 103, and an image correction module 104, in which:

The data acquisition module 101 is configured to acquire the point cloud data of the scanned scene and the initial scanned image of the scanned scene;

The curved surface detection module 102 is configured to perform curved surface detection on the point cloud data to obtain initial curved surface data;

The curved surface optimization module 103 is configured to optimize the curved surface represented by the initial curved surface data to obtain target curved surface data;

The image correction module 104 is configured to correct the pixel coordinates of the pixels in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain the target scanned image.

In some embodiments, the curved surface detection module 102 is configured to: perform cylindrical shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object; according to the multiple characteristic parameter values, from the point cloud The initial curved surface data is determined in the data.

In some embodiments, the shape of the target object is a cylinder, and the multiple characteristic parameter values include the axial direction, the axial range, the tangential direction, and the tangential range of the cylinder; the curved surface optimization module 103 is used for : According to the axial direction, the axial range, the tangential direction, the tangential range and the specific meshing interval, mesh the curved surface represented by the initial curved surface data to obtain N meshes, N is an integer greater than 0; according to the three-dimensional coordinates of the sampling points in the initial surface data, the optimal solution search is performed on each vertex of each grid to obtain the optimal three-dimensional coordinates of the corresponding vertex; The optimal three-dimensional coordinates of is determined as the target surface data.

In some embodiments, the multiple characteristic parameter values further include the radius of the cylinder and the center position of the bottom surface of the cylinder; the curved surface optimization module 103 is configured to: according to the curved surface where the j-th vertex of the i-th mesh is located The position of the cylinder, the radius of the cylinder and the center position of the bottom of the cylinder determine the radial vector where the j-th vertex is located; where i is an integer greater than 0 and less than or equal to N, and j is greater than 0 And an integer less than or equal to 4; determine the search space from the initial surface data according to the radial vector; determine the optimal three-dimensionality of the j-th vertex according to the three-dimensional coordinates of the sampling points in the search space coordinate.

In some embodiments, the search space adopts a cubic bounding box, and the curved surface optimization module 103 is used to:

Taking the j-th vertex as the center point of the cube bounding box, according to the length of the axial direction, the meshing interval in the tangential direction, and the size of the radial vector, the initial surface data is determined The cube bounding box.

In some embodiments, the curved surface optimization module 103 is configured to: determine the three-dimensional coordinates of the center of gravity of the search space according to the three-dimensional coordinates of the sampling points in the search space; project the three-dimensional coordinates of the center of gravity to the diameter In the direction vector, the optimal three-dimensional coordinate of the j-th vertex is obtained.

In some embodiments, the curved surface optimization module 103 is configured to: perform Poisson surface reconstruction according to the three-dimensional coordinates of the sampling points in the initial curved surface data to obtain an equivalent surface; The position of the curved surface, the radius of the cylinder and the center position of the bottom surface of the cylinder determine the radial vector at which the j-th vertex is located; where i is an integer greater than 0 and less than or equal to N, and j is An integer greater than 0 and less than or equal to 4; determine the intersection point between the radial vector and the isosurface; determine the three-dimensional coordinate of the intersection on the isosurface as the j-th vertex The optimal three-dimensional coordinates.

In some embodiments, the curved surface optimization module 103 is used to back-project the three-dimensional coordinates of the k-th sampling point on the radial vector where the j-th vertex of the i-th grid is located to the imaging plane of each camera. Above, obtain the corresponding pixel coordinates; where i is an integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4; according to a specific sampling window, determine that each pixel coordinate is collected by the corresponding camera Determine the degree of correlation between each of the regional blocks; in the case that the degree of correlation does not meet specific conditions, backproject the three-dimensional coordinates of the next sampling point of the radial vector respectively On the imaging plane of each camera, until the determined correlation degree satisfies the specific condition, the three-dimensional coordinate of the corresponding sampling point is determined as the optimal three-dimensional coordinate of the j-th vertex.

In some embodiments, the image correction module 104 is configured to determine the target surface data and the target surface data according to the optimal three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data. The transformation relationship between the reference surface data; according to the transformation relationship, the pixel coordinates of the pixels in the initial scanned image are corrected to obtain the target scanned image.

The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be noted that, in the embodiments of the present application, if the above-mentioned image scanning method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to enable An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a robot, a drone, etc.) executes all or part of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), magnetic disk or optical disk and other media that can store program codes. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, an embodiment of the application provides an electronic device. FIG. 11 is a schematic diagram of a hardware entity of the electronic device according to an embodiment of the application. As shown in FIG. 11, the hardware entity of the electronic device 110 includes: a memory 111 and a processor. 112. The memory 111 stores a computer program that can run on the processor 112, and the processor 112 implements the steps in the image scanning method provided in the foregoing embodiment when the processor 112 executes the program.

The memory 111 is configured to store instructions and applications executable by the processor 112, and can also cache data to be processed or processed by each module in the processor 112 and the electronic device 110 (for example, image data, audio data, voice communication data, and Video communication data) can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).

Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the image scanning method provided in the foregoing embodiments are implemented.

It should be pointed out here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application. The implementation process constitutes any limitation. The serial numbers of the foregoing embodiments of the present application are for description only, and do not represent the superiority of the embodiments.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, It also includes other elements that are not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation, such as: multiple modules or components can be combined, or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or modules, and may be electrical, mechanical, or other forms. of.

The modules described above as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the embodiments of the present application may all be integrated into one processing unit, or each module may be individually used as a unit, or two or more modules may be integrated into one unit; the above-mentioned integration The module can be implemented in the form of hardware, or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps in the above method embodiments can be implemented by a program instructing relevant hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the execution includes The steps of the foregoing method embodiment; and the foregoing storage medium includes: various media that can store program codes, such as a mobile storage device, a read only memory (Read Only Memory, ROM), a magnetic disk, or an optical disk.

Alternatively, if the aforementioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to enable An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a robot, a drone, etc.) executes all or part of the method described in each embodiment of the present application. The aforementioned storage media include: removable storage devices, ROMs, magnetic disks, or optical disks and other media that can store program codes.

The methods disclosed in the several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

The above are only the implementation manners of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Covered in the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Industrial applicability

In the embodiment of the present application, after obtaining the point cloud data of the scanned scene and the initial scanned image of the scanned scene, the electronic device performs curved surface detection on the point cloud data, and optimizes the detected initial curved surface data to obtain the target curved surface Data; in this way, on the one hand, the electronic device can correct the pixel coordinates in the initial scan image according to the three-dimensional coordinates in the target surface data, so as to obtain more accurate and detailed scan results (ie, the target scan image); on the other hand, In terms of surface detection on point cloud data, compared to plane detection, it can be applied to more scanning scenes, for example, it can scan the surface of cylinders and nearly cylinders.

Claims

An image scanning method, the method includes:

Acquiring the initial scanned image of the scanned scene and the point cloud data of the scanned scene;

Performing surface detection on the point cloud data to obtain initial surface data;

Optimizing the curved surface represented by the initial curved surface data to obtain target curved surface data;

According to the three-dimensional coordinates in the target curved surface data, the pixel coordinates of the pixel points in the initial scanned image are corrected to obtain the target scanned image.
The method according to claim 1, wherein said performing curved surface detection on said point cloud data to obtain initial curved surface data comprises:

Performing cylindrical shape detection on the point cloud data to obtain multiple characteristic parameter values of the target object;

According to the plurality of characteristic parameter values, the initial curved surface data is determined from the point cloud data.
The method according to claim 2, wherein the shape of the target object is a cylinder, and the plurality of characteristic parameter values include axial, axial, tangential, and tangential ranges of the cylinder, and the alignment The surface represented by the initial surface data is optimized to obtain the target surface data, including:

According to the axial direction, the axial range, the tangential direction, the tangential direction range, and the specific meshing interval, the curved surface represented by the initial curved surface data is meshed to obtain N meshes, N Is an integer greater than 0;

Performing an optimal solution search for each vertex of each grid according to the three-dimensional coordinates of the sampling points in the initial surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex;

The optimal three-dimensional coordinate of each vertex is determined as the target curved surface data.
The method according to claim 3, wherein the plurality of characteristic parameter values further include the radius and the center position of the cylinder, and the three-dimensional coordinates of the sampling points in the initial curved surface data are used for each grid Perform an optimal solution search for a vertex to obtain the optimal three-dimensional coordinates of the corresponding vertex, including:

According to the position of the curved surface where the j-th vertex of the i-th mesh is located, the radius of the cylinder, and the center position of the bottom surface of the cylinder, determine the radial vector where the j-th vertex is located; where i is greater than An integer that is 0 and less than or equal to N, and j is an integer that is greater than 0 and less than or equal to 4;

Determining a search space from the initial surface data according to the radial vector;

According to the three-dimensional coordinates of the sampling points in the search space, the optimal three-dimensional coordinates of the j-th vertex are determined.
The method according to claim 4, wherein the search space adopts a cubic bounding box, and the determining the search space from the initial surface data according to the radial vector comprises:

Taking the j-th vertex as the center point of the cube bounding box, according to the length of the axial direction, the meshing interval in the tangential direction, and the size of the radial vector, the initial surface data is determined The cube bounding box.
The method according to claim 4, wherein the determining the optimal three-dimensional coordinates of the j-th vertex according to the three-dimensional coordinates of the sampling points in the search space comprises:

Determine the three-dimensional coordinates of the center of gravity of the search space according to the three-dimensional coordinates of the sampling points in the search space;

Projecting the three-dimensional coordinates of the center of gravity onto the radial vector to obtain the optimal three-dimensional coordinates of the j-th vertex.
The method according to claim 3, wherein the plurality of characteristic parameter values further include the radius and the center position of the cylinder, and the three-dimensional coordinates of the sampling points in the initial curved surface data are used for each grid Perform an optimal solution search for a vertex to obtain the optimal three-dimensional coordinates of the corresponding vertex, including:

Perform Poisson surface reconstruction according to the three-dimensional coordinates of the sampling points in the initial curved surface data to obtain an equivalent surface;

According to the position of the curved surface where the j-th vertex of the i-th grid is located, the radius of the cylinder and the center position of the bottom surface of the cylinder, determine the radial vector where the j-th vertex is located; where i is An integer greater than 0 and less than or equal to N, j is an integer greater than 0 and less than or equal to 4;

Determining the intersection point of the radial vector and the isosurface;

The three-dimensional coordinates of the intersection on the isosurface are determined as the optimal three-dimensional coordinates of the j-th vertex.
The method according to claim 3, wherein the searching for the optimal solution for each vertex of each grid according to the three-dimensional coordinates of the sampling points in the initial curved surface data to obtain the optimal three-dimensional coordinates of the corresponding vertex comprises:

The three-dimensional coordinates of the k-th sampling point on the radial vector where the j-th vertex of the i-th grid is located are back-projected onto the imaging plane of each camera to obtain the corresponding pixel coordinates; where i is greater than 0 And an integer less than or equal to N, j is an integer greater than 0 and less than or equal to 4, and k is an integer greater than 0;

According to a specific sampling window, determine the area block of each pixel coordinate on the image collected by the corresponding camera;

Determine the degree of correlation between each of said regional blocks;

In the case that the degree of correlation does not satisfy a specific condition, the three-dimensional coordinates of the next sampling point of the radial vector are back projected onto the imaging plane of each camera, until the determined degree of correlation satisfies the specific condition , Determining the three-dimensional coordinate of the corresponding sampling point as the optimal three-dimensional coordinate of the j-th vertex.
The method according to any one of claims 3 to 8, wherein the correcting the pixel coordinates of the pixels in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain the target scanned image includes:

Determine the transformation relationship between the target surface data and the reference surface data according to the optimal three-dimensional coordinates of each vertex in the target surface data and the three-dimensional coordinates of the corresponding vertex in the reference surface data;

According to the transformation relationship, the pixel coordinates of the pixels in the initial scanned image are corrected to obtain the target scanned image.
An image scanning device, including:

A data acquisition module for acquiring the point cloud data of the scanned scene and the initial scanned image of the scanned scene;

The curved surface detection module is used to perform curved surface detection on the point cloud data to obtain initial curved surface data;

A curved surface optimization module for optimizing the curved surface represented by the initial curved surface data to obtain target curved surface data;

The image correction module is used to correct the pixel coordinates of the pixels in the initial scanned image according to the three-dimensional coordinates in the target curved surface data to obtain the target scanned image.
An electronic device, comprising a memory and a processor, the memory storing a computer program that can run on the processor, and the processor implements the image scanning method according to any one of claims 1 to 9 when the processor executes the program A step of.
A computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the image scanning method according to any one of claims 1 to 9 are realized.