WO2023173572A1

WO2023173572A1 - Real-time panoramic imaging method and device for underwater cleaning robot

Info

Publication number: WO2023173572A1
Application number: PCT/CN2022/092839
Authority: WO
Inventors: 杨灿军; 李佳; 陈燕虎; 赵一冰; 刘斯悦; 苏杭; 夏庆超; 郑方舟
Original assignee: 浙江大学
Priority date: 2022-03-17
Filing date: 2022-05-13
Publication date: 2023-09-21
Also published as: CN114742941A

Abstract

A real-time panoramic imaging method and device for an underwater cleaning robot. The method comprises: constructing a three-dimensional projection curved surface of panoramic imaging on the basis of a working scene of a cleaning robot on a marine conduit; acquiring in real time multi-angle images collected by a camera on the cleaning robot, and splicing the multi-angle images to obtain a spliced panorama; performing conduit edge detection on the spliced panorama to obtain a real-time conduit edge line; constructing a transformation equation on the basis of pixel points on the real-time conduit edge line, a homography matrix, and pixel points on a reference conduit edge line, and solving for the transformation equation to obtain a real-time rotation angle contained in the homography matrix; and calling, from a texture mapping library according to the real-time rotation angle, a texture mapping matrix corresponding to the real-time rotation angle, and mapping the spliced image onto the three-dimensional projection curved surface on the basis of the texture mapping matrix to realize panoramic imaging. The real-time panoramic imaging method and device for the underwater cleaning robot can improve the efficiency and the accuracy of panoramic imaging of marine jacket scenes.

Description

Real-time panoramic imaging method and device for underwater cleaning robot

Technical field

The invention belongs to the field of panoramic imaging, and specifically relates to a real-time panoramic imaging method and device for an underwater cleaning robot.

Background technique

Offshore oil platforms have made great progress as important equipment for exploiting offshore oil and gas resources. The jacket is an indispensable support structure for most offshore oil platforms. It has been immersed in seawater for a long time and has become a breeding ground for the reproduction and growth of marine life. A large number of marine organisms such as oysters and barnacles grow on the surface of the jacket. These marine organisms adhere to the surface of the jacket. During their metabolism, they produce various chemical substances, and the acids among them have a great impact on the jacket. The corrosive nature of the jacket reduces the service life of the jacket. In order to alleviate the adverse effects of marine organisms on the jacket, it is crucial to regularly clean the marine organisms attached to the surface of the jacket on offshore oil platforms.

Underwater cleaning robots can regularly clean marine organisms attached to the surface of the jacket of offshore oil platforms. However, existing underwater cleaning robots use a reflective panoramic imaging system. By installing a reflector on the top of the cleaning robot, image information such as the robot's position and posture on the pipeline can be obtained. However, this method encounters big problems in practice. First, the field of view seen by the robot through the reflector is very small, and it can only observe image information within 20cm around the machine. Secondly, installing a reflector on the top of the cleaning robot will cause the height of the whole machine to be high. When encountering a surge, the cleaning robot will be subject to a large impact, which may easily cause the robot to detach from the wall, fall into the sea, and other dangerous situations. , the robot's obstacle-crossing performance and cross-pipe performance also have great limitations.

Three-dimensional panoramic imaging technology is used to replace the mirror method to solve the above problems. Three-dimensional panoramic imaging technology shoots existing scenes from multiple angles, stitches and fuses multiple pictures, and finally projects them onto a curved surface to create an effect similar to the real world. Three-dimensional panoramic imaging technology is mainly divided into two types: columnar panorama and spherical 360° panorama. Cylindrical panorama projects the panoramic picture taken along the horizontal direction onto a cylindrical plane, and then places the perspective inside the cylinder to achieve the purpose of horizontal panorama. The spherical 360° panorama uses multi-angle panoramic shooting along the horizontal and vertical directions. The final imaging surface is a sphere, that is, a projection surface with countless symmetry axes, which expands the imaging perspective.

However, in marine jacket scenarios, the projection surface is only symmetrical with respect to the x-axis and y-axis. When the cleaning robot is in different postures on the round pipe, the perspectives of each camera of the cleaning robot are different, and the mapping relationship between the camera and the three-dimensional model is also different. Simply use the established projection surface and panoramic image directly. The mapping relationship between them is not enough to correctly map the image texture to the projection surface to obtain a correct panoramic image.

Contents of the invention

In view of the above technical problems, the purpose of the present invention is to provide a real-time panoramic imaging method and device for an underwater cleaning robot. By detecting the posture of the cleaning robot on the pipeline in real time, and adjusting the texture mapping relationship in real time according to the posture, the image texture can be corrected. Ground mapping onto the projection surface to achieve panoramic imaging in ocean jacket scenes.

In order to achieve the above-mentioned object of the invention, the embodiment provides a real-time panoramic imaging method for an underwater cleaning robot, which includes the following steps:

Construct a three-dimensional projection surface for panoramic imaging based on the working scene of the cleaning robot on the marine pipeline;

Acquire multiple images collected by the camera on the cleaning robot in real time, and stitch the multiple images to obtain a stitched panorama;

Perform catheter edge detection on the stitched panorama to obtain real-time catheter edge lines;

Construct a transformation equation based on the pixels on the real-time catheter edge line, the homography matrix and the pixel points on the reference catheter edge line, and solve the transformation equation to obtain the real-time rotation angle contained in the homography matrix;

According to the real-time rotation angle, the texture mapping matrix corresponding to the real-time rotation angle is retrieved from the texture mapping library, and based on the texture mapping matrix, the splicing map is mapped to the three-dimensional projection surface to achieve panoramic imaging.

In one embodiment, the constructed projection curved surface is a curved surface formed by intercepting the hemispherical curved surface with a cylinder. Preferably, when intercepting, the height of the cylinder is parallel to the diameter of the hemispherical curved surface.

In one embodiment, the EDLines algorithm is used to perform catheter edge detection on the stitched panorama, and a threshold detection method is used for the detected line segments to filter long line segments as real-time catheter edge lines.

In one embodiment, images from four directions collected by four cameras are acquired in real time, and the four images are spliced to obtain a stitched panorama.

In one embodiment, a cleaning robot model is built in the three-dimensional projection curved surface. After obtaining the real-time rotation angle, the position and posture of the cleaning robot model is adjusted according to the real-time rotation angle to obtain a complete real-time panoramic view.

In order to achieve the above object of the invention, another embodiment provides a real-time panoramic imaging device for an underwater cleaning robot, including:

The three-dimensional projection surface building module is used to construct a three-dimensional projection surface for panoramic imaging based on the working scene of the cleaning robot on the ocean pipeline;

The stitching panorama building module is used to obtain multi-angle images collected by the camera on the cleaning robot in real time, and stitch the multi-angle images to obtain a stitched panorama;

The edge detection module is used to perform catheter edge detection on the stitched panorama to obtain real-time catheter edge lines;

The real-time rotation angle calculation module is used to construct a transformation equation based on the pixels on the real-time catheter edge line, the homography matrix and the pixel points on the reference catheter edge line, and solve the transformation equation to obtain the real-time rotation contained in the homography matrix. angle;

The panoramic imaging module is used to retrieve the texture mapping matrix corresponding to the real-time rotation angle from the texture mapping library according to the real-time rotation angle, and map the splicing map to the three-dimensional projection surface based on the texture mapping matrix to achieve panoramic imaging.

Compared with the prior art, the beneficial effects of the present invention include at least:

After the spliced panorama is obtained through real-time collection of multi-angle images, the real-time catheter edge line is obtained through edge detection. Based on the mapping relationship between the pixels of the real-time catheter edge line and the reference catheter edge line on the homography matrix, a transformation equation is constructed. By solving Transform the equation to obtain the real-time rotation angle, and use the pre-built texture mapping library to call the corresponding texture mapping matrix for panoramic imaging, which can improve the efficiency and accuracy of panoramic imaging of ocean jacket scenes.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of the real-time panoramic imaging method of the underwater cleaning robot provided by the embodiment;

Figure 2 is a three-dimensional projection curved surface provided by the embodiment;

Figure 3 is a schematic diagram of catheter edge detection provided by the embodiment, in which (a) is a spliced panorama, (b) is an edge detection result diagram, (c) is a threshold screening detection catheter straight line, and (d) is a real-time catheter edge line ;

Figure 4 is the dimensional relationship of robots in different states provided by the embodiment;

FIG. 5 is a schematic structural diagram of the real-time panoramic imaging device of the underwater cleaning robot provided by the embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and do not limit the scope of the present invention.

In order to solve the problem of inaccurate panoramic imaging caused by the inaccurate real-time panoramic imaging of the cleaning robot in the ocean pipe scene. The embodiment provides a real-time panoramic imaging method for an underwater cleaning robot.

Figure 1 is a flow chart of the real-time panoramic imaging method of an underwater cleaning robot provided by the embodiment. As shown in Figure 1, the real-time panoramic imaging method of an underwater cleaning robot provided by the embodiment includes the following steps:

Step 1: Construct a three-dimensional projection surface for panoramic imaging based on the working scene of the cleaning robot on the ocean pipeline.

In the embodiment, the cleaning robot is equipped with multiple cameras for collecting images. Specifically, the cameras can be distributed in four directions, front, rear, left and right, for collecting images from different angles. The cleaning robot crawls on the marine conduit. Marine conduits are generally cylindrical, so the constructed projection surface is based on the hemispherical surface and is formed by intercepting the hemispherical surface with a cylinder, as shown in Figure 2. The specific construction process is:

The surface equation of the cylinder is: x ₁ ² +y ₁ ² =r, and the surface equation of the sphere is:

When y ₁ > y ₂ , the projection surface is composed of the following formula, which is used to draw the cylindrical texture:

In the above formula, i, j are the sequence of surface nodes, (x, y, z) are the spatial coordinates of the projected surface, r is the cylinder radius, h is the cylinder height, m ₁ is the number of axial vertices, and n ₁ is The number of vertices in the circumferential direction.

When y ₁ > y ₂ , the projection surface is composed of the following formula, which is used to draw spherical texture:

In the above formula, R is the radius of the sphere, r ₁ is the radius of the curved surface circle at a certain height, and is given by

Calculated, m is the number of longitude lines, and n ₂ is the number of latitudinal lines.

Step 2: Acquire multiple images collected by the camera on the cleaning robot in real time, and stitch the multiple images to obtain a stitched panorama.

In the embodiment, multi-threading is used to control multiple cameras to simultaneously collect multiple images. It should be noted that during the actual acquisition process, the angles between the multiple cameras and the catheter are the same to ensure the accuracy of the stitched panorama. The angle can be set to 45°, 60°, 120° and 135°, and the stitched panorama obtained from different angles will be different.

In the embodiment, two-dimensional panoramic imaging technology is used to stitch multiple images to form a stitched panorama. Specifically, it includes: camera distortion correction, local feature point detection, feature point matching, image splicing, and seam melting processing on multiple images to obtain a spliced panorama.

Step 3: Perform catheter edge detection on the stitched panorama to obtain real-time catheter edge lines.

In the embodiment, first, the EDLines algorithm is used to detect the edge of the catheter on the spliced panorama. After detecting the edge points, the straightness criterion and the least squares straight line fitting method are used to extract line segments from the generated pixel chain. Then, the line segments are obtained based on the detection. The line segments use threshold detection method to filter long line segments as real-time catheter edge lines, as shown in Figure 3. This edge detection method is fast and highly accurate.

Step 4: Construct a transformation equation based on the pixels on the real-time catheter edge line, the homography matrix and the pixel points on the reference catheter edge line, and solve the transformation equation to obtain the real-time rotation angle contained in the homography matrix.

In the embodiment, after performing edge detection on the catheter and extracting the edge line of the catheter, the angle between the camera and the catheter can be obtained through homography transformation, and the position and posture of the cleaning robot on the catheter and the texture mapping matrix in three-dimensional imaging can be adjusted. This achieves the purpose of displaying panoramic images in real time.

In the embodiment, the constructed transformation equation is:

q _b = H _ba q _a

Among them, q _b represents the pixel coordinates on the real-time catheter edge line, q _a represents the pixel coordinates on the reference edge line, H _ba represents the homography matrix, and the rotation of the two cameras corresponding to pixel plane b and pixel plane a Matrix R _ba and translation _matrix t _ab , two camera intrinsic parameter matrices Ka and K _b , pixel plane parameters (

d _a ) consisting of, where,

Represents the transpose of the normal vector of the pixel plane a, d _a represents the distance to the pole in polar coordinates, and the homography matrix H _ba is:

During the movement of the cleaning robot on the catheter, there is no relative movement between the cleaning robot coordinate space and the catheter coordinate space in the x and y directions, that is, the center of the cleaning robot and the center of the catheter are always aligned. The cleaning robot will only rotate and move along the radius of the conduit due to the change of the cleaning robot's posture, as shown in Figure 4, where (a) is a schematic diagram of the vertical state, (b) is a dimensional relationship diagram of the vertical state, (c) is a schematic diagram of the horizontal state, and (d) is a dimensional relationship diagram of the horizontal state.

When the robot is in the vertical state and the horizontal state respectively, it corresponds to the maximum and minimum distance between the robot and the circular tube in the radial direction. It can be seen from Figure 4:

Among them, R _c is the radius of the conduit, r is the wheel diameter of the cleaning robot, W is the axial distance between the left wheel and the right wheel of the cleaning robot, and L is the distance between the center of the front wheel and the center of the rear wheel of the cleaning robot; therefore, there is a translation vector t _ab = [0 0 z].

When running the panoramic imaging algorithm under different catheter scenarios, the only variable in the z _max formula is the catheter radius R _c .

The various parameters of the cleaning robot are shown in Table 1:

Table 1

Substituting various parameters of the cleaning robot into the z _max formula, and deriving the derivative of R, it is found that z _max is monotonically decreasing with respect to R. Therefore, when R is the smallest, z _max is the largest. In this scenario, the single direction of the cleaning robot along the radial direction of the cleaning robot is the largest, and has the greatest impact on the solution of the rotation angle of the cleaning robot. In the panoramic imaging system, the distance between the cleaning robot's viewpoint and the circular tube is greater than 1000mm. Compared to the maximum displacement, the motion of the cleaning robot is ignored, so the translation vector t _ab = (0,0,0). The above formula can be simplified to:

After simplification, the rotation matrix R _ba is:

_Rba _＝ _Kb ^- _1HbaKa

K _a and K _b are the internal parameter matrices of the camera, expressed as:

f _xa , f _xb , f _ya , f _yb are the focal length parameters of the camera, u _0a , v _0a , u _0b , v _0b are the main point displacement vectors. When performing the projection transformation matrix, the focal length parameters f _x and f _y of the two cameras are both Equal to 1, the displacement vector u _0a , v _0a , u _0b , v _0b is half the length and width of the spliced panorama;

It is also known that the expression of the rotation matrix is:

Then the solution is:

Among them, α, β, and γ represent the rotation angles in the x, y, and z directions respectively.

Step 5: According to the real-time rotation angle, the texture mapping matrix corresponding to the real-time rotation angle is retrieved from the texture mapping library, and based on the texture mapping matrix, the splicing map is mapped to the three-dimensional projection surface to achieve panoramic imaging.

When the cleaning robot is in different postures on the pipe, the perspectives of each camera of the cleaning robot are different, and the mapping relationship between the camera and the three-dimensional projection surface is also different. Therefore, it is necessary to be able to detect the posture of the robot on the circular pipe in real time. Only by adjusting the texture mapping relationship in real time according to the posture can panoramic imaging in ocean jacket scenes be achieved.

During real-time three-dimensional imaging, the texture mapping matrix corresponding to the real-time rotation angle is retrieved from the texture mapping library according to the real-time rotation angle, and based on the texture mapping matrix, the splicing map is mapped to the three-dimensional projection surface to achieve panoramic imaging.

It should be noted that the texture mapping matrix in the texture mapping library is mapped through the OpenGL texture mapping principle. Specifically, the texture mapping process includes: before texture mapping, the mapping relationship between three-dimensional points and pixels needs to be established. The mapping relationship between them is achieved through texture coordinates. By specifying the texture coordinates of several key points on a three-dimensional model, a one-to-one correspondence between three-dimensional points and two-dimensional pixels is achieved, and the remaining points are interpolated through the fragment shader to achieve complete mapping of the image. The value range of texture coordinates is 0~1, and the lower left corner of the texture image is the origin of texture coordinates.

In the embodiment, a cleaning robot model is also built in the three-dimensional projection surface. After obtaining the real-time rotation angle, the position and posture of the cleaning robot model are adjusted according to the real-time rotation angle to obtain a complete real-time panoramic view.

The real-time panoramic imaging method of the underwater cleaning robot provided by the above embodiments detects the posture of the cleaning robot on the pipeline in real time, and adjusts the texture mapping relationship in real time according to the posture to correctly map the image texture onto the projection surface, thereby realizing the vision in the ocean. Panoramic imaging of jacket scenes.

FIG. 5 is a schematic structural diagram of the real-time panoramic imaging device of the underwater cleaning robot provided by the embodiment. As shown in Figure 5, the real-time panoramic imaging device provided by the embodiment includes:

It should be noted that when the real-time panoramic imaging device of the underwater cleaning robot provided in the above embodiment performs real-time panoramic imaging, the division of the above functional modules should be used as an example. The above functions can be allocated to different functional modules as needed. Complete, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the real-time panoramic imaging device of the underwater cleaning robot provided in the above embodiments and the real-time panoramic imaging method embodiment of the underwater cleaning robot belong to the same concept. For details of the implementation process, please refer to the real-time panoramic imaging method embodiment of the underwater cleaning robot. I won’t go into details here.

The above-described specific embodiments describe in detail the technical solutions and beneficial effects of the present invention. It should be understood that the above are only the most preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, equivalent substitutions, etc. made within the scope of the invention shall be included in the protection scope of the present invention.

Claims

A real-time panoramic imaging method for underwater cleaning robots, which is characterized by including the following steps:

Construct a three-dimensional projection surface for panoramic imaging based on the working scene of the cleaning robot on the marine pipeline;

Acquire multiple images collected by the camera on the cleaning robot in real time, and stitch the multiple images to obtain a stitched panorama;

Perform catheter edge detection on the stitched panorama to obtain real-time catheter edge lines;

Construct a transformation equation based on the pixels on the real-time catheter edge line, the homography matrix and the pixel points on the reference catheter edge line, and solve the transformation equation to obtain the real-time rotation angle contained in the homography matrix;

According to the real-time rotation angle, the texture mapping matrix corresponding to the real-time rotation angle is retrieved from the texture mapping library, and based on the texture mapping matrix, the splicing map is mapped to the three-dimensional projection surface to achieve panoramic imaging.
The real-time panoramic imaging method of an underwater cleaning robot according to claim 1, characterized in that the constructed projection curved surface is a curved surface formed by intercepting the hemispherical curved surface with a cylinder. During the interception, the height of the cylinder is equal to the hemispherical curved surface. diameter is parallel.
The real-time panoramic imaging method of an underwater cleaning robot according to claim 1, characterized in that the EDLines algorithm is used to perform conduit edge detection on the spliced panorama, and a threshold detection method is used for the detected line segments to screen long line segments as real-time conduits. edge line.
The real-time panoramic imaging method of underwater cleaning robots according to claim 1, characterized in that the constructed transformation equation is:

q b = H ba q a

Among them, q b represents the pixel coordinates on the real-time catheter edge line, q a represents the pixel coordinates on the reference edge line, H ba represents the homography matrix, and the rotation of the two cameras corresponding to pixel plane b and pixel plane a Matrix R ba and translation matrix t ab , two camera internal parameter matrices Ka and K b , pixel plane parameters
consists of,
Represents the transpose of the normal vector of the pixel plane a, d a represents the distance to the pole in polar coordinates, and the homography matrix H ba is:
The real-time panoramic imaging method of underwater cleaning robots according to claim 4, characterized in that when solving the transformation equation, let t ab = (0,0,0), then the homography matrix H ba is simplified to:

After simplification, the rotation matrix R ba is:

Rba ＝ Kb - 1HbaKa

K a and K b are the internal parameter matrices of the camera, expressed as:

f xa , f xb , f ya , f yb are the focal length parameters of the camera, u 0a , v 0a , u 0b , v 0b are the main point displacement vectors. When performing the projection transformation matrix, the focal length parameters f x and f y of the two cameras are both Equal to 1, the displacement vector u 0a , v 0a , u 0b , v 0b is half the length and width of the spliced panorama;

It is also known that the expression of the rotation matrix is:

Then the solution is:

Among them, α, β, and γ represent the rotation angles in the x, y, and z directions respectively.
The real-time panoramic imaging method of an underwater cleaning robot according to claim 1, characterized in that images in four directions collected by four cameras are acquired in real time, and the four images are spliced to obtain a spliced panorama.
The real-time panoramic imaging method of an underwater cleaning robot according to claim 1, characterized in that a cleaning robot model is also built in the three-dimensional projection curved surface. After the real-time rotation angle is obtained, the cleaning robot model is imaged according to the real-time rotation angle. Position adjustment to obtain a complete real-time panorama.
A real-time panoramic imaging device for an underwater cleaning robot, which is characterized by including:

The three-dimensional projection surface building module is used to construct a three-dimensional projection surface for panoramic imaging based on the working scene of the cleaning robot on the ocean pipeline;

The stitching panorama building module is used to obtain multi-angle images collected by the camera on the cleaning robot in real time, and stitch the multi-angle images to obtain a stitched panorama;

The edge detection module is used to perform catheter edge detection on the stitched panorama to obtain real-time catheter edge lines;

The real-time rotation angle calculation module is used to construct a transformation equation based on the pixels on the real-time catheter edge line, the homography matrix and the pixel points on the reference catheter edge line, and solve the transformation equation to obtain the real-time rotation contained in the homography matrix. angle;

The panoramic imaging module is used to retrieve the texture mapping matrix corresponding to the real-time rotation angle from the texture mapping library according to the real-time rotation angle, and map the splicing map to the three-dimensional projection surface based on the texture mapping matrix to achieve panoramic imaging.