CN115514885A - Monocular and binocular fusion-based remote augmented reality follow-up perception system and method - Google Patents

Abstract

The invention discloses a remote augmented reality follow-up sensing system and a remote augmented reality follow-up sensing method based on monocular and binocular fusion, which belong to the field of intelligent engineering machinery, wherein the system comprises a monocular and binocular fusion follow-up intelligent sensing module, a server-side intelligent processing module and a user-side augmented reality module, wherein the monocular and binocular fusion follow-up intelligent sensing module is used for acquiring RGB information and depth data of a construction scene and sending the RGB information and the depth data to the edge end of a server through a wireless transmission technology; the server-side intelligent processing module is used for executing calculations required by bucket attitude estimation, bucket tip positioning, accurate environment perception and augmented reality; the user-side augmented reality module comprises three-dimensional display equipment and an operation console, the three-dimensional display equipment is used for displaying the three-dimensional information fusion image processed by the server-side algorithm processing module, and the operation console is used for controlling engineering machinery operation on a construction site. The invention is based on the remote follow-up intelligent sensing technology, and can solve the problems of absence of telepresence and distance sense of an operator and the like.

Description

Remote augmented reality follow-up perception system and method based on single and binocular fusion

technical field

The invention relates to the field of intelligent construction machinery, in particular to a remote augmented reality follow-up perception system and method based on single and binocular fusion.

Background technique

In the domestic and international markets, the demands of construction machinery users for equipment are constantly diversifying, and people's requirements for operating comfort are getting higher and higher. Especially in arduous, high-risk, and repetitive operating environments, due to the inability to guarantee the health of workers and the serious shortage of skilled operators, it is becoming more and more difficult for construction companies to recruit workers. In these environments and accidental disasters, a large number of construction machinery such as loaders and excavators are often required for emergency rescue such as site cleaning and road restoration. People are increasingly eager to obtain a comfortable working environment and the same work efficiency. However, at present, the environment perception of construction machinery remote control system generally adopts visible light camera or laser radar whose position is fixed relative to the machine body, which limits the field of view of intelligent perception to a certain extent.

The invention with the application number 201810268923.3 discloses a stereo vision follow-up system applied to disaster area search. The operator wears VR glasses and Bluetooth headsets to obtain real-time audio and video in the disaster area, and uses the head posture to control the synchronous movement of the UAV camera. ; The invention with application number 202010882933.3 discloses a humanoid binocular follow-up virtual reality system suitable for remote operation of robots. The binocular camera is placed on a two-dimensional platform, and the binocular camera is controlled by the follow-up mechanism to follow the movement of the operator. Head movements synchronize pitch and rotation, thereby changing the binocular camera perspective. Although the above technology provides operators with an immersive three-dimensional sense, there are still some shortcomings: only directly providing binocular images for operators will still lead to the problem of lack of presence and distance.

Contents of the invention

The technical problem to be solved in the present invention is to provide a remote augmented reality follow-up perception system and method based on single and binocular fusion. The system is based on remote follow-up intelligent perception technology and can provide the operator with real-time attitude estimation of the bucket And important information such as the positioning of the bucket tip of the bucket solves the problems of the operator's lack of sense of presence and distance.

In order to solve the problems of the technologies described above, the technical solution adopted in the present invention is:

A remote augmented reality follow-up perception system based on single-eye fusion, including an edge-side single-eye fusion follow-up intelligent perception module, a server-side intelligent processing module, and a user-end augmented reality module;

The single and binocular fusion follow-up intelligent perception module at the edge end collects RGB information and depth data of the construction scene based on the single and binocular fusion method, and sends them to the server through wireless transmission;

The server-side intelligent processing module is used to perform calculations required for bucket attitude estimation, bucket tip positioning, precise environment perception, and augmented reality;

The user-side augmented reality module includes a three-dimensional display device for displaying the three-dimensional information fusion image processed by the server-side algorithm processing module and a console for controlling construction machinery operations on the construction site.

The further improvement of the technical solution of the present invention lies in that: the edge-end single- and binocular fusion follow-up intelligent perception module includes a single- and binocular vision sensor, an edge-end AI processor, and a follow-up pan/tilt; the edge-end AI processor is used to realize Fusion perception of monocular and binocular RGB information, depth data, camera pose information and key target information, and read the head posture of the operator wearing video glasses on the user end, and then control the single and double The follow-up pan/tilt of the visual sensor quickly synchronizes the operator's head posture to achieve a follow-up effect.

The further improvement of the technical solution of the present invention lies in that: the edge AI processor is connected to the single and binocular vision sensor through the USB data line, and reads monocular RGB information, binocular grayscale information and camera pose information in real time; The binocular grayscale information uses the stereo matching algorithm to restore the depth map, converts the 2D points on the depth map to the 3D points in the world coordinate system through the internal and external parameters of the binocular depth camera, and then converts the world coordinate system through the internal and external parameters of the monocular RGB camera. The 3D points below are projected onto the RGB image to achieve single and binocular information fusion; real-time detection of key target information based on the target detection algorithm, and the position of the key target is sent to the server-side intelligent processing module for bucket tip positioning.

The further improvement of the technical solution of the present invention lies in: the server-side intelligent processing module, the bucket attitude estimation and positioning algorithm adopts an efficient template matching method based on a sparse area and a real-time lightweight deep learning network to track the bucket state in real time, accurately The environment perception algorithm adopts the monocular vision SLAM algorithm, which can provide the necessary environment map information and construction machinery attitude information for safe operation.

The further improvement of the technical solution of the present invention lies in that: while displaying the fused image processed by the server, the 3D display device can accurately capture the posture of the operator's head, and send it to the server in real time, and then be read by the edge-side AI processor. Take, control the follow-up pan-tilt to follow; the console can also be used to provide the operator with a real manipulation environment, and the effect of being on the scene can be achieved by cooperating with video glasses.

A remote augmented reality follow-up perception method based on single and binocular fusion, comprising the following steps:

Step 1. Place the edge-end single and binocular fusion follow-up intelligent perception module in the construction machinery cab. After ensuring that the front glass of the cab is unobstructed, turn on the power of the edge-end processor and put it in a waiting state for intelligent processing with the server. The module establishes a communication connection;

Step 2, turn on the power of the server, make it in the monitoring state, and wait for the establishment of a communication connection with the edge-side single-eye fusion follow-up intelligent perception module and the user-side augmented reality module;

Step 3, the operator enters the console, wears video glasses, waits for the construction interface to appear on the 3D display device, and then starts the remote control operation;

Step 4, the edge processor reads the operator's head posture data, controls the follow-up pan/tilt to update the posture in real time, and sends the RGB information and depth data of the construction scene to the server through wireless transmission technology;

Step 5, the server-side intelligent processing module performs bucket attitude estimation and bucket tip positioning through the received RGB information and depth data, and constructs environmental map information, and finally integrates bucket attitude information, bucket tip position information and bucket The fusion image of the actual distance information between the tip and the surrounding objects such as dump trucks is sent to the client for display;

Step 6: The operator at the user end uses the fused image and on-site 3D information displayed by the video glasses to cooperate with the console to remotely control the construction machinery operations, and at the same time the video glasses capture the operator's head posture in real time and send it to the server for processing at the edge device read;

Step 7, repeat steps 4 to 6.

A method for tracking bucket posture based on sparse areas, wherein a remote augmented reality follow-up perception system based on single-eye fusion is used, comprising the following steps:

S1, place the bucket under natural light, avoid reflective objects around, use camera equipment to circle around the bucket, and take 30 photos, and keep the bucket at the center of the image when shooting;

S2, open the RealityCapture software, use 30 photos of the bucket to generate a three-dimensional model of the bucket, and the three-dimensional model is exactly the same as the real bucket;

S3, place virtual cameras at 2562 different positions around the 3D model to render the 3D model, use the rendered image to collect sparse contour points of the bucket in its current posture, and back-project the contour points to the coordinate system of the bucket 3D model for storage , while saving the normal vector of the contour point and the direction vector of the current attitude; finally generating 2562 template views;

S4. Given the initial pose of the bucket, multiply the direction vectors of all template views by the initial pose to find a template view that is consistent with the initial pose; project the contour points of the template view onto the current real image, and the contour points along the The first 18 pixels in the normal direction are designated as bucket pixels, and the last 18 pixels are designated as background pixels, and the bucket is separated from the background to obtain the real outline of the bucket;

S5, using the distance between the model contour point and the real contour point along the normal direction to estimate the real posture of the bucket, and then realize the bucket tracking.

Owing to having adopted above-mentioned technical scheme, the technical progress that the present invention obtains is:

1. Due to the adoption of single and binocular follow-up intelligent perception technology in the present invention, the operator can get rid of the limitation of the fixed viewing angle of the traditional remote operation in the over-the-horizon situation, and can freely obtain different viewing angles of the construction site through the change of the head posture Monocular RGB information and binocular depth data.

2. Since the present invention adopts an efficient and real-time bucket tracking and bucket tip positioning algorithm, it can capture the posture of the bucket and the spatial position of the bucket tip in real time through images, and then cooperate with augmented reality technology to provide operators with real-time information of the bucket. Attitude information and position information get rid of the limitation that traditional teleoperation cannot provide operators with a sense of distance.

3. Since the present invention adopts the SLAM+YOLO environment perception algorithm, it can transform the local coordinates of the bucket tip to the starting point coordinate system of the camera in real time, thereby providing global environment perception and providing a basis for efficient operation of construction machinery.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings on the premise of not paying creative labor;

Figure 1 is a schematic diagram of the overall structure of a remote augmented reality follow-up perception system based on single-eye and binocular fusion in an embodiment of the present invention;

Fig. 2 is a schematic diagram 2 of the overall structure of the remote augmented reality follow-up perception system based on single and binocular fusion in an embodiment of the present invention;

Fig. 3 is the structural diagram of the follow-up pan-tilt platform of the remote augmented reality follow-up perception system based on single-eye and binocular fusion in an embodiment of the present invention;

FIG. 4 is a schematic diagram of single-eye fusion of a remote augmented reality follow-up perception system based on single-eye fusion in an embodiment of the present invention;

Fig. 5 is a schematic diagram of bucket tracking of the remote augmented reality follow-up perception system based on single-eye and binocular fusion in an embodiment of the present invention;

Fig. 6 is a schematic diagram of bucket tip positioning of the remote augmented reality follow-up perception system based on single- and binocular fusion in an embodiment of the present invention;

Among them, 1. Edge AI processor, 2. Single and binocular vision sensor, 3. Y-axis brushless DC motor, 4. X-axis DC brushless motor, 5. RGB camera, 6. Left-eye depth camera, 7. Right-eye Depth camera.

detailed description

It should be noted that the terms "comprising" and "having" in the description and claims of the present invention and the above drawings, as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, including a series of steps or units A process, method, system, product or device is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to the process, method, product or device.

Below in conjunction with accompanying drawing and embodiment the present invention is described in further detail:

As shown in Figures 1 and 2, a remote augmented reality follow-up perception system based on single- and binocular fusion includes an edge-end single- and binocular fusion follow-up intelligent perception module, a server-side intelligent processing module, and a user-side augmented reality module;

The edge-side single-eye fusion follow-up intelligent perception module is placed in the cab of construction machinery to collect RGB information and depth data of the construction scene, and send them to the server through wireless transmission technology;

The server-side intelligent processing module is used to perform calculations required for bucket attitude estimation, bucket tip positioning, precise environment perception, and augmented reality;

The user-side augmented reality module includes video glasses and a control console. The video glasses are used to display the 3D information fusion images processed by the server-side algorithm processing module, and the control console is used to control construction machinery operations on the construction site.

As shown in Figure 3, the servo gimbal integrates a single-eye and binocular vision sensor 2, an edge-side AI processor 1, an x-axis brushless DC motor 4, and a y-axis brushless DC motor 3. The two degrees of freedom of the axis and the y-axis, through the motor control technology (FOC motor control technology), make the DC motor have small torque fluctuations, high efficiency, low noise, fast dynamic response, and can quickly synchronize the operator's head posture.

As shown in Figure 4, the single and binocular vision sensor 2 has three cameras, namely an RGB camera 5, a left-eye depth camera 6, and a right-eye depth camera 7. The RGB camera 5 can collect RGB information including the bucket and transmit it wirelessly. Send it to the server-side intelligent processing module for bucket tracking. The left-eye depth camera 6 and right-eye depth camera 7 can collect grayscale information including the bucket, and maintain the same resolution as the RGB image through upsampling. SGBM binocular stereo The matching algorithm performs depth restoration based on the right-eye image, and finally re-projects the pixels of the depth image onto the RGB image through the coordinate transformation relationship between the right-eye depth camera 7 and the RGB camera 5 for single and binocular information fusion, so that the attitude information of the bucket can be transformed In the RGB coordinate system, it is convenient for subsequent positioning of the bucket tip.

As shown in Figure 5, the bucket attitude estimation algorithm deployed by the server-side intelligent processing module can use the three-dimensional model of the bucket to generate template views of each attitude, and then match the real view of the bucket returned by the pan/tilt, and then Accurate estimation of the attitude of the bucket is obtained, and then the rendered image of the current attitude is superimposed on the real image to visualize the tracking results.

As shown in (a) of Figure 6, the bucket tip positioning algorithm deployed by the server-side intelligent processing module can use the 2D image rendered by the 3D model of the bucket to obtain the 2D image plane coordinates of the bucket tip, and then use the virtual camera The internal and external parameters of the bucket are obtained in the three-dimensional coordinates of the bucket tip in the three-dimensional model coordinate system. As shown in (b) in Figure 6, the bucket tip coordinates in the 3D model coordinate system can be transformed into the camera coordinate system through the camera pose, and at the same time, the YOLOv4 target detection algorithm deployed by the edge AI processor 1 detects the position of the dump truck. Position, combined with the depth data to obtain the three-dimensional coordinates of the truck center point in the camera coordinate system, and then calculate the Euclidean distance from the bucket tip to the truck center point, and provide the operator with relative distance information from the bucket tip to the truck. When the relative distance is less than the set When a certain threshold is reached, the operator is alerted to avoid a collision. As shown in (c) of Figure 6, through the SLAM+YOLO algorithm deployed on the server side, the points outside the box represent the detected static feature points, and the points inside the box represent the dynamic feature points of the bucket and its surroundings. Calculate the transformation relationship between the current frame and the first frame through the static feature points of each frame, so as to convert the coordinates of the bucket tip at each moment into the coordinates in the camera coordinate system at the initial moment, that is, the coordinates in the global coordinate system. The global bucket coordinates can better estimate the construction status of the current construction machinery.

A remote augmented reality follow-up perception method based on single and binocular fusion, comprising the following steps:

Step 1. Place the edge-end single and binocular fusion follow-up intelligent perception module in the construction machinery cab. After ensuring that the front glass of the cab is unobstructed, turn on the power of the edge-end processor and put it in a waiting state for intelligent processing with the server. The module establishes a communication connection;

Step 2, turn on the power of the server, make it in the monitoring state, and wait for the establishment of a communication connection with the edge-side single-eye fusion follow-up intelligent perception module and the user-side augmented reality module;

Step 3, the operator enters the console, wears video glasses, waits for the construction interface to appear on the 3D display device, and then starts the remote control operation;

Step 4, the edge processor reads the operator's head posture data, controls the follow-up pan/tilt to update the posture in real time, and sends the RGB information and depth data of the construction scene to the server through wireless transmission technology;

Step 5, the server-side intelligent processing module performs bucket attitude estimation and bucket tip positioning through the received RGB information and depth data, and constructs environmental map information, and finally integrates bucket attitude information, bucket tip position information and bucket The fusion image of the actual distance information between the tip and the surrounding objects such as dump trucks is sent to the client for display;

Step 6: The operator at the user end uses the fused image and on-site 3D information displayed by the video glasses to cooperate with the console to remotely control the construction machinery operations, and at the same time the video glasses capture the operator's head posture in real time and send it to the server for processing at the edge device read;

Step 7, repeat steps 4 to 6.

Specifically, when the construction machinery performs remote control operations, the operator is located on the console and wears video glasses. At this time, the image displayed by the video glasses is the fused image processed by the server-side intelligent processing module, which includes the attitude information of the bucket, the distance information between the bucket tip and the follow-up platform, and the distance between the bucket tip and the dump truck The operator can use this information to judge the construction status of the construction site and control the construction machinery operations on the construction site through the console. The follow-up pan/tilt placed in the construction machinery cab on the construction site can rotate freely according to the operator's head posture, collect the RGB image and depth image of the site in real time, and then send them to the server-side intelligent processing module through wireless transmission technology for attitude estimation, Tip positioning, environment perception and image fusion based on augmented reality technology, and finally the server sends the fused image to the video glasses for display.

The over-the-horizon remote augmented reality follow-up intelligent perception system and method based on single and binocular fusion can be used in the existing actual intelligent engineering machinery teleoperation system, and the intelligent follow-up pan-tilt can replace the drone for environment perception. The vision-based bucket tracking method can replace the IMU sensor-based tracking method, and the video glasses can replace the large-screen two-dimensional display interface, providing teleoperators with an immersive sense of presence.

The present invention also provides a bucket posture tracking method based on a sparse area, using a remote augmented reality follow-up perception system based on single and binocular fusion. The tracking method realizes the function through the following steps:

S1, place the bucket under natural light, and try to avoid reflective objects around. Use any camera equipment to circle around the bucket and take 30 photos. When shooting, try to keep the bucket in the center of the image.

S2, open the RealityCapture software, use 30 photos of the bucket to generate a three-dimensional model of the bucket, and the three-dimensional model is exactly the same as the real bucket.

S3, place virtual cameras at 2562 different positions around the 3D model to render the 3D model, use the rendered image to collect sparse contour points of the bucket in its current posture, and back-project the contour points to the coordinate system of the bucket 3D model for storage , while saving the normal vector of the contour point and the direction vector of the current pose. Finally, 2562 template views can be generated.

S4, given the initial pose of the bucket, multiply the direction vectors of all template views with the initial pose to find the template view closest to the initial pose. Project the contour points of the template view onto the current real image. The first 18 pixels of the contour point along the normal direction are designated as bucket pixels, and the last 18 pixels are designated as background pixels. Then the bucket and the background can be separated, and then obtained The true outline of the bucket.

S5, using the distance between the model contour point and the real contour point along the normal direction to estimate the real posture of the bucket, and then realize the bucket tracking.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (7)

1. The utility model provides a long-range augmented reality follow-up perception system based on monocular and binocular fusion which characterized in that: the system comprises an edge end single-binocular fusion follow-up intelligent sensing module, a server end intelligent processing module and a user end augmented reality module;

the edge end single-binocular fusion follow-up intelligent sensing module is used for acquiring RGB information and depth data of a construction scene based on a single-binocular fusion method and sending the RGB information and the depth data to the server end through wireless transmission;

the server-side intelligent processing module is used for executing calculation required by bucket attitude estimation, bucket tip positioning, accurate environment perception and augmented reality;

the user-side augmented reality module comprises a three-dimensional display device and a control console, wherein the three-dimensional display device is used for displaying the three-dimensional information fusion image processed by the server-side algorithm processing module, and the control console is used for controlling the operation of the engineering machinery on a construction site.

2. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 1, wherein: the edge end single-binocular fusion follow-up intelligent sensing module comprises a single-binocular vision sensor, an edge end AI processor and a follow-up holder; the edge terminal AI processor is used for realizing the fusion perception of monocular and binocular RGB information, depth data, camera pose information and key target information, reading the head posture of an operator wearing the video glasses at a user terminal, and controlling a follow-up pan-tilt carrying monocular and binocular vision sensors to quickly synchronize the head posture of the operator by a direct current brushless motor control method, so that a follow-up effect is achieved.

3. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 2, wherein: the edge terminal AI processor is connected with the monocular and binocular vision sensor through a USB data line, and reads monocular RGB information, binocular gray scale information and camera pose information in real time; the binocular gray scale information adopts a stereo matching algorithm to recover a depth map, 2D points on the depth map are converted into 3D points under a world coordinate system through internal and external parameters of a binocular depth camera, and then the 3D points under the world coordinate system are projected onto an RGB image through the internal and external parameters of a monocular RGB camera, so that single and binocular information fusion is realized; and detecting the information of the key target in real time based on a target detection algorithm, and sending the position of the key target to a server-side intelligent processing module for positioning the bucket tip.

4. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 1, wherein: the intelligent processing module at the server side adopts an efficient sparse region template matching method and a real-time lightweight deep learning network based bucket attitude estimation and positioning algorithm to track the state of the bucket in real time, and the accurate environment perception algorithm adopts a monocular vision SLAM algorithm, so that necessary environment map information and engineering machinery attitude information can be provided for safe operation.

5. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 1, wherein: the three-dimensional display equipment can accurately capture the head posture of an operator while displaying the fused image processed by the server, and sends the head posture to the server in real time, so that the head posture is read by the edge terminal AI processor to control the follow-up cradle head to follow; the control console can also be used for providing a real control environment for an operator, and can achieve the effect of being personally on the scene by matching with video glasses.

6. A perception method of the monocular and binocular fusion based remote augmented reality follow-up perception system according to any one of claims 1-5, wherein: the method comprises the following steps:

step 1, placing an edge end single-binocular fusion follow-up intelligent sensing module in a cab of engineering machinery, and after ensuring that glass in front of the cab is not shielded, turning on a power supply of an edge end processor to enable the edge end processor to be in a waiting state, and waiting for establishing communication connection with a server end intelligent processing module;

step 2, turning on a power supply of the server side to enable the server side to be in a monitoring state, and waiting for establishing communication connection with the edge end single-binocular fusion follow-up intelligent sensing module and the user side augmented reality module;

step 3, an operator enters the console, wears the video glasses and starts remote control operation after the three-dimensional display equipment has a construction interface;

step 4, reading the head posture data of an operator by the edge end processor, controlling the servo cradle head to update the posture in real time, and simultaneously sending RGB information and depth data of a construction scene to the server end through a wireless transmission technology;

step 5, the server-side intelligent processing module carries out attitude estimation and bucket tip positioning of the bucket and environment map information construction through the received RGB information and depth data, and finally sends a fused image fused with the bucket attitude information, bucket tip position information and actual distance information between the bucket tip and objects around the dump truck to a user side for display;

step 6, the user side operator remotely controls the engineering machinery operation in a manner of being matched with the console through the fused image and the on-site three-dimensional information displayed by the video glasses, and meanwhile, the video glasses capture the head posture of the operator in real time and send the head posture to the server side to wait for the edge side processor to read;

and 7, repeating the step 4 to the step 6.

7. A bucket attitude tracking method based on sparse areas, wherein a monocular and binocular fusion based remote augmented reality follow-up perception system according to any one of claims 1 to 5 is used, characterized in that: the method comprises the following steps:

s1, placing a bucket under natural illumination, avoiding reflective objects around the bucket, taking 30 pictures by using photographing equipment to surround the bucket for a circle, and enabling the bucket to be located at the center of an image during photographing;

s2, opening RealityCapture software, and generating a bucket three-dimensional model by using 30 bucket photos, wherein the three-dimensional model is completely the same as a real bucket in proportion;

s3, placing virtual cameras at 2562 different positions around the three-dimensional model to render the three-dimensional model, acquiring sparse contour points of the bucket in the current posture by using a rendering map, back-projecting the contour points to a coordinate system of the bucket three-dimensional model for storage, and simultaneously storing normal vectors of the contour points and direction vectors of the current posture; finally, 2562 template views are generated;

s4, giving an initial posture of the bucket, multiplying the direction vectors of all the template views by the initial posture, and finding out the template view consistent with the initial posture; projecting the contour point of the template view onto a current real image, wherein the front 18 pixels of the contour point along the normal direction are designated as bucket pixels, the rear 18 pixels are designated as background pixels, and the bucket and the background are segmented to obtain the real contour of the bucket;

and S5, estimating the real posture of the bucket by using the distance between the model contour point and the real contour point along the normal direction, and further realizing the bucket tracking.

Images