CN115514885A - Monocular and binocular fusion-based remote augmented reality follow-up perception system and method - Google Patents
Monocular and binocular fusion-based remote augmented reality follow-up perception system and method Download PDFInfo
- Publication number
- CN115514885A CN115514885A CN202211037134.1A CN202211037134A CN115514885A CN 115514885 A CN115514885 A CN 115514885A CN 202211037134 A CN202211037134 A CN 202211037134A CN 115514885 A CN115514885 A CN 115514885A
- Authority
- CN
- China
- Prior art keywords
- bucket
- follow
- information
- monocular
- augmented reality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 51
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 42
- 230000008447 perception Effects 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 27
- 238000010276 construction Methods 0.000 claims abstract description 24
- 238000005516 engineering process Methods 0.000 claims abstract description 15
- 230000005540 biological transmission Effects 0.000 claims abstract description 9
- 238000004364 calculation method Methods 0.000 claims abstract description 4
- 239000011521 glass Substances 0.000 claims description 23
- 239000013598 vector Substances 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 4
- 238000009877 rendering Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 3
- 238000005286 illumination Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 2
- 238000007500 overflow downdraw method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004886 head movement Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000033001 locomotion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000007115 recruitment Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
Abstract
The invention discloses a remote augmented reality follow-up sensing system and a remote augmented reality follow-up sensing method based on monocular and binocular fusion, which belong to the field of intelligent engineering machinery, wherein the system comprises a monocular and binocular fusion follow-up intelligent sensing module, a server-side intelligent processing module and a user-side augmented reality module, wherein the monocular and binocular fusion follow-up intelligent sensing module is used for acquiring RGB information and depth data of a construction scene and sending the RGB information and the depth data to the edge end of a server through a wireless transmission technology; the server-side intelligent processing module is used for executing calculations required by bucket attitude estimation, bucket tip positioning, accurate environment perception and augmented reality; the user-side augmented reality module comprises three-dimensional display equipment and an operation console, the three-dimensional display equipment is used for displaying the three-dimensional information fusion image processed by the server-side algorithm processing module, and the operation console is used for controlling engineering machinery operation on a construction site. The invention is based on the remote follow-up intelligent sensing technology, and can solve the problems of absence of telepresence and distance sense of an operator and the like.
Description
Technical Field
The invention relates to the field of intelligent engineering machinery, in particular to a monocular and binocular fusion-based remote augmented reality follow-up sensing system and method.
Background
In domestic and international markets, the requirements of engineering machinery users on equipment are continuously diversified, and the requirements of people on operation comfort are higher and higher. Especially in hard, high-risk and repetitive operation environments, because the health of operators cannot be guaranteed, skilled operators are seriously in short supply, and the recruitment of workers in construction enterprises is more and more difficult. In these environments and accident disasters, a large number of engineering machines such as loaders and excavators are often required to perform emergency rescue such as on-site cleaning and road restoration, and people are increasingly eager for obtaining comfortable working environment and equivalent working efficiency by remotely and intelligently remotely controlling the engineering machines. However, the environment sensing of the teleoperation system of the current engineering machinery generally adopts a visible light camera or a laser radar with a fixed position relative to the machine body, and limits the field range of intelligent sensing to a certain extent.
The invention with the application number of 201810268923.3 discloses a stereoscopic vision follow-up system applied to disaster area searching, which obtains disaster area field audio and video in real time by wearing VR glasses and Bluetooth earphones by an operator, and controls the synchronous motion of an unmanned aerial vehicle camera by using the head posture; the invention with the application number of 202010882933.3 discloses a humanoid binocular follow-up virtual reality system suitable for teleoperation of a robot, wherein a binocular camera is placed on a two-dimensional platform, and the binocular camera is controlled by a follow-up mechanism to synchronously pitch and rotate along with the head movement of an operator, so that the visual angle of the binocular camera is changed. Although the technology provides an immersive stereoscopic impression for operators, the technology has the following defects: only binocular images are directly provided for operators, and the problems of absence of presence and distance are still caused.
Disclosure of Invention
The system is based on a remote follow-up intelligent sensing technology, can provide important information such as real-time posture estimation of a bucket and bucket tip positioning of the bucket for an operator, and solves the problems of absence of telepresence and distance sense of the operator.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
a remote augmented reality follow-up sensing system based on monocular and binocular fusion comprises an edge end monocular and binocular fusion follow-up intelligent sensing module, a server end intelligent processing module and a user end augmented reality module;
the edge end single-binocular fusion follow-up intelligent sensing module is used for acquiring RGB information and depth data of a construction scene based on a single-binocular fusion method and sending the RGB information and the depth data to the server end through wireless transmission;
the server-side intelligent processing module is used for executing calculations required by bucket attitude estimation, bucket tip positioning, accurate environment perception and augmented reality;
the user-side augmented reality module comprises a three-dimensional display device and a control console, wherein the three-dimensional display device is used for displaying the three-dimensional information fusion image processed by the server-side algorithm processing module, and the control console is used for controlling the operation of the engineering machinery on a construction site.
The technical scheme of the invention is further improved as follows: the edge end single-binocular fusion follow-up intelligent sensing module comprises a single-binocular vision sensor, an edge end AI processor and a follow-up holder; the edge terminal AI processor is used for realizing the fusion perception of single and double eye RGB information, depth data, camera pose information and key target information, reading the head posture of an operator wearing video glasses at a user terminal, and further controlling a follow-up cradle head carrying a single and double eye vision sensor to quickly synchronize the head posture of the operator by a direct current brushless motor control method, so that a follow-up effect is achieved.
The technical scheme of the invention is further improved as follows: the edge terminal AI processor is connected with the monocular and binocular vision sensor through a USB data line, and reads monocular RGB information, binocular gray scale information and camera pose information in real time; the binocular gray scale information adopts a stereo matching algorithm to recover a depth map, 2D points on the depth map are converted into 3D points under a world coordinate system through internal and external parameters of a binocular depth camera, and then the 3D points under the world coordinate system are projected onto an RGB image through the internal and external parameters of a monocular RGB camera, so that single and binocular information fusion is realized; and detecting the information of the key target in real time based on a target detection algorithm, and sending the position of the key target to a server-side intelligent processing module for positioning the bucket tip.
The technical scheme of the invention is further improved as follows: the intelligent processing module at the server side adopts an efficient sparse region template matching method and a real-time lightweight deep learning network based bucket attitude estimation and positioning algorithm to track the state of the bucket in real time, and the accurate environment perception algorithm adopts a monocular vision SLAM algorithm, so that necessary environment map information and engineering machinery attitude information can be provided for safe operation.
The technical scheme of the invention is further improved as follows: the three-dimensional display equipment can accurately capture the head posture of an operator while displaying the fused image processed by the server end, and sends the head posture to the server end in real time, so that the head posture is read by the edge end AI processor and the follow-up holder is controlled to follow; the control console can also be used for providing a real control environment for an operator, and can achieve an effect of being personally on the scene by matching with the video glasses.
A remote augmented reality follow-up perception method based on monocular and binocular fusion comprises the following steps:
step 1, placing an edge end single-binocular fusion follow-up intelligent sensing module in a cab of engineering machinery, and after ensuring that glass in front of the cab is not shielded, turning on a power supply of an edge end processor to enable the edge end processor to be in a waiting state, and waiting for establishing communication connection with a server end intelligent processing module;
step 5, the server-side intelligent processing module carries out attitude estimation and bucket tip positioning of the bucket and environment map information construction through the received RGB information and depth data, and finally sends a fused image fused with the bucket attitude information, bucket tip position information and actual distance information of surrounding objects such as the bucket tip and a dump truck to a user side for display;
step 6, the user side operator remotely controls the engineering machinery operation in a manner of being matched with the console through the fused image and the on-site three-dimensional information displayed by the video glasses, and meanwhile, the video glasses capture the head posture of the operator in real time and send the head posture to the server side to wait for the edge side processor to read;
and 7, repeating the step 4 to the step 6.
A bucket attitude tracking method based on a sparse region uses a remote augmented reality follow-up perception system based on single and binocular fusion, and comprises the following steps:
s1, placing a bucket under natural illumination, avoiding reflective objects around the bucket, taking 30 pictures by using photographing equipment to surround the bucket for a circle, and enabling the bucket to be located at the center of an image during photographing;
s2, opening RealityCapture software, and generating a bucket three-dimensional model by utilizing 30 bucket photos, wherein the three-dimensional model is completely the same as the real bucket in proportion;
s3, placing virtual cameras at 2562 different positions around the three-dimensional model to render the three-dimensional model, acquiring sparse contour points of the bucket in the current posture by using a rendering map, back-projecting the contour points to a coordinate system of the bucket three-dimensional model for storage, and simultaneously storing normal vectors of the contour points and direction vectors of the current posture; 2562 template views are finally generated;
s4, giving an initial posture of the bucket, multiplying the direction vectors of all the template views by the initial posture, and finding out the template view consistent with the initial posture; projecting the contour point of the template view onto a current real image, wherein the front 18 pixels of the contour point along the normal direction are designated as bucket pixels, the rear 18 pixels are designated as background pixels, and the bucket and the background are segmented to obtain the real contour of the bucket;
and S5, estimating the real posture of the bucket by using the distance between the model contour point and the real contour point along the normal direction, and further realizing the bucket tracking.
Due to the adoption of the technical scheme, the invention has the technical progress that:
1. due to the adoption of the monocular and binocular follow-up intelligent sensing technology, the invention realizes that the operator gets rid of the limitation of fixed visual angle of the traditional teleoperation under the condition of over-sight distance, and monocular RGB information and binocular depth data of different visual angles of a construction site can be freely acquired through the change of the head posture.
2. Due to the adoption of the efficient and real-time bucket tracking and bucket tip positioning algorithm, the posture of the bucket and the spatial position of the bucket tip can be captured in real time through images, so that the real-time posture information and the position information of the bucket are provided for operators by matching with an augmented reality technology, and the limitation that the distance sense cannot be provided for the operators by the traditional teleoperation is overcome.
3. Due to the adoption of the SLAM + YOLO environment perception algorithm, the local coordinates of the bucket tip of the bucket can be transformed to the starting point coordinate system of the camera in real time, so that the global environment perception is provided, and a foundation is provided for the efficient operation of the engineering machinery.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts;
fig. 1 is a schematic diagram of an overall structure of a remote augmented reality follow-up sensing system based on monocular and binocular fusion in an embodiment of the present invention;
fig. 2 is a schematic diagram of a general structure of a remote augmented reality follow-up sensing system based on monocular and binocular fusion in the embodiment of the present invention;
fig. 3 is a structure diagram of a servo pan-tilt of a remote augmented reality servo perception system based on monocular and binocular fusion in the embodiment of the present invention;
FIG. 4 is a schematic diagram of monocular and binocular fusion of the remote augmented reality follow-up sensing system based on monocular and binocular fusion in the embodiment of the present invention;
FIG. 5 is a schematic diagram of bucket tracking of a remote augmented reality follow-up sensing system based on monocular and binocular fusion according to an embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating positioning of a bucket tip of a bucket of a remote augmented reality follow-up sensing system based on monocular and binocular fusion in the embodiment of the present invention;
the device comprises an edge terminal AI processor 1, a monocular and binocular vision sensor 2, a 3, y-axis brushless DC motor, a 4, x-axis brushless DC motor, a 5, RGB camera, a 6, left eye depth camera and a 7, right eye depth camera.
Detailed Description
It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The invention is described in further detail below with reference to the following figures and examples:
as shown in fig. 1 and 2, a monocular and binocular fusion based remote augmented reality follow-up sensing system includes an edge end monocular and binocular fusion follow-up intelligent sensing module, a server end intelligent processing module and a user end augmented reality module;
the edge end single-binocular fusion follow-up intelligent sensing module is placed in a cab of the engineering machinery, is used for acquiring RGB information and depth data of a construction scene and sending the RGB information and the depth data to the server end through a wireless transmission technology;
the server-side intelligent processing module is used for executing calculation required by bucket attitude estimation, bucket tip positioning, accurate environment perception and augmented reality;
the user side augmented reality module comprises video glasses and a console, the video glasses are used for displaying the three-dimensional information fusion image processed by the server side algorithm processing module, and the console is used for controlling engineering machinery operation of a construction site.
As shown in fig. 3, the servo-actuated tripod head integrates a monocular and binocular vision sensor 2, an edge AI processor 1, an x-axis dc brushless motor 4 and a y-axis dc brushless motor 3, adopts the self-design of solidworks software, has two degrees of freedom of the x-axis and the y-axis, and enables the dc motor to have small torque fluctuation, high efficiency, small noise and fast dynamic response through a motor control technology (FOC motor control technology), thereby being capable of rapidly synchronizing the head posture of an operator.
As shown in fig. 4, the monocular and binocular vision sensor 2 has three cameras, which are an RGB camera 5, a left eye depth camera 6 and a right eye depth camera 7, the RGB camera 5 can collect RGB information including a bucket, and sends the RGB information to the server-side intelligent processing module for bucket tracking through wireless transmission, the left eye depth camera 6 and the right eye depth camera 7 can collect gray scale information including the bucket, the resolution of the RGB image is kept consistent through upsampling, depth recovery is performed based on the right eye image by using an SGBM binocular stereo matching algorithm, and finally, a depth image pixel point is re-projected onto the RGB image through a coordinate transformation relation between the right eye depth camera 7 and the RGB camera 5 for monocular and binocular information fusion, so that bucket attitude information can be transformed into an RGB coordinate system, and the bucket point can be conveniently located subsequently.
As shown in fig. 5, a bucket attitude estimation algorithm deployed by the server-side intelligent processing module can generate template views of each attitude by using a three-dimensional model of the bucket, and then match the template views with a real view of the bucket returned by the servo pan-tilt to obtain accurate estimation of the bucket attitude, and then superimpose and display a rendering image of the current attitude and the real image to visualize a tracking result.
As shown in fig. 6 (a), the bucket tip positioning algorithm deployed by the server-side intelligent processing module can obtain two-dimensional image plane coordinates of the bucket tip by using a two-dimensional image rendered by the bucket three-dimensional model, and then obtain three-dimensional coordinates of the bucket tip in the three-dimensional model coordinate system through internal and external parameters of the virtual camera. As shown in fig. 6 (b), the bucket tip coordinates in the three-dimensional model coordinate system may be transformed to the camera coordinate system through the camera pose, the position of the dump truck is detected through the YOLOv4 target detection algorithm deployed by the edge AI processor 1, the three-dimensional coordinates of the truck center point in the camera coordinate system are obtained by matching with the depth data, the euclidean distance between the bucket tip and the truck center point is further calculated, the information of the relative distance between the bucket tip and the truck is provided for the operator, and when the relative distance is smaller than the set threshold, an alarm may be issued to the operator to avoid collision. As shown in (c) in fig. 6, by using the SLAM + YOLO algorithm deployed on the server side, the points outside the box represent the detected static feature points, the points inside the box represent the bucket and the dynamic feature points around the bucket, and the transformation relationship of the current frame with respect to the first frame is calculated by using the static feature points of each frame, so that the tip coordinates at each time are converted into the coordinates in the camera coordinate system at the initial time, that is, the coordinates in the global coordinate system. The global bucket coordinate can better estimate the construction state of the current engineering machine.
A remote augmented reality follow-up perception method based on monocular and binocular fusion comprises the following steps:
step 1, placing an edge end single-binocular fusion follow-up intelligent sensing module in a cab of engineering machinery, and after ensuring that glass in front of the cab is not shielded, turning on a power supply of an edge end processor to enable the edge end processor to be in a waiting state, and waiting for establishing communication connection with a server end intelligent processing module;
step 5, the server-side intelligent processing module carries out attitude estimation and bucket tip positioning of the bucket and environment map information construction through the received RGB information and depth data, and finally sends a fused image fused with the bucket attitude information, bucket tip position information and actual distance information of surrounding objects such as the bucket tip and a dump truck to a user side for display;
step 6, the user side operator remotely controls the engineering machinery operation in a manner of being matched with the console through the fused image and the on-site three-dimensional information displayed by the video glasses, and meanwhile, the video glasses capture the head posture of the operator in real time and send the head posture to the server side to wait for the edge side processor to read;
and 7, repeating the step 4 to the step 6.
Specifically, when the engineering machinery carries out remote control operation, an operator is positioned on the control console and wears the video glasses. The image displayed by the video glasses is a fused image processed and completed by the server-side intelligent processing module, the fused image comprises attitude information of the bucket, distance information between the bucket tip and the servo pan-tilt, distance information between the bucket tip and the dump truck and global environment information of the bucket, and an operator can judge the construction state of a construction site by using the information and control the operation of engineering machinery of the construction site through the console. The follow-up cradle head placed in the engineering machinery cab of the construction site can freely rotate according to the head posture of an operator, on-site RGB images and depth images are collected in real time, then the images are sent to the server-side intelligent processing module through a wireless transmission technology to be subjected to posture estimation, bucket tip positioning, environment perception and image fusion based on the augmented reality technology, and finally the server side sends the fused images to video glasses to be displayed.
The beyond-the-horizon remote augmented reality follow-up intelligent sensing system and method based on monocular and binocular fusion can be used in the existing practical intelligent engineering machinery teleoperation system, an intelligent follow-up cradle head can replace an unmanned aerial vehicle to sense the environment, a bucket tracking method based on vision can replace a tracking method based on an IMU sensor, video glasses can replace a large-screen two-dimensional display interface, and the telepresence sense of being personally on the scene is provided for teleoperators.
The invention also provides a bucket attitude tracking method based on the sparse region, which uses a remote augmented reality follow-up perception system based on monocular and binocular fusion, and the tracking method realizes functions through the following steps:
s1, placing the bucket under natural illumination, and avoiding reflective objects around the bucket as much as possible. 30 pictures are taken with any photographing device around the bucket, with the bucket as centered as possible in the image.
And S2, opening RealityCapture software, and generating a bucket three-dimensional model by using 30 bucket photos, wherein the three-dimensional model is completely the same as a real bucket in proportion.
And S3, placing 2562 virtual cameras at different positions around the three-dimensional model to render the three-dimensional model, acquiring sparse contour points of the bucket in the current posture by using a rendering graph, back-projecting the contour points to a coordinate system of the bucket three-dimensional model for storage, and simultaneously storing normal vectors of the contour points and direction vectors of the current posture. Finally 2562 template views can be generated.
And S4, giving the initial posture of the bucket, multiplying the direction vectors of all the template views by the initial posture, and finding out the template view closest to the initial posture. And projecting the contour point of the template view onto the current real image, wherein the front 18 pixels of the contour point along the normal direction are designated as bucket pixels, and the rear 18 pixels are designated as background pixels, so that the bucket and the background can be segmented, and the real contour of the bucket is obtained.
And S5, estimating the real posture of the bucket by using the distance between the model contour point and the real contour point along the normal direction, and further realizing the bucket tracking.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and these modifications or substitutions do not depart from the spirit of the corresponding technical solutions of the embodiments of the present invention.
Claims (7)
1. The utility model provides a long-range augmented reality follow-up perception system based on monocular and binocular fusion which characterized in that: the system comprises an edge end single-binocular fusion follow-up intelligent sensing module, a server end intelligent processing module and a user end augmented reality module;
the edge end single-binocular fusion follow-up intelligent sensing module is used for acquiring RGB information and depth data of a construction scene based on a single-binocular fusion method and sending the RGB information and the depth data to the server end through wireless transmission;
the server-side intelligent processing module is used for executing calculation required by bucket attitude estimation, bucket tip positioning, accurate environment perception and augmented reality;
the user-side augmented reality module comprises a three-dimensional display device and a control console, wherein the three-dimensional display device is used for displaying the three-dimensional information fusion image processed by the server-side algorithm processing module, and the control console is used for controlling the operation of the engineering machinery on a construction site.
2. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 1, wherein: the edge end single-binocular fusion follow-up intelligent sensing module comprises a single-binocular vision sensor, an edge end AI processor and a follow-up holder; the edge terminal AI processor is used for realizing the fusion perception of monocular and binocular RGB information, depth data, camera pose information and key target information, reading the head posture of an operator wearing the video glasses at a user terminal, and controlling a follow-up pan-tilt carrying monocular and binocular vision sensors to quickly synchronize the head posture of the operator by a direct current brushless motor control method, so that a follow-up effect is achieved.
3. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 2, wherein: the edge terminal AI processor is connected with the monocular and binocular vision sensor through a USB data line, and reads monocular RGB information, binocular gray scale information and camera pose information in real time; the binocular gray scale information adopts a stereo matching algorithm to recover a depth map, 2D points on the depth map are converted into 3D points under a world coordinate system through internal and external parameters of a binocular depth camera, and then the 3D points under the world coordinate system are projected onto an RGB image through the internal and external parameters of a monocular RGB camera, so that single and binocular information fusion is realized; and detecting the information of the key target in real time based on a target detection algorithm, and sending the position of the key target to a server-side intelligent processing module for positioning the bucket tip.
4. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 1, wherein: the intelligent processing module at the server side adopts an efficient sparse region template matching method and a real-time lightweight deep learning network based bucket attitude estimation and positioning algorithm to track the state of the bucket in real time, and the accurate environment perception algorithm adopts a monocular vision SLAM algorithm, so that necessary environment map information and engineering machinery attitude information can be provided for safe operation.
5. The monocular and binocular fusion based remote augmented reality follow-up perception system according to claim 1, wherein: the three-dimensional display equipment can accurately capture the head posture of an operator while displaying the fused image processed by the server, and sends the head posture to the server in real time, so that the head posture is read by the edge terminal AI processor to control the follow-up cradle head to follow; the control console can also be used for providing a real control environment for an operator, and can achieve the effect of being personally on the scene by matching with video glasses.
6. A perception method of the monocular and binocular fusion based remote augmented reality follow-up perception system according to any one of claims 1-5, wherein: the method comprises the following steps:
step 1, placing an edge end single-binocular fusion follow-up intelligent sensing module in a cab of engineering machinery, and after ensuring that glass in front of the cab is not shielded, turning on a power supply of an edge end processor to enable the edge end processor to be in a waiting state, and waiting for establishing communication connection with a server end intelligent processing module;
step 2, turning on a power supply of the server side to enable the server side to be in a monitoring state, and waiting for establishing communication connection with the edge end single-binocular fusion follow-up intelligent sensing module and the user side augmented reality module;
step 3, an operator enters the console, wears the video glasses and starts remote control operation after the three-dimensional display equipment has a construction interface;
step 4, reading the head posture data of an operator by the edge end processor, controlling the servo cradle head to update the posture in real time, and simultaneously sending RGB information and depth data of a construction scene to the server end through a wireless transmission technology;
step 5, the server-side intelligent processing module carries out attitude estimation and bucket tip positioning of the bucket and environment map information construction through the received RGB information and depth data, and finally sends a fused image fused with the bucket attitude information, bucket tip position information and actual distance information between the bucket tip and objects around the dump truck to a user side for display;
step 6, the user side operator remotely controls the engineering machinery operation in a manner of being matched with the console through the fused image and the on-site three-dimensional information displayed by the video glasses, and meanwhile, the video glasses capture the head posture of the operator in real time and send the head posture to the server side to wait for the edge side processor to read;
and 7, repeating the step 4 to the step 6.
7. A bucket attitude tracking method based on sparse areas, wherein a monocular and binocular fusion based remote augmented reality follow-up perception system according to any one of claims 1 to 5 is used, characterized in that: the method comprises the following steps:
s1, placing a bucket under natural illumination, avoiding reflective objects around the bucket, taking 30 pictures by using photographing equipment to surround the bucket for a circle, and enabling the bucket to be located at the center of an image during photographing;
s2, opening RealityCapture software, and generating a bucket three-dimensional model by using 30 bucket photos, wherein the three-dimensional model is completely the same as a real bucket in proportion;
s3, placing virtual cameras at 2562 different positions around the three-dimensional model to render the three-dimensional model, acquiring sparse contour points of the bucket in the current posture by using a rendering map, back-projecting the contour points to a coordinate system of the bucket three-dimensional model for storage, and simultaneously storing normal vectors of the contour points and direction vectors of the current posture; finally, 2562 template views are generated;
s4, giving an initial posture of the bucket, multiplying the direction vectors of all the template views by the initial posture, and finding out the template view consistent with the initial posture; projecting the contour point of the template view onto a current real image, wherein the front 18 pixels of the contour point along the normal direction are designated as bucket pixels, the rear 18 pixels are designated as background pixels, and the bucket and the background are segmented to obtain the real contour of the bucket;
and S5, estimating the real posture of the bucket by using the distance between the model contour point and the real contour point along the normal direction, and further realizing the bucket tracking.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211037134.1A CN115514885B (en) | 2022-08-26 | 2022-08-26 | Remote augmented reality follow-up sensing system and method based on monocular and binocular fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211037134.1A CN115514885B (en) | 2022-08-26 | 2022-08-26 | Remote augmented reality follow-up sensing system and method based on monocular and binocular fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115514885A true CN115514885A (en) | 2022-12-23 |
| CN115514885B CN115514885B (en) | 2024-03-01 |
Family
ID=84501858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211037134.1A Active CN115514885B (en) | 2022-08-26 | 2022-08-26 | Remote augmented reality follow-up sensing system and method based on monocular and binocular fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115514885B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117197769A (en) * | 2023-11-03 | 2023-12-08 | 江苏智能无人装备产业创新中心有限公司 | Loader front image generation system and method based on bucket position observation |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015199470A1 (en) * | 2014-06-25 | 2015-12-30 | 한국과학기술원 | Apparatus and method for estimating hand position utilizing head mounted color depth camera, and bare hand interaction system using same |
| CN107034942A (en) * | 2017-05-24 | 2017-08-11 | 成都跟驰科技有限公司 | Virtual reality system for excavator remote control |
| CN107882103A (en) * | 2017-10-26 | 2018-04-06 | 南京工业大学 | A kind of excavator 3 d pose is shown and Remote Automatic Control System |
| CN107888895A (en) * | 2017-10-17 | 2018-04-06 | 三重机有限公司 | Excavator tele-control system, method and excavator |
| US20180197345A1 (en) * | 2016-09-13 | 2018-07-12 | Youngzone Culture (Shanghai) Co., Ltd. | Augmented reality technology-based handheld viewing device and method thereof |
| CN108797669A (en) * | 2018-06-20 | 2018-11-13 | 清华大学 | A kind of autonomous 3D excavations construction robot |
| CN109828658A (en) * | 2018-12-17 | 2019-05-31 | 彭晓东 | A kind of man-machine co-melting long-range situation intelligent perception system |
| US20200005520A1 (en) * | 2018-06-27 | 2020-01-02 | Shanghai United Imaging Healthcare Co., Ltd. | Method and system for fusing image data |
| CN112116631A (en) * | 2020-09-07 | 2020-12-22 | 江苏瑞科科技有限公司 | Industrial augmented reality combined positioning system |
| CN112554253A (en) * | 2020-11-27 | 2021-03-26 | 徐工集团工程机械有限公司 | Multifunctional emergency rescue vehicle and control method thereof |
| WO2021098441A1 (en) * | 2019-11-20 | 2021-05-27 | Oppo广东移动通信有限公司 | Hand posture estimation method and apparatus, device and computer storage medium |
| US20210311320A1 (en) * | 2020-04-06 | 2021-10-07 | Pike Enterprises, Llc | Virtual reality tracking system |
| CN113723279A (en) * | 2021-08-30 | 2021-11-30 | 东南大学 | Multi-target tracking acceleration method based on time-space optimization in edge computing environment |
-
2022
- 2022-08-26 CN CN202211037134.1A patent/CN115514885B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015199470A1 (en) * | 2014-06-25 | 2015-12-30 | 한국과학기술원 | Apparatus and method for estimating hand position utilizing head mounted color depth camera, and bare hand interaction system using same |
| US20180197345A1 (en) * | 2016-09-13 | 2018-07-12 | Youngzone Culture (Shanghai) Co., Ltd. | Augmented reality technology-based handheld viewing device and method thereof |
| CN107034942A (en) * | 2017-05-24 | 2017-08-11 | 成都跟驰科技有限公司 | Virtual reality system for excavator remote control |
| CN107888895A (en) * | 2017-10-17 | 2018-04-06 | 三重机有限公司 | Excavator tele-control system, method and excavator |
| CN107882103A (en) * | 2017-10-26 | 2018-04-06 | 南京工业大学 | A kind of excavator 3 d pose is shown and Remote Automatic Control System |
| CN108797669A (en) * | 2018-06-20 | 2018-11-13 | 清华大学 | A kind of autonomous 3D excavations construction robot |
| US20200005520A1 (en) * | 2018-06-27 | 2020-01-02 | Shanghai United Imaging Healthcare Co., Ltd. | Method and system for fusing image data |
| CN109828658A (en) * | 2018-12-17 | 2019-05-31 | 彭晓东 | A kind of man-machine co-melting long-range situation intelligent perception system |
| WO2021098441A1 (en) * | 2019-11-20 | 2021-05-27 | Oppo广东移动通信有限公司 | Hand posture estimation method and apparatus, device and computer storage medium |
| US20210311320A1 (en) * | 2020-04-06 | 2021-10-07 | Pike Enterprises, Llc | Virtual reality tracking system |
| CN112116631A (en) * | 2020-09-07 | 2020-12-22 | 江苏瑞科科技有限公司 | Industrial augmented reality combined positioning system |
| CN112554253A (en) * | 2020-11-27 | 2021-03-26 | 徐工集团工程机械有限公司 | Multifunctional emergency rescue vehicle and control method thereof |
| CN113723279A (en) * | 2021-08-30 | 2021-11-30 | 东南大学 | Multi-target tracking acceleration method based on time-space optimization in edge computing environment |
Non-Patent Citations (1)
| Title |
|---|
| 陈熙霖等: "面向真实世界的智能感知与交互", 中国科学：信息科学, 31 August 2016 (2016-08-31) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117197769A (en) * | 2023-11-03 | 2023-12-08 | 江苏智能无人装备产业创新中心有限公司 | Loader front image generation system and method based on bucket position observation |
| CN117197769B (en) * | 2023-11-03 | 2024-01-26 | 江苏智能无人装备产业创新中心有限公司 | Loader front image generation system and method based on bucket position observation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115514885B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111062873B (en) | Parallax image splicing and visualization method based on multiple pairs of binocular cameras | |
| CN108780586B (en) | Image processing method, display device and inspection system | |
| CN110458897B (en) | Multi-camera automatic calibration method and system and monitoring method and system | |
| TWI617277B (en) | Pose estimation apparatus and vacuum cleaner system | |
| US10008028B2 (en) | 3D scanning apparatus including scanning sensor detachable from screen | |
| CN113570721B (en) | Method and device for reconstructing three-dimensional space model and storage medium | |
| US7825948B2 (en) | 3D video conferencing | |
| CN206162398U (en) | A stereovision follow -up display system for heavy machinery is long -range unattended | |
| CN112634318B (en) | Teleoperation system and method for underwater maintenance robot | |
| CN105666505A (en) | Robot system having augmented reality-compatible display | |
| CN108093244B (en) | Remote follow-up stereoscopic vision system | |
| JP2010109783A (en) | Electronic camera | |
| CN111383348A (en) | Method for remotely and synchronously controlling robot through virtual reality | |
| CN105373130A (en) | Special device accident on-site information detection system based on stereo modeling | |
| CN115514885B (en) | Remote augmented reality follow-up sensing system and method based on monocular and binocular fusion | |
| JP2015118442A (en) | Information processor, information processing method, and program | |
| CN111947650A (en) | Fusion positioning system and method based on optical tracking and inertial tracking | |
| JP2016541042A (en) | Method and system for providing position or motion information for controlling at least one function of a vehicle | |
| WO2017155005A1 (en) | Image processing method, display device, and inspection system | |
| CN106989730A (en) | A kind of system and method that diving under water device control is carried out based on binocular flake panoramic vision | |
| Stricker et al. | From interactive to adaptive augmented reality | |
| McMurrough et al. | 3D point of gaze estimation using head-mounted RGB-D cameras | |
| CN115890693A (en) | Mobile explosion-removing device and method based on immersive perception and bionic control | |
| EP3336801A1 (en) | Method and apparatus for constructing lighting environment representations of 3d scenes | |
| JP2001239983A (en) | Outfit assisting system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |