CN117162109A - Building spraying robot and spraying method thereof - Google Patents
Building spraying robot and spraying method thereof Download PDFInfo
- Publication number
- CN117162109A CN117162109A CN202311006244.6A CN202311006244A CN117162109A CN 117162109 A CN117162109 A CN 117162109A CN 202311006244 A CN202311006244 A CN 202311006244A CN 117162109 A CN117162109 A CN 117162109A
- Authority
- CN
- China
- Prior art keywords
- spraying
- area
- door
- depth camera
- path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005507 spraying Methods 0.000 title claims abstract description 229
- 239000007921 spray Substances 0.000 claims abstract description 95
- 238000000034 method Methods 0.000 claims description 17
- 238000013136 deep learning model Methods 0.000 claims description 14
- 230000009466 transformation Effects 0.000 claims description 13
- 238000010276 construction Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000010422 painting Methods 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 240000007651 Rubus glaucus Species 0.000 description 1
- 235000011034 Rubus glaucus Nutrition 0.000 description 1
- 235000009122 Rubus idaeus Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000003973 paint Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Abstract
The invention provides a building spraying robot and a spraying method thereof, wherein the spraying method comprises the steps of acquiring an initial operation area from a video image of a depth camera, identifying a door and window hole of the initial operation area, acquiring a target operation area which does not comprise the door and window hole, dividing the target operation area according to the door and window hole, planning a path according to a dividing result to obtain an initial operation path, acquiring a relative position relation between the depth camera and a spray gun, determining a target operation path of the building spraying robot according to the relative position relation and the initial operation path, and controlling the robot to spray according to the target operation path. According to the invention, the door and window holes can be identified by only selecting the initial operation area to be sprayed on the display device by an operator, the target operation area to be operated and the planned target operation path are determined according to the door and window holes, and the robot can automatically move to spray according to the target operation path, so that the operation is convenient and the accuracy is high.
Description
Technical Field
The invention belongs to the technical field of building spraying, and particularly relates to a building spraying robot and a spraying method thereof.
Background
The building spraying robot for building is used for automatically spraying the surface of a structure. Because the surface of a building generally has areas such as door and window holes and the like which do not need to be sprayed, the areas of the door and window holes need to be identified and avoided in advance during spraying operation, the whole spraying path is planned in advance, the spraying time is shortened, the spraying efficiency is improved, and the leakage spraying and the spraying to the door and window holes are avoided. At present, the planning combination is determined by means of manpower or related components, and the problems of low operation efficiency, low operation accuracy and the like exist.
Disclosure of Invention
The invention aims to solve the technical problems of low operation efficiency and low operation accuracy in spraying planning on a plane with a door and window hole.
The embodiment of the invention is realized in such a way that the spraying method is applied to a building spraying robot, the building spraying robot comprises a movable chassis, a mechanical arm connected with the movable chassis and a spraying module positioned at the tail end of the mechanical arm, the spraying module comprises a spray gun and a depth camera, and the spraying method comprises the following steps:
acquiring an initial operation area from a video image of the depth camera;
carrying out door and window hole identification on the initial operation area to obtain a target operation area, wherein the target operation area does not comprise the door and window hole;
dividing the target operation area according to the door and window holes, and planning a path according to the dividing result to obtain an initial operation path;
acquiring the relative position relation between the depth camera and the spray gun, and determining a target operation path of the building spraying robot according to the relative position relation and the initial operation path;
and controlling the building spraying robot to spray according to the target operation path.
Preferably, the acquiring the initial working area from the video image of the depth camera includes:
displaying the video image shot by the depth camera;
and acquiring a selected operation on the video image, determining a selected area according to the selected operation, and taking the selected area as an initial operation area.
Preferably, the initial working area is a rectangular plane, the acquiring a selected operation on the video image, and determining the selected area according to the selected operation includes:
acquiring a selected operation on the video image, the selected operation comprising three anchor points selected on the video image;
calculating the space coordinates of the three positioning points in a camera coordinate system of the depth camera according to a pixel coordinate-space coordinate transformation function of the depth camera;
according to the space coordinates of the three positioning points, determining the space coordinates of four corner points of a rectangle enclosed by the three positioning points;
and determining the selected area according to the space coordinates of the four corner points.
Preferably, said determining the selected region according to the spatial coordinates of the four corner points comprises:
acquiring a plurality of adjacent image frames in the video image;
acquiring corner coordinates of the four corners in a plurality of image frames respectively;
taking the median of a plurality of image frames as a reference value, deleting the data of the image frames when the deviation between the rest of the image frames and the reference value is larger than a first preset distance, and taking the average value of the data of the rest of the image frames as the space coordinates of four corner points of the selected area;
and determining the selected area according to the space coordinates of the four corner points.
Preferably, the spraying method further comprises:
according to the space coordinates of the preset corner points of the initial operation area, calculating a plane equation of a rectangular plane where the initial operation area is located and a transformation matrix from a camera coordinate system of the depth camera to the rectangular plane;
according to the plane equation and the transformation matrix, three Euler angles required to rotate the depth camera to be parallel to the rectangular plane are calculated;
taking the three Euler angles as rotation angles required by the spray gun to be perpendicular to the initial operation area;
and controlling the mechanical arm to rotate so as to enable the spray gun to rotate to a position corresponding to the rotation angle.
Preferably, the performing door and window hole recognition on the initial operation area, and obtaining the target operation area includes:
acquiring a preset deep learning model;
and carrying out door and window hole identification on the initial operation area by using the deep learning model to obtain a target operation area which does not contain the door and window hole.
Preferably, after the door and window hole recognition is performed on the initial working area by using the deep learning model, the method further includes:
acquiring space coordinates of four corner points of the identified door and window hole;
and projecting the space coordinates of the four corner points of the door and window hole to the initial operation area, and correcting according to a projection result to obtain the corrected space coordinates of the four corner points of the door and window hole.
Preferably, the acquiring the relative positional relationship between the depth camera and the spray gun, and determining the target working path of the building spraying robot according to the relative positional relationship and the initial working path includes:
obtaining the maximum spraying rectangular area of the spray gun;
determining overall spraying path coordinates of the target working area according to the maximum spraying rectangular area and the initial working path;
obtaining the vertical distance between the spray gun and the spraying plane where the target operation area is located;
translating the overall spraying path coordinates along the normal direction of the spraying plane by the vertical distance to obtain a first space coordinate sequence of the spray gun in a camera coordinate system of the depth camera;
acquiring a relative position relation between the depth camera and the spray gun, and performing coordinate conversion on the first space coordinate sequence according to the relative position relation to obtain a second space coordinate sequence under a mechanical arm coordinate system of a mechanical arm where the spray gun is positioned;
according to the functional relation between the Cartesian space coordinates of the spray gun and the mechanical arm joint driving space coordinates, calculating the mechanical arm joint driving space coordinates corresponding to each point in the second space coordinate sequence;
and determining a target operation path of the building spraying robot according to the mechanical arm joint driving space coordinates.
The embodiment of the invention also provides a building spraying robot, which comprises a movable chassis, a mechanical arm connected with the movable chassis, a spraying module and a control module, wherein the spraying module is positioned at the tail end of the mechanical arm, the spraying module comprises a spray gun and a depth camera, and the control module is used for executing the spraying method according to any one of claims 1 to 8.
Preferably, the spraying module is provided with a biaxial holder, the depth camera is fixed on the biaxial holder, and when the biaxial holder is positioned at the initial position, the optical axis of the depth camera is parallel to the spraying direction of the spray gun.
Compared with the prior art, the invention has the beneficial effects that: according to the embodiment of the invention, an initial operation area is obtained from a video image of a depth camera, door and window holes are identified for the initial operation area, a target operation area which does not comprise the door and window holes is obtained, the target operation area is divided according to the door and window holes, a path planning is carried out according to a division result to obtain an initial operation path, the relative position relation between the depth camera and the spray gun is obtained, a target operation path of a building spraying robot is determined according to the relative position relation and the initial operation path, and the building spraying robot is controlled to spray according to the target operation path. According to the embodiment of the invention, the video image shot by the depth camera is displayed on the related display device in real time, the door and window holes can be identified on the initial operation area only by an operator selecting the initial operation area to be sprayed on the display device, the target operation area to be operated and the target operation path of the planning building spraying robot are determined according to the door and window holes, and the building spraying robot can automatically move to perform full-automatic spraying according to the target operation path, so that the operation is convenient and the accuracy is high.
Drawings
FIG. 1 is a flow chart of a spray coating method according to an embodiment of the present invention;
FIG. 2 is a graph of the recognition result of a door/window hole according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a target job path according to another embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The building spraying robot of the embodiment is composed of a movable chassis, a folding or telescopic mechanical arm and a spraying module. The mechanical arm can turn, fold and stretch, and the spraying module arranged at the tail end of the mechanical arm can be moved to a designated position in a working range. And the spraying module of the building spraying robot is used for automatically spraying the surface of a structure. The spray module is typically comprised of a rail and a plurality of spray gun systems side by side. The spray gun system can freely move along the direction of the guide rail, and the spray width can be controlled in real time by the number of spray guns which are simultaneously opened.
In general, the area of a region to be sprayed on a construction site is large, and the region needs to be further divided into a plurality of regions for spraying. The first-stage subarea is formed by a range which can be sprayed on different points of the mobile chassis. The second-stage subarea is formed by a range which can be sprayed by the mechanical arm at different points when the movable chassis is fixed. The range in which the mechanical arm can spray at a certain point is the range in which the spray gun in the spraying module at the tail end of the mechanical arm can move for spraying. Because the spraying plane generally has areas such as door and window holes and the like which do not need to be sprayed, the door and window hole areas need to be recognized and avoided in advance during spraying operation, the whole spraying path is planned in advance, the spraying time is shortened, the spraying efficiency is improved, and the leakage spraying and the spraying to the door and window holes are avoided. And then the space coordinates of the planned spraying paths are provided for the mechanical arm, and the spraying module at the tail end of the mechanical arm is moved to a designated point for spraying. The space coordinates of the spraying module are required to enable the vertical distance between the spraying module spray gun and the spraying surface to reach a given value, so that the position error of the spraying module can directly influence the spraying quality. In the construction spraying scene, the position accuracy requirement of the spraying module is generally about 5 cm.
In the conventional technology, the spraying planning of a building spraying robot for a building on a plane with a door and window hole has the following spraying schemes: (1) The spraying operation is carried out by means of manual remote control by avoiding the door and window holes, and the defects of insufficient precision, difficulty in controlling the distance between the spray gun and the spraying surface, low operation efficiency and the like exist. (2) Positioning is carried out by means of a GPS, IMU, RTK positioning system, and the defects of insufficient precision and difficult signal guarantee exist; (3) By means of laser radar scanning, the defects of low accuracy of identifying door and window holes by point clouds, unobvious point cloud characteristic points and high cost exist.
Based on the background, the embodiment of the invention provides a spraying method for intelligently identifying a door and window hole on a plane with the door and window hole, positioning and navigating a spraying module at the tail end of a mechanical arm and planning a spraying path by utilizing an RGB-D depth camera. The spraying method is economical, convenient and fast, has low requirements on operators, has low precision, is suitable for building spraying (within 5 cm), and is suitable for the working scene of typical building spraying robots such as bridges, tunnels, workshops and the like.
The spraying method can display construction scene videos shot by the RGB-D depth camera on a display device (including a tablet personal computer, a notebook personal computer, a mobile phone, a desktop personal computer and the like) in real time, and can automatically move to perform full-automatic spraying only by an operator selecting a range needing to be sprayed on an upper frame of the display device, and a spraying module at the tail end of the mechanical arm and a spray gun on a spraying guide rail, and meanwhile intelligently identify and avoid door and window holes needing no spraying.
Fig. 1 shows a spraying method according to an embodiment of the present invention, where the spraying method is applied to a building spraying robot, the building spraying robot includes a movable chassis, a mechanical arm connected to the movable chassis, and a spraying module located at an end of the mechanical arm, the spraying module includes a spray gun and a depth camera, and the spraying method includes:
s101, an initial work area is acquired from a video image of a depth camera.
In this step, the depth camera photographs in real time the area to be subjected to the painting operation, and transmits the video image photographed in real time to the display device. The depth camera is an RGB-D depth camera, and the display device can be a terminal with a display function, such as a tablet computer, a desktop computer, a notebook computer, a mobile phone and the like. The display device displays the video image shot by the depth camera. After the operator performs a selection operation on the video image returned by the depth camera, the operator obtains a selection operation on the video image, determines a selected area according to the selection operation, and takes the selected area as an initial operation area.
In this step, the initial working area selected by the operator is a rectangular plane, and the acquiring the selected operation on the video image, and determining the selected area according to the selected operation includes: a selected operation on the video image is acquired, the selected operation including three anchor points selected on the video image. And calculating the spatial coordinates of the three positioning points in the camera coordinate system of the depth camera according to the pixel coordinate-spatial coordinate transformation function of the depth camera. And determining the space coordinates of four corner points of the rectangle enclosed by the three positioning points according to the space coordinates of the three positioning points. And determining the selected area according to the space coordinates of the four corner points. In this embodiment, when the operator selects the area to be subjected to the spraying operation, three points may be selected only on the area to be subjected to the spraying operation, and then a primary selected operation area of a rectangular plane is determined according to the three points selected by the operator. Through the step, an operator can conveniently select an initial operation area.
In other embodiments, in this step, in order to determine the initial working area more accurately, the determining the selected area according to the spatial coordinates of the four corner points specifically includes: a number of adjacent image frames in the video image are acquired. Acquiring corner coordinates of the four corners in a plurality of image frames respectively; and taking the median of the image frames as a reference value, deleting the data of the image frames when the deviation between the rest of the image frames and the reference value is larger than a first preset distance, and taking the average value of the data of the rest of the image frames as the space coordinates of the four corner points of the selected area. And determining the selected area according to the space coordinates of the four corner points. In this embodiment, by acquiring several continuous images and processing and calculating the several continuous images, the initial operation area can be acquired more accurately.
S102, door and window holes are identified in the initial operation area, and a target operation area is obtained, wherein the target operation area does not comprise the door and window holes.
In this step, door and window tunnel recognition is performed on the initial operation area, and obtaining the target operation area includes: acquiring a preset deep learning model; and carrying out door and window hole identification on the initial operation area by using the deep learning model to obtain a target operation area which does not contain the door and window hole. Further, in order to obtain the space coordinates of the door and window hole more accurately, after the door and window hole is identified in the initial operation area by using the deep learning model, the method further comprises: acquiring space coordinates of four corner points of the identified door and window hole; and projecting the space coordinates of the four corner points of the door and window hole to the initial operation area, and correcting according to the projection result to obtain the corrected space coordinates of the four corner points of the door and window hole.
S103, dividing the target operation area according to the door window hole, and planning a path according to the division result to obtain an initial operation path.
In this step, after the target work area is determined, the target work area may be divided using a conventional dividing method. In this embodiment, in order to facilitate the operation of the construction painting robot, the target operation area is divided according to four sides of the door and window hole, and the path planning is performed according to a plurality of areas obtained by the division, so as to obtain an initial operation path.
S104, acquiring the relative position relation between the depth camera and the spray gun, and determining a target working path of the building spraying robot according to the relative position relation and the initial working path.
In this step, the obtaining the relative positional relationship between the depth camera and the spray gun, and determining the target working path of the building spray robot according to the relative positional relationship and the initial working path includes: and obtaining the maximum spraying rectangular area of the spray gun. And determining the overall spraying path coordinates of the target working area according to the maximum spraying rectangular area and the initial working path. And obtaining the vertical distance between the spray gun and the spraying plane where the target working area is located. And translating the overall spraying path coordinate along the normal direction of the spraying plane by the vertical distance to obtain a first space coordinate sequence of the spray gun in a camera coordinate system of the depth camera. And acquiring the relative position relation between the depth camera and the spray gun, and carrying out coordinate conversion on the first space coordinate sequence according to the relative position relation to obtain a second space coordinate sequence under the mechanical arm coordinate system of the mechanical arm where the spray gun is positioned. And calculating the mechanical arm joint driving space coordinate corresponding to each point in the second space coordinate sequence according to the functional relation between the Cartesian space coordinate of the spray gun and the mechanical arm joint driving space coordinate. And determining a target operation path of the building spraying robot according to the space coordinates driven by the mechanical arm joint.
S105, controlling the building spraying robot to spray according to the target operation path.
In this step, after the target working path is determined, the construction painting robot is controlled to paint according to the target working path. Wherein the spraying process of the building spraying robot includes, but is not limited to: and determining a first-stage sub-region of the range which can be sprayed by the mobile chassis at different points. And when the movable chassis is fixed, determining a second-stage subarea of the range which can be sprayed by the mechanical arm at different points. And determining the related operation point positions according to the plurality of point positions of the first-stage sub-region and the second-stage sub-region.
The embodiment of the invention provides a spraying method for intelligently identifying a door and window hole on a plane with the door and window hole by using an RGB-D depth camera, positioning and navigating a spraying module at the tail end of a mechanical arm and planning a spraying path. According to the spraying method, construction scene videos shot by the RGB-D depth camera can be displayed on the display device in real time, all door and window holes in the spraying range can be intelligently identified only by selecting a range needing to be sprayed on an upper frame of the display device by an operator, the door and window holes are avoided for spraying path planning, space coordinates of the path points of the terminal spraying module obtained through planning are transmitted to the mechanical arm, the mechanical arm moves to a designated position for spraying, and meanwhile, the spraying module spray gun is guaranteed to be perpendicular to a spraying surface.
In some embodiments of the invention, subsequent work has been done in order to advance the positioning of the construction painting robot. After determining the initial work area, further comprising: according to the space coordinates of the preset corner points of the initial operation area, calculating a plane equation of a rectangular plane where the initial operation area is located and a transformation matrix from a camera coordinate system of the depth camera to the rectangular plane. And according to the plane equation and the transformation matrix, three Euler angles of rotation required by the depth camera to be parallel to the rectangular plane are calculated. The three euler angles are taken as the rotation angles required for the spray gun to be perpendicular to the initial working area. And controlling the mechanical arm to rotate so as to enable the spray gun to rotate to a position corresponding to the rotation angle.
Embodiments of the present invention are further described below.
The spraying method provided by the invention mainly comprises the following steps:
step one, selecting a spraying area (namely an initial operation area), calculating a spraying plane space equation where the spraying area is located, and calculating the relative pose (Euler angle) of a spraying plane and a spray gun. The spray gun is positioned in a spraying module on the tail end of the manipulator.
And secondly, intelligently identifying the door and window holes in the spraying area by using a deep learning method, obtaining a target spraying area (namely a target operation area), and marking and obtaining the space coordinates of four corner points of the door and window holes.
And thirdly, carrying out spray path planning by avoiding the door and window holes, obtaining space coordinates of spray path points required by the spray module, converting the space coordinates into coordinates of a mechanical arm coordinate system, and reversely calculating the amount of rotation or expansion of each joint of the mechanical arm, controlling the spray module at the tail end of the mechanical arm to move to the corresponding space points, and realizing the accurate positioning navigation of the spray module.
More specifically, the spraying method can be implemented according to the following steps:
1. the RGB-D depth camera shoots in real time, and video pictures of the spraying plane shot by the RGB-D depth camera are displayed on the tablet personal computer in real time. The operator selects a rectangular area (initial work area) on the tablet computer to be sprayed. An operator selects a rectangular area to be sprayed on the tablet personal computer, and a rectangle can be determined by selecting three points, for example, the operator firstly selects two points to determine the length of one side of the rectangle, and then selects a third point to determine the height of the rectangle.
2. The spatial coordinates of the three points selected in the step 1 in the camera coordinate system can be calculated through the pixel coordinate-spatial coordinate transformation function of the RGB-D depth camera, and further the spatial coordinates of four corner points of the rectangular area determined by the three points can be calculated.
3. In order to reduce the distance measurement error of the depth camera, four corner coordinates are calculated for 5 adjacent frames in the video image, the median of the 5 frames is taken as a reference value, when the deviation between the rest 4 frames and the reference value is greater than 5cm, the frame data are removed, and finally the average value of the rest frame data is calculated to be taken as the space coordinates of four corner points of the rectangular area.
4. From the space coordinates of the four corner points of the rectangular area, a plane equation ax+by+cz+d=0 of the rectangular plane where the rectangular area is located and a transformation matrix R from the depth camera coordinate system to the rectangular plane can be calculated, so that three euler angles required to rotate the depth camera to be parallel to the rectangular plane can be obtained. Since the depth camera is arranged in the embodiment, the optical axis of the depth camera is parallel to the direction of the spray gun in the spraying module at the tail end of the manipulator, and the three Euler angles are the angles required to rotate the spray gun in the spraying module perpendicular to the spraying surface (the rectangular plane).
5. And identifying the door and window holes in the selected rectangular area (initial operation area) by using the trained deep learning model, and framing out 4 sides of the identified door and window holes in the rectangular area to obtain a target operation area. In the implementation, a computer vision deep learning method is adopted to train and identify a deep learning model of the door and window hole. Firstly, a large number of photos of various door and window holes are collected on site to serve as training data, and a vision-transformation deep learning model is adopted to train and identify deep learning model parameters of the door and window holes.
6. And calculating the space coordinates of the four corner points of the identified door and window hole by using a depth camera pixel coordinate-space coordinate transformation function.
7. In order to further reduce the error of the spatial coordinates of four corner points of the identified door and window hole, the spatial coordinates of the four corner points of the door and window hole can be projected onto the spraying plane where the determined target operation area is located for correction by utilizing the characteristic that the door and window hole is parallel to the spraying plane where the determined target operation area is located, and the corrected spatial coordinates of the corner points of the door and window hole are obtained.
8. And 5, dividing the target operation area determined in the step into a plurality of rectangles along the four sides of the identified door and window hole to respectively carry out spray coating path planning, and finally summarizing to form an overall spray coating path planning.
9. The maximum area which can be sprayed by the spray guns on the spraying module at the tail end of the manipulator is A, all spray guns are opened at the moment, and the spray gun system moves from one end of the guide rail to the other end of the guide rail. In order to reduce the number of movements of the mechanical arm and improve the spraying efficiency, each rectangle in step 8 should include as many rectangles a with the largest spraying area as possible, and the rest can be sprayed by adjusting the opening number of the spray guns and the travel of the spraying system on the guide rail.
10. In order to reduce the possibility of spraying the spray gun onto the door and window hole, each rectangle in the step 8 is sprayed from a corner close to the door and window hole, the area of the rectangle is smaller than the largest module spraying area A, the number of spray guns and the guide rail travel required to be opened for spraying can be calculated according to the length and width of the rectangle and the parameters of the spraying module, and each rectangle is ensured to be accurately sprayed without multiple spraying and leakage spraying.
11. And combining the steps to obtain the overall spraying path coordinates of the target operation area. Because the distance between the spraying module and the spraying plane is required to be ensured to be s during spraying, the total spraying path coordinate is required to be translated along the normal direction of the spraying plane by the distance s, and the space coordinate sequence of the spraying module at the tail end of the mechanical arm in the camera coordinate system is obtained.
12. And (3) carrying out coordinate transformation on the coordinate sequence in the camera coordinate system obtained in the step (11) according to the relative position relation between the installation position of the depth camera and the terminal spraying module, so as to obtain the space coordinate sequence of the terminal spraying module under the mechanical arm coordinate system.
13. According to the functional relationship between the cartesian space coordinates of the spraying module at the tail end of the mechanical arm and the driving space coordinates of the mechanical arm joint, the driving space coordinates (joint rotation amount or expansion amount) of the mechanical arm joint corresponding to each point in the space coordinate sequence in the step 12 can be calculated. And transmitting the rotation or expansion amount of the mechanical arm joint corresponding to each point position to the mechanical arm controller, so that the spraying module at the tail end of the mechanical arm can move along the planned path point position to perform spraying operation.
14. The spraying path coordinates of the mechanical arm coordinate system are converted into the site global coordinates for storage, so that whether a certain spraying point position is sprayed or not can be inquired and judged, repeated spraying can be avoided, and the spraying efficiency is improved.
According to the embodiment provided by the invention, the automatic spraying operation can be carried out on the area to be sprayed in various typical building spraying scenes, so that the position of the door and window opening is automatically avoided, and the terminal spraying module can accurately move and spray along the planned spraying path. According to the spraying method, the RGB-D depth camera is used for spraying planning, the area needing to be sprayed is selected on the display device, automatic spraying can be performed after clicking, operation is simple and convenient, and the accuracy meets the spraying construction requirement.
FIG. 2 shows door and window openings identified by a deep learning method, the door and window openings being labeled with windows 0.95 and windows 0.96, the numbers representing the confidence of the identification.
Fig. 3 shows a schematic view displayed on a display device after the relevant path determination is made, wherein the outermost box is the selected spray area and the small circles are spray path points. The small rectangle is a spraying rectangle of each spraying path point, a certain overlap joint length is needed between the spraying rectangles, a line segment between the spraying path points is a spraying module moving path at the tail end of the mechanical arm, and the middle lamp strip represents the identified door and window hole.
The embodiment of the invention also provides a building spraying robot which comprises a movable chassis, a mechanical arm connected with the movable chassis, a spraying module and a control module, wherein the spraying module is positioned at the tail end of the mechanical arm, the spraying module comprises a spray gun and a depth camera, and the control module is used for executing the spraying method provided by any embodiment.
In the embodiment of the invention, the spraying module is provided with a biaxial holder, the depth camera is fixed on the biaxial holder, and when the biaxial holder is positioned at the initial position, the optical axis of the depth camera is parallel to the spraying direction of the spray gun.
In the building spraying robot provided by the embodiment of the invention, a spraying module is arranged at the tail end of a mechanical arm, and the spraying module consists of a guide rail and a plurality of spray gun systems which are arranged side by side. And a biaxial holder is arranged on the spraying module, the depth camera is fixed on the holder, and the optical axis of the depth camera is ensured to be parallel to the direction of the spray gun when the holder is at the initial position. The depth camera is connected with the raspberry pie 4B through a USB line (the depth camera needs to be connected with a computer to read video data of the depth camera), and the depth camera wirelessly transmits RGB images, depth images and camera intrinsic parameters to a display device through a router of a mechanical arm control cabinet. The display device is provided with pre-written software, the software can display an RGB image and a depth image shot by the depth camera in real time, can input relevant parameters of the mechanical arm and relevant parameters of the tail end spraying module, can load a trained door and window hole recognition deep learning model, and all relevant calculation is carried out on the tablet personal computer. The operator selects a required spraying area on the display device, and clicks a start button, so that the position of the door and window hole can be automatically identified and displayed, and the planned spraying path is displayed for the operator to preview and confirm. After the operator confirms, the display device sends the mechanical arm driving space coordinates corresponding to the spraying paths to the mechanical arm controller, and the mechanical arm tail end spraying module starts to move and spray according to the planned paths.
After the spraying is finished, the operator can continue to select points for spraying. The completed spraying coordinates can be converted into field global coordinates for storage, and when the sprayed point positions appear, operators can be reminded of spraying at the positions, so that the situation of repeated spraying is effectively avoided, and the spraying efficiency and the spraying quality are improved.
According to the embodiment provided by the invention, the spray area is selected through the RGB depth camera, a space equation of a spray face is obtained, the relative pose of the spray gun and the spray face is calculated, and the spray module space positioning is performed. And a deep learning method is adopted for the spraying surface video shot by the RGB depth camera, the position of the door and window opening is automatically identified and accurately framed, the angular point space coordinates of the door and window opening are calculated, effective correction is carried out, and the accuracy is improved. And (3) carrying out spray path planning by avoiding the door and window holes, obtaining the space coordinates of the spray path points required by the spray module, calculating the rotation or expansion quantity required by each joint of the mechanical arm, controlling the spray module to accurately move to the corresponding space points, and realizing the accurate positioning and navigation of the spray module.
In the several embodiments provided in the present invention, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another apparatus, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules.
The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of the various embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, randomAccessMemory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present invention is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all required for the present invention.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
The foregoing describes a construction painting robot and a painting method thereof, and those skilled in the art, based on the concepts of the embodiments of the present invention, may vary in terms of specific embodiments and application ranges, and in summary, the disclosure should not be construed as limiting the invention.
Claims (10)
1. The spraying method is applied to a building spraying robot, the building spraying robot comprises a movable chassis, a mechanical arm connected with the movable chassis and a spraying module arranged at the tail end of the mechanical arm, the spraying module comprises a spray gun and a depth camera, and the spraying method comprises the following steps:
acquiring an initial operation area from a video image of the depth camera;
carrying out door and window hole identification on the initial operation area to obtain a target operation area, wherein the target operation area does not comprise the door and window hole;
dividing the target operation area according to the door and window holes, and planning a path according to the dividing result to obtain an initial operation path;
acquiring the relative position relation between the depth camera and the spray gun, and determining a target operation path of the building spraying robot according to the relative position relation and the initial operation path;
and controlling the building spraying robot to spray according to the target operation path.
2. The spray method of claim 1, wherein the acquiring an initial work area from a video image of the depth camera comprises:
displaying the video image shot by the depth camera;
and acquiring a selected operation on the video image, determining a selected area according to the selected operation, and taking the selected area as an initial operation area.
3. The spray method of claim 2 wherein said initial working area is a rectangular plane, said obtaining a selected operation on said video image, determining a selected area based on said selected operation comprising:
acquiring a selected operation on the video image, the selected operation comprising three anchor points selected on the video image;
calculating the space coordinates of the three positioning points in a camera coordinate system of the depth camera according to a pixel coordinate-space coordinate transformation function of the depth camera;
according to the space coordinates of the three positioning points, determining the space coordinates of four corner points of a rectangle enclosed by the three positioning points;
and determining the selected area according to the space coordinates of the four corner points.
4. A spray method according to claim 3, wherein said determining said selected area from the spatial coordinates of said four corner points comprises:
acquiring a plurality of adjacent image frames in the video image;
acquiring corner coordinates of the four corners in a plurality of image frames respectively;
taking the median of a plurality of image frames as a reference value, deleting the data of the image frames when the deviation between the rest of the image frames and the reference value is larger than a first preset distance, and taking the average value of the data of the rest of the image frames as the space coordinates of four corner points of the selected area;
and determining the selected area according to the space coordinates of the four corner points.
5. The spray coating method according to any one of claims 1 to 4, further comprising:
according to the space coordinates of the preset corner points of the initial operation area, calculating a plane equation of a rectangular plane where the initial operation area is located and a transformation matrix from a camera coordinate system of the depth camera to the rectangular plane;
according to the plane equation and the transformation matrix, three Euler angles required to rotate the depth camera to be parallel to the rectangular plane are calculated;
taking the three Euler angles as rotation angles required by the spray gun to be perpendicular to the initial operation area;
and controlling the mechanical arm to rotate so as to enable the spray gun to rotate to a position corresponding to the rotation angle.
6. The spray method of claim 1, wherein the performing door and window cavity recognition on the initial work area, obtaining a target work area, comprises:
acquiring a preset deep learning model;
and carrying out door and window hole identification on the initial operation area by using the deep learning model to obtain a target operation area which does not contain the door and window hole.
7. The spray method of claim 6, wherein after the door and window cavity identification is performed on the initial work area using the deep learning model, further comprising:
acquiring space coordinates of four corner points of the identified door and window hole;
and projecting the space coordinates of the four corner points of the door and window hole to the initial operation area, and correcting according to a projection result to obtain the corrected space coordinates of the four corner points of the door and window hole.
8. The spray coating method of claim 1, wherein the obtaining the relative positional relationship of the depth camera and the spray gun, and determining the target working path of the construction spray coating robot based on the relative positional relationship and the initial working path comprises:
obtaining the maximum spraying rectangular area of the spray gun;
determining overall spraying path coordinates of the target working area according to the maximum spraying rectangular area and the initial working path;
obtaining the vertical distance between the spray gun and the spraying plane where the target operation area is located;
translating the overall spraying path coordinates along the normal direction of the spraying plane by the vertical distance to obtain a first space coordinate sequence of the spray gun in a camera coordinate system of the depth camera;
acquiring a relative position relation between the depth camera and the spray gun, and performing coordinate conversion on the first space coordinate sequence according to the relative position relation to obtain a second space coordinate sequence under a mechanical arm coordinate system of a mechanical arm where the spray gun is positioned;
according to the functional relation between the Cartesian space coordinates of the spray gun and the mechanical arm joint driving space coordinates, calculating the mechanical arm joint driving space coordinates corresponding to each point in the second space coordinate sequence;
and determining a target operation path of the building spraying robot according to the mechanical arm joint driving space coordinates.
9. A building spraying robot, characterized in that the building spraying robot comprises a movable chassis, a mechanical arm connected with the movable chassis, a spraying module and a control module, wherein the spraying module is positioned at the tail end of the mechanical arm, the spraying module comprises a spray gun and a depth camera, and the control module is used for executing the spraying method according to any one of claims 1 to 8.
10. The architectural spraying robot of claim 9, wherein the spraying module is provided with a two-axis cradle head, the depth camera is fixed on the two-axis cradle head, and when the two-axis cradle head is positioned at the initial position, the optical axis of the depth camera is parallel to the spraying direction of the spray gun.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311006244.6A CN117162109A (en) | 2023-08-10 | 2023-08-10 | Building spraying robot and spraying method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311006244.6A CN117162109A (en) | 2023-08-10 | 2023-08-10 | Building spraying robot and spraying method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117162109A true CN117162109A (en) | 2023-12-05 |
Family
ID=88942117
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311006244.6A Pending CN117162109A (en) | 2023-08-10 | 2023-08-10 | Building spraying robot and spraying method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117162109A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117649545A (en) * | 2024-01-30 | 2024-03-05 | 武汉市双桥科技有限公司 | Spray track planning method and system based on artificial intelligence |
-
2023
- 2023-08-10 CN CN202311006244.6A patent/CN117162109A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117649545A (en) * | 2024-01-30 | 2024-03-05 | 武汉市双桥科技有限公司 | Spray track planning method and system based on artificial intelligence |
| CN117649545B (en) * | 2024-01-30 | 2024-04-12 | 武汉市双桥科技有限公司 | Spray track planning method and system based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210109548A1 (en) | Heading generation method and system of unmanned aerial vehicle | |
| EP3705239A1 (en) | Calibration system and method for robotic cells | |
| DE112007003293B4 (en) | Optical instrument and method for obtaining distance and image information | |
| US20170259920A1 (en) | Automatic painting system with drone, user interface and computer vision | |
| CN117162109A (en) | Building spraying robot and spraying method thereof | |
| CN111932609A (en) | Cloud deck calibration method and device for valve hall equipment inspection robot and storage medium | |
| CN110815205A (en) | Calibration method, system and device of mobile robot | |
| CN105607651A (en) | Rapid aligning system and method via visual guiding | |
| CN110490942A (en) | A kind of mobile camera calibration method based on the second arm of SCARA manipulator | |
| CN111571082A (en) | Automatic welding method and device, mobile terminal and readable storage medium | |
| CN112743548A (en) | Method, system and terminal for unifying hand-eye calibration of two mechanical arms | |
| CN110039520B (en) | Teaching and processing system based on image contrast | |
| JP7334460B2 (en) | Work support device and work support method | |
| CN113450414A (en) | Camera calibration method, device, system and storage medium | |
| US20230123629A1 (en) | 3d computer-vision system with variable spatial resolution | |
| CN113906358B (en) | Control method, device and system for movable platform | |
| CN113470121B (en) | Autonomous mobile platform, external parameter optimization method, device and storage medium | |
| WO2022105575A1 (en) | Image processing method and related device | |
| CN113280790B (en) | Building mapping device based on laser positioning | |
| CN114800574A (en) | Robot automatic welding system and method based on double three-dimensional cameras | |
| Kim et al. | Rover mast calibration, exact camera pointing, and camera handoff for visual target tracking | |
| CN113858214A (en) | Positioning method and control system for robot operation | |
| WO2020172878A1 (en) | Method and device for shooting and aiming control of movable platform, and readable storage medium | |
| CN111025324A (en) | Household pattern generating method based on distance measuring sensor | |
| US11983821B2 (en) | Control method and apparatus for movable platform, and control system thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |