RU2804654C9 - Method and device for controlling a loader based on binocular vision - Google Patents

Abstract

FIELD: mining.

SUBSTANCE: method and apparatus for controlling a roadheader based on binocular vision, in which status information corresponding to the cutting surface area and status information corresponding to the material laying area are determined by binocular vision technology, then a target task to be performed by the roadheader is determined, and a target position corresponding to the target task according to the status information corresponding to the cutting surface area and the status information corresponding to the material laying area. Then, the roadheader is guided from the starting position to the target position, which corresponds to the cutting surface area or to the material laying area, by tracking its movement through binocular vision, thereby realizing identification and positioning for the target task, which corresponds to the cutting surface area or the material laying area located in a tunnel, using binocular vision technology. The roadheader is then directed to move to the target position, so that the roadheader performs the target task, and after the target task is completed, the roadheader is controlled according to the map data of the environment surrounding the roadheader to move from the target position to the starting position along the path movement.

EFFECT: improved efficiency.

10 cl, 8 dwg

Description

CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to Chinese Patent Application No. 202210502665.7, filed on May 10, 2022, entitled "Method and apparatus for bottomhole identification and bottomhole monitoring based on binocular vision for a roadheader", the entire contents of which are incorporated by reference in their entirety herein .

TECHNICAL FIELD

[0002] This application relates to the field of roadheaders and, in particular, relates to a method and apparatus for controlling a roadheader based on binocular vision.

BACKGROUND OF THE ART

[0003] Currently, the face of an intelligent roadheader may have problems such as uneven cutting surfaces, poor contouring, and large accumulations of material on both sides, which make the automatic cutting process of the roadheader difficult. In such a case, the roadheader must move to the target area to process it. For example, during automatic cutting by an intelligent roadheader, it is possible that some of the discharged materials cannot be collected by the brush device at the base of the roadheader. As cutting continues, a certain material stacking area will be formed on both sides of the face, which obstructs the movement of the punching attachment and the progress of the roadheader, resulting in the need to collect these materials manually, which reduces the speed of automatic operation and the efficiency of the roadheader.

DISCLOSURE OF THE INVENTION

[0004] This document provides a method and apparatus for controlling a roadheader based on binocular vision, which can identify and position a cutting surface area and a material laying area located in a tunnel through binocular vision technology, and then guide the roadheader to move it into a blanking surface area and/or a material laying area, thereby completing the blanking of a blanking surface area and/or collecting material in the material laying area by the roadheader.

[0005] In a first aspect, a method for controlling a roadheader based on binocular vision is provided, including:

[0006] obtaining state information corresponding to the cutting surface area and state information corresponding to the material laying area according to binocular vision images in the tunnel;

[0007] determining a target task to be performed by the roadheader and a target position corresponding to the target task according to state information corresponding to the cutting surface area and state information corresponding to the material laying area;

[0008] directing the roadheader according to tracking its movement through binocular vision to move it from a starting position to a target position, and mapping map data of the environment and the moving path of the roadheader while it is moving;

[0009] controlling the roadheader to perform a target task provided that the roadheader moves to the target position; And

[0010] controlling the roadheader according to the mapped environment data to move it from a target position to a home position along a moving path provided that the roadheader completes the target task.

[0011] In addition, the status information includes information about the degree of formation of edges, information about the degree of alignment and information about the first position of the cutting surface area, as well as information about the height and information about the second position of the material laying area, while obtaining information about condition corresponding to the cutting surface area, and condition information corresponding to the material laying area, according to binocular vision images in the tunnel includes:

[0012] obtaining information about the face point cloud according to binocular vision images in the tunnel;

[0013] separating the cutting surface area from the material laying area according to the information about the face point cloud;

[0014] determining information about the degree of formation of edges, information about the degree of alignment and information about the first position of the region of the cutting surface according to the point cloud data in the region of the cutting surface;

[0015] determining height information and second position information of the material laying area according to point cloud data in the material laying area;

[0016] determining the target task to be performed by the roadheader and the target position corresponding to this target task, according to the state information corresponding to the cutting surface area, and the state information corresponding to the material laying area, includes:

[0017] determining the target task to be performed by the roadheader according to information about the degree of formation of edges, information about the degree of leveling of the cutting surface area, and information about the height of the material laying area; wherein the target task comprises at least one of a punching task for the punching surface area and a material collection task for the material laying area; And

[0018] determining the target position according to the first position information and/or the second position information corresponding to the target task.

[0019] In addition, obtaining information about the face point cloud according to binocular vision images in the tunnel includes:

[0020] obtaining a dense parallax map of the face according to binocular vision images in the tunnel;

[0021] calculating the spatial coordinates of each point cloud data on the dense parallax map to obtain information about the face point cloud on the dense parallax map;

[0022] separating the cutting surface area from the material placement area according to the face point cloud information, including:

[0023] calculating, according to the face point cloud information on the dense parallax map, a spatial plane corresponding to the point cloud information in the cutting surface area, and determining point cloud data corresponding to the cutting surface area;

[0024] using as the initial point set of the specified data a cloud of points, the distance of which from the spatial plane corresponding to the area of the cutting surface is less than a threshold value;

[0025] projecting point cloud data within the original point set onto a spatial plane corresponding to the surface area of the cutting to obtain a depth map;

[0026] training a semantic segmentation model based on map label data collected from the depth map; And

[0027] separating the cutting surface area from the material placement area on the depth map based on the semantic segmentation model.

[0028] In addition, determining information about the degree of formation of edges, information about the degree of alignment and information about the first position of the cutting surface area according to the point cloud data in the cutting surface area includes:

[0029] projecting point cloud data corresponding to the cutting surface region onto an actual physical coordinate system to obtain actual spatial point cloud data in the cutting surface region;

[0030] obtaining information about the first position corresponding to the area of the surface of the notch, according to the cloud data of actual spatial points in the area of the surface of the notch;

[0031] determining height information and second position information in the material laying area according to point cloud data in the material laying area, including:

[0032] projecting point cloud data corresponding to the material placement area onto an actual physical coordinate system to obtain actual spatial point cloud data in the material placement area, and

[0033] obtaining information about the second position corresponding to the material laying area according to the actual spatial point cloud data in the material laying area.

[0034] In addition, guiding the roadheader according to its movement tracking by binocular vision to move from a starting position to a target position includes:

[0035] obtaining real-time point cloud data in the area of the cutting surface through binocular cameras attached to both sides of the roadheader; And

[0036] calculating the relative position between the roadheader and the cutting surface area according to real-time point cloud data in the material laying area to achieve tracking of the movement of the roadheader.

[0037] In addition, guiding the roadheader according to tracking its movement through binocular vision to move it from a starting position to a target position and mapping the cartographic data of the environment and the moving path of the roadheader during its movement includes:

[0038] obtaining a body area and an initial rotation angle of the roadheader, wherein the body area is a covering area of the body of the roadheader;

[0039] defining a virtual area that moves with the roadheader, according to the area of the body of the roadheader;

[0040] directing the roadheader to move directly in the direction of the starting rotation angle according to the movement tracking of the virtual area through binocular vision, and directing the roadheader when the virtual area touches the side wall to move it by rotation until the roadheader moves to the target position;

[0041] mapping in real time a moving path of the roadheader from a starting position to a target position according to a straight path along which the roadheader is guided to move directly in the direction of the starting angle of rotation, and a rotation path along which the roadheader is moved by rotation; And

[0042] mapping real-time cartographic data about the environment of the roadheader as it moves from a starting position to a target position according to binocular images of the roadheader during its movement obtained through binocular vision.

[0043] In addition, the direction of the roadheader to move straight in the direction of the initial rotation angle, and the direction of the roadheader when the virtual area touches the side wall to move it by rotation according to the tracking of the movement of the virtual area by binocular vision until the roadheader will move to the target position where the material placement area is located, includes:

[0044] preliminary determining the number of turning points in the trajectory of movement of the roadheader from the starting position to the target position; And

[0045] determining the rotation angle that guides one rotational movement of the roadheader, according to the initial rotation angle and the number of rotation points in the path, provided that the virtual region touches the side wall.

[0046] In a second aspect, a device for controlling a mining machine based on binocular vision is provided, comprising:

[0047] a state information acquiring module for obtaining state information corresponding to a cutting surface area and state information corresponding to a material laying area according to binocular vision images in the tunnel;

[0048] a target task determination module for determining a target task to be performed by the roadheader and a target position corresponding to the target task according to state information corresponding to the cutting surface area and state information corresponding to the material laying area;

[0049] a navigation module for guiding the roadheader according to tracking its movement through binocular vision to move from a starting position to a target position and mapping map data of the environment and the moving path of the roadheader while it is moving;

[0050] a first control module for controlling the roadheader to perform a target task, provided that the roadheader moves to the target position; And

[0051] a second control module for controlling the roadheader according to the mapped environment data to move it from a target position to a home position along a movement path provided that the roadheader completes the target task.

[0052] In a third aspect, there is provided a computing device comprising a memory device, a processor, and a computer program stored in the memory device and executed by the processor, wherein when the processor executes the computer program, the steps of the above-mentioned binocular vision-based method of controlling a road miner are executed.

[0053] In a fourth aspect, there is provided a computer-readable storage medium in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned binocular vision-based method of controlling a mining machine are carried out.

[0054] In the solutions implemented in the above-described manner and by a binocular vision-based roadheader control device, as well as a computing device and a storage medium, state information corresponding to the cutting surface area and state information corresponding to the material laying area, are determined by binocular vision technology, then the target task to be performed by the roadheader and the target position corresponding to the target task are determined according to the state information corresponding to the cutting surface area and the state information corresponding to the material laying area, then the roadheader is guided from starting position to a target position corresponding to the punching surface area or material laying area when tracking by binocular vision, and thus achieving identification and positioning of the target task corresponding to the punching surface area or material laying area located in the tunnel through binocular vision technology, then the roadheader is guided to move to the target position, so that the roadheader performs the target task, and after the target task is completed, the roadheader is controlled according to the mapping data of the roadhead's environment to move from the target position to the starting position along the path movement.

BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, a brief description will be given below of the accompanying drawings necessary for use in describing the embodiments of the present invention. It will be appreciated that the drawings in the following description are only some of the embodiments of the present invention, and other drawings can be obtained by one skilled in the art in accordance with these accompanying drawings without creative work.

[0056] In FIG. 1 is a flow diagram of a method for controlling a roadheader based on binocular vision in one embodiment of the present invention;

[0057] In FIG. 2 is a block diagram of a specific embodiment of step S101 of FIG. 1;

[0058] In FIG. 3 is a block diagram of a specific embodiment of step S1012 of FIG. 2;

[0059] In FIG. 4 is a block diagram of a specific embodiment of step S103 of FIG. 1;

[0060] In FIG. 5 is a schematic diagram of a roadheader moving from a starting position to a target position in the area of material placement;

[0061] In FIG. 6 is a schematic block diagram of a device for controlling a mining machine based on binocular vision in one embodiment of the present invention;

[0062] In FIG. 7 is a schematic block diagram of a computing device in one embodiment of the present invention; And

[0063] In FIG. 8 is another schematic block diagram of a computing device in one embodiment of the present invention.

IMPLEMENTATION OF THE INVENTION

[0064] The technical solutions in the embodiments of the present application will be explicitly and fully described below in connection with the accompanying drawings in the embodiments of the present application. It will be appreciated that the described embodiments are part of the embodiments of this application and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by a person skilled in the art from the prior art without creative work will be within the scope of protection of this application.

[0065] In one embodiment of the present application, a method for controlling a roadheader based on binocular vision is provided, as shown in FIG. 1, including the following steps.

[0066] In step S101, state information corresponding to the cutting surface area and state information corresponding to the material laying area are obtained according to the binocular vision images in the tunnel.

[0067] Currently, navigation aids for the positioning of existing roadheaders mainly rely on external calibration tables or laser radar, which is associated with high costs and high calibration requirements. In addition, known roadheaders do not yet have effective technology for identifying the area of the cutting surface in the tunnel and the area where the material is laid. In view of this, herein, by identifying the tunnel using binocular vision technology and obtaining the state information corresponding to the cutting surface area and the state information corresponding to the material laying area, the identification and positioning of the cutting surface area and the material laying area in the tunnel can be realized , and at the same time, by inertial navigation integrated positioning of the roadheader using binocular vision technology, the costs associated with navigation aids for positioning the roadheader can be further reduced.

[0068] Here, the status information includes edge formation degree information, leveling degree information, first position information of a cutting surface area, height information, and second position information of a material stacking area, and, as shown in FIG. 2, obtaining state information corresponding to the cutting surface area and state information corresponding to the material laying area according to binocular vision images in the tunnel includes the following steps.

[0069] In step S1011, information about the face point cloud is obtained according to binocular vision images in the tunnel.

[0070] Here, a binocular camera equipped with a multi-point infrared uniform illumination supplement function is attached to both sides of the roadheader, where the binocular camera is designed to obtain binocular images of the tunnel face on the left front side and the right front side of the roadheader.

[0071] In some embodiments, obtaining information about the face point cloud according to the binocular vision images in the tunnel may include: obtaining a dense parallax map of the face according to the binocular vision images in the tunnel; and calculating the spatial coordinates of the data of each point cloud in the dense parallax map to obtain information about the face point cloud in the dense parallax map.

[0072] It should be noted that the binocular image is calculated by the SGM algorithm to obtain a dense parallax map corresponding to the face, and then the spatial coordinates of the individual point cloud data in the dense parallax map are calculated to obtain information about the point cloud of the face in the dense parallax map.

[0073] Here, the point cloud information on the dense parallax map contains the spatial coordinates of the point cloud data on the dense parallax map, and at the time of calculating the spatial coordinates of the point cloud on the dense parallax map, the spatial coordinates {x _k , y _k , z _k }k =1, 2, …, K point cloud data on a dense parallax map are calculated according to the formula

Where:

l - base distance;

disp - parallax value;

ƒ _x , and ƒ _y are the focal lengths in pixels for the point cloud data; And

x _p , and y _p are the pixel coordinates for the point cloud data.

[0074] In step S1012, the cutting surface area is separated from the material laying area according to the face point cloud information.

[0075] Here, the cutting surface area is separated from the material laying area according to the face point cloud information, so that the state information of the separated cutting surface area and the material laying area are evaluated separately, thereby obtaining accurate state information corresponding to the cutting surface area and areas for laying the material, respectively.

[0076] In some embodiments, with reference to FIG. 3, separating the cutting surface area from the material laying area according to the information about the face point cloud includes:

[0077] Step S1012-1, in which, according to the face point cloud information on the dense parallax map, a spatial plane corresponding to the point cloud information in the cutting surface area is calculated, and point cloud data corresponding to the cutting surface area is determined.

[0078] When calculating the spatial plane corresponding to the point cloud information of the notch surface area, this calculation is performed by combining RANSAC with the least squares method to fit the equation of the spatial plane corresponding to the point cloud information of the notch surface area. The specific calculation method is as follows.

[0079] It may be assumed that the spatial plane equation corresponding to the point cloud information in the notch surface region is xr _x +yr _y +zr _z =1, and the set of point clouds of the notch surface region is {x _k , y _k , z _k }k=1, 2, …, K, then:

[0080] , let , A

in this case r=(V ^T V) ^-1 V ^T 1, i.e. a spatial plane corresponding to information about the point cloud in the area of the cutting surface is obtained, where r denotes a spatial plane corresponding to information about the point cloud in the area of the cutting surface.

[0081] In step S1012-2, point cloud data whose distance from the spatial plane corresponding to the cutting surface area is less than a threshold value is taken as the initial point set.

[0082] Since there is a certain distance between the punch surface area and the material laying area, by separately calculating the spatial planes corresponding to the point cloud information in the punch surface area and comparing the resulting spatial planes with a threshold value, the resulting point set can have point cloud data, appropriate areas for laying material within the threshold range.

[0083] In this embodiment, point cloud data whose distance from the spatial plane r corresponding to the point cloud information in the area of the cutting surface is within the threshold value range (0 m to 1 m) is selected as the initial point set P ₀ .

[0084] In step S1012-3, point cloud data within the original point set is projected onto a spatial plane corresponding to the cutting surface area to obtain a depth map.

[0085] Here, each point cloud data within the original point set P ₀ is projected onto a spatial plane r corresponding to the point cloud information in the area of the cutting surface to obtain a depth map.

[0086] In step S1012-4, a semantic segmentation model is trained based on the map label data collected on the depth map.

[0087] It should be noted that the semantic segmentation model herein is a U-net semantic segmentation model.

[0088] In step S1012-5, the cutting surface area is separated from the material laying area on the depth map based on the semantic segmentation model.

[0089] Here, during the separation of the cutting surface area from the material laying area on the depth map, the cutting surface area and the material laying area on the depth map are identified through a semantic segmentation model, and then the point cloud data in the cutting surface area and the material laying area are respectively re-identified projecting into corresponding physical coordinate systems that correspond to the cutting surface region and the material laying region, thereby obtaining corresponding actual spatial point cloud data corresponding to the cutting surface region and the material laying region, to complete the separation of the cutting surface region from the material laying region.

[0090] In step S1013, edge formation degree information, leveling degree information, and first position information of the cutting surface area are determined according to the point cloud data of the cutting surface area.

[0091] In some embodiments, determining the edge formation degree information, the leveling degree information, and the first position information of the notch surface region according to the point cloud data of the notch surface region includes: projecting the point cloud data corresponding to the notch surface region onto an actual physical coordinate system to obtain cloud data of actual spatial points of the cutting surface area and obtain information about the first position corresponding to the cutting surface area according to the data of the cloud of actual spatial points of the cutting surface area.

[0092] In this case: it is determined that the edge of this area of the cutting surface should be punched out under the condition that the edge formation degree information has a degree value that is greater than a threshold value; and it is determined that the cutting surface of this region of the cutting surface should be cut under the condition that the leveling degree information has a degree value that is greater than the threshold value.

[0093] In step S1014, height information and second position information of the material stacking area are determined according to the point cloud data of the material stacking area.

[0094] It should be noted that in order to prevent the roadheader from collecting materials for a material laying area with a low height, determining the height information and the second position information of the material laying area according to the point cloud data of the material laying area further includes: comparing the height information of the material laying area with a predetermined height threshold and using a material stacking area whose height is not less than the predetermined height threshold as a material stacking area in which the materials are to be collected. The height is set as an estimate for the area where material is to be collected. The material stacking area is considered to exceed a threshold value, and the subsequent material collection step is performed under the condition that the detected height of the material stacking area is greater than the predetermined height threshold, otherwise the material collection job is skipped.

[0095] Determining the height information and the second position information of the material laying area according to the point cloud data of the material laying area includes: projecting the point cloud data corresponding to the material laying area onto the actual physical coordinate system to obtain the actual spatial point cloud data of the material laying area, and obtaining information about the second position corresponding to the material laying area, according to the data of the cloud of actual spatial points of the material laying area.

[0096] In step S102, the target task to be performed by the roadheader and the target position corresponding to the target task are determined according to the state information corresponding to the cutting surface area and the state information corresponding to the material laying area.

[0097] It should be noted that the target task includes at least one of a punching task for a punching surface area and a material collection task for a material laying area, and the target position includes a position of a punching surface area for which the roadheader performs a punching task, and the position of the material laying area for which the roadheader performs the task of collecting material. For example, when the target task is a material collection task for a material stacking area in a tunnel, the target position is a position of a material stacking area in a tunnel.

[0098] In some embodiments, determining a target task to be performed by the roadheader and a target position corresponding to the target task, according to state information corresponding to the cutting surface area, and state information corresponding to the material placement area, may include: determining the target the task to be performed by the roadheader, according to information about the degree of formation of edges and information about the degree of leveling of the cutting surface area and information about the height of the material laying area; wherein the target task comprises at least one of a punching task for the punching surface area and a material collection task for the material laying area; and determining a target position according to the first position information and/or the second position information corresponding to the target task.

[0099] Here, after extracting the point cloud data of the notch surface area, the specified point cloud data of the notch surface area is used to fit the edge profile of the notch surface area, and then the edge formation degree of the notch surface area is evaluated according to the deviation of the fitting result from the target result, thus obtaining information about the degree of edge formation of the cutting surface area; calculating the average value and variance of the distance from the point cloud data of the cutting surface area to a spatial plane r corresponding to the point cloud information in the cutting surface area, and comprehensively evaluating the leveling degree of the cutting surface area through the average value and variance, thereby obtaining information about the leveling degree of the area cutting surfaces.

[0100] In step S103, according to the tracking of the movement of the roadheader by means of binocular vision, the roadheader is guided to move from a starting position to a target position, and mapping data of the environment and the trajectory of the roadheader during its movement is carried out.

[0101] It should be noted that while guiding the roadheader to move from the starting position to the target position of the material laying area, the hydraulic motor equipped with the roadheader for driving the roadheader causes the roadheader to move from the starting position to the target position of the laying area material according to information about the detection of the cutting surface area and the material laying area in the tunnel using binocular vision.

[0102] In this case, by monitoring the movement of the roadheader in real time through binocular vision, the position of the roadheader during its movement can be determined, and the direction of movement of the roadheader from the starting position to the target position of the material laying area can be realized according to the position of the roadheader obtained by tracking, i.e. real-time direction of movement of the roadheader when the tracked position of the roadheader deviates from moving towards the target position, to ensure that the roadheader moves to the target position where the material laying area is located.

[0103] In some embodiments, according to binocular vision tracking of a roadheader, directing the roadheader to move from a home position to a target position includes: obtaining real-time point cloud data in a region of the cutting surface via binocular cameras attached to both sides of the roadhead combine; and calculating a relative position between the roadheader and the cutting surface area according to real-time point cloud data in the material laying area to achieve tracking of the movement of the roadheader.

[0104] Obtaining real-time point cloud data in the cutting surface area during the movement of the roadheader through binocular cameras attached on both sides of the roadheader not only provides real-time acquisition of the relative position between the roadheader and the cutting surface area for implementation tracking the movement of the roadheader, but also facilitates the mapping of real-time environmental mapping data as the roadheader moves from a starting position to a target position through real-time binocular visual images.

[0105] In order for the roadheader to move exactly to the target position of the material placement area, with reference to FIG. 4, according to the tracking of the movement of the roadheader by binocular vision, directing the roadheader to move from a starting position to a target position and mapping the cartographic data of the environment and the moving path of the roadheader during its movement may include the following steps.

[0106] In step S1031, the body area and initial rotation of the roadheader are obtained, wherein the body area is a body covering area of the roadheader.

[0107] As shown in FIG. 5, the material laying area is located on the left side of the drawing, i.e. area S ₀ is the area of the roadheader body, and θ ₀ is the initial rotation angle of the roadheader. In this case, the initial rotation angle is the angle at which the roadheader moves when it begins to move from the starting position to the target position of the material placement area.

[0108] In step S1032, a virtual area is set that moves with the roadheader according to the area of the body of the roadheader.

[0109] Region S ₁ in FIG. 5 is a virtual area defined according to the body area of the roadheader. In this case, the virtual area can move with the roadheader, the width of the virtual area is greater than the width of the body area, and the roadheader can reach different end positions by selecting the width of the virtual area.

[0110] In step S1033, directing the roadheader to move directly in the direction of the starting rotation angle according to the movement tracking of the virtual area by binocular vision, and when the virtual region touches the side wall, guiding the roadheader to move in rotation until it moves to the target position.

[0111] The above can be understood as follows: first, the roadheader is directed to move by rotation to the initial rotation angle θ ₀ ; then direct the roadheader to move straight forward in the direction of the original steering angle; detect in real time whether the area S ₁ touches the side wall (i.e., whether the point cloud entry into the area S ₁ is detected) through a binocular camera located on the roadheader, and if the area S ₁ touches the side wall, direct the roadheader to turning until the miner's deflection angle is 0. At this time, the miner moves straight to reach the target position, where the material laying area is located on the side of the face.

[0112] In addition, by tracking the movement of the virtual area in real time through binocular vision, the position of the roadheader during movement can be determined, and according to the position of the roadheader obtained by said tracking, the roadheader can be moved from the starting position to the target position in the laying area material is controlled in real time.

[0113] Preferably, directing the roadheader to move straight in the direction of the original rotation angle, and when the virtual area touches the side wall, directing the roadheader to move it by rotating according to tracking the movement of the virtual area through binocular vision until the roadheader moves to the target position in which the material laying area is located includes: preliminary determination of the number of turning points of the trajectory of movement of the roadheader from the starting position to the target position; and determining a rotation angle that directs one rotational movement of the roadheader according to the initial rotation angle and the number of rotation points of the path under the condition that the virtual region touches the side wall.

[0114] In this case, the predetermined number of turning points of the trajectory is n, and the turning angle that guides the rotational movement of the roadheader when the virtual area touches the side wall is

[0115] In step S1034, real-time mapping of the moving path of the roadheader from the starting position to the target position is carried out according to the straight path along which the roadheader is guided to move directly in the direction of the starting rotation angle, and the rotation path along which the roadheader moves rotation.

[0116] In step S1035, real-time mapping data of the environment of the roadheader while it is moving from the starting position to the target position is performed according to the binocular vision images of the roadheader while it is moving obtained by binocular vision.

[0117] In this case, while guiding the roadheader to move, by collecting ORB feature points using binocular cameras, a feature point map is constructed in real time and optimized using VA.

[0118] In step S104, the roadheader is controlled to perform the target task provided that the roadheader moves to the target position.

[0119] It should be noted that the roadheader is equipped with a hydraulic motor for performing general movement of the roadheader, a cutting mechanism for cutting out the area of the cutting surface, and a material collecting mechanism for collecting materials and placing them in the material laying area. When the roadheader is driven by the hydraulic motor to move to a target position, the material collection mechanism of the roadheader is controlled to collect materials for placement in the material laying area.

[0120] In this case, when the target task is the task of collecting material to be placed in the material stacking area of the tunnel, and the target position is the position in the material stacking area of the tunnel, after the roadheader is moved to the target position, the material collecting mechanism collects materials laid down in the material stowage area until the height of the laid materials is less than the corresponding threshold value; and when the target task is a punching task for a punching surface area, and the target position is a position of a punching surface area, after the roadheader is moved to the target position, the punching mechanism punches out the punching surface area. In this case, when it is necessary to punch out the edge of the punching surface area, the punching mechanism punches out the edge of the punching surface area until the punching surface area has an edge forming degree value that is less than the corresponding threshold value; and when it is necessary to punch out the punching surface of the punching surface area, the punching mechanism punches the punching surface of the punching surface area until the punching surface of the punching surface area has a leveling degree value that is less than a corresponding threshold value.

[0121] In step S105, the roadheader is controlled according to the environmental map data to move from a target position to a home position along a moving path provided that the roadheader completes the target task.

[0122] After the roadheader has completed collecting materials for the material laying area, it can be moved from the target position to the starting position by directing the roadheader in the opposite direction to move according to the navigation path. In this case, the real-time position of the roadheader during its movement to the starting position is obtained in real time by additionally collecting in real time characteristic points of the roadheader's environment during its movement through cameras located on the roadheader and matching the obtained features environment with a map of characteristic points that make up the movement trajectory.

[0123] This document provides a method for controlling a roadheader based on binocular vision. Unlike the prior art, in the present document, state information corresponding to the cutting surface area and state information corresponding to the material laying area are determined by binocular vision technology, then the target task to be performed by the roadheader and the target position corresponding to this target task is determined according to the state information that corresponds to the cutting surface area and the state information that corresponds to the material laying area, then guides the roadheader from the starting position to the target position that corresponds to the cutting surface area or the material laying area, tracking by binocular vision, thereby achieving the identification and positioning of the target task, which corresponds to the cutting surface area or material laying area located in the tunnel, using binocular vision technology, then directing the roadheader to move to the target position, so that the roadheader performs the target task , and after the target task is completed, the roadheader is controlled according to map data about the environment of the roadheader to move it from a target position to a home position along a moving path.

[0124] It should be understood that the order of the steps in the above embodiments does not imply the order of execution, but the order of execution of each process should be determined by its function and inherent logic, without constituting a limitation on the implementation of the process according to these embodiments of the present invention.

[0125] In one embodiment, a device is provided for controlling a roadheader based on binocular vision, as shown in FIG. 6, wherein the material collection system contains a positioning module based on binocular vision, a navigation module and a control module. Each functional module is described in detail below.

[0126] A state information acquiring unit 61 for obtaining state information that corresponds to a cutting surface area and state information that corresponds to a material laying area according to binocular vision images in the tunnel.

[0127] A target task determination unit 62 for determining a target task to be performed by the roadheader and a target position that corresponds to the target task according to state information corresponding to the cutting surface area and state information corresponding to the material laying area.

[0128] A navigation module 63 for guiding the roadheader according to tracking its movement through binocular vision to move from a starting position to a target position and mapping map data about the environment and the trajectory of the roadheader as it moves.

[0129] A first control module 64 for controlling the roadheader when it performs a target task, provided that the roadheader moves to a target position.

[0130] A second control module 65 for controlling the roadheader according to the mapped environment data to move it from a target position to a home position along its moving path provided that the roadheader completes the target task.

[0131] In addition, the status information includes edge formation degree information, leveling degree information, first position information in the cutting surface area, height information, and second position information in the material laying area, wherein the receiving unit 61 status information contains:

[0132] a point cloud data acquiring unit for obtaining point cloud information of a face according to binocular vision images in a tunnel;

[0133] a separating unit for separating the cutting surface area from the material laying area according to the face point cloud information;

[0134] a first information determining unit for determining edge formation degree information, leveling degree information, and first position information in the punching surface area according to the point cloud data in the punching surface area; And

[0135] a second information determining unit for determining height information and second position information in the material laying area according to the point cloud data in the material laying area;

[0136] wherein the target task definition module 62 contains:

[0137] a target task determination unit for determining a target task to be performed by the roadheader according to information about the degree of formation of edges, information about the leveling degree of the cutting surface area, and information about the height of the material laying area; wherein the target task comprises at least one of a punching task for the punching surface area and a material collection task for the material laying area; And

[0138] a target position determining unit for determining a target position according to the first position information and/or the second position information corresponding to the target task.

[0139] In addition, the point cloud data acquisition block contains:

[0140] a dense parallax map acquisition subunit for obtaining a dense parallax map of the face according to binocular vision images in the tunnel and

[0141] a first point cloud data obtaining subunit for calculating the spatial coordinates of each point cloud data on the dense parallax map to obtain information about the face point cloud on the dense parallax map;

[0142] the split block contains:

[0143] a second point cloud data obtaining subunit for calculating a spatial plane corresponding to point cloud information in a cutting surface region, said calculation being performed according to the point cloud information of a cutting surface on a dense parallax map, and determining point cloud data corresponding to a cutting surface region ;

[0144] an initial point set obtaining subunit for use as an initial point set of point cloud data whose distance from the spatial plane corresponding to the cutting surface area is less than a threshold value;

[0145] a depth map obtaining subunit for projecting point cloud data within the original point set onto a spatial plane corresponding to the cutting surface area to obtain a depth map;

[0146] a semantic segmentation model training subunit for training a semantic segmentation model based on map label data collected from the depth map and

[0147] A partitioning subunit for separating the cutting surface area from the material placement area in the depth map based on a semantic segmentation model.

[0148] In addition, the first information definition block contains:

[0149] a first actual spatial point cloud data acquisition unit for projecting point cloud data corresponding to the cutting surface area onto an actual physical coordinate system to obtain actual spatial point cloud data in the cutting surface area, and

[0150] a first position information acquiring unit for obtaining information about a first position corresponding to the cutting surface area according to the actual spatial point cloud data in the cutting surface area;

[0151] the second information definition block contains:

[0152] a second actual spatial point cloud data acquisition unit for projecting point cloud data corresponding to the material laying area onto an actual physical coordinate system to obtain actual spatial point cloud data in the material laying area, and

[0153] a second position information obtaining unit for obtaining information about a second position corresponding to the material laying area according to the actual spatial point cloud data in the material laying area.

[0154] In addition, the navigation module 63 includes:

[0155] a real-time point cloud data acquisition unit for obtaining real-time point cloud data in a cutting surface area through binocular cameras attached to both sides of the roadheader; And

[0156] a motion tracking unit for calculating the relative position between the roadheader and the cutting surface area according to real-time point cloud data in the material laying area to achieve motion tracking of the roadheader.

[0157] In addition, the navigation module 63 includes:

[0158] a roadheader information obtaining unit for obtaining a body area and an initial rotation angle of the roadheader, wherein the body area is a covering area of the body of the roadheader;

[0159] a virtual area setting unit for specifying a virtual area that moves with the roadheader according to the area of the body of the roadheader;

[0160] a navigation unit for guiding the roadheader to move directly in the direction of the original rotation angle according to the movement tracking of the virtual area through binocular vision, and when the virtual area touches the side wall, guiding the roadheader to move it by rotating until the roadheader will move to the target position;

[0161] a motion path mapping unit for mapping in real time the path of motion of the roadheader from a starting position to a target position according to a straight path along which the roadheader is directed to move directly in the direction of the starting angle of rotation, and a rotation path along which the roadheader moves rotation; And

[0162] an environmental mapping data mapping unit for mapping real-time environmental mapping data of a road miner while it is moving from a starting position to a target position according to binocular vision images of the road miner while it is moving obtained by binocular vision.

[0163] In addition, the navigation block contains:

[0164] a subunit for pre-determining the number of turning points in a path for pre-determining the number of turning points in a path of movement of the roadheader from a starting position to a target position; And

[0165] A rotation angle determining unit for determining a rotation angle that guides one rotational movement of the roadheader according to the initial rotation angle and the number of rotation points in the path under the condition that the virtual region touches the side wall.

[0166] Herein, a device for controlling a mining machine based on binocular vision is proposed. Compared with the prior art, in the present document, the state information corresponding to the cutting surface area and the state information corresponding to the material laying area are determined through binocular vision technology, then the target task to be performed by the roadheader, and the target position corresponding to the target task are determined according to the state information corresponding to the cutting surface area and the state information corresponding to the material laying area, then the roadheader is guided from the starting position to the target position corresponding to the cutting surface area or the material laying area while tracking through binocular vision, and In this way, identification and positioning of the target task corresponding to the cutting surface area or material laying area located in the tunnel is achieved through binocular vision technology, then the roadheader is directed to move to the target position, so that the roadheader performs the target task, and after the target task is completed, the roadheader is controlled according to the map data about the environment of the roadheader to move it from the target position to the home position along the moving path.

[0167] Regarding the specific limitations on the device for controlling a road miner based on binocular vision, reference may be made to the limitations on the method for controlling a road miner based on binocular vision in the previous content, which will not be repeated here. The modules of the above-mentioned device for controlling a mining machine based on binocular vision can be fully or partially implemented using software, hardware, and combinations of the above. Each of the modules described above may be built into or independent of a processor in a computing device in the form of hardware, or may be stored in a memory of the computing device in the form of software such that the processor can retrieve the software to perform operations corresponding to each of modules described above.

[0168] In one embodiment, a computing device is provided, which may be a back end, and its internal structure may be as shown in FIG. 7. The computing device contains a processor, a storage device, a network interface and a database connected to each other via a system bus. The processor of the computing device is used to provide computing and control functions. A storage device of a computing device includes a non-transitory and/or non-transitory storage medium and an internal storage device. Durable storage media stores the operating system, computer program, and database. Internal storage provides an environment for running an operating system and a computer program in a durable storage medium. The network interface of a computing device is used to exchange data with an external client via a network connection. The computer program is executed by a processor to implement the functions or steps of the server part of a method for controlling a roadheader based on binocular vision.

[0169] In one embodiment, a computing device is provided that may be a client, and its internal structure may be as shown in FIG. 8. A computing device includes a processor, a storage device, a network interface, a display screen and an input device connected to each other via a system bus. The processor of the computing device is used to perform computing and control functions. A storage device in a computing device includes a non-transitory storage medium and an internal storage device. Durable storage media stores the operating system and computer program. Internal storage provides an environment for running an operating system and a computer program in a durable storage medium. The network interface of a computing device is used to exchange data with an external server through a network connection. The computer program is executed by the processor to implement the functions or stages of the client part of the method for controlling a roadheader based on binocular vision.

[0170] In one embodiment, a computing device is provided that includes a memory device, a processor, and a computer program stored in the memory device and executed by the processor, which when executed by the processor implements the following steps:

[0171] obtaining state information corresponding to the cutting surface area and state information corresponding to the material laying area according to the binocular vision image in the tunnel;

[0172] determining a target task to be performed by the roadheader and a target position corresponding to the specified target task according to state information that corresponds to the cutting surface area and state information that corresponds to the material laying area;

[0173] guiding the roadheader according to tracking its movement through binocular vision to move it from a starting position to a target position, and mapping map data of the environment and the moving path of the roadheader while it is moving;

[0174] controlling the roadheader to perform a target task, provided that the roadheader moves to the target position; And

[0175] controlling the roadheader according to the mapped environment data to move from a target position to a home position along a movement path provided that the roadheader completes the target task.

[0176] In one embodiment, another computer-readable storage medium is provided that stores a computer program that, when executed by a processor, performs the following steps:

[0177] obtaining state information corresponding to the cutting surface area and state information corresponding to the material laying area according to binocular vision images in the tunnel;

[0178] determining a target task to be performed by the roadheader and a target position corresponding to this target task according to state information corresponding to the cutting surface area and state information corresponding to the material laying area;

[0179] guiding the roadheader according to tracking its movement through binocular vision to move it from a starting position to a target position, and mapping map data about the environment and the moving path of the roadheader while it is moving;

[0180] controlling the roadheader to perform a target task, provided that the roadheader moves to the target position; And

[0181] controlling the roadheader according to the mapped environment data to move it from a target position to a home position along a moving path, provided that the roadheader completes the target task.

[0182] It should be noted that with respect to the above-mentioned functions or steps that can be performed by a computer-readable storage medium or computing device, reference may be made to the corresponding description of the server part and the client part in the previous embodiments of the method, which will not be described in this case, so as not to repeat myself.

[0183] One skilled in the art may appreciate that a given embodiment of all or part of the processes in the methods of the embodiments described above may be implemented by a computer program instructing associated hardware, and the computer program may be stored in a non-transitory computer-readable storage medium, and wherein the computer program, when executed, may include processes of embodiments of the methods described above. Any reference to a memory device, storage device, database, or other media used in the embodiments presented herein may include long-term and/or short-term storage. The long-term storage device may include read-only memory (ROM), programmable memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), or flash memory. The short-term storage device may include random access memory (RAM) or external cache memory. By way of illustration, and not limitation, random access memory (RAM) may be available in a variety of forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic memory (DRAM), SDRAM, Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), Memory Rambus direct random access memory (RDRAM), Rambus direct dynamic random access memory (DRDRAM), Rambus direct random access memory (RDRAM), etc.

[0184] One skilled in the art may clearly understand that for convenience and brevity of description, the above-mentioned division of each functional block and module is given as an example only. In actual application cases, the above functions can be assigned to be performed by different function blocks and modules according to the needs, i.e. The internal structure of the device is divided into various functional blocks or modules to implement all or part of the functions described above.

[0185] The embodiments described above are used only to illustrate the technical solutions of the present invention, but not to limit them. Although the present application has been described in detail with reference to the above embodiments, one skilled in the art will appreciate that the technical solutions in the above embodiments may, however, be modified, or equivalent substitutions may be made in some of the technical features, contained in these embodiments; and these changes or substitutions do not cause the essence of the corresponding technical solutions to deviate from the idea and scope of protection of these technical solutions contained in these embodiments described herein, but shall be included in the scope of protection of the present invention.

Claims (52)

1. A method for controlling a roadheader based on binocular vision, including: obtaining state information corresponding to the cutting surface area and state information corresponding to the material laying area according to binocular vision images in the tunnel; determining a target task to be performed by the roadheader and a target position corresponding to this target task according to state information corresponding to the cutting surface area and state information corresponding to the material laying area; directing the roadheader according to tracking its movement through binocular vision to move from a starting position to a target position and mapping the cartographic data of the environment and the moving path of the roadheader during its movement; controlling the roadheader to perform a target task, provided that the roadheader moves to the target position; And controlling the roadheader according to the mapping data of the environment to move it from a target position to a home position along a moving path, provided that the roadheader completes the target task. 2. The control method according to claim 1, wherein the state information includes edge formation degree information, leveling degree information and first position information of the cutting surface area, as well as height information and second position information of the material laying area, wherein obtaining information about the state corresponding to the area of the cutting surface, and information about the state corresponding to the area of laying the material, according to binocular vision images in the tunnel includes: obtaining information about the cloud of face points according to binocular vision images in the tunnel; separating the area of the cutting surface from the area of laying the material according to information about the cloud of points of the face; determining information about the degree of formation of edges, information about the degree of alignment and information about the first position of the cutting surface area according to the point cloud data in the cutting surface area; determining height information and second position information of the material laying area according to the point cloud data in the material laying area; Determining the target task to be performed by the roadheader and the target position corresponding to this target task, according to the state information corresponding to the cutting surface area, and the state information corresponding to the material laying area, includes: determining the target task to be performed by the roadheader, according to information about the degree of formation of edges, information about the degree of leveling of the cutting surface area and information about the height of the material laying area; wherein the target task comprises at least one of a punching task for the punching surface area and a material collection task for the material laying area; And determining a target position according to the first position information and/or the second position information corresponding to the target task. 3. The control method according to claim 2, according to which obtaining information about the cloud of face points according to binocular vision images in the tunnel includes: obtaining a dense parallax map of the face according to binocular vision images in the tunnel; calculating the spatial coordinates of each point cloud data on the dense parallax map to obtain information about the face point cloud on the dense parallax map; separation of the cutting surface area from the material laying area according to information about the cloud of face points, including: calculating, according to the point cloud information of the cutting face on the dense parallax map, a spatial plane corresponding to the point cloud information in the cutting surface area, and determining point cloud data corresponding to the cutting surface area; using as the initial point set of the specified data a cloud of points, the distance of which from the spatial plane corresponding to the area of the cutting surface is less than a threshold value; projecting point cloud data within the original point set onto a spatial plane corresponding to the surface area of the cutting to obtain a depth map; training a semantic segmentation model based on map label data collected from a depth map; And separation of the cutting surface area from the material laying area on the depth map based on a semantic segmentation model. 4. The control method according to claim 3, according to which determining information about the degree of formation of edges, information about the degree of alignment and information about the first position of the cutting surface area according to the point cloud data in the cutting surface area includes: projecting point cloud data corresponding to the cutting surface area onto a system of actual physical coordinates to obtain cloud data of actual spatial points in the cutting surface area; obtaining information about the first position corresponding to the area of the cutting surface, according to the data of the cloud of actual spatial points in the area of the cutting surface; determining the height information and the second position information of the material laying area according to the point cloud data of the material laying area, including: projecting point cloud data corresponding to the material laying area onto a system of actual physical coordinates to obtain cloud data of actual spatial points of the material laying area; And obtaining information about the second position corresponding to the material laying area, according to the data of the cloud of actual spatial points in the material laying area. 5. The control method according to claim 3, according to which the direction of the roadheader according to tracking its movement through binocular vision to move it from the starting position to the target position includes: obtaining point cloud data in real time in the area of the cutting surface using binocular cameras attached to both sides of the roadheader; And Calculating the relative position between the roadheader and the cutting surface area according to real-time point cloud data in the material laying area to achieve tracking of the roadheader's movement. 6. The control method according to claim 1, according to which the direction of the roadheader according to tracking its movement through binocular vision to move it from a starting position to a target position and mapping cartographic data about the environment and the trajectory of movement of the roadheader during its movement includes: obtaining the body area and the initial rotation angle of the roadheader, wherein the body area is the covering area of the body of the roadheader; setting a virtual area that moves with the roadheader, according to the area of the roadheader body; directing the roadheader to move directly in the direction of the starting rotation angle according to the movement tracking of the virtual area by binocular vision, and directing the roadheader when the virtual area touches the side wall to move it by rotation until the roadheader moves to the target position; real-time mapping of a moving path of the roadheader from a starting position to a target position according to a straight path along which the roadheader is directed to move directly in the direction of the starting angle of rotation, and a rotation path along which the roadheader is moved by rotation; And mapping real-time cartographic data about the environment of the roadheader as it moves from a starting position to a target position according to the binocular vision images of the roadheader during its movement obtained through binocular vision. 7. The control method according to claim 6, wherein the direction of the roadheader to move directly in the direction of the initial rotation angle and the direction of the roadheader when the virtual area touches the side wall to move it by rotation according to tracking the movement of the virtual area through binocular vision until until the roadheader moves to the target position where the material placement area is located, includes: preliminary determination of the number of turning points in the trajectory of movement of the roadheader from the starting position to the target position; And determining the rotation angle that guides one rotational movement of the roadheader, according to the initial rotation angle and the number of rotation points in the trajectory, provided that the virtual area touches the side wall. 8. A device for controlling a roadheader based on binocular vision, containing: a state information acquiring module for obtaining state information corresponding to a cutting surface area and state information corresponding to a material laying area according to binocular vision images in the tunnel; a target task determination module for determining a target task to be performed by the roadheader and a target position corresponding to the target task according to state information corresponding to the cutting surface area and state information corresponding to the material laying area; a navigation module for guiding the roadheader according to tracking its movement through binocular vision to move it from a starting position to a target position and mapping cartographic data about the environment and the moving path of the roadheader while it is moving; a first control module for controlling the roadheader to perform a target task provided that the roadheader moves to a target position; And a second control module for controlling the roadheader according to the mapped environment data to move it from a target position to a home position along a moving path provided that the roadheader completes the target task. 9. Computing device containing: one or more processors; one or more storage devices; And one or more programs stored in a storage device and configured to be executed by one or more processors, wherein the program contains instructions for implementing a method for controlling a roadheader based on binocular vision according to any one of claims. 1-7. 10. A computer-readable storage medium containing a computer program for use in combination with a navigation device in a vehicle, wherein the computer program is executable by a processor for implementing the binocular vision-based method of controlling a roadheader according to any one of claims. 1-7.