WO2023056720A1

WO2023056720A1 - Visual boundary detection-based robot control system and method

Info

Publication number: WO2023056720A1
Application number: PCT/CN2022/070672
Authority: WO
Inventors: 张伟; 吴一飞; 鲍鑫亮; 陈越凡; 申中一
Original assignee: 邦鼓思电子科技（上海）有限公司
Priority date: 2021-10-09
Filing date: 2022-01-07
Publication date: 2023-04-13
Also published as: CN113910225A

Abstract

A visual boundary detection-based robot control system, comprising: an image acquisition module (110), a visual boundary recognition module (120), and a movement control module (130); the image acquisition module (110) acquires image information of the surrounding environment of a robot in real time; the visual boundary recognition module (120) recognizes the working boundary of a working region according to the image information acquired by the image acquisition module (110), and forms corresponding working region boundary information; the movement control module (130) performs calculation according to the working region boundary information of the working region recognized by the visual boundary recognition module (120), determines whether the robot reaches the boundary of the working region, if the robot reaches the boundary of the working region, adjusts the working state of the robot to prevent the robot from crossing the boundary of the working region, and if the robot does not reach the boundary of the working region, controls the robot to maintain the current working state. The robot control system can be well integrated with a visual boundary detection scheme, for an outdoor working region without any preset physical boundary, the corresponding working boundary of the outdoor working region can be precisely recognized, and after it is detected that a device reaches the boundary of the working region, a subsequent forward route is precisely adjusted. Also provided is a visual boundary detection-based robot control method.

Description

A robot control system and method based on visual boundary detection

technical field

The invention relates to the technical field of robot control, in particular to a control technology based on visual boundary detection.

Background technique

With the progress of science and the development of technology, more and more automated mobile robots appear in people's daily life. These automatic mobile robots rely on their own control system to complete the related tasks set by people in a fixed area without human operation and intervention. The existing mainstream control technology includes collision control based on collision sensors.

In indoor mobile robots, due to the existence of natural physical barriers such as walls, collision-based control is mostly used. When the robot hits an obstacle, the collision sensor sends a signal to control the direction of the robot. However, in an outdoor mobile robot, the collision sensor does not work when it reaches the boundary of the work area due to the absence of physical obstacles such as indoor walls.

According to the characteristics of the outdoor environment, people have designed a visual boundary detection scheme to perceive the surrounding environment; to detect whether the current robot has reached the boundary of the working area, but once the boundary detection signal is triggered, if the robot continues to move forward, it will cross the boundary, so Serious accidents may result.

However, the existing control scheme cannot be well integrated with the visual boundary detection scheme, so that the robot based on visual boundary detection can accurately adjust the subsequent forward route after detecting the boundary of the working area.

It can be seen that there is an urgent need in this field for a robot control scheme based on visual boundary detection.

Contents of the invention

Aiming at the problems existing in the mobile control scheme of the existing outdoor automatic mobile equipment based on visual detection of area boundaries, the purpose of the present invention is to provide a robot control system based on visual boundary detection and a corresponding control method to realize After the robot detects the boundary of the work area, it can make precise adjustments to the subsequent forward route based on the detected convenience.

In order to achieve the above object, the robot control system based on visual boundary detection provided by the present invention includes: an image acquisition module, a visual boundary recognition module and a movement control module;

The image acquisition module obtains image information of the surrounding environment of the robot in real time;

The visual boundary recognition module recognizes the working boundary of the working area according to the image information collected by the image collection module;

The movement control module calculates and judges whether the mower reaches the boundary of the working area according to the working boundary information of the working area identified by the visual boundary recognition module, and if it reaches the boundary of the working area, adjusts the working state of the robot to prevent the robot from crossing the boundary of the working area; If the boundary of the working area is not reached, the robot is controlled to maintain the current working state.

Further, the visual boundary identification module identifies the boundary of the working area based on a deep neural network and an image processing method.

Further, the visual boundary recognition module segments the obtained image of the surrounding environment of the robot based on the deep neural network to obtain a corresponding neural network segmentation map, and forms a corresponding workable area and an unworkable area in the neural network segmentation map. The boundaries between them form the working area boundaries.

Further, the working area boundary information includes working area boundary contour information.

Further, the movement control module obtains the boundary contour information of the working area identified by the visual boundary recognition module, and some contour point information corresponding to the contour of the robot, and calculates the relative relationship between several contour points and the boundary contour of the working area accordingly , judge whether the robot advances to the boundary position of the working area according to the calculation result, and control the robot to adjust the forward direction according to the working mode and the boundary information of the working area when the robot advances to the boundary of the working area.

In order to achieve the above object, the robot control method based on visual boundary detection provided by the present invention includes:

Real-time acquisition of image information of the surrounding environment of the device;

Identify the natural working boundary of the working area according to the collected image information, and generate recognizable working area boundary information from the natural working boundary;

Based on the generated working area boundary information, calculate and judge whether the lawn mower has reached the working area boundary. If it reaches the working area boundary, adjust the working state of the robot to prevent the robot from crossing the working area boundary; if it does not reach the working area boundary, control the robot to keep the current work state.

Further, the method generates identifiable working area boundary information based on a deep neural network and an image processing algorithm.

Further, the method is based on a deep neural network to segment the acquired image of the surrounding environment of the robot to obtain a corresponding neural network segmentation map, and forms a corresponding workable area and an unworkable area in the neural network segmentation map. The boundaries of form the working area boundary.

Further, the method acquires the identified working area boundary contour information in real time, as well as some contour point information corresponding to the robot contour, and calculates the relative relationship between several contour points and the working area boundary contour based on this, and judges according to the calculation results Whether the robot advances to the boundary of the working area, and when the robot reaches the boundary of the working area, control the robot to adjust the forward direction according to the working mode and the boundary information of the working area.

The solution provided by the present invention can be well integrated with the visual boundary detection solution. For outdoor work areas without any preset physical boundaries, it can accurately identify the corresponding work boundaries of the outdoor work area, and detect that the equipment reaches the boundary of the work area After that, make precise adjustments to the follow-up route.

The solution provided by the present invention can accurately identify the relative position between the equipment and the boundary of the working area, and can accurately control the forward route of the equipment when the equipment reaches the boundary of the working area, avoiding the equipment from crossing the boundary of the working area, and improving the reliability and safety of automatic operation of the equipment sex.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is the composition principle diagram of the robot control system based on visual boundary detection in the example of the present invention;

Fig. 2 is an example diagram of the working route of the robot on the lawn without any preset boundary marks in the example of the present invention;

Fig. 3 is an example diagram of a visual perception picture obtained at point B of the boundary in an example of the present invention;

Fig. 4 is an example rendering of the neural network segmentation diagram formed for the visual perception picture shown in Fig. 3 in the example of the present invention;

Fig. 5 is an example diagram of a visual perception picture obtained by turning in an example of the present invention;

Fig. 6 is an example rendering of the neural network segmentation diagram formed for the visual perception picture shown in Fig. 5 in the example of the present invention;

Fig. 7 is an example diagram of a visual perception picture obtained in mode 1 in the example of the present invention;

Fig. 8 is an example rendering of the neural network segmentation diagram formed for the visual perception picture shown in Fig. 7 in the example of the present invention;

Fig. 9 is an example rendering of a device out of bounds in an example of the present invention;

Fig. 10 is an example rendering of the device not crossing the boundary in the example of the present invention;

Fig. 11 is an example rendering of a device partially out of bounds in an example of the present invention.

Detailed ways

In order to make the technical means, creative features, goals and effects achieved by the present invention easy to understand, the present invention will be further described below in conjunction with specific illustrations.

The outdoor working area, because it does not have a specific boundary limiter such as an indoor wall, brings a lot of problems to the outdoor automatic operation equipment. In order to well limit the outdoor automatic operation equipment in the outdoor working area, the conventional method presets the corresponding physical boundary for the boundary of the outdoor working area, and cooperates with the sensing technology of the outdoor automatic operation equipment to realize the equipment Confined within the intended work area. As an example, the physical boundaries here are pre-buried boundary copper wires and the like.

This scheme abandons the outdoor automatic operation equipment control scheme that requires artificially preset corresponding physical boundaries at the boundary of the outdoor work area, and proposes an outdoor automatic operation equipment control scheme based on visual boundary detection, which can realize the control of outdoor automatic operation equipment without presetting any physical boundaries. Under certain circumstances, accurately identify the working boundary corresponding to the outdoor working area in the outdoor working area, and after detecting that the equipment reaches the working area boundary, make precise adjustments to the subsequent forward route to prevent the equipment from crossing the outdoor working area boundary and ensure outdoor automatic operation The reliability and safety of equipment work.

The control scheme of outdoor automatic operation equipment based on visual boundary detection given in this scheme obtains the surrounding environment image information of the equipment (that is, outdoor automatic operation equipment) in real time;

Based on the generated boundary information of the working area, it is judged whether the mower has reached the boundary. If it reaches the boundary of the working area, the working state of the robot is adjusted to prevent the robot from crossing the boundary; if it does not reach the boundary of the working area, the robot is controlled to maintain the current working state and continue to move forward. Work.

The working status here includes, but is not limited to, the forward direction and forward speed of the robot.

In this solution, when acquiring the image information of the surrounding environment of the equipment, the boundary of the working area is identified based on the deep neural network and image processing algorithm to form the corresponding boundary information of the working area.

When generating the identifiable boundary information of the working area in this solution, based on the deep neural network, the acquired surrounding environment image of the equipment is segmented to obtain the corresponding neural network segmentation map.

The corresponding workable area and non-workable area are formed in the neural network segmentation diagram, and the boundary between the two is the working area boundary.

As an example, two different colors can be used to represent the workable area and the non-workable area, and the boundary of the two colors is the boundary of the natural workable area.

As shown in Figure 4, it shows the neural network segmentation diagram obtained by performing neural network segmentation on the surrounding environment image of the equipment shown in Figure 3. The blue area in the figure indicates the working area, and the gray area indicates the non-working area. The boundaries between the color areas are the boundaries of the natural working area.

On the basis, the boundary information of the working area is further calculated, and the boundary information of the working area here includes boundary contour information and distance information between the boundary and the equipment.

It should be noted here that the boundary information of the working area can also be calculated to obtain other information as required.

When making precise adjustments to the forward route, this solution obtains the results of visual recognition and detection of the boundary of the working area in real time, calculates and judges whether the equipment has advanced to the boundary of the working area based on the obtained results, and when the equipment advances to the boundary of the working area, timely Adjust the working status of the equipment to avoid the equipment crossing the working area boundary.

Further, this solution obtains the boundary information through the results of visual recognition and detection of the boundary of the working area. The boundary information here is mainly the outline information of the working area boundary; and based on the outline information of the working area boundary, the parameters of the image acquisition device on the equipment Calculates the distance of the device relative to the work area boundary.

On the basis, combined with the obtained working area outline information and the distance information of the equipment relative to the working area boundary, it can be judged in real time whether the equipment has reached the boundary, and when it reaches the boundary, it can be adjusted according to the obtained working area boundary outline information.

Regarding the above-mentioned method of determining whether the equipment is relative to the boundary of the working area, this solution also provides a method of directly judging whether the equipment reaches the boundary of the working area based on the contour information of the boundary of the working area.

As an example, in this way, this solution obtains boundary information through the results of visual recognition and detection of the boundary of the working area. The boundary information here is specifically the contour information of the boundary of the working area; at the same time, the visual recognition and detection of the boundary of the working area is carried out for the surrounding environment images of the equipment. , the generated neural network segmentation diagram, select the corresponding contour point in the neural network segmentation diagram, and determine the position information of the contour point; then combine the position information of the contour point with the contour information based on the determined working area boundary to calculate and judge Whether the position of the contour point satisfies the corresponding threshold value is used to determine whether the equipment has reached the boundary of the working area. When reaching the boundary, it is adjusted according to the obtained boundary contour information of the working area.

In view of the above-mentioned method of determining whether the equipment is relative to the boundary of the working area, this solution also provides a method of directly judging whether the equipment reaches the boundary of the working area based on the area of the working area.

As an example, in this way, this solution obtains boundary information through the results of visual recognition and detection of the boundary of the working area. The boundary information here is specifically the outline information of the boundary of the working area; the neural network segmentation is determined based on the outline information of the boundary of the working area. The area of the working area in the figure, the neural network segmentation diagram here is determined when the visual recognition and detection of the working area boundary is performed on the surrounding environment image of the equipment; then calculate and judge whether the determined working area area operates the corresponding threshold value, from this Determine whether the device has reached the boundary of the working area. When reaching the boundary, it is adjusted according to the obtained boundary contour information of the working area.

For the calculation and determination of the area of the working area in the neural network segmentation map, in this example, it is preferred to determine the area of the working area in the neural network segmentation map by counting the number of pixels in the working area in the corresponding neural network segmentation map, which can ensure the calculation of the area. Accuracy, but also to ensure the speed of calculation, to avoid excessive consumption of computing power of the processor.

Furthermore, when adjusting the working state of the equipment, this solution forms a control command based on the results of visual recognition and detection combined with the working mode set by the equipment, so as to control the equipment to adjust the moving state and continue to move forward in the working area.

The visual recognition and detection results of the boundary of the working area here are the boundary contour information determined based on the neural network segmentation map and the state information of the equipment relative to the boundary contour.

The status information of the device relative to the boundary contour here may be the distance information of the device relative to the boundary contour, the area information of the working area between the device and the boundary contour, and the like.

As an example, this solution can calculate the working area boundary visual recognition detection result based on the processed working area neural network segmentation map and working area boundary information, that is, obtain the corresponding boundary contour information and working area area information.

In order to ensure the accuracy of the adjustment when adjusting the working state, this scheme first calculates and determines the direction information of the working area relative to the current working state of the equipment based on the obtained boundary contour information and the area information of the working area; on this basis, the determined The direction information of the working area is fused with the preset working mode of the equipment to form an instruction for adjusting the equipment movement status, so as to realize precise adjustment of the equipment movement status.

This outdoor automatic operation equipment control scheme based on visual boundary detection, in specific applications, can constitute a corresponding software program, presented with a corresponding outdoor automatic operation equipment control system, and can run in the outdoor automatic operation equipment to realize the aforementioned based on Visual boundary detection is a scheme for realizing the working area of outdoor automatic working equipment.

Referring to Fig. 1, it shows the compositional system diagram of the outdoor automatic operation equipment control system based on visual boundary detection provided by this scheme.

The outdoor automatic operation equipment control system 100 based on visual boundary detection mainly includes three functional modules: an image acquisition module 110, a visual boundary recognition module 120 and a mobile control module 130. The control of automatic operation equipment is limited to the outdoor working area.

For the convenience of expression, the outdoor automatic operation equipment will be referred to as equipment for short.

The image acquisition module 110 in this system acquires the image information of the surrounding environment of the outdoor automatic operation equipment in real time.

The visual boundary identification module 120 in this system performs data interaction with the image acquisition module 110, and can identify the working boundary of the working area according to the image information collected by the image acquisition module.

Specifically, the visual boundary identification module 120 can identify the boundary of the working area based on the deep neural network and image processing algorithm based on the image information of the surrounding environment of the equipment acquired by the image acquisition module 110, so as to form corresponding boundary information of the working area.

When the visual boundary identification module 120 recognizes and detects and generates identifiable work area boundary information, it specifically segments the acquired equipment surrounding environment image based on a deep neural network to obtain a corresponding work area neural network segmentation map, and performs the work area neural network The corresponding workable area and non-workable area are formed in the segmentation map, and the boundary between the two is the working area boundary.

The movement control module 130 in this system interacts with the visual boundary recognition module 120 for data, and performs data interaction with the drive control part of the device.

The movement control module 130 can control the equipment to continue moving according to the working boundary information of the working area identified by the visual boundary identification module 120, and adjust the working state of the equipment according to the calculation result.

The working state here includes, but is not limited to: the forward direction, angle, moving distance, etc. of the equipment.

Specifically, the mobile control module 130 obtains the visual recognition detection result of the visual boundary recognition module 120 in real time, calculates and judges whether the equipment advances to the boundary position of the working area based on the obtained result, and when the equipment advances to the boundary position of the working area, Adjust the working status of the equipment in time to avoid the equipment from crossing the boundary of the working area.

The manner and process for the mobile control module 130 to specifically determine whether the device has reached the boundary of the working area are as described above, and will not be repeated here.

The movement control module 130 also forms a control instruction based on the result of the visual recognition detection by the visual boundary recognition module 120 and the working mode set by the device, so as to control the device to adjust the moving state and continue to move forward in the working area.

The movement control module 130 specifically controls the way and process of adjusting the movement state of the device when the device reaches the boundary of the working area, as described above, and will not be described here.

The resulting outdoor automatic operation equipment control system 100 based on visual boundary detection can directly run in the outdoor work area without any artificial boundary calibration or setting when running the corresponding outdoor operation equipment (such as a robot), which can automatically Detect the working boundary of the working area, and generate the working area boundary information recognizable by the machine from the natural boundary. When receiving the corresponding working area boundary information, it can carry out precise behavior control and make precise adjustments to the subsequent forward route to ensure The robot always works in the corresponding working area to prevent the risk caused by the robot crossing the boundary.

As an example, when the outdoor automatic operation equipment control system 100 is running in a corresponding outdoor operation robot, based on the size parameter data (such as width data, etc.) of the outdoor automatic operation equipment itself and the focal length parameter information of the image acquisition module carried, the Obtain the range of the drivable area in the real-time collected image, count the pixels in the drivable area to determine the area of the drivable area, and further judge the number of pixels in the statistical drivable area, if the conditions are met You can continue to move forward without reaching the boundary; otherwise, it is judged that the outdoor automatic operation equipment has reached the boundary.

Continuing with the above example, the following further illustrates the process of the outdoor automatic operation equipment control system 100 controlling the automatic operation of the outdoor operation robot in the working area.

The control system obtains the surrounding environment image information of the equipment through the image acquisition module carried on the equipment, and the environment image here is preferably the environment image in the forward direction of the equipment (as shown in Fig. 9a, Fig. 10a, Fig. 11a).

The visual boundary recognition module 120 in the system segments the acquired device surrounding environment image based on the deep neural network to obtain the corresponding neural network segmentation diagram (as shown in Fig. 9b, Fig. 10b, Fig. 11b).

The corresponding workable area and non-workable area are formed in the neural network segmentation diagram, and the boundary between the two is the working area boundary. Here, two different colors are used to indicate the workable area and the non-workable area, and the boundary of the two colors is the boundary of the natural working area.

Further combining the focal length parameter information of the image acquisition module carried by the device, the coordinates of each pixel in the neural network segmentation map in the visual imaging coordinate system are calculated.

On this basis, the distance between each pixel of the neural network segmentation map and the device can be further calculated, of course including the distance between the boundary line in the picture and the device.

After the neural network segmentation map processing of the surrounding environment image is completed (that is, the visual recognition detection is completed), based on the obtained results, it is judged whether the device reaches the boundary of the working area or whether it crosses the boundary.

For example, based on the obtained neural network segmentation map, it can be judged whether there is a working area. If there is no working area, it indicates that the head of the device has crossed the boundary.

Referring to Figure 9, the neural network segmentation map (Figure 9b) corresponding to the surrounding environment image shown in Figure 9a does not have any workable area pixels (blue), indicating that the head of the device has crossed the boundary as a whole. According to a certain working mode, control the driving control parts in the equipment to adjust the moving state of the equipment.

Based on the obtained neural network segmentation map, it is judged that it has a workable area, and then it is further judged whether the device has reached the boundary and whether the moving state needs to be adjusted.

First, according to the size parameter data of the device itself, the contour points corresponding to the device position are obtained in the neural network segmentation map, such as points A and B in Figure 10.

As an example, point A and point B here are determined based on the width parameters of the device, and the line between point A and point B corresponds to the width side of the front end of the device, so by determining the points A and B in the neural network segmentation diagram The specific location information can determine the location status information of the device relative to the boundary.

Based on the processing of the aforementioned neural network segmentation map, the coordinate information of the contour point A and the contour point B in the graph can be determined, and further based on the coordinate information of the contour point A and the contour point B, the state of the device relative to the boundary can be judged.

(1) By comparison at this time, if the contour point A and the contour point B are both in the working area of the neural network segmentation map, it means that the device has not crossed the boundary. At this time, the distance or neural network of the device relative to the boundary line of the working area can be further calculated. The area of the working area in the network segmentation diagram is used to judge whether the device reaches the boundary. If it does not reach the boundary and there is enough moving area in the front working area, the control device will continue to move while maintaining the current moving state; if it reaches the boundary, the front working area If there is not enough moving area, the control device will adjust the current moving state and continue moving.

Referring to FIG. 10 , in the neural network segmentation map corresponding to the surrounding image shown in FIG. 10 a (as shown in FIG. 10 b ), it is determined that the contour point A and the contour point B in the figure correspond to the current position of the device.

Based on the processing of the aforementioned neural network segmentation graph, the coordinate information of the contour point A and the contour point B in the graph can be determined. At the same time, in the neural network segmentation diagram shown in FIG. 10 b , the coordinate information of points C and D corresponding to the outermost points of contour point A and contour point B in the working area in the diagram is obtained.

By comparing the coordinate information of point C and point D with the coordinate information of contour point A and contour point B, it is determined whether the device is out of bounds.

Take the content shown in Figure 10 as an example, get the value of the leftmost coordinate in the working area in the split map, that is, the coordinate value of point C in the picture, and get the value of the rightmost coordinate of the working area in the split map, that is, point D in the picture coordinate value. By comparing and calculating the coordinate values of point C and point A, if it is determined that point C is on the left of point A, then it is determined that there is still a working area on the left side of the equipment, and the equipment does not cross the boundary in the left area. At the same time, it is calculated by comparing the coordinates of point D with the coordinate information of point B, and the coordinates of point D are on the right side of the coordinates of point B, so it is determined that there is a lawn on the right side of the device, and the area on the right side of the device is not out of bounds.

On this basis, it can be further judged whether the front of the equipment reaches the boundary of the working area, and/or whether there is enough moving area, so as to control whether the equipment needs to adjust the moving state.

(2) At this time, by comparison, if only one of the contour points A and B is within the working area of the neural network segmentation map, it means that some areas of the device have crossed the boundary, and the current moving state needs to be adjusted to continue move.

Referring to FIG. 11 , in the neural network segmentation map corresponding to the surrounding image shown in FIG. 11 a (as shown in FIG. 11 b ), it is determined that the contour point A and the contour point B in the figure correspond to the current position of the device.

Based on the processing of the aforementioned neural network segmentation graph, the coordinate information of the contour point A and the contour point B in the graph can be determined. At the same time, in the neural network segmentation diagram shown in FIG. 11 b , the coordinate information of points C and D corresponding to the outermost points of contour point A and contour point B in the working area in the diagram is obtained.

Take the content shown in Figure 11 as an example, obtain the value of the leftmost coordinate in the working area in the split graph, that is, the coordinate value of point C in the graph, and obtain the value of the rightmost coordinate of the working area in the split graph, that is, point D in the graph coordinate value. By comparing and calculating the coordinate values of point C and point A, if it is determined that point C is on the right side of point A, then it is determined that there is no working area on the left side of the equipment, and the area on the left side of the equipment has crossed the boundary. At the same time, it is calculated by comparing the coordinates of point D with the coordinate information of point B, and the coordinates of point D are on the right side of the coordinates of point B, so it is determined that there is a lawn on the right side of the device, and the area on the right side of the device is not out of bounds.

Based on this, it is judged that the left side of the device has crossed the border, and the right side has a working area, and if there is no border crossing, the device is controlled to adjust the moving state so that it moves towards the current right area.

It can be seen from the above that the outdoor automatic operation equipment control system can integrate the size parameter data of the outdoor automatic operation equipment itself and the focal length parameter information of the mounted image acquisition module to complete the precise automatic operation of the outdoor automatic operation equipment in specific applications. control.

In order to make the purpose of this solution, the technical solution and a bit clearer, the following takes the lawn robot to work on a specific outdoor lawn as an example to demonstrate the solution provided by the present invention.

In this example scheme, the outdoor automatic operation equipment control system based on visual boundary detection provided by this scheme is run in the lawn robot, and a corresponding visual boundary sensor is formed in the lawn robot.

This example is aimed at the outdoor specific lawn, without any artificial boundary calibration or setting.

According to this setting, the lawn robot 200 works on the specific outdoor lawn. At the beginning, the lawn robot 200 is at any position in the map.

As shown in Figure 2, the initial position is A, and A is a random position in the image, moving in a random direction. For the convenience of display, it is assumed to be the direction from point A to point B shown in the figure.

When the mower reaches point B at the boundary of the working area, the visual boundary sensor in the lawn robot 200 will analyze according to the obtained visual perception picture, as shown in FIG. 3 .

Based on the visual boundary detection, the neural network segmentation diagram shown in Figure 4 is formed, and the working area information, non-working area information, and boundary contour information are obtained based on this, so that the recognized natural boundaries are generated into machine-recognizable working area boundary information ,As shown in Figure 4.

At this time, the visual boundary sensor in the lawn robot 200 judges whether the lawn robot 200 has reached the boundary based on the processed boundary information of the working area. For example, based on the neural network segmentation diagram shown in Figure 4, statistically analyze the lawn pixels in the drivable area in the diagram, and judge that the robot has not reached the working boundary according to the statistical results, and the control system continues to control the lawn robot to work forward based on the current driving state.

When the lawn mower reaches the boundary B of the working area, the visual boundary sensor in the lawn robot 200 will analyze according to the obtained visual perception picture, as shown in FIG. 5 .

Based on the visual boundary detection, the neural network segmentation diagram shown in Figure 6 is formed, and the working area information, non-working area information, and boundary contour information are obtained based on this, so that the recognized natural boundaries are generated into machine-recognizable working area boundary information ,As shown in Figure 6.

In this state, the visual boundary sensor in the lawn robot 200 judges whether the lawn robot 200 has reached the boundary based on the processed boundary information of the working area. For example, based on the neural network segmentation diagram shown in Figure 6, statistically analyze the lawn pixels in the drivable area, and determine that the lawn pixels in the drivable area do not meet the conditions according to the statistical results, and the left body of the lawn robot 200 will cross the boundary if it continues to move forward. At this time, the visual boundary sensor in the lawn robot 200 sends a control adjustment signal to the drive control system in the lawn robot 200, and the control system starts to adjust the forward direction of the lawn mower, turning as needed. At the same time, according to the statistical distribution of lawn pixels in the drivable area, it is determined that there is a workable area on the right side of the lawn robot 200 and the lawn pixels in the workable area meet the conditions, thereby determining the turning direction according to the boundary information Turn clockwise.

Furthermore, the lawn robot 200 obtains the surrounding corresponding visual perception pictures in real time when turning, as shown in FIG. 7 .

In this process, synchronously for the obtained visual perception pictures, based on the visual boundary detection, a neural network segmentation diagram as shown in Figure 8 is formed, and the working area information, non-working area information and boundary contour information are obtained based on this, so that the recognized The natural boundaries of generate machine-recognizable working area boundary information, as shown in Figure 8.

During this process, the visual boundary sensor in the lawn robot 200 also synchronously judges whether the lawn robot 200 has reached the boundary based on the processed boundary information of the working area. For example, based on the neural network segmentation diagram shown in Figure 8, the lawn pixels in the drivable area are statistically analyzed, and according to the statistical results, it is determined that the lawn pixels in the drivable area in front of the current state of the lawn robot 200 meet the conditions, and the lawn robot can pass , will send a signal to the driving control system in the lawn robot 200, and the control system controls the lawn robot to stop turning and start to move forward.

As an example, the control system can precisely adjust the subsequent forward path of the lawn robot through the following modes.

Mode 1: The control system is along the boundary mode.

In this mode, after the control system receives the detection signal from the visual boundary sensor, it controls the lawn robot to move forward along the boundary line after turning.

For example, at boundary B, the result of visual boundary analysis is a counterclockwise turn. At the same time, the visual boundary sensor analyzes the angle of the turn. When the forward direction of the lawn robot is horizontal to the boundary line after the turn, the lawn robot continues to advance straight along the direction of the boundary line BC. When advancing along the boundary line, the surrounding pictures are visually perceived in real time, as shown in Figure 7, and the obtained boundary information is analyzed in real time, as shown in Figure 8. The control system maintains the distance between the lawn robot and the boundary according to the boundary information obtained by analysis.

Mode 2: The control system is in automatic turning mode.

In this mode, the control system of the lawn robot starts to turn after receiving the boundary signal detected by the visual boundary sensor. At the same time, the visual boundary sensor analyzes the turning angle, firstly to ensure that the lawn mower will not go beyond the boundary after turning, and then control the lawn robot to continue Turn at a random angle, and then the control system controls the lawn robot to move forward in a straight line, and controls the lawn robot to drive away from the boundary.

Mode 3: The control system is in the preset distance mode.

In this mode, after the control system of the lawn robot receives the detection signal from the visual boundary sensor, it controls the lawn robot to turn first. The turning angle and direction are determined by the boundary image. For example, at boundary B, the visual boundary analysis result is a counterclockwise turn. At the same time, the visual boundary sensor analyzes the turning angle. When the forward direction of the lawn robot is horizontal to the boundary line after the turn, the lawn robot is controlled to continue to advance straight along the boundary line; the control system presets the distance value along the boundary, and will monitor the lawn in real time. When the distance value that the robot advances along the boundary reaches the preset value, the control system controls the robot to turn according to the visual analysis sensor. After turning at a random angle, the mower leaves the boundary and moves straight forward.

It can be seen from the above example that the solution provided by the present invention can accurately identify the relative position between the equipment and the boundary of the working area, and can precisely control the forward route of the equipment when the equipment reaches the boundary of the working area, so as to prevent the equipment from crossing the boundary of the working area and improve the automatic operation of the equipment. reliability and safety.

The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims

The robot control system based on visual boundary detection is characterized in that it includes: an image acquisition module, a visual boundary recognition module and a mobile control module;

The image acquisition module obtains image information of the surrounding environment of the robot in real time;

The visual boundary identification module identifies the working boundary of the working area according to the image information collected by the image acquisition module, and forms corresponding working area boundary information;

The movement control module calculates and judges whether the robot reaches the boundary of the working area according to the working area boundary information of the working area identified by the visual boundary identification module, and if it reaches the boundary of the working area, adjusts the working state of the robot to prevent the robot from crossing the boundary of the working area; if If the boundary of the working area is not reached, the robot is controlled to maintain the current working state.
The robot control system based on visual boundary detection according to claim 1, wherein the visual boundary recognition module recognizes the boundary of the working area based on a deep neural network and an image processing method.
The robot control system based on visual boundary detection according to claim 2, wherein the visual boundary recognition module segments the obtained robot surrounding environment image based on a deep neural network to obtain a corresponding neural network segmentation map, and The corresponding workable area and non-workable area are formed in the segmentation map of the neural network, and the boundary between the two forms the boundary of the workable area.
The robot control system based on visual boundary detection according to claim 1, wherein the boundary information of the working area includes boundary contour information of the working area.
The robot control system based on visual boundary detection according to claim 4, wherein the mobile control module obtains the boundary contour information of the working area identified by the visual boundary recognition module, and several contour point information corresponding to the contour of the robot, Based on this, calculate the relative relationship between several contour points and the boundary contour of the working area, judge whether the robot advances to the boundary position of the working area according to the calculation results, and when the robot advances to the boundary of the working area, according to the working mode and the boundary information of the working area, Control the robot to adjust its forward direction.
The robot control method based on visual boundary detection is characterized in that, comprising:

Real-time acquisition of image information of the surrounding environment of the device;

Identify the natural working boundary of the working area according to the collected image information, and generate recognizable working area boundary information from the natural working boundary;

Based on the generated working area boundary information, calculate and judge whether the robot has reached the working area boundary. If it reaches the working area boundary, adjust the working state of the robot to prevent the robot from crossing the working area boundary; if it does not reach the working area boundary, control the robot to maintain the current working state.
The robot control method based on visual boundary detection according to claim 6, characterized in that the method generates identifiable working area boundary information based on a deep neural network and an image processing algorithm.
The robot control method based on visual boundary detection according to claim 7, wherein the method is based on a deep neural network to segment the acquired robot surrounding environment image to obtain a corresponding neural network segmentation map, and segment the image in the neural network The corresponding workable area and non-workable area are formed in the figure, and the boundary between them forms the working area boundary.
The robot control method based on visual boundary detection according to claim 6, characterized in that, the working area boundary information includes working area boundary contour information.
The robot control method based on visual boundary detection according to claim 9, characterized in that, the method acquires the identified boundary contour information of the working area in real time, and several contour point information corresponding to the contour of the robot, and calculates several contours accordingly The relative relationship between the point and the boundary contour of the working area, judge whether the robot advances to the boundary position of the working area according to the calculation results, and when the robot advances to the boundary of the working area, control the robot to adjust the forward direction according to the working mode and the boundary information of the working area .