WO2023280533A1

WO2023280533A1 - Method and apparatus for controlling self-moving device

Info

Publication number: WO2023280533A1
Application number: PCT/EP2022/066239
Authority: WO
Inventors: Davide DALFRA
Original assignee: Positec Power Tools Suzhou Co Ltd; Positec Italia Srl
Current assignee: Positec Power Tools Suzhou Co Ltd; Positec Italia Srl
Priority date: 2021-07-07
Filing date: 2022-06-14
Publication date: 2023-01-12
Anticipated expiration: 2024-01-07
Also published as: CN115657648A

Abstract

This application provides a method and apparatus for controlling a self-moving device, which can improve working safety of a self-moving device. The method includes periodically performing a control operation, the control operation including: receiving sensor data from one or more sensors; providing the sensor data as a sensor input to a control process; and controlling the self-moving device to move according to the control process. The method further includes: periodically performing a verification operation, the verification operation comprising: providing a verification input to the control process, wherein the verification input is associated with a pre-stored expected result; processing, by the control process, the verification input to produce an identification result; comparing the identification result to the pre-stored expected result, wherein if the identification result does not correspond to the pre-stored expected result: determining that a working status of the self-moving device is abnormal; and controlling the self-moving device to stop moving.

Description

METHOD AND APPARATUS FOR CONTROLLING SELF-MOVING

DEVICE

TECHNICAL FIELD

This application relates to the field of intelligent robot technologies, and specifically, to a method and apparatus for controlling a self-moving device.

BACKGROUND

Because a self-moving device does not require manual operation, the self-moving device may move and work within a predetermined working region. At present, a working region is generally determined by a physical boundary line, such as a boundary sensor. The self- moving device determines a working region by detecting the physical boundary line, which is inflexible and inconvenient.

With the development of new technologies, the working region may be determined by applying computer vision. Using computer vision does not require setting a physical boundary line, and as such, computer vision methods for determining a working region are more convenient and flexible for use. However, using computer vision to determine the working region without a physical boundary line poses a further problem regarding the working safety of the self-moving device.

It has thus been appreciated that there is a need to be able to use computer vision to determine a working region in a safe and reliable manner.

SUMMARY

In view of this, embodiments of this application are dedicated to providing a method and apparatus for controlling a self-moving device, which can improve working safety of a self- moving device.

According to a first aspect, a method for verifying correct operation of a self-moving device is defined, the method comprising periodically performing a control operation, the control operation including: receiving sensor data from one or more sensors; providing the sensor data as a sensor input to a control process; and controlling the self-moving device to move according to the control process; the method further comprising: periodically performing a verification operation, the verification operation comprising: providing a verification input to the control process, wherein the verification input is associated with a pre-stored expected result; processing, by the control process, the verification input to produce an identification result; comparing the identification result to the pre-stored expected result, wherein if the identification result does not correspond to the pre-stored expected result: determining that a working status of the self-moving device is abnormal; and controlling the self-moving device to stop moving.

This method advantageously checks and verifies that the control process responsible for controlling the self-moving device to move based on sensor input is functioning correctly, by periodically performing a verification operation. The verification operation effectively provides a sensor input including sensor data that should be processed by the control process to provide a certain pre-stored expected result. The actual result returned by the control process, also called an identification result, is compared to the expected result. If these results match, then the verification operation determines that the control process is functioning correctly. However, if the verification operation determines that the actual result does not match the expected result, then it is deemed that the control process is not functioning correctly. In order to maintain safety of the self-moving device, the self-moving device is instructed to stop moving.

The sensor input may be from any sensor suitable for performing identification and/or navigation tasks, such as an imaging sensor such as a camera, a radar, LiDAR, infrared sensor or the like. The control process may be any process which takes an input from one or more sensors and produces an output which is then used to determine how to control the self- moving device. For example, the control process could be an image recognition or identification algorithm, an object identification algorithm, a navigation algorithm or the like. Such algorithms can be implemented in an artificial intelligence (AI) model using deep- learning and/or machine learning techniques, neural networks and the like. The AI model may be trained by training data and routinely verified by the verification operation to ensure that the result provided by the AI model for control purposes is one that matches an expected result from a known input.

Preferably the method may further comprise performing the verification operation according to a verification rate such that a maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations.

The verification rate is effectively the number of times that a verification operation is performed in its entirety for a measure of time. For example, the verification rate may be 2 or 3 times per second. When the sensor data used is an image from an imaging sensor such as a camera, the verification rate can be defined in terms of frames per second, wherein the verification image is a verification frame. As such, a verification rate of 2 or 3 frames per second means that the verification operation is performed 2 or 3 times per second. By linking the verifications rate to maximum distance threshold, which may be for example, 0.25m 0.5m, lm, 2m, 5m, 10m or 100m, it is possible to ensure that the self-moving device does not move beyond said maximum threshold distance between verification operations. This means that, even if the control process is not functioning correctly, the self-moving device will not move excessively according to poor function of the control process. This makes the self-moving device safer as it is less likely to cause accidents, mistakes, collisions or the like in a shorter distance. For example, the verification operation may occur at a rate of one time per second. Between each verification operation, there may be one or more control operations. For instance, the verification operation may be performed, followed by 4 control operations, before repeating in a cycle. Even if the 4 control operations are erroneous based on poor function of the control process, the fact that the verification rate is set to once per second, based on a maximum threshold distance, the 4 erroneous control operations will not amount to erroneous behavior of the self-moving device beyond the maximum threshold distance, since the next verification operation will identify that the control process if not functioning correctly, and the self-moving device will be stopped.

Preferably the method may further comprise detecting a speed of the self-moving device; setting and/or adjusting the verification rate based on the speed of the self-moving device such that the maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations.

This advantageously allows the method to be used on self-moving devices of a varying speed, to ensure that a distance travelled by the self-moving device between verification operations, according to intermediate control operations, does not exceed the maximum distance threshold.

Preferably the method may further comprise detecting the speed of the self-moving device in real or near-real time, and adjusting the verification rate comprises dynamically adjusting the verification rate based on the detected speed of the self-moving device, such that the maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations.

This advantageously allows the method to be used on self-moving devices of a varying speed in real-time, to ensure that a distance travelled by the self-moving device between verification operations, according to intermediate control operations, does not exceed the maximum distance threshold.

Preferably the method may further comprise detecting a speed of the self-moving device; limiting or reducing the speed of the self-moving device below a maximum safe speed, such that the maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations performed at the verification rate, the maximum safe speed determined in dependence on the verification rate and the maximum threshold distance.

Therefore, not only can the verification rate be adjusted, but also, the speed of the self- moving vehicle can be adjusted to ensure that the maximum threshold distance is not exceeded between verification operations. The verification rate may thus stay constant and the speed modified to ensure that it is not possible to travel further than the maximum threshold distance between verification operations.

Preferably the method may further comprise performing a combination of: setting and/or adjusting the verification rate based on the speed of the self-moving device; and limiting or reducing the speed of the self-moving device, such that the maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations.

This means that both the verification rate and the speed of the self-moving device can be modified to ensure that it is not possible to travel further than the maximum threshold distance between verification operations performing a combination of actions such as this is advantageous because it means the verification rate and the speed do not have to be changed as much individually compared to when these aspects are changed alone. This means that the verification rate can be optimized for processing efficiency, whilst speed can be optimized to ensure the self-moving device is not too slow.

Preferably, the method further comprises: adjusting the maximum threshold distance dependent on at least one of: a size of the self-moving device; an environmental parameter relating to a feature of the environment of the self-moving device; and/or a braking distance of the self-moving device.

Other factors may affect the distance travelled by the self-moving device between verification operations. This feature advantageously takes these factors into consideration to ensure that it is not possible to travel further than the maximum threshold distance between verification operations, even in differing conditions.

Preferably, the method further comprises adjusting the maximum threshold distance dependent on a braking distance of the self-moving device by reducing the maximum threshold distance by the braking distance, wherein the method further comprises: obtaining the braking distance from a look-up table.

This ensures that it is not possible to travel further than the maximum threshold distance between verification operations, even when the instruction to stop the self-moving device does not result in its immediate stop. By accounting for the braking distance, the maximum threshold distance is not exceeded. The look-up table may be a database, or any accessible memory containing data relating to a braking distance. The look-up table may be predefined or populated based on calibration tests and/or historical data from previous operations.

Preferably, the look-up table stores a plurality of values for the braking distance, each value in the look-up table being associated with a speed or range of speeds, the method further comprising: obtaining the value for the braking distance from the plurality of values for braking distance based on the speed of the self-moving device.

The look-up table may thus store different braking distances for different speeds, which ensures that the maximum threshold distance is not exceeded even for higher speeds, since the corresponding larger braking distance will be subtracted from the original maximum threshold distance to obtain a maximum threshold distance appropriate for the speed of the self-moving device. Alternatively, the look-up table may only store one braking distance, corresponding to a braking distance for the maximum speed of the self-moving device. This ensures safety is maintained whilst freeing up storage and processing power by not having to store multiple pieces of data.

Preferably, the method further comprises performing a braking distance calibration, the braking distance calibration comprising: controlling the self-moving device to move at a predefined speed; instructing the self-moving device to brake or stop moving from at a first recorded time; determining that the self-moving device becomes stationary; determining a distance travelled from the first recorded time; and recording, in the look-up table, the distance travelled as the braking distance for the predefined speed. This advantageously allows for real-world braking distances to be obtained for future use.

Preferably the method further comprises detecting the environmental parameter relating to a feature of the environment of the self-moving device; comparing the environmental parameter to an environmental parameter threshold; and if the environmental parameter meets the environmental parameter threshold, decreasing the maximum threshold distance by an environmental parameter scaling factor.

The environmental parameter may be used with or without the braking distance to modify the maximum threshold distance. For example, when used with the braking distance, the environmental parameter may be considered to effectively increase the braking distance. The environmental parameter threshold may be considered a threshold above which the difference caused to the braking distance and/or maximum threshold distance, by the environmental parameter, is not negligible. The environmental parameter threshold may be pre-stored, set by a user, or based on calibration tests in different environmental conditions. Preferably, the environmental parameter includes one or more of: a moisture content of a surface on which the self-moving device moves; a gradient of the surface on which the self- moving device moves; and a temperature of the environment. The moisture content of the surface can affect braking distance for example. When the surface is wet, the braking distance is higher. The gradient can affect the braking distance and speed of the self-moving device. When the gradient is a downhill slope, the self-moving device may move faster and have a higher braking distance. The temperature can affect the braking distance, based on the temperature of the brakes and the hardness of the surface. When the brakes are relatively cooler and the surface harder, the braking distance may increase. By factoring these considerations into calculation or determination of the maximum distance threshold, the maximum distance threshold is more accurate.

Preferably, performing of the control operation and performing of the verification operation are mutually exclusive, such that the verification operation is performed cyclically after every N control operations, where N is a positive integer. This effectively means that the self-moving device performs a verification operation followed by one or more control operations in a repeated cycle.

Preferably, adjusting the verification rate by increasing a control operation rate and/or decreasing the number of control operations N between verification operations. The verification rate can be increased by increasing the control operation rate for a given number of control operations per verification operation. For example, when the control operation rate is 4 per second, and 3 control operations are performed every second, this only leaves space for 1 verification operation per second. Increasing the control operation rate to 6 per second, and maintaining 3 control operations performed every second, frees up space and leaves 0.5s to perform more than one verification operation. Similarly, the number of control operations per verifications operation can be changed to increase the verification rate. For example, 4 control operations per verification operation may be reduced to 2 control operations per verification operation, meaning verification operations are performed more frequently. These changes may be made according to the speed of self-moving device as discussed above.

Preferably, the one or more sensors include an imaging sensor such that the method comprises: periodically performing the control operation, the control operation including: receiving image data from the imaging sensor; providing the image data as the sensor input to the control process, wherein the control process is an image recognition or image- processing process; processing, by the control process, the image data to obtain an image- processing result; controlling the self-moving device to move according to the image- processing result; the method further comprising: periodically performing the verification operation, the verification operation comprising: providing a verification image input to the control process, wherein the verification image input is associated with a pre-stored expected result; processing, by the control process, the verification image input to produce an identification result; comparing the identification result to the pre-stored expected result, wherein if the identification result does not correspond to the pre-stored expected result: determining that a working status of the self-moving device is abnormal; and controlling the self-moving device to stop moving.

The imaging device may be an imaging sensor such as a camera. The image recognition or processing process may be an AI model including an image classification, image recognition, object detection algorithm or the like. The verification image may be an image with a pre-determined or pre-known classification or detection result. Using a vision module including an imaging sensor allows the self-moving device to use images to perform the method.

Preferably, the image-processing result is a determination of a type of surface of the environment in the vicinity and/or upcoming path of the self-moving device. The determination may distinguish whether the self-moving device is capable to move to the type of surface or whether the self-moving device is not authorized or supposed to move on the type of surface. The determination may determine whether the surface is grass or not grass, a border of a lawn area or not a border, a paved surface or not a paved surface, for example.

Preferably, the verification image input is an image of a type of surface and the pre stored expected result is a result indicating the type of surface present in the verification image input. The verification image is thus linked directly to the purpose of the control process, such that the verification operation directly targets the core functioning of the control process.

Preferably, the determination of the type of surface is a determination of a lawn or non lawn region present in the image data. The self-moving device may be allowed to move in the lawn region but not the non-lawn region.

Preferably, the verification image input is an image of region including lawn or non lawn, and the pre-stored expected result is a result indicating lawn or non-lawn respectively.

Preferably, controlling the self-moving device to stop moving comprises controlling the self-moving device to perform at least one of: stopping any motion completely; reducing movement speed below a safety-threshold speed; stopping motion of one or more actuators; and/or stopping a grass-cutting mechanism. The safety-threshold speed may be much lower than an average speed of the self-moving device and may be set by the user or pre-set. The one or more actuators may be a motor or engine responsible for propelling the self-moving device. The grass-cutting mechanism may be a sharp tool for shearing grass, a spinning blade or wire or the like.

Preferably, the control process is an artificial intelligence model. The AI model allows the control operations to be performed autonomously, by collecting the sensor input and deciding, determining or classifying, by the AI model, the sensor input to produce an AI output the AI output causes the self-moving device to control how the self-moving device moves and operates.

According to a second aspect, there is provided a self-moving device, comprising: a sensor module comprising one or more sensors configured to obtain sensor data; a motor module configured to move the self-moving device; and a control module, communicatively coupled to the sensor module and the motor module; wherein the control module is configured to perform the method of the first aspect. The sensor module may be a vision module in the case that the sensor is an imaging device such as a camera. The motor module may be a walking/movement motor. The control module may include a processor and memory, wherein the processor is configured to execute the method of the first aspect. The control process, or AI model, may be loaded onto the memory and executed by the processor. Preferably, the one or more sensors include one or more of: an imaging sensor, a radar sensor, a light detection and ranging LiDAR sensor, and/or an infrared sensor. Any one or more of these may be sued to provide the sensor input for the control process. The verification operation includes providing a verification input corresponding to the type of sensors.

Preferably, the self-moving device is a robot lawnmower comprising a cutting instrument configured to cut lawn. The robot lawnmower may include wheels and a motor allowing the robot lawnmower to move around a lawn.

Preferably, if the identification result does not correspond to the pre-stored expected result, the control module is configured to control the self-moving device to stop moving by being configured to: cause the motor module to stop moving the self-moving device; and/or cause the cutting instrument to disengage a surface and/or stop actuating. The cutting instrument may disengage by altering its position or by turning off.

According to a third aspect, there is provided a computer readable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of the first aspect above. The computer readable medium may be any suitable memory. A computer-readable storage media can be any available storage media that may be accessed by a computer. By way of example, and not limitation, such computer- readable storage media may comprise RAM, ROM, EEPROM, flash memory or other memory devices, CD-ROM or other optical disc storage, magnetic disc storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disc and disk, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray (RTM) disc (BD). Further, a propagated signal is not included within the scope of computer-readable storage media. Computer- readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fibre optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media.

In a further example, a method for controlling a self-moving device is provided. The method is applicable to a control module of the self-moving device, the method including: receiving a plurality of types of indication information sent by a vision module, where the indication information is used for indicating working statuses of the vision module; detecting whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information; and controlling the self-moving device to stop working in a case that an anomaly of the vision module is detected.

In a possible implementation, the plurality of types of indication information include verification information and/or heartbeat information, the verification information is used for indicating a verification result of the vision module for a verification image, and the heartbeat information is used for indicating a communication status of the control module and the vision module; the detecting whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information includes: detecting whether identification of the vision module is abnormal according to a receiving status of the verification information; and/or detecting whether communication of the vision module is abnormal according to a receiving status of the heartbeat information.

In a possible implementation, the detecting whether identification of the vision module is abnormal according to a receiving status of the verification information includes: determining that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that an identification result of the vision module for the verification image is inconsistent with a pre stored result corresponding to the verification image.

In a possible implementation, the detecting whether identification of the vision module is abnormal according to a receiving status of the verification information includes: determining that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range. In a possible implementation, the heartbeat information includes a first heartbeat packet; and the detecting whether communication of the vision module is abnormal according to a receiving status of the heartbeat information includes: determining that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range.

In a possible implementation, before the receiving a plurality of types of indication information sent by a vision module, the method further includes: sending a second heartbeat packet to the vision module; correspondingly, the heartbeat information comprises a reply packet for the second heartbeat packet; and the detecting whether communication of the vision module is abnormal according to a receiving status of the heartbeat information includes: determining that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range.

In a possible implementation, in a case that no anomaly of the vision module is detected, the method further includes: controlling the self-moving device to stop working in a case that warning information sent by the vision module is received, where the warning information is used for indicating that the vision module identifies that the self-moving device is in a non working region.

According to a second further example, a method for controlling a self-moving device is provided. The method is applicable to a vision module in the self-moving device, the method including: generating a plurality of types of indication information, where the indication information is used for indicating working statuses of the vision module; sending the plurality of types of indication information to a control module, to cause the control module to detect whether the vision module is abnormal based on a receiving status of at least one of the plurality of types of indication information.

In a possible implementation, the plurality of types of indication information include verification information and/or heartbeat information, the verification information is used for indicating a verification result of the vision module for a verification image, and the heartbeat information is used for indicating a communication status of the control module and the vision module; the generating a plurality of types of indication information includes: generating the verification information; and correspondingly, the sending the plurality of types of indication information to a control module, to cause the control module to detect whether the vision module is abnormal based on a receiving status of at least one of the plurality of types of indication information includes: sending the verification information to the control module, to cause the control module to detect whether identification of the vision module is abnormal based on a receiving status of the verification information; and/or the generating a plurality of types of indication information includes: generating the heartbeat information; and correspondingly, the sending the plurality of types of indication information to a control module, to cause the control module to detect whether the vision module is abnormal based on a receiving status of at least one of the plurality of types of indication information includes: sending the heartbeat information to the control module, to cause the control module to detect whether communication of the vision module is abnormal based on a receiving status of the heartbeat information.

In a possible implementation, the generating the verification information includes: inputting the verification image into an artificial intelligence (AI) model to obtain an identification result; verifying whether the identification result is consistent with a pre-stored result corresponding to the verification image, to obtain the verification result; and generating the verification information according to the verification result; and correspondingly, the sending the verification information to the control module, to cause the control module to detect whether identification of the vision module is abnormal based on a receiving status of the verification information includes: sending the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the identification result is inconsistent with the pre-stored result corresponding to the verification image.

In a possible implementation, the verification image includes an image in a working region and an image in a non-working region, a pre-stored result corresponding to the image in the working region is a first pre-stored result, and a pre-stored result corresponding to the image in the non-working region is a second pre-stored result; the inputting the verification image into an AI model to obtain an identification result includes: inputting the image in the working region into the AI model to obtain a first identification result; and/or inputting the image in the non-working region into the AI model to obtain a second identification result; correspondingly, the verifying whether the identification result is consistent with a pre-stored result corresponding to the verification image includes: verifying whether the first identification result is consistent with the first pre-stored result; and/or verifying whether the second identification result is consistent with the second pre-stored result; and correspondingly, the sending the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the identification result is inconsistent with the pre-stored result corresponding to the verification image includes: sending the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the first identification result is inconsistent with the first pre-stored result and/or the second identification result is inconsistent with the second pre-stored result.

In a possible implementation, the sending the verification information to the control module, to cause the control module to detect whether identification of the vision module is abnormal based on a receiving status of the verification information includes: sending the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range.

In a possible implementation, the heartbeat information includes a first heartbeat packet; and the sending the heartbeat information to the control module, to cause the control module to detect whether communication of the vision module is abnormal based on a receiving status of the heartbeat information includes: sending the first heartbeat packet to the control module, to cause the control module to determine that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range. In a possible implementation, before the sending the heartbeat information to the control module, the method further includes: receiving the second heartbeat packet sent by the control module; correspondingly, the heartbeat information includes a reply packet for the second heartbeat packet; and the sending the heartbeat information to the control module, to cause the control module to detect whether communication of the vision module is abnormal based on a receiving status of the heartbeat information includes: sending the reply packet to the control module, to cause the control module to determine that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range.

In a possible implementation, in a case that no anomaly of the vision module is detected by the control module, the method further includes: obtaining a current image at a position of the self-moving device; inputting the current image into an AI model to obtain an identification result; and sending warning information to the control module in a case that the identification result indicates that the self-moving device is in a non-working region, to cause the control module to control the self-moving device to stop working in a case that the warning information is received.

According to a further example, a method for controlling a self-moving device is provided. The self-moving device includes a vision module and a control module, the method including: sending, by the vision module, a plurality of types of indication information to the control module, where the indication information is used for indicating working statuses of the vision module; detecting, by the control module, whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information; and controlling, by the control module, the self-moving device to stop working in a case that an anomaly of the vision module is detected.

In a possible implementation, the plurality of types of indication information include verification information and/or heartbeat information, the verification information is used for indicating a verification result of the vision module for a verification image, and the heartbeat information is used for indicating a communication status of the control module and the vision module; the sending, by the vision module, a plurality of types of indication information to the control module includes: sending, by the vision module, the verification information to the control module; and correspondingly, the detecting, by the control module, whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information includes: detecting, by the control module, whether identification of the vision module is abnormal according to a receiving status of the verification information; and/or the sending, by the vision module, a plurality of types of indication information to the control module includes: sending, by the vision module, the heartbeat information to the control module; and correspondingly, the detecting, by the control module, whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information includes: detecting, by the control module, whether communication of the vision module is abnormal according to a receiving status of the heartbeat information.

In a possible implementation, the sending, by the vision module, the verification information to the control module includes: inputting, by the vision module, the verification image into an AI model to obtain an identification result; verifying, by the vision module, whether the identification result is consistent with a pre-stored result corresponding to the verification image to obtain the verification result; generating, by the vision module, the verification information according to the verification result; and sending, by the vision module, the verification information to the control module; and correspondingly, the detecting, by the control module, whether identification of the vision module is abnormal according to a receiving status of the verification information includes: determining, by the control module, that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the identification result is inconsistent with the pre-stored result corresponding to the verification image.

In a possible implementation, the verification image includes an image in a working region and an image in a non-working region, a pre-stored result corresponding to the image in the working region is a first pre-stored result, and a pre-stored result corresponding to the image in the non-working region is a second pre- stored result; the inputting, by the vision module, the verification image into an AI model to obtain an identification result includes: inputting, by the vision module, the image in the working region into the AI model to obtain a first identification result; and/or inputting, by the vision module, the image in the non working region into the AI model to obtain a second identification result; correspondingly, the verifying, by the vision module, whether the identification result is consistent with a pre stored result corresponding to the verification image includes: verifying, by the vision module, whether the first identification result is consistent with the first pre-stored result; and/or verifying whether the second identification result is consistent with the second pre stored result; and correspondingly, the determining, by the control module, that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the identification result is inconsistent with the pre-stored result corresponding to the verification image includes: determining, by the control module, that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the first identification result is inconsistent with the first pre-stored result and/or the second identification result is inconsistent with the second pre-stored result.

In a possible implementation, the detecting, by the control module, whether identification of the vision module is abnormal according to a receiving status of the verification information includes: determining, by the control module, that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range.

In a possible implementation, the heartbeat information includes a first heartbeat packet, and the sending, by the vision module, the heartbeat information to the control module includes: sending, by the vision module, the first heartbeat packet to the control module; and correspondingly, the detecting, by the control module, whether communication of the vision module is abnormal according to a receiving status of the heartbeat information includes: determining, by the control module, that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range. In a possible implementation, before the sending, by the vision module, the heartbeat information to the control module, the method further includes: sending, by the control module, a second heartbeat packet to the vision module; correspondingly, the heartbeat information includes a reply packet for the second heartbeat packet, and the sending, by the vision module, the heartbeat information to the control module includes: sending, by the vision module, the reply packet to the control module; and correspondingly, the detecting, by the control module, whether communication of the vision module is abnormal according to a receiving status of the heartbeat information includes: determining, by the control module, that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range.

In a possible implementation, in a case that no anomaly of the vision module is detected by the control module, the method further includes: obtaining, by the vision module, a current image at a position of the self-moving device; inputting, by the vision module, the current image into an AI model to obtain an identification result; sending, by the vision module, warning information to the control module in a case that the identification result indicates that the self-moving device is in a non-working region; and controlling, by the control module, the self-moving device to stop working in a case that the warning information is received.

According to a fourth further example, an apparatus for controlling a self-moving device is provided. The apparatus includes a control module; and the control module is configured to: receive a plurality of types of indication information sent by a vision module, where the indication information is used for indicating working statuses of the vision module; detect whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information; and control the self-moving device to stop working in a case that an anomaly of the vision module is detected.

According to a fifth further example, an apparatus for controlling a self-moving device is provided. The apparatus includes a vision module; and the vision module is configured to: generate a plurality of types of indication information, where the indication information is used for indicating working statuses of the vision module; and send the plurality of types of indication information to a control module of the self-moving device, to cause the control module to detect whether the vision module is abnormal based on a receiving status of at least one of the plurality of types of indication information.

According to a sixth further example, a self-moving device is provided, including: a vision module, configured to send a plurality of types of indication information to a control module, where the indication information is used for indicating working statuses of the vision module; and the control module being configured to: detect whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information; and control the self-moving device to stop working in a case that an anomaly of the vision module is detected.

Accordingly, the status of the vision module is detected, and in a case that an anomaly of the vision module is detected, the self-moving device is controlled to stop working, thereby preventing the self-moving device from at least one of randomly moving outside a boundary, causing safety problems to a user, and causing damage to a working region when the vision module is abnormal, and improving working safety of the self-moving device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an automatic working system according to an embodiment.

FIG. 2 is a schematic structural diagram of a smart lawnmower according to an embodiment.

FIG. 3 is a schematic structural diagram of a self-moving device according to an embodiment.

FIG. 4 is a schematic flowchart of a method for controlling a self-moving device according to an embodiment.

FIG. 5 is a schematic flowchart of another method for controlling a self-moving device according to an embodiment.

FIG. 6 is a schematic flowchart of still another method for controlling a self-moving device according to an embodiment.

FIG. 7 is a schematic block diagram of an apparatus for controlling a self-moving device according to an embodiment.

FIG. 8 is a schematic block diagram of another apparatus for controlling a self-moving device according to an embodiment.

FIG. 9 is a schematic block diagram of a self-moving device according to an embodiment.

DETAILED DESCRIPTION

The technical solutions in embodiments of this application are clearly and completely described below with reference to the accompanying drawings. It will be apparent that the embodiments described here are exemplary.

A self-moving device in the embodiments of this application may be an outdoor robot, for example, a smart device with an automatic moving/movement function such as a smart lawnmower, a smart snow sweeper, a smart sweeper, and a smart floor scrubber.

Using a smart lawnmower as an example, an automatic working system 100 shown in FIG. 1 may include a smart lawnmower 1 and a boundary 6, where the boundary defines a working region 7 of the smart lawnmower 1. The boundary 6 may be a boundary line between a lawn region and a non-lawn region, whereby the lawn region is the working region 7 contained within the boundary line. The smart lawnmower 1 may work in the working region 7 defined by the boundary 6, and perform functions such as automatically completing mowing. The boundary 6 may separate the working region 7 from a non-working region.

In the working region 7 of the smart lawnmower 1, there are some obstacle regions that affect working of the smart lawnmower. The obstacle regions may include, for example, a region 3 where houses are located, a pit region 4, and a region 5 where trees are located. During working, the smart lawnmower 1 may bypass these regions to work and mow the lawn region.

The automatic working system 100 may further include a charging station 2 configured to recharge the smart lawnmower 1. The smart lawnmower 1 may automatically perform work tasks without human supervision, and may automatically return to the charging station 2 for charging when power is insufficient.

The charging station 2 may be located on the boundary 6, such that a part of the charging station 2 is located in the working region 7, and the other part is located in the non-working region; or the charging station 2 may be entirely located in the working region 7; or the charging station 2 may alternatively be entirely located in the non-working region.

FIG. 2 is a schematic structural diagram of a smart lawnmower 1. The automatic lawnmower may include a housing 16, a mobile module, a task execution module, an energy module, a control module, and the like.

The mobile module is configured to drive the smart lawnmower to walk/move in the working region 7. The mobile module generally includes a wheel set mounted on the smart lawnmower and walking/movement motors that drive the wheel set to walk/move and steer. The wheel set includes driving wheels 14 connected to the walking/movement motors and auxiliary wheels 15 mainly for auxiliary support. The auxiliary wheels 15 may also be referred to as driven wheels, and the auxiliary wheels 15 may be universal wheels. There may be two driving wheels 14, which are located on two sides of the housing 16 respectively. There may be two walking/movement motors, which are connected to the two driving wheels respectively. There may be one or two auxiliary wheels. When there are two auxiliary wheels 15, the two auxiliary wheels 15 may be located on two sides of the front of the smart lawnmower. The auxiliary wheels 15 are not connected to the walking/movement motors, but are driven to roll and walk/move when supporting the smart lawnmower to walk/move. Through the setting of the foregoing structure, the smart lawnmower may flexibly walk/move and steer in the working region 7 under the control of the control module.

The task execution module may include a cutting assembly that may be configured to perform mowing. The cutting assembly may be disposed on a chassis of the smart lawnmower, and may be located between the driving wheels 14 and the auxiliary wheels 15.

The energy module may include a battery pack configured to provide electric energy for movement and working of the smart lawnmower 1. For example, the energy module may provide electric energy for a motor, so that the motor may drive the cutting assembly to work.

The control module may be electrically connected to the mobile module, the task execution module, and the energy module, to control the mobile module to drive the smart lawnmower 1 to move, and control the task execution module to execute a work task.

The control module may be configured to control the smart lawnmower to automatically walk/move, work, replenish energy, and the like, and is a core component of the smart lawnmower. Functions performed by the control module may include: controlling the task execution module to start or stop working, generating a walking/moving path and controlling the mobile module to walk/move according to the path, determining power of the energy module and instructing the smart lawnmower to return to a docking station in time for automatic docking and charging, controlling the smart lawnmower to return to the working region when it is detected that the smart lawnmower is at a boundary position or in the non working region, and the like.

Still referring to FIG. 1, the boundary 6 may limit movement of the smart lawnmower 1 within the working region 7 defined by the boundary 6, preventing the smart lawnmower 1 from moving to a region outside the boundary. As noted in the background section above, the boundary 6 is typically identified by a physical boundary line. For example, a boundary line may be arranged at the position of the boundary 6, and the smart lawnmower may identify the boundary line by detecting intensity of a magnetic field signal formed by current in the boundary line. However, the foregoing setting manner requires manual arrangement of the boundary line, which is inconvenient and inflexible to use, and the laying and maintenance costs are high.

Embodiments of the present disclosure include a vision module used for identifying a position of the self-moving device. This manner of identifying a position of the self-moving device does not require setting a physical boundary line, but uses an image photographed by the vision module to identify whether the self-moving device is in the working region 7, which is convenient and flexible for use. The vision module may be mounted on the self-moving device and obtain a current image at the position of the self-moving device. The current image may also be referred to as an environment image around the body. According to the current image, the vision module may determine whether the self-moving device is in the working region 7.

As shown in FIG. 3, a self-moving device shown in FIG. 3 may include a vision module 11. The vision module 11 may include an image acquisition apparatus 110 and an image processing module 113. The image acquisition apparatus 110 may be configured to acquire a current image at a position of the self-moving device. The image processing module 113 may be configured to analyze and process the current image, such as by using an artificial intelligence (AI) model to identify the current image, to identify the environment around the body. For example, the image processing module 113 may process the current image to identify whether the self-moving device is in a working region 7. Using the smart lawnmower 1 as an example, if the current image includes a relevant feature of grass, it may be determined that the smart lawnmower is in the working region 7; and if the current image includes a relevant feature of a road surface, or the current image does not include the relevant feature of the grass, it may be determined that the smart lawnmower is in a non-working region.

The image acquisition apparatus 110 may be, for example, various image acquisition devices such as a camera. The image acquisition apparatus 110 may include an optical system 111 and an image sensor 112. The optical system 111 may include single-convex, single concave, double-convex, double-concave lenses, and the like. The image sensor 112 may be a complementary metal oxide semiconductor (CMOS).

This embodiment of this application does not limit a shooting direction of an image captured by the image acquisition apparatus 110. The image acquisition apparatus 110 may capture images vertically downwards. For example, the vision module including the image acquisition apparatus 110 may be disposed on a chassis of the self-moving device. The image acquisition apparatus 110 may alternatively capture images in a direction inclined relative to a vertical direction. As shown in FIG. 2, the vision module 11 may be mounted on a front end of the self-moving device, the image acquisition apparatus may capture images obliquely forward, and an angle between the shooting direction of the image acquisition apparatus and the vertical direction is a, 0<a<90°.

The shooting direction of the image acquisition apparatus may be fixed or adjustable. For example, a position of the image acquisition apparatus may be changed through operations such as rotation and/or movement, thereby changing the shooting direction of the image acquisition apparatus.

An identification range of the image acquisition apparatus may be fixed or adjustable. This is not specifically limited in this embodiment of this application. Using FIG. 2 as an example, the identification range (that is, a viewing angle range) of the image acquisition apparatus shown in FIG. 2 is b, where b may be a fixed value or a variable value. When the identification range is b, the image acquisition apparatus may capture an image in a region d.

Still referring to FIG. 3, the image processing module 113 may be a system-on-chip (SOC). The image processing module 113 may include an image signal processing (ISP) module 114 and a central processing unit (CPU) 115. The ISP module 114 may be configured to process a signal outputted by the image sensor 112, and input the processed signal into the CPU 115 for image processing. The CPU 115 may perform AI identification and the like on an image.

The vision module of this embodiment of this application may identify whether the self- moving device is in the working region 7 by performing the AI identification on the current image. The AI identification may refer to a process of analyzing and processing the image by using an AI model. Before the self-moving device works, the AI model may be trained through sample images, so that the trained model can accurately identify a feature of each object in the image. A training process may include, for example, obtaining sample images, where the sample images may include images in a working region 7 and/or images in a non working region; and inputting the sample images into a first AI model for training, to obtain a second AI model. The first AI model may be an initial AI model, and the second AI model is a target AI model after training. To test the accuracy of the second AI model, a part of the sample images may be reserved. After the second AI model is obtained, the second AI model is tested by using the reserved sample images, to evaluate the response accuracy of the second AI model. After the training is completed, the self-moving device may use the second AI model to identify the current image. In other words, the second AI model represents the trained AI model that is used to identify images in real or near-real time whilst the self moving device including the vision module is moving. More detail regarding how the second AI model is routinely checked and/or trained further during use, using the sample images kept in reserve, is provided below.

The vision module may identify the working region 7 and/or the non-working region according to a key feature in the working region and/or the non-working region. The key feature may be a marker, an obstacle, or the like in the working region 7 and/or non-working region.

For the smart lawnmower 1, the key feature may include a feature of grass, and the vision module may use an image processing technology to identify a lawn region and a non-lawn region. For example, if the vision module identifies that the current image includes the feature of grass, it may be determined that the smart lawnmower is in the working region 7; and if the vision module identifies that the current image does not include the feature of grass, or include a feature of a road surface, it may be determined that the smart lawnmower is in a non-working region.

A manner in which the vision module identifies the working region is not specifically limited in this embodiment of this application. For example, the vision module may perform identification according to feature points such as texture, color, surface flatness, and degree of clutter of an object in the image. Using texture as an example, the vision module may identify texture of an object in the current image, and compare an identification result (for example, a feature value) with a preset range. If the identification result is within the preset range, it indicates that the self-moving device is in the working region 7; and if the identification result is outside the preset range, it indicates that the self-moving device is in the non-working region.

A position of the vision module is not specifically limited in this embodiment of this application. For example, the vision module may be disposed at a front end of the self-moving device, to obtain an environment image in a forward direction of the self-moving device, so as to ensure that the self-moving device does not work beyond the boundary 6 in the forward direction. In another example, the vision module may alternatively be disposed at a back end of the self-moving device, to obtain an environment image in a backward direction of the self-moving device, so as to ensure that the self-moving device does not work beyond the boundary in the backward direction. In still another example, the vision module may alternatively be disposed on two sides of the body of the self-moving device.

There may be one or more vision modules disposed on the self-moving device. For example, vision modules may be disposed on both the front and back ends of the self-moving device. In another example, vision modules may be disposed on the front end and the two sides of the self-moving device. A plurality of vision modules are disposed, so that the accuracy of boundary identification can be improved.

The identification result of the current image can provide a basis for path planning of the self-moving device, and ensure the working safety of the self-moving device. The path planning may be implemented by the control module in the self-moving device, and the control module may be disposed on a main board of the self-moving device. The control module may be a processor of the self-moving device, and the processor may be a microcontroller unit (MCU). When the obtained current image is different, a moving path of the self-moving device controlled by the control module may also be different.

For example, if the current image indicates that the self-moving device is in the working region 7, the control module may control the self-moving device to continue to move forward and work. In another example, if the current image indicates that the self-moving device is outside the working region (that is, in the non-working region), the control module may control the self-moving device to return to the working region, or control the self-moving device to move to a charging station, or control the self-moving device to stop working, or the like. In still another example, if the current image includes a region that does not need to be processed, the control module may control the self-moving device to bypass the region. The region that does not need to be processed may include at least one of the following: a region that has been processed, a boundary, a marker, an obstacle, and the like. Although the embodiments discussed here describe a vision module and an image acquisition apparatus, it is to be understood that these features are not required to capture images in the visible range of wavelengths of the electromagnetic spectrum, and may instead capture images in other wavelengths, such as radio, microwave, and infrared for example. In this respect, the image acquisition apparatus is not required to be a camera or similar optical imaging apparatus, and may instead be a sensor capable of transducing the environment of the self-moving device to determine whether the self-moving device is in the working region 7 or a non-working region. Such a sensor may include a radar or LIDAR device, for example.

For a self-moving device provided with a vision module, how to improve working safety of the self-moving device is a problem to be resolved urgently at present.

Based on this, an embodiment of this application provides a method for controlling a self-moving device, which may improve working safety of a self-moving device. The self- moving device in this embodiment of this application may include a vision module and a control module. The vision module may be any vision module described above, and the control module may be any control module described above. As shown in FIG. 4, the method may include steps S410 to S420.

S410: A vision module sends a plurality of types of indication information to a control module, where the indication information is used for indicating working statuses of the vision module.

S420: The control module may detect whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information.

The plurality of types of indication information may include verification information and/or heartbeat information, and the like. Correspondingly, the working statuses of the vision module may include an identification status of the vision module and/or a communication status of the vision module, and the like. When the indication information is verification information, the verification information may be used for indicating the identification status of the vision module. When the indication information is heartbeat information, the heartbeat information may be used for indicating the communication status between the control module and the vision module.

The verification information may be used for indicating a verification result of the vision module for a verification image. According to this embodiment of this application, the verification image may be pre-stored in the self-moving device (for example, the vision module), and the vision module may identify the verification image, and compare the identification result with a pre-stored result corresponding to the verification image to obtain the verification result. This process is explained in more detail later.

The heartbeat information may refer to a heartbeat packet between the control module and the vision module. According to this embodiment of this application, a communication link may be added between the control module and the vision module, and the control module may determine whether the vision module is abnormal by monitoring heartbeat information of the vision module. The heartbeat information may refer to a heartbeat packet sent by the vision module to the control module, or may refer to a reply packet of the vision module to the heartbeat packet sent by the control module.

S430: The control module controls the self-moving device to stop working in a case that an anomaly of the vision module is detected.

The anomaly of the vision module may include at least one of a communication anomaly and an identification anomaly. The communication anomaly may refer to a communication anomaly between the vision module and the control module, for example, the control module cannot receive the heartbeat information sent by the vision module. The identification anomaly may refer to an anomaly of the identification result of the vision module for the verification image.

Stopping working may include at least one of the following: stopping movement, stopping working of the task execution module, and shutdown. Controlling the shutdown of the self-moving device can further improve the safety of the self-moving device.

It can be learned from the foregoing description that the vision module plays a vital role in the work of the self-moving device. Whether the vision module is abnormal or not affects the control of the self-moving device by the control module. If the vision module is abnormal, the self-moving device may move randomly and run out of the boundary. Meanwhile, if the self-moving device continues to work when the vision module is abnormal, safety problems may be caused to a user, or damage may be caused to the working region 7. Therefore, considering the foregoing important role of the vision module, according to this embodiment of this application, the status of the vision module is detected, and in a case that an anomaly of the vision module is detected, the self-moving device is controlled to stop working, thereby preventing the self-moving device from at least one of moving outside the boundary (or moving outside the boundary too far), causing safety problems to the user, and causing damage to the working region 7, and improving working safety of the self-moving device.

The control module may detect whether the vision module is abnormal according to a receiving status of any of the plurality of types of indication information. For example, the control module may detect whether the vision module is abnormal according to a receiving status of the verification information, and control the self-moving device to stop working when an anomaly of the vision module is detected based on the verification information. In another example, the control module may detect whether the vision module is abnormal according to a receiving status of the heartbeat packet, and control the self-moving device to stop working when an anomaly of the vision module is detected based on the heartbeat packet.

The control module may alternatively detect whether the vision module is abnormal according to receiving statuses of the plurality of types of indication information. For example, the plurality of types of indication information may include both verification information and heartbeat information, so that the control module may detect whether the vision module is abnormal according to receiving statuses of both the verification information and the heartbeat information. Specifically, the control module may control the self-moving device to stop working in a case that an anomaly of the vision module is detected based on any of the foregoing indication information.

According to this embodiment of this application, through the dual-channel detecting manner with the verification information and the heartbeat information, the working status of the vision module is detected simultaneously from a software dimension of image identification and a hardware dimension of the communication link, which is conducive to improving the working safety of the self-moving device.

The following describes the verification information and the heartbeat information in detail with reference to FIG. 5 and FIG. 6.

Referring to FIG. 5, the plurality of types of indication information may include verification information, and the verification information may be used for indicating a verification result of the vision module for a verification image. A method shown in FIG. 5 includes steps S510 to S530.

S510: A vision module may send verification information to a control module.

S520: The control module may detect whether identification of the vision module is abnormal according to a receiving status of the verification information.

S530: The control module may control the self-moving device to stop working in a case that an anomaly of the vision module is detected.

During working of the self-moving device, the vision module may capture a current image at a position of the self-moving device, and identify the current image through an AI model to determine whether or not the self-moving device is in the working region 7. However, during working of the vision module, parameters in the AI model may change due to some factors. For example, in a high temperature environment or when computing power of a processor changes, the parameters in the AI model may change. If the parameters of the AI model change, an identification result of the AI model for the current image may be affected. If the AI model is still used to identify the current image, an erroneous identification result may be obtained, thus affecting the working safety of the self-moving device.

Based on this, one or more verification images may be pre-stored in the self-moving device (or the vision module) to verify the AI model that is used by the self-moving device. Not only the verification image is pre-stored in the self-moving device, but also a pre-stored result corresponding to the verification image is pre-stored. During working of the self- moving device, the AI model may be verified by using the verification image to ensure the working safety of the self-moving device. This discussion corresponds to the training and/or checking of the second AI model discussed above.

Specifically, the verification image may be inputted into the AI model to obtain an identification result. Further, whether the identification result is consistent with the pre-stored result corresponding to the verification image may be verified, to obtain the verification result. The verification result is thus simply the result of a comparison made between the identification result, produced by the AI model with respect to the verification image, and the pre-stored result. When the identification result matches the pre-stored result, the verification result indicates that the AI model successfully identified the verification image/ features of the verification image. On the other hand, when the identification result does not match the pre-stored result, the verification result indicates that the AI model has unsuccessfully identified the verification image/ features of the verification image.

In other words, if the identification result is consistent with the pre-stored result corresponding to the verification image, the verification result indicates that identification of the AI model is normal; and if the identification result is inconsistent with the pre-stored result corresponding to the verification image, the verification result indicates that identification of the AI model is abnormal, that is, working of the vision module is abnormal. For example, if the verification image is an image in the working region, and the identification result of the vision module for the verification image is also the working region, the identification result of the vision module for the verification image is consistent with the pre-stored result. In this case, the outputted verification information may be 1. In another example, if the verification image is an image in the working region, and the identification result of the vision module for the verification image is the non-working region, the identification result of the vision module for the verification image is inconsistent with the pre-stored result. In this case, the outputted verification information may be 0.

The verification image may include an image in the working region and/or an image in the non-working region, there may be one or more images in the working region, and there may be one or more images in the non-working region. It is to be understood that the verification images do not have to be from the same working region 7 as the self-moving device is moving in. The verification images may be pre-stored prior to deployment of the self-moving device in the working region 7. A pre-stored result corresponding to the image in the working region is recorded as a first pre-stored result, and a pre-stored result corresponding to the image in the non-working region is recorded as a second pre-stored result. The vision module may verify only the image in the working region, or only the image in the non-working region, or verify the image in the working region and the image in the non-working region simultaneously. Certainly, the vision module may alternatively randomly select an image for verification, and this is not specifically limited in this embodiment of this application.

For example, in the process of verifying the AI model by using the verification image, the vision module may input the image in the working region into the AI model to obtain a first identification result; and/or input the image in the non-working region into the AI model to obtain a second identification result. The vision module may then verify whether the first identification result is consistent with the first pre-stored result; and/or verify whether the second identification result is consistent with the second pre-stored result.

After obtaining the verification result, the vision module may generate verification information according to the verification result, and send the verification information to the control module. The control module may detect whether the vision module is abnormal according to the verification information. For example, the control module may determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the first identification result is inconsistent with the first pre-stored result and/or the second identification result is inconsistent with the second pre-stored result.

If the identification result is inconsistent with the pre-stored result, the AI model in the vision module may be abnormal. If the AI model is still used to identify the current image, an erroneous identification result is obtained, and the working safety of the self-moving device is affected. Therefore, when the identification result for the verification image is inconsistent with the pre-stored result, controlling the self-moving device to stop working can improve the working safety of the self-moving device. Safety-critical features of the self- moving device, such as a blade in the instance of a smart lawnmower 1, may be shut down. Furthermore, walking/movement motors may also be shutdown. Before restarting the blade and/or motors, it may be necessary to obtain a further identification result that is consistent with the pre-stored result.

In addition to the case that the identification result is inconsistent with the pre-stored result, the control module may alternatively determine that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range. The first preset time range may be greater than a time interval between sending of the verification information by the vision module. If the vision module does not obtain the verification result, the vision module may not send the verification information to the control module. In this case, the control module may determine that the vision module (for example, an AI operating system or software) is abnormal. Further, the control module may control the self-moving device to stop working, so as to improve the working safety of the self-moving device.

To further ensure the working safety of the self-moving device, in this embodiment of this application, the AI model may be periodically verified. For example, each time before identifying the current image, the vision module verifies the AI model, and when the AI model is normal, the AI model is used to identify the current image. Certainly, the vision module may alternatively verify the AI model each time after identifying the current image, to determine the accuracy of the identification result for the current image. Optionally, to save computing power and reduce operating power consumption, the AI model may alternatively be periodically verified at a longer time interval. A person skilled in the art may make a selection according to actual situations, and this is not limited in this application.

To improve the safety of the self-moving device, the vision module and the AI model used to identify a current image captured by the vision module may be verified regularly, in a period that is dependent on the speed of the self-moving device and the working frame-rate of the AI model. In particular, the AI model is configured to analyze a certain number of frames per unit of time. For example, the AI model may analyze five frames captured by the image acquisition apparatus 110 of the vision module 11 every second. It is to be understood that the AI model could analyze more or fewer frames every second. The working frame rate of the AI model is the rate at which the AI model analyzes an image presented to it, to determine the identification result. The working frame rate is the same as the frame rate of the image acquisition apparatus 110, or rather, the AI model is configured to analyze frames at the same rate as they are acquired by the image acquisition apparatus 110. The frame rate of the image acquisition apparatus may also be dynamically adjusted to match the working frame rate of the AI model. Having the frame rate of the image acquisition apparatus match the working frame rate of the AI model ensures that no frames acquired by the image acquisition apparatus are skipped by the AI model, and that the AI model analyzes the frames in real or near real time. This ensures that the stream of frames that are analyzed by the AI model correspond to consecutively acquired image frames.

A verification image may be included periodically in the frames being analyzed by the AI model in order to obtain an identification result and subsequently the verification result. For example, a verification image may be included in every fifth frame analyzed by the AI model. This means that four captured images are analyzed from four frames obtained by the image acquisition apparatus for every one verification image provided to the vision module. Using the above example of a frame rate of 5 frames per second, this means that one verification image is analyzed by the AI model to obtain an identification result and subsequently a verification result every second to verify the proper working and function of the AI model in performing its task.

As noted above, the frequency of analyzing a verification image may be dependent on the frame rate of the image acquisition apparatus, the working frame rate of the AI model, and the speed of the self-moving device. Using the same example as above, if the speed of the self-moving device is 1 meter per second, 1 m/s, and the working frame rate of the AI model is 5 frames per second, 5 fps, then the difference between the position of capture of each frame, or in other words, the distance travelled by the self-moving device between each frame, is equal to/approximately equal to 0.2 meters, calculated by dividing the speed by the working frame rate. If only one frame of the five frames per second is devoted to verifying the AI model using a verification image, it follows that the AI model is only verified once for every meter travelled by the self-moving device. Safety issues may arise from this configuration in certain scenarios. For example, when the self-moving device is a smart lawnmower, only verifying the AI model for every meter travelled may be deemed to be unsafe, because the smart lawnmower could leave a working region and enter a non-working region within this distance. If the AI model was not working correctly, and was misclassifying captured images, it could cause damage to itself, the non-working region, or other features of the environment before the problem is identified in a later verification of the AI model.

To remedy this potential issue, several parameters may be modified. Firstly, the working frame rate of the AI model may be increased, and/or more frames may be provided per measure of time for performing verification on verification images. For example, the working frame rate of the AI model and the frame rate of the image acquisition apparatus 110 may be increased from 5 fps to lOfps. Maintaining the ratio of 1 :4 for verification images to captured images to be provided to the AI model, would allow two frames per second to be provided for performing verification of the AI model on two verification images. This has the effect of halving the distance travelled between analyses of verification images, such that, at the same speed of lm/s, the AI model is verified every 0.5m that the self-moving device travels. The ratio of verification images to captured images may also be modified to reduce this distance further. For example, if the ratio is increased from 1:4 to 3:7, three verification frames are provided to the AI model every second, meaning, at a speed of lm/s, the distance travelled between verifications of the AI model is 0.33m. It is to be understood that any combination of parameters may be modified to reduce the distance travelled between verifications of the AI model to a level that is appropriate for the particular application of the self-moving device. It is to be understood that in some applications, verification of the AI model may be required at least once every second, but in other applications, a verification every minute, hour, day, or each time the self-moving device is powered on, may also be appropriate. Similarly, the distance travelled by the self-moving device per verification may be appropriate if under 0.1m, 0.2m, 0.5m, lm, 5m, 10m, 50m or 100m or more, depending on the application.

Whilst increasing the frame rate of the AI model and the image acquisition apparatus, and/or adjusting the ratio of verification images to captured images to be provided to the AI model is beneficial in that it provides more frequent verifications, these adjustments may require a greater computational burden, since the AI module and the image acquisition apparatus are required to function faster. This may make the equipment associated with these features more expensive.

To avoid the requirement of overly complex and expensive equipment, a further parameter - the speed of the self-moving device - may be modified to reduce the distance travelled by the self-moving device between verifications of the AI model. For example, by reducing the speed of the self-moving device from lm/s to 0.5m/s, the distance travelled between verifications of the AI model is halved without adjusting any further parameters such as the frame rate of the image acquisition apparatus and the AI model, or the frequency of interjected verification images. If the frame rate is maintained at 5fps, with the ratio of 1 :4 for verification images to captured images to be provided to the AI model, each verification is performed after 0.5m of distance moved by the self-moving device rather than lm.

It is to be understood that a combination of adjustments can be made to the above parameters to improve the safety of the self-moving device. For example, the speed can be reduced and the frame rate can be increased. This has a compound effect on reducing the distance travelled by the self-moving device. For example, if the frame rate is doubled from 5fps to lOfps, and the speed is halved from 1ms to 0.5ms, for a ratio of 1:4 for verification images to captured images to be provided to the AI model, a verification is performed every 0.25m travelled. The ratio may also be increased to allow for more frequent verifications.

The speed of the self-moving device may be adjusted in real-time. In this case, to ensure that the distance travelled between verifications of the AI model, using the verification images, does not increase, the working frame rate of the AI model and the frame rate of the image acquisition apparatus may be adjusted in real or near real time to compensate for the change in speed. The speed may be directly sampled by a sensor or the like on the self- moving device. The sampled speed may then be input to the control module, which subsequently computes the necessary frame rate required to maintain the distance travelled between verifications of the AI model to acceptably low values. The control module may then instruct the AI model and the image acquisition apparatus to start obtaining and analyzing images according to the computed frame rate. This implementation is adaptive and beneficial in that it can be used more efficiently than maintain a constant frame rate. In particular, when the self-moving device is moving relatively slowly, there is no need for high frame rates in order to keep the distance travelled by the self-moving device to acceptably low values. Thus, the frame rate may be reduced at lower movement speeds, which is computationally efficient.

The control module may store one or more maximum distance thresholds that indicate the maximum acceptable distance that the self-moving device may move between verifications of the AI model. If speed is adjusted as noted above, the control module may compute a corresponding adjusted working frame rate that is required to keep the distance travelled by the self-moving device between verifications equal to/below the maximum distance threshold.

A size of the self-moving device may also be a parameter that is factored into the adjustments made to the parameters to improve the safety of the self-moving device. The size of the device may refer to an absolute size of the device or a size relative to a distance between the image acquisition apparatus and an end of the self-moving device. The end of the self- moving device may be the end in the direction of travel. For example, the front end may be the relevant end for determining the relative size of the self-moving device when the self- moving device is moving forwards. The size of the self-moving device may also be determined by taking into account the distance between the front wheels of the self-moving device, the distance from the front wheels to the absolute front end of the self-moving device, or any combination of such distances discussed here. The setting of the maximum distance threshold may be dependent on the absolute or relative size of the self-moving device, such that larger sizes mean the maximum distance threshold is reduced in order to increase safety for such larger devices. The speed and frame rate may also be dependent on the absolute/relative size of the self-moving device, such that larger sizes require lower speeds and/or higher frame rates.

Generally, the size and speed of the self-moving device share a positive correlation with the number of verifications that are required to maintain safety, such that more speed or larger self-moving devices require more frequent verifications. The frame rate or the working frame rate of the AI model has a negative correlation with verifications, in that a higher frame rate/ working rate allows the ratio of verification images to captured images to be provided to the AI model to decrease, such that more captured images are analyzed for each verification image. Since the frame rate is increased, the actual number of verifications per unit of time can still be maintained.

The frame rate of the image acquisition apparatus and the working frame rate of the AI model in analysis frames may also be adjusted dynamically dependent on the motion of the self-moving device. For example, when the motion of the self-moving device is a rotational or turning motion, the frame rate may be adjusted to increase to account for the angular velocity or rotational speed of the self-moving device. Performing a turn or rotation usually results in the environment in the field of view of the image acquisition apparatus changing rapidly when compared to movement in a straight line. The environment in the field of view of the image acquisition apparatus may completely change in a relatively short amount of time, which means there is less probability that features found in previous frames exist in following frames, It is thus important that the AI model is verified regularly during a turn or rotation to ensure that the AI model correctly identifies its newly observed environment. To provide more verifications per measure of time, the frame rate is increased and/or the frequency of interjected verification frames is increased. The control module of the self- moving device may automatically increase these parameters when a turn instruction is provided to the motors of the self-moving device, or alternatively, sensors can directly sample the degree of rotation/turn in the movement of the self-moving device. These sensors may then provide the sampled rotation data to the control module, which is configured to respond accordingly by adjusting the frame rate at a rate that is proportional to the angular velocity.

In some embodiments, a braking distance is also considered in the computation and adjustment of frame rates and speed of the self-moving device. In order to obtain a value or data for the braking distance, a look-up table or database of speed against braking distance may be stored in memory accessible by the control module. The control module may then, from knowledge of the speed of the self-moving device, obtained via sensors or otherwise, look up the value of the speed in the look-up table to obtain a corresponding braking distance. The braking distance may be affected by other environmental parameters such as slope, weather and temperature. For example, in wet conditions, the braking distance may increase. In order to negate the effect of these environmental parameters, each of these environmental parameters may be sensed by one or more sensors on board the self-moving device, or data corresponding to these parameters may be communicated to the self-moving device. The obtained braking distance value from the look-up table may then be multiplied by an environmental parameter scaling factor, whereby the environmental parameter scaling factor includes one or more components corresponding to one or more of the environmental parameters such as slope, weather, temperature and the like. The environmental parameter scaling factor modifies the braking distance value obtained from the look-up table to provide an environment-adjusted braking distance value. A benefit of using the environment-adjusted braking distance value is that it is more likely to be accurate and comparable to a real-world braking distance.

Alternatively, a braking distance value may be obtained via a braking distance calibration test during an initial calibration of the self-moving device, for example, when the self-moving device leaves a charging station or when the self-moving device is powered on. Alternatively, the braking distance calibration test may be performed periodically. The braking distance calibration test includes determining a braking distance autonomously, by the self-moving device, using one or more sensors on board the self-moving device. At a first time value, known to the control module of the self moving device, the control module sends an instruction to the brakes of the self-moving device to brake from a predefined initial control speed. The control module then records a second time value, when the self-moving device is determined as coming to rest and becoming stationary. Using one or more of a positioning sensor, such as GPS, an inertial measurement unit (IMU), accelerometers, and/or the vision module, the control module is configured to determine a braking distance value corresponding to the distance travelled by the self-moving device between the first time value and the second time value. For example, using the time data, initial speed, and deceleration data (from an IMU or accelerometer), it is possible to calculate a braking distance. Similarly, from using a series of frames obtained via the vision module, it is possible to measure a change in distance/triangulation with respect to observed features in each frame between the first time value and the second time value to obtain a braking distance.

It is to be understood that, although the term 'braking distance' is used above, it is not necessary for the self-moving device to include brakes. The braking distance may instead simply refer to the distance travelled by the self-moving device after turning off or stopping the walking/movement motors.

The braking distance may also be affected by the weight of the self-moving device. In this respect, calculation of the braking distance may include a weight factor. The weight factor may be estimated according to a detected absolute/relative size of the self-moving device, since weight generally correlates with size.

Once a braking distance value has been obtained by any of the methods set out above, or otherwise, the braking distance value may, in some embodiments, be incorporated into the maximum distance threshold. As notes above, the maximum distance threshold indicates the maximum distance between verifications of the AI model, to ensure safe operation of the self-moving device. As an example, the maximum distance threshold may be lm. This means that the parameters of the self-moving device such as the working frame rate and/or the speed of the self-moving device should be selected/operated such that the self-moving device travels a maximum of lm between verifications of the AI model. When the maximum distance threshold incorporates the braking distance value, which for example may be 0.5m, the maximum distance threshold is adjusted to produce an adjusted distance threshold. The adjusted distance threshold is equal to the maximum distance threshold minus the braking distance value. In this example, the adjusted distance threshold is therefore 0.5m. The control module is configured to compute this adjusted distance threshold and modify the parameters of the self-moving device, such as the AI model working frame rate, the frame rate of the image acquisition apparatus, and the speed of the self-moving device, such that the distance travelled between verifications of the AI model is equal or less than the adjusted distance threshold. In this example, reducing the distance from lm to 0.5m to compensate for the braking distance of 0.5m could be done by halving the speed, or doubling the working frame rate, for example. It is to be understood that any combination of parameter modifications could be done to reduce the distance between verifications, and that different values for the maximum distance threshold, the braking distance, and the adjusted distance threshold are envisaged.

The braking distance may also be used when turning at the boundary of the working region. When an image captured by the image acquisition apparatus is identified by the vision module as including a boundary, a verification image may be inteijected into the stream of frames being processed by the AI model to verify the AI. The interjection of this verification image may be done before a braking distance from the observed boundary, to ensure that the AI model is working properly (verified), before responding to the presence of the boundary. Inteijecting the verification frame into the stream of frames before the braking distance from the boundary ensures that the control module has time to perform an action in response to the success or failure of the verification before the self-moving devices reaches the boundary.

Although the above description relating to verification of the AI model refers to a working AI frame rate, it is to be understood that this parameter may be any parameter indicative of the processing power available to the AI model. As noted above, the working frame rate of the AI model corresponds to the number of image frames that the AI model is able to analyze and determine identification and ultimately verification results for. This is intrinsically linked and dependent on the processing power available to the AI model afforded by the vision module of the self-moving device. Instead of the working frame rate, this parameter may be referred to as computational power or random access memory (RAM) afforded to the AI model. Thus, adjusting this parameter to analyse frames faster may include providing more computational power or RAM to the AI model. Similarly, the AI model may be any suitable type of AI model for performing verification and image analysis, such as a neural network, or AI algorithm, for example.

Furthermore, although the above description focuses on providing the AI model in a vision module, and providing the AI model with visual data collected from an image acquisition apparatus, it is to be understood that embodiments are not limited to such aspects. In particular, the image acquisition apparatus is not required to collect data in the optical or visible wavelengths of the electromagnetic spectrum as noted above, and may instead collect data at other wavelengths. Furthermore, the image acquisition apparatus may be substituted for any other sensor capable of detecting features in the environment of the self-moving device. For example, a radar, or LIDAR may be used instead of the image acquisition apparatus. In the case of a radar, detected waveforms may be analyzed in the same way as the frames mentioned above, using the AI model to determine whether a detected feature is lawn/not lawn. Similarly, pre-known verification waveforms may be provided to the AI model, in order to verify whether the AI model is functioning correctly.

Embodiments may thus include an AI model that is trained according to the sensor it is to be paired with, for specific applications. If for example, the AI model is to be used with a radar, it is trained with radar waveforms, and, when in use, verified with radar waveforms. If the AI model is to be used with a LIDAR, the AI model is trained with LIDAR waveforms and, when in use, verified with LIDAR waveforms. This applies to any type of suitable sensor.

The above embodiments concerning verification of the AI model advantageously check in a periodic manner whether the AI model is functioning correctly. The AI model may, for several reasons, function incorrectly without prior warning. This could be related to a software or hardware fault. For example, RAM/memory being used by the AI model may become corrupted or otherwise unusable. Verification of the AI model determines that the AI model is functioning correctly or incorrectly, based on its response to verification images or other verification sensor data that has a known or expected result. When the identification result produced by the AI model does not match the known or expected result, the verification result indicates that the AI model is not working correctly. At this instance, action is taken to ensure safety is maintained. The action may include shutting down the self-moving device, restarting the self-moving device, optionally in a safe-mode to debug the AI model, stopping the self-moving device by applying the brakes and/or deactivating the walking/movement motors, and stopping any lawn-cutting features such as a blade. Verification is a means of ensuring proper function of the AI model and thus proper function of the self-moving device. It is thus important that the verification process occurs regularly, according to a period that is informed by several parameters. In particular, to ensure that the self-moving device does not move a significant distance when the AI model may not be working correctly, a verification frame or verification sensor data is interjected in the stream of frames or stream of sensor data being analyzed by the AI model in a manner and frequency that is proportional to parameters such as the speed of the self-moving device and the frame rate/ sensing rate of the image acquisition apparatus/sensor. To reduce the distance that can be travelled between such inteijections of verification sensor data, the speed of the self moving device may be reduced, and/or the frequency of inteijections or the ratio of interjections to sensor data may be increased. This verification process ultimately routinely tests the AI model to ensure it is working accurately and to ensure safety of the self moving device is maintained.

The embodiments including the self-moving device discussed above may be subject to particular constraints or values of working parameters of the self-moving device to ensure safety is maintained in a manner that meets necessary safety requirements. For example, the speed of the self-moving device may be in the range of 0.35 to 0.5 m/s, a shooting distance of the image acquisition apparatus 110 may be in the range of 0.6 to 0.9 m, such as 0.7m, and the frame rate may be 10 frames per second. Therefore, according to the method set out above, if an abnormal status of the self-moving device is detected, the self-moving device may only move a short distance between consecutive verification processes, such as less than 0.1 m. This distance is acceptable in terms of safety requirements of the self-moving device. It is to be understood that other parameters may be selected such that they are within acceptable safety requirements, and further, the values of the speed, shooting distance, and frame rate may be set differently based on different safety requirements.

Referring to FIG. 6, the plurality of types of indication information may include heartbeat information, and the heartbeat information may be used for indicating a communication status between the control module and the vision module. A method shown in FIG. 6 includes steps S610 to S630.

S610: A vision module may send heartbeat information to a control module.

S620: The control module may detect whether communication of the vision module is abnormal according to a receiving status of the heartbeat information.

S630: The control module may control the self-moving device to stop working in a case that an anomaly of the vision module is detected.

The heartbeat information may be a heartbeat packet (or referred to as a heartbeat package) sent by the vision module to the control module, or may be a reply packet of the vision module to the heartbeat packet sent by the control module. It may be understood that, the heartbeat packet is information sent by a sending end to a receiving end at a certain time interval, and the information may be code. The two cases are described separately below.

For example, the vision module may send a first heartbeat packet to the control module. The control module may receive the first heartbeat packet to determine whether the vision module is normal. The control module may determine that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range. The first preset time range may be greater than a time interval between sending of the first heartbeat packet by the vision module. In this case, the control module may control the self-moving device to stop working, to ensure the working safety of the self- moving device.

In another example, the control module may send a second heartbeat packet to the vision module, and after receiving the second heartbeat packet, the vision module may send a reply packet for the second heartbeat packet to the control module. The control module may determine whether the communication of the vision module is abnormal according to a receiving status of the reply packet. The control module may determine that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range. The third preset time range may be greater than a time interval between sending of the second heartbeat packet by the control module.

According to this embodiment of this application, whether the vision module is abnormal may be detected based on the heartbeat information. If the control module cannot detect the heartbeat information sent by the vision module, it indicates that there may be a problem with the connection between the vision module and the control module. In this case, controlling the self-moving device to stop working can ensure the working safety of the self-moving device. A manner of communication between the vision module and the control module is not specifically limited in this embodiment of this application. The vision module may send indication information to the control module in a wired manner, for example, via an electrical signal. The vision module and the control module may be electrically connected. Alternatively, the vision module may send indication information to the control module in a wireless manner, such as via Bluetooth and Wi-Fi. For the heartbeat information and the verification information, the vision module may send the heartbeat information and the verification information in the same manner or in different manners. For example, the vision module may send the heartbeat information and the verification information through electrical signals. In another example, the vision module may send the verification information through an electrical signal, and send the heartbeat information through a wireless signal.

In addition, even if there is no anomaly in the vision module, the self-moving device may move to the non-working region. The self-moving device in the non-working region may also cause safety problems or cause damage to the non-working region. Based on this, according to this embodiment of this application, the self-moving device may alternatively be controlled to stop working when the self-moving device is in the non-working region, to further improve the working safety of the self-moving device.

For example, the vision module may capture a current image at a position of the self- moving device, and input the current image into the AI model to identify the current image, thereby determining the position of the self-moving device. If the current image includes a feature of an object (for example, grass) in the working region 7, it may be determined that the self-moving device is in the working region 7. If the current image does not include a feature of an object (for example, grass) in the working region, it may be determined that the self-moving device is in the non-working region. The vision module may send an identification result to the control module, and the control module may control the self- moving device according to the identification result. If the vision module identifies that the self-moving device is in the non-working region, the vision module may send warning information to the control module, where the warning information is used for indicating that the vision module identifies that the self-moving device is in the working region 7. The control module may control the self-moving device to stop working in a case that the warning information is received.

The foregoing describes the method embodiments of this application in detail with reference to FIG. 1 to FIG. 6, and the following describes the apparatus embodiments of this application in detail with reference to FIG. 7 to FIG. 9. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, for parts that are not described in detail, reference may be made to the method embodiments above. FIG. 7 shows an apparatus for controlling a self-moving device according to an embodiment of this application. The apparatus 700 may include a control module 710, and the control module 710 may be any control module described above.

The control module 710 may be configured to: receive a plurality of types of indication information sent by a vision module, where the indication information is used for indicating working statuses of the vision module; detect whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information; and control the self-moving device to stop working in a case that an anomaly of the vision module is detected.

Optionally, in some embodiments, the plurality of types of indication information include verification information and/or heartbeat information, the verification information is used for indicating a verification result of the vision module for a verification image, and the heartbeat information is used for indicating a communication status of the control module and the vision module; the control module 710 may be configured to detect whether identification of the vision module is abnormal according to a receiving status of the verification information; and/or detect whether communication of the vision module is abnormal according to a receiving status of the heartbeat information.

Optionally, in some embodiments, the control module 710 may be configured to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that an identification result of the vision module for the verification image is inconsistent with a pre stored result corresponding to the verification image.

Optionally, in some embodiments, the control module 710 may be configured to determine that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range.

Optionally, in some embodiments, the heartbeat information comprises a first heartbeat packet; and the control module 710 may be configured to determine that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range.

Optionally, in some embodiments, the control module 710 may be configured to send a second heartbeat packet to the vision module; correspondingly, the heartbeat information comprises a reply packet for the second heartbeat packet; and the control module 710 may be configured to determine that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range.

Optionally, in some embodiments, in a case that no anomaly of the vision module is detected, the control module 710 may be configured to control the self-moving device to stop working in a case that warning information sent by the vision module is received, where the warning information is used for indicating that the vision module identifies that the self- moving device is in a non-working region.

FIG. 8 shows another apparatus for controlling a self-moving device according to an embodiment of this application. The apparatus 800 may include a vision module 810, and the vision module 810 may be any vision module described above.

The vision module 810 may be configured to: generate a plurality of types of indication information, where the indication information is used for indicating working statuses of the vision module; and send the plurality of types of indication information to a control module of the self-moving device, to cause the control module to detect whether the vision module is abnormal based on a receiving status of at least one of the plurality of types of indication information.

Optionally, in some embodiments, the plurality of types of indication information include verification information and/or heartbeat information, the verification information is used for indicating a verification result of the vision module for a verification image, and the heartbeat information is used for indicating a communication status of the control module and the vision module; and the vision module 810 may be configured to: generate the verification information; and send the verification information to the control module, to cause the control module to detect whether identification of the vision module is abnormal based on a receiving status of the verification information; and/or the vision module 810 may be configured to: generate the heartbeat information; and send the heartbeat information to the control module, to cause the control module to detect whether communication of the vision module is abnormal based on a receiving status of the heartbeat information.

Optionally, in some embodiments, the vision module 810 may be configured to: input the verification image into an AI model to obtain an identification result; verify whether the identification result is consistent with a pre-stored result corresponding to the verification image, to obtain the verification result; generate the verification information according to the verification result; and send the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the identification result is inconsistent with the pre-stored result corresponding to the verification image.

Optionally, in some embodiments, the verification image includes an image in a working region and an image in a non-working region, a pre-stored result corresponding to the image in the working region is a first pre-stored result, and a pre-stored result corresponding to the image in the non-working region is a second pre-stored result; and the vision module 810 may be configured to: input the image in the working region into the AI model to obtain a first identification result; and/or input the image in the non-working region into the AI model to obtain a second identification result; verify whether the first identification result is consistent with the first pre-stored result; and/or verify whether the second identification result is consistent with the second pre-stored result; and send the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the first identification result is inconsistent with the first pre-stored result and/or the second identification result is inconsistent with the second pre-stored result.

Optionally, in some embodiments, the vision module 810 may be configured to send the verification information to the control module, to cause the control module to determine that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range.

Optionally, in some embodiments, the heartbeat information includes a first heartbeat packet; and the vision module 810 may be configured to send the first heartbeat packet to the control module, to cause the control module to determine that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range.

Optionally, in some embodiments, the vision module 810 may be configured to receive the second heartbeat packet sent by the control module; correspondingly, the heartbeat information includes a reply packet for the second heartbeat packet; and the vision module 810 may be configured to send the reply packet to the control module, to cause the control module to determine that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range.

Optionally, in some embodiments, in a case that no anomaly of the vision module is detected by the control module, the vision module 810 may be configured to: obtain a current image at a position of the self-moving device; input the current image into an AI model to obtain an identification result; and send warning information to the control module in a case that the identification result indicates that the self-moving device is in a non-working region, to cause the control module to control the self-moving device to stop working in a case that the warning information is received. FIG. 9 is a schematic block diagram of a self-moving device according to an embodiment of this application. The self-moving device 900 may include a vision module 910 and a control module 920. The vision module 910 may be any vision module described above, and the control module 920 may be any control module described above.

The vision module 910 may be configured to send a plurality of types of indication information to a control module, where the indication information is used for indicating working statuses of the vision module.

The control module 920 may be configured to: detect whether the vision module is abnormal according to a receiving status of at least one of the plurality of types of indication information; and control the self-moving device to stop working in a case that an anomaly of the vision module is detected.

Optionally, in some embodiments, the plurality of types of indication information include verification information and/or heartbeat information, the verification information is used for indicating a verification result of the vision module for a verification image, and the heartbeat information is used for indicating a communication status of the control module and the vision module; the vision module 910 may be configured to send the verification information to the control module; correspondingly, the control module 920 may be configured to detect whether identification of the vision module is abnormal according to a receiving status of the verification information; and/or the vision module 910 may be configured to send the heartbeat information to the control module; and correspondingly, the control module 920 may be configured to detect whether communication of the vision module is abnormal according to a receiving status of the heartbeat information.

Optionally, in some embodiments, the vision module 910 may be configured to: input the verification image into an AI model to obtain an identification result; verify whether the identification result is consistent with a pre-stored result corresponding to the verification image, to obtain the verification result; generate the verification information according to the verification result; and send the verification information to the control module correspondingly, the control module 920 may be configured to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the identification result is inconsistent with the pre-stored result corresponding to the verification image.

Optionally, in some embodiments, the verification image includes an image in a working region and an image in a non-working region, a pre-stored result corresponding to the image in the working region is a first pre-stored result, and a pre-stored result corresponding to the image in the non-working region is a second pre-stored result; and the vision module 910 may be configured to: input the image in the working region into the AI model to obtain a first identification result; and/or input the image in the non-working region into the AI model to obtain a second identification result; verify whether the first identification result is consistent with the first pre-stored result; and/or verify whether the second identification result is consistent with the second pre-stored result. Correspondingly, the control module 920 may be configured to determine that the identification of the vision module is abnormal in a case that the verification information is received and the verification information indicates that the first identification result is inconsistent with the first pre-stored result and/or the second identification result is inconsistent with the second pre-stored result.

Optionally, in some embodiments, the control module 920 may be configured to determine that the identification of the vision module is abnormal in a case that the verification information is not received within a first preset time range.

To improve the safety of the self-moving device, the vision module 910 and the AI model used to identify a current image captured by the vision module 910 may be verified regularly, in a period that is dependent on the speed of the self-moving device and the working frame- rate of the AI model. In particular, the AI model is configured to analyze a certain number of frames per unit of time. For example, the AI model may analyze five frames captured by the image acquisition apparatus 110 of the vision module 910 every second. It is to be understood that the AI model could analyze more or less frames every second. The working frame rate of the AI model is the rate at which the AI model analyzes an image presented to it, to determine the identification result. The working frame rate is the same as the frame rate of the image acquisition apparatus 110, or rather, the AI model is configured to analyze frames at the same rate as they are acquired by the image acquisition apparatus 110. The frame rate of the image acquisition apparatus may also be dynamically adjusted to match the working frame rate of the AI model. Having the frame rate of the image acquisition apparatus match the working frame rate of the AI model ensures that no frames acquired by the image acquisition apparatus are skipped by the AI model, and that the AI model analyzes the frames in real or near real time. This ensures that the stream of frames that are analyzed by the AI model correspond to consecutively acquired image frames.

In terms of verification, the same features, processes and modifications that are described above with reference to the method are also applicable here with reference to the apparatus. Such features are not limited to a particular embodiment.

Optionally, in some embodiments, the heartbeat information includes a first heartbeat packet, and the vision module 910 may be configured to send the first heartbeat packet to the control module. Correspondingly, the control module 920 may be configured to determine that the communication of the vision module is abnormal in a case that the first heartbeat packet is not received within a second preset time range.

Optionally, in some embodiments, before the vision module sends the heartbeat information to the control module, the control module 920 may be configured to send a second heartbeat packet to the vision module; correspondingly, the heartbeat information includes a reply packet for the second heartbeat packet, and the vision module 910 may be configured to send the reply packet to the control module; and correspondingly, the control module 920 may be configured to determine that the communication of the vision module is abnormal in a case that the reply packet is not received within a third preset time range.

Optionally, in some embodiments, in a case that no anomaly of the vision module is detected by the control module, the vision module 910 may be configured to: obtain a current image at a position of the self-moving device; input the current image into an AI model to obtain an identification result; and send warning information to the control module in a case that the identification result indicates that the self-moving device is in a non- working region; and the control module 920 may be configured to control the self-moving device to stop working in a case that the warning information is received. The foregoing descriptions are merely preferred embodiments of this application, but are not intended to limit this application.

Claims

1. A method for verifying correct operation of a self-moving device, the method comprising: periodically performing a control operation, the control operation including: receiving sensor data from one or more sensors; providing the sensor data as a sensor input to a control process; and controlling the self-moving device to move according to the control process; the method further comprising: periodically performing a verification operation, the verification operation comprising: providing a verification input to the control process, wherein the verification input is associated with a pre-stored expected result; processing, by the control process, the verification input to produce an identification result; comparing the identification result to the pre-stored expected result, wherein if the identification result does not correspond to the pre-stored expected result: determining that a working status of the self-moving device is abnormal; and controlling the self-moving device to stop moving.

2. The method of claim 1 wherein performing the verification operation comprises performing the verification operation according to a verification rate such that a maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations.

3. The method of claim 2, further comprising: detecting a speed of the self-moving device; and either: setting and/or adjusting the verification rate based on the speed of the self-moving device such that the maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations; and/or limiting or reducing the speed of the self-moving device below a maximum safe speed, such that the maximum threshold distance is not exceeded by the self-moving device moving between successive verification operations performed at the verification rate, the maximum safe speed determined in dependence on the verification rate and the maximum threshold distance.

4. The method of claim 3, wherein detecting the speed comprises detecting the speed of the self-moving device in real or near-real time.

5. The method of any of claims 2 to 4, wherein the maximum distance threshold is 0.1 m or less.

6. The method of any of claims 2 to 5, further comprising: adjusting the maximum threshold distance dependent on at least one of: a size of the self-moving device; an environmental parameter relating to a feature of the environment of the self-moving device; and/or a braking distance of the self-moving device.

7. The method of claim 6, comprising adjusting the maximum threshold distance dependent on a braking distance of the self-moving device by reducing the maximum threshold distance by the braking distance, wherein the method further comprises: obtaining the braking distance from a look-up table based on the speed of the self- moving device, wherein the look-up table stores a plurality of values for the braking distance, each value in the look-up table being associated with a speed or range of speeds; and/or performing a braking distance calibration, the braking distance calibration comprising: controlling the self-moving device to move at a predefined speed; instructing the self-moving device to brake or stop moving from at a first recorded time; determining that the self-moving device becomes stationary; determining a distance travelled from the first recorded time; and recording the distance travelled as the braking distance for the predefined speed.

8. The method of any of claims 6 to 7, further comprising detecting the environmental parameter relating to a feature of the environment of the self-moving device; comparing the environmental parameter to an environmental parameter threshold; and if the environmental parameter meets the environmental parameter threshold, decreasing the maximum threshold distance by an environmental parameter scaling factor.

9. The method of claim 8 wherein the environmental parameter includes one or more of: a moisture content of a surface on which the self-moving device moves; a gradient of the surface on which the self-moving device moves; and a temperature of the environment.

10. The method of any of claims 2 to 9, wherein performing of the control operation and performing of the verification operation are mutually exclusive, such that the verification operation is performed cyclically after every N control operations, where N is a positive integer.

11. The method of claim 10, further comprising adjusting the verification rate by increasing a control operation rate and/or decreasing the number of control operations N between verification operations.

12. The method of any preceding claim wherein the one or more sensors include an imaging sensor such that the method comprises: periodically performing the control operation, the control operation including: receiving image data from the imaging sensor; providing the image data as the sensor input to the control process, wherein the control process is an image recognition or image-processing process; processing, by the control process, the image data to obtain an image-processing result; controlling the self-moving device to move according to the image-processing result; the method further comprising: periodically performing the verification operation, the verification operation comprising: providing a verification image input to the control process, wherein the verification image input is associated with a pre- stored expected result; processing, by the control process, the verification image input to produce an identification result; comparing the identification result to the pre-stored expected result, wherein if the identification result does not correspond to the pre-stored expected result: determining that a working status of the self-moving device is abnormal; and controlling the self-moving device to stop moving.

13. The method according to claim 12, wherein the image-processing result is a determination of a type of surface of the environment in the vicinity and/or upcoming path of the self-moving device.

14. The method of claim 13 wherein the verification image input is an image of a type of surface and the pre-stored expected result is a result indicating the type of surface present in the verification image input.

15. The method of claim 13 or 14, wherein the determination of the type of surface is a determination of a lawn or non-lawn region present in the image data; and wherein the verification image input is an image of region including lawn or non-lawn, and the pre-stored expected result is a result indicating lawn or non-lawn respectively.

16. The method of any preceding claim, wherein controlling the self-moving device to stop moving comprises controlling the self-moving device to perform at least one of: stopping any motion completely; reducing movement speed below a safety-threshold speed; stopping motion of one or more actuators; and/or stopping a grass-cutting mechanism.

17. The method of any preceding claim wherein the control process is an artificial intelligence model.

18. A self-moving device, comprising: a sensor module comprising one or more sensors configured to obtain sensor data; a motor module configured to move the self-moving device; and a control module, communicatively coupled to the sensor module and the motor module; wherein the control module is configured to perform the method of any of claims 1 to 17.

19. The self-moving device of claim 18, wherein the one or more sensors include one or more of: an imaging sensor, a radar sensor, a light detection and ranging LiDAR sensor, and/or an infrared sensor.

20. The self-moving device of claim 18 or claim 19, wherein the self-moving device is a robot lawnmower comprising a cutting instrument configured to cut lawn.

21. The self-moving device of claim 20 wherein if the identification result does not correspond to the pre-stored expected result, the control module is configured to control the self-moving device to stop moving by being configured to: cause the motor module to stop moving the self-moving device; and/or cause the cutting instrument to disengage a lawn and/or stop actuating.

22. A computer readable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any of claims 1 to 17.