WO2023217193A1

WO2023217193A1 - Robot and method for robot to recognise fall

Info

Publication number: WO2023217193A1
Application number: PCT/CN2023/093320
Authority: WO
Inventors: 骆张强; 许鲤蓉
Original assignee: 神顶科技（南京）有限公司
Priority date: 2022-05-10
Filing date: 2023-05-10
Publication date: 2023-11-16
Also published as: CN117064255A

Abstract

A robot and a method for the robot to recognise a fall, relating to the technical field of artificial intelligence. The method for a robot to recognise a fall comprises: a robot moves position and executes detection of a first target part of a foot sole or a shoe sole, and records the position and posture of the detected first target part (101); on the basis of the position and posture of the first target part, controlling the movement position and/or changing the posture of the robot so as to execute detection of other target parts, and recording the positions of the detected other target parts (102); and, on the basis of the positions of the detected target parts, determining whether the current object is in a fall state (103). Automatic fall detection and recognition by the robot can be implemented, better monitoring the safety of family members whilst taking care of cleaning.

Description

Robots and methods for robots to recognize falls

Technical field

This application relates to the field of artificial intelligence technology, specifically to robots and robot fall recognition technology.

Background technique

With the development of artificial intelligence and the improvement of living standards, people have higher and higher requirements for smart home appliances, and various robots have emerged. The sweeping robot is a type of robot. It is a smart home appliance that can clean the indoor floor environment and bring a certain degree of convenience to people's lives. However, existing robots can only achieve cleaning functions and do not have Some other safety monitoring functions, such as fall detection and alarm, are especially targeted at scenarios where the elderly live alone or only the elderly and children are at home, so they are not fully intelligent and convenient. In addition, most of the existing safety detection functions or fall detection functions are implemented using some gravity sensors or cameras with higher field of view, but these are not suitable for sweeping robots.

Contents of the invention

The purpose of this application is to provide a robot and a method for the robot to identify falls, which can realize automatic detection and identification of falls by the robot, and can better monitor the safety of family members while taking into account its own service functions.

This application discloses a method for a robot to identify falls, which includes the following steps:

Robot A moves position, detects the first target part of the foot or sole, and records the position and attitude of the detected first target part;

B. Control the robot to move the position and/or change the posture according to the position and attitude of the first target part to perform detection of other target parts, and record the detected positions of other target parts;

C determines whether the current object is in a fallen state based on the detected position of the target part.

In a preferred example, the positions of the detected target parts are respectively the positions of their centers of gravity;

The step C further includes the following steps:

Determine whether the line connecting the center of gravity positions of all the target parts tends to the same straight line or the same plane and the angle between the straight line or plane and the ground is less than the threshold angle. If so, it is determined that the current object is in a fallen state.

In a preferred example, step B further includes the following steps:

Determine whether the current object is suspected of falling based on the position and posture of the first target part, and estimate the positions of other target parts of the suspected falling object;

The robot is controlled to move the position and/or change the posture according to the estimated position to perform detection of other target parts, and record the detected positions of other target parts.

In a preferred example, step B further includes the following steps:

When the other target part is detected, continuously collect multiple frames of data of the other target part and record the position of the other target part in each frame;

The step C further includes the following steps:

It is determined whether the current object is in a fallen state according to the detected changes in the positions of the other target parts. If the positions of the other target parts in each frame change within a preset range, it is determined that the current object is in a fallen state.

In a preferred example, step B further includes the following steps:

According to the position and attitude of the first target part, the robot is controlled to perform N-circle position movements around the outline of the object associated with the first target part, and correspondingly performs N times of detection and recording of the target part each time The location of the detected target part;

The step C further includes the following steps:

If the positions of the target parts detected N times are all the same or change within a preset range, the current object is determined to be in a fallen state.

In a preferred example, when detecting the first target part of the foot sole or shoe sole, the following steps are further included:

If the sole of the foot or sole is detected, the binocular vision unit of the robot is used to collect the depth information of the sole of the foot or sole, and the size of the sole of the foot or sole is determined based on the collected depth information;

Determine whether the size of the sole of the foot or sole meets the preset condition. If so, determine the sole of the foot or sole as the first target part and record the position of the first target part. Otherwise, continue to move the position and execute the sole of the foot or sole. Detection of the first target part.

In a preferred example, the other target parts include the face and/or legs.

In a preferred example, step B further includes the step of: recording the detected postures of other target parts when detecting other target parts;

The step C also includes the following step: judging whether the current object is in a fallen state according to the detected position and posture of the target part.

In a preferred example, the robot uses a target recognition model to detect the target part;

The target recognition model is trained according to the following steps:

The robot is used to collect a set of sample images. The sample images include facial images, foot images and leg images associated with each foot image. The facial images include images with different shooting distances and different facial angles and postures, so The above-mentioned foot images include images of soles and shoe soles with different bottom orientations, and each image is marked with a category label and posture label;

Use the sample image set to train a deep neural network to obtain the target recognition model.

In a preferred example, the robot is a robot that moves close to the ground.

In a preferred example, the robot is a sweeping robot.

This application also discloses a robot including:

The target part detection module is used to detect the first target part of the sole or shoe sole when the robot moves, and record the detected position and posture of the first target part, and record the position and posture of the first target part according to the position and position of the first target part. Attitude controls the robot to move its position and/or change its attitude to perform detection of other target parts, and record the detected positions of other target parts;

A fall determination module is used to determine whether the current object is in a fallen state according to the detected position of the target part.

The fall determination module is also used to determine whether the lines connecting the center of gravity positions of all the target parts tend to be in the same straight line or the same plane and the angle between the straight line or the plane and the ground is less than a threshold angle. If so, it is determined that the current object is in a fallen state. .

In a preferred example, the target part detection module is also used to determine whether the current object is suspected of falling based on the position and posture of the first target part, and to estimate the positions of other target parts of the suspected falling object, and based on the estimated The position control robot moves the position and/or changes the attitude to perform detection of other target parts, and records the detected positions of other target parts.

In a preferred example, the target part detection module is also configured to continuously collect multiple frames of data of the other target parts and record the position of the other target parts in each frame when the other target parts are detected; and,

The fall determination module is also used to determine whether the current object is in a fallen state based on the detected change in the position of the target part. If the positions of other target parts in each frame change within a preset range, it is determined that the current object is in a fall state. state.

In a preferred example, the target part detection module is also used to control the robot to surround the outline of an object associated with the first target part according to the position and attitude of the first target part. Carry out N circle position movements, and perform corresponding detection of the target part N times and record the position of the target part detected each time; and,

The fall determination module is also used to determine that the current object is in a fallen state if the positions of the target parts detected N times are all the same or change within a preset range.

In a preferred example, the robot further includes a binocular vision unit for collecting depth information of the soles of the feet or shoe soles;

The target part detection module is also configured to determine the size of the sole or sole based on the depth information of the sole or sole collected by the binocular vision unit if the sole or sole is detected, and determine the size of the sole or sole. Whether the size of the shoe sole meets the preset conditions, if so, the sole or sole is determined to be the first target part and the position of the first target part is recorded; otherwise, the detection of the first target part is continued.

In a preferred example, the other target parts include the face and/or legs.

In a preferred example, the target part detection module is also used to record the detected postures of other target parts when performing detection of other target parts;

The fall determination module is also configured to determine whether the current object is in a fallen state based on the detected position and posture of the target part.

In a preferred example, the system further includes an image acquisition unit, and the target part detection module further includes a target recognition model;

The image acquisition unit is used to collect a set of sample images. The sample images include facial images, foot images and leg images associated with each foot image. The facial images include images from different shooting distances and different facial angles and postures. Image, the foot image includes sole and sole images of different bottom orientation postures, each image is marked with a category label and posture label, and the target recognition module is obtained by training a deep neural network using the sample image set;

The image acquisition unit is also used to collect images when the robot moves its position, and the collected images are The image is input into the target recognition model to detect the target part.

Compared with the existing technology, the embodiments of the present application include at least the following advantages and beneficial effects: Aiming at the unique low viewing angle of robots that move close to the ground (such as sweeping robots), the present invention proposes low-view image processing and recognition dedicated to robots. This method realizes the automatic detection and recognition of falls by the robot, and can better monitor the safety of family members (especially the elderly and children) while taking into account cleaning. Use the associated combined data of feet and legs and face data as sample data, and use categories and postures as labels to train the model. The trained model can directly identify the category and posture of the target part, and thus can effectively determine the identity of family members. Fall situations. The fall situation is determined based on the detected position, position change, or combination of position and posture of the target parts of the face, legs, and feet, with high accuracy.

The description of this application records a large number of technical features, which are distributed in various technical solutions. If we want to list all possible combinations of technical features (ie, technical solutions) of this application, the description will be too lengthy. In order to avoid this problem, the technical features disclosed in the above-mentioned summary of the present invention, the technical features disclosed in the various embodiments and examples below, and the technical features disclosed in the drawings can be freely combined with each other to form Various new technical solutions (these technical solutions are deemed to have been recorded in this specification), unless this combination of technical features is technically unfeasible. For example, feature A+B+C is disclosed in one example, and feature A+B+D+E is disclosed in another example. However, features C and D are equivalent technical means that play the same role. Technically, as long as you choose It can be used as soon as it is used, and it is impossible to use it at the same time. Feature E can technically be combined with feature C. Then, the solution A+B+C+D should not be regarded as documented because it is technically unfeasible, while A+B+ C+E's plan should be deemed to have been documented.

Description of the drawings

Figure 1 is a schematic flowchart of a method for identifying falls by a robot according to the first embodiment of the present application.

Figure 2 is a schematic flowchart of a method for a robot to identify falls according to an embodiment of the present application.

Figure 3 is a schematic flowchart of a method for a robot to identify falls according to another embodiment of the present application.

Figure 4 is a schematic structural diagram of a robot according to the second embodiment of the present application.

Detailed ways

In the following description, many technical details are provided to enable readers to better understand this application. However, those of ordinary skill in the art can understand that the technical solution claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The first embodiment of the present application relates to a method for a robot to identify falls. The process is shown in Figure 1. The method includes the following steps:

Step 101: The robot moves its position, detects the first target part of the sole or sole, and records the position and posture of the detected first target part;

Step 102: Control the robot to move the position and/or change the posture according to the position and attitude of the first target part to perform detection of other target parts, and record the detected positions of other target parts;

Step 103: Determine whether the current object is in a fallen state according to the detected position of the target part.

The specific description is as follows:

In step 101, the robot performs position movement, performs detection of the first target part of the sole or shoe sole, and records the position and posture of the detected first target part. The robot can be a robot that moves close to the ground, with its camera positioned close to the ground. In one embodiment, the robot may be a sweeping robot usually used for indoor cleaning, and its height is usually less than 20 cm.

Optionally, when performing the detection of the first target part of the sole or shoe sole in step 101, the following steps are also included:

If a foot sole or shoe sole is detected, the robot's binocular vision unit is used to collect depth information of the foot sole or shoe sole, and the size of the foot sole or shoe sole is determined based on the collected depth information;

Determine whether the size of the sole of the foot or sole meets the preset conditions. If so, determine the sole of the foot or sole as the first target part and record the position of the first target part. Otherwise, continue to move the position and execute the first goal of the sole of the foot or sole. Detection of parts.

In step 102, the robot is controlled to move the position and/or change the posture according to the position and attitude of the first target part to perform detection of other target parts, and record the detected positions of other target parts.

Wherein, the other target parts in step 102 can be any part on the human body, and the other target parts include, for example, the face and/or the legs.

Optionally, this step 102 may further include the following sub-steps 102a and 102b:

Step 102a: Determine that the current object is suspected of falling based on the position and posture of the first target part, and estimate the positions of other target parts of the suspected fallen object;

Step 102b: Control the robot to move the position and/or change the posture according to the estimated position to perform detection of other target parts, and record the detected positions of other target parts.

Optionally, the robot can detect the position of the target part in an implementation manner. For example, but not limited to, the binocular vision unit can use a disparity map of the target part to obtain the position information of the target part.

In step 103, it is determined whether the current object is in a fallen state according to the detected position of the target part.

Step 103 can be implemented in various specific ways. In one embodiment, the detected positions of the target parts are respectively the positions of their centers of gravity; in this optional embodiment, step 103 may further include the following steps: determining whether the lines connecting the positions of the centers of gravity of all the target parts tend to be same straight line or same A plane and the angle between the straight line or plane and the ground is less than the threshold angle. If so, the current object is determined to be in a fallen state.

In another embodiment, the following steps may be included after step 102: when the other target part is detected, continuously collecting multiple frames of data of the other target part and recording the position of the other target part in each frame; and, step 103 It can be further implemented as: judging whether the current object is in a fallen state based on the detected changes in the positions of the other target parts. For example, but not limited to, if it is detected that the positions of the other target parts change within a preset range in each frame, then it is judged that the current object is in a fallen state. Falling state.

In yet another embodiment, step 102 may be further implemented as: controlling the robot to perform N circle position movements around the outline of the object associated with the first target part according to the position and attitude of the first target part, and correspondingly executing N Detect the target part several times and record the position of the target part detected each time; and step 103 may further include the following steps: If the positions of the target part detected N times are all the same or change within a preset range, then Determine the current object to be in a fallen state.

In yet another embodiment, step 102 may also include the step of: recording the detected postures of other target parts when detecting other target parts; and step 103 may be further implemented as: according to the detected target part The position and posture of the object are used to determine whether the current object is in a fallen state.

Optionally, the robot uses a target recognition model to detect the target part.

Optionally, the target recognition model is trained in the following manner: using the robot to collect a sample image set, the sample image includes a face image, a foot image and a leg image associated with each foot image, and the face image contains different Images taken at different distances and different facial angles and postures. The foot images include images of soles and shoe soles with different bottom orientations. Each image is marked with a category label. This sample image collection is used to train a deep neural network to obtain the target recognition model. Optionally, each image can also be identified with a pose label, that is, both a category label and a pose label are identified.

Furthermore, considering that the viewing angle of sweeping robots is generally low, basically about 6-7cm or lower from the ground, at such a low viewing angle, when the distance is far, the human body seen is not clear enough and cannot be accurately seen. Recognition; when the distance is close, the human body cannot be completely within the viewing angle. Conventional models that use the human face as a detection or the entire human body as a training model cannot be applied to sweeping robots. Optionally, the combination of foot and leg sample data is used as the main sample data for training, and the facial data is used as the auxiliary sample data to train the model.

Moreover, when using the sweeping robot to collect the sample image set, for the collection of facial image data in the sample data, consider that the human face seen from the sweeping robot's perspective is generally very close to the camera, has a low viewing angle, is tilted, or has only partial ears. Or half-face outline features; to address this feature or limitation, image collection can be performed, for example, at long distances, ultra-close ranges, lying-down angles, or face-down angles. And, for example, after collecting the photos, perform image processing on them: cut out the pictures to obtain facial features; enlarge the facial features; and perform image amplification on the enlarged facial features. For the collection of foot and leg image data in the sample data, the foot features seen from the perspective of the sweeping robot need to be divided into two types: with shoes and without shoes. Both types of data require regular angle, abnormal angle, and bottom-up collection and data enhancement for model training. During the recognition process, the depth information obtained by the binoculars will be combined for further judgment. Usually the characteristic width of the feet is within a certain range (31cm). If the width information of the object obtained by the binoculars exceeds this value, the object is considered not to be a "foot". department". The leg features seen from the perspective of the sweeping robot are similar to those of objects such as furniture legs when the distance is far away. When the leg features are close to the sweeping robot, they are close to the features of objects such as bases. Therefore, for the characteristics of the legs, it is necessary to synchronously determine whether there are associated foot characteristics after detection. Usually the foot characteristics are associated with the leg characteristics. Once marked as leg characteristics, the location of the leg will be Record and save.

Optionally, the data can be processed before training the above model. Specifically, the main control chips used by robots generally only support low computing power. Some main control chips do not even have an NPU and can only use the CPU to load and calculate some models. Therefore, when training the model, we must take into account the two dimensions of the model's computing power consumption and accuracy. Model training first requires preprocessing of the collected data, which is called data cleaning, that is, various checks and reviews of the data are performed to correct missing values and normalize/standardize the values to make them comparable. After obtaining the processed data, perform data labeling, such as labeling The set object categories to be recognized. For example, the data set can be divided into 3 parts. The larger data subset is used as the training set, accounting for 95% of the original data. The second part is usually a smaller subset used for model verification. The three parts are usually smaller subsets used for model testing.

Alternatively, the target recognition model can be trained and recognized as a separate self-learning model, and the detection and recognition of other obstacles can be treated as another model without self-learning capabilities. Because the robot's viewing angle is low, obstacles or objects that usually affect the robot's work are generally relatively small in size. However, a robot that can recognize falls needs to recognize some human body features. Therefore, when selecting a model, two different factors need to be taken into consideration. Scenes. Therefore, the present invention adopts the method of superimposing two models. Based on two superimposed model systems, there will also be corresponding processes for subsequent processing of recognized objects. The overall flow chart is as follows: First, the robot will perform target detection of indoor objects, and will call the obstacle detection model when detecting regular obstacles. , after the robot detects the object trained in the model, the robot will record and mark the information of this type of obstacle, and then update the information to the robot's map. The robot will make corresponding actions based on the characteristics of the obstacle. In the present invention, the obstacle is basically bypassed. If no obstacles of the category defined in the model are identified, the robot continues cleaning and performs target detection. For fall detection, the robot will use the face detection model for identification. The condition for falling is that the features of "face", "legs" and "feet" are detected at the same time, and the centers of gravity of these three areas are based on the same horizontal line. After the robot detects the fall signal, it will trigger the alarm module on the robot and send out an alarm signal.

In the present invention, after recognizing the fall status of a family member, for example, an alarm signal or alarm information can be sent to the mobile phones of other family members, such as but not limited to sending alarm information in the form of mobile phone text messages.

As shown in Figures 2 and 3, Figure 2 shows a flow chart of a method for a robot to recognize a fall according to one embodiment of the present application, and Figure 3 shows a flow chart of a method for a robot to recognize a fall according to another embodiment of the present application. The details listed in these two embodiments are mainly for ease of understanding and are not intended to limit the protection of this application. Scope limitations.

The second embodiment of the present application relates to a robot, the structure of which is shown in Figure 4. The robot includes a target part detection module and a fall judgment module.

The target part detection module is used to detect the first target part of the sole or shoe sole when the robot moves position, and record the position and attitude of the detected first target part, and the position and attitude according to the first target part. Control the robot to move the position and/or change the posture to perform detection of other target parts, and record the detected positions of other target parts; the fall determination module is used to determine whether the current object has fallen based on the detected position of the target part. state.

Optionally, the target part detection module is also used to determine whether the current object is suspected of falling based on the position and attitude of the first target part, estimate the positions of other target parts of the suspected fallen object, and control the movement of the robot according to the estimated positions. position and/or change posture to perform detection of other target parts, and record the detected positions of other target parts.

Optionally, the robot further includes a binocular vision unit, and the robot obtains the position information of the target part by using, for example, but not limited to, the disparity map of the target part using the binocular vision unit.

Optionally, the binocular vision unit is also used to collect the depth information of the sole of the foot or the sole of the shoe; the target part detection module is also used to: if the sole of the foot or the sole of the shoe is detected, based on the depth information of the sole of the foot or the sole of the shoe collected by the binocular vision unit The depth information determines the size of the sole of the foot or sole, and determines whether the size of the sole of the foot or sole meets the preset conditions. If so, determines the sole of the foot or sole as the first target part and records the position of the first target part. Otherwise, continue execution. Detection of the first target site.

The other target parts in this application can be any part on the human body, and the other target parts include, for example, the face and/or the legs.

Optionally, the system further includes an image acquisition unit, and the target part detection module further includes a target recognition model. On the one hand, the image acquisition unit is used to collect a sample image set. The sample image includes a facial image, a foot image, and a leg image associated with each foot image. The facial image includes images of different shooting distances and different facial perspective postures. , the foot image contains different bottom orientations Images of soles and soles of postures. Each image is marked with a category label and posture label. The target recognition module uses the sample image collection to train a deep neural network. On the other hand, the image acquisition unit is also used to collect images when the robot moves. Image, input the collected image into the target recognition model to detect the target part.

In one embodiment, the detected positions of the target parts are respectively the positions of their centers of gravity; the fall determination module is also used to determine whether the lines connecting the positions of the centers of gravity of all the target parts tend to be the same straight line or the same plane and the straight line or The angle between the plane and the ground is less than the threshold angle. If so, the current object is determined to be in a fallen state.

In another embodiment, the target part detection module is also configured to continuously collect multiple frames of data of the other target part and record the position of the other target part in each frame when the other target part is detected; and the fall determination module It is also used to determine whether the current object is in a fallen state based on the detected change in the position of the target part. If the positions of other target parts in each frame change within a preset range, it is determined that the current object is in a fallen state.

In yet another embodiment, the target part detection module is also used to control the robot to perform N-circle position movements around the outline of the object associated with the first target part according to the position and attitude of the first target part, and perform corresponding execution Detect the target part N times and record the position of the target part detected each time; and, the fall judgment module is also used to if the positions of the target part detected N times are all the same or change within a preset range, then Determine the current object to be in a fallen state.

In yet another embodiment, the target part detection module is also used to record the detected postures of other target parts when performing the detection of other target parts; and the fall determination module is also used to record the detected postures of the target part according to the detected target part. The position and posture of the object are used to determine whether the current object is in a fallen state.

The first embodiment is a method implementation corresponding to this embodiment. The technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment.

It should be noted that the robot of the present invention can include all functional modules of existing robots. and units, such as but not limited to, may include battery units, power management and generator drives, motor units, sensor matrix units, storage units, navigation and positioning units, mobile modem units, wifi units, control units, etc.

It should be noted that those skilled in the art should understand that the implementation functions of each module shown in the embodiment of the robot can be understood with reference to the relevant description of the method for identifying a fall by the robot. The functions of each module shown in the above embodiments of the robot can be implemented by programs (executable instructions) running on the processor, or by specific logic circuits. If the above-mentioned robot in the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or those that contribute to the existing technology. The computer software products are stored in a storage medium and include a number of instructions to A computer device (which may be a personal computer, a server, a network device, etc.) is caused to execute all or part of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), magnetic disk or optical disk and other media that can store program code. As such, embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, embodiments of the present application also provide a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method implementations of the present application are implemented. Computer-readable storage media includes permanent and non-transitory, removable and non-removable media and may be implemented by any method or technology to store information. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cassettes, disk storage or other magnetic storage devices or any other non-transmission medium that can be used for storage that can be accessed by a computing device Ask for information. As defined in this article, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

In addition, embodiments of the present application also provide a robot, which includes a memory for storing computer-executable instructions, and a processor; the processor is used to implement the above methods when executing the computer-executable instructions in the memory. steps in the way. Among them, the processor can be a central processing unit (Central Processing Unit, referred to as "CPU"), a graphics processor (Graphic Processing Unit, referred to as "GPU"), a digital signal processor (Digital Signal Processor, referred to as "DSP"), Microcontroller Unit (MCU), Neural Network Processor (NPU), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array, (referred to as "FPGA") or other programmable logic devices, etc. The aforementioned memory can be read-only memory (read-only memory, referred to as "ROM"), random access memory (random access memory, referred to as "RAM"), flash memory (Flash), hard disk or solid state drive, etc. The steps of the method disclosed in each embodiment of the present invention can be directly implemented by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Furthermore, the terms "comprises,""comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a" does not exclude the presence of additional identical elements in a process, method, article, or device that includes the stated element. In the application documents of this patent, if it is mentioned that an act is performed based on a certain element, it means that the act is performed based on at least that element, which includes two situations: performing the act based on that element only, and performing the act based on both that element and Other elements perform this behavior. Multiple, many times, various Expressions include 2, 2 times, 2 kinds, and 2 or more, 2 or more times, and 2 or more kinds.

All documents mentioned in this application are considered to be included in the disclosure of this application in their entirety so as to serve as a basis for modifications when necessary. In addition, it should be understood that the above descriptions are only preferred embodiments of this specification and are not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the protection scope of one or more embodiments of this specification.

Claims

A method for a robot to identify falls, which is characterized by including the following steps:

Robot A moves position, detects the first target part of the foot or sole, and records the position and attitude of the detected first target part;

B. Control the robot to move the position and/or change the posture according to the position and attitude of the first target part to perform detection of other target parts, and record the detected positions of other target parts;

C determines whether the current object is in a fallen state based on the detected position of the target part.
The method according to claim 1, characterized in that the positions of the detected target parts are respectively the positions of their centers of gravity;

The step C further includes the following steps:

Determine whether the line connecting the center of gravity positions of all the target parts tends to the same straight line or the same plane and the angle between the straight line or plane and the ground is less than the threshold angle. If so, it is determined that the current object is in a fallen state.
The method of claim 1, wherein step B further includes the following steps:

Determine whether the current object is suspected of falling based on the position and posture of the first target part, and estimate the positions of other target parts of the suspected falling object;

The robot is controlled to move the position and/or change the posture according to the estimated position to perform detection of other target parts, and record the detected positions of other target parts.
The method of claim 3, wherein step B further includes the following steps:

When the other target part is detected, continuously collect multiple frames of data of the other target part and record the position of the other target part in each frame;

The step C further includes the following steps:

It is determined whether the current object is in a fallen state according to the detected changes in the positions of the other target parts. If the positions of the other target parts in each frame change within a preset range, it is determined that the current object is in a fallen state.
The method of claim 1, wherein step B further includes the following steps:

According to the position and attitude of the first target part, the robot is controlled to perform N-circle position movements around the outline of the object associated with the first target part, and correspondingly performs N times of detection and recording of the target part each time The location of the detected target part;

The step C further includes the following steps:

If the positions of the target parts detected N times are all the same or change within a preset range, the current object is determined to be in a fallen state.
The method according to claim 1, characterized in that when performing the detection of the first target part of the foot sole or shoe sole, it further includes the following steps:

If the sole of the foot or sole is detected, the binocular vision unit of the robot is used to collect the depth information of the sole of the foot or sole, and the size of the sole of the foot or sole is determined based on the collected depth information;

Determine whether the size of the sole of the foot or sole meets the preset condition. If so, determine the sole of the foot or sole as the first target part and record the position of the first target part. Otherwise, continue to move the position and execute the sole of the foot or sole. Detection of the first target part.
The method of claim 1, wherein the other target parts include the face and/or the legs.
The method of claim 1, wherein step B further includes the step of: recording the detected postures of other target parts when detecting other target parts;

The step C also includes the following steps: according to the detected position and posture of the target part Determine whether the current object is in a fallen state.
The method according to any one of claims 1 to 8, characterized in that the robot uses a target recognition model to detect the target part;

The target recognition model is trained according to the following steps:

The robot is used to collect a set of sample images. The sample images include facial images, foot images and leg images associated with each foot image. The facial images include images with different shooting distances and different facial angles and postures, so The above-mentioned foot images include images of soles and shoe soles with different bottom orientations, and each image is marked with a category label and posture label;

Use the sample image set to train a deep neural network to obtain the target recognition model.
The method according to any one of claims 1 to 8, characterized in that the robot is a robot that moves close to the ground.
The method according to any one of claims 10, wherein the robot is a sweeping robot.
A robot is characterized by including:

Memory for storing computer-executable instructions; and,

A processor, coupled to the memory, for implementing the steps of the method of any one of claims 1 to 11 when executing the computer-executable instructions.
A computer-readable storage medium, characterized in that computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, the computer-executable instructions implement any one of claims 1 to 11. steps in the method described above.