CN114246526B

CN114246526B - Sweeping robot control method, electronic equipment and storage medium

Info

Publication number: CN114246526B
Application number: CN202111671104.1A
Authority: CN
Inventors: 唐世长
Original assignee: Sipic Technology Co Ltd
Current assignee: Sipic Technology Co Ltd
Priority date: 2021-12-31
Filing date: 2021-12-31
Publication date: 2023-08-01
Anticipated expiration: 2041-12-31
Also published as: CN114246526A

Abstract

The invention discloses a control method of a sweeping robot, electronic equipment and a storage medium. In the method, environmental audio of the environment where the sweeping robot is located is collected; identifying whether abnormal environmental sounds exist in the environmental audio; when the abnormal environmental sound exists, triggering the sweeping robot to carry out cruising operation, and executing image acquisition operation when the sweeping robot cruises. Therefore, the sound around the sweeping robot can be continuously monitored, the sweeping robot is actively triggered to carry out cruising and photographing operations when abnormal sounds exist in the home environment, and sudden events in the home environment can be automatically and timely found.

Description

Sweeping robot control method, electronic equipment and storage medium

Technical Field

The invention belongs to the technical field of Internet, and particularly relates to a control method of a sweeping robot, electronic equipment and a storage medium.

Background

With the continuous development of intelligent household products, the sweeping robot has become a good helper for cleaning household environments in many families.

The robot of sweeping floor of linkage security protection supports remote control patrol on the basis of the original clean function of robot of sweeping floor, has realized the wireless interactive function of robot of sweeping floor, has received more favor of user.

However, these robots need to rely on human commands and then the robots simply execute the corresponding commands, relatively passive. Especially, aiming at the security part, some emergencies are usually detected, and the emergencies or problems in the household environment can not be found in time at all by means of artificial remote control patrol or timing patrol.

In view of the above problems, currently, there is no preferred solution provided in the industry.

Disclosure of Invention

The embodiment of the invention provides a control method of a sweeping robot, electronic equipment and a storage medium, which are used for at least solving one of the technical problems.

In a first aspect, an embodiment of the present invention provides a method for controlling a sweeping robot, including: collecting environmental audio of the environment where the sweeping robot is located; identifying whether abnormal environmental sounds exist in the environmental audio; when the abnormal environmental sound exists, triggering the sweeping robot to carry out cruising operation, and executing image acquisition operation when the sweeping robot cruises.

In a second aspect, an embodiment of the present invention provides an electronic device, including: the system comprises at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method described above.

In a third aspect, embodiments of the present invention provide a storage medium having stored therein one or more programs including execution instructions capable of being read and executed by an electronic device (including, but not limited to, a computer, a server, or a network device, etc.) for performing the steps of the above-described method of the present invention.

In a fourth aspect, embodiments of the present invention also provide a computer program product comprising a computer program stored on a storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the steps of the above-described method.

The embodiment of the invention has the beneficial effects that:

the method comprises the steps of collecting environmental audio of an environment where the sweeping robot is located, identifying whether abnormal environmental sound exists in the environmental audio, triggering the sweeping robot to conduct cruising operation when the abnormal environmental sound exists, and executing image collecting operation when the sweeping robot cruises. Therefore, the sound around the sweeping robot can be continuously monitored, the sweeping robot is actively triggered to carry out cruising and photographing operations when abnormal sounds exist in the home environment, and sudden events in the home environment can be automatically and timely found.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 shows a flowchart of an example of a control method of a sweeping robot according to an embodiment of the present invention;

fig. 2 is a flowchart showing an example of a control method of the sweeping robot in a normal service mode according to an embodiment of the present invention;

FIG. 3 is a flowchart showing an example of a control method of a sweeping robot in a security mode according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of an embodiment of an electronic device of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other.

The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In the present invention, "module," "system," and the like refer to a related entity, either hardware, a combination of hardware and software, or software in execution, as applied to a computer. In particular, for example, an element may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. Also, the application or script running on the server, the server may be an element. One or more elements may be in processes and/or threads of execution, and elements may be localized on one computer and/or distributed between two or more computers, and may be run by various computer readable media. The elements may also communicate by way of local and/or remote processes in accordance with a signal having one or more data packets, e.g., a signal from one data packet interacting with another element in a local system, distributed system, and/or across a network of the internet with other systems by way of the signal.

Finally, it is also noted that, in this document, the terms "comprises," comprising, "and" includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

It should be noted that, in the related art at present, there are some robots for sweeping floor that have a photographing function and a cruising function. However, the camera of the sweeping robot can only monitor the condition in the sight range, and the home environment often comprises a plurality of rooms, the view is blocked by obstacles, and if the robot is fixed in one place for a long time, the condition behind other rooms or obstacles cannot be found in time. In addition, simply relying on manual remote control patrol or timing patrol often results in an emergency that cannot be found in time.

Fig. 1 shows a flowchart of an example of a control method of a sweeping robot according to an embodiment of the present invention. It should be noted that, the execution body of the method embodiment of the present invention may be a controller module integrated in the sweeping robot, or a processor unit disposed outside the sweeping robot and communicating with the sweeping robot, or the like.

As shown in fig. 1, in step 110, environmental audio of an environment in which the robot is located is collected.

In some examples of the embodiment of the invention, a camera and a microphone array can be added on the basis of the hardware of the original sweeping robot, so that the sweeping robot has a photographing function and an audio acquisition function. Here, each microphone (three or more) in the microphone array may be disposed above the sweeping robot, and may have a ring-shaped or polygonal layout, so as to preferably pick up sound source signals of various directions.

In step 120, it is identified whether an abnormal ambient sound is present in the ambient audio.

Here, the abnormal environmental sound may be identified in various manners, for example, the environmental audio may be transmitted to a server (a robot cleaner server or a voice platform server) to identify whether the abnormal environmental sound exists in the environmental audio by the server, for example, a machine learning model with training convergence may be used to identify the abnormal sound in the audio. In addition, a corresponding abnormal voice recognition module can be set locally on the sweeping robot so as to perform corresponding recognition operation locally, or the two implementations are fused, and the method and the device belong to the implementation range of the embodiment of the invention.

In step 130, when there is an abnormal environmental sound, the sweeping robot is triggered to perform a cruising operation, and an image capturing operation is performed while the sweeping robot cruises.

On one hand, the sweeping robot can adopt a whole house for cruising, and a camera is called to perform image acquisition operation in the cruising process. On the other hand, the sweeping robot can also directionally cruise aiming at the abnormal sound source, and the sweeping robot belongs to the implementation range of the embodiment of the invention.

Through the embodiment of the application, the abnormal sound event in the home environment is rapidly determined by utilizing the voice collection and abnormal sound identification of the microphone, and then the sweeping robot is invoked to carry out cruising operation and image collection operation, so that a command is not required to be manually sent, the user can independently cruise when the abnormal event exists in the home environment, and image data in the home environment is collected.

In this application embodiment, the camera on sweeping robot plays control and keeps away the barrier function, and microphone array not only can receive speaker instruction and fall the function of making an uproar to the pronunciation, can also monitor the environmental sound, uses degree of depth neural network to judge the category of environmental sound (such as the water pipe leaks, rains, the air-out, the sound of opening the window, broken sound of glass etc.) with the camera mutually support, all-round guarantee home security.

In some cases, the above-described security functions of the sweeping robot are expected to be closable, for example when an adult user is present at home. Specifically, the sweeping robot is configured to have a variety of alternative or switchable modes of operation, such as a security mode and a normal service mode.

Regarding the implementation details of the step 110, the working mode of the sweeping robot may be detected, and when the working mode of the sweeping robot is the security mode, the environmental audio of the environment where the sweeping robot is located is collected.

Fig. 2 is a flowchart illustrating an example of a control method of the sweeping robot in a normal service mode according to an embodiment of the present invention.

As shown in fig. 2, in step 210, environmental audio of an environment in which the robot is located is collected.

In step 220, it is identified whether the environmental audio meets a preset voice wake-up condition.

In step 230, when the voice wake condition is met, a user manipulation instruction is acquired, and an operation is performed based on the user manipulation instruction.

When the sweeping robot is in the common service mode, the sweeping robot does not need to perform an autonomous security function and can execute a voice interaction function with a user. By way of example, the user can wake up the sweeping robot through a specific wake-up keyword, such as a small cloth, and give a voice control instruction when the sweeping robot is in a voice wake-up state, so that voice control of the sweeping robot is realized, and user experience of the sweeping robot can be improved to a greater extent.

Fig. 3 is a flowchart showing an example of a control method of the sweeping robot in a security mode according to an embodiment of the present invention.

As shown in fig. 3, in step 310, environmental audio of the environment in which the robot is located is collected.

In step 320, a neural network is identified based on the environmental sounds, and an environmental sound category corresponding to the environmental sounds is determined. Specifically, the environmental sound recognition neural network may adopt a deep neural network, and the training sample set at least comprises various abnormal sound samples, such as water leakage, rain, wind, window opening and door opening sound or glass breaking sound of a tap water pipe, and the like. In this way, training the converged ambient sound recognition neural network can effectively recognize various sound types in the ambient sound.

In step 330, based on the ambient sound category, it is identified whether an abnormal ambient sound exists in the ambient audio.

On the one hand, the training sample set of the environment sound recognition neural network only comprises abnormal sound samples, and the environment sound recognition neural network can only be used for recognizing the type of the abnormal environment sound at the moment, and the existence of the abnormal environment sound can be determined as long as the type is successfully recognized. On the other hand, the training sample set of the environmental sound recognition neural network has an abnormal sound sample and a normal sound sample at the same time, at this time, the output result of the environmental sound recognition neural network may be a normal sound type or an abnormal sound type, and the abnormal sound type may be compared with each preset abnormal type to determine whether the corresponding recognition result is an abnormal sound type.

In step 340, when there is an abnormal environmental sound, abnormal sound azimuth information corresponding to the abnormal environmental sound is determined. Here, the direction or position of the abnormal ambient sound may be determined using various sound source localization methods, for example, azimuth calculation may be performed using MUSIC (Multiple Signal Classification multi-signal classification) method, or spatial filtering may be performed based on beam forming method, etc.

In step 350, the sweeping robot is triggered to perform cruising operation according to the abnormal sound azimuth information. Therefore, the directional cruising device can reach the position of the abnormal sound source earlier and reach the destination of the emergency rapidly compared with the whole house cruising.

In step 360, the environmental image acquired when the sweeping robot cruises is uploaded to the server, so that the server performs an early warning operation based on the environmental image. The server may send the environment image to the client of the binding user, so that the user can quickly learn about the emergency in the home environment, and further degradation of the emergency is effectively avoided.

The embodiment of the invention solves the problem that the security robot cannot find the security problem in real time, effectively improves the instantaneity of processing the security problem, and can participate in home care, such as selecting window opening ventilation or timely closing window to prevent house water inflow according to weather conditions, and timely finding whether tap water leaks, whether a kitchen and the like have fire, and the like; if the baby is in the family, the baby cradle can also be monitored and nursed, for example, the baby crys or the baby wakes up or the baby turns over and climbs to cause the cradle to shake, and the like.

In some specific business scenarios, the general service mode is mainly used for receiving a user command and performing voice interaction with a user when the host is at home in order to understand the intention of the user. The security mode is used for monitoring household conditions in real time under the condition that a user is not at home, and timely finding and processing emergency conditions, and is mainly used for carrying out cooperative interaction by the environment sound category classification module, the image/video acquisition module and the communication module.

In the embodiment of the invention, the environment sound monitoring function is added on the basis of the original robot, the robot can be in a standby state at ordinary times, once the environment sound abnormality such as glass breaking sound is detected, the robot can distinguish the sound direction and actively cruises, take a photograph and report information and the like, so that timeliness of finding an emergency event (lower delay) can be reduced while less active cruises can be effectively achieved.

In some examples of embodiments of the present application, after step 320, it may also be identified whether a preset allied ambient sound category exists in the ambient sound category. When the joint control environment sound category exists, a joint control instruction is sent to the household equipment corresponding to the joint control environment sound category, so that the household equipment executes corresponding operation according to the joint control instruction. Here, the joint control environmental sound type may be a part of abnormal environmental sound types, for example, under some specific environmental sound types, other intelligent devices need to be controlled in a joint way to perform operation so as to respond to an emergency in time, and thus home care is achieved. By means of environment sound recognition, the floor sweeping robot can timely inform a user of treatment when detecting water leakage/fire and the like of tap water; in addition, when the conditions of wind blowing, rain blowing and the like are found, the intelligent window/curtain can be controlled to operate, and the house is prevented from being filled with water.

In this application embodiment, for the robot that sweeps floor newly-increased environmental sound class detection module, just like having installed a pair of ear for the robot that sweeps floor, can discover the condition outside the camera sight scope in real time, combine the image signal that the camera gathered to carry out the overall analysis to reach best decision-making, make the robot that sweeps floor more intelligent, the range of application is wider.

It should be noted that, for simplicity of description, the foregoing method embodiments are all illustrated as a series of acts combined, but it should be understood and appreciated by those skilled in the art that the present invention is not limited by the order of acts, as some steps may be performed in other orders or concurrently in accordance with the present invention. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present invention. In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

In some embodiments, embodiments of the present invention provide a non-transitory computer readable storage medium having stored therein one or more programs including execution instructions that are readable and executable by an electronic device (including, but not limited to, a computer, a server, or a network device, etc.) for performing any of the above-described methods of the present invention.

In some embodiments, embodiments of the present invention also provide a computer program product comprising a computer program stored on a non-volatile computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform any one of the above-described methods of controlling a floor sweeping robot.

In some embodiments, the present invention further provides an electronic device, including: the system comprises at least one processor and a memory communicatively connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a sweeping robot control method.

Fig. 4 is a schematic hardware structure of an electronic device for executing a control method of a sweeping robot according to another embodiment of the present application, as shown in fig. 4, the device includes:

one or more processors 410, and a memory 420, one processor 410 being illustrated in fig. 4.

The apparatus for performing the robot controlling method may further include: an input device 430 and an output device 440.

The processor 410, memory 420, input device 430, and output device 440 may be connected by a bus or other means, for example in fig. 4.

The memory 420 is used as a non-volatile computer readable storage medium, and may be used to store a non-volatile software program, a non-volatile computer executable program, and modules, such as program instructions/modules corresponding to the sweeping robot control method in the embodiment of the present application. The processor 410 executes various functional applications of the server and data processing, namely, implements the above-described method embodiment of the robot cleaner control method by running the nonvolatile software programs, instructions, and modules stored in the memory 420.

Memory 420 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the robot cleaner control device, and the like. In addition, memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 420 optionally includes memory remotely located with respect to processor 410, which may be connected to the sweeping robot control device through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 430 may receive input numerical or character information and generate signals related to user settings and function control of the floor sweeping robot control apparatus. The output 440 may include a display device such as a display screen.

The one or more modules are stored in the memory 420, which when executed by the one or more processors 410, perform the method of controlling the floor sweeping robot in any of the method embodiments described above.

The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment may be found in the methods provided in the embodiments of the present application.

The electronic device of the embodiments of the present application exist in a variety of forms including, but not limited to:

(1) Mobile communication devices, which are characterized by mobile communication functionality and are aimed at providing voice, data communication. Such terminals include smart phones, multimedia phones, functional phones, low-end phones, and the like.

(2) Ultra mobile personal computer equipment, which belongs to the category of personal computers, has the functions of calculation and processing and generally has the characteristic of mobile internet surfing. Such terminals include PDA, MID, and UMPC devices, etc.

(3) Portable entertainment devices such devices can display and play multimedia content. The device comprises an audio player, a video player, a palm game machine, an electronic book, an intelligent toy and a portable vehicle navigation device.

(4) Other on-board electronic devices with data interaction functions, such as on-board devices mounted on vehicles.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

From the above description of embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus a general purpose hardware platform, or may be implemented by hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the related art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A method of controlling a sweeping robot, comprising:

detecting the working mode of the sweeping robot;

when the working mode of the sweeping robot is a security mode, acquiring environmental audio of the environment where the sweeping robot is located based on a microphone array; each microphone in the microphone array is arranged above the sweeping robot and is in an annular or polygonal layout for picking up sound source signals in different directions;

identifying whether abnormal environmental sound exists in the environmental audio, more specifically, identifying a neural network based on the environmental sound, determining an environmental sound category corresponding to the environmental audio, and identifying whether the abnormal environmental sound exists in the environmental audio based on the environmental sound category;

identifying whether the environment sound category exists a preset joint control environment sound category or not; the joint ambient sound category includes at least one selected from the group consisting of: baby crying, wind and rain sounds;

when the joint control environment sound category exists, sending a joint control instruction to household equipment corresponding to the joint control environment sound category, so that the household equipment executes corresponding operation according to the joint control instruction; the home device includes at least one selected from the group consisting of: smart windows, smart curtains and infant cradles;

when the abnormal environmental sound exists, azimuth calculation is carried out according to a multi-signal classification mode aiming at the picked sound source signals in different directions, or spatial filtering is carried out based on a beam forming mode, so that abnormal sound azimuth information corresponding to the abnormal environmental sound is determined;

triggering and controlling the sweeping robot to perform cruising operation according to the abnormal sound azimuth information, and executing image acquisition operation when the sweeping robot cruises;

when the working mode of the sweeping robot is a common service mode, collecting environmental audio of the environment where the sweeping robot is located;

identifying whether the environmental audio accords with a preset voice wake-up condition;

and when the voice wake-up condition is met, acquiring a user control instruction, and executing operation based on the user control instruction.

2. The method of claim 1, further comprising:

and uploading the environment image acquired when the sweeping robot cruises to a server, so that the server executes early warning operation based on the environment image.

3. The method of claim 2, wherein the server is configured to send the environmental image to a client of the binding user to perform the early warning operation.

4. An electronic device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1-3.

5. A storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the method of any of claims 1-3.