WO2022000757A1

WO2022000757A1 - Ar-based robot internet of things interaction method and apparatus, and medium

Info

Publication number: WO2022000757A1
Application number: PCT/CN2020/112502
Authority: WO
Inventors: 王龙龙; 高明; 金长新
Original assignee: 济南浪潮高新科技投资发展有限公司
Priority date: 2020-06-29
Filing date: 2020-08-31
Publication date: 2022-01-06
Also published as: CN111784797A

Abstract

Disclosed are an AR-based robot Internet of Things (IOT) interaction method and apparatus, and a medium. The method comprises the step of obtaining a navigation path. Because the obtained navigation path is drawn by a terminal device according to a scene map and the scene map is set according to a scene picture acquired by a camera in the terminal device and data of an IMU sensor in the terminal device, a robot can complete an IOT task according to a pre-planned navigation path after obtaining the IOT task, apparatuses such as a laser radar or a camera do not need to be configured at a robot end, repeated use of hardware apparatuses can be achieved by using the built-in camera and IMU sensor of the terminal device, the hardware composition of the robot is simplified, and hardware costs are reduced. In addition, an ROS operation system does not need to be configured at the robot end, path planning is transferred to the terminal device to be completed, operation is more skilled, convenient and rapid because a user uses the terminal device in daily life, and the use experience of the user is improved.

Description

An AR-based robot Internet of Things interaction method, device and medium

technical field

The present application relates to the field of information technology, and in particular, to an AR-based robot Internet of Things interaction method, device, and medium.

Background technique

Augmented Reality (AR), also known as augmented reality, AR technology is a technology that calculates the position and angle of the camera in real time and superimposes text, images, videos, and 3D models on the real environment. The surrounding environment and predicting the direction of light, the virtual objects are accurately "placed" in the real environment, the virtual objects are integrated with the real environment with the help of display devices, and a new environment with real sensory effects is presented to the user.

The Internet of Things (IOT) technology refers to the connection of any object with the network through information sensing equipment and according to the agreed protocol, and the object exchanges and communicates with the information transmission medium to realize intelligent identification, positioning, tracking, monitoring and other functions. IOT equipment, that is, Internet of Everything network equipment. IOT devices have the characteristics of low power consumption, wide coverage, multiple connections, and low cost, and are widely used in intelligent transportation, smart home, public safety and other fields.

A robot is an intelligent machine that can work semi-autonomously or fully autonomously, with basic characteristics such as perception, decision-making, and execution.

With the development of Internet of Things technology and robotics, the concept of Robot Internet of Things has been proposed. Through the connection of robots and IOT devices, the intelligence of robots can be expanded, and the purpose of intelligent decision-making and manipulation of actual objects can be achieved. However, at this stage, the robot and the IOT device operate independently, and there is a lack of interaction between the two. During the interaction process, the ROS operating system and devices such as lidar or camera for data collection need to be configured on the robot side to create a scene map that can be used for navigation. The rendering of the robot makes the hardware composition of the robot very complex and the hardware cost is very high.

In view of the above-mentioned prior art, it is an urgent problem for those skilled in the art to seek a method that can simplify the hardware composition of the robot and simultaneously realize barrier-free human-machine interaction.

SUMMARY OF THE INVENTION

The purpose of this application is to provide an AR-based robot IoT interaction method, device and medium.

In order to solve the above technical problems, the present application provides an AR-based robot Internet of Things interaction method, including:

Obtain a navigation path and an IOT task, wherein the navigation path is drawn by the terminal device according to a scene map, and the scene map is set according to the scene picture collected by the camera in the terminal device and the data of the IMU sensor in the terminal device;

receiving the current location determined by the camera;

From the current location, travel according to the navigation path to complete the IOT task.

Preferably, the device information of the IOT device is stored in the form of a two-dimensional code, so that the terminal device scans the two-dimensional code and registers the IOT device in the scene map drawn by the terminal device.

Preferably, the device information specifically includes function information and location information of the IOT device.

Preferably, it also includes:

Get the charging path;

Check the current power;

Determine whether the current power is less than the set value, and if so, interrupt the IOT task and proceed to the charging location for charging according to the charging path;

After the charging is completed, the IOT task continues to be executed.

Preferably, it also includes:

Write the process of executing the IOT task into the task record.

Preferably, it also includes:

The execution progress of the IOT task and the IOT device to be accessed are displayed through the terminal device.

Preferably, it also includes:

When a task repetition signal is received, the IOT task is stored.

In order to solve the above technical problems, the present application also provides an AR-based robot Internet of Things interactive device, including:

The acquisition module is used to acquire the navigation path and the IOT task, wherein the navigation path is drawn by the terminal device according to the scene map, and the scene map is based on the scene picture collected by the camera in the terminal device and the IMU sensor in the terminal device. data settings;

a receiving module for receiving the current location determined by the camera;

A travel module, configured to travel according to the navigation path from the current location to complete the IOT task.

In order to solve the above technical problems, the present application also provides an AR-based robot Internet of Things interaction device, including a memory for storing computer programs;

The processor is configured to implement the steps of the AR-based robot Internet of Things interaction method when executing the computer program.

In order to solve the above technical problems, the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the AR-based robot as described above is realized. The steps of the networking interaction method.

In the AR-based robot Internet of Things interaction method provided by this application, the acquired navigation path is drawn by the terminal device according to the scene map, and the scene map is based on the scene picture collected by the camera in the terminal device and the inertial measurement in the terminal device. The data setting of the Inertial measurement unit (IMU) sensor, so the robot can complete the IOT task according to the pre-planned navigation path after acquiring the IOT task, without the need to configure the lidar or camera on the robot side, and use the built-in device in the terminal device. The camera and IMU sensor can realize the repeated use of the hardware device, which simplifies the hardware structure of the robot and reduces the hardware cost. In addition, there is no need to configure the ROS operating system on the robot side, and the path planning is transferred to the terminal device to complete. Because the user uses the terminal device on a daily basis, the operation is more proficient, convenient and quick, which improves the user experience.

Description of drawings

In order to describe the embodiments of the present application more clearly, the following will briefly introduce the drawings that are used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, which are not relevant to ordinary skills in the art. As far as personnel are concerned, other drawings can also be obtained from these drawings on the premise of no creative work.

1 is a flowchart of an AR-based robot Internet of Things interaction method provided by an embodiment of the present application;

FIG. 2 is a flowchart of another AR-based robot Internet of Things interaction method provided by an embodiment of the present application;

3 is a schematic diagram of a robot task line provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an AR-based robot Internet of Things interaction device provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of another AR-based robot Internet of Things interaction device according to an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative work fall within the protection scope of the present application.

The core of this application is to provide an AR-based robot IoT interaction method, device and medium, wherein the AR-based robot IoT interaction method utilizes the built-in camera and IMU sensor in the terminal device to realize the repeated use of hardware devices, simplifying the The hardware composition of the robot is reduced, and the hardware cost is reduced.

In order to make those skilled in the art better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

It should be noted that the terminal device mentioned in the present invention is a mobile or fixed networked computing device, including products such as smart phones and tablet computers, and the corresponding products above include cameras and IMU sensors. In addition, an AR application is installed in the terminal device. The AR application can be developed based on Google's mobile ARCore SDK, or based on software development kits such as Apple's ARKit SDK or Unity's ARFundation SDK, which does not affect this technology. implementation of the plan. The terminal device can use the ordinary Android system or the iOS system, the robot uses the programmable control robot, and the IOT devices in the surrounding environment include 3D printers, humidity sensors, etc. Terminal devices, robots and IOT devices communicate and connect in the same local area network. It is understandable that the AR-based robot IoT interaction method mentioned in this application can be implemented by a micro control unit (MCU) or other types of control devices in the robot, which does not affect the implementation of the technical solution.

FIG. 1 is a flowchart of an AR-based robot IoT interaction method provided by an embodiment of the present application. As shown in Figure 1, the method includes:

S10: Get the navigation path and the IOT task.

The navigation path is drawn by the terminal device according to the scene map, and the scene map is set according to the scene picture collected by the camera in the terminal device and the data of the IMU sensor in the terminal device.

In a specific implementation, during the movement of the terminal device, its built-in camera captures the surrounding scene images, and the AR application marks the feature points in the scene images, and tracks the movement process of these points over time. Combining the movement of these points with data from the IMU sensor in the end device estimates the camera's position and screen orientation as the end device moves, and detects the floor plane so that virtual content can be rendered from the correct perspective and overlaid onto the camera On the acquired scene picture, the coordinate system in the real environment is restored.

After drawing the scene map according to the surrounding environment, the user can plan the navigation path in the path planning interface of the AR application. It should be noted that the path can be planned by hand-drawing and hand-held movement. The hand-drawn way is to add the robot's navigation path by hand-drawn curves on the screen. The internal implementation takes the terminal device plane as the starting point. According to the points drawn by the user's finger during the drawing process, the ray is calculated from the terminal device plane, projected on the target plane, and the path drawing is completed by connecting the points into a line. In the hand-held movement mode, the user holds the terminal device and walks along the expected navigation path, and the terminal device records the entire path. Both methods have their own advantages. The hand-drawn method is suitable for small areas, which is convenient and quick; the hand-held movement method is suitable for larger work areas and is more direct.

In a specific implementation, after the setting of the navigation path is completed, the user places the terminal device at a designated position on the robot, and the position may be a groove or a water platform, which is not limited in this application. The robot communicates with the terminal device through the local area network, and obtains the IOT task from the terminal device.

S11: Receive the current location determined by the camera.

S12: Start from the current location and travel according to the navigation path to complete the IOT task.

In the specific implementation, the robot obtains information such as the current location and the current posture through the built-in camera of the terminal device, and then travels according to the set navigation path to sequentially access the IOT device and complete the corresponding IOT task.

In the AR-based robot IoT interaction method provided by the embodiments of the present application, the acquired navigation path is drawn by the terminal device according to the scene map, and the scene map is based on the scene picture collected by the camera in the terminal device and the scene map in the terminal device. The data setting of the IMU sensor, so the robot can complete the IOT task according to the pre-planned navigation path after obtaining the IOT task, without the need to configure the lidar or camera on the robot side, and use the built-in camera and IMU sensor in the terminal device. The repeated use of the hardware device is realized, the hardware structure of the robot is simplified, and the hardware cost is reduced. In addition, there is no need to configure the ROS operating system on the robot side, and the path planning is transferred to the terminal device to complete. Because the user uses the terminal device on a daily basis, the operation is more proficient, convenient and quick, which improves the user experience.

Based on the above embodiment, the device information of the IOT device is stored in the form of a two-dimensional code, so that the terminal device scans the two-dimensional code and registers the IOT device in the scene map drawn by the terminal device.

The device information specifically includes function information and location information of the IOT device.

In the specific implementation, the device information of the IOT device specifically includes its IP address, interaction protocol and location information, etc. These information can be stored in the form of a QR code or barcode, and made into a sticker and attached to the IOT device. After scanning by the terminal device You can obtain the corresponding information, register the IOT device on the established scene map, and generate a virtual icon at the location of the device. The user clicks the virtual icon to pop up the function menu, and at the same time, a pointer to the device will be generated in front of the device. Virtual guide route to form visual guide.

The AR-based robot Internet of Things interaction method provided by the embodiments of the present application stores the device information of the IOT device in the form of a two-dimensional code, so that the user can directly obtain the relevant information by scanning, and the operation is proficient, convenient and fast, and the user's experience is improved. Use experience.

FIG. 2 is a flowchart of another AR-based robot IoT interaction method provided by an embodiment of the present application. As shown in Figure 2, on the basis of the above embodiment, the method further includes:

S13: Obtain the charging path.

In the specific implementation, the planning of the charging path is consistent with the planning method of the navigation path in the above-mentioned embodiment. Therefore, please refer to the description of the embodiment in the navigation path section, which will not be repeated here.

S14: Detect the current power.

S15: Determine whether the current power is less than the set value, if yes, go to S16, if not, go back to S14.

S16: Interrupt the IOT task and perform charging according to the charging path to the charging location.

S17: Continue to execute the IOT task after the charging is completed.

In the specific implementation, when completing the task, the robot will detect its own state at any time. When the power is less than the set value, the robot will request to temporarily interrupt the task, go to a nearby charging point for charging, and continue to execute the unfinished task after completing the charging. Task.

It should be noted that the purpose of the set value is to remind the robot to interrupt the IOT task when the power of the robot drops to this value, so the set value can be set to a fixed value.

In the AR-based robot IoT interaction method provided by the embodiments of the present application, since the robot will suspend the task and go to the charging location according to the charging path when the current power is less than the set value, the normal interaction between the robot and the IoT device is ensured, and no Losing unsaved IOT task progress improves user experience.

On the basis of the above-mentioned embodiment, the method also includes:

Write the process of executing the IOT task into the task record.

In the specific implementation, the data of the robot during the execution of the IOT task will be automatically uploaded to the terminal device, and the full name of the task execution process will be recorded, which is convenient for users to analyze and optimize the task later.

On the basis of the above-mentioned embodiment, the method also includes:

Display the progress of the IOT task execution and the IOT devices to be accessed through the terminal device.

FIG. 3 is a schematic diagram of a task line of a robot according to an embodiment of the present application. As shown in FIG. 3 , in order to visualize the entire workflow, the present application also provides a task management method. All tasks are connected in series by a task line. The task line represents a task flow and displays the current robot task progress and added tasks. The task line can display the current task progress and the IOT device to be accessed in real time. Users can edit the path, add or delete IOT tasks, and set the task completion time and repetition times. After the task editing is completed, the user can simulate the task process in the AR application of the terminal device before executing the task, and the whole process is presented on the terminal device interface in an augmented reality manner.

The AR-based robot IoT interaction method provided by the embodiments of the present application can display the IOT task execution progress through the terminal device, which facilitates the user to understand the task progress and edit the task in time, and improves the user experience.

On the basis of the above-mentioned embodiment, the method also includes:

When a task repetition signal is received, the IOT task is stored.

In the specific implementation, when encountering repetitive tasks, such as moving the printed model of the 3D printer on the assembly line to the sorting point, the user can set a task repetition signal for the task. When the robot receives the signal, The IOT task will be stored and executed for the specified number of runs or repeated loops according to the prompt of the signal.

The AR-based robot IoT interaction method provided by the embodiments of the present application enhances the interaction between the robot and the surrounding IOT devices because the robot can repeatedly perform or perform a certain task according to the user's needs, thereby expanding the human and the surrounding environment. interactive capabilities.

In the above embodiments, the AR-based robot IoT interaction method is described in detail, and the present application also provides embodiments corresponding to the AR-based robot IoT interaction device. It should be noted that this application describes the embodiments of the device part from two perspectives, one is based on the perspective of functional modules, and the other is based on the perspective of hardware.

FIG. 4 is a schematic structural diagram of an AR-based robot Internet of Things interaction device according to an embodiment of the present application. As shown in Figure 4, based on the perspective of functional modules, the device includes:

The first acquisition module 10 is used to acquire the navigation path and the IOT task, wherein the navigation path is drawn by the terminal device according to the scene map, and the scene map is set according to the scene picture collected by the camera in the terminal device and the data of the IMU sensor in the terminal device.

The receiving module 11 is used for receiving the current location determined by the camera.

The traveling module 12 is used for traveling according to the navigation path from the current location to complete the IOT task.

As a preferred embodiment, it also includes:

The second acquisition module is used to acquire the charging path.

The detection module is used to detect the current power.

The judgment module is used to judge whether the current power is less than the set value.

The charging module is used to interrupt the IOT task and perform charging according to the charging path to the charging location.

The execution module is used to continue to execute the IOT task after charging is completed.

Since the embodiment of the apparatus part corresponds to the embodiment of the method part, for the embodiment of the apparatus part, please refer to the description of the embodiment of the method part, which will not be repeated here.

In the AR-based robot IoT interaction device provided by this application, the acquired navigation path is drawn by the terminal device according to the scene map, and the scene map is based on the scene picture collected by the camera in the terminal device and the IMU sensor in the terminal device. Therefore, the robot can complete the IOT task according to the pre-planned navigation path after obtaining the IOT task. It is not necessary to configure the lidar or camera and other devices on the robot side. The built-in camera and IMU sensor in the terminal device can be used to realize the hardware. The repeated use of the device simplifies the hardware structure of the robot and reduces the hardware cost. In addition, there is no need to configure the ROS operating system on the robot side, and the path planning is transferred to the terminal device to complete. Because the user uses the terminal device on a daily basis, the operation is more proficient, convenient and quick, which improves the user experience.

FIG. 5 is a structural diagram of an AR-based robot Internet of Things interaction device provided by another embodiment of the present application. As shown in FIG. 5 , based on the hardware structure, the device includes: a memory 20 for storing computer programs;

The processor 21 is configured to implement the steps of the AR-based robot Internet of Things interaction method in the foregoing embodiment when executing the computer program.

The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 21 can use at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish. The processor 21 may also include a main processor and a co-processor. The main processor is a processor used to process data in the wake-up state, also called CPU (Central Processing Unit, central processing unit); the co-processor is A low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 21 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

Memory 20 may include one or more computer-readable storage media, which may be non-transitory. Memory 20 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, where, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the AR-based robot Internet of Things interaction method disclosed in any of the foregoing embodiments . In addition, the resources stored in the memory 20 may also include an operating system 202, data 203, etc., and the storage mode may be short-term storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, and the like. The data 203 may include, but is not limited to, location information of the IOT device, and the like.

In some embodiments, the bus 22 may also be a peripheral component interconnect standard (peripheral component interconnect, referred to as PCI) bus or an extended industry standard architecture (extended industry standard architecture, referred to as EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 5, but it does not mean that there is only one bus or one type of bus.

Those skilled in the art can understand that the structure shown in FIG. 5 does not constitute a limitation on the AR-based robot Internet of Things interactive device, and may include more or less components than those shown.

The AR-based robot Internet of Things interaction device provided by the embodiment of the present application includes a memory and a processor. When the processor executes a program stored in the memory, the processor can implement the following method: obtain a navigation path, and the navigation path is drawn by a terminal device according to a scene map obtained, and the scene map is set according to the scene picture collected by the camera in the terminal device and the data of the IMU sensor in the terminal device, so the robot can complete the IOT task according to the pre-planned navigation path after acquiring the IOT task, without needing to perform the IOT task on the robot side. By configuring devices such as lidar or camera, and using the built-in camera and IMU sensor in the terminal device, the hardware device can be reused, which simplifies the hardware structure of the robot and reduces the hardware cost. In addition, there is no need to configure the ROS operating system on the robot side, and the path planning is transferred to the terminal device to complete. Because the user uses the terminal device on a daily basis, the operation is more proficient, convenient and quick, which improves the user experience.

Finally, the present application also provides an embodiment corresponding to a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps described in the foregoing method embodiments are implemented.

It can be understood that, if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The AR-based robot Internet of Things interaction method, device and medium provided in this application have been described in detail above. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method. It should be pointed out that for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can also be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or operations. There is no such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Claims

An AR-based robot Internet of Things interaction method, characterized in that it includes:

Obtain a navigation path and an IOT task, wherein the navigation path is drawn by the terminal device according to a scene map, and the scene map is set according to the scene picture collected by the camera in the terminal device and the data of the IMU sensor in the terminal device;

receiving the current location determined by the camera;

From the current location, travel according to the navigation path to complete the IOT task.
The AR-based robot Internet of Things interaction method according to claim 1, wherein the device information of the IOT device is stored in the form of a two-dimensional code, so that the terminal device scans the two-dimensional code and the IOT device Registered in the scene map drawn by the terminal device.
The AR-based robot IoT interaction method according to claim 2, wherein the device information specifically includes function information and location information of the IoT device.
The AR-based robot Internet of Things interaction method according to claim 1, further comprising:

Get the charging path;

Check the current power;

Determine whether the current power is less than the set value, and if so, interrupt the IOT task and proceed to the charging location for charging according to the charging path;

After the charging is completed, the IOT task continues to be executed.
The AR-based robot Internet of Things interaction method according to claim 1, further comprising:

Write the process of executing the IOT task into the task record.
The AR-based robot Internet of Things interaction method according to claim 1, further comprising:

The execution progress of the IOT task and the IOT device to be accessed are displayed through the terminal device.
The AR-based robot Internet of Things interaction method according to any one of claims 1 to 6, further comprising:

When a task repetition signal is received, the IOT task is stored.
An AR-based robot Internet of Things interactive device, characterized in that it includes:

The acquisition module is used to acquire the navigation path and the IOT task, wherein the navigation path is drawn by the terminal device according to the scene map, and the scene map is based on the scene picture collected by the camera in the terminal device and the IMU sensor in the terminal device. data settings;

a receiving module for receiving the current location determined by the camera;

A travel module, configured to travel according to the navigation path from the current location to complete the IOT task.
An AR-based robot Internet of Things interaction device, characterized in that it includes a memory for storing computer programs;

The processor is configured to implement the steps of the AR-based robot Internet of Things interaction method according to any one of claims 1 to 7 when executing the computer program.
A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the AR-based storage medium according to any one of claims 1 to 7 is implemented. Steps of the Robot IoT Interaction Method.