KR102650464B1 - Method for updating map based on image deep learning using autonomous mobile robot and system for monitoring driving using the method - Google Patents

Abstract

According to an embodiment of the present invention, a method of controlling a self-driving robot through image learning-based map updating performed by a server includes: a) periodically autonomously driving in a specific space and selecting the space from a mapping robot that senses the space; Collecting sensing data including captured image data in real time; b) comparing the sensing data with sensing data collected in a previous cycle to recognize environmental changes occurring in the space and determine the location of the environmental changes; c) Among the image data, the frame in which the location of the environmental change is captured is input into an environmental change classifier built by a preset image learning model to determine the type of environmental change, and the judgment result is stored on a map of the space previously stored. updating the map based on the reflection; and d) controlling autonomous driving of a service robot to perform a preset service via the location of the environmental change according to the determination result.

Description

Image learning-based map update method using autonomous driving robot and driving monitoring system using the same {METHOD FOR UPDATING MAP BASED ON IMAGE DEEP LEARNING USING AUTONOMOUS MOBILE ROBOT AND SYSTEM FOR MONITORING DRIVING USING THE METHOD}

The present invention relates to an image learning-based map update method using an autonomous driving robot and a driving monitoring system using the same.

Recently, the use of self-driving robots is increasing in various industries and service sites. However, due to the variety of unexpected situations and failure conditions that can occur due to autonomous driving, the places and services where autonomous driving is actually implemented are very limited.

In this regard, self-driving robots equipped with sensors or cameras to recognize surrounding obstacles and to avoid them are being developed. However, these robots require expensive equipment, resulting in a significant economic burden, and there is a risk of defects due to their complex design.

Additionally, technology is being designed to create or update maps by reflecting images of objects acquired while the robot is driving. However, despite the wide variety of factors that affect driving, it is still limited to the abstract concept of identifying the existence or type of an object and reflecting it on the map, which limits its effectiveness.

In order to improve this situation, there is a demand for the introduction of robot-infrastructure linkage technology that can be used in actual fields rather than a simple concept by comprehensively reflecting various unexpected situations that may occur during driving on the map.

The present invention is intended to solve the problems of the prior art described above, and updates the map to reflect various types of environmental changes through machine learning on images collected through the driving of a mapping robot, and updates a number of maps according to the type of environmental change. The purpose is to provide technology to control and manage the driving of service robots.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

According to an embodiment of the present invention, a method of controlling a self-driving robot through image learning-based map updating performed by a server includes: a) periodically autonomously driving in a specific space and selecting the space from a mapping robot that senses the space; Collecting sensing data including captured image data in real time; b) comparing the sensing data with sensing data collected in a previous cycle to recognize environmental changes occurring in the space and determine the location of the environmental changes; c) Among the image data, the frame in which the location of the environmental change is captured is input into an environmental change classifier built by a preset image learning model to determine the type of environmental change, and the judgment result is stored on a map of the space previously stored. updating the map based on the reflection; and d) controlling autonomous driving of a service robot to perform a preset service via the location of the environmental change according to the determination result.

According to one embodiment of the present invention, the mapping robot includes a position sensor that detects location data of the mapping robot in real time, a spatial recognition sensor that acquires three-dimensional spatial data about the surrounding terrain of the mapping robot, and the mapping robot. It includes a 360-degree camera sensor that acquires image data about the surroundings, and the 3D spatial data and image data are matched and stored for each location data.

According to one embodiment of the present invention, the service robot is designed to be inexpensive and does not include the spatial recognition sensor and the 360-degree camera sensor.

According to one embodiment of the present invention, before step a), the server generates and stores a map of the space based on the location data, the 3D space data, and the image data.

According to one embodiment of the present invention, step b) includes comparing the image data with the image data collected in the previous cycle for the same location data, and identifying location data in which different frames are found as a result of the comparison; determining the location of the environmental change based on identified location data and three-dimensional spatial data matched to the identified location data; and extracting coordinate data of the map corresponding to the location of the environmental change.

According to one embodiment of the present invention, before step a), the server collects learning images containing objects representing preset types of environmental changes, and when one learning image is input, one of the types of environmental changes is selected. The environmental change classifier is constructed by repeating machine learning through the image learning model so that at least one is output.

According to one embodiment of the present invention, in step c), the type is preset as a standard by which the data input to the environmental change classifier is classified, such as obstacles, weather, texture of the road surface, density of people, and theft. It includes a type of conflict, and when the frame is input to the environment change classifier, it is classified into at least one type.

According to an embodiment of the present invention, in step c), when the frame is input to the environment change classifier and classified into an obstacle type, obstacles that can be handled and obstacles that cannot be handled are based on feature values of the frame representing the obstacle. It is further classified into one subtype.

According to one embodiment of the present invention, in step d), if the frame is classified into a subtype of an obstacle that can be handled, the service robot is set so that the service robot does not pass the location of the environmental change for a predetermined period of time. Controlling autonomous driving, and if the frame is classified as a subtype of an obstacle that cannot be handled, generating an avoidance route for the location of the environmental change and transmitting it to the service robot.

According to one embodiment of the present invention, in step c), when the frame is input to the environmental change classifier and classified into a weather type, the weather situation in the space is identified based on the feature value of the frame indicating the weather. And depending on the weather conditions, it is further classified into one subtype of drivable weather and non-drivable weather.

According to one embodiment of the present invention, in step d), if the frame is classified into the subtype of weather where driving is possible, a signal of low speed driving is transmitted to the service robot and the frame is classified into the subtype of weather where driving is not possible. When classified, it includes transmitting a stop signal to the service robot.

According to one embodiment of the present invention, in step c), when the frame is input to the environmental change classifier and classified into a road surface texture type, the road surface condition is determined based on the feature value of the frame representing the road surface texture. It is identified and further classified into one subtype of drivable road surface and non-drivable road surface according to the road surface condition.

According to an embodiment of the present invention, in step d), if the frame is classified into a subtype of a road surface that can be driven, a signal of low speed driving is transmitted to the service robot, and the frame is classified into a subtype of a road surface that is not drivable. If classified, it includes generating an avoidance route for the location of the environmental change and transmitting it to the service robot.

According to one embodiment of the present invention, in step c), when the frame is input to the environmental change classifier and classified into a human density type, the drivable area and driving range are based on the characteristic values of the frame representing the person. It is further classified into one subtype of the impossible area.

According to an embodiment of the present invention, in step d), if the frame is classified into a subtype of an area where driving is possible, a signal of low speed driving is transmitted to the service robot, and the frame is classified into a subtype of an area where driving is not possible. If classified as, creating an avoidance route for the location of the environmental change and transmitting it to the service robot, or controlling autonomous driving of the service robot so that the service robot does not pass the location of the environmental change for a predetermined period of time. Includes.

According to one embodiment of the present invention, in step c), when the frame is input to the environmental change classifier and feature values with reduced resolution are extracted, the frame is classified into theft and collision types.

According to one embodiment of the present invention, in step d), if the frame is classified as a theft or collision type, the location of the environmental change and a danger signal are provided to the administrator terminal, and the service robot is operated for a predetermined period of time. and controlling autonomous driving of the service robot to avoid the location of the environmental change.

According to an embodiment of the present invention, a self-driving robot control server includes a memory storing a program that performs a self-driving robot control method through image learning-based map update; and a processor executing the program, wherein the method includes: a) periodically autonomously driving in a specific space and collecting sensing data including image data captured in the space from a mapping robot that senses the space in real time; steps; b) comparing the sensing data with sensing data collected in a previous cycle to recognize environmental changes occurring in the space and determine the location of the environmental changes; c) Among the image data, the frame in which the location of the environmental change is captured is input into an environmental change classifier built by a preset image learning model to determine the type of environmental change, and the judgment result is stored on a map of the space previously stored. updating the map based on the reflection; and d) controlling autonomous driving of a service robot to perform a preset service via the location of the environmental change according to the determination result.

According to one embodiment of the present invention, it is possible to create and update a map for a specific place using an autonomous robot without having to visit or perform the task in person.

According to an embodiment of the present invention, the types of environmental changes or unexpected situations that can occur during autonomous driving are specifically specified, and learning data corresponding to them is presented.

According to one embodiment of the present invention, objects that interfere with driving are identified in real time in images transmitted by a robot through a classifier built through machine learning, and their types are determined.

According to one embodiment of the present invention, not only tangible objects are identified, but also intangible environmental changes such as weather, texture, density, and external conditions are identified, and such various changes are reflected in the map.

According to one embodiment of the present invention, the map updated based on image learning is shared with other autonomous robots that are driving.

According to an embodiment of the present invention, each of the running robots can be controlled with an operation appropriate for factors of environmental change.

According to one embodiment of the present invention, even if only a few expensive robots are used to generate or update maps, the driving of multiple robots performing services can be managed simultaneously. Therefore, safe driving is promoted and service robots can be designed at low cost, making it economical.

In this way, one embodiment of the present invention guarantees the processing speed and accuracy of the image learning classification model, and is therefore effective in applying to an actual driving monitoring system, making it easy to predict and respond to unexpected situations for driving robots. do.

1 is a structural diagram of a self-driving robot control system according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining the structure of a self-driving robot control server according to an embodiment of the present invention.
Figure 3 is an operation flowchart of a method for controlling a self-driving robot according to an embodiment of the present invention.
Figure 4 is a conceptual diagram for explaining the process of building an environmental change classifier according to an embodiment of the present invention.
Figure 5 is an example diagram of a map generated by driving a mapping robot according to an embodiment of the present invention.
Figure 6 is an example diagram of an updated map according to an embodiment of the present invention.
Figure 7 is an example of a frame classified into an obstacle type through an environmental change classifier according to an embodiment of the present invention.
Figure 8 is an example of a frame classified by weather type through an environmental change classifier according to an embodiment of the present invention.
Figure 9 is an example of a frame classified by road surface texture type through an environmental change classifier according to an embodiment of the present invention.
Figure 10 is an example of a frame classified by human density type through an environmental change classifier according to an embodiment of the present invention.
Figure 11 is an example of a frame classified into theft and collision types through an environmental change classifier according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Hereinafter, an embodiment of the present invention will be described in detail using the attached drawings.

1 is a structural diagram of a self-driving robot control system according to an embodiment of the present invention.

Referring to FIG. 1, the system includes a server 100, a mapping robot 200, and a service robot 300.

According to one embodiment, the mapping robot 200 is a device that periodically autonomously travels a specific space along a preset path and takes pictures of the surroundings or senses geographical features. The mapping robot 200 performs wireless communication with the server 100 and can transmit sensing data detected by various installed sensors to the server 100 in real time.

According to one embodiment, the mapping robot 200 is equipped with a location sensor such as GPS that detects location data of the mapping robot 200 in real time, and a lider that acquires three-dimensional spatial data about surrounding features. It may include a spatial recognition sensor and a 360-degree camera sensor that acquires image data about the surroundings. Sensing data detected by each sensor is transmitted to the server 100, and the server 100 can match and store 3D spatial data and image data for each location data. Meanwhile, the mapping robot 200 may include more diverse accessories such as an ultrasonic sensor and an inertial measurement unit (IMU), and is not limited to the configurations listed above.

According to one embodiment, the service robot 300 is a device designed to periodically perform preset services while traveling in part or all of the driving space of the mapping robot 200, where the services include delivery, delivery, transportation, It may vary, including guidance, process, and beautification, and is not particularly limited.

According to one embodiment, the service robot 300 is connected to the server 100 that monitors driving and service performance and performs wireless communication, and accordingly operates in real time by control commands transmitted from the server 100. It can be controlled.

According to one embodiment, a map of the driving space of the mapping robot 200 is stored in the service robot 300. When the map is updated by the server 100, the service robot 300 may remove the previously stored map from memory and receive and store the updated map from the server 100.

According to one embodiment, the service robot 300 may receive an updated map and simultaneously receive data about environmental changes reflected in the map from the server 100. Data on environmental changes may include, but are not limited to, command values for the type, location, and response actions of environmental changes.

According to one embodiment, a plurality of service robots 300 may be provided on the system, and each service robot 300 has unique identification data such as a robot ID or code set in advance and stored in the server 100. You can.

According to one embodiment, the service robot 300 may be equipped with a location sensor such as a GPS sensor. Through this, the server 100 can determine in real time where each service robot 300 is located and which service robot 300 starts driving or performs a designated service.

According to one embodiment, unlike the mapping robot 200, the service robot 300 may be designed not to include a spatial recognition sensor and a 360-degree camera sensor. In other words, the service robot 300 can be manufactured at a low cost to consist only of equipment essential for autonomous driving and service provision. Accordingly, the economic burden on service providers can be alleviated, and since it is relatively easy to provide a plurality of service robots 300, more improved services can be provided to the public.

The server 100 may refer to a server operated by a company that provides control and monitoring services for self-driving robots. Accordingly, the server 100 can be defined as an autonomous robot control server.

Figure 2 is a block diagram for explaining the structure of a self-driving robot control server according to an embodiment of the present invention.

According to one embodiment, the server 100 may include a communication module 110, memory 120, processor 130, and database 140.

The communication module 110 processes data communication with each mapping robot 200 and service robot 300.

The memory 120 stores a program (or application) for performing a self-driving robot control method, and the processor 130 executes the program stored in the memory 120 to provide various types of self-driving robot control methods. handle the process

One embodiment of the overall process processed by the processor 130 is as follows. The processor 130 generates and stores a map of the space in which the mapping robot 200 runs based on location data, 3D spatial data, and image data received in real time from the mapping robot 200. The processor 130 collects learning images containing objects representing preset types of environmental changes. The processor 130 builds an environmental change classifier by repeating machine learning through an image learning model so that when one learning image is input, at least one type is output. The processor 130 collects sensing data including image data from the mapping robot 200 in real time. The processor 130 compares the currently collected sensing data with the sensing data collected in the previous cycle to recognize environmental changes that have occurred in the space and determine their location. The processor 130 determines the type of environmental change by inputting a frame of image data in which the location of the environmental change is captured into an environmental change classifier. The processor 130 updates the map by reflecting the judgment result in a previously created map that is already stored. The processor 130 controls the autonomous driving of the service robot 300 that passes or is scheduled to pass through the location of the environmental change according to the judgment result.

In the database 140 according to one embodiment, various types of data generated as the self-driving robot control method is performed may be stored. Specifically, learning images collected by the processor 110 are stored. When learning images are learned by an image model and classified into a specific type of environmental change, each learning image is matched with the classified type and stored. The map generated by the processor 110 is stored. Sensing data received from the mapping robot 200 is stored in real time, and the sensing data is classified and stored according to the driving cycle of the mapping robot 200. When an environmental change is recognized by the processor 110, the corresponding frame, the type into which the frame is classified by the environmental change classifier, and the location of the environmental change are stored. Based on this, when the map is updated by the processor 110, the previously stored map is updated with the updated map.

According to one embodiment, the system may further include a manager terminal (not shown) of a manager that performs autonomous driving monitoring. The manager terminal may be connected to the server 100 to perform wireless communication, and a program that monitors the driving and service performance of the mapping robot 200 and the service robot 300 may be installed in the manager terminal. Additionally, the administrator terminal may receive a user interface for creating and changing a map from the server 100. The map may be automatically created or updated by a process of the server 100, but may also be manually manipulated by an administrator.

Meanwhile, the “administrator terminal” may be implemented as a computer or portable terminal that can connect to a server or other terminals through a network. Here, the computer is, for example, a laptop equipped with a web browser, a desktop, a laptop, a VR HMD (e.g., HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.), etc. may include. Here, VR HMD is for PC (e.g. HTC VIVE, Oculus Rift, FOVE, Deepon, etc.), mobile (e.g. GearVR, DayDream, Storm Magic, Google Cardboard, etc.), and console (PSVR). Includes independently implemented Stand Alone models (e.g. Deepon, PICO, etc.). Portable terminals are, for example, wireless communication devices that ensure portability and mobility, including smart phones, tablet PCs, and wearable devices, as well as Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, and ultrasonic devices. , may include various devices equipped with communication modules such as infrared, WiFi, and LiFi. In addition, “network” refers to a connection structure that allows information exchange between nodes such as terminals and servers, including a local area network (LAN), a wide area network (WAN), and the Internet. (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. This includes, but is not limited to, communication, Visible Light Communication (VLC), LiFi, etc.

Hereinafter, an embodiment of a method for controlling an autonomous robot will be described using FIGS. 3 to 11. Figure 3 is an operation flowchart of a method for controlling a self-driving robot according to an embodiment of the present invention.

In step S300, the server 100 may collect learning images containing objects representing preset types of environmental changes. For example, each learning image may be collected by an administrator terminal with the type of environmental change labeled, or may be collected by the server 100 itself through collection techniques such as Internet crawling and scraping, and the collection method may be consistent with the present invention. It is not intended to limit.

As an example of a specific learning image, referring to FIG. 4, the learning image is an image containing various types of obstacles, an image expressing the weather, an image expressing the texture of the road surface or floor, and a plurality of people or objects. It may include images of space and images of people or objects that are blurred and out of focus. Here, obstacle is a general term for facilities or terrain features that can affect autonomous driving.

The server 100 may perform image learning using training data. As shown in FIG. 4, a preset machine learning model based on deep learning may be applied to image learning. The machine learning model may be, for example, a deep learning model specialized in image processing of the CNN (Convolutional neural network) series, but is not limited thereto.

According to one embodiment, image learning may be repeatedly performed through a machine learning model so that when one learning image is input, at least one type of environmental change is output. Types here may include obstacles, weather, road texture, human density, and types of theft and collisions. For example, the server 100 may repeat image learning so that when a learning image depicting a rainy situation and including a bump is input to a machine learning model, the type of obstacle and weather is output.

According to one embodiment, the feature values of each learning image are extracted by a machine learning model, and the server 100 performs image learning to further classify one learning image into at least one subtype based on the feature values. It can get worse. That is, obstacles, weather, road surface texture, human density, theft, and collision are the main types, and each main type can be further classified into at least one or more subtypes. For example, when a specific learning image is input to a machine learning model and the type of “weather” is output, a result value representing at least one weather situation may be further output based on feature values representing the weather. Here, the weather situation is a preset one, and examples include, but are not limited to, snow, rain, yellow dust, fog, and darkness. Additionally, based on the output weather situation, one of the subtypes of drivable weather and non-drivable weather may be further output. In other words, when a learning image depicting rainy weather is input to the machine learning model, the server 100 can repeatedly perform image learning so that “weather”, “rain”, and “driving weather (or non-driving weather)” are output. there is.

The server 100 may build an environmental change classifier as a result of image learning. In other words, the environmental change classifier is designed to classify a specific random image into at least one type of environmental change when input. Additionally, according to the above-described embodiment, the environmental change classifier may be designed to further classify a random specific image into at least one subtype when input.

According to one embodiment, the environmental change classifier may be constructed in the form of a table where one axis is composed of environmental change types and the other axis is composed of subtypes. In other words, when a specific image is input into an environmental change classifier, at least one table value can be extracted. For example, when an image containing a “revolving door” is input, a table value of “obstacle-unprocessable obstacle” can be extracted. there is.

In step S301, the server 100 controls autonomous driving of the mapping robot 200 to a space where services are to be provided through autonomous driving in the future. The server 100 may create a driving route that passes through each area of the space as a whole and transmit it to the mapping robot 200. The mapping robot 200 drives along the driving path and transmits sensing data that senses the space in real time to the server 100, and the server 100 can generate a map of the space based on the received sensing data. there is. For example, the server 100 can identify spatial terrain, road surface conditions, and fixed features through image data captured by a 360 camera sensor. The server 100 can determine the shape, size, and location of each feature based on 3D spatial data whose depth is detected by a spatial recognition sensor and real-time location data of the robot according to a position sensor. The server 100 can create and store a map by mapping the identified data in a three-dimensional space that shares the coordinate system set in the location sensor. An example of the map 50 is shown in FIG. 5. The server 100 may share the generated map by transmitting it to the mapping robot 200 and each service robot 300.

After the map is created, in step S302, the mapping robot 200 periodically senses the space of the map while driving along the driving path. The server 100 collects sensing data including image data captured in the corresponding space from the mapping robot 200 in real time.

In step S303, the server 100 compares the sensing data with the sensing data collected in the previous cycle to recognize environmental changes that have occurred in the space and determine their location.

According to one embodiment, the server 100 may compare image data for the same location data with image data collected in the previous cycle, and determine the location data of the mapping robot 200 in which different frames are found as a result of the comparison. The server 100 may determine the location of the environmental change based on the identified location data and the three-dimensional spatial data matched thereto. The server 100 may extract map coordinate data corresponding to the determined location.

In step S304, the server 100 determines the type of environmental change by inputting a frame in which the location of the environmental change is captured among the image data into an environmental change classifier. In other words, frames in which locations recognized as environmental changes are captured are input to the environmental change classifier in time series, and each frame is classified into at least one type accordingly. According to one embodiment, each frame may be further classified into at least one subtype, and correspondingly, at least one “type-subtype” table value may be output.

In step S305, the server 100 updates the map by reflecting environmental changes on the map, and transmits the updated map to the mapping robot 200 and each service robot 300 to share them. For example, if a frame is classified as an obstacle type, the server 100 determines the shape and size of the obstacle based on the three-dimensional spatial data of the corresponding location data, and stores a predetermined object that matches the shape and size and is stored. is additionally displayed at the coordinates of the map corresponding to the location data. If the frame is classified into a weather type, the server 100 changes the background of the map or additionally displays an image object representing a predetermined weather situation at the coordinates where the mapping robot 200 is currently located. If the frame is classified by type of human density and further classified by subtype of non-drivable area, the server 100 additionally displays a preset non-divable image object at the corresponding coordinates in the map. If the frame is classified into the type of theft or collision, the server 100 additionally displays a preset warning image object at the corresponding coordinates in the map. If the frame is classified by the type of road surface texture, the server 100 determines the road surface condition and additionally displays a preset texture representing the road surface condition at the coordinates of the corresponding road surface in the map. When a frame is classified by the type of road surface texture, an example of an updated map 60 in which the corresponding part 61 of the existing map has been changed is shown in FIG. 6.

In step S306, the server 100 may inform the manager by providing the type and location of the environmental change to the manager terminal. Administrators can immediately check this and perform an on-site check. In addition, environmental changes identified by the mapping robot 200 are immediately shared with a plurality of service robots 300, and accordingly, each service robot 300 drives in a direction that minimizes the impact of environmental changes. A safe service is achieved. That is, the server 100 can control the driving motion of the service robot 300 that passes or is scheduled to pass through the corresponding location depending on the type of environmental change, and specific embodiments of the control method for each type are shown in FIGS. 7 to 7. Please use 11 to explain.

Figure 7 is an example of a frame classified into an obstacle type through an environmental change classifier according to an embodiment of the present invention.

In step S303, a frame containing more random objects than the frame of the previous cycle may be found among the image data for specific location data. Alternatively, image data of location data that was not collected in the previous cycle may be discovered. As shown in FIG. 7, objects appearing in a frame are processed as bounding boxes, and bounding boxes that were not in the frame of the previous cycle for the same location data are detected or image data at a location that was not previously collected is detected. If detected, the server 100 may input the frame or all frames of the image data into the environmental change classifier.

In step S304, when the frame is input to the environment change classifier, it is classified into an obstacle type, and the image of the detected bounding box is extracted as a feature value. For example, in FIG. 7, a construction information board is extracted as feature values in FIG. 7(a), bumps and risers are extracted in FIG. 7(b), and a revolving door is extracted as feature values in FIG. 7(c). Based on these feature values, the frame may be further classified into one subtype of handleable obstacles and non-handleable obstacles.

In step S306, when the frame is classified as a handleable obstacle, as in the case of FIG. 7(a), the server 100 provides service so that the running service robots 300 do not pass the location of the environmental change for a predetermined period of time. Autonomous driving of the robot 300 can be controlled. Here, the predetermined time may be the time taken for the obstacle to be handled by the manager, and is preset and stored in the server 100.

Conversely, as in the case of FIGS. 7(b) and 7(c), if the frame is classified as an obstacle that cannot be processed, the server 100 creates an avoidance path to avoid the location of the environmental change and moves the running service robot. It can be sent to (300). That is, the existing driving path stored in each service robot 300 is updated to an avoidance path, and accordingly, the service robot 300 drives along the avoidance path.

Figure 8 is an example of a frame classified by weather type through an environmental change classifier according to an embodiment of the present invention.

In step S303, a frame in which it is determined that there is a change in weather for specific location data, as shown in FIG. 8, may be found. For example, a frame in which pixel values of a preset area or more differ by more than a preset range may be detected compared to the frame of the previous cycle for the same location data. Alternatively, image data of location data that was not collected in the previous cycle may be discovered. In this case, the server 100 may input the corresponding frames into an environmental change classifier.

In step S304, when the frame is input to the environmental change classifier, it is classified into a weather type, and the part representing the weather or the part with a different pixel value is extracted as a feature value, and the weather situation can be identified based on this. Weather conditions are preset classification values when building an environmental change classifier, and may include, for example, snow, rain, fog, yellow dust, and darkness as shown in Figure 8, but are not limited thereto. Additionally, depending on the identified weather situation, the frame may be further classified into one subtype of drivable weather and non-drivable weather.

In step S306, if the frame is classified as driving weather, the server 100 may transmit a signal of low-speed driving to the driving service robot 300. Each service robot 300 travels at a reduced speed according to a low-speed driving signal. Conversely, if the frame is classified as undriving weather, the server 100 may transmit a stop signal to the driving service robot 300. Each service robot 300 performs a stop operation after receiving a signal, and when the weather is determined to be suitable for driving by the mapping robot 200, it resumes driving under the control of the server 100.

Figure 9 is an example of a frame classified by road surface texture type through an environmental change classifier according to an embodiment of the present invention.

In step S303, a frame in which it is determined that there is a change in the state of the road surface can be found for specific location data, as shown in FIG. 9. For example, a frame in which pixel values of a preset area or more differ by more than a preset range may be detected compared to the frame of the previous cycle for the same location data. Alternatively, image data of location data that was not collected in the previous cycle may be discovered. In this case, the server 100 may input the corresponding frames into an environmental change classifier.

In step S304, when the frame is input to the environmental change classifier, it is classified into the texture type of the road surface, and the part representing the road surface or the part with different pixel values is extracted as a feature value, and the road surface condition can be identified based on this. The road surface condition is a preset classification value when building an environmental change classifier, and as shown in FIG. 9, for example, it may include sand, gravel, grass, puddles, ice, etc., but is not limited thereto. Additionally, depending on the identified road surface condition, the frame may be further classified into one subtype of a drivable road surface and a non-drivable road surface.

In step S306, when the frame is classified as a drivable road surface, such as the grass in FIG. 9(a), the server 100 may transmit a low-speed driving signal to the driving service robot 300. Each service robot 300 travels at a reduced speed according to a low-speed driving signal. On the contrary, when the road surface is classified as undrivable due to heavy gravel as shown in FIG. 9(b), the server 100 creates an avoidance route to avoid the location of environmental changes and transmits it to the running service robot 300. You can. That is, the existing driving path stored in each service robot 300 is updated to an avoidance path, and accordingly, the service robot 300 drives along the avoidance path.

Figure 10 is an example of a frame classified by human density type through an environmental change classifier according to an embodiment of the present invention.

In step S303, a frame in which a large number of people are distributed can be found, as shown in FIG. 10. For example, people appearing in a frame are processed as bounding boxes, and the server 100 can input a frame with multiple bounding boxes into an environmental change classifier.

In step S304, when the frame is input to the environmental change classifier, it is classified into the type of human density, and the distribution state of the bounding box can be extracted as a feature value. For example, the distribution state of the bounding boxes may include at least one of the number of bounding boxes, the distance between adjacent bounding boxes, and the ratio of the area without bounding boxes to the area of the entire frame, but is not limited to this. Based on these feature values, the frame may be further classified into one subtype of drivable area and non-drivable area. For example, the environmental change classifier detects an undriveable area when the number of bounding boxes is more than a preset number, the distance between bounding boxes is less than a preset length value, or the ratio of the area without bounding boxes is less than a preset threshold. It can be learned to be classified as.

In step S306, when the frame is classified as a drivable area, as in the case of FIG. 10(a), the server 100 may transmit a low-speed driving signal to the driving service robot 300. Each service robot 300 travels at a reduced speed according to a low-speed driving signal. Conversely, if the frame is classified as an area that cannot be driven, the server 100 may create an avoidance route to avoid the location of environmental changes and transmit it to the driving service robot 300. Alternatively, the server 100 may control the autonomous driving of the service robots 300 so that the traveling service robots 300 do not pass a location where the environment changes for a predetermined period of time. Here, the predetermined time may be the time taken until the mapping robot 200 determines that the density of people has been reduced.

Figure 11 is an example of a frame classified into theft and collision types through an environmental change classifier according to an embodiment of the present invention.

For example, when a person applies force to the mapping robot 200 in order to steal the robot or a loaded object, or when the mapping robot 200 collides with a specific object, it includes a part where the focus is shaken from the mapping robot 200. A frame may be transmitted.

Referring to FIG. 11, in step S304, when a frame containing an out-of-focus portion 10 is input to an environmental change classifier and feature values with reduced resolution are extracted, the frame can be classified into theft and collision types. there is. Here, the range of reduced resolution can be set in advance in the environmental change classifier.

In step S306, if the frame is classified as theft or collision type, the server 100 may provide the location of the environmental change and a preset danger signal to the administrator terminal. Accordingly, managers recognize the risk and take immediate action. Alternatively, the server 100 may control the autonomous driving of each service robot 300 so that the driving service robot 300 avoids the location of environmental changes for a predetermined period of time. For example, the service robot 300 may be controlled by the administrator to avoid the location for a predetermined period of time when the theft and collision situation is resolved.

Although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be.

Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: server
200: Mapping robot
300: Service robot
50: map
60: Update map

Claims (18)

In a method of controlling a self-driving robot through image learning-based map update performed by a server,
a) collecting sensing data including image data taken of the space in real time from a mapping robot that periodically autonomously drives through a specific space and senses the space;
b) comparing the sensing data with sensing data collected in a previous cycle to recognize environmental changes occurring in the space and determine the location of the environmental changes;
c) Among the image data, the frame in which the location of the environmental change is captured is input into an environmental change classifier built by a preset image learning model to determine the type of environmental change, and the judgment result is stored on a map of the space previously stored. updating the map based on the reflection; and
d) controlling autonomous driving of a service robot to perform a preset service via the location of the environmental change according to the determination result;
Including,
In step c),
The type is preset as a standard by which data input to the environmental change classifier is classified and includes obstacles, weather, road surface texture, human density, and types of theft and collision,
When the frame is input into the environmental change classifier, it is classified into at least one type,
When the frame is input to the environmental change classifier and the people included in the frame are processed as bounding boxes and classified into human density types, drivable areas and non-drivable areas are based on the characteristic values of the frame representing people. It is further classified into one subtype of:
The feature value of the frame representing the person is extracted as the distribution state of bounding boxes, including at least one of the number of bounding boxes, the distance between adjacent bounding boxes, and the ratio of the area without bounding boxes to the area of the entire frame. In,A self-driving robot control method. According to clause 1,
The mapping robot includes a position sensor that detects the location data of the mapping robot in real time, a spatial recognition sensor that acquires 3D spatial data about the surrounding terrain of the mapping robot, and a sensor that acquires image data about the surroundings of the mapping robot. Includes a 360-degree camera sensor,
The 3D spatial data and image data are matched and stored for each location data. According to clause 2,
The service robot is designed to be inexpensive and does not include the spatial recognition sensor and the 360-degree camera sensor. According to clause 2,
Before step a), generating and storing a map of the space based on the location data, the three-dimensional space data, and the image data. A method for controlling a self-driving robot, further comprising: According to clause 2,
In step b),
Comparing the image data with image data collected in a previous cycle for each same location data, and identifying location data in which different frames are found as a result of the comparison;
determining the location of the environmental change based on identified location data and three-dimensional spatial data matched to the identified location data; and
extracting coordinate data of the map corresponding to the location of the environmental change;
A self-driving robot control method including. According to clause 1,
Before step a), learning images containing objects representing preset types of environmental changes are collected, and when one learning image is input, machine learning is performed using the image learning model so that at least one of the types of environmental changes is output. Constructing the environment change classifier by repeating; Further comprising, a self-driving robot control method. delete According to clause 1,
In step c), when the frame is input to the environment change classifier and classified into an obstacle type, it is further classified into one subtype of a treatable obstacle and an unprocessable obstacle based on the characteristic value of the frame representing the obstacle. A method for controlling a self-driving robot. According to clause 8,
In step d),
If the frame is classified as a subtype of a handleable obstacle, controlling autonomous driving of the service robot so that the service robot does not pass the location of the environmental change for a predetermined period of time,
If the frame is classified as a subtype of an obstacle that cannot be handled, generating an avoidance path for the location of the environmental change and transmitting it to the service robot. According to clause 1,
In step c), when the frame is input to the environmental change classifier and classified into a weather type, the weather situation in the space is identified based on the feature value of the frame representing the weather, and the weather condition that can be driven according to the weather situation is determined. and a method for controlling an autonomous robot, which is further classified into one subtype of non-driving weather. According to clause 10,
In step d),
If the frame is classified as a subtype of drivable weather, transmitting a signal of low-speed driving to the service robot,
If the frame is classified as a subtype of weather that is impossible to drive, transmitting a stop signal to the service robot. A method for controlling an autonomous robot, including. According to clause 1,
In step c), when the frame is input to the environmental change classifier and classified into a road surface texture type, the road surface condition is identified based on the feature value of the frame representing the road surface texture, and the road surface condition is determined to allow driving according to the road surface condition. A method of controlling an autonomous robot, which is further classified into one subtype of road surface and non-divable road surface. According to clause 12,
In step d),
If the frame is classified into a subtype of a drivable road surface, a signal of low-speed driving is transmitted to the service robot,
If the frame is classified as a subtype of an undriveable road surface, generating an avoidance route for the location of the environmental change and transmitting it to the service robot. A method for controlling an autonomous driving robot comprising a. delete According to clause 1,
In step d),
If the frame is classified into a subtype of a drivable area, a signal of low-speed driving is transmitted to the service robot,
If the frame is classified as a subtype of an untravelable area, an avoidance route for the location of the environmental change is created and transmitted to the service robot, or the service robot is prevented from passing the location of the environmental change for a predetermined period of time. A method for controlling an autonomous driving robot, comprising: controlling the autonomous driving of a service robot. According to clause 1,
In step c), when the frame is input to the environment change classifier and feature values with reduced resolution are extracted, the frame is classified into theft and collision types. According to clause 16,
In step d), if the frame is classified as a theft or collision type, the location of the environmental change and a danger signal are provided to the administrator terminal, and the service robot avoids the location of the environmental change for a predetermined period of time. A method for controlling an autonomous driving robot, comprising: controlling the autonomous driving of a service robot. In the autonomous robot control server,
Memory storing a program that performs a self-driving robot control method through image learning-based map update; and
Includes a processor that executes the program,
The above method is,
a) collecting sensing data including image data taken of the space in real time from a mapping robot that periodically autonomously drives through a specific space and senses the space;
b) comparing the sensing data with sensing data collected in a previous cycle to recognize environmental changes occurring in the space and determine the location of the environmental changes;
c) Among the image data, the frame in which the location of the environmental change is captured is input into an environmental change classifier built by a preset image learning model to determine the type of environmental change, and the judgment result is stored in the previously stored map of the space. updating the map based on the reflection; and
d) controlling autonomous driving of a service robot to perform a preset service via the location of the environmental change according to the determination result;
Including,
In step c),
The type is preset as a standard by which data input to the environmental change classifier is classified and includes obstacles, weather, road surface texture, human density, and types of theft and collision,
When the frame is input into the environmental change classifier, it is classified into at least one type,
When the frame is input to the environmental change classifier and the people included in the frame are processed as bounding boxes and classified into human density types, drivable areas and non-drivable areas are based on the characteristic values of the frame representing people. It is further classified into one subtype of:
The feature value of the frame representing the person is extracted as the distribution state of bounding boxes, including at least one of the number of bounding boxes, the distance between adjacent bounding boxes, and the ratio of the area without bounding boxes to the area of the entire frame. In,autonomous robot control server.

Images