KR102181815B1 - Personalized arrangement robot and arrangement method using the same - Google Patents

Abstract

The present invention relates to a personalized organizing robot that recognizes personal preferences and organizes household items, thereby reducing a sense of heterogeneity in organizing things, and helping users find things easily. To this end, the present invention, an image photographing unit for photographing an object; An object recognition unit for recognizing an object photographed through the image capturing unit; A moving unit for moving the object; An object location map unit for storing the predicted position of the object; And a control unit for controlling the movement performing unit by comparing the predicted position by the object location map unit with the actual position of the object photographed by the image capturing unit.
Therefore, according to the present invention, it is possible to provide an organization robot that can organize items used by individuals, and there is an advantage in that items can be arranged in the most preferred location by grasping the personal tendency.

Description

Personalized arrangement robot and arrangement method using the same}

The present invention relates to a personalized organizing robot and a method for organizing objects using the same, and specifically, an individual who helps users to find things easily, reducing the sense of disparity in organizing things by identifying personal preferences and organizing household items. It relates to a customized organizing robot and an object organizing method using the same.

With the development of robot technology, not only industrial robots but also home robots are increasingly being studied. In particular, household robots are a field in which cleaning robots are most actively commercialized. These cleaning robots store indoor maps to understand the structure of the room, and they move around themselves to absorb dust or wipe the floor using a vacuum absorber.

As a prior art, referring to Registration Patent No. 10-0877072 shown in FIG. 1, a mobile robot for cleaning includes a feature generation unit that generates a feature map for recognizing the location of the mobile robot, and the feature map. A path map generation unit that includes a plurality of cells based on the obtained attitude information of the mobile robot, and each cell generates a path map having obstacle presence information and path information, and the mobile robot according to the obstacle presence information and path information. It consists of a movement control unit that moves.

According to the patent, since a cleaning path is generated using a feature map and a path map, there is an advantage in that cleaning time is significantly reduced.

However, such a prior art only serves to clean the floor according to a predetermined path, and there is no method or configuration for organizing objects according to personal preferences.

Therefore, there is a need for research on a robot that can organize personal items by grasping the propensity of an individual, not a fixed route.

The present invention is to solve the above problems, and it is an object of the present invention to provide a personalized organizing robot for organizing and arranging personal objects in a preferred position by self-learning personal preferences.

In addition, it is another object of the present invention to provide information that can help a user when searching for an object by locating a personal object.

In order to achieve the above object, the present invention, an image photographing unit for photographing an object; An object recognition unit for recognizing an object photographed through the image capturing unit; A moving unit for moving the object; An object location map unit for storing the predicted position of the object; And a control unit for controlling the movement performing unit by comparing the predicted position by the object location map unit with the actual position of the object photographed by the image capturing unit.

It is preferable that the object location map unit includes the position and recognition time of the object recognized through the object recognition unit as input data, and includes the predicted position or discarding status of each object as output data to perform machine learning.

The control unit determines the presence or absence of a corresponding predicted position with respect to the object recognized by the object recognition unit by referring to the object location map unit, and if the predicted position exists, calculates an error distance between the predicted position and the measured position, and the When the error distance is more than a certain distance, it is preferable to control the moving unit to move the object to the predicted position.

The personalized organizing robot further includes a personal propensity determining unit, and the personal propensity determining unit calculates the number of movements in which the error distance is greater than or equal to a certain distance, and when the number of movements is greater than a certain number and the measured position is within a certain range, the The measured position can be changed to the predicted position.

The personal preference determining unit may update the object location map unit by receiving feedback on the object moved by the moving unit.

The personalized organizing robot further includes a display unit for giving information to a user, and the control unit searches for the object location map unit when the position information of a specific object is requested and displays the position information of the object through the display unit. desirable.

It is preferable that the object recognition unit classifies the types of objects photographed by the image capturing unit, and the object location map unit stores predicted positions for each type of object recognized by the object recognition unit.

The object recognition unit stores the photographing date of the object photographed by the image photographing unit, and the control unit compares the setting date and the photographing date with respect to the object recognized by the object recognition unit, and the photographing date is equal to or equal to the setting date. After that, it is preferable to control the moving part to move the object to the predicted position.

According to the present invention, further, the step of photographing an object by an image photographing unit; Recognizing an object photographed through the image capturing unit by an object recognition unit; Storing the predicted position of the object in an object location map unit; And the control unit compares the predicted position by the object location map unit with the measured position of the object photographed by the image capturing unit, and controls the moving unit to move the object. Provides.

It is preferable that the object location map unit includes the position and recognition time of the object recognized through the object recognition unit as input data, and includes the predicted position or discarding status of each object as output data to perform machine learning.

According to the present invention, it is possible to provide an organization robot capable of organizing items used by an individual, and there is an advantage in that items can be arranged in a most preferred location by grasping an individual's tendency.

In addition, according to the present invention, it is possible to organize things efficiently, and when a user finds a specific object, it is advantageous in that it can be conveniently used by displaying information on the object.

1 is a configuration diagram showing the configuration of a cleaning robot according to the prior art;
Figure 2 is a configuration diagram showing the configuration of a personalized organizing robot according to the present invention;
3 is a block diagram showing the configuration of an object recognition unit in FIG. 2;
4 is an exemplary view showing an example of a personalized organizing robot according to the present invention;
5 is an exemplary view showing a learning method of an object location map unit according to the present invention;
6 is an explanatory diagram showing a method of arranging a personalized organization robot according to the present invention.

The configuration and operation of the embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 and 6, the personalized organizing robot according to the present invention includes an image capturing unit 110, an object recognition unit 120, a movement performing unit 130, an object location map unit 140, and a personal preference determination unit. 150, the control unit 160, the communication unit 170, it is configured to include a driving unit 180.

The image capturing unit 110 may be configured to include one or more sensors capable of capturing a front image, and may include an RGB camera, an RGBD camera, a lidar, an infrared scanner, an ultrasonic sensor, an LRF sensor, and the like. I can.

The object recognition unit 120 generates a label by recognizing an object included in the image using an image captured by the image capturing unit 110.

Specifically, referring to FIG. 3, the object recognition unit 120 includes an image receiving unit 121, a feature point extracting unit 122, an outline extracting unit 123, and a label matching unit 124.

The image receiving unit 121 may receive an image photographed by the image capturing unit 110 or may receive an externally photographed image through the communication unit 160. In the image received through the image receiving unit 121, feature points are extracted from the feature point extracting unit 122. The feature points may be points having a large change in pixel (color or brightness), and may be, for example, corners or vertices of an object.

In addition, the outline extracting unit 123 serves to extract the appearance of the photographed object. A segmentation technique may be used as a method of extracting the contour.

The image information of the object separated by the contour extraction unit 123 is classified by the label matching unit 124 to generate a label.

The label matching unit 124 may be composed of an artificial intelligence neural network such as deep learning including an input layer, a hidden layer, and an output layer, and weight information is stored in the hidden layer using the learned data.

The object recognition unit can classify the types of objects photographed by the image photographing unit in this way, and store the photographing date of the object photographed by the image photographing unit.

The movement performing unit 130 is formed in front of the main body as shown in FIG. 4 and serves to grip an object photographed from the image capturing unit 110, and is preferably provided with a soft gripper. That is, when the object photographed from the image capturing unit 110 can be grasped under the control of the control unit, the movement performing unit 130 holds the object and releases the gripping of the object after moving to a specific place by the driving unit 180. can do.

The movement performing unit 130 may be provided in various forms, such as a robot hand imitating five fingers, an absorber capable of vacuum adsorption, and robot tongs provided in two directions to pick up objects according to design conditions.

The control unit may determine whether gripping is possible through the image of the object captured by the image capturing unit. That is, the control unit classifies the type of the object by matching the image of the object captured by the image capturing unit with the image information of each object with a database of the object type stored in the storage unit, and classifies the weight of the object or According to the shape, it is determined whether the grip is possible through the moving part.

In addition, the control unit can learn the object when it is determined that it can be gripped by the moving unit. Here, the controller can learn the object through the GAN-based picking learning policy network. For example, the control unit learns about an object to be actually gripped through a generator and generates false data based on this. In addition, the control unit performs learning to determine whether the data provided by the creator is real or false through a discriminor. In this way, the controller can process a large amount of data without a purification process, and by learning by itself, it is possible to solve the problem that it is difficult to secure learning data.

According to the present invention, a large amount of data can be managed by setting a management server separately and establishing a database on the management server. In other words, the management server receives the image of the object detected in front from the personalized organizing robot, or transmits or grips the object type determined by the server control unit or the weight or shape according to the object type back to the personalized organizing robot. You can send data about the availability.

In addition, the server controller may identify the type of the object included in the image by matching the image of the object received from the personalized organizing robot with image information stored in the database.

The object location map unit 140 serves to store and update the predicted location of the user object. The predicted position may be a user's preferred position for each object, and may be viewed as data obtained by matching a place where the object should be located to a predetermined indoor location. In particular, the object location map unit 140 may store the predicted position for each type of object recognized by the object recognition unit.

The object location map unit 140 stores an indoor map including an indoor structure, and also learns the position and pose of the object through machine learning such as an artificial neural network or deep learning as shown in FIG. 5 to predict the position of each object. It will store the information. For example, when the object recognition unit recognizes the cup, the predicted position of the cup may be stored in the kitchen, and when the clothes are recognized, the predicted position of the clothes may be stored in a laundry room or the like.

The indoor map may store location and structure information of a room, living room, kitchen, and toilet, and may store pre-made map information or generate an indoor map based on image information acquired through a driving unit. In addition, the predicted position of an object may correspond to a two-dimensional or three-dimensional coordinate to each object based on an indoor map, or to a functional partition space such as a room, a kitchen, and a toilet.

When learning the predicted position, the input layer may include the location and recognition time of the object acquired through the image capturing unit 110 as input data, and the output layer may include the predicted position or discarding of each object as output data. have.

For example, when a bottle is recognized through an imaging unit, the volume of the contents is estimated by analyzing the color of the bottle, and if the volume of the contents is determined to be less than a certain volume, it may be determined to be discarded. In addition, in the case of a cookie box, if the box recognized through the image capturing unit is open, it may be determined whether there is a cookie, and if there is no cookie, it may be determined to be discarded.

As described above, in the case of an object determined to be discarded, the predicted position is stored in the trash, so that the moving unit can grasp the object and move it to the trash.

In addition, even for the same object, the predicted position may vary according to the state of the object recognized by the object recognition unit 120. For example, when an object is recognized as a coke bottle by the object recognition department, if it is recognized that the coke bottle is full of coke, the predicted position may be a refrigerator, and if the coke bottle is recognized as empty, it is discarded. Judge.

The driving unit 180 is configured to move the main body, and may be composed of a wheel installed under the main body and a motor for driving the wheel, or may be composed of a leg capable of walking.

The main body is equipped with devices or software such as an image capturing unit, object recognition unit, and moving unit, and the main body is capable of adjusting height or maintaining a certain height in order to recognize and move indoor objects at a certain height. It is desirable to be configured.

In addition, the communication unit 170 provided in the main body is provided to perform wireless communication such as Bluetooth, Wi-Fi, and LTE, and may communicate with a management server through a communication network. In particular, the communication unit 170 transmits the image of the object in front photographed by the image capturing unit 110 to the management server, or the type of object corresponding to the image transmitted from the management server or whether or not the object can be gripped. Data can be received.

In addition, an input unit 190 and a display unit 195 may be provided in the main body. The input unit 190 may include one or more buttons. For example, the input unit 190 may include a button for inputting a command for confirming a predicted location, image information, an object location, or the like, or a button for resetting the command, a delete button, a cleanup start button, a stop button, and the like. In addition, the input unit 190 may include a button for setting or deleting reservation information. The input unit 190 may include at least one of a keypad and a touch pad that generate an input signal according to a user's touch or manipulation. In this case, the input unit is configured in the form of a single touch panel together with the display unit to perform input and display functions at the same time.

The display unit 195 displays reservation information, battery status, arrangement method, driving method, predicted position, and object position on the screen. The display unit may output a current state and an organized state of each unit of the main body.

The controller 160 serves to control the operation of each component unit of the main body. In addition, the main body moves around the indoor space through the driving unit and scans indoor objects periodically or at a set time.

The controller 160 recognizes the object by analyzing the image captured by the object recognition unit based on the image captured through the image capture unit, and determines whether there is a predicted position corresponding to the recognized object by referring to the object location map unit. .

At this time, when there is a predicted position corresponding to the recognized object, an error distance between the predicted position and the measured position photographed through the image capturing unit is calculated. The error distance may be calculated as a straight line distance with reference to the indoor map, but is not limited thereto and may be calculated in a manner suitable for a situation. For example, it can be calculated as the shortest distance taking into account an indoor wall or door, not the distance of coordinates, but the distance between functional compartments such as rooms, kitchens, and toilets. The functional partition space can be divided into functional objects such as a bed, a desk, and a chair even within a room. If the predicted position and the measured position of the recognized object are the same, the error distance is calculated as 0.

The control unit calculates the error distance for each object and stores it in the storage unit in the same manner as described above. If the error distance is greater than a certain distance, the moving unit can control the object to hold the object and then move the object to the predicted position.

At this time, the error distance may be different from the general distance concept. For example, if there is a desk next to a bed in the same room, even if the measured distance measured by a ruler is a short distance, the error distance may be stored as a very large distance if the functional partition space is changed. That is, the error distance can be calculated by multiplying the weight of the functional partition space by the measured distance measured with a ruler. When the functional partition space changes, the weight is set to increase. That is, if the functional partition space of the predicted position and the measured position is different, the weight may be multiplied by a value greater than 1, and if the functional partition space of the predicted position and the measured position are the same, the weight may be multiplied by a value less than 1.

When the entire indoor scan is finished, the control unit sorts the objects with the largest error distances, and then can start organizing the objects with the largest error distance.

On the other hand, the setting date or the setting number of times may be stored according to the recognized object. The setting date or number of times of setting is the date or number of times to take some action on the object. For example, when a specific coat is recognized, the controller may compare a set date with a shooting date recognized by the object recognition unit and perform a set operation if the shooting date is the same as or after the set date. The set operation may be an operation such as moving to the laundry room using the moving unit. In addition, when a specific food item is recognized through the object recognition unit, when a set date is reached, an operation of moving the food item to the trash bin through the moving unit may be performed.

The number of settings indicates the number of times recognized by the object recognition unit. In this case, it is preferable to increase the number of settings when the shooting date is changed.

The set date or set number may be input by a user or may be set automatically. For example, when the recognized object is a specific shirt, the number of settings is automatically determined, and when the number of times the shirt is recognized reaches the number of settings, the control unit moves the shirt to the laundry room using the movement unit.

The personal propensity determination unit 150 reflects personal propensity for household objects, unlike a general environment, and analyzes the image captured by the image capturing unit to determine the number of times the error distance for each object is more than a certain distance. That is, when the number of movements is calculated for a certain period of time and the number of movements is greater than or equal to a certain number of movements, and the measured position is within the measured error range, the measured position is changed to the predicted position in the object location map unit through the control unit.

The measured error range may mean a range within a certain distance from the stored final measured position, and also in this case, the weight may be multiplied according to a functional partition space like the error distance. That is, if the final measured position and the current measured position are the same functional partition space, the weight may be less than 1, and if the final measured position and the current measured position are the same functional partition space, the weight may be greater than 1.

The personal preference determination unit 150 may also receive a personal preference location through an external terminal. For example, a preferred location of a specific object may be input through a terminal such as a smartphone, and in this case, the personal preference determination unit changes the preferred location received through the external terminal into the predicted location of the corresponding object.

In addition, the personal preference determination unit 150 may update the object location map unit by receiving feedback on the object moved by the movement performing unit 130 and using it as input data for machine learning. In this case, it is preferable to use artificial intelligence learning based on Generative Adversarial Networks (GAN) for machine learning.

Meanwhile, the display unit 195 displays various types of information to the user, and the controller may display corresponding information through the display unit in response to a request received through the input unit 190 or an external terminal.

The object location map unit 140 stores the final position of the scanned object or the object moved through the movement performing unit. In this case, when a request for location information of a specific object is received, the control unit may search the object location map unit and display the location information of the object through the display unit.

Although the above has been described with reference to embodiments of the present invention, those of ordinary skill in the art can variously modify and modify the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. You will understand that you can change it.

110: imaging unit 120: object recognition unit
130: movement performing unit 140: object location map unit
150: personal preference determination unit 160: control unit
170: communication unit 180: driving unit
190: input unit 195: display unit

Claims (10)

delete An image photographing unit for photographing objects;
An object recognition unit for recognizing an object photographed through the image capturing unit;
A moving unit for moving the object;
An object location map unit for storing the predicted position of the object; And
And a control unit for controlling the movement performing unit by comparing the predicted position by the object location map unit and the actual measured position of the object photographed by the image capturing unit,
The object location map unit includes the position and recognition time of the object recognized through the object recognition unit as input data, and includes the predicted position or discarding of each object as output data to perform machine learning. robot.
An image photographing unit for photographing objects;
An object recognition unit for recognizing an object photographed through the image capturing unit;
A moving unit for moving the object;
An object location map unit for storing the predicted position of the object; And
And a control unit for controlling the movement performing unit by comparing the predicted position by the object location map unit and the actual measured position of the object photographed by the image capturing unit,
The control unit determines the presence or absence of a corresponding predicted position with respect to the object recognized by the object recognition unit by referring to the object location map unit, and if the predicted position exists, calculates an error distance between the predicted position and the measured position, and the Personally customized organizing robot, characterized in that when the error distance is more than a certain distance, the object is moved to a predicted position by controlling a moving unit.
The method of claim 3,
The personalized organizing robot further includes a personal preference determination unit,
The personal preference determining unit calculates the number of movements in which the error distance is more than a certain distance, and changes the measured position to a predicted position when the number of movements is more than a certain number and the measured position is within a certain range. Organizing robot.
The method of claim 4,
And the personal preference determining unit receives feedback on the moved object by the moving unit and updates the object location map unit.

An image photographing unit for photographing objects;
An object recognition unit for recognizing an object photographed through the image capturing unit;
A moving unit for moving the object;
An object location map unit for storing the predicted position of the object; And
A control unit for controlling the movement performing unit by comparing the predicted position by the object location map unit and the actual measured position of the object photographed by the image capturing unit;
It includes a display to give information to the user,
When a request for location information of a specific object is received, the control unit searches the object location map unit and displays the location information of the object through the display unit.
An image photographing unit for photographing objects;
An object recognition unit for recognizing an object photographed through the image capturing unit;
A moving unit for moving the object;
An object location map unit for storing the predicted position of the object; And
And a control unit for controlling the movement performing unit by comparing the predicted position by the object location map unit and the actual measured position of the object photographed by the image capturing unit,
The object recognition unit classifies the type of object photographed by the image capturing unit,
The object location map unit, the personalized organizing robot, characterized in that to store the predicted position for each type of object recognized by the object recognition unit.
The method of claim 3,
The object recognition unit stores the photographing date of the object photographed by the image photographing unit, and the control unit compares the setting date and the photographing date with respect to the object recognized by the object recognition unit, and the photographing date is equal to or equal to the setting date. Then, by controlling the movement performing unit to move the object to the predicted position personalized personalized organizing robot.
delete Photographing an object by an image capturing unit;
Recognizing an object photographed through the image capturing unit by an object recognition unit;
Storing the predicted position of the object in an object location map unit; And
The control unit performs the step of moving the object by comparing the predicted position by the object location map unit and the actual measured position of the object photographed by the image capturing unit, and controlling the moving unit,
The object location map unit includes the position and recognition time of the object recognized through the object recognition unit as input data, and includes the predicted position or discarding of each object as output data to perform machine learning. How to organize things by robots.

Images