KR20200062770A - Non-contact type recognition apparatus and method of object's attibutes - Google Patents

Abstract

The present invention relates to a non-contact type object attribute recognition apparatus capable of inferring an attribute of an object and performing a necessary operation without directly carrying the object and to a method thereof. To this end, the apparatus of the present invention comprises: an image photographing unit photographing an object; a label generation unit recognizing the object photographed by the image photographing unit, classifying the type of the object, and recognizing a corresponding label; and an attribute estimation unit estimating the attribute corresponding to the label in accordance with the label recognized in the label recognition unit. Therefore, according to the present invention, the object is recognized as an image and grasped, and the behavior of a robot corresponding to the inferred attribute may be appropriately controlled by inferring the attribute of the identified object.

Description

Non-contact type recognition apparatus and method of object's attibutes}

The present invention relates to a non-contact object property recognition device and method, and specifically, to a non-contact object property recognition device and method capable of inferring the property of an object and performing necessary operations without having to directly listen to the object.

Research and development of a device for working on behalf of a person, that is, a robot, has been continuously performed. The robot recognizes the front using a camera sensor or the like, and controls its own movement using various motors.

Recently, due to the development of artificial intelligence, many methods for self-learning about recognition and classification of objects have been studied.

According to Patent No. 10-1803471 (a deep learning system using a convolutional neural network-based image patterning and an image learning method using the same), the image input unit for inputting an input image and the input image received from the image input unit are patterned CNN learning unit based on the convolutional neural network (CNN) that learns the patterning module generated from multiple pattern images, the input image received from the image input unit and the pattern image received from the patterning module, and learning information from the CNN learning unit And a deep learning system including a CNN execution unit that receives an input image received from the image input unit, and a final classification unit that receives image information from the CNN execution unit and classifies objects of image information by type. Using such a system, it is possible to learn to discern objects based on the received image.

However, such a system only provides a function to discriminate objects, and it is impossible to induce robots to infer about future actions related to objects, and thus there is a limit to apply them to robots targeting actual commercialization.

The present invention is to solve the problems as described above, by classifying and grasping the type of the object, and further inferring the property of the identified object to properly control the behavior of the robot according to the inferred property.

In order to achieve the above object, the present invention is an image photographing unit for photographing an object; A label generation unit that recognizes an object photographed by the image photographing unit, classifies the type of the object, and recognizes a corresponding label; It provides a non-contact object property recognition device comprising an attribute estimation unit for estimating an attribute corresponding to the label according to the label recognized by the label generator.

The non-contact object attribute recognition device includes an attribute database in which attributes corresponding to the label are stored, and the attribute estimator determines whether the label is a constant weight label, and if the weight is determined to be a constant weight label, the object is recognized from the captured image. It is preferable to perform a step of calculating a volume, detecting a weight attribute of the label from the attribute database and estimating the weight of the object using the calculated volume.

The attribute estimator determines whether the label is a constant weight label, and if it is determined that the weight is not constant, detecting the color difference of the object from the captured image, and estimating the volume of the content from the color difference; Estimating the weight of the object from the estimated volume may be performed.

At this time, the non-contact object attribute recognition device preferably includes an attribute database in which attributes corresponding to the label are stored, and the attribute estimator searches the attribute database to calculate the weight corresponding to the estimated volume.

According to the present invention, further, an image photographing unit for photographing an object; A moving unit for grabbing and moving the object; A label generation unit that recognizes an object photographed by the image photographing unit, classifies the type of the object, and recognizes a corresponding label; An attribute estimation unit for estimating the weight corresponding to the label according to the label recognized by the label generator; It is configured to include a control unit for controlling the moving performance unit, the control unit provides an object moving device for changing the control method of the moving performance unit according to the weight estimated by the attribute estimation.

The moving performance unit includes a plurality of holding means for holding the object, and it is preferable that the object moving device changes the holding means according to the weight estimated by the attribute estimation.

And, according to the present invention, the step of photographing the object by the image taking unit; Recognizing an object photographed by the image photographing unit by a label generator and classifying the type of the object and recognizing a corresponding label; Provided is a method for recognizing a non-contact object property that performs a step of estimating an attribute corresponding to the label according to a label recognized by the label generator by the attribute estimator.

The attribute estimator determines whether the label is a constant weight label, and if the weight is determined to be a constant label, calculating the volume of the object from the captured image, and the weight of the label from the attribute database storing the attribute corresponding to the label It is preferable to perform the step of detecting an attribute and estimating the weight of the object using the calculated volume.

The attribute estimator determines whether the label is a constant weight label, and if it is determined that the weight is not constant, detecting the color difference of the object from the captured image, and estimating the volume of the content from the color difference; Estimating the weight of the object from the estimated volume may be performed.

It is preferable that the attribute estimator calculates a weight corresponding to the estimated volume by searching an attribute database storing attributes corresponding to the label.

According to the present invention, it is possible to appropriately control the behavior of the robot corresponding to the inferred attribute by recognizing and grasping the object as an image and inferring the attribute of the identified object.

Therefore, even when the object first recognized when there is a need to grab or listen to the actual object, the robot can be properly acted without wasting resources without having to undergo trial and error.

1 is a configuration diagram showing the configuration of a non-contact object property recognition apparatus according to the present invention;
Figure 2 is a block diagram showing the configuration of the label generating unit in Figure 1;
3 is an exemplary view showing an object image taken by the image taking unit according to the present invention;
4 is an exemplary view showing an example of an attribute database in FIG. 1;
5 to 7 are explanatory diagrams for explaining the weight estimation process by attribute estimation in FIG. 1;
8 is a flowchart for explaining a method for recognizing a non-contact object property according to the present invention;
9 is a block diagram showing the configuration of an object moving apparatus according to the present invention;
10 is an exemplary view showing an example of a holding means in FIG. 8;

The configuration and operation of the embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, the non-contact object property recognition apparatus 100 according to the present invention includes an image photographing unit 110, a label generating unit 120, an attribute estimation unit 130, an attribute database 140, and a control unit 150 , It comprises a communication unit 160, a storage unit 170.

The image photographing unit 110 may include one or more sensors capable of photographing a front image, and may include an RGB camera, an RGBD camera, a lidar, an infrared scanner, an ultrasonic sensor, and an LRF sensor. Can be.

The label generator 120 recognizes an object included in the image using the image captured by the image photographing unit 110 to generate a label.

Specifically, referring to FIG. 2, the label generation unit 120 includes an image receiving unit 121, a feature point extraction unit 122, a contour extraction unit 123, and a label matching unit 124.

The image receiving unit 121 may receive an image photographed by the image photographing unit 110 or may receive an image photographed from the outside through the communication unit 160. In the image received through the image receiving unit 121, feature points are extracted from the feature point extraction unit 122. The feature points may be points in which a change in pixel (color or contrast) is large, for example, an edge or a vertex of an object.

In addition, the outline extraction unit 123 serves to extract the appearance of the photographed object. A segmentation technique may be used as a method of extracting contours.

The image information of the object separated by the contour extraction unit 123 is classified by the label matching unit 124 to generate a label.

The label matching unit 124 may be composed of an artificial intelligence neural network such as a deep learning including an input layer, a hidden layer, and an output layer, and weight information is stored in the hidden layer using the learned data.

For example, when the object shown in FIG. 3 is photographed by the image receiving unit 121, the label generating unit 120 recognizes and classifies the object after extracting the contour of the object. Labels corresponding to the classified output results may be'apples' and'cola bottles', and the attribute database 140 stores labels and attribute information corresponding to the labels.

Referring to FIG. 4, in the attribute database 140, labels generated by the label generator 120 are classified and stored according to attributes, and each label stores a corresponding attribute.

Specifically, the label is largely divided into a fixed weight label and a variable weight label and stored. The weight of the fixed weight label is estimated immediately according to the measured external volume, and the weight of the variable weight label is estimated by measuring the volume of the external volume and contents.

For example, the fixed weight label may be fruit or paper, and the weight of the fixed weight label is directly estimated based on the size (or volume) measured from the outside.

In addition, the variable weight label may be a bottle such as bottled water or coke bottle, and if the label generated by the label generator 120 corresponds to the variable weight label, as well as the size (or volume) measured from the outside, therein The estimation of the volume (or height) of the estimated contents is performed once again.

The attribute estimator 130 estimates a new attribute for the label generated by the label generator 120. The estimation may be performed using the attribute database 140 or may be performed by combining artificial intelligence algorithms using artificial neural networks such as deep learning.

For example, the attribute estimator 130 may estimate the weight of the label generated by the label generator 120.

First, in the case of the label'apple' generated in FIG. 3, referring to the attribute database 140, it is classified as a fixed weight label, and the weight is estimated according to the size of the apple. In this case, the attribute estimation unit 130 estimates the size or volume of the apple using the image taken by the image photographing unit 110. At this time, since the volume of the apple requires a 360° stereoscopic image, if a very precise estimation is not required, the weight is estimated from the size of the 2D image. That is, the attribute estimation unit estimates the size of the apple after extracting the contour of the apple from the 2D image. The size of the apple is determined by using the distance between the image capturing unit and the apple or comparing the size of other objects to be compared. The size of the apple can be estimated.

As the distance between the image capturing unit and the object increases, the size of the object in the captured image decreases, and the closer the distance between the image capturing unit and the object increases, the larger the object size in the captured image increases. If the distance between the image capturing unit and the object is set in advance, the size of the real object can be estimated from the captured image.

In this embodiment, the size of the apples stored in the attribute database 140 is divided into 9 steps and stored, and steps 1 to 9 may be a range of sizes or volumes. For example, if the size of the apple measured by analyzing the image photographed by the image photographing unit belongs to step 5, the weight of the apple may be estimated to be 210 g.

On the other hand, in the case of the label'Cola bottle' generated in FIG. 3, referring to the attribute database 140, it is classified as a variable weight label, and the weight is estimated through a process of further measuring the appearance size and volume of the contents of the Cola bottle. .

That is, the attribute estimation unit 130 estimates the external size of the Coke bottle classified as a variable weight label from the image photographed by the image photographing unit. At this time, since the size of the coke bottle is standardized, the outer size can be estimated by measuring only the height of the coke bottle.

Next, the attribute estimation unit 130 measures the size or volume of the contents accommodated in the variable weight label. At this time, the attribute estimator 130 may detect the color difference of the variable weight label from the image photographed by the image photographing unit to estimate the size or volume of the contents.

Referring to FIG. 5, the attribute estimation unit 130 determines the boundary B of the contents using the color difference of the Coke bottle. At this time, it is preferable to determine the condition of the boundary determination as first, whether the boundary is in the horizontal direction, and second, whether the color difference is greater than or equal to a threshold color difference based on the boundary.

In FIG. 5, since the color close to black is displayed below the boundary B and the color close to the background color is displayed above the boundary B, the color difference is more than a critical color difference, and the boundary is horizontal. Therefore, the attribute estimation unit 130 determines the B line as a boundary and measures the height H from the floor to the boundary. The volume of the contents can also be estimated by measuring the height of the coke bottle because it is standardized.

However, in the case of a bottle that is not standardized, the volume is estimated by measuring the width (W). Then, in the case of a cylindrical bottle, the volume can be calculated as πW ² H/4.

When the volume (or height) of the contents is calculated in this way, the attribute estimation unit 130 calculates the volume occupancy of the contents by comparing the external size (volume) of the variable weight label with the contents volume. For example, in Figure 5, the volume occupancy of the content may be calculated as 70%.

Then, the attribute estimator 130 estimates the weight corresponding to the volume occupancy by referring to the attribute database 140. At this time, if there is no accurate weight data corresponding to the calculated volume occupancy, the weight can be estimated using an interpolation method.

For example, using the linear interpolation method, if the weight is y and the volume occupancy is x, a graph of y = 11x + 300 is obtained from 50% and 100% volume occupancy data of the Coke bottle stored in the attribute database 140, If the volume occupancy is 70%, using this graph, the weight can be estimated to be 1,070g.

On the other hand, referring to FIG. 6, when the cover is covered on the outside of the bottle, the attribute estimation unit may recognize this as the content, and an error may occur. Therefore, it is desirable for the attribute estimation unit 130 to determine a boundary by adding one more condition to the image captured by the image photographing unit 110.

That is, referring to FIG. 7, the attribute estimator detects whether a step D exists in a portion determined as a boundary candidate. Referring to FIG. 7 in which the boundary of FIG. 6 is enlarged, it can be seen that there is a groove or step in the outer edge of the boundary. In addition, the color of the lower part of the step is outwardly left and right.

Therefore, in the attribute estimation, a step exists near the boundary, and when the color of the lower portion is outside the step, it is not judged as content.

Referring to Fig. 8, a method for recognizing a non-contact object property according to the present invention will be described.

When an object in front is photographed by the image photographing unit, an image of the photographed object is acquired and transmitted to the label generating unit. At this time, a plurality of images of the same object may be photographed, and a plurality of images photographed from different angles may be transmitted.

The label generator receives the object image, detects feature points and contours, and uses artificial intelligence such as deep learning to identify and create the label of the object.

Thereafter, the attribute estimator determines whether the label generated by referring to the attribute database is a fixed weight label or a variable weight label, and if it is determined that the label is a fixed weight label, calculates the size or volume of the photographed image to estimate the weight.

On the other hand, if it is determined that the generated label is a variable weight label, the attribute estimation unit grasps the category (type) of the object. For example, after grasping the type of the beverage bottle or the cookie bag, the state of the object is estimated.

In the case of a drink bottle, the ratio of the present weight to the basic weight of the 100% state is estimated by estimating what percentage of the contents of the beverage bottle. At this time, the ratio of the current weight can be estimated by referring to the attribute database or by using artificial intelligence such as deep learning.

Next, an object moving device capable of moving an object using the non-contact object property recognition device according to the present invention will be described.

The object moving apparatus 300 according to the present invention includes an image photographing unit 110, a label generating unit 120, an attribute estimation unit 130, a moving execution unit 310, a control unit 150, a communication unit 160, and storage It comprises a portion 170. The components of the same name as the non-contact object property recognition device described above have the same function, and thus the description is omitted.

It is preferable that the moving performance unit 310 is provided in the form of a robot arm so as to be able to hold an object. That is, the movement performing unit 310 may include a robot arm having a plurality of joints and various types of holding means that are provided at the ends of the robot arm to hold an object.

Referring to FIG. 9, the holding means may be provided in various forms, such as a robot hand mimicking five fingers, an adsorber capable of vacuum adsorption, and robot tongs provided in two directions to pick up an object.

The holding means have respective features, and in this embodiment, the non-contact object property recognition apparatus moves the object using the most suitable holding means according to the situation.

Specifically, the robot tongs have the advantage of reliably holding a simple object, but if the gripping force is too large, there is a fear that the object may break, so that the strength of the object must be additionally recognized. In addition, there is a concern that a complex-shaped object is difficult to grasp the gripping point and cannot lift the object properly.

The robot hand is suitable for holding a complex object, but there is a disadvantage in that an additional learning process is required by using artificial intelligence such as deep learning or using separate data to determine the gripping point and gripping power of the object. In addition, it is very difficult to hold thin paper or film with a robot hand.

The adsorber has an advantage in that a thin object or a relatively light object such as paper or film can be moved without fear of being broken. However, heavy or prone objects are difficult to move to the adsorber.

The object moving apparatus according to the present embodiment photographs an object in front of the image photographing unit 110 while moving by a moving means such as a wheel or a leg, and generates a label of the object photographed in the label generating unit 120. Then, the attribute estimator 130 estimates the attribute of the object corresponding to the label.

Then, the control unit 150 determines the object movement method of the movement performing unit 310 according to the label generated by the label generating unit 120 or the attribute estimated by the attribute estimation unit, and then moves the object in the most suitable way. . The attributes may be the appearance, weight, type, material, etc. of the object.

For example, if the label generated by the label generator 120 is a simple cylindrical glass bottle, the control unit may use a robot tongs to hold the object. In addition, if the label generated by the label generating unit 120 is a confectionery bag, the control unit may move the object using an adsorber.

In addition, the control unit controls the object moving method according to the weight of the object estimated by the attribute estimation unit 130. For example, the upper limit of the weight of an object that can be moved to the adsorber is stored in the storage unit. If the weight of the object estimated by the attribute estimation exceeds the upper limit of the weight, the controller excludes the adsorber and uses other holding means.

As described above, the object moving apparatus according to the present invention includes a plurality of holding means, generates a label through the captured image, estimates attributes according to the label, and selects the most appropriate holding means to organize the objects most efficiently. There is an advantage.

Although described above with reference to the embodiments of the present invention, those skilled in the art can variously modify the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can change it.

100: non-contact object property recognition device 110: image recording unit
120: label generating unit 130: attribute estimation
140: attribute database unit 150: control unit
160: communication unit 170: storage unit
300: object moving device 310: moving unit

Claims (10)

An image photographing unit for photographing objects;
A label generation unit that recognizes an object photographed by the image photographing unit, classifies the type of the object, and recognizes a corresponding label;
And a property estimator for estimating an attribute corresponding to the label according to the label recognized by the label generator.
According to claim 1,
The non-contact object attribute recognition device includes an attribute database in which attributes corresponding to the label are stored,
The attribute estimator determines whether the label is a constant weight label, and when it is determined that the weight is a constant label, calculating the volume of the object from the captured image,
And detecting the weight attribute of the label from the attribute database and estimating the weight of the object using the calculated volume.
According to claim 1,
The attribute estimator determines whether the label is a constant weight label, and when it is determined that the weight is not constant, detecting the color difference of the object from the captured image.
Estimating the volume of the content from the color difference;
And estimating the weight of the object from the estimated volume.
According to claim 3,
The non-contact object attribute recognition device includes an attribute database in which attributes corresponding to the label are stored,
The attribute estimator retrieves the attribute database and calculates a weight corresponding to the estimated volume.
An image photographing unit for photographing objects;
A moving unit for grabbing and moving the object;
A label generation unit that recognizes an object photographed by the image photographing unit, classifies the type of the object, and recognizes a corresponding label;
An attribute estimation unit for estimating the weight corresponding to the label according to the label recognized by the label generator;
It comprises a control unit for controlling the moving performance unit,
The control unit moves the object moving device to change the control method of the moving unit according to the weight estimated by the attribute estimator.
The method of claim 5,
The moving performance unit is provided with a plurality of holding means for holding the object,
The object moving device changes the holding means according to the weight estimated by the attribute estimation device.
Photographing an object by the image photographing unit;
Recognizing an object photographed by the image photographing unit by a label generator and classifying the type of the object and recognizing a corresponding label;
A method for recognizing a property of a non-contact object, comprising estimating an attribute corresponding to the label according to a label recognized by the label generator by the attribute estimator.
The method of claim 7,
The attribute estimator determines whether the label is a constant weight label, and when it is determined that the weight is a constant label, calculating the volume of the object from the captured image,
A method for recognizing a non-contact object property, comprising detecting a weight property of the label from the property database storing the property corresponding to the label and estimating the weight of the object using the calculated volume.
The method of claim 7,
The attribute estimator determines whether the label is a constant weight label, and when it is determined that the weight is not constant, detecting the color difference of the object from the captured image.
Estimating the volume of the content from the color difference;
And estimating the weight of the object from the estimated volume.
The method of claim 9,
The attribute estimator retrieves an attribute database storing attributes corresponding to the label and calculates a weight corresponding to the estimated volume.

Images