KR102537295B1 - Loaded object recognition device and object recognition method - Google Patents

Abstract

An object recognition device is disclosed. The object recognizing device may include a camera for acquiring a color image and a depth image of a loaded object; an angle map generating unit generating an angle map for an angle between the object and the camera based on a point cloud collected from the depth image; a background removal unit for removing an angle of a background space other than the object based on the angle map; a histogram generation unit that generates a histogram of depth values for each layer in which the object is loaded based on the angle map from which the background space is removed; and an image extraction unit generating a mask for each layer based on the depth value histogram and generating an object image for each layer by combining the mask and the color image.

Description

Loaded object recognition device and object recognition method {LOADED OBJECT RECOGNITION DEVICE AND OBJECT RECOGNITION METHOD}

The present invention relates to a loaded object recognizing device and an object recognizing method.

As the size of the online market grows, the delivery volume is also increasing exponentially. In line with this, logistics automation systems are also greatly developed.

Currently, most of these logistics unloading operations are performed manually, and this operation requires continuous intense labor intensity. In order to solve this problem, an automatic getting off system is being developed. On the other hand, the first task to be performed in automatic unloading is to grasp the loading status of logistics.

As described above, the first step to achieve automation of logistics unloading is to grasp the loading status of logistics and individual logistics.

In order to grasp the current loading status of cargo, there are a method of grasping the loading status by rotating a 3D camera or a 2D laser scanner and a method of grasping the loading status by learning the shape of the distribution. It does not exclude objects other than the area of interest that uses 3D cameras to understand the logistics loading situation, and 2D laser scanners are also the same.

However, in order to understand the loading status of logistics and individual logistics, parts other than frontline logistics, which is a target of interest, must first be removed from the image.

Accordingly, a technique for removing exterior walls and floors from an image using point cloud information obtained from a 3D camera is required.

Korea Patent Registration No. 10-2178013 (2020.11.06)

An object to be solved by the present invention is to provide an object recognition device and object recognition method capable of removing outer walls and floors from an image using point cloud information.

An object to be solved by the present invention is to provide an object recognizing apparatus and object recognizing method capable of generating mask data for each layer of a cargo by utilizing the removed depth information.

An object to be solved by the present invention is to provide an object recognition device and object recognition method capable of extracting images for each cargo layer by combining generated mask data and loaded cargo images.

According to an embodiment of the present invention, an object recognizing apparatus is provided. The object recognizing device may include a camera for acquiring a color image and a depth image of a loaded object; an angle map generating unit generating an angle map for an angle between the object and the camera based on a point cloud collected from the depth image; a background removal unit for removing an angle of a background space other than the object based on the angle map; a histogram generation unit that generates a histogram of depth values for each layer in which the object is loaded based on the angle map from which the background space is removed; and an image extraction unit generating a mask for each layer based on the depth value histogram and generating an object image for each layer by combining the mask and the color image.

Coordinates of the color image and the depth image may be corrected.

The angle map generating unit may derive a normal vector based on one coordinate in the point cloud, a right coordinate rightwardly adjacent to the coordinates, and a lower coordinate lowerwise adjacent to the coordinates, and the normal vector and the Z coordinate of the camera. The angle between them can be derived.

The background removal unit may remove a coordinate corresponding to an angle within a predetermined range based on a reference angle in the angle map from the angle map. The reference angle may be 0 degrees between the normal vector and the Z coordinate of the camera.

The background remover may correct a relative angle of the camera to the reference angle and then remove coordinates corresponding to angles within the predetermined range from the angle map.

The image extraction unit may generate the object image by combining the average position value of the layer and the color image.

Also, according to another embodiment of the present invention, a method for recognizing a loaded object is provided. The object recognition method may include an image acquisition step of acquiring a color image and a depth image of a loaded object from a camera; generating an angle map for an angle between the object and the camera based on a point cloud collected from the depth image; a background removal step of removing an angle of a background space other than the object based on the angle map; a histogram generating step of generating a histogram of depth values for each layer in which the object is loaded based on the angle map from which the background space is removed; and an image extraction step of generating a mask for each layer based on the depth value histogram and generating an object image for each layer by combining the mask and the color image.

In the angle map generating step, a normal vector may be derived based on one coordinate in the point cloud, a right coordinate adjacent to the right side of the coordinates, and a bottom coordinate adjacent to the bottom side of the coordinates. The angle between the Z coordinates can be derived.

In the removing of the background, a coordinate corresponding to an angle within a predetermined range based on a reference angle in the angle map may be removed from the angle map. The reference angle may be 0 degrees between the normal vector and the Z coordinate of the camera.

The removing of the background may include correcting a relative angle of the camera with respect to the reference angle and then removing coordinates corresponding to an angle within the predetermined range from the angle map.

According to an embodiment of the present invention, an object recognizing apparatus and object recognizing method capable of removing an outer wall and a floor from an image using point cloud information are provided.

According to an embodiment of the present invention, an object recognizing apparatus and an object recognizing method capable of generating mask data for each layer of a cargo by utilizing the removed depth information are provided.

According to an embodiment of the present invention, an object recognizing apparatus and an object recognizing method capable of extracting images for each cargo layer by combining generated mask data and loaded cargo images are provided.

As a result, it is possible to accurately and quickly grasp logistics disposed in the rear, thereby increasing the speed of disembarking the logistics.

1 is a control block diagram of an object recognition apparatus according to an embodiment of the present invention.
2 is a diagram illustrating a color image and an angle map according to an embodiment of the present invention.
3 is a diagram illustrating a depth histogram according to an embodiment of the present invention.
4 is a diagram illustrating mask and object images for each layer according to an embodiment of the present invention.
5 is a control flowchart illustrating an object recognizing method of an object recognizing apparatus according to an embodiment of the present invention.
6 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification and drawings (hereinafter referred to as 'this specification'), redundant descriptions of the same components are omitted.

In addition, in this specification, when a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but another component in the middle It should be understood that may exist. On the other hand, in this specification, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In addition, terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention.

Also, in this specification, a singular expression may include a plurality of expressions unless the context clearly indicates otherwise.

In addition, in this specification, terms such as 'include' or 'having' are only intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more It should be understood that the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any item among a plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the subject matter of the present invention will be omitted.

1 is a control block diagram of an object recognition apparatus according to an embodiment of the present invention.

As shown, the object recognition apparatus according to the present embodiment may include a camera 10, an angle map generator 20, a background remover 30, a histogram generator 40, and an image extractor 50. can The object recognition device may be composed of a camera and a single module capable of processing an image thereof, and may be attached to a robot that loads or unloads loaded cargo, or may be implemented as a computer capable of communicating with such a robot.

Each component included in the object recognition device is functionally classified and described for convenience of description, and does not mean physical distinction. The components may be composed of a plurality of modules that perform at least one function, or may be implemented as a single module.

The camera 10 may be composed of a plurality of cameras that acquire color images and depth images of loaded objects, and individually acquire color images and depth images of objects, and simultaneously capture color images and depth images of objects. It can also be implemented as a 3D sensor that does.

Although not shown, an image correction unit may be further included to unify pixel information in the image through calibration of the color image and the depth image obtained from the camera 10 . This image correction unit may be provided as a separate component, or may be implemented as a single module merged with the camera 10 . Such image correction may be performed by the angle map generator 20 .

Point cloud information about the loaded cargo may be derived through such an image obtained from the camera 10 .

A point cloud refers to data collected by a sensor (a camera in this embodiment) capable of obtaining a 3D image, and refers to a set of several points spread over a 3D space. Unlike 2D images, these point clouds contain depth, that is, Z-axis information.

The angle map generator 20 may generate an angle map for an angle between the object and the camera 10 based on the point cloud collected from the depth image.

The angle map generation unit 20 obtains two vectors using the point cloud coordinates and then generates a law vector. An angle between the normal vector and the coordinate system of the camera 10 may be derived, and an angle map may be created using this.

In Equation 1, v _{(i, j)} represents the normal vector of the corresponding angle map, d _{(i, j)} is the coordinate of the corresponding pixel, and d _{(i + 1, j)} is d _{(i, j)} The coordinates of the right pixel located on the right, d _{(i, j + 1),} represents the coordinates of the lower pixel located below d _{(i, j)} .

Calculating the angle between the derived normal vector and the z-axis of the camera coordinate system is as shown in Equation 2.

In Equation 2, z represents the unit vector of the camera z-axis corresponding to the front of the camera, and the outer wall and the floor can be removed using the angle map generated in this way.

2 is a diagram illustrating a color image and an angle map according to an embodiment of the present invention.

2 (a) shows a color image of an object captured by the camera 10, that is, a cargo, and (b) shows an angle map generated by the angle map generator 20.

As shown in (b) of FIG. 2, the outer wall and the floor are displayed in gray on the angle map generated by the angle map generator 20, and by removing these parts, an accurate region of interest for the cargo can be extracted.

The background removal unit 30 may remove angles of background spaces other than the object based on the angle map. When the background space is removed by the background removal unit 30, an angle map as shown in (c) of FIG. 2 may be generated.

To this end, the background removal unit 30 may remove coordinates corresponding to angles within a predetermined range based on the reference angle in the angle map from the angle map.

In this case, the reference angle may be set to an angle between the normal vector derived from the image of the cargo and the Z coordinate of the camera 10 being 0 degrees.

That is, if the camera 10 is located in front of the cargo and the image is obtained at the reference angle, the background removal unit 30 may be determined as an outer wall, for example, between 0 and 10 degrees or 170 and 180 degrees. angle can be removed.

At this time, the background removal unit 30 may remove the angle determined to be the outer wall by reflecting the reference angle and the relative angle of the camera. For example, if the rotation of the camera 10 generates a relative angle to the cargo area, which is the region of interest, that is, when the camera 10 is not at the reference angle position, the camera 10 is within a predetermined range by the distorted angle. may also change.

Alternatively, the background removal unit 30 may regenerate an angle map by correcting the position of the camera 10 using an angle sensor attached to the camera 10, and remove the remaining background parts except for cargo from the regenerated angle map. can

As described above, since the camera 10 may be mounted on a moving robot, it may not satisfy the reference angle because it is not fixed. In this case, according to the present embodiment, the reference angle and the distorted angle of the camera 10 may be corrected, and based on this, a background image such as an outer wall may be removed.

The histogram generation unit 40 may generate a depth value histogram for each layer in which objects are loaded based on the angle map from which the background space is removed.

3 is a diagram illustrating a depth histogram according to an embodiment of the present invention.

FIG. 3(a) shows a histogram of depth values before and after removing the outer wall, and FIG. 3(b) shows a layer determined based on the generated depth data.

As shown in (a) of FIG. 3 , when the outer wall data is removed, the layer can be more easily determined from the depth data, and based on this, a depth value for each layer can be derived.

At this time, an average position value of each layer may be calculated.

The image extractor 50 may generate a mask for each layer based on the depth value histogram, and generate an object image for each layer by combining the mask for each layer and the color image.

4 is a diagram illustrating mask and object images for each layer according to an embodiment of the present invention.

The black and white image shown on the leftmost side of FIG. 4 represents a mask for each layer, and the rightmost image represents an object image derived for each layer.

The image extraction unit 50 may generate a mask for each layer based on the average position value of the layer. The user can use only the mask for the layer he/she wants, or can recognize the loaded cargo through the mask for each layer and make a loading and unloading plan.

An object image as shown in FIG. 4 is generated by synthesizing the mask generated for each layer and the color image of the cargo.

Through the last image of FIG. 4 , it can be determined that the current cargo is loaded in three layers, and the number and size of cargo for each layer can be identified.

The identification result of these cargoes can be provided to a mechanical system for cargo unloading, and based on this, automatic unloading by the machine can be performed.

According to an example, the floor or ceiling may be removed by adjusting or controlling the angle removed from the angle map, and even if the cargo is not a confined space such as a vehicle or a room, an external background may be displayed based on the color image and depth value information. may be deleted.

5 is a control flowchart illustrating an object recognizing method of an object recognizing apparatus according to an embodiment of the present invention. Referring to FIG. 5 , the object recognition method according to the present embodiment is summarized as follows.

First, a color image and a depth image of a loaded object may be acquired from a camera, for example, a 3D camera or 3D sensor capable of capturing a 3D image of the object (S510).

According to an example, the object may be cargo that is loaded in layers, that is, a layer in the depth direction when viewed from the front of the camera, and the camera takes out the loaded cargo to move or unload the loaded cargo to another place. direction, that is, it can be installed in the vertical direction of the layer.

These cameras can be installed on robots that unload cargo, or they can be installed arbitrarily in the vertical direction of a layer.

When an image of an object is captured by the 3D camera, a depth image including depth information in addition to a color image of the object is obtained, and point cloud information having coordinate values for each pixel may be derived from the depth image.

The color image and the depth image may be derived through one camera or acquired through different cameras.

In the image acquisition step, a process of calibrating the coordinates of the color image and the depth image, that is, calibrating the coordinates of the color image and the depth image for each pixel, may be performed.

Thereafter, an angle map for an angle between the object and the camera may be generated based on the point cloud collected from the depth image (S520).

This can be done by the angle map generator of the object recognition device, and the angle map generator generates a normal vector based on one coordinate in the point cloud, the right coordinate adjacent to the right side of the point cloud, and the lower coordinate adjacent to the lower side of the coordinate. and derive the angle between the normal vector and the Z coordinate of the camera to create an angle map.

When the angle map for the direction in which the camera looks at the object is generated in this way, the angle for the background space other than the object may be removed based on the angle map (S530).

For example, if cargo is loaded in a space such as an outer wall such as a cargo vehicle or a warehouse, a ceiling, and a floor, an image corresponding to the corresponding portion may be removed in the angle removal step.

The background removal unit may remove coordinates corresponding to angles within a predetermined range based on the reference angle in the angle map from the angle map. In this case, the reference angle is an angle between the normal vector for each coordinate and the Z coordinate of the camera is 0 degrees. can be set to That is, an angle when the camera is located in front of the loaded cargo may be the reference angle.

When the reference angle is determined in this way, the background removal unit can remove coordinates between 0 and 10 or 170 and 180 that can be determined as an outer wall from the angle map. That is, by removing the portion determined to be the outer wall, the loaded cargo may be limited to the region of interest.

In this case, the background remover may remove coordinates corresponding to a specific angle from the angle map by reflecting the reference angle and the relative angle of the camera. For example, if a color image and a depth image are acquired from a direction rotated by about 10 degrees, rather than from the front of the screen on which the camera is loaded, the background removal unit reflects these relative angles and adjusts them between -10 and 0 or 180 and 190. coordinates can be removed.

Alternatively, according to another example, the background remover may correct a relative angle of the camera to the reference angle and then remove coordinates corresponding to angles within a predetermined range from the angle map. For example, if a color image and a depth image are obtained from a direction rotated by about 10 degrees instead of from the front of the screen on which the camera is loaded, the coordinates of the depth map are corrected in advance by reflecting this, and then, 0 to 10, Alternatively, coordinates between 170 and 180 can be removed from the angle map.

If the direction of the loaded cargo and the camera does not satisfy the reference angle, it is automatically corrected and removed to the background, such as an outer wall.

A histogram of depth values for each layer may be generated based on the angle map from which the background space is removed (S540).

Layers of loaded cargo can be distinguished through the histogram of FIG. 3 . At this time, an average position value of each radar may be calculated.

Thereafter, a mask for each layer is generated based on the depth value histogram, and an object image for each layer may be generated by combining the mask and the color image as shown in FIG. 4 (S550).

In this case, the object image may be derived by combining the average position value of the layer and the color image.

Cargo in the foremost layer, that is, the first layer, is intuitively recognized through the object image created in this way, and the robot performing unloading of the cargo can unload the cargo of the corresponding layer based on this.

When the first layer is unloaded in this way, the cargo of the second layer may be unloaded based on the object image for the second layer.

In this way, only object images for each layer can be extracted, and loaded cargoes can be classified for each layer.

As described above, the present invention utilizes a 3D camera to remove an outer wall of a loading space that may interfere with image recognition, and an object recognition device capable of grasping the loading situation for each layer of loaded cargo using depth data and suggest a method.

6 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 6 may be a device (eg, an object recognition device, etc.) described in this specification.

In the embodiment of FIG. 6 , the computing device TN100 may include at least one processor TN110, a transceiver TN120, and a memory TN130. In addition, the computing device TN100 may further include a storage device TN140, an input interface device TN150, and an output interface device TN160. Elements included in the computing device TN100 may communicate with each other by being connected by a bus TN170.

The processor TN110 may execute program commands stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Processor TN110 may be configured to implement procedures, functions, methods, and the like described in relation to embodiments of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of read only memory (ROM) and random access memory (RAM).

The transmitting/receiving device TN120 may transmit or receive a wired signal or a wireless signal. The transmitting/receiving device TN120 may perform communication by being connected to a network.

Meanwhile, the embodiments of the present invention are not implemented only through the devices and/or methods described so far, and may be implemented through a program that realizes functions corresponding to the configuration of the embodiments of the present invention or a recording medium in which the program is recorded. And, such implementation can be easily implemented by those skilled in the art from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. belong to the scope of the invention.

Claims (10)

a camera for obtaining a color image and a depth image of the loaded object;
an angle map generating unit generating an angle map for an angle between the object and the camera based on a point cloud collected from the depth image;
a background removal unit for removing an angle of a background space other than the object based on the angle map;
a histogram generating unit generating a histogram of depth values for each layer in which the object is loaded based on the angle map from which the background space is removed;
An image extraction unit generating a mask for each layer based on the depth value histogram, and generating an object image for each layer by combining the mask and the color image.
Object recognition device comprising a.
According to claim 1,
The object recognition device, characterized in that the coordinates of the color image and the depth image are calibrated.
According to claim 1,
The angle map generator derives a normal vector based on one coordinate in the point cloud, a right coordinate adjacent to the right side of the coordinates, and a bottom coordinate adjacent to the bottom side of the coordinates,
The object recognition device, characterized in that for deriving an angle between the normal vector and the Z coordinate of the camera.
According to claim 3,
The background removal unit removes a coordinate corresponding to an angle within a predetermined range based on a reference angle in the angle map from the angle map,
The reference angle is an object recognition device, characterized in that the angle between the normal vector and the Z coordinate of the camera is 0 degree.
According to claim 4,
The background removing unit corrects the relative angle of the camera with respect to the reference angle and then removes coordinates corresponding to angles within the predetermined range from the angle map.
According to claim 3,
The object recognition device, characterized in that the image extraction unit generates the object image by combining the average position value of the layer and the color image.
In the method for recognizing an object loaded with an object recognition device,
an image acquisition step of acquiring, by a camera included in the object recognizing device, a color image and a depth image of the loaded object;
generating an angle map of an angle between the object and the camera based on a point cloud collected from the depth image by an angle map generator included in the object recognition device;
a background removing step of removing, by a background removing unit included in the object recognizing device, an angle of a background space other than the object based on the angle map;
a histogram generating step of generating, by a histogram generator included in the object recognition device, a histogram of depth values for each layer in which the object is loaded based on the angle map from which the background space is removed;
An image extraction step of generating, by an image extraction unit included in the object recognition device, a mask for each layer based on the depth value histogram, and generating an object image for each layer by combining the mask and the color image.
Object recognition method comprising a.
According to claim 7,
In the step of generating the angle map,
The angle map generation unit derives a normal vector based on one coordinate in the point cloud, a right coordinate adjacent to the right side of the coordinates, and a bottom coordinate adjacent to the bottom side of the coordinates,
The method of object recognition, characterized in that for deriving an angle between the normal vector and the Z coordinate of the camera.
According to claim 8,
The background removal step is
Characterized in that the background removal unit removes a coordinate corresponding to an angle within a predetermined range based on a reference angle in the angle map from the angle map,
The reference angle is an object recognition method, characterized in that the angle between the normal vector and the Z coordinate of the camera is 0 degree.
According to claim 9,
The background removal step is
The object recognition method of claim 1 , wherein the background remover corrects the relative angle of the camera to the reference angle and then removes a coordinate corresponding to an angle within the predetermined range from the angle map.

Images