KR102660089B1 - Method and apparatus for estimating depth of object, and mobile robot using the same - Google Patents

Abstract

The present invention relates to an object depth estimation method and device for estimating an optimal depth value using a low-cost sensor, and a mobile device using the same. The object depth estimation device according to the present invention includes the steps of acquiring an RGB color image and depth information, designating a region of interest through object recognition in the RGB color image, and mapping the distribution of depth values for pixels within the region of interest using the depth information. It includes a step of generating and estimating the depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object.

Description

Method and apparatus for estimating depth of object, and mobile robot using the same}

The present invention relates to a method for estimating the depth of an object, and more specifically, to an object depth estimation method and device for estimating an optimal depth value using a low-cost sensor, and a mobile device using the same.

Used in various fields such as robot vision, human computer interface, intelligent visual surveillance, 3D image acquisition, intelligent driver assistance system, etc. Active research is being conducted on methods for estimating three-dimensional depth information.

Most traditional methods for estimating 3D depth information rely on multiple images, such as stereo vision. Stereo matching is a method of estimating depth using binocular disparity that occurs in images obtained from two cameras. Although this method has many advantages, it has a fundamental limitation of requiring a pair of images obtained from two cameras for the same scene.

As an alternative to the method using binocular disparity, monocular methods are also being studied. For example, the depth from defocus (DFD) method is a single camera-based depth estimation method that estimates the degree of defocus blur using pairs of images with different focuses taken in the same scene. However, this method has the limitation of requiring a fixed camera view to capture multiple defocused images.

Korean Patent Publication No. 10-2016-0041441 (2016.04.18.)

Therefore, the purpose of the present invention is to provide an object depth estimation method and device for estimating an optimal depth value using a low-cost sensor, and a mobile device using the same.

The object depth estimation method according to the present invention includes the steps of acquiring an RGB color image and depth information, designating a region of interest through object recognition in the RGB color image, and using the depth information to determine the depth of a pixel in the region of interest. Generating a distribution chart of values, and estimating the depth value of an area in the distribution chart where the frequency is higher than a preset value as the depth of the object.

In the object depth estimation method according to the present invention, the step of specifying the region of interest is characterized by designating the region of interest through deep learning-based object recognition.

In the object depth estimation method according to the present invention, the step of generating the distribution diagram is characterized by calculating the distribution diagram through a histogram for all pixels in the region of interest.

In the object depth estimation method according to the present invention, the estimating step is characterized by estimating the average depth value of a region in the distribution map where the frequency is higher than a preset value as the depth of the object.

In the object depth estimation method according to the present invention, in the acquisition step, an RGB color image and depth information are acquired through an RGB-D camera.

An object depth estimation device according to the present invention includes a camera that acquires an RGB color image and depth information, designates an area of interest through object recognition in the RGB color image obtained from the camera, and uses the depth information to determine the area of interest within the area of interest. It is characterized in that it includes a control unit that generates a distribution chart of depth values for all pixels and estimates the depth value of an area in the distribution chart where the frequency is higher than a preset value as the depth of the object.

A mobile device according to the present invention includes a camera that acquires RGB color images and depth information, designates a region of interest through object recognition in the RGB color image obtained from the camera, and uses the depth information to determine all pixels within the region of interest. An object depth estimation device including a control unit that generates a distribution map of depth values and estimates the depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object, and detects an object by the object depth estimation device. It is characterized in that it includes a robot control unit that.

The object depth estimation method according to the present invention acquires RGB color images and depth information through a low-cost RGB-D camera, generates a distribution chart of depth values for pixels within the area of interest, and generates a distribution chart of depth values for pixels in the area of interest, and the frequency in the distribution chart is higher than a preset value. By estimating the depth value of the area as the depth of the object, an accurate depth value can be estimated in a simple way.

1 is a block diagram showing the configuration of a mobile device according to an embodiment of the present invention.
Figure 2 is a block diagram showing the configuration of an object depth estimation device according to an embodiment of the present invention.
Figure 3 is a flowchart showing an object depth estimation method according to an embodiment of the present invention.
Figure 4 is a flowchart specifically showing step S200 of Figure 3.
Figure 5 is an example diagram for explaining an object depth estimation method according to an embodiment of the present invention.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, the technical terms used in this specification, unless specifically defined in a different way in this specification, should be interpreted as meanings commonly understood by those skilled in the art in the technical field to which the present invention pertains, and are not overly comprehensive. It should not be interpreted in a literal or excessively reduced sense. Additionally, if the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that can be correctly understood by those skilled in the art. In addition, general terms used in the present invention should be interpreted according to the definition in the dictionary or according to the context, and should not be interpreted in an excessively reduced sense.

Additionally, as used herein, singular expressions include plural expressions, unless the context clearly dictates otherwise. In this application, terms such as “consists of” or “have” should not be construed as necessarily including all of the various components or steps described in the specification, and only some of the components or steps are included. It may not be possible, or it should be interpreted as including additional components or steps.

Additionally, terms including ordinal numbers, such as first, second, etc., used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected to or connected to the other component, but other components may also exist in between. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of the reference numerals, and duplicate descriptions thereof will be omitted. Additionally, when describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the attached drawings are only intended to facilitate easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the attached drawings. The spirit of the present invention should be construed as extending to all changes, equivalents, or substitutes other than the attached drawings.

Figure 1 is a block diagram showing the configuration of a mobile device according to an embodiment of the present invention.

Referring to FIG. 1, a movement device 300 according to an embodiment of the present invention may be configured to include an object depth estimation device 100 and a movement control unit 200.

The object depth estimation device 100 is mounted on the mobile device 300, detects an object while the mobile device 300 moves, or recognizes an object in a captured image so that it can avoid the detected object, and detects the depth of the object. can be estimated.

Here, the mobile device 300 may be a variety of devices that require object recognition, such as a mobile robot or self-driving car.

This object depth estimation device 100 first acquires an image of the object. Here, the object depth estimation device 100 can acquire RGB color images and depth information.

Here, the object depth estimation device 100 can acquire RGB color images and depth information by moving an RGB-D camera capable of simultaneously acquiring RGB color images and depth information. However, it is not limited to this, and the object depth estimation device 100 may acquire an RGB camera and depth information through an RGB camera capable of acquiring an RGB color image and a depth camera capable of acquiring depth information, respectively.

Additionally, the object depth estimation device 100 can designate a region of interest (Bounding Box) through object recognition in the previously acquired RGB color image. Here, the object depth estimation device 100 can designate an area of interest through deep learning-based object recognition. For example, the object depth estimation device 100 may recognize an object in an image using a CNN algorithm and designate an area of interest.

Additionally, the object depth estimation apparatus 100 may generate a distribution chart of depth values for pixels within a designated area of interest using the acquired depth information. At this time, the object depth estimation device 100 may calculate the distribution through the histogram for all pixels within the area of interest.

Additionally, the object depth estimation device 100 may estimate the depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. Specifically, the object depth estimation device 100 may estimate the average depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. That is, the object depth estimation device 100 can estimate the depth value of the object more accurately by removing the noise about the depth value within the area of interest by estimating the average depth value for the area above the frequency set value in the histogram distribution as the depth of the object. there is.

The movement control unit 200 may control the movement of the movement device 300 by detecting an object using the object depth estimation device 100 described above.

Hereinafter, the configuration of the object depth estimation device 100 according to an embodiment of the present invention will be described in detail.

Figure 2 is a block diagram showing the configuration of an object depth estimation device according to an embodiment of the present invention.

Referring to FIG. 2, the object depth estimation device 100 according to an embodiment of the present invention includes a camera 110, a storage unit 120, and a control unit 130.

The camera 110 acquires an image of an object. Here, the camera 110 can acquire RGB color images and depth information. Here, the camera 110 can acquire RGB color images and depth information by moving an RGB-D camera capable of simultaneously acquiring RGB color images and depth information. However, it is not limited to this, and the camera 110 may acquire RGB camera and depth information through an RGB camera capable of acquiring RGB color images and a depth camera capable of acquiring depth information, respectively.

The storage unit 120 is a device for storing data and stores application programs necessary for functional operation of the object depth estimation device 100. In particular, the storage unit 120 may store an algorithm for extracting a region of interest, an algorithm for generating a distribution map, an algorithm for estimating object depth, etc. This storage unit 120 may be a flash memory type, hard disk type, multimedia card micro type, or card type memory (for example, SD or XD memory, etc.). , RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory , may include at least one storage medium among a magnetic disk and an optical disk.

The control unit 130 may estimate the depth of the object through the image captured by the camera 110. This control unit 130 may be configured to include a region of interest extraction module 131, a distribution map generation module 132, and a depth estimation module 133.

The region of interest extraction module 131 can specify a region of interest (Bounding Box) through object recognition in the RGB color image acquired through the camera 110. Here, the region of interest extraction module 131 can specify the region of interest through deep learning-based object recognition. For example, the region-of-interest extraction module 131 may recognize objects in an image using a CNN algorithm and designate a region of interest.

First, the region of interest extraction module 131 can convert an input image with a specific resolution to a resolution that matches the input data of the CNN.

For example, if the input image is a high-resolution (e.g., FHD, QHD, etc.) image and the input data of the CNN is a low-resolution (e.g., SD, etc.) image, the region of interest extraction module 131 lowers the resolution of the input image. It can be converted to have a resolution according to the CNN input data by scaling. At this time, the input image may be consecutive frames that make up a video (for example, when the current time point is t, three frames with time points corresponding to t-1, t, and t+1).

Next, the region-of-interest extraction module 131 passes the pre-processed input image to a pre-trained CNN, and as the output of the CNN, bounding box candidates for several pixel positions and the probability that the object of interest is located in the bounding box candidate are calculated. A level of reliability can be obtained.

Here, CNN can estimate the location and size of a bounding box using feature maps of various sizes and output bounding box candidates composed of the estimated bounding boxes. At this time, CNN can also output the reliability of the bounding box candidates. Additionally, the coefficient values that make up the CNN can be determined by learning in advance using big data. In addition, since the configuration and operation of CNN for object detection can be easily understood by those skilled in the art, detailed description will be omitted.

Additionally, the region of interest extraction module 131 may determine the region of interest by performing a non-max suppression algorithm to select a bounding box candidate whose reliability is higher than a preset threshold.

The distribution map generation module 132 may generate a distribution map of depth values for pixels within a designated area of interest using depth information acquired through the camera 110. At this time, the distribution chart creation module 132 may calculate the distribution chart through the histogram for all pixels within the region of interest.

Additionally, the depth estimation module 133 may estimate the depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. Specifically, the depth estimation module 133 may estimate the average depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. That is, the depth estimation module 133 estimates the average depth value for an area whose frequency is greater than the value set in the histogram distribution chart as the depth of the object, thereby removing noise about the depth value within the area of interest to more accurately estimate the depth value of the object. .

Hereinafter, the object depth estimation method according to an embodiment of the present invention will be described in detail.

FIG. 3 is a flowchart showing an object depth estimation method according to an embodiment of the present invention, FIG. 4 is a flowchart specifically showing step S200 of FIG. 3, and FIG. 5 illustrates an object depth estimation method according to an embodiment of the present invention. This is an example for doing so.

Referring to FIGS. 3 to 5, first, the object depth estimation device may acquire RGB color images and depth information in step S100.

Next, an area of interest (Bounding Box) can be designated through object recognition in the RGB color image obtained in step S200. Here, the object depth estimation device can specify an area of interest through deep learning-based object recognition. For example, an object depth estimation device can use a CNN algorithm to recognize objects in an image and designate an area of interest.

Specifically, in step S210, the object depth estimation device may perform preprocessing to convert an input image with a specific resolution to a resolution suitable for the input data of the CNN.

For example, if the input image of the object depth estimation device is a high-resolution (e.g., FHD, QHD, etc.) image, and the input data of the CNN is a low-resolution (e.g., SD, etc.) image, the object depth estimation device downscales the resolution of the input image to create a CNN. It can be converted to have a resolution according to the input data. At this time, the input image may be consecutive frames that make up a video (for example, when the current time point is t, three frames with time points corresponding to t-1, t, and t+1).

Next, in step S220, the object depth estimation device passes the preprocessed input image to the pre-trained CNN, and as the output of the CNN, bounding box candidates for several pixel positions and the probability that the object of interest is located in the bounding box candidate are calculated. A level of reliability can be obtained.

Here, CNN can estimate the location and size of a bounding box using feature maps of various sizes and output bounding box candidates composed of the estimated bounding boxes. At this time, CNN can also output the reliability of the bounding box candidates. Additionally, the coefficient values that make up the CNN can be determined by learning in advance using big data. In addition, detailed descriptions of the configuration and operation of CNN for object detection will be omitted since those skilled in the art can easily understand them.

And in step S230, the object depth estimation device can determine the region of interest by performing a non-max suppression algorithm to select a bounding box candidate whose reliability is higher than a preset threshold.

Next, in step S300, the object depth estimation device may generate a distribution chart of depth values for pixels within the designated area of interest using the depth information acquired in step S100. At this time, the object depth estimation device can calculate the distribution through the histogram for all pixels within the area of interest.

Next, in step S400, the object depth estimation device may estimate the depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. Specifically, the object depth estimation device can estimate the average depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. In other words, the object depth estimation device can estimate the depth value of the object more accurately by removing noise about the depth value within the area of interest by estimating the average depth value of the area above the frequency set value in the histogram distribution as the depth of the object.

For example, as shown in FIG. 5, the object depth estimation device generates a histogram of depth values for all pixels and averages the depth values for areas with depth values exceeding a preset depth value (A). It can be estimated by the depth of the object.

In this way, the object depth estimation method according to an embodiment of the present invention acquires RGB color images and depth information through a low-cost RGB-D camera, generates a distribution map of depth values for pixels within the area of interest, and calculates the frequency numbers in the distribution map. By estimating the depth value of an area higher than a preset value as the depth of the object, an accurate depth value can be estimated in a simple way.

In addition, in describing embodiments according to the present invention, operations are depicted in the drawings in a specific order, but this means that in order to obtain desirable results, such operations must be performed in the specific order or sequential order shown, or all of the illustrated operations must be performed. They should not be understood as something that must be done. In certain cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it is possible.

100: object depth estimation device 110: camera
120: storage unit 130: control unit
131: Region of interest extraction module 132: Distribution map generation module
133: Depth estimation module 200: Robot control unit
300: moving device

Claims (7)

Obtaining RGB color image and depth information;
Designating a region of interest through object recognition in the RGB color image;
generating a distribution chart of depth values for pixels within the region of interest using the depth information;
A step of estimating the depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object,
The estimation step is,
An object depth estimation method characterized by estimating the average depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. According to paragraph 1,
The step of specifying the area of interest is,
An object depth estimation method characterized by specifying a region of interest through deep learning-based object recognition. According to paragraph 1,
The step of generating the distribution diagram is,
An object depth estimation method characterized by calculating a distribution through a histogram for all pixels within the region of interest. delete According to paragraph 1,
In the obtaining step,
An object depth estimation method characterized by acquiring RGB color images and depth information through an RGB-D camera. A camera that acquires RGB color images and depth information;
Designate an area of interest through object recognition in the RGB color image obtained from the camera, generate a distribution chart of depth values for all pixels within the area of interest using the depth information, and set the frequency in the distribution chart to a preset value. It includes a control unit that estimates the depth value of the higher area as the depth of the object,
The control unit,
An object depth estimation device characterized by estimating the average depth value of an area in the distribution map where the frequency is higher than a preset value as the depth of the object. A camera that acquires RGB color images and depth information, designates a region of interest through object recognition in the RGB color image obtained from the camera, and uses the depth information to create a distribution chart of depth values for all pixels in the region of interest. an object depth estimation device including a control unit that generates and estimates a depth value of a region in the distribution map where the frequency is higher than a preset value as the depth of the object;
It includes a movement control unit that detects an object and controls its movement by the object depth estimation device,
The control unit,
A mobile device characterized in that the average depth value of an area in the distribution map where the frequency is higher than a preset value is estimated as the depth of the object.