KR102366396B1 - RGB-D Data and Deep Learning Based 3D Instance Segmentation Method and System - Google Patents

Abstract

The embodiment relates to a 3D object region segmentation method and system using RGB-D data and deep learning.
Specifically, these methods and systems utilize RGB-D cameras and recent deep learning techniques to segment the object area in three dimensions to find the xyz and RGB values of the object.
Specifically, it is characterized in that xyz and RGB color information of the object can be obtained by performing 3D object region segmentation of the object by RGB image and depth data and deep learning of the object to be recognized.
Therefore, through this, the embodiment finally finds and expresses the 3D xyz data (ie, three-dimensional information) and RGB color information of the object while using the camera, in the case of robot vision monitoring such as automation, to display the object in three dimensions. Recognize and distinguish

Description

3D object region segmentation method and system using RGB-D data and deep learning {RGB-D Data and Deep Learning Based 3D Instance Segmentation Method and System}

The content disclosed in the present specification relates to a system for classifying an object by applying a robot vision technology for recognizing an object in three dimensions, and relates to a technology field belonging to the field of robot vision used for automation and the like.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and inclusion in this section is not an admission that it is prior art.

In general, object recognition in a robot is to determine what an object is detected by the robot, and the robot can perform an appropriate task as programmed through the determination result.

On the other hand, as a method for recognizing an object, a so-called model-based object recognition method, in which a large amount of information about a three-dimensional object is stored in a database, is compared with an object recognized by the robot, and acquired information of the object is generally is being used as The model-based object recognition method is largely divided into the storage step of information about the model object in the model database and the

It is divided into a processing step for comparison with a model object from the input information obtained from the system, that is, a feature extraction step and a recognition step.

The vision-based object recognition method used in such a conventional robot captures an object with a camera to obtain image data, processes the image data to extract the feature points of the object, and compares it with the feature points of the objects stored in the database to determine the similarity. Object recognition was performed through the process of judgment.

The prior art documents in this background are about the following patent documents.

(Patent Document 1) KR100920457 Y1

For reference, the technology of Patent Document 1 uses an RFID code detected through an RFID reader attached to a service robot, unlike conventional methods in which an object was recognized by extracting feature points from an image image obtained through a camera attached to the robot. is used to recognize objects.

On the other hand, in particular, in relation to the content disclosed in this specification, in more detail, techniques for recognizing an object using a camera have been used in the prior art, but it is difficult for a robot to fully recognize an object as two-dimensional information.

And, in the technology using deep learning, most of the areas for the object in the two-dimensional image are derived, and as a calibration process in advance, the correlation between the size of the pixel in the image and the size of the actual xy is derived with a formula Thus, the method of finding the xy region for the region to which the object belongs is utilized.

Also, in the prior art, there is a case in which z-axis information of an object is also found in a fixed environment through a correction process.

However, in the case of not undergoing a calibration process or in general object recognition, obtaining three-dimensional object area information with a camera has not been solved by the prior art.

Also, even when using an RGB-D camera, it may be possible to feel a sense of distance by using depth information, but three-dimensional information including the xyz value of a specific object area is obtained by classifying a specific object. There is still no prior art for obtaining it.

The disclosed content is a 3D object using RGB-D data and deep learning that uses an RGB-D camera and uses a recent deep learning technology to segment an object region in three dimensions to find xyz values and RGB values The present invention intends to provide a domain division method and its system.

3D object region segmentation method and system using RGB-D data and deep learning according to the embodiment,

By using RGB-D camera and recent deep learning technology, it is possible to find the xyz value and RGB value of the object by segmenting the object area in three dimensions.

Specifically, it is characterized in that xyz and RGB color information of the object can be obtained by performing 3D object region segmentation of the object by RGB image and depth data and deep learning of the object to be recognized.

According to the embodiments, when finally performing robot vision monitoring such as automation, the object is displayed in three dimensions by finding and expressing 3D xyz data (ie, three-dimensional information) and RGB color information of the object while using a camera. Recognize and differentiate

Therefore, further industrially, there is an effect that can be utilized as a vision system for 3D object recognition. And, technically, as 3D-based object recognition becomes possible, it can be used in various ways, such as a vision system for robot vision monitoring.

Additionally, it relates to a system for classifying an object by applying a robot vision technology that recognizes an object in three dimensions using an RGB-D camera, and xyz information corresponding to one or some objects in recognizing a general object and it acts as a vision system to obtain RGB information.

1A and 1B are diagrams for explaining a 3D object region division system using RGB-D data and deep learning according to an embodiment;
2 is a view showing an object division learning process (that is, the process of constructing the object 2D region division deep learning model described above) applied to the 3D object region division system of FIGS. 1A and 1B.
3 and 4 are diagrams for explaining a 3D object region division method using RGB-D data and deep learning according to an embodiment applied to the system of FIGS. 1A and 1B
5 is a view showing RGB color images and depth data applied to the system of FIGS. 1A and 1B.
6 and 7 are diagrams showing the deep learning object 2D region division process by the object 2D region division deep learning model used in the system of FIGS. 1A and 1B
8 and 9 are diagrams graphically expressing the result of 3D object region division by the system of FIGS. 1A and 1B.

1A and 1B are diagrams for explaining a 3D object region division system using RGB-D data and deep learning according to an embodiment.

Specifically, FIG. 1A is a diagram schematically illustrating the concept of such a 3D object region division system. And, FIG. 1B is a block diagram showing the configuration of the 3D object region division system.

1A and 1B, the 3D object area division system according to an embodiment performs robot vision monitoring by recognizing an object in 3D from vision data when monitoring robot vision such as automation. It presupposes a system that makes it possible.

In this state, the 3D object region division system according to an embodiment divides the object region in three dimensions using an RGB-D camera and recent deep learning technology to find the xyz value and RGB value of the object. will be.

Specifically, the system of one embodiment includes an RGB-D camera 110 that collects RGB image and depth data of an object to be recognized, and performs 3D object region segmentation of the object by RGB image and depth data and deep learning. It includes a central control panel 120 that acquires xyz and RGB color information of , and a display device 130 thereof.

The RGB-D camera 110 collects RGB images and depth data of an object to be recognized first in order to distinguish objects using robot vision, etc., and provides these two pieces of information to the central control panel. In this case, the RGB image and the depth data are used for 3D object region division of an object according to an embodiment. Additionally, these RGB-D cameras utilize Kinect, RealSense, and the like.

The central control panel 120 performs 3D object region segmentation of an object by applying deep learning only to the RGB image and depth data, so that xyz and RGB color information of the object can be obtained. Specifically, the central control panel 120 a) firstly calculates the deep learning parameters for the object region through deep learning learning for 2D object region division from only a plurality of different RGB images in advance, so that the object 2D region division deep Build a learning model. Then, the central control panel 120 performs b) object 2D region division from the RGB image from the RGB-D camera by the object 2D region division deep learning model. Next, c) the object depth information is obtained by performing logical AND between the object 2D region division result and the depth data from the RGB-D camera. Then, the central control panel 120 d) applies the perspective transformation to the depth information of the object to perform 3D object region division of the object, so that xyz and RGB color information of the object can be obtained. Therefore, through this, the system according to an embodiment divides the area of the object in three dimensions using an RGB-D camera and recent deep learning technology, and finally finds and expresses the 3D xyz data and RGB color information of the object.

When data including xyz and RGB color information of an object is extracted by the central control panel 120, the display device 130 expresses and provides the result of dividing the 3D object region of the object.

And, in this case, the central control panel 120 is configured as follows, for example.

That is, the central control panel 120 interfaces with the RGB-D camera 110 as data to receive the RGB image and depth data of the object, the I/O module 121, and the RGB image and depth data of the object. It includes a DSP unit 122 that obtains xyz and RGB color information of an object by performing digital signal processing to perform 3D object region division of an object, and a CPU 123 that controls each unit. In this case, the digital signal processing includes a deep learning operation.

Additionally, in this case, the central control panel 120 further includes a storage unit 124 that stores and provides the object 2D region division deep learning model in the DSP unit 122 through the CPU 123 in advance. do.

FIG. 2 is a diagram showing an object division learning process (ie, the process of constructing the object 2D region division deep learning model described above) applied to the 3D object region division system of FIGS. 1A and 1B .

As shown in Figure 2, the object division learning process according to an embodiment goes through a learning process of deep learning object region division using only RGB for deep learning object 2D region division.

Specifically, in the object segmentation learning process according to an embodiment, only RGB images are first collected through an RGB-D camera for an object to be recognized.

Next, data for learning is created by defining and displaying the area of the object in the RGB image in advance and designating the type of the object. Additionally, this step is a data set creation step, which is a step to create training data in a deep learning model, and is a step to define and display the area of an object in an image in advance and designate the type of the object.

Then, by applying a preset object segmentation deep learning model to these training data, only the area of the object is segmented. For example, the object segmentation deep learning model is a mask R-CNN (Mask R-CNN) is representative, and other models having the same function may be used.

On the other hand, the data set generated through the divided result is applied to a preset deep learning model for 2D object region segmentation to learn. Additionally, this step is a learning step by applying the data set to the Deep Runim model for learning. In this learning stage, the object segmentation deep learning model must be composed of S/W or H/W, or online learning is included.

Then, the deep learning parameters for the object domain are calculated from these learning results. In addition, this step is a parameter extraction step, which is a step of constructing parameters appearing as a result of the learning as soon as the learning of the deep learning model is completed. The parameter is also referred to as a weight file.

So, an object 2D region division deep learning model is constructed from the deep learning parameters related to the object region. This step is a step of constructing a S/W or H/W system to apply and utilize the extracted parameters as the deep learning system step. This system is utilized in deep learning object 2D segmentation for 3D object segmentation.

For example, the object 2D region division deep learning model extracts a representative threshold value for each 2D object region through mask R-CNN type deep learning learning for 2D object region division from a number of different RGB images in advance to create a binary mask. By creating it, it consists of a mask R-CNN based object 2D region segmentation deep learning model.

3 and 4 are diagrams for explaining a 3D object region segmentation method using RGB-D data and deep learning according to an embodiment.

Specifically, FIG. 3 is a flowchart sequentially illustrating a 3D object region segmentation method using RGB-D data and deep learning according to an embodiment applied to the system of FIGS. 1A and 1B (see FIG. 2 ). And, Figure 4 is a view for explaining the perspective transformation applied to this method.

First, as shown in FIG. 3, the 3D object area division method according to an embodiment can perform robot vision monitoring by first recognizing an object in 3D from vision data when monitoring robot vision such as automation. presupposes that it be

And, this object recognition is entirely performed by the above-described central control panel.

In this state, the 3D object region segmentation method according to an embodiment preferentially calculates deep learning parameters for the object region through deep learning learning for 2D object region segmentation from only a plurality of different RGB images in advance to calculate the object 2D region A segmented deep learning model is built (S301). For reference, the construction will be described later in more detail with reference to FIG. 5 .

And, in this method, object 2D region division from the RGB image from the RGB-D camera (S302) is performed by the object 2D region division deep learning model (S303).

Next, by performing a logical AND between the object 2D region division result and the depth data from the RGB-D camera, depth information of the object is obtained (S304). That is, only the area where the depth data and the object 2D area division overlap is divided.

Then, by performing a 3D object region segmentation of the object by applying perspective transformation to the depth information of the object (S405), xyz and RGB color information of the object can be obtained. That is, in this perspective transformation step, xyz information is extracted by using perspective transformation on the region-divided depth information shown in the previous step.

In general, perspective transformation can be described with a drawing such as FIG. 4 .

The depth data of RGB-D data provides coordinates (u,v) in the image plane (U,V), the image coordinates (u,v), and a depth value ( d ), as shown in FIG. 10 . From this information, the actual coordinates ( x,y,z ) based on the origin (0,0,0) of the camera can be obtained from the following equation.

When the distance from the origin (0,0,0) to ( u,v ) is l , f is the focal length, and d is the depth value, l is expressed as . At this time, the position ( x,y,z ) of the object in the (X, Y, Z) coordinate system with respect to the origin can be calculated as , , .

From this equation, (x, y, z) can be obtained from the (u, v) and d information provided by the RGB-D data in the coordinate system with respect to the camera.

Using this, the set of (x,y,z) coordinates of the real object area can be obtained from the set of (u,v) and d, which are the area of the object.

So, if you go through that step, only the area where the object exists can be divided into 3D, that is, (x,y,z) (3D object division is complete).

In the 3D object region division, 3D xyz data of a finally designated object and color information of the corresponding object can be simultaneously expressed through this process.

Accordingly, the 3D object region segmentation method according to an embodiment divides the object region in three dimensions using an RGB-D camera and recent deep learning technology, thereby finally generating 3D xyz data and RGB color information of the object. find and express

As described above, one embodiment uses an RGB-D camera and recent deep learning technology to segment an object region in three dimensions to find the xyz value and RGB value of the object.

Specifically, an embodiment enables to obtain xyz and RGB color information of the object by performing 3D object region segmentation of the object by deep learning and RGB image and depth data of the object to be recognized.

Therefore, in the case of robot vision monitoring such as automation through this, 3D xyz data (ie, three-dimensional information) and RGB color information of an object are found and expressed while using a camera, thereby recognizing an object in three dimensions. separate

Therefore, further industrially, there is an effect that can be utilized as a vision system for 3D object recognition. And, technically, as 3D-based object recognition becomes possible, it can be used in various ways, such as a vision system for robot vision monitoring.

5 is a view showing an RGB color image and depth data applied to the system of FIG. 2 .

As shown in FIG. 5 , the data according to this embodiment is an example of the result of collecting RGB-D data, that is, the result of expressing the RGB image and the depth data in black and white.

And, in this case, the depth data becomes data indicating depth (distance) according to brightness. In addition, the depth data includes a case of having a high resolution according to the performance of the camera.

Specifically, in one embodiment, matching information matching resolution information corresponding to a plurality of different camera (performance) specifications is preset and registered.

Then, the depth data from the RGB-D camera is extracted as a resolution corresponding to the currently used camera (performance) specification according to the matching information.

Therefore, by performing logical AND between the object 2D region division result and the extracted depth data from the RGB-D camera, depth information of the object is obtained.

6 and 7 are diagrams showing a deep learning object 2D region division process by the object 2D region division deep learning model used in the system of FIGS. 1A and 1B.

Specifically, FIG. 6 is a basic region representation through the deep learning object 2D region segmentation process, and FIG. 7 shows regions separately for each object.

6 and 7, the deep learning object 2D region division process according to an embodiment is basically expressed as in FIG. 6 by the object 2D region division deep learning model of an embodiment completed in the learning process described above. Alternatively, as shown in FIG. 7 , a region is expressed separately for each object.

8 and 9 are diagrams graphically expressing the result of 3D object region division by the system of FIGS. 1A and 1B.

Specifically, FIG. 8 is an example in which RGB image and depth data are expressed in an XYZ coordinate system and expressed graphically, and FIG. 9 is a diagram expressing a result of dividing a 3D area graphically.

8, when the system according to an embodiment divides the 3D object region as described above, through this, the RGB image and depth data are converted into 3D xyz data and RGB color information of the object, that is, the XYZ coordinate system and the represent graphically.

And, as shown in FIG. 9 , in the 3D object region division according to an embodiment, the result of dividing the 3D region by dividing only the object region and the background region, for example, is graphically expressed.

Therefore, in the 3D object region division according to an embodiment, the 3D xyz data of the finally designated object and the color information of the corresponding object can be simultaneously expressed through this process.

On the other hand, the 3D object region division system according to an embodiment has another embodiment in which, for example, the above-described central control panel of FIG. 2 is configured in the form of a PLC controller.

Specifically, in this other embodiment, the central control panel is composed of a PLC controller.

In this case, the PLC controller basically includes a power supply module, an I/O module that integrates input/output of various analog and digital information such as water and sewage status information, a converting module that converts data, and a CPU module that is a central processing unit, as in the past. include And, it includes a communication module for communicating with various control objects including the OS of the central control center.

For reference, as in the prior art, the I/O module includes an A/I module, an A/O module, a D/A module, and a D/O module, and the converting module includes an A/D module and a D/O module. A module is included.

In this state, also in the embodiment, the PLC controller further includes an LTE module that wirelessly communicates with the manager mobile terminal (this LTE module will be described in more detail later).

As described above, the PLC controller according to the embodiment has the following configuration.

a) a power module for supplying power to the PLC itself;

b) an A/D module that converts the electrical analog signal of the infrastructure and the water and sewage status analog information of the sensor unit into digital;

c) D/A module for converting digital information of digital unit into analog;

d) a D/I module for receiving an electrical digital signal of the infrastructure and digital information of the water and sewage status of the sensor unit;

e) a D/O module for outputting digital information including a driving operation control value of the central control center according to the state of water and sewage of the sensor unit to a previously connected driving target;

f) a communication module providing the operation operation control value of the central control center or the on-site control panel by communicating with a corresponding pre-registered control target including the MCU;

g) an LTE module for notifying a pre-registered manager mobile terminal when the water and sewage state value of the sensor unit is a preset state risk level; and

h) a CPU module for controlling each module; includes

In addition, if we explain further in this regard, modules with various functions are required for the existing PLC system used in various environments, and accordingly, PLC manufacturers provide various modules that satisfy the requirements of users. For example, a module having various functions, such as a digital input/output module, an analog input/output module, and a communication module, is used in a PLC system, and a system desired by a user is built through these various modules.

For example, the technology of patent document KR101778333 Y1 is an invention registered as such a technology, and specifically relates to a PLC system having a diagnostic module for diagnosing whether an output module of the PLC is defective in operation.

The above-described LTE module according to an embodiment provides a PLC that provides an LTE function from the LTE module by using these points, and furthermore, through this, monitoring and control can be easily performed by an administrator.

Accordingly, through this, the PLC controller according to an embodiment is provided with a (wireless) internet connection by itself through the LTE module, so that the dangerous state data of water and sewage detected in real time can be directly transmitted to the pre-registered manager mobile terminal. carry out

On the other hand, in this case, when the system is connected to the manager mobile terminal through the LTE module, the CPU module secures a real-time connection with the manager mobile terminal, so that an immediate notification can be made in case of a dangerous state of water and sewage.

To this end, when the CPU module notifies the manager mobile terminal, when the manager mobile terminal is connected through a wireless communication network, the system connects to an individual IP address of the corresponding wireless communication network, and when the wireless communication network is not connected, the mobile communication It connects to the terminal identification subscriber number of the data network to secure a real-time connection and alarm.

Additionally, for example, in the system according to an embodiment, an A/I module among the input modules of the PLC controller receives analog information about fine dust around a sensor unit, for example, a fine dust detection sensor. Then, the D/I module receives the sensor unit, for example, solar digital information in the vicinity.

In this case, the input module is defined as including an I/O card as an input module of the PLC.

And, when a detection signal is inputted through the input module, the output module of the PLC controller transmits a control signal to a driving target through the CPU module, for example, to a fine dust reduction device.

On the other hand, in addition to this, in this system, the LTE module of the PLC controller makes a voice alarm in relation to the data collected from the I/O card of the PLC, etc. in addition to the above.

To this end, the LTE module has its own TTS engine, and when the level of the surrounding environment information is a preset driving risk level, a voice alarm is issued.

In this case, the voice comment is made by, for example, registering in a flash voice memory provided in the LTE module itself.

So, through this, a voice alarm is made in relation to data collected from an I/O card of the PLC, etc.

In this regard, the LTE module additionally performs a noise canceling function for an external voice including the manager's voice.

Specifically, the LTE module receives the voice of an administrator related to vision monitoring, performs noise cancellation, and outputs audio.

For example, in this case, noise cancellation is performed by receiving an external audio input, and audio output is performed through an audio amplifier provided in the LTE module itself.

On the other hand, the system according to the above-described embodiment is different from the above PLC controller and has the following configuration.

That is, this system is different from the above-described PLC, and external input/output ports, that is, digital input, digital output, analog input (Analoge Input: 4-20mA input, etc.) communication ports are RS- 485 communication port, RS-232 communication port, LAN port, audio port, etc. to provide a system that can directly process the camera unit.

To this end, such a system includes a data collection unit that receives analog data and digital data, respectively, from an external measurement device, and a commercial CPU accordingly.

The data collection unit includes an ADC converter that converts a signal of an analog sensor into a digital signal, a GPIO port for receiving an electrical input or output to control an external sensor by interworking with viewer software, and a method for receiving digital data from the external measuring device An RS485 port, an RS232 port for communication with the data processing unit and communication with the viewer software, and an MCU connected to the ADC converter, GPIO port, RS485 port, and RS232 port to process data and perform control commands of the viewer software may include

Therefore, through this, the external input/output port, i.e., digital input, digital output, analog input, RS-485 communication port, RS232 communication port, LAN port, audio port, etc. can be directly processed by the system, so that different specifications can be negotiated for each NVR company. It has no effect.

In addition, data output (D/O (data out)) for controlling external measuring devices, contact output, port for controlling alarm, etc., data input (D/I (data in)), analog input (Analog Input: 4-20mA input, etc.) is pre-processed by MCU and delivered to CPU, RS-485 communication port, LAN port, audio port, etc.

By trending data directly from Mera, administrators can easily grasp data such as flow rate, pressure, water level, and water quality, which has the effect of providing a method to efficiently control external systems.

And, in this case, the above-described system directly database data input to the external input/output port in its own memory, and displays the data trend in graph form when displaying it as an image on the monitor.

To this end, the system stores the analog or digital data of the surrounding environment information processed together with the collected image as a database in SD, and the database with the image can be displayed on the screen of the central control center as a data trend in text or graph format. there is.

The content displayed on the screen is a database of various situations in the surrounding environment.

Suga is displayed together with the video, and when text is displayed on the video, it can be set as the text of the text, the unit of the text value, the location of the text on the screen, font, color, etc. And, at the time of warning, the expression text is displayed in any one of the cases of flickering in a specified color or flickering in which the text is still in a specified color and the entire screen flickers in a specified color or black and white. In addition, when displaying a database for various situations in a graph-type data trend, the text, unit, and color of the data can be displayed as set by the administrator. Alternatively, if the trend bar displayed on the image is moved at a desired time, the time value of the trend where the moved bar is located is displayed, and the data value located on the moving bar automatically disappears after confirming the data. And, at the time of warning, the expression text includes a function to be displayed in either case of flickering in a specified color, or text is still in a specified color and the entire screen flickers in a specified color or black and white.

A method of a program for setting an expression method when expressing data trends in the form of data characters and graphs on the image consists of general, data analog, data digital input, and digital output.

General settings are made in general, and the camera unit selects the type of the connected camera unit, the Address is the network address of the selected camera unit, and the Protocol is a camera such as LSIS, Modbus, Profibus, etc. You can select the one that matches the part 50, Communication can select one of RS-232, RS485, and LAN communication, and the Communication Port (Comm. Port) selects a communication port (COM1, COM2) , ... , COM10), IP address is the IP address of the camera, History Trend is Live or saved previous data is selected from History, History is selected from the date of previous data search, Period is the search date period.

You can choose.

In Analog, Enable means whether the displayed data is used or not, String is the name of the data, Measure is the data unit, X-axis is the coordinate of the X-axis to display text on the screen, Y-axis displays the text on the screen The coordinates of the Y axis to be used, Size is the size of the character, Color is the color of the character, Display Status is either Text or Trend, and the Minimum Range is the minimum value of data. , Range Max is the highest value of data, and Display Time is the time when the X coordinate of the time domain is set among trend coordinates when displaying a trend.

Enabling of digital input indicates whether or not display data is used, string is data name, X-axis is X-axis coordinate to display text on the screen, Y-axis is Y-axis coordinate to display text on screen, size is text size, color is text Color and Effect set the alarm method (character fast blinking, slow blinking, screen blinking), Display Status, Text or Trend, and Display Time, time among trend coordinates when displaying a trend. The area X-coordinate is expressed as the set time.

Enabling of digital output indicates whether the displayed data is used or not, string is the name of the data, X-axis is the X-axis coordinate to display text on the screen, Y-axis is the Y-axis coordinate to display text on the screen, size is the size of the text, and color is the text The color and effect of , set the alarm method, and the Control Status is a function that allows the administrator to arbitrarily set the program that selects on/off control of the control system.

On the other hand, additionally, object recognition applied to such a system consists of, for example, the following configuration.

Such object recognition specifies an object area using a Region Proposal Network (RPN), and stores the result of recognizing an object included in an image by using an object recognition algorithm in the specified object area. That is, the object is recognized by convolving the RPN and R-CNN algorithm that specifies the object. Then, the recognition result, that is, the integrated image of the object is displayed as a 3D stereoscopic image.

For example, in this case, the image may be an image provided by various terminals or information processing devices.

On the other hand, in object recognition according to another embodiment, the region corresponding to the object to be recognized in the image is specified using RPN, and the R-CNN algorithm learned for the object within the specified region is applied to the object characteristics stored in the object database. By comparison, it is specified as an object in the image.

Specifically, in order to specify a region corresponding to an object included in an image, an object region is extracted using RPN.

Next, in order to recognize the object in the extracted object area as one object, it is learned using R-CNN algorithm, which is a deep learning algorithm, and compared with several objects in the stored object database.

Therefore, if the object in the image is recognized as an object of the object database as a result of the comparison, the result of the name of the recognized object is stored in the object database to upgrade the learning result.

That is, in the object recognition, the object in the image and the object stored in the object database are various objects stored in the object database for faster object recognition through convolutional combination of the RPN for specifying the object area and Fast R-CNN, which is an algorithm for recognizing the specified object, into one network. It reduces the time required for many operations that occur by comparing with information. And, according to this, it is possible to further recognize an object much faster than an environment in which the system performance is low.

Therefore, in order to implement a three-dimensional image image in an environment where the performance of the conventional system is low, a lot of load is generated and object recognition for an object to be searched is separately performed in order to solve the problem that object recognition takes a lot of time. We use the FAST R-CNN algorithm performed using a neural network pre-trained by a deep learning algorithm.

Accordingly, there is an effect that a 3D image can be easily and quickly implemented in real time even in an environment with low system performance.

110: RGB-D camera 120: central control panel
130: display device 121: I/O module unit
122: DSP unit 123: control unit
124: storage

Claims (10)

In the case of robot vision monitoring, the method for performing robot vision monitoring by recognizing an object in three dimensions from vision data,
The first to construct an object 2D region division deep learning model by calculating deep learning parameters for the object region through deep learning learning for 2D object region division from only a number of different RGB images in advance before dividing the object region step;
a second step of performing object 2D region division from the RGB image from the RGB-D camera by the object 2D region division deep learning model;
a third step of obtaining depth information of an object by performing a logical AND between the object 2D region division result and depth data from the RGB-D camera; and
a fourth step of obtaining xyz and RGB information of the object by performing a 3D object region division of the object by applying perspective transformation to the depth information of the object; including,

The first step is
The object 2D region division deep learning model generates a binary mask by extracting a representative threshold for each 2D object region through R-CNN type deep learning learning in advance, a mask for 2D object region division from a number of different RGB images, It is an object 2D region segmentation deep learning model based on R-CNN,

The first step is
Step 1-1 of collecting only RGB images of an object to be recognized through an RGB-D camera;
a first 1-2 step of creating learning data by defining and displaying an area of an object in the RGB image in advance and designating a type of the object;
Steps 1-3 of dividing only an object region by applying a preset object division deep learning model to the training data;
Steps 1-4 of learning by applying the dataset obtained through the division result to a preset deep learning model for 2D object region division;
Steps 1-5 of calculating a deep learning parameter for an object region from the learning result; and
Steps 1-6 of constructing an object 2D region division deep learning model from the deep learning parameters related to the object region; includes,

The fourth step is
When the perspective transformation is performed, the depth data of RGB-D data provides the coordinates (u,v) in the image plane (U,V), the image coordinates (u,v), and the depth value ( d ). to get the actual coordinates ( x,y,z ) based on the origin (0,0,0) of the camera,
When real coordinates are obtained, the distance from the origin (0,0,0) to ( u,v ) is expressed as l , f is the focal length, and d is the depth value. Obtain (x, y, z) in the coordinate system based on the camera from , v) and d information,
Obtain the set of (x,y,z) coordinates of the real object area from the set of (u,v) and d, which are the area of the object from the corresponding information,
expressing 3D xyz data of the object and color information of the object by dividing only the region where the object exists by the set into 3D (x, y, z); A 3D object region segmentation method using RGB-D data and deep learning, characterized by
delete delete The method according to claim 1,
The third step is
a 3-1 step of presetting and registering matching information that matches resolution information corresponding to a plurality of different camera (performance) specifications;
a 3-2 step of extracting the depth data from the RGB-D camera as a resolution corresponding to the currently used camera (performance) specification according to the matching information; and
a step 3-3 of obtaining depth information of an object by performing a logical AND between the object 2D region division result and the extracted depth data from the RGB-D camera; 3D object region segmentation method using RGB-D data and deep learning, comprising: delete In the case of robot vision monitoring, the system for performing robot vision monitoring by recognizing objects in three dimensions from vision data,
RGB-D camera that collects RGB image and depth data of object to be recognized;
a central control panel for obtaining xyz and RGB information of an object by performing 3D object region division of an object only with the RGB image and depth data; and
a display device that provides a result of dividing the 3D object region of the object; contains,
The central control panel
a) Before dividing the area of the object, the deep learning parameters for the object area are calculated through deep learning for 2D object area segmentation from only a number of different RGB images in advance to construct an object 2D area segmentation deep learning model, ,
b) performing object 2D region division from the RGB image from the RGB-D camera by the object 2D region division deep learning model;
c) obtaining depth information of an object by performing a logical AND between the object 2D region division result and depth data from the RGB-D camera;
d) By applying perspective transformation to the depth information of the object to perform 3D object region segmentation of the object, xyz and RGB information of the object can be obtained,

The object 2D region segmentation deep learning model is
Mask R-CNN-type deep learning learning for 2D object region segmentation from multiple different RGB images in advance to extract representative threshold values for each 2D object region and generate a binary mask, mask R-CNN-based object 2D region segmentation It is a deep learning model,

Building the object 2D region segmentation deep learning model (a))
a-1) Collect RGB images through RGB-D camera for the object to be recognized,
a-2) Create learning data by defining and displaying the area of an object in the RGB image in advance and specifying the type of the object;
a-3) Applying a preset object division deep learning model to the training data to divide only the area of the object,
a-4) learning by applying the dataset obtained through the division result to a preset deep learning model for 2D object region division,
a-5) Calculating a deep learning parameter for an object area from the learning result,
a-6) constructing an object 2D region division deep learning model from the deep learning parameters related to the object region,

The perspective transformation is
From the depth data of RGB-D data, the coordinates (u,v) in the image plane (U,V), the image coordinates (u,v), and the depth value ( d ) are provided, and the origin (0, get real coordinates ( x,y,z ) relative to 0,0),
When real coordinates are obtained, the distance from the origin (0,0,0) to ( u,v ) is expressed as l , f is the focal length, and d is the depth value. Obtain (x, y, z) in the coordinate system based on the camera from , v) and d information,
Obtain the set of (x,y,z) coordinates of the real object area from the set of (u,v) and d, which are the area of the object from the corresponding information,
expressing 3D xyz data of the object and color information of the object by dividing only the region where the object exists by the set into 3D (x, y, z); A 3D object region segmentation system using RGB-D data and deep learning, characterized by
delete delete 7. The method of claim 6,
Acquiring the depth information of the object (c))
c-1) preset and register matching information that matches resolution information corresponding to a number of different camera (performance) specifications;
c-2) extracting the depth data from the RGB-D camera as a resolution corresponding to the currently used camera (performance) specification according to the matching information;
c-3) obtaining depth information of an object by performing a logical AND between the object 2D region segmentation result and the extracted depth data from the RGB-D camera; A 3D object region segmentation system using RGB-D data and deep learning, characterized by delete

Images