KR102453834B1 - A method for structuring the output information of multiple thermal and image cameras as input data of a deep neural network model - Google Patents

Abstract

In the method for structuring into input data of a deep neural network model according to an embodiment of the present invention, when a two-dimensional image is input by the plane classification unit, the information included in the image is classified based on the properties of the information included in the image, , generating a plurality of planar images each including classified information, and a stereoscopic forming unit stacking the plurality of planar images in a plurality of layers to generate a three-dimensional image having a three-dimensional structure.

Description

A method for structuring the output information of multiple thermal and image cameras as input data of a deep neural network model

The present invention relates to a technique for generating input data of a deep neural network model, and more particularly, input of a deep neural network model for estimating position information, posture information, and object information from output information of a plurality of thermal images and video cameras It is about a method for structuring into data.

Instead of the existing image processing, various methods for identifying objects through deep neural networks are being introduced.

Korean Patent Laid-Open Patent No. 2020-0035536 published on April 06, 2020 (Title: System and method for controlling bus-only lane violations using infrared camera and infrared lighting)

An object of the present invention is to provide a deep neural network for estimating an absolute or relative position change amount, posture information, and object information of an object using a plurality of heterogeneous cameras attached to, connected to, or built-in to a device in indoor and outdoor environments. To provide a method of deriving input data for learning

The method for structuring into input data of a deep neural network model according to a preferred embodiment of the present invention for achieving the above object is based on the properties of information included in the image when the plane classification unit receives a two-dimensional image. classifying the information included in the image, generating a plurality of planar images each including the classified information, and a stereoscopic image having a three-dimensional structure by stacking the plurality of planar images in a plurality of layers by a stereoscopic forming unit It includes the step of creating

The generating of the plurality of flat images may include: when the input image is a thermal image, classifying the pixels of the thermal image by the plane classification unit into a plurality of unit sections based on the temperature of the pixel value; and generating, by the plane classification unit, a plane image including pixels having a pixel value of a temperature included in the unit section for each of the plurality of classified unit sections, thereby generating a plurality of planar images.

에 따라 소정의 분할 수의 단위 구간으로 구분하는 것을 특징으로 하고, 상기 Q는 분할 수이고, 상기 Tstart는 단위 구간의 시작 온도이고, 상기 Tend는 단위 구간의 종료 온도이고, 상기 Tmax는 상기 온도 분포의 최대값인 것을 특징으로 한다. In the step of classifying into the plurality of unit sections, after the plane classifying unit sets the maximum value of the temperature distribution,

characterized in that it is divided into unit sections of a predetermined number of divisions according to the It is characterized in that it is the maximum value of .

The generating of the plurality of plane images includes: replacing, by the plane classification unit, a pixel value having a temperature equal to or greater than the maximum value of the temperature distribution with an effective maximum temperature; generating a plurality of flat images by constructing a flat image including a feature pixel having a pixel value of , and a padding pixel obtained by converting a pixel value of a temperature not included in a unit section to 0; Normalizing the pixel values of ? and converting them to a range of [0.0, 1.0).

In the generating of the plurality of images, when the input image includes a color image, a depth image, and an infrared image, the plane classifying unit includes a plurality of planes in which pixel values have at least one of color information, distance information, and brightness information. generating an image, and converting pixel values of all pixels of a plurality of planar images into a range of [0.0, 1.0) by normalizing the pixel values.

The generating of the plurality of plane images includes an R channel plane image having only pixel values of an R (Red, red) channel, a G channel plane image having only pixel values of a G (Green, green) channel, and a B (Blue, blue) generating a B-channel plane image having only pixel values of the channel or generating a grayscale plane image composed of grayscale values, and an upper depth plane image and a lower generating a depth plane image or generating a quantized depth plane image by quantizing pixel values; and generating a quantized quantized brightness plane image.

The method includes the steps of: a grid forming unit arranging a plurality of stereoscopic images in a grid to form a grid image; The method may further include matching heights of a plurality of stereoscopic images included in the grid image by erasing the plane image of , or adding at least one padding image to the at least one stereoscopic image.

In the method, when a two-dimensional image is input, the plane classifying unit generates a plurality of planar images, the three-dimensional forming unit generates the three-dimensional image from the plurality of planar images, and the grid forming unit generates the plurality of three-dimensional images. Forming a grid image by arranging them in a grid, and collecting, by a learning unit, at least one of position information, posture information, and object information of an object as label data in response to input data that is the stereoscopic image or the grid image; The method further includes: a learning unit labeling the label data on the input data.

According to the present invention, it is possible to process the output information of a plurality of thermal imagers or video cameras, convert it into an input data structure, and fuse it to estimate the absolute position or relative position change amount, posture information, etc. of a device or object in an indoor or outdoor environment. It enables efficient training of neural network models. When a deep neural network is trained using the training data generated by the present invention, it is possible to conveniently and quickly estimate a position and posture without calculating a complex image processing algorithm.

1 is a conceptual diagram for explaining a method of deriving information of an arithmetic device according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining the configuration of an apparatus for structuring output information of a plurality of thermal imaging and imaging cameras into input data of a deep neural network model according to an embodiment of the present invention.
FIG. 3 is a view for explaining the detailed configuration of a control unit of an apparatus for structuring output information of a plurality of thermal images and video cameras into input data of a deep neural network model according to an embodiment of the present invention.
4 is a diagram for explaining an example of a deep neural network model for estimating position, posture, and object information based on output information of a plurality of thermal images and video cameras according to an embodiment of the present invention.
5 is a flowchart illustrating a method for structuring output information of a plurality of thermal imaging and imaging cameras into input data of a deep neural network model according to an embodiment of the present invention.
6 is a view for explaining a method of generating a stereoscopic image from output information of a plurality of thermal imaging cameras according to the first embodiment of the present invention.
7 is a diagram for explaining a method of generating a stereoscopic image from output information of a plurality of image cameras according to a second embodiment of the present invention.
8 is a diagram for explaining a method of generating a grid image from output information of a plurality of thermal images and image cameras according to an embodiment of the present invention.
9 is a flowchart illustrating a method of generating a stereoscopic image according to the first embodiment of the present invention.
10 is a flowchart illustrating a method of generating a stereoscopic image according to a second embodiment of the present invention.
11 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical ideas of the present invention, so various equivalents that can replace them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, a method of deriving information based on input data generated based on images of a plurality of thermal imaging and imaging cameras through a deep neural network according to an embodiment of the present invention will be described. . 1 is a conceptual diagram for explaining a method of deriving information of an arithmetic device according to an embodiment of the present invention. FIG. 2 is a diagram for explaining the configuration of an apparatus for structuring output information of a plurality of thermal imaging and imaging cameras into input data of a deep neural network model according to an embodiment of the present invention. FIG. 3 is a view for explaining the detailed configuration of a control unit of an apparatus for structuring output information of a plurality of thermal images and video cameras into input data of a deep neural network model according to an embodiment of the present invention. 4 is a diagram for explaining an example of a deep neural network model for estimating position, posture, and object information based on output information of a plurality of thermal images and video cameras according to an embodiment of the present invention.

Referring to FIG. 1 , basically, the computing device 10 is for learning a deep neural network according to an embodiment of the present invention. It can be installed in the robot (R) or the computing unit (C) to learn deep neural networks. If it is installed in the computing unit (C) to learn the deep neural network, the deep neural network can be ported to the robot (R). Such a deep neural network can be trained to identify position information of the robot R or objects (obj1, obj2, ob3), estimate posture information, or identify objects (obj1, obj2, ob3). To this end, the present invention provides input data generated based on a color image, a depth image, an infrared image, and a thermal image, and location information for the input data; Learning data labeled with posture information and entity information is used.

Referring to FIG. 2 , an apparatus (10: hereinafter abbreviated as 'structuring apparatus') for structuring output information of a plurality of thermal imaging and imaging cameras into input data of a deep neural network model according to an embodiment of the present invention is a camera It includes a unit 11 , an input unit 12 , a display unit 13 , a storage unit 14 , a communication unit 15 , a sensor unit 16 , a location information unit 17 , and a control unit 18 .

The camera unit 11 is for capturing an image. The camera unit 11 includes an image camera 110 and a thermal image camera 120 . The video camera 110 captures a subject and outputs at least one of a color image, a depth image, and an infrared image. The thermal imaging camera 120 outputs a thermal image by photographing a subject.

The input unit 12 may receive a user's manipulation for controlling the arithmetic unit 10 , generate an input signal, and transmit it to the control unit 18 . The input unit 12 includes various kinds of buttons and keys for controlling the arithmetic unit 10 . As for the input unit 12, when the display unit 13 is formed of a touch screen, the functions of various keys may be performed on the display unit 13, and when all functions can be performed only with the touch screen, the input unit 12 may be omitted. may be

The display unit 13 is for screen display, and may visually provide a menu of the computing device 10 , input data, function setting information, and other various information to the user. The display unit 13 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 13 may be implemented as a touch screen. In this case, the display unit 13 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 18 . In particular, when the display unit 13 is formed of a touch screen, some or all of the functions of the input unit 12 may be performed through the display unit 13 .

The storage unit 14 serves to store programs and data necessary for the operation of the arithmetic unit 10 . The storage unit 14 may store an image captured by the camera unit 11 and inertial data sensed by the sensor unit 12 for a predetermined period of time. Various types of data stored in the storage unit 14 may be deleted, changed, or added according to a user's operation.

The communication unit 15 is for communicating with another object 20 such as, for example, a robot. The communication unit 15 may receive from the object 20 the location information of the object received by the object 20 through the GPS receiver, inertial sensor information of the object 20 measured through the inertial sensor of the object 20, and the like. have. The communication unit 15 includes an RF transmitter for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver for low-noise amplifying and down-converting a received signal. In addition, the communication unit 15 includes a modem that modulates a transmitted signal and demodulates a received signal.

The sensor unit 16 is for measuring inertia. The sensor unit 16 includes a speed sensor, an acceleration sensor, an angular velocity sensor, a gyro sensor, an inertial measurement unit (IMU), a Doppler Velocity Log (DVL), and an attitude and heading reference. System : AHRS). Such a sensor may be implemented as micro electro-mechanical systems (MEMS).

The location information unit 17 is for receiving a Global Positioning System (GPS) signal. For example, the location information unit 17 continuously receives a GPS signal from a GPS satellite or the like, and derives location information from the received GPS signal. The derived location information is transmitted to the control unit 18 . Such location information may be coordinates such as latitude, longitude, and altitude.

The controller 18 may control the overall operation of the arithmetic unit 10 and the signal flow between the internal blocks 11 to 18 of the arithmetic unit 10 , and may perform a data processing function of processing data. In addition, the control unit 18 basically serves to control various functions of the arithmetic unit 10 . The control unit 18 may be exemplified by a central processing unit (CPU), a digital signal processor (DSP), or the like.

Next, referring to FIG. 3 , the controller 18 includes a configuration unit 200 , a learning unit 300 , a deep neural network (DNN) 400 , and an estimator 500 .

The configuration unit 200 is for generating input data input to the deep neural network 400 from the image captured by the camera unit 11 . The configuration unit 200 receives a two-dimensional image captured by the camera unit 11, configures the input two-dimensional image into one or more three-dimensional stereoscopic images (SI: Stereo Image), and comprises a plurality of A grid image (GI) is generated by arranging a stereoscopic image (SI) in a grid. Such a stereo image (SI) or a grid image (GI) may be used as input data input to the deep neural network 400 .

To this end, the constituent unit 200 includes a planar forming unit 210 , a three-dimensional forming unit 220 , and a grid forming unit 230 . When a two-dimensional image is input, the plane forming unit 210 classifies information included in the image based on the properties of the information included in the input image, and a plurality of plane images FI each including the classified information. flat image). The three-dimensional forming unit 220 generates a three-dimensional image SI having a three-dimensional structure by stacking the plurality of planar images FI generated by the planar forming unit 210 in a plurality of layers. The grid forming unit 230 forms a grid image GI by arranging a plurality of three-dimensional shapes in a grid. The operation of the component 200 including the planar forming unit 210 , the three-dimensional forming unit 220 , and the grid forming unit 230 will be described in more detail below.

The learning unit 300 generates learning data for the deep neural network 400 by performing labeling to give a label to the input data, and uses this to learn the deep neural network 400 .

The deep neural network 400 is to provide a probability for estimating or classifying the position, posture, and object information of an object included in an image. The deep neural network 400 may be a convolutional neural network, such as a Convolution Neural Network (CNN), or a recurrent neural network, such as a Recurrent Neural Network (RNN) or Long Short-Term Memory (LTSM). However, the deep neural network 400 is not limited thereto, and all kinds of artificial neural networks in which the hidden layer is composed of a plurality of layers may be the deep neural network 400 according to an embodiment of the present invention.

According to an embodiment of the present invention, when the deep neural network 400 is a convolutional neural network, the deep neural network 400 is an input layer (IL), at least one pair of alternatingly repeated convolution layers (convolution layer: CL) ), a pooling layer (PL), and at least one fully-connected layer (FL) and an output layer (OL). As shown in FIG. 4 , the deep neural network 400 according to an embodiment of the present invention sequentially includes an input layer (IL), a convolution layer (CL), a pooling layer (PL), a fully connected layer (FL), and and an output layer OL.

The convolution layer CL and the pooling layer PL include at least one feature map (FM). The feature map FM is derived as a result of receiving a value applied with a weight and a threshold to the operation result of the previous layer, and performing an operation on the input value. These weights are applied through a filter or kernel W, which is a weight matrix of a predetermined size. In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer CL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When input data (a matrix or vector column of a predetermined size) is input to the input layer IL, the convolution layer CL performs convolution using the first filter W1 on the input data of the input layer IL. At least one first feature map FM1 is derived by performing an operation using an operation and an activation function. Next, the pooling layer PL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature map FM1 of the convolutional layer CL to obtain at least one A second feature map FM2 is derived.

The final connection layer FL includes a plurality of operation nodes f1 to fx. The plurality of operation nodes f1 to fx of the final connection layer FL calculates a plurality of operation values through an operation using an activation function with respect to at least one second feature map FM2 of the pooling layer PL.

The output layer OL includes a plurality of output nodes g1 to gy. Each of the plurality of operation nodes f1 to fx of the final connection layer FL is connected to the output nodes g1 to gy of the output layer OL through a channel having a weight (W). In other words, a weight is applied to a plurality of operation values of the plurality of operation nodes f1 to fx and input to each of the plurality of output nodes g1 to gy. Accordingly, the plurality of output nodes g1 to gy of the output layer OL calculates an output value through an activation function operation for a plurality of calculated values to which the weight of the final connection layer FL is applied.

Activation functions used in the above-described convolutional layer (CL), final connection layer (FL) and output layer (OL) are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. may be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the final connection layer FL, and the output layer OL.

In summary, as described above, the deep neural network 400 includes a plurality of layers. In addition, the plurality of layers of the deep neural network 400 includes a plurality of operations. The calculation result of each of the plurality of layers is transmitted to the next layer by applying parameters, ie, weights, thresholds, and the like. Accordingly, the deep neural network 400 may calculate an output value by performing a plurality of operations to which the weights of a plurality of layers are applied to the input data, and may output the calculated output value.

The estimator 500 is for estimating or classifying the position, posture, and object information of the object included in the image by using the deep neural network 400 . The output value of the deep neural network 400 may be a probability. The estimator 500 may estimate or classify the position, posture, and object information of the object included in the image based on the probability output from the deep neural network 400 .

Next, a method for structuring output information of a plurality of thermal images and video cameras into input data of a deep neural network model according to an embodiment of the present invention will be described. 5 is a flowchart illustrating a method for structuring output information of a plurality of thermal imaging and imaging cameras into input data of a deep neural network model according to an embodiment of the present invention. 6 is a view for explaining a method of generating a stereoscopic image from output information of a plurality of thermal imaging cameras according to the first embodiment of the present invention. 7 is a diagram for explaining a method of generating a stereoscopic image from output information of a plurality of image cameras according to a second embodiment of the present invention. 8 is a diagram for explaining a method of generating a grid image from output information of a plurality of thermal images and image cameras according to an embodiment of the present invention.

Referring to FIG. 5 , the configuration unit 200 receives a plurality of images IMG photographed through the plurality of image cameras 110 and the plurality of thermal imaging cameras 120 of the camera unit 11 in step S110 . can As described above, the camera unit 11 includes a plurality of video cameras 110 and a plurality of thermal imaging cameras 120 . Accordingly, the configuration unit 200 includes a plurality of images including a color image (CI), a depth image (DI), and an infrared image (II), and a plurality of thermal images (TI). Thermal Image) can be input.

In this way, when a plurality of images (IMG: CI, DI, II, TI) are input, as shown in FIGS. 6 and 7 , the plane classification unit 210 of the configuration unit 200 generates a plurality of images in step S120 . Information included in each of a plurality of images (IMG: CI, DI, II, TI) is classified based on the properties of the information included in each of the images (IMG: CI, DI, II, TI), and each classified information Generates a plurality of flat images (FI) including A plurality of stereo images (SI: Stereo Images) having a three-dimensional structure are generated by stacking them. In the embodiment of the present invention, the planar image F1 is an image in which a plurality of pixels are arranged in two dimensions, and each pixel value of the plurality of pixels contains predetermined information such as temperature, color, depth (distance), brightness, etc. have In addition, in the embodiment of the present invention, a stereo image (SI) refers to an image in which a plurality of flat images F1 are stacked and all pixels included in the plurality of flat images F1 constitute a three-dimensional shape. do.

The above-described step S120 is performed in the first embodiment assuming that the input image is a thermal image (TI) as shown in FIG. 6 , and as shown in FIG. 7 , in which the input image is a color image (CI). , a second embodiment that assumes a case including a depth image DI and an infrared image II is included. The first and second embodiments of step S120 will be described in more detail below.

When a plurality of stereoscopic images SI are generated according to the first and second embodiments, the grid forming unit 230 arranges the plurality of stereoscopic images SI in a grid in step S130 to form a grid image (GI). to form

For example, it is assumed that the camera unit 110 includes a first image camera 111 and a second image camera 112 , and a first thermal imager 121 and a second thermal imager 122 . In addition, the first imaging camera 111 and the second imaging camera 112 and the first thermal imaging camera 121 and the second thermal imaging camera 122 photographing the subject at the same point in time photograph the subject at the same time Thus, the first image camera 111 outputs a first image including a color image CI, a depth image DI, and an infrared image II, and the second image camera 112 outputs a color image CI, A second image including a depth image DI and an infrared image II is output, the first thermal imaging camera 121 outputs a first thermal image TI1, and the second thermal imaging camera 121 is It is assumed that the second thermal image TI2 is output. Accordingly, as shown in FIG. 8 , the configuration unit 200 performs the first stereoscopic image SI1 from the first image, the second stereoscopic image SI2 from the second image, and the second stereoscopic image SI1 from the first thermal image TI1. A fourth stereoscopic image SI4 may be generated from the third stereoscopic image SI3 and the second thermal image TI2 . Then, the grid forming unit 230 arranges the first stereoscopic image SI1 , the second stereoscopic image SI2 , the third stereoscopic image SI3 , and the fourth stereoscopic image SI4 in a grid as shown in the figure. A lattice image GI is formed.

Meanwhile, when at least one of the plurality of stereoscopic images SI1 , SI2 , SI3 and SI4 included in the grid image GI has a different height in step S130 , the grid forming unit 230 performs at least one stereoscopic image SI. A plurality of stereoscopic images SI1, SI2, included in a grid image by deleting at least one planar image FI from The heights of SI3 and SI4) can be matched. Here, the padding image PI refers to a flat image FI in which pixel values of all pixels are constant.

Then, the above-described first and second embodiments will be described in more detail. First, a method for generating a stereoscopic image SI according to a first embodiment of the present invention will be described. 9 is a flowchart illustrating a method of generating a stereoscopic image (SI) according to the first embodiment of the present invention.

6 and 9 , as described above, in the first embodiment, it is assumed that an image input to the configuration unit 200 is a thermal image TI. When the thermal image TI is input, the plane classifying unit 210 classifies the pixel of the thermal image into a plurality of unit sections based on the temperature indicated by the pixel value of the pixel of the thermal image in step S210 .

At this time, the plane classification unit 210 is the temperature distribution {T | 0 ≤ T < Tmax}, and derive the maximum value Tmax of the temperature distribution according to the set temperature distribution. Then, the plane classifying unit 210 divides the pixels of the thermal image into unit sections of a predetermined number of divisions based on the temperature indicated by the pixel value according to Equation 1 below.

In Equation 1, Q is the number of divisions and is preset. In addition, Tstart is the start temperature of the unit section, and Tend represents the end temperature of the unit section. Tmax means the maximum value of the temperature distribution. The temperature distribution may be basically determined based on the pixel values of all pixels, that is, the mean and standard deviation of the temperature. Alternatively, when the temperature distribution must include a specific temperature range or temperature according to the purpose of training the deep neural network 400 , the temperature distribution may be set based on the corresponding temperature range or temperature. For example, if the deep neural network 400 is to detect an overheated battery or to detect the stability of a hydrogen storage tank, the temperature distribution should be relatively high or have a relatively low temperature distribution than the general room temperature range. . Therefore, it is preferable to set the temperature distribution according to the purpose of learning the deep neural network 400 .

Next, the plane classifying unit 210 replaces a pixel value indicating a temperature equal to or higher than the maximum value Tmax of the temperature distribution with a pixel value indicating the effective maximum temperature Tmax-1 in step S220. previously set temperature distribution {T | As 0 ≤ T < Tmax}, since the end range of the temperature distribution is a closed section, the effective maximum temperature (Tmax-1) subtracts 1 from the maximum value (Tmax) of the temperature distribution.

Then, the plane classification unit 210 constructs a plane image composed of pixels representing the temperature included in the unit section for each of the plurality of (Q) unit sections classified in step S230 by constructing a planar image of a plurality of (Q) planes. Create an image (FI). In this case, the plane classifying unit 210 replaces the pixel value of the pixel representing the temperature at which the pixel value is not included in the unit section with 0. In summary, in step S230 , the plane classifying unit 210 generates a plurality of (Q) plane images by constructing the plane image FI for each of the classified plurality (Q) unit sections. In particular, each plane image FI includes a feature pixel representing a temperature in which a pixel value is included in a unit section and a padding pixel obtained by converting a pixel value of a pixel representing a temperature in which a pixel value is not included in the unit section to 0. For example, any one plane image FI shown in FIG. 6 includes a feature pixel FX whose pixel value indicates a temperature included in a unit section of the plane image, and the remaining areas other than the feature pixel FX All are padding pixels PX, and all pixel values of the padding pixels PX are replaced with 0.

를 차감한다. 이와 같이, 특징 픽셀(FX)은 각 단위 구간의 시작 온도를 차감하고, 나머지 패딩 픽셀(PX)의 픽셀값은 0이기 때문에 모든 평면 영상(FI)의 모든 픽셀의 픽셀값은

범위를 가진다. 그러면, 평면분류부(210)는 평면 영상(F1)의 모든 픽셀을

로 나눈다. 그러면, 모든(Q개) 평면 영상(FI)의 모든 픽셀의 픽셀값은 [0.0, 1.0)의 범위로 변환된다. 이러한 S240 단계의 정규화 과정은 선택적인 것으로 생략될 수 있다. Next, the plane classifying unit 210 normalizes the pixel values of the pixels of the plurality of (Q) plane images FI in step S240 and converts them to a range of [0.0, 1.0). The normalization will be described in more detail as follows. The plane classifying unit 210 determines the starting temperature of the unit section of the corresponding plane image from the pixel values of all the characteristic pixels FX of each of the plurality of (Q) plane images FI.

deduct the As such, since the feature pixel FX subtracts the start temperature of each unit section, and the pixel value of the remaining padding pixels PX is 0, the pixel values of all pixels of all flat images FI are

have a range Then, the plane classification unit 210 selects all pixels of the plane image F1.

Divide by Then, the pixel values of all pixels of all (Q) plane images FI are converted into the range of [0.0, 1.0). The normalization process of step S240 is optional and may be omitted.

Next, the three-dimensional forming unit 220 generates a three-dimensional image SI having a three-dimensional structure by stacking the plurality of (Q) planar images FI previously generated in step S250 in a plurality of layers, that is, in the depth direction. . Subsequently, the stereoscopic forming unit 220 may optionally add at least one padding image PI to the stereoscopic image SI in operation S260 . Here, the padding image PI means a flat image FI in which pixel values of all pixels are constant C (here, C is a real number). The padding image PI is used to provide a reference value for the depth direction.

Next, a method for generating a stereoscopic image SI according to a second embodiment of the present invention will be described. 10 is a flowchart illustrating a method of generating a stereoscopic image (SI) according to a second embodiment of the present invention.

Referring to FIGS. 7 and 10 , as described above, in the second embodiment, an image input to the configuration unit 200 includes a plurality of images including a color image CI, a depth image DI, and an infrared image II. Assume that it is an image. When these images (CI, DI, II) are input, the plane classification unit 210 determines that pixel values have at least one of color information, distance information, and brightness information based on color information, distance information, and brightness information in step S310 . Generate a plurality of flat images. In more detail with respect to this step S310, as follows.

, 8비트의 G(Green, 녹색) 채널의 픽셀값만을 가지는 G 채널 평면 영상

및 8비트의 B(Blue, 청색) 채널의 픽셀값만을 가지는 B 채널 평면 영상

을 생성할 수 있다. 또한, 평면분류부(210)는 3개 채널의 컬러 영상의 정보를 축약하기 위해서 컬러 영상(CI)의 픽셀값을 3채널의 RGB 값에서 그레이스케일 값으로 변환하여 픽셀값이 그레이스케일의 값으로 이루어진 그레이스케일 평면 영상

을 생성할 수 있다. First, in the color image CI, each pixel is expressed as a combination of 8-bit R (Red, Red), G (Green, Green), and B (Blue, Blue) values. Accordingly, the plane classification unit 210 is an R channel plane image having only 8-bit R (Red, red) channel pixel values from the color image CI.

, G-channel plane image with only 8-bit G(Green) channel pixel values

and a B channel plane image having only 8-bit B (Blue, Blue) channel pixel values.

can create In addition, the plane classification unit 210 converts the pixel value of the color image CI from the RGB value of the three channels to the grayscale value in order to abbreviate the information of the color image of the three channels, and the pixel value is converted to the value of the grayscale. A grayscale plane image made up of

can create

및 8비트로 거리를 나타내는 하위 깊이 평면 영상

을 생성할 수 있다. 또는, 평면분류부(210)는 깊이 영상(DI)에 대해 정보를 축약하기 위해서 깊이 영상(DI)의 픽셀값을 8비트로 양자화(Quantization) 하여 양자화 깊이 평면 영상

을 생성할 수 있다. In addition, the pixel value of the depth image DI basically represents a distance at which each pixel has a size between 8 bits (256 range) and 16 bits (65536 range). Therefore, the plane classifying unit 210 determines the upper depth plane image indicating the distance in 16 bits (65536 range) from the depth image DI according to the size of the pixel value indicating the distance.

and a sub-depth plane image representing the distance in 8 bits.

can create Alternatively, the plane classifying unit 210 quantizes the pixel value of the depth image DI to 8 bits in order to abbreviate information about the depth image DI, and then quantizes the quantized depth plane image.

can create

및 16비트(65536 범위)로 밝기를 표현하는 하위 평면 영상

을 생성할 수 있다. 또는, 평면분류부(210)는 적외선 영상(II)에 대해 정보를 축약하기 위해서 적외선 영상(II)의 픽셀값을 8비트로 양자화(Quantization) 하여 픽셀값을 양자화한 양자화 밝기 평면 영상

을 생성할 수 있다. The pixel value of the infrared image (II) basically expresses brightness having a size between 8 bits (256 range) and 16 bits (65536 range). Accordingly, the plane classifying unit 210 expresses the brightness in 16 bits (range 65536) according to the size of the pixel value representing the brightness from the infrared image II.

and a sub-plane image representing brightness in 16 bits (range 65536)

can create Alternatively, the plane classifying unit 210 quantizes the pixel values of the infrared image II into 8 bits in order to abbreviate information about the infrared image II and quantizes the pixel values.

can create

As described above, in the second embodiment of the present invention, the plane classification unit 210 classifies the pixels of each image output from the image camera 110 by channel or byte order, and generates a plane image FI according to the classification. do.

Next, the plane classifying unit 210 normalizes the pixel values of the pixels of the plurality of plane images FI in step S320 and converts them to a range of [0.0, 1.0). At this time, the plane classifying unit 210 obtains the maximum pixel value of the pixel of each plane, and divides the pixel value of all the pixels by the maximum pixel value. Accordingly, the values of all pixels are converted to the range of [0.0, 1.0). The normalization process of step S320 is optional and may be omitted.

Next, the three-dimensional forming unit 220 generates the three-dimensional image SI having a three-dimensional structure by stacking the plurality of previously generated planar images FI in a plurality of layers, that is, in the depth direction in step S330 . As described above, the plane classification unit 210 includes three or one planar image for the color image CI, two or one planar image for the depth image DI, and the infrared image II. Two or one planar images were generated for each image. Accordingly, when the three-dimensional forming unit 220 generates the three-dimensional image SI, the depth varies according to the number of the two-dimensional image FI generated by the plane classifying unit 210 . The stereoscopic image (SI) of the deepest three-dimensional structure is composed of seven layers having a depth as shown in Equation 2 below.

In addition, the 3D image SI of the shallowest 3D structure is composed of three layers in depth and is expressed by Equation 3 below.

The order of each layer in Equations 2 and 3 may be mutually changed.

과, 상수 1인 평면 영상

이 부가될 수 있다. 각 층의 순서는 상호간에 바뀔 수 있다. Next, the stereoscopic forming unit 220 may optionally add at least one padding image PI to the stereoscopic image SI in operation S340 . Here, the padding image PI means a flat image FI in which pixel values of all pixels are constant C (here, C is a real number). The padding image PI is used to provide a reference value for the depth direction. For example, as shown in FIG. 7 , a flat image with a constant 0

and a flat image with constant 1

This can be added. The order of each floor can be changed with each other.

In summary, a plurality of stereoscopic images from a plurality of images (IMG: CI, DI, II, TI) captured by the thermal imaging camera 120 or the imaging camera 110 according to the first and second embodiments of the present invention (SI) can be created. And as described above with reference to FIG. 5 , when a plurality of stereoscopic images SI are generated according to the first and second embodiments, the grid forming unit 230 arranges the plurality of stereoscopic images SI in a grid to create a grid. An image GI may be formed.

The above-described stereoscopic image (SI) or grid image (GI) is input data for the deep neural network 400 , and may be used as learning data for the deep neural network 400 by assigning a label to the input data. A method of generating such training data will be described. 11 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.

Referring to FIG. 11 , the configuration unit 200 includes a plurality of images (IMG: CI, DI, II and TI), a plurality of stereoscopic images SI may be generated, and the plurality of stereoscopic images SI may be arranged in a grid to form a grid image GI. The plurality of stereoscopic images (SI) or lattice images (GI) are used as input data to the deep neural network 400 .

Then, the learning unit 300 collects label data corresponding to the time t in step S420 . Such label data may vary depending on the purpose for training the deep neural network 400 . The label data may include position information including an absolute position or relative position change amount of the object 20 , posture information of the object 20 , and identification information of the object 20 .

Then, the learning unit 300 generates learning data by setting the previously collected label data as a label for the input data in step S430. That is, the learning unit 300 labels input data generated from a plurality of images (IMG: CI, DI, II, TI) captured at time t with label data collected at time t.

According to an embodiment, the learning unit 300 prepares learning data for estimating the positions of the robot R and the objects obj1, obj2, and obj3 in which the deep neural network 400 is equipped with the computing device 10. . To this end, when the configuration unit 200 generates input data, the learning unit 300 generates the inertia measured by the sensor unit 16 in response to the timing when the configuration unit 200 generates the input data, that is, the time t. Extracted from the GPS signal received by the information and location information unit 17, the inertial information and GPS signal of the object obj1 received from the robot object obj1, and the depth image captured by the camera unit 11 Absolute position expressed by three axes (x, y, z) on the Cartesian coordinate system of the robot (R) and the objects (obj1, obj2, obj3) based on the depth of the feature point (FP) of the object (obj1, obj2, obj3) and position information including a relative position change amount in which the position is relatively changed from the previous position is acquired as label data. Then, the learning unit 300 labels the position information obtained with respect to the input data at time t.

According to another embodiment, the learning unit 300 may provide training data for the deep neural network 400 to estimate the postures of the robot R and the objects obj1, obj2, and obj3 in indoor and outdoor spaces. To this end, when the configuration unit 200 generates input data, the learning unit 300 generates the inertia measured by the sensor unit 16 in response to the timing when the configuration unit 200 generates the input data, that is, the time t. Information (eg, yaw, roll, pitch), angular velocity and angular acceleration of the motor of each joint of the robot R, inertia information of the object obj1 received from the robot object obj1, the angle of the robot R Based on the depth of the feature point (FP) of the object (obj1, obj2, obj3) extracted from the depth image taken by the camera unit 11, the angular velocity, angular acceleration, etc. of the motor of the joint motion capture device, AHRS ( Through the attitude/heading reference system) device, the attitude information expressed in 6 degrees of freedom of the robot (R) or the object (obj1) or the attitude information indicating the depth of the feature point (FP) of the objects (obj2, obj3) can be acquired as label data. can Next, the learning unit 300 labels the posture information obtained with respect to the input data at time t.

According to another embodiment, the learning unit 300 includes the area occupied by the objects (objects 1, 2, 3) on the 2D image or 3D projection space where the deep neural network 400 is photographed through the camera unit 11 and It can be trained to identify the types of objects (object 1, 2, 3) (Robot 5911TARGA4S, tumbler, Human). Accordingly, when the construction unit 200 generates input data, the learning unit 320 automatically uses an image processing algorithm for a plurality of plane images FI of the input data to automatically classify the objects obj1, obj2, and obj3. (Robot 5911TARGA4S, tumbler, Human) and area box (B) can be detected. As an alternative to this, the classes (Robot 5911TARGA4S, tumbler, Human) and the areabox (Robot 5911TARGA4S, tumbler, Human) and the domain box ( B) is designated, and the learning unit 300 may receive it. Accordingly, the learning unit 300 acquires object information obtained for each of the one or more objects obj1 , obj2 , and obj3 included in the plurality of plane images FI of the input data, that is, the class and region of the object. Label the Bounding Box.

When the training data is prepared as described above, the learning unit 300 may train the deep neural network 400 in step S440 . For example, the learning unit 300 inputs the input data to the deep neural network 400, and when the deep neural network 400 calculates an output value through a plurality of operations of a plurality of layers to which each weight is applied, the output value and the label data The deep neural network 400 can be trained through optimization of modifying the parameters of the deep neural network 400 so that the difference between .

As described above, when the learning is completed, the configuration unit 200 of the computing device 10 is photographed through the thermal imaging camera 120 or the video camera 110 according to the first and second embodiments of the present invention. A plurality of stereoscopic images SI may be generated from a plurality of images IMG: CI, DI, II, and TI, and a grid image GI may be formed by arranging the plurality of stereoscopic images SI in a grid.

Then, the configuration unit 200 may input a stereoscopic image (SI) or a grid (GI) as input data to the deep neural network 400 . Then, the deep neural network 400 will output the probability of position information, posture information, or object information as an output value through a plurality of operations to which a plurality of layer weights are applied according to the learned parameters. Then, the estimator 500 estimates the absolute position or the relative position change amount, the posture of the robot R or the object (obj1, obj2, obj3) through the probability that is the output value, or identifies the object (obj1, obj2, obj3). can

Meanwhile, the above-described method according to an embodiment of the present invention may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of the program instruction may include not only machine language such as generated by a compiler, but also a high-level language that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: arithmetic unit 11: camera unit
12: input unit 13: display unit
14: storage unit 15: communication unit
16: sensor unit 17: location information unit
18: control unit 110: video camera
120: thermal imaging camera 200: components
210: plane classification unit 220: three-dimensional forming unit
230: grid forming unit 300: learning unit
400: deep neural network 500: estimator

Claims (8)

In a method for structuring as input data of a deep neural network model,
classifying the information included in the image based on the properties of the information included in the image when the plane classifier receives a two-dimensional image, and generating a plurality of plane images each including the classified information; and
generating a three-dimensional image having a three-dimensional structure by a three-dimensional forming unit stacking the plurality of planar images in a plurality of layers;
includes,
The step of generating the plurality of plane images
When the input image is a thermal image,
classifying, by the plane classification unit, the pixels of the thermal image into a plurality of unit sections based on the temperature of the pixel value; and
generating, by the plane classification unit, a plane image including pixels having a pixel value of a temperature included in the unit section for each of the plurality of classified unit sections, thereby generating a plurality of planar images;
characterized in that it comprises
A method for structuring as input data of a deep neural network model. delete According to claim 1,
The step of classifying into the plurality of unit sections is
The plane classification unit
After setting the maximum value of the temperature distribution,
formula

It is characterized in that it is divided into unit sections of a predetermined number of divisions according to
wherein Q is the number of divisions,
The Tstart is the starting temperature of the unit section,
The Tend is the end temperature of the unit section,
The Tmax is the maximum value of the temperature distribution
characterized by
A method for structuring as input data of a deep neural network model. 4. The method of claim 3,
The step of generating the plurality of plane images
The plane classification unit
replacing a pixel value having a temperature equal to or greater than the maximum value of the temperature distribution with an effective maximum temperature;
By constructing a flat image including a feature pixel having a pixel value of a temperature included in the unit section for each of the classified plurality of unit sections and a padding pixel obtained by converting a pixel value of a temperature not included in the unit section into 0, a plurality of planes generating an image;
Normalizing pixel values of all pixels of a plurality of plane images and converting them into a range of [0.0, 1.0);
characterized in that it comprises
A method for structuring as input data of a deep neural network model. According to claim 1,
The step of generating the plurality of images
generating, by the plane classification unit, a plurality of plane images in which pixel values have at least one of color information, distance information, and brightness information when the input image includes a color image, a depth image, and an infrared image; and
Normalizing pixel values of all pixels of a plurality of plane images and converting them into a range of [0.0, 1.0);
characterized in that it comprises
A method for structuring as input data of a deep neural network model. 6. The method of claim 5,
The step of generating a plurality of plane images is
From a color image, an R channel plane image having only R(Red, red) channel pixel values, a G channel plane image having only G(Green, green) channel pixel values, and a B (Blue, blue) channel pixel value only. generating a channel plane image or a grayscale plane image composed of grayscale values;
generating an upper depth plane image and a lower depth plane image according to the size of a pixel value indicating a distance from the depth image, or generating a quantized depth plane image by quantizing pixel values; and
generating an upper luminance plane image and a lower luminance plane image according to the size of a pixel value indicating brightness from the infrared image, or generating a quantized luminance plane image obtained by quantizing pixel values;
characterized in that it comprises
A method for structuring as input data of a deep neural network model. According to claim 1,
forming a grid image by arranging a plurality of stereoscopic images in a grid by a grid forming unit; and
When at least one of the plurality of stereoscopic images included in the grid image has a different height, at least one planar image is deleted from the at least one stereoscopic image or by adding at least one padding image to the at least one stereoscopic image matching the heights of a plurality of stereoscopic images included in the image;
characterized in that it further comprises
A method for structuring as input data of a deep neural network model. 8. The method of claim 7,
When a two-dimensional image is input, the plane classification unit generates a plurality of planar images, the three-dimensional forming unit generates the three-dimensional image from the plurality of planar images, and the grid forming unit arranges the plurality of three-dimensional images in a grid. to form a grid image;
collecting, by a learning unit, at least one of position information, posture information, and object information of an object as label data in response to input data that is the stereoscopic image or the grid image; and
labeling, by the learning unit, the label data to the input data;
characterized in that it further comprises
A method for structuring as input data of a deep neural network model.

Images