KR102108953B1 - Robust camera and lidar sensor fusion method and system - Google Patents

Abstract

Disclosed is a deep learning based camera, a rider sensor fusion recognition method and system that is robust to sensor quality degradation. The cognitive method performed by the cognitive system according to an embodiment may include obtaining different feature maps by inputting different data into each deep neural network; Fusing each of the acquired feature maps through a fusion network; And detecting an object based on a new feature map fused through the fusion network.

Description

Deep learning-based camera, rider sensor convergence recognition method and system robust to sensor quality deterioration {ROBUST CAMERA AND LIDAR SENSOR FUSION METHOD AND SYSTEM}

The following description relates to a deep learning technique for recognizing objects by fusing different data.

Object recognition technology through image information is an area that has been actively studied in various approaches in the field of computer vision. Typically, there is a method of extracting features that can help to recognize an object in an image from the image and determining the position and type of the object through a classifier. Recently, due to the development of deep learning technology that utilizes a deep neural network structure, a method for learning features from a large amount of image data and detecting an object with a very high level of accuracy has been proposed. As an example, it is a technology that recognizes or detects an object by finding out features such as the structure or shape of a line of an image and comparing it with a template that is already known.Scale invariant feature transform (SIFT) and history of oriented HOG gradients). SIFT is a method of extracting feature vectors having directionality for local patches centered on each feature point after selecting feature points that are easily identified, such as corner points, in the image. This is a method using a feature that expresses the direction of brightness change and the sudden degree of brightness change. HOG is a method using a vector that divides the target region into cells of a predetermined size, obtains a histogram for the direction of the boundary pixels for each cell, and then connects the histogram bin values in a row. Since HOG uses boundary direction information, it can be viewed as a kind of boundary-based template matching method. In addition, since it uses silhouette information of an object, it is a suitable method for identifying an object having unique unique contour information without complicated internal patterns such as a person or a car. Since these technologies utilize only the known information of the object about the edge or shape in advance, the recognition performance is greatly improved when the feature value deviates from the predicted distribution when various illuminance changes, shape distortion, noise, and occlusion occur in the image data. There is a downside.

Deep learning technology, which has been studied a lot in recent years, has the advantage of maintaining good performance in various environments or changes by using a method of directly expressing data, that is, extracting features from a large amount of data. In particular, the convolutional neural network (CNN) structure can hierarchically extract the local characteristics of a 2D image to effectively obtain high-dimensional semantic information necessary for image recognition. Recently, as computer technology capable of parallel computing has been developed and an infrastructure capable of utilizing large amounts of data has been made available, such deep learning technology is being used as a very effective recognition technology.

Meanwhile, the sensor fusion technology refers to a technology that combines and utilizes data acquired from various sensors. Sensor fusion is an important technology in the field of autonomous driving or image recognition because it is possible to increase the performance and reliability of cognition by using various types of sensors with different characteristics. In order to fuse sensor data of various distributions, various fusion techniques are possible. Generally, there are early stage fusion, middle stage fusion, and late stage fusion. Early stage fusion is a fusion method in which one or more data is pre-combined and processed at the same time, and late stage fusion is a technique in which each data is processed and combined after obtaining a final recognition result. Intermediate-stage fusion is a method located in the middle of these two methods. In the early stage fusion, it is difficult to obtain good fusion performance due to heterogeneous data characteristics, and the later stage fusion has a disadvantage in that it cannot utilize the high-order semantic correlation of data well. As sensor fusion technology based on deep learning has recently emerged, the middle-stage fusion technique that combines each sensor signal through a separate CNN and processes it through the CNN is combined with the intermediate feature and the combined feature value in the last stage is good. It shows performance. Since it is possible to obtain sensor measurement data having different distributions for the same environment from these various sensors, the question of how to optimize sensor performance by integrating these different types of sensor data is becoming important. Basically, it is necessary to develop a sensor fusion technique that effectively combines data by reflecting the characteristics of each sensor performance rather than processing each sensor data separately and reflecting the results.

In the field of autonomous driving, deep learning technology has been applied to cognitive technology based on sensor fusion. In particular, research is being actively conducted to improve the reliability of cognitive performance by integrating a camera mounted on a vehicle with a rider sensor. As a technology that applies deep learning for fusion of these cameras and rider sensors, a method of creating a top-view image with rider points and fusion using a fully connected layer in the camera image and network, and second, a top-view made with rider points There is a method of converting and converting an image into a front image.

However, the following problems may occur in object recognition technology through sensor fusion. After extracting the feature map of each sensor data using CNN, in the case of the technology that fuses the camera and the rider sensor at the network stage, after learning through various data, fixed network coefficients are used. In this case, when both sensor data are intact, they exhibit good performance, but when the quality of one sensor data is deteriorated, the network does not handle this situation well, resulting in a problem of deterioration in performance. Accordingly, in a self-driving environment where sensor data is frequently degraded, it can be a cause of a fatal problem.

References: KR10-2018-0003535 (2018.01.09.), KR10-1714233 (2017.03.02.), KR10-2016-0096460 (2016.08.16)

It is possible to provide a method and system for detecting an object based on a new feature map constructed by fusing each feature map obtained by inputting different data into a deep neural network through a fusion network.

 In addition, it is possible to provide a deep learning-based convergence cognitive technology that is robust to deterioration in sensor quality. In other words, it is possible to provide a sensor fusion method and system that effectively combines data by reflecting characteristics of different types of sensor performance. Specifically, through the deep neural network for fusion of the camera's image data and the rider's point data, the object's cognitive performance is improved, and even if some of the camera or rider sensor data is degraded, a method of exerting a high level of object recognition performance and System.

The cognitive method performed by the cognitive system includes: inputting different data into each deep neural network to obtain respective feature maps; Fusing each of the acquired feature maps through a fusion network; And detecting an object based on a new feature map fused through the fusion network.

The step of acquiring each feature map by inputting the different data into each deep neural network may include converting 2 data according to a pre-processing process for the rider when the different data is data related to a rider and a camera. And passing the dimensional three-channel image and the camera image for the camera through each CNN.

The step of acquiring each feature map by inputting the different data into each deep neural network maps the 3D point information acquired from the rider to the 2D image of the camera, but the location information of the 3D point information It may include the step of performing a pre-processing process of generating a coordinate of a 2D image by multiplying matrices for converting from rider coordinates to camera coordinates.

In the step of fusing each of the acquired feature maps through a convergence network, after assigning a weight to any one or more feature maps of the respective feature maps by determining the quality of the respective feature maps, each feature map It may include the step of fusion through the fusion network.

In the step of fusing each acquired feature map through a convergence network, if each acquired feature map is a feature map of the camera and a feature map of the rider, the feature map of the camera and the feature map of the rider are fused As passing through the network, each feature map may be fused in parallel to generate a new feature map.

In the step of fusing each of the acquired feature maps through a fusion network, the camera's feature map and the rider's feature map are first fused in parallel, and the first fused feature map is a plurality of 3X3 size kernels. And passing through a deep neural network having a plurality of sigmoid functions and determining the robustness of the rider or the camera.

In the step of fusing each of the acquired feature maps through a fusion network, as the first fused feature map passes a deep neural network having a plurality of 3 × 3 kernels and a sigmoid function, the camera and the rider The reliability of the data may include outputting a value between 0 and 1 for each pixel.

In the step of fusing each of the acquired feature maps through a convergence network, a value obtained by multiplying the reliability of the camera by the feature map of the camera and the multiplied reliability of the rider by the feature map of the rider in parallel 2 It may include a step of fusion and deriving a new feature map for the feature map of the camera and the feature map of the rider by performing a third fusion of the second fused feature map through a deep neural network of a 1x1 kernel. have.

In the step of detecting an object based on a new feature map fused through the convergence network, generating data for reducing the quality of some of the different data and controlling to learn the degraded data It may include steps.

In the step of detecting an object based on a new feature map fused through the convergence network, as the deep neural network and the sensor convergence network are simultaneously learned, the new feature map is processed to simultaneously determine the location and type associated with the object. It may include the steps.

The cognitive system may include a feature map acquiring unit configured to input different data into each deep neural network to obtain a first feature map and a second feature map; A fusion unit that fuses the obtained first feature map and second feature map through a fusion network; And a detection unit detecting an object based on a new feature map fused through the fusion network.

The feature map acquiring unit converts the two-dimensional three-channel image and the image of the second data by performing preprocessing on the first data as the first data and the second data are input as the different data. Can be passed through each CNN.

The fusion unit may assign a weight to one or more feature maps of each feature map according to determining the quality of the first feature map and the second feature map, and then fuse each feature map through a fusion network. have.

The fusion unit first fuses the first feature map and the second feature map in parallel, and the first fused feature map passes through a deep neural network having a plurality of 3X3 kernels and a sigmoid function. As a reliability of the data of the camera and the rider, a value between 0 and 1 may be output for each pixel.

The fusion unit secondarily fuses the multiplied value of the reliability of the camera to the feature map of the camera and the multiplied value of the reliability of the rider to the feature map of the rider in parallel, and 1x1 of the second fused feature map. Through the deep neural network of the kernel of the size, the third feature may be fused to derive a new feature map for the feature map of the camera and the feature map of the rider.

The recognition system according to an embodiment recognizes and detects an object by recognizing an object based on a new feature map fused through a fusion network, each feature map obtained by inputting different data into each deep neural network. Improve it.

In addition, since the different feature maps are weighted and combined in the convergence network, the optimal sensor convergence is possible by adjusting the contribution to the deteriorated feature map. In other words, when applying a network that has been trained without a conventional convergence network (GFU), sensor quality degradation such as brightness change, occlusion, noise, and failure occurs in one of the sensor data, and the combined feature map is affected. It can solve the problem that the overall cognitive performance is lowered.

In addition, there is an advantage that the information included in the rider and the camera can be used as it is, by using deep learning techniques, excellent classification and generalization performance of deep learning.

1 is a view for explaining the general operation of the cognitive system according to an embodiment.
2 is a block diagram illustrating the configuration of a cognitive system according to an embodiment.
3 is a flowchart illustrating an object recognition method of a cognitive system according to an embodiment.
4 is a diagram for explaining the structure of a deep learning-based object recognition algorithm in a cognitive system according to an embodiment.
5 is a diagram for explaining the structure of GFU for fusion of different data in a cognitive system according to an embodiment.
6 is an example of deteriorating the performance of a camera image in order to evaluate the performance of a cognitive system according to an embodiment.
7 is a table for explaining the performance of a cognitive system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a view for explaining the general operation of the cognitive system according to an embodiment.

In the IoT environment for autonomous driving or smart homes, we propose a deep learning-based object recognition algorithm (100) that can recognize objects and understand the environment by integrating cameras and rider sensors. The deep learning-based object recognition algorithm 100 may be operated by a cognitive system. At this time, data having different characteristics / structures may recognize an object (eg, an object, an object, etc.) by performing a deep learning-based object recognition algorithm. In the following embodiment, a method of recognizing an object using data of a camera and data of a rider sensor (hereinafter referred to as 'rider') will be described as an example.

In the following embodiment, a sensor fusion deep learning technique for simultaneously using two-dimensional point image information obtained from a rider and image information acquired through a camera as input of a deep neural network will be described. In addition, even if degradation occurs in the sensor data of one of the rider or the camera, it exhibits good performance based on the other sensor data.

The cognitive system may recognize an object based on a deep learning-based object recognition algorithm 100 by performing a data pre-processing process as data of the camera 110 and the rider 120 are acquired. The camera data obtained from the camera 110 may be recognized based on the deep learning-based object recognition algorithm 100. At this time, the camera data is a two-dimensional image, and may be composed of an RGB image 111. The 3D point data (3D point cloud) 121 obtained from the rider 120 is converted into a 2D image (front-view image) 122 to convert the object based on the deep learning-based object recognition algorithm 100. Can be recognized. Specifically, the data pre-processing process may be performed on the camera or / and the rider. Accordingly, a pre-processing process for data obtained from the rider will be described. For example, the 3D spatial information obtained from the rider may be converted into an image including at least one of 2D depth, height, or reflectance, and imaged in 3 channels. The two-dimensional three-channel image on which the pre-processing has been performed may be used as an input of two convolutional neural networks (CNN) by using the RGB image 111 together. Feature maps obtained from two CNNs are combined and final information related to object detection is extracted using the combined feature maps. When combining the feature maps obtained from two CNNs, in order to obtain robust performance when the quality of one sensor signal deteriorates in poor conditions, appropriate weight values are applied to the feature maps obtained from the sensor signals of Kyder and the camera. The sensor fusion technique to apply and fuse is applied. These weight values are automatically calculated using the feature map as inputs, and the convergence network that calculates the weights in the training step is simultaneously trained with the two previous CNN networks.

The cognitive system may detect the cognitive result 130 and the object according to learning the camera data and the rider data based on the deep learning-based object recognition algorithm 100. Accordingly, it is possible to obtain good recognition performance when quality deterioration occurs in data of a specific sensor as well as intact data as compared to the existing rider sensor and camera sensor fusion technology.

2 is a block diagram illustrating a configuration of a cognitive system according to an embodiment, and FIG. 3 is a flowchart illustrating an object recognition method of a cognitive system according to an embodiment.

The cognitive system 200 may include a feature map acquisition unit 210, a fusion unit 220, and a detection unit 230. These components may be expressions of different functions performed by a processor according to a control command provided by program code stored in the cognitive system 200. The components may control the cognitive system 200 to perform steps 310 to 330 included in the object recognition method of FIG. 3. At this time, the components may be implemented to execute instructions according to the code of the operating system included in the memory and the code of at least one program.

The processor of the recognition system 200 may load program code stored in a file of a program for an object recognition method into a memory. For example, when a program is executed in the cognitive system 200, the processor may control the cognitive system to load program code from a file of the program into a memory under the control of the operating system. At this time, each of the feature map acquisition unit 210, the fusion unit 220, and the detection unit 230 included in the processor and the processor executes an instruction of a corresponding portion of the program code loaded in the memory, and then performs the following steps 310 to 330 ) May be different functional representations of the processor for executing.

In step 310, the feature map acquiring unit 210 may acquire different feature maps by inputting different data into each deep neural network. The feature map acquiring unit 210 is configured to perform a pre-processing process on the data of the first sensor as the first sensor and the second sensor are input as different data, and convert the two-dimensional three-channel image and the second sensor. Images can be passed through each CNN. For example, when the different data is data related to the rider and the camera, the feature map acquiring unit 210 respectively converts the two-dimensional three-channel image and the camera image for the camera by performing a pre-processing process for the rider. Can pass through CNN. At this time, the feature map acquiring unit 210 maps the 3D point information obtained from the rider to the 2D image of the camera, but multiplies the location information of the 3D point information from the rider coordinates to the camera coordinates to multiply the 2D image. A pre-processing process generated by coordinates can be performed.

In step 320, the fusion unit 220 may fuse each acquired feature map through a fusion network. As the quality of each feature map is determined, the fusion unit 220 may assign weights to one or more feature maps of each feature map, and then fuse each feature map through a fusion network. When each acquired feature map is a feature map of the camera and a feature map of the rider, the fusion unit 220 fuses each feature map in parallel by passing the feature map of the camera and the feature map of the rider through the fusion network. To create a new feature map. The fusion unit 220 first fuses the feature map of the camera and the rider's feature map in parallel, and passes the first fused feature map through a deep neural network having a plurality of 3X3 kernels and a plurality of sigmoid functions. The toughness of the rider or the camera can be determined. The fusion unit 220 is the reliability of the data of the camera and the rider as it passes through the deep neural network having a plurality of 3X3 size kernels and the sigmoid function, and the primary fused feature map is a value between 0 and 1 for each pixel. Can be printed. The fusion unit 220 secondarily fuses the multiplied value of the reliability of the camera to the feature map of the camera and the multiplied value of the reliability of the rider to the feature map of the rider in parallel, and the second fused feature map is 1x1 in size. Through the kernel's deep neural network, it is possible to derive a new feature map for the camera's feature map and the rider's feature map by tertiary fusion.

In step 330, the detector 230 may detect the object based on the new feature map fused through the fusion network. As the learning unit 230 simultaneously learns the deep neural network and the sensor fusion network, the new feature map can be processed to simultaneously determine the location and type associated with the object. The detector 230 may generate data for deteriorating the quality of some of the different data, and may control to learn the degraded data.

The deep learning-based object recognition algorithm has the advantage that the information included in the rider and the camera can be utilized by using the deep classification and general classification performance of deep learning as it is.

4 is a diagram for explaining a structure of an object recognition algorithm based on deep learning of a cognitive system according to an embodiment.

를 도출할 수 있다. 영상 좌표는 영상의 각 픽셀의 위치에 해당하게 되고, 픽셀값은 깊이, 높이, 반사율 정보 값으로부터 획득될 수 있다. 예를 들면, 매우 가까이 있는 포인트에 대해서는 255에 가까운 밝기로, 매우 먼 경우에 있는 포인트에 대해서는 0에 가까운 밝기로 변환될 수 있다. 이 경우, 포인트를 이용하여 2차원 공간을 모두 채울 수 없기 때문에 포인트가 이미지 상에 생긴 형태로 드문드문(sparse) 나타나게 된다. 포인트가 존재하지 않는 픽셀의 경우 0으로 채울 수 있다. 이미 카메라의 좌표와 라이더의 좌표가 정합이 되어 있을 경우, 앞서 설명한 바와 같은 방법으로 라이더의 데이터를 변환하여 새로운 2차원의 3채널 이미지를 추가적으로 획득할 수 있다. 이와 같이, 라이더에 대하여 전처리 과정을 수행함에 따라 변환된 라이더에 대한 2차원의 3채널 이미지(122)와 카메라의 카메라 이미지(111)를 각각의 딥 뉴럴 네트워크(예를 들면, CNN)에 통과시킬 수 있다. 도 4와 같이, 딥 뉴럴 네트워크의 입력 부분에는 라이더의 3채널 이미지와 카메라의 카메라 이미지를 각각 독립적인 CNN을 통과시킨 후, 어느 시점 이후의 네트워크 노드에서 획득되는 특징맵(feature map)을 융합 네트워크(GFU)(410)를 통하여 융합하여 새로운 특징맵을 구성할 수 있다. 이러한 특징맵을 처리함에 따라 객체의 위치와 종류를 동시에 판별(420)할 수 있다. 이때, 예를 들면, Single Shot Detector(SSD) 방법을 통하여 객체가 검출될 수 있으며, SSD 방법 이외에도 다양한 방법을 통하여 객체를 검출할 수도 있다. Referring to FIG. 4, a structure 400 of a deep learning-based object recognition algorithm is illustrated. As the sensor data of the camera and the rider are first passed through a plurality of deep neural networks (for example, CNN), the quality of each data is determined, and then the two data are fused to recognize the object. In other words, the RGB image of the camera (hereinafter referred to as the camera image) 111 and the rider's lidar image 122 may be input to each deep neural network. At this time, a pre-processing process may be performed on the rider's data for sensor fusion technology. The 3D point information acquired through the rider can be mapped to the 2D image of the camera. 2D image coordinates by multiplying the coordinate information p = (x, y, z) of the 3D point by a matrix that converts from rider coordinates to camera coordinates

Can be derived. The image coordinates correspond to the position of each pixel in the image, and the pixel value can be obtained from depth, height, and reflectance information values. For example, it may be converted to a brightness close to 255 for a point very close, and to a brightness close to 0 for a point very distant. In this case, since the two-dimensional space cannot be filled by using the points, the points appear sparse in the form formed on the image. Pixels with no points can be filled with zeros. If the coordinates of the camera and the rider's coordinates have already been matched, a new two-dimensional three-channel image can be additionally obtained by converting the rider's data in the same manner as described above. As such, as the pre-processing process is performed on the rider, the two-dimensional three-channel image 122 and the camera image 111 of the converted rider are passed through each deep neural network (for example, CNN). Can be. As shown in FIG. 4, a fusion network uses a feature map obtained from a network node after a certain point after passing an independent CNN through a 3-channel image of a rider and a camera image of a camera, respectively, in an input portion of a deep neural network. It is possible to construct a new feature map by fusion through (GFU) 410. As the feature map is processed, the location and type of the object may be simultaneously determined (420). At this time, for example, an object may be detected through a Single Shot Detector (SSD) method, and an object may be detected through various methods in addition to the SSD method.

Referring to FIG. 5, it is a view for explaining the structure of the GFU for fusion of the feature map 510 of the camera and the feature map 520 of the rider. After determining the quality of each feature map, after assigning a weight to one or more feature maps of each feature map, each feature map may be fused through a fusion network (GFU 410). Specifically, first, the feature map 510 of the camera and the feature map 520 of the rider are merged in parallel. Then, the feature maps (primary fusion feature maps) merged in parallel pass through the CNN and sigmoid functions, each with a 3X3 kernel. Through this process, it is possible to determine which of the camera and rider data is more reliable data. At this time, as the feature map (primary fusion feature map) merged in parallel passes through the CNN and sigmoid functions having a 3X3 kernel, the reliability of the data for each pixel is mapped to a value between 0 and 1 to output the output data. Can be. In other words, the output passed through the CNN and sigmoid functions with a 3X3 kernel size can be derived per pixel by 0 to 1 as the reliability of the data. The reliability can be multiplied back to each feature map of the camera and the rider, and the multiplied feature maps can be combined again in parallel. Then, after passing through a CNN with a 1X1 kernel, the camera's feature map and the rider's feature map are finally fused. Specifically, the value multiplied by the reliability of the camera is multiplied by the camera's feature map and the value of the reliability of the rider by the feature map of the rider is secondly fused in parallel again, and the second fused feature map is deep in the kernel of 1X1 size. A new feature map for the feature map of the camera and the feature map of the rider may be constructed by tertiary fusion through the neural network (eg, CNN). Finally, the GFU coefficient, the CNN corresponding to each sensor, and the network coefficient for extracting the detection result from the fused feature map are simultaneously learned. Accordingly, an object may be detected based on the new feature map.

At this time, when the sensor quality of the camera and the rider is deteriorated, a data augmentation technique is applied to apply the deterioration of the quality to one of the camera and the rider in order to learn a function of assigning a weight appropriate thereto. For example, referring to FIG. 6, this is an example of deteriorating an image of a camera in order to evaluate the performance of the deep learning-based object recognition algorithm proposed in the embodiment. FIG. 6 (a) is image data whose brightness is manipulated, FIG. 6 (b) is image data that is partially obscured, FIG. 6 (c) is image data including noise, and FIG. 6 (d) is an original image Data. You can create additional data to give robustness through operations such as partially obscuring the image data, adjusting the brightness, or removing one of the two data, and train the network through this. By lowering the quality of some of the data and learning the data, it is possible to adaptively assign a weight as well as a network structure according to the quality of the image.

In the case of sensor fusion through deep learning, the cognitive system according to an embodiment combines the feature map acquired through CNN to obtain a new feature map, and additionally applies the CNN layer to the acquired new feature map to further apply sensor fusion. Based object detection is performed. At this time, since an appropriate weight is applied to each feature map in the fusion network (GFU) to combine them, optimal sensor fusion is achieved by controlling the contribution to the deteriorated feature map. In addition, it is possible to exert fusion using the advantages of data for the rider and the camera, and even when a degradation occurs in one sensor data, optimal fusion is possible.

7 is a table for explaining the performance of a cognitive system according to an embodiment.

Object recognition and detection performance can be improved through deep learning techniques for fusion of camera and rider data. For example, using the KITTI benchmark data set, after matching the rider coordinates and the camera coordinates, the 3D point data of the rider sensor is converted into a 2D image, and then the 2D image of the rider sensor and the image of the camera are sent to CNN. You can enter each. At this time, the quality of sensor data is determined and fused through the fusion network (GFU) to recognize the object. 7 shows the results of comparing the performance for GFU with the case where GFU is included in the base network and the case where GFU is not included. For example, to compare performance, the camera image may be empty, partially obscured, partially brightened, or noise may be mixed, and in addition, the rider image may be empty or partially obscured. Accordingly, as a result of the comparison, it can be seen that the network in which GFU exists in all items showed better performance, and in particular, when noise is mixed, shows a larger performance difference.

The cognitive system according to an embodiment may be applied to autonomous driving in which a rider and a camera sensor are expected to be simultaneously used in recent autonomous driving, and may be applied to various artificial intelligence fields that recognize an environment or an object as well as autonomous driving.

Typical applications include autonomous vehicles, robotics and vehicle monitoring systems. First, in autonomous vehicles, object recognition is the first technology that must be carried out, and it serves to determine whether there are objects and pedestrians in the vicinity and whether there are any dangerous factors. Through this, you can drive safely and avoid accidents. Can be prevented. It is also used to check road information required for driving, such as traffic lights and signs, to assist driving. Second, in robotics, object recognition helps the robot's visual activity like the human eye. Through object recognition, the robot can check surrounding conditions and objects to perform functions suitable for the purpose. Also, object recognition technology can be applied to vehicle monitoring systems. As an example of a parking management system, it is possible to assist with parking by recognizing the type and vehicle number of a car entering the parking lot and further checking the current vacant situation.

In order to apply this technology, there is a method of acquiring data in real time using a camera and a rider sensor and performing an object recognition algorithm in an embedded system equipped with a graphic processor unit (GPU). To do this, it is necessary to acquire data on various environments in advance and learn the structure of a deep neural network. The trained deep neural network is stored as an optimized network coefficient and applied to an embedded system to perform object recognition algorithm on test data input in real time to obtain the result.

In addition, object recognition algorithms based on cameras and rider sensors can be applied to current autonomous driving or mobile robots, and are expected to perform more complex functions such as object tracking and future prediction beyond future recognition. For example, automated monitoring, traffic monitoring and vehicle navigation or traffic light control considering traffic conditions on the road will be possible. In addition, object recognition plays an important role in understanding the surrounding environment, and it can be applied to home appliances that combine IoT. As described above, deep learning-based object recognition algorithm is a technology that is based on future technologies and can be said to be one of the representative artificial intelligence technologies.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In the cognitive method performed by the cognitive system,
Obtaining different feature maps by inputting different data into each deep neural network;
Fusing each of the acquired feature maps through a fusion network; And
Detecting an object based on a new feature map fused through the convergence network
Including,
The step of acquiring each feature map by inputting the different data into each deep neural network,
When the different data is data related to the rider and the camera, the converted 2-dimensional three-channel image and the camera image for the camera are passed through each CNN by performing a pre-processing process on the rider, and from the rider Mapping the acquired 3D point information to the 2D image of the camera, but performing a pre-processing process of generating location information of the 3D point information from rider coordinates to camera coordinates and generating 2D image coordinates by multiplying the matrix.
Cognitive method comprising a. delete delete According to claim 1,
The step of fusing each of the acquired feature maps through a fusion network,
After determining the quality of each feature map, assigning a weight to one or more feature maps of the feature maps, and then fusing each feature map through a convergence network
Cognitive method comprising a. The method of claim 4,
The step of fusing each of the acquired feature maps through a fusion network,
When each of the acquired feature maps is a feature map of the camera and a feature map of the rider, as the feature map of the camera and the feature map of the rider pass through the fusion network, each feature map is fused in parallel to generate new features. Steps to create a map
Cognitive method comprising a. In the cognitive method performed by the cognitive system,
Obtaining different feature maps by inputting different data into each deep neural network;
Fusing each of the acquired feature maps through a fusion network; And
Detecting an object based on a new feature map fused through the convergence network
Including,
The step of fusing each of the acquired feature maps through a fusion network,
As the quality of each feature map is determined, weights are applied to any one or more feature maps of the feature maps, and then each feature map is fused through a fusion network, and each acquired feature map is a camera. In the case of the feature map of the rider and the feature map of the rider, as the feature map of the camera and the feature map of the rider pass through the fusion network, each feature map is fused in parallel to generate a new feature map, and the features of the camera The map and the rider's feature map are first fused in parallel, and the first fused feature map is passed through a deep neural network with a plurality of 3X3 size kernels and a plurality of sigmoid functions to improve the robustness of the rider or the camera. Judging steps
Cognitive method comprising a. The method of claim 6,
The step of fusing each of the acquired feature maps through a fusion network,
Outputting a value of 0 to 1 for each pixel as a reliability of data of the camera and the rider by passing the primary fused feature map through a deep neural network having a plurality of 3X3 kernels and a sigmoid function.
Cognitive method comprising a. The method of claim 7,
The step of fusing each of the acquired feature maps through a fusion network,
Secondly, the second multiplicity of the multiplied value of the reliability of the camera is multiplied by the feature map of the camera and the multiplied value of the reliability of the rider by the feature map of the rider, and the second fused feature map of the kernel of 1x1 size. Step 3 through a deep neural network to derive a new feature map for the feature map of the camera and the feature map of the rider
Cognitive method comprising a. According to any one of claims 1 or 6,
The step of detecting an object based on a new feature map fused through the convergence network includes:
Generating data for reducing the quality of some of the different data, and controlling to learn the deteriorated data
Cognitive method comprising a. According to any one of claims 1 or 6,
The step of detecting an object based on a new feature map fused through the convergence network includes:
By simultaneously learning the deep neural network and the convergence network, processing the new feature map to simultaneously determine the location and type associated with the object.
Cognitive method comprising a. In the cognitive system,
A feature map acquiring unit for inputting different data into each deep neural network to obtain a first feature map and a second feature map;
A fusion unit that fuses the obtained first feature map and second feature map through a fusion network; And
A detection unit that detects an object based on a new feature map fused through the convergence network
Including,
The fusion unit,
The first and second feature maps are first fused in parallel, and the data of the camera and the rider as the first fused feature map passes through a deep neural network and a sigmoid function having a plurality of 3X3 kernels. Reliability for outputting values between 0 and 1 for each pixel
Cognitive system. The method of claim 11,
The feature map acquisition unit,
As the first data and the second data are input as the different data, the converted 2-dimensional three-channel image and the image for the second data are passed through each CNN as a pre-processing process for the first data. Letting
Cognitive system characterized in that. The method of claim 11,
The fusion unit,
As the quality of the first feature map and the second feature map is determined, weighting is applied to one or more feature maps of each feature map, and then each feature map is fused through a convergence network.
Cognitive system characterized in that. delete The method of claim 11,
The fusion unit,
Secondly, the second multiplicity of the multiplied value of the reliability of the camera is multiplied by the feature map of the camera and the multiplied value of the reliability of the rider by the feature map of the rider, and the second fused feature map of the kernel of 1x1 size. Through the deep neural network, the third feature is fused to derive a new feature map for the camera feature map and rider feature map.
Cognitive system characterized in that.

Images