KR20190131207A - Robust camera and lidar sensor fusion method and system - Google Patents

Abstract

Disclosed are a deep learning-based camera robust to the sensor quality degradation and a method and system for recognizing the LIDAR sensor fusion. According to one embodiment of the present invention, the method performed by the system may comprise the steps of: inputting different data to each deep neural network to obtain each feature map; fusing the obtained feature maps through a fusion network; and detecting an object based on a new feature map fused through the fusion network.

Description

Deep learning based camera and rider sensor fusion recognition method and system robust against deterioration of sensor quality {ROBUST CAMERA AND LIDAR SENSOR FUSION METHOD AND SYSTEM}

The following description relates to a deep learning technique for recognizing objects by fusing different data.

Object recognition technique using image information is a field that has been actively studied in various approaches in the field of computer vision. Typically, there is a method of extracting features from an image that can help to recognize an object in the image and determining the position and type of the object through a classifier. Recently, due to the development of deep learning technology using a deep neural network structure, a method of learning features from a large amount of image data and detecting an object with a very high level of accuracy has been proposed. For example, it is a technology that recognizes or detects an object by comparing features of a line structure or shape of an image and comparing it with a template that is already known, and typical feature extraction techniques include SIFT (Scale invariant feature transform) and HOG (Historgram of oriented). gradients), etc. SIFT is a method of extracting feature vectors with directivity for local patches around each feature point after selecting feature points that can be easily identified such as corner points in an image. This is a method using a feature that expresses the direction of the change in brightness of the surroundings and the degree of sudden change in brightness. HOG divides the target area into cells of a certain size, obtains a histogram of the direction of boundary pixels for each cell, and then uses a vector in which these histogram bin values are connected in a row. HOG can be viewed as a kind of boundary-based template matching method because it uses the direction information of the boundary. In addition, since the silhouette information of the object is used, it is a suitable method for identifying an object having unique unique contour information without complicated internal patterns such as a person or a car. Since these technologies utilize only the known information of the object about the edge or shape in advance, the recognition performance is greatly increased when the feature values deviate from the predicted distribution when various illumination changes, shape distortions, noise, and occlusion occur in the image data. There is a downside to falling.

Deep learning technology, which has been studied a lot recently, has the advantage of maintaining good performance in various environments or changes by using a method of directly expressing data extraction, that is, feature extraction from a large amount of data. In particular, the convolutional neural network (CNN) structure can extract high-level semantic information necessary for image recognition by hierarchically extracting local features of two-dimensional images. Recently, with the development of computer technology that can perform parallel computation and the infrastructure that can utilize large amounts of data, such deep learning technology is being used as a very effective recognition technology.

Meanwhile, the sensor fusion technology refers to a technology that combines and uses data acquired from various sensors. Sensor fusion is an important technology in the field of autonomous driving or image recognition because it is possible to improve the performance and reliability of cognition by using various kinds of sensors having different characteristics. Various fusion techniques are possible to fuse sensor data of various distributions. Generally, early fusion, intermediate fusion, and late fusion are available. Early stage fusion is a fusion method in which one or more data is precombined and processed simultaneously, and late stage fusion is a technique of combining each data after the final recognition result is obtained. Intermediate stage fusion is a method located in the middle of these two methods. In the early stage fusion, it is difficult to obtain good fusion performance due to heterogeneous data characteristics, and the late stage fusion does not use the higher semantic correlation of data well. Recently, the sensor fusion technique by deep learning technique has emerged, and the middle stage fusion technique that passes each sensor signal through a separate CNN, combines them in the middle, and processes the combined feature values at the last stage through the CNN is good. Performance is shown. Since sensor measurement data having different distributions for the same environment can be obtained from these various sensors, the question of how to fuse these different types of sensor data to optimize sensor performance becomes important. Basically, rather than processing each sensor data separately and reflecting the results, sensor fusion technology needs to be developed to effectively combine the data by reflecting the characteristics of each sensor performance.

Recently, deep learning technology has been applied to sensor convergence-based cognitive technology in autonomous driving. In particular, research is being actively conducted to increase the reliability of cognitive performance by fusing cameras and rider sensors installed in automobiles. Deep learning is applied to the fusion of camera and rider sensors, and a top-view image is created by using a rider point and fused using a fully connected layer in the camera image and network. Second, the top-view is made by a rider point. There is a method in which an image is converted into a front image and fused.

However, the object recognition technology through sensor fusion can cause the following problems. After extracting feature maps of each sensor data using CNN, the technology of fusion of camera and rider sensors at the network stage uses fixed network coefficients after learning through various data. In this case, when both sensor data are intact, they show good performance. However, when the quality of one sensor data is degraded, the network does not handle this situation well, resulting in a problem of poor performance. Accordingly, in an autonomous driving environment in which deterioration occurs frequently in sensor data, it may be a cause of fatal problem.

Reference: KR10-2018-0003535 (2018.01.09.), KR10-1714233 (2017.03.02.), KR10-2016-0096460 (2016.08.16)

Provided are a method and system for detecting an object based on a new feature map configured by fusing different feature maps obtained by inputting different data into a deep neural network.

 In addition, it is possible to provide a deep learning-based fusion recognition technology that is robust against deterioration of sensor quality. In other words, it is possible to provide a sensor fusion method and system that effectively combines data by reflecting characteristics of different types of sensor performance. Specifically, a method of improving object recognition performance through a deep neural network for fusing camera image data and point data of a rider and exhibiting a high level of object recognition performance even when some data of the camera or rider sensor is degraded. A system can be provided.

A cognitive method performed by a cognitive system includes: inputting different data into each deep neural network to obtain each feature map; Fusing the acquired respective feature maps through a fusion network; And detecting an object based on a new feature map fused through the fusion network.

The step of acquiring each feature map by inputting the different data to each deep neural network includes: when the different data is data related to a rider and a camera, 2 Transformed by performing a preprocessing process on the rider; And passing the three-dimensional image of the dimension and the camera image for the camera through each CNN.

Acquiring each feature map by inputting the different data into each deep neural network may include mapping 3D point information acquired from the rider to a 2D image of the camera, wherein location information of the 3D point information is mapped. And multiplying the matrix from the rider coordinates to the camera coordinates to generate coordinates of the 2D image.

In the fusion of each of the acquired feature maps through a fusion network, weighting any one or more feature maps of each feature map according to the quality of each feature map is determined, and then each feature map. May include fusing through a fusion network.

The fusing of each acquired feature map through the fusion network may include fusing the feature map of the camera and the feature map of the rider when the acquired feature map is a feature map of a camera and a rider. As passing through the network, each feature map may be fused in parallel to generate a new feature map.

The fusion of each of the acquired feature maps through a fusion network may include: primary fusion of the feature map of the camera and the feature map of the rider in parallel, and converting the primary fused feature map into a plurality of 3X3 kernels. And passing through a deep neural network having a plurality of sigmoid functions and determining the robustness of the rider or the camera.

The fusion of each of the acquired feature maps through a fusion network may include passing the first fused feature map through a deep neural network having a plurality of 3 × 3 kernels and a sigmoid function. And outputting a value between 0 and 1 for each pixel as reliability of the data.

The fusion of each of the acquired feature maps through a fusion network may include: multiplying a value obtained by multiplying the reliability of the camera by a feature map of the camera and a value obtained by multiplying the reliability of the rider by the feature map of the rider in parallel. And fusion the second and second fusion feature maps through a deep neural network of 1x1 size kernels to derive new feature maps for the feature map of the camera and the rider. have.

The detecting of the object based on the new feature map fused through the fusion network may include generating data for reducing the quality of data of some of the different data and controlling learning of the degraded data. It may include a step.

In the detecting of the object based on the new feature map fused through the fusion network, the deep neural network and the sensor fusion network are simultaneously learned to process the new feature map to simultaneously determine the location and type associated with the object. It may include the step.

The cognitive system includes: a feature map obtainer for inputting different data into each deep neural network to obtain a first feature map and a second feature map; A fusion unit configured to fuse the obtained first and second feature maps through a fusion network; And a detector configured to detect an object based on a new feature map fused through the fusion network.

The feature map obtaining unit may include a two-dimensional three-channel image and an image of the second data converted as the first data and the second data are input as the different data, and then the preprocessing process is performed on the first data. Can be passed to each CNN.

The fusion unit may weight the at least one feature map of each feature map according to the quality of the first feature map and the second feature map, and then fuse the feature maps through the fusion network. have.

The fusion unit may first fuse the first feature map and the second feature map in parallel, and pass the first fused feature map through a deep neural network having a plurality of 3 × 3 kernels and a sigmoid function. The reliability of the data of the camera and the rider may be output as a value between 0 and 1 for each pixel.

The fusion unit performs second-order fusion in parallel with the value obtained by multiplying the reliability map of the camera by the feature map of the camera and the reliability map of the rider by the feature map of the rider, and 1x1 the second fused feature map. Through the deep neural network of the kernel of size, the third fusion can be derived to derive new feature maps for the feature map of the camera and the feature map of the rider.

The cognitive system according to an embodiment recognizes and detects an object by recognizing an object based on a new feature map fused through a fusion network by inputting different feature data into each deep neural network. Can improve.

In addition, in the converged network, different feature maps are weighted and combined to adjust the contribution to the degraded feature map, thereby enabling optimal sensor fusion. In other words, when the training is completed without a conventional fusion network (GFU), when the sensor quality deterioration such as brightness change, occlusion, noise, or failure occurs in one of the sensor data, the combined feature map is affected. This can solve the problem of lowering the overall cognitive performance.

In addition, the information contained in the rider and the camera can be used to utilize deep learning techniques to utilize the excellent classification and generalization performance of deep learning.

1 is a diagram for describing an operation of a cognitive system according to an exemplary embodiment.
2 is a block diagram illustrating a configuration of a cognitive system according to an exemplary embodiment.
3 is a flowchart illustrating an object recognition method of a recognition system according to an exemplary embodiment.
4 is a diagram illustrating a structure of a deep learning-based object recognition algorithm in a cognitive system according to an exemplary embodiment.
5 is a diagram illustrating a structure of a GFU for fusion of different data in a cognitive system according to an embodiment.
6 illustrates an example of degrading the performance of a camera image in order to evaluate the performance of a cognitive system according to an exemplary embodiment.
7 is a table for explaining a performance of a cognitive system according to an exemplary embodiment.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram for describing an operation of a cognitive system according to an exemplary embodiment.

In the IoT environment for autonomous driving or smart home, a deep learning-based object recognition algorithm 100 capable of recognizing objects and understanding the environment by fusing a camera and a rider sensor is proposed. The deep learning based object recognition algorithm 100 may be operated by a cognitive system. In this case, data having different characteristics / structures may recognize an object (eg, an object or an object) by performing a deep learning based object recognition algorithm. In the following embodiment, a method of recognizing an object using data of a camera and data of a rider sensor (hereinafter, referred to as a rider) will be described as an example.

In the following embodiment, a sensor fusion deep learning technique for simultaneously utilizing two-dimensional point image information acquired from a rider and image information acquired through a camera as an input of a deep neural network will be described. In addition, even if a degradation occurs in the sensor data of one of the riders or cameras, the performance is good based on the other sensor data.

As the recognition system acquires data of the camera 110 and the rider 120, the recognition system may perform data preprocessing to recognize an object based on the deep learning based object recognition algorithm 100. The object may be recognized based on the deep learning based object recognition algorithm 100 based on the camera data acquired from the camera 110. At this time, the camera data is a two-dimensional image, it may be composed of an RGB image (111). 3D point data (three-dimensional point cloud) 121 obtained from the rider 120 is converted into a two-dimensional image (front-view image) 122 to convert the object based on the deep learning-based object recognition algorithm 100 It can be recognized. Specifically, the data preprocessing process may be performed on the camera or / and the rider. Thus, the preprocessing process for the data obtained from the rider will be described. For example, the 3D spatial information obtained from the rider may be converted into an image including at least one or more information of 2D depth, height, or reflectance, and imaged into 3 channels. The two-dimensional three-channel image subjected to such a preprocessing process may be used as an input of two convolutional neural networks (CNNs) by using the same as the RGB image 111. The feature maps obtained from the two CNNs are combined and final information related to object detection is extracted using the combined feature maps. When combining the feature maps obtained from two CNNs, a weight value appropriate to the feature maps obtained from the sensor signals of the camera and the camera is obtained in order to obtain robust performance when the quality of one sensor signal is degraded in a poor situation. The sensor fusion technique is applied. These weight values are automatically calculated by inputting the feature map, and trained the converged network that calculates the weights in the training phase together with the two previous CNN networks.

The cognitive system may detect the recognition result 130 and the object of learning the data of the camera and the data of the rider based on the deep learning based object recognition algorithm 100. Accordingly, compared to conventional rider sensor and camera sensor fusion technology, good recognition performance can be obtained even when quality deterioration occurs in not only intact data but also data of a specific sensor.

2 is a block diagram illustrating a configuration of a cognitive system according to an embodiment, and FIG. 3 is a flowchart illustrating an object recognition method of a cognitive system according to an embodiment.

The recognition system 200 may include a feature map acquirer 210, a fusion unit 220, and a detector 230. These components may be representations of different functions performed by a processor in accordance with control instructions provided by program code stored in cognitive system 200. The components may control the recognition system 200 to perform the steps 310 to 330 included in the object recognition method of FIG. 3. In this case, the components may be implemented to execute instructions according to code of an operating system included in a memory and code of at least one program.

The processor of the recognition system 200 may load program code stored in a file of a program for an object recognition method into a memory. For example, when a program is executed in the cognitive system 200, the processor may control the cognitive system to load program code from a file of a program into a memory under control of an operating system. In this case, each of the processor and the feature map acquirer 210, the fuser 220, and the detector 230 included in the processor executes instructions of a corresponding part of the program code loaded in the memory, and then executes steps 310 to 330. May be different functional representations of a processor for executing < RTI ID = 0.0 >

In operation 310, the feature map acquirer 210 may obtain different feature maps by inputting different data into each deep neural network. The feature map acquirer 210 performs a preprocessing process on the data of the first sensor as the first sensor and the second sensor are input as different data, and thus the 2D 3-channel image and the second sensor You can pass an image through each CNN. For example, when the different data is data related to the rider and the camera, the feature map acquisition unit 210 performs a preprocessing process on the rider, and converts the two-dimensional three-channel image and the camera image of the camera, respectively. Pass it to CNN. At this time, the feature map acquisition unit 210 maps the 3D point information obtained from the rider to the 2D image of the camera, and multiplies the matrix for converting the position information of the 3D point information from the rider coordinates to the camera coordinates to obtain the 2D image. A preprocessing process can be performed to generate coordinates.

In operation 320, the fusion unit 220 may fuse each acquired feature map through a fusion network. The fusion unit 220 may assign a weight to any one or more feature maps of each feature map according to the quality of each feature map, and then fuse the feature maps through the fusion network. When each of the acquired feature maps is the feature map of the camera and the feature map of the rider, the fusion unit 220 merges the feature maps in parallel as the feature map of the camera and the rider feature map pass through the fusion network. To create a new feature map. The fusion unit 220 first fuses the feature map of the camera and the feature map of the rider in parallel, and passes the first fused feature map through a deep neural network having a plurality of 3X3 kernels and a plurality of sigmoid functions. The robustness of the rider or the camera can be determined. As the fusion unit 220 passes the first fused feature map through a deep neural network having a plurality of 3X3 kernels and a sigmoid function, the fusion unit 220 has a value between 0 and 1 for each pixel as a reliability of the data of the camera and the rider. Can be printed. The fusion unit 220 performs second-order fusion in parallel with the value obtained by multiplying the reliability of the camera by the feature map of the camera and the value of the rider's reliability by the feature map of the rider, and converts the second-fused fused feature map into a 1x1 size. Through the kernel's deep neural network, the third fusion can be used to derive new feature maps for the camera's feature map and the rider's feature map.

In operation 330, the detector 230 may detect an object based on the new feature map fused through the converged network. As the detector 230 simultaneously learns the deep neural network and the sensor fusion network, the detector 230 may process a new feature map to simultaneously determine the location and type of the object. The detector 230 may generate data for lowering the quality of some of the different data, and may control to learn data whose quality is degraded.

Deep learning-based object recognition algorithms have the advantage of using deep learning techniques to take advantage of the deep classification and generalization of information contained in riders and cameras.

4 is a diagram illustrating a structure of an object recognition algorithm based on deep learning of a cognitive system, according to an exemplary embodiment.

를 도출할 수 있다. 영상 좌표는 영상의 각 픽셀의 위치에 해당하게 되고, 픽셀값은 깊이, 높이, 반사율 정보 값으로부터 획득될 수 있다. 예를 들면, 매우 가까이 있는 포인트에 대해서는 255에 가까운 밝기로, 매우 먼 경우에 있는 포인트에 대해서는 0에 가까운 밝기로 변환될 수 있다. 이 경우, 포인트를 이용하여 2차원 공간을 모두 채울 수 없기 때문에 포인트가 이미지 상에 생긴 형태로 드문드문(sparse) 나타나게 된다. 포인트가 존재하지 않는 픽셀의 경우 0으로 채울 수 있다. 이미 카메라의 좌표와 라이더의 좌표가 정합이 되어 있을 경우, 앞서 설명한 바와 같은 방법으로 라이더의 데이터를 변환하여 새로운 2차원의 3채널 이미지를 추가적으로 획득할 수 있다. 이와 같이, 라이더에 대하여 전처리 과정을 수행함에 따라 변환된 라이더에 대한 2차원의 3채널 이미지(122)와 카메라의 카메라 이미지(111)를 각각의 딥 뉴럴 네트워크(예를 들면, CNN)에 통과시킬 수 있다. 도 4와 같이, 딥 뉴럴 네트워크의 입력 부분에는 라이더의 3채널 이미지와 카메라의 카메라 이미지를 각각 독립적인 CNN을 통과시킨 후, 어느 시점 이후의 네트워크 노드에서 획득되는 특징맵(feature map)을 융합 네트워크(GFU)(410)를 통하여 융합하여 새로운 특징맵을 구성할 수 있다. 이러한 특징맵을 처리함에 따라 객체의 위치와 종류를 동시에 판별(420)할 수 있다. 이때, 예를 들면, Single Shot Detector(SSD) 방법을 통하여 객체가 검출될 수 있으며, SSD 방법 이외에도 다양한 방법을 통하여 객체를 검출할 수도 있다. Referring to FIG. 4, the structure 400 of the deep learning based object recognition algorithm is illustrated. The sensor data of the camera and the rider are first passed through a plurality of deep neural networks (eg, CNNs) to determine the quality of each data, and then the two data may be fused to recognize the object. In other words, an RGB image (hereinafter referred to as a camera image) 111 of a camera and a lidar image 122 of a rider may be input to each deep neural network. In this case, a preprocessing process may be performed on the data of the rider for the sensor fusion technology. Three-dimensional point information acquired through the rider can be mapped to the two-dimensional image of the camera. Two-dimensional image coordinates by multiplying coordinate information p = (x, y, z) by three-dimensional points by a matrix that converts from rider coordinates to camera coordinates

Can be derived. The image coordinates correspond to the position of each pixel of the image, and the pixel value may be obtained from depth, height, and reflectance information values. For example, it can be converted to a brightness close to 255 for very close points and to a brightness close to zero for points that are very far away. In this case, since the point cannot be used to fill all of the two-dimensional space, the point appears sparse in the form of the image. Pixels that do not have a point can be filled with zeros. If the coordinates of the camera and the coordinates of the rider are already matched, a new two-dimensional three-channel image may be additionally acquired by converting the rider's data as described above. As such, as the preprocessing process is performed on the rider, the two-dimensional three-channel image 122 of the converted rider and the camera image 111 of the camera are passed through each deep neural network (for example, CNN). Can be. As shown in FIG. 4, the input part of the deep neural network passes a three-channel image of a rider and a camera image of a camera through independent CNNs, and then a feature map obtained at a network node after a certain point is converged. A new feature map may be constructed by fusing through the (GFU) 410. As the feature map is processed, the location and type of the object may be simultaneously determined 420. In this case, for example, the object may be detected through a single shot detector (SSD) method, and the object may be detected through various methods in addition to the SSD method.

Referring to FIG. 5, a diagram illustrating a structure of a GFU for fusion of a feature map 510 of a camera and a feature map 520 of a rider is illustrated. After determining the quality of each feature map, one or more feature maps of each feature map may be weighted, and then each feature map may be fused through the fusion network (GFU 410). Specifically, first, the feature map 510 of the camera and the feature map 520 of the rider are merged in parallel. Thereafter, the combined feature maps (primary fusion feature maps) pass through the CNN and sigmoid functions, each with a 3X3 kernel. Through this process, it is possible to determine the robustness of which of the data of the camera and the rider is more reliable data. At this time, as the feature maps (primary fusion feature maps) merged in parallel pass through the CNN and sigmoid function having a 3X3 kernel, the output data can be output by mapping to a value between 0 and 1 as the reliability of the data for each pixel. Can be. In other words, a CNN with a 3 × 3 kernel and an output through the sigmoid function can be derived per pixel with a value between 0 and 1 as the reliability of the data. This reliability may be multiplied by each feature map of the camera and the rider, and the multiplication feature map may be combined again in parallel. Then, the CNN with a 1 × 1 kernel is finally fused with the camera's feature map and the rider's feature map. Specifically, secondly fusion in parallel the value obtained by multiplying the camera's reliability map with the camera's feature map and the rider's reliability with the rider's feature map in parallel, and dividing the second fused feature map in the kernel of 1 × 1 size. The third feature may be fused through a neural network (eg, a CNN) to form a new feature map for the feature map of the camera and the feature map of the rider. Finally, the GFU coefficient, the CNN corresponding to each sensor, and the coefficient of the network extracting the detection result from the fused feature map are simultaneously learned. Accordingly, the object may be detected based on the new feature map.

In this case, when the sensor quality of the camera and the rider is deteriorated, a data augmentation technique is required to apply the deterioration of the quality to one of the camera and the rider in order to learn a function of assigning a weight appropriate thereto. For example, referring to FIG. 6, in order to evaluate the performance of the deep learning-based object recognition algorithm proposed in the embodiment, the performance of the camera image is reduced. FIG. 6 (a) is image data in which brightness is manipulated, FIG. 6 (b) is image data in which part is masked, FIG. 6 (c) is image data including noise, and FIG. 6 (d) is original image. Data. By performing operations such as partially obscuring image data, adjusting brightness, or removing one of the two data, additional data generation for robustness can be performed and the corresponding network can be learned. By lowering the quality of some of the data to learn the data, the weight of the network may be adaptively assigned according to the quality of the image.

In the case of sensor fusion through deep learning, the cognitive system combines a feature map obtained through CNN to obtain a new feature map, and additionally applies the acquired CNN layer to the sensor fusion Based object detection. In this case, since each feature map is combined with an appropriate weight in the fusion network (GFU), optimal sensor fusion is achieved by controlling the contribution to the degraded feature map. In addition, it is possible to achieve the fusion using the advantages of the data for the rider and the camera, it is possible to optimize the fusion even in the case of a degradation in one sensor data.

7 is a table for explaining a performance of a cognitive system according to an exemplary embodiment.

Deep learning techniques for fusion of camera and rider data can improve object recognition and detection performance. For example, using a KITTI benchmark data set, rider coordinates and camera coordinates are matched, and then the 3D point data of the rider sensor is converted into a 2D image, and then the 2D image of the rider sensor and the camera image are transferred to the CNN. You can enter each one. At this time, the quality of the sensor data is determined and fused through the fusion network (GFU) to recognize the object. In FIG. 7, the performance of the GFU is compared with the case where the GFU is included in the base network and the case where the GFU is not included. For example, to compare performance, the camera image may be empty, partially obscured, partially lightened, or mixed with noise. In addition, the rider image may be empty or partially obscured. Accordingly, as a result of the comparison, the network with the GFU in all items showed better performance, and especially, when the noise is mixed, it can be seen that the larger performance difference.

A cognitive system according to an embodiment may be applied to autonomous driving in which riders and camera sensors are expected to be used simultaneously in recent autonomous driving, and may be applied to various artificial intelligence fields that recognize environments or objects as well as autonomous driving.

Typical applications include autonomous vehicles, robotics and vehicle monitoring systems. Firstly, object recognition is the first technology to be involved in autonomous cars, and it determines the location of nearby objects and pedestrians to determine if there are any dangerous factors. It can prevent. It is also used to check road information needed for driving such as traffic lights and signs for driving assistance. Secondly, in robotics, object recognition helps the robot's visual activity like the human eye. Object recognition enables the robot to check the surroundings and objects and perform functions according to the purpose. Object recognition technology is also applicable to vehicle monitoring systems. For example, the parking management system may help parking by recognizing the type and vehicle number of the car entering the parking lot, and even checking the current empty space.

To apply this technology, there is a method of acquiring data in real time using a camera and a rider sensor, and performing an object recognition algorithm in an embedded system equipped with a graphic processor unit (GPU). To this end, data on various environments are secured in advance and the deep neural network structure is learned through this. The trained deep neural network is stored with optimized network coefficients and applied to the embedded system to perform the object recognition algorithm on the test data input in real time to obtain the result.

In addition, camera and rider sensor-based object recognition algorithms can be applied to autonomous driving or mobile robots and are expected to perform more complex functions such as object tracking and future prediction beyond recognition. For example, automated monitoring, traffic monitoring and vehicle navigation, or traffic light control based on traffic conditions on the road will also be possible. In addition, object recognition plays an important role in understanding the environment and can be applied to home appliances that combine IoT. As such, deep learning-based object recognition algorithm is one of the representative artificial intelligence technologies based on future technologies.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments are, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs). Can be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. It can be embodied in. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (15)

In the cognitive method performed by the cognitive system,
Inputting different data to each deep neural network to obtain each feature map;
Fusing the acquired respective feature maps through a fusion network; And
Detecting an object based on a new feature map fused through the fusion network;
Cognitive method, characterized in that. The method of claim 1,
Obtaining each feature map by inputting the different data to each deep neural network,
When the different data is data related to a rider and a camera, passing the two-dimensional three-channel image and the camera image of the camera, which have been converted as a preprocessing process is performed on the rider, through each CNN.
Cognitive method comprising a. The method of claim 2,
Obtaining each feature map by inputting the different data to each deep neural network,
The 3D point information obtained from the rider is mapped to the 2D image of the camera, and a preprocessing process of generating the coordinates of the 2D image by multiplying a matrix for converting the position information of the 3D point information from the rider coordinates to the camera coordinates. Steps to perform
Cognitive method comprising a. The method of claim 1,
Converging the obtained respective feature maps through a fusion network,
Weighting any one or more feature maps of each feature map according to the quality of each feature map, and then fusing each feature map through a fusion network
Cognitive method comprising a. The method of claim 4, wherein
Converging the obtained respective feature maps through a fusion network,
When each of the acquired feature maps is a feature map of a camera and a feature map of a rider, the feature map of the camera and the feature map of the rider are passed through the fusion network, and the feature maps are fused in parallel to form a new feature. Steps to Create a Map
Cognitive method comprising a. The method of claim 5,
Converging the obtained respective feature maps through a fusion network,
The feature map of the camera and the feature map of the rider are primarily fused in parallel, and the rider or the fused feature map is passed through a deep neural network having a plurality of 3X3 kernels and a plurality of sigmoid functions. Steps to determine the robustness of the camera
Cognitive method comprising a. The method of claim 6,
Converging the obtained respective feature maps through a fusion network,
Outputting the first fused feature map as a value between 0 and 1 for each pixel as reliability of data of the camera and the rider as a deep neural network having a plurality of 3X3 kernels and a sigmoid function are passed through
Cognitive method comprising a. The method of claim 7, wherein
Converging the obtained respective feature maps through a fusion network,
The second multiplication is performed again in parallel by multiplying the reliability of the camera by the feature map of the camera and the reliability of the rider by the feature map of the rider, and converting the second fused feature map into a 1x1 kernel. Deriving a new feature map for the feature map of the camera and the rider's feature map by performing the third fusion through a deep neural network
Cognitive method comprising a. The method of claim 1,
Detecting an object based on a new feature map fused through the fusion network,
Generating data for lowering the quality of some of the different data, and controlling to learn the degraded data;
Cognitive method comprising a. The method of claim 1,
Detecting an object based on a new feature map fused through the fusion network,
Simultaneously determining the deep neural network and the sensor fusion network to determine the location and type associated with the object by processing the new feature map.
Cognitive method comprising a. In the cognitive system,
A feature map acquisition unit for inputting different data into each deep neural network to obtain a first feature map and a second feature map;
A fusion unit configured to fuse the obtained first and second feature maps through a fusion network; And
A detector for detecting an object based on a new feature map fused through the fusion network
Cognitive system comprising a. The method of claim 11,
The feature map acquisition unit,
As the first data and the second data are input as the different data, the two-dimensional three-channel image and the image of the second data, which are converted as the preprocessing process is performed on the first data, are passed through the respective CNNs. Letting
Cognitive system, characterized in that. The method of claim 11,
The fusion unit,
Weighting at least one feature map of each feature map according to the quality of the first feature map and the second feature map, and then fusing each feature map through a fusion network.
Cognitive system, characterized in that. The method of claim 11,
The fusion unit,
The camera and the rider as the first and second feature maps are firstly fused in parallel, and the first and second feature maps are passed through a deep neural network having a plurality of 3X3 kernels and a sigmoid function. As the reliability of the data of
Cognitive system, characterized in that. The method of claim 14,
The fusion unit,
The second multiplication is performed again in parallel by multiplying the reliability of the camera by the feature map of the camera and the reliability of the rider by the feature map of the rider, and converting the second fused feature map into a 1x1 kernel. The third feature is fused through a deep neural network to derive a new feature map for the feature map of the camera and the feature map of the rider.
Cognitive system, characterized in that.

Images