KR102108951B1 - Deep learning-based object detection method and system utilizing global context feature of image - Google Patents

Abstract

An object detection method according to an embodiment may include: acquiring a feature map by passing image information through a first deep neural network; Passing the acquired feature map through a second deep neural network and a third deep neural network, respectively; Fusing each feature map obtained by passing through the second deep neural network and the third deep neural network through a convergence network; And detecting an object from the image information based on the fused feature map.

Description

DEEP LEARNING-BASED OBJECT DETECTION METHOD AND SYSTEM UTILIZING GLOBAL CONTEXT FEATURE OF IMAGE}

The following description relates to a technique for detecting an object using global context information of an image.

The object detector method refers to a technique for classifying the position and species of an object to be detected from a camera image. Object detection technology through image information is a field that has been studied for a long time in the field of computer vision. In recent years, as a deep learning technology using a deep neural network structure has been developed, a method of detecting an object with a very high level of accuracy by learning a high-dimensional feature value from a large amount of image data has been proposed. These technologies are actively applied to various fields such as autonomous driving, robotics, medical care, and security. In recent years, the vehicle is required to mount a pedestrian safety system, and it is also used in a technology to avoid danger by recognizing the location and condition of a pedestrian in a driver assistance system for safety. In the case of robots used in factories, it is also used as an important technique for learning the changes and characteristics of the factory environment by using image-based object recognition technology and ultimately maximizing the intelligence and processing performance of robots. In the medical imaging field, it is used as a means to assist the diagnosis of experts by using deep learning-based object recognition techniques to reduce simple labor for doctors. As such, with the development of cognitive technology in the field of imaging, the need for object detection technology through understanding of the environment and situation is increasing.

As an example, techniques for extracting key features of data and designing sophisticated techniques to obtain image feature values and to recognize or detect objects through statistical classifiers have been mainly used. For example, it is a technology that recognizes or detects an object by finding out features such as the structure or shape of a line of an image and comparing it with a template that is already known. Typical feature extraction technologies include scale invariant feature transform (SIFT) and histogram of oriented gradients (HOG). SIFT is a method of extracting feature vectors having directionality for local patches centered on each feature point after selecting feature points that are easily identified, such as corner points, in the image. Use expressive features. HOG is a method using a vector that divides the target region into cells of a certain size, obtains a histogram of the direction of the boundary pixels for each cell, and then uses these histogram bin values in a line, and uses boundary direction information. Perform template matching. In addition, since HOG uses silhouette information of an object, it is suitable for identifying an object having unique unique contour information without complicated internal patterns such as a person or a car. Since these technologies utilize only the known information of the object for the edge or shape in advance, the recognition performance is greatly improved when the feature value deviates from the predicted distribution when various illuminance changes, shape distortion, noise, and occlusion occur in the image data. There is a downside.

In order to overcome the limitations of the feature value-based object detection technique, the deep learning technique learns a method of expressing data, i.e., feature values, from a large amount of data using a structure of a deep neural network, a more generalized learning model. In this case, since a method of expressing data is obtained through various types of data, good performance can be obtained for various changes and malicious elements of image information. A conventional object detection method using a deep learning technique has an R-CNN technique. The R-CNN method receives an input from the camera sensor, first goes through a pre-processing network that proposes an area that can be an object candidate in the image, and then passes through the object classification network, which uses information about the area as an input of the deep neural network. It is a technique. In addition, the Fast R-CNN technique, which improves the shortcomings of R-CNN, uses the CNN network to find the feature values to reduce the computational complexity of the existing R-CNN technique, and then uses the CNN network for the object candidate region in the feature map stage of the middle node. Object detection is performed using both the area suggestion network that extracts information and the convolutional network that determines the type of object. Recently, these two networks are integrated into a single step detector (SSD) and deep learning based object detection called You only look once (YOLO), which combines the steps of finding the location of the object area and classifying the types of objects. Techniques have been proposed.

However, the feature map is extracted from the image using a convolutional neural network (CNN), and deep learning to find the bounding box and class information of the object using the local information corresponding to a specific object from the feature map. The conventional object detection technology based on has a limitation in utilizing only local information corresponding to a specific object because it determines the region and type of the object by matching the feature values corresponding to the object from the feature values obtained from the CNN. For example, in the case of a real person, global information, such as the surrounding environment or background, is very helpful in detecting an object. When the surrounding background is a road, it is very likely that the object is a traffic-related object, and when the surrounding is an indoor environment, it is most likely an object that we see indoors. Conventional deep learning-based object detection technology does not utilize such global context information, so there is a problem in that it does not actively utilize all available surrounding information.

References: KR 10-2016-0099289 (2016.08.22), KR 10-1836742 (2018.03.02), KR 10-2015-0050134 (2015.05.08)

We would like to propose a deep learning-based object detection method and system that uses global information on the surrounding environment of an object together with local information on a part of the object.

The object detection method performed by the object detection system includes: obtaining a feature map by passing image information through a first deep neural network; Passing the acquired feature map through a second deep neural network and a third deep neural network, respectively; Fusing each feature map obtained by passing through the second deep neural network and the third deep neural network through a convergence network; And detecting an object from the image information based on the fused feature map.

The step of passing the acquired feature map through the second deep neural network and the third deep neural network includes: a second deep neural network and global information for obtaining a feature map for expressing regional information on the acquired feature map. And passing through each of the third deep neural networks for obtaining a feature map expressing.

The second deep neural network is configured with a six-layer CNN structure to obtain feature values for expressing regional information, extracts a six-layer feature map from the CNN structure, detects an object, and detects the third dip The neural network is composed of a five-layer CNN structure to obtain a feature map for expressing global information, and an object can be detected using the feature map of one of the five layers in the CNN structure.

Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information, and the second deep neural network and the third The step of fusing each feature value obtained by passing through the deep neural network through the convergence network includes the type and location of the object as each feature map of the regional information passes through the convolution filter according to the location. To obtain the object information value, expand the feature values of the feature map of the global information, and connect with the embedding vector to pass through the complete connection layer to obtain the object information value including the type and location of the object, and And combining an object information value for local information and an object information value for the global information.

Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information, and the second deep neural network and the third The step of fusing each feature map obtained by passing through the deep neural network through a convergence network is a matrix that makes feature values corresponding to the location of the feature map of the regional information zero in the feature map of the global information. The method may include using information on the object as a feature map of the local information using a masking technique that multiplies by and using environment information other than the object as a feature map of the global information.

The step of fusing each feature value obtained by passing through the second deep neural network and the second deep neural network through a convergence network passes each feature map of the regional information through a convolution filter according to a location. Obtain the object information value including the type and location of the object according to the seam, expand the feature values of the feature map of the global information to which the masking technique is applied, and then pass the complete connection layer and obtain the value and the local information It may include the step of combining the object information value for.

Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information, and the image is based on the fused feature map. The step of detecting an object from the information may include detecting the object from the image information by combining the object information value for the local information and the object information value for the global information and passing it through a complete connection layer. have.

The object detection system includes: a feature map acquiring unit that acquires a feature map by passing image information through a first deep neural network; An input unit for passing the acquired feature map through a second deep neural network and a third deep neural network, respectively; A fusion unit that fuses each feature map obtained by passing through the second deep neural network and the third deep neural network through a fusion network; And a detection unit detecting an object from the image information based on the fused feature map.

The input unit may pass the acquired feature map through each of a second deep neural network for obtaining a feature map for expressing regional information and a third deep neural network for obtaining a feature map representing global information. have.

The second deep neural network is configured with a six-layer CNN structure to obtain feature values for expressing regional information, extracts a six-layer feature map from the CNN structure, detects an object, and detects the third dip The neural network is composed of a five-layer CNN structure to obtain a feature map for expressing global information, and an object can be detected using the feature map of one of the five layers in the CNN structure.

Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information, and the fusion unit is a feature map of the regional information As each passes through a convolution filter according to the location, an object information value including the type and location of the object is obtained, and the feature values of the feature map of the global information are expanded, and then connected with an embedding vector to the full connection layer. Upon passing, an object information value including the type and location of an object may be obtained, and an object information value for the local information and an object information value for the global information may be combined.

Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information, and the fusion unit features the global information The information on the object is used as a feature map of the regional information by using a masking technique of multiplying a matrix that makes feature values corresponding to the location of the feature map of the regional information zero, and the feature map of the global information It is possible to use environment information other than the above object.

The fusion unit acquires an object information value including the type and location of the object by passing the feature map of the regional information to the convolution filter according to the location, and of the feature map of the global information to which the masking technique is applied. After expanding the feature values, the value obtained by passing through the complete connection layer and the object information value for the local information may be combined.

Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information, and the detection unit is an object for the regional information. The object value can be detected from the image information by combining an information value and an object information value for the global information and passing it through a complete connection layer.

The object detection system according to an embodiment may improve detection performance by using both local and global information for image information.

The object detection system according to an embodiment configures a network for acquiring regional information where an object is located and a network for acquiring global information about the surrounding environment to which the object belongs, and each obtained by passing through the configured network By combining the feature maps through the convergence network, it is possible to perform effective object recognition based on the understanding of the surrounding environment for the object.

1 is a diagram for explaining a structure of a deep neural network performing object detection in an object detection system according to an embodiment.
2 is a diagram for explaining a structure of a deep neural network that fuses local information and global information using an embedding vector in an object detection system according to an embodiment.
3 is a diagram for explaining the structure of a deep neural network that fuses local information and global information using a masking technique in an object detection system according to an embodiment.
4 is a table for explaining the performance of detecting an object in the object detection system according to an embodiment.
5 is a block diagram illustrating the structure of an object detection system according to an embodiment.
6 is a flowchart illustrating a method of recognizing an object in an object detection system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a structure of a deep neural network performing object detection in an object detection system according to an embodiment.

In performing object detection using a deep neural network, the object detection system may improve image-based object detection performance by fusing global information including the surrounding environment with local information about the object. Accordingly, a structure of a deep neural network (eg, CNN) is used to extract local information and global information, and a part of the Single Shot Detector (SSD) method can be used as a structure for detecting an object from a feature map. have.

Referring to FIG. 1, a structure of a deep neural network for detecting an object by fusing global information and local information is illustrated. Image information of the camera may be input to a deep neural network (eg, CNN). Hereinafter, in the following description, a deep neural network in which image information is initially input will be described as a first deep neural network for convenience. As the image information passes through the first deep neural network, the output value may be divided into a second deep neural network and a third deep neural network and passed. For example, the RGB image information 110 of the camera may be passed through the VGG-16 CNN 101. The feature value as the image information is passed through the VGG-16 CNN 101 is divided into the regional CNN 102 and the global CNN 103 for passing the feature information map and the global information feature map, respectively. have. In the structure of the local CNN, the object map is detected by extracting the feature maps of the six layers (a), (b), (c), (d), (e), and (f), and in the structure of the global CNN, the total The feature map (g) of the 4th layer among the 5 layers will be used. Objects may be detected 120 by fusing each of the feature maps through the fusion network 200. As described above, a method for detecting an object by simultaneously using a local feature map and a global feature map will be described in FIGS. 2 and 3.

2 is a diagram for explaining a structure of a convergence network (deep neural network) in which local information and global information are fused using an embedding vector in an object detection system according to an embodiment. As an example, referring to FIG. 2, a method of fusing two different feature maps using an embedding vector of location information for regional information.

The feature map of the regional information may be passed through a convolution filter according to the location, such as the SSD method, to obtain an object information value including the type and location of the object corresponding to the regional information. In addition, the feature map of the global information expands the values of the feature map of the global information and connects with the embedding vector to pass through the fully-connected layer to obtain the object information value including the type and location of the object. can do. In this way, the object information value obtained from the feature map of the local information and the object information value (eg, score value) obtained from the feature map of the global information are concatenated, and then passed through the complete connection layer again. A final detection may be performed by obtaining a value (for example, a score value) for the object type and coordinates. Here, the embedding vector is a technique of expressing information about location coordinates as a vector of a specific dimension. In an embodiment, a feature map of local information may be used for the purpose of activating global information corresponding to the location of the convolution filter. . At this time, the embedding vector may also be learned together in the training process of the network, and the center coordinates in which the convolution filter is located may be obtained from the feature map of the local information and the embedding vector may be obtained based on the center coordinates.

FIG. 3 is a diagram for explaining a structure of a converged network (deep neural network) in which regional information and global information are fused using a masking technique in an object detection system according to an embodiment.

Referring to FIG. 3, in order to use only the information on the environment in addition to the object information, the feature map of the global information multiplies the matrix multiplying the matrix generating feature values corresponding to the location of the local information in the feature map of the global information ( Using the masking technique, regional and global information can be fused. At this time, when the masking technique detects an object using the feature map of the local information and the feature map of the global information, the feature map of the local information uses only the information about the object, and the feature map of the global information uses the object map Since the detection is performed using only environment information other than related information, it is a technique that can derive the effect of removing object-related information from the feature map of global information and maintaining only environment information.

The feature map of the regional information may be passed through a convolution filter according to the location, such as the SSD method, to obtain an object information value including the type and location of the object corresponding to the regional information.

In the feature map of the global information, multiply the matrix that generates the feature value corresponding to the location of the local information by 0, expand the values of the feature map of the global information, and then pass the complete connection layer to obtain the value and the local information. After concatenating the object information value (eg, score value) obtained from the feature map of, after passing through the complete connection layer again, the final detection of the object is performed.

4 is a table for explaining the performance of detecting an object in the object detection system according to an embodiment.

A table comparing the performance of the object detection technique proposed in the embodiment with the conventional object detection algorithm. In the table, the performance of the detector is expressed as mean average precision (mAP), and PASCAL VOC is used for training data. As shown in the table, it can be confirmed that the proposed technology that utilizes global context information shows a better detection performance of an average of 1% or more compared to the existing Single Shot Detector (SSD) method that does not utilize global information.

Conventional deep learning-based object recognition technology performs object detection using only regional information in a feature map of a deep neural network, but the method proposed in the embodiment is a network trained to acquire regional information from a network node after the middle stage. To acquire global information, object detection is performed using information about the surrounding background by using a trained network together.

In addition, the conventional deep learning-based object detection network performs training through supervised learning to perform detection corresponding to the corresponding label using the data with the label of the object, whereas it is used in the technique proposed in the embodiment The global information is data without label information, but the data corresponding to the global information utilizes the interaction with the local information data. Accordingly, when using the regional information of the feature map, the interaction between the local information and the global information can be maximized by using the embedding vector for the location of the evolution filter. Accordingly, the cognitive detection performance of the object may be improved by using additional information about the environment compared to the object detection method using only local information.

The object detection system according to an embodiment may improve detection performance by using both local and global information when performing object detection using a deep learning technique. In the proposed method, a network for acquiring regional information with an object and a network for acquiring global information about the surrounding environment to which the object belongs are configured in a deep neural network using camera sensor data as input. By effectively fusion, it is possible to perform effective object recognition based on an understanding of the surrounding environment of an object.

The object detection system according to an embodiment will be used to perform an object detection and an understanding of the surrounding environment to which an object belongs when performing object detection using camera sensor data in a recent smart home environment or an autonomous driving environment. It is expected, and the proposed method can provide a solution to effectively understand these objects and their surroundings. In addition, it can be applied not only to smart homes or autonomous driving, but also to various artificial intelligence technologies that recognize environments or objects.

5 is a block diagram illustrating a structure of an object detection system according to an embodiment, and FIG. 6 is a flowchart illustrating a method of recognizing an object in an object detection system according to an embodiment.

The object detection system 500 may include a feature map acquisition unit 510, an input unit 520, a fusion unit 530, and a detection unit 540. These components may be representations of different functions performed by a processor according to a control instruction provided by program code stored in the object detection system 500. Components may control the object detection system 500 to perform steps 610 to 640 included in the object detection method of FIG. 6. At this time, the components may be implemented to execute instructions according to the code of the operating system included in the memory and the code of at least one program.

The processor of the object detection system 500 may load program code stored in a file of a program for an object detection method into a memory. For example, when a program is executed in the object detection system 500, the processor may control the object detection system to load program code from a file of the program into a memory under control of an operating system. At this time, each of the feature map acquisition unit 510, the input unit 520, the fusion unit 530, and the detection unit 540 included in the processor and the processor executes an instruction of a corresponding part of the program code loaded in the memory, and then executes the subsequent steps. It may be different functional representations of the processor for executing the fields (610 to 640).

In step 610, the feature map acquiring unit 510 may acquire the feature map by passing the image information through the first deep neural network.

In step 620, the input unit 520 may pass the acquired feature map to the second deep neural network and the third deep neural network, respectively. The input unit 520 may pass the acquired feature map to each of the second deep neural network for obtaining a feature map for representing regional information and the third deep neural network for obtaining a feature map representing global information. have. At this time, the second deep neural network is composed of a six-layer CNN structure to obtain a feature value for expressing regional information, and an object can be detected by extracting a six-layer feature map from the CNN structure. The deep neural network is composed of a five-layer CNN structure to obtain a feature map for expressing global information, and an object can be detected using a feature map of one of the five layers in the CNN structure.

In step 630, the fusion unit 530 may fuse each feature map obtained by passing through the second deep neural network and the third deep neural network through the fusion network. The fusion unit 530 locates the feature map of the local information if each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of the local information and a feature map of the global information. According to each passing through the convolution filter, the object information value including the type and location of the object is obtained, and the feature values of the feature map of the global information are expanded and connected to the embedding vector to pass through the complete connection layer. Accordingly, an object information value including an object type and location may be obtained, and an object information value for local information and an object information value for global information may be combined. The fusion unit 530 uses the information on the object as a feature map of the local information by using a masking technique that multiplies the feature map corresponding to the location of the feature map of the local information from the feature map of the global information to 0. , Environment information other than objects can be used as a feature map of global information. The fusion unit 530 obtains an object information value including the type and location of the object as each feature map of the regional information passes through the convolution filter according to the location, and of the feature map of the global information to which the masking technique is applied. After unfolding the feature values, the values obtained by passing through the complete connection layer and the object information values for the local information can be combined.

In operation 640, the detector 540 may detect an object from image information based on the fused feature map. The detection unit 540 may combine an object information value for local information and an object information value for global information and detect an object from image information based on the obtained object information value by passing it through the complete connection layer.

The object detection technique using global context information according to an embodiment may be applied to various fields. Typical applications include smart home cameras, autonomous vehicles, and robotics systems. First, object detection in a smart home camera is the most important technology to be performed, and suggests or functions suitable for the purpose based on information on the types and locations of objects in the indoor environment where the smart home system is implemented. Make it possible. Second, in self-driving cars, it plays a role in determining where other vehicles, objects, and pedestrians in the vicinity exist, and whether or not there is a risk factor, which enables safe driving and prevents accidents. In addition, the object detection technology acquires data using the camera sensor mounted on the robot in the field of robotics and recognizes the location information, size, and type of obstacles or obstacles in the robot's movement to perform the task appropriate for the purpose. Provides the information necessary to perform. In order to apply this technology, there is a method of acquiring data in real time using a camera sensor and performing an object recognition algorithm in an embedded system equipped with a graphic processor unit (GPU). To do this, in advance, secure data on various environments and learn the deep neural network structure through this. The trained deep neural network is stored as an optimized network coefficient, and it is applied to an embedded system to perform object recognition algorithm on test data input in real time to obtain the result.

The object detection technology using global information based on deep learning can be applied to smart home cameras, autonomous driving, mobile robots, etc. Based on this technology, in the future, it will track objects beyond recognition and grasp the relationship between objects and understand the environment. It is expected to perform more complex functions such as predicting the future through understanding. For example, based on the understanding of the environment using global information in a smart home environment, it is possible to predict the future position of a moving object by predicting and preventing a dangerous situation and understanding relationships between objects using only a camera sensor. In addition, in an autonomous driving environment, it can be used for advanced tasks such as automated monitoring and traffic monitoring. As described above, an object detection algorithm using global information is a technology that is based on future technologies and can be said to be one of the representative artificial intelligence technologies.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

In the object detection method performed by the object detection system,
The object detection system comprising: acquiring a feature map by passing image information through a first deep neural network;
In the object detection system, Passing the obtained feature map through a second deep neural network and a third deep neural network, respectively;
In the object detection system, Fusing each feature map obtained by passing through the second deep neural network and the third deep neural network through a fusion network; And
In the object detection system, detecting an object from the image information based on the fused feature map
Including,
Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information,
In the object detection system, the step of fusing each feature value obtained by passing through the second deep neural network and the third deep neural network through a convergence network may include:
The object information value including the type and location of the object is obtained by passing the feature map of the regional information to the convolution filter according to the location, and after expanding the feature values of the feature map of the global information, the embedding vector and By connecting and passing through the complete connection layer, obtaining an object information value including the type and location of the object, and combining the object information value for the local information and the object information value for the global information
Object detection method comprising a. According to claim 1,
In the object detection system, the step of passing the acquired feature map through the second deep neural network and the third deep neural network,
Passing the acquired feature map through each of a second deep neural network for obtaining a feature map for representing local information and a third deep neural network for obtaining a feature map for representing global information.
Object detection method comprising a. According to claim 1,
The second deep neural network is configured with a six-layer CNN structure to obtain feature values for representing regional information, and an object is detected by extracting a six-layer feature map from the CNN structure,
The third deep neural network is configured with a 5-layer CNN structure to obtain a feature map for expressing global information, and detecting an object by utilizing a feature map of one of the 5 layers in the CNN structure , Object detection method. delete In the object detection method performed by the object detection system,
The object detection system comprising: acquiring a feature map by passing image information through a first deep neural network;
In the object detection system, Passing the obtained feature map through a second deep neural network and a third deep neural network, respectively;
In the object detection system, Fusing each feature map obtained by passing through the second deep neural network and the third deep neural network through a fusion network; And
In the object detection system, detecting an object from the image information based on the fused feature map
Including,
Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information,
In the object detection system, the step of fusing each feature map obtained by passing through the second deep neural network and the third deep neural network through a convergence network,
In the object detection system, the object is mapped to the object as a feature map of the local information by using a masking technique of multiplying a feature matrix corresponding to the location of the feature map of the local information from a feature map of the global information to 0. Using information about the environment and using environment information other than the object as a feature map of the global information
Object detection method comprising a. The method of claim 5,
In the object detection system, the step of fusing each feature value obtained by passing through the second deep neural network and the third deep neural network through a convergence network may include:
As the feature map of the regional information is passed through the convolution filter according to the location, an object information value including the type and location of the object is obtained, and the feature values of the feature map of the global information to which the masking technique is applied are expanded. Then combining the values obtained by passing through the complete connection layer and the object information values for the local information
Object detection method comprising a. The method of claim 1 or 5,
Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information,
In the object detection system, detecting an object from the image information based on the fused feature map,
Combining the object information value for the local information and the object information value for the global information and passing it through a complete connection layer to detect the object from the image information
Object detection method comprising a. In the object detection system,
A feature map acquiring unit for acquiring a feature map by passing the image information through the first deep neural network;
An input unit for passing the acquired feature map through a second deep neural network and a third deep neural network, respectively;
A fusion unit that fuses each feature map obtained by passing through the second deep neural network and the third deep neural network through a fusion network; And
A detection unit that detects an object from the image information based on the fused feature map
Including,
Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information,
The fusion unit,
The object information value including the type and location of the object is obtained by passing the feature map of the regional information to the convolution filter according to the location, and after expanding the feature values of the feature map of the global information, the embedding vector and By connecting and passing through the complete connection layer, an object information value including the type and location of the object is obtained, and the object information value for the local information and the object information value for the global information are combined.
Object detection system. The method of claim 8,
The input unit,
Passing the acquired feature map through each of the second deep neural network for obtaining a feature map for expressing regional information and the third deep neural network for obtaining a feature map for expressing global information.
Object detection system, characterized in that. The method of claim 8,
The second deep neural network is configured with a six-layer CNN structure to obtain feature values for representing regional information, and an object is detected by extracting a six-layer feature map from the CNN structure,
The third deep neural network is configured with a 5-layer CNN structure to obtain a feature map for expressing global information, and detecting an object by utilizing a feature map of one of the 5 layers in the CNN structure
Object detection system, characterized in that. delete In the object detection system,
A feature map acquiring unit for acquiring a feature map by passing the image information through the first deep neural network;
An input unit for passing the acquired feature map through a second deep neural network and a third deep neural network, respectively;
A fusion unit that fuses each feature map obtained by passing through the second deep neural network and the third deep neural network through a fusion network; And
A detection unit that detects an object from the image information based on the fused feature map
Including,
Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information,
The fusion unit,
In the feature map of the global information, the information on the object is used as a feature map of the regional information by using a masking technique that multiplies a matrix that makes feature values corresponding to the location of the feature map of the regional information equal to 0, and the Using environment information other than the above object as a feature map of global information
Object detection system, characterized in that. The method of claim 12,
The fusion unit,
As the feature map of the regional information is passed through the convolution filter according to the location, an object information value including the type and location of the object is obtained, and the feature values of the feature map of the global information to which the masking technique is applied are expanded. After that, the value obtained by passing through the complete connection layer is combined with the object information value for the local information.
Object detection system, characterized in that. The method of claim 8 or 12,
Each feature map obtained by passing through the second deep neural network and the third deep neural network is a feature map of regional information and a feature map of global information,
The detection unit,
The object information value for the local information and the object information value for the global information are combined and passed through a complete connection layer to detect the object from the image information.
Object detection system, characterized in that.

Images