KR102314520B1 - Apparatus and method for detecting object - Google Patents

Abstract

The present invention relates to a deep learning-based real-time object detection apparatus and method specialized for extracting vehicle information as an object in order to detect a parked vehicle from an image obtained through a fisheye lens camera installed on the ceiling of a parking lot and track a driving vehicle. it's about
The object detection apparatus according to an embodiment of the present invention extracts features of an image by applying a deep learning-based convolutional neural network model to an image captured by a plurality of fisheye lens cameras provided on the ceiling of a parking lot. A multi-preprocessing unit that merges features extracted from at least one of the uppermost layer, the next higher layer, and the next higher layer of the feature extraction unit, and receives the merged features from the multiple pre-processing unit to display one or more objects as a bounding box, and the center of the bounding box It includes a multi-classifier for estimating coordinates, object width, object height, reliability of an object included in the bounding box, probability that an object exists in a cell is a specific class, and the direction of the object.

Description

Object detection apparatus and method {APPARATUS AND METHOD FOR DETECTING OBJECT}

The present invention relates to a deep learning-based real-time object detection apparatus and method specialized for extracting vehicle information as an object in order to detect a parked vehicle from an image obtained through a fisheye lens camera installed on the ceiling of a parking lot and track a driving vehicle. it's about

Convolutional neural network-based object recognition technology needs to extract rich features for recognizing various objects in order to recognize various objects in various environments. In order to extract rich features, the number of convolution layers must be very large, and the number of filters used in each convolution layer must also be large. This has a problem that requires a large amount of computation.

The above-mentioned background art is technical information possessed by the inventor for derivation of the present invention or acquired in the process of derivation of the present invention, and cannot necessarily be said to be a known technique disclosed to the general public prior to the filing of the present invention.

Domestic Patent Publication No. 2018-0136720

The present invention has been devised to solve the above-described problems and/or limitations, and an object of the present invention according to one aspect is to detect a parked vehicle from an image obtained through a fisheye lens camera installed on the ceiling of a parking lot and drive a vehicle In order to track the vehicle, the goal is to achieve a high detection rate with a small amount of computation by developing a deep learning-based real-time object detection technology specialized in extracting vehicle information as an object.

The object detection apparatus according to an embodiment of the present invention extracts features of the image by applying a deep learning-based convolutional neural network model to an image captured by a plurality of fisheye lens cameras provided on the ceiling of a parking lot. extraction unit; a multiple pre-processing unit merging the features extracted from at least one of the uppermost layer, the next-level layer, and the next-level layer of the feature extraction unit; and receiving the merged features from the multi-preprocessor to represent one or more objects as a bounding box, the central coordinates of the bounding box, the width of the object, the height of the object, the reliability of the object included in the bounding box, and the cell. It may include; a multi-classifier for estimating the probability that the object to be present is a specific class and the moving direction of the object.

The feature extraction unit includes 19 convolution layers that perform filtering by continuously performing 3×3 and 1×1 convolutions on the image output from the fisheye lens camera, and a 2×2 filter. It may include darknet-19, which includes 5 max pooling layers that perform image downsampling.

The multi-preprocessor includes 1024 features of the corresponding cell output from the 19th convolutional layer of the uppermost layer among the feature extractors including 19 convolutional layers and 5 maximal pooling layers, and the second layer corresponding to the uppermost cell. a first pre-processing unit that merges 256 features extracted from 4 cells of ; and a second preprocessing unit for merging 1024 features of the corresponding cell output from the 13th convolutional layer of the next higher layer among the feature extracting unit and 256 features extracted from 4 cells of the next higher layer corresponding to the next higher layer cell. may include.

The multi-classifier, from 1024 features obtained by passing 1024 1×1×1280 synthetic product filters for 1280 features merged by the first preprocessor, the central coordinates (x,y) of the bounding box, and the area of the object (w), the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) of the object existing in the cell, and the moving direction (θ) of the object are estimated a first classification unit; and the center coordinates (x,y) of the bounding box, the area of the object (w), A second classification for estimating the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) of the object existing in the cell being of a specific class, and the moving direction (θ) of the object may include;

The apparatus may further include a non-maximum suppression (NMS) processing unit that removes a portion where multiple bounding boxes overlap the object from the estimation result of the first classifier and the estimation result of the second classifier.

In the object detection method according to an embodiment of the present invention, a deep learning-based convolutional neural network model is applied to an image captured by a plurality of fisheye lens cameras provided on the ceiling of a parking lot by a feature extraction unit to apply the image extracting the features of merging the features extracted from one or more of the uppermost layer, the next-level layer, and the next-order layer by the multiple preprocessor; and by the multi-classifier, receiving the merged features from the multi-preprocessing unit and representing one or more objects as a bounding box, including the central coordinates of the bounding box, the width of the object, the height of the object, and the bounding box. It may include; estimating the reliability of the object, the probability that the object existing in the cell is a specific class, and the moving direction of the object.

In the step of extracting the feature, 19 convolution layers that perform filtering by performing continuous 3×3 and 1×1 convolution on the image output from the fisheye lens camera, and a 2×2 filter and extracting the feature using darknet-19 including five max pooling layers for downsampling the image.

In the merging step, 1024 features of the cell output from the 19th convolutional layer of the uppermost layer among the feature extraction unit including 19 convolutional layers and 5 maximal pooling layers by the first preprocessor; merging 256 features extracted from 4 cells of the second layer corresponding to the uppermost cell; and by the second pre-processing unit, 1024 features of the corresponding cell output from the 13th convolutional layer of the second-order layer among the feature extraction unit and 256 features extracted from 4 cells of the next-order layer corresponding to the second-order cell merging; may include.

In the estimating step, the center coordinates (x) of the bounding box from 1024 features obtained by passing the 1280 features merged by the first preprocessor through 1024 1×1×1280 synthetic product filters by the first classification unit , y), the width (w) of the object, the height (h) of the object, the reliability (C) of the object included in the bounding box, and the probability (Pi) that the object exists in the cell is a specific class and the object estimating a moving direction (θ) of and the center coordinate (x,y) of the bounding box from 1024 features obtained by passing 1024 1×1×1280 synthetic product filters for 1280 features merged by the second preprocessing unit by the second classification unit, The width (w) of the object, the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) of the object existing in the cell being a specific class, and the moving direction (θ) of the object ) estimating; may include.

The method may further include: removing, by a non-maximum suppression (NMS) processing unit, a portion where multiple bounding boxes overlap the object from the estimation result of the first classification unit and the estimation result of the second classification unit; have.

In addition to this, other methods for implementing the present invention, other systems, and computer programs for executing the methods may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to embodiments, a high detection rate can be achieved with a small amount of computation by developing a deep learning-based real-time object detection technology specialized for recognizing a vehicle as an object from an image obtained through a fisheye lens camera installed on the ceiling of a parking lot. In particular, similar detection performance can be achieved with about 70% of the computational amount of YOLOv3, which has superior performance compared to the computational amount among previously announced detection devices.

In addition, it is possible to increase the reliability of vehicle tracking in the parking lot by recognizing the traveling direction of the vehicle.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating performance versus computational amount of various object detectors based on a convolutional neural network.
2 is a diagram schematically illustrating a configuration of an object detection apparatus according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating image quality improvement performed by an image processing unit of the object detection apparatus of FIG. 2 .
4 and 5 are diagrams for schematically explaining a detailed configuration of an object detection unit in the object detection apparatus of FIG. 2 .
FIG. 6 is a diagram illustrating an object detection result in the object detection apparatus of FIG. 2 .
7 is a flowchart illustrating an object detection method according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will become apparent with reference to the detailed description in conjunction with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention, and to fully inform those of ordinary skill in the art to the scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals, and overlapping descriptions thereof are omitted. decide to do

With the development of deep learning learning techniques and the ability to collect large amounts of image data, the technology for recognizing objects in images has achieved high recognition performance. Among the deep learning-based object detection methods, the Faster R-CNN method proposed by Ren et al. is a method that shows fast detection speed and high performance. In this method, features are extracted with a pyramid-structured convolutional neural network, and k anchor boxes are moved in the image to determine whether an object exists in the anchor box. At this time, the existence of the object is determined by features extracted from the neural network in the anchor box area. Finally, the size and position of the region including the object, the reliability of the object existing in the region, and the probability that the object is a feature type are calculated. However, this method has a problem in that a large amount of computation is required to be implemented in real time.

Rendman et al., instead of finding the region of an object while moving the anchor box, use the Bro bounding box in each cell of the top layer (resolution: 7×7) of the convolutional neural network to determine the position and size of the object region and the reliability that the bounding box contains the object. , the probability that the included object is a specific type was calculated as a fully connected network. This method is called YOLO (you only look once) because object detection is performed only once in a fixed 7×7×b bounding box instead of moving the anchor box. This method requires very little computation and real-time object detection is possible. However, there is a problem in that the detection performance of small objects is low.

Liu et al. proposed a method that can effectively detect even small objects. This method is called SSD (single-shot detector) because it processes the position, size, and probability of each type of object all at once in the bounding box. This method implemented a multiscale detector that detects objects in various layers from the high-resolution layer to the lowest-resolution layer of the convolutional neural network. This method shows a high detection rate with a computational amount similar to that of YOLO.

Redmon et al. constructed a convolutional neural network for feature detection with 19 convolution layers and 5 max pooling layers consisting of continuous 3×3 and 1×1 convolutions, and the top layer The resolution was increased to 13×13, and this neural network was named darknet-19. As a classifier for object detection, we proposed YOLOv2 consisting of a convolutional neural network (CNN) instead of a fully connected network. This method realized better detection performance than SSD with less computation than YOLO.

He et al. proposed a convolutional neural network composed of very deep layers that can extract rich features required for detecting thousands of objects, which is called ResNet. This neural network has a structure that can learn very deep layers stably and shows high performance in object detection and recognition. However, there is a problem in that the amount of computation increases as the layer becomes deeper.

Lin et al. found that the number of object samples is very small compared to the number of background samples in learning a neural network in a method of detecting an object using an anchor box, and thus the performance of the learned detector is deteriorated. method was presented. This is called RetinaNet.

Redman et al. used 53 convolutional layers in a mixed form of darknet-19 and ResNet to extract richer features, which they call darknet-53. Darknet-53 uses more computation than darknet-19 and less computation than ResNet. Redman et al. extracted features with darknet-53 and created a classifier for object detection in three layers: top layer, second layer, and next layer to detect objects of various sizes. This method is called YOLOv3.

1 is a diagram illustrating performance versus computational amount of various object detectors based on a convolutional neural network according to the prior art. Referring to FIG. 1 , it is shown that YOLOv3 is significantly superior to other methods in detection performance compared to the amount of computation.

Such deep learning-based object recognition technology has a limitation in that its performance is lowered when it tries to recognize various objects in various environments. However, if the object to be recognized is specialized, the viewpoint and type of a camera that captures the object, and the surrounding environment are specialized, object detection with a high detection rate is possible. In this embodiment, we propose a deep learning-based real-time object detection technology specialized for detecting a vehicle as an object in an image obtained through a fisheye lens camera installed on the ceiling of an underground parking lot.

2 is a diagram schematically illustrating a configuration of an object detection apparatus according to an embodiment of the present invention. Referring to FIG. 2 , the object detecting apparatus 1 may include an image capturing unit 100 , an image processing unit 200 , an object detecting unit 300 , and a display unit 400 .

The image photographing unit 100 may photograph an image through N fisheye lens cameras 100_1 to 100_N (eg, 16) installed on the ceiling of the indoor parking lot. In the present embodiment, each of the fisheye lens cameras 100_1 to 100_N may capture an image by condensing incident light within a viewing angle range of 150 degrees or more and converting it into an electrical signal.

The image processing unit 200 may improve the image quality of the image captured by the image capturing unit 100 . Even if the lighting is evenly arranged inside the indoor parking lot, the vehicle parked in the corner of the building is dark in lighting, so it is difficult to recognize the outline of the rear of the object as shown in FIG. 3(a). When an image including an object is input to the convolutional neural network-based object detector 300 , the edge of the object outline is emphasized when an output value is visualized in the intermediate layer of the neural network. That is, in order to improve the object detection rate, it is necessary to improve the image quality to make the outline of the object clear. Of course, when learning the object detector, a data augmentation method that changes the intensity, hue, etc. of the input image is used, but in general, if the change range of the object to be detected is large, the performance of object detection is deteriorated.

In this embodiment, the image processing unit 200 applies gamma correction and edge enhancement to improve the quality of the input image. Gamma correction is expressed by Equation (1).

은 입력 영상의 밝기,

는 입력 영상의 밝기 범위,

는 감마 보정된 영상의 밝기,

는 보정된 영상의 밝기 범위를 각각 나타내며, R,G,B 각각 독립적으로 보정한다.in Equation 1

is the brightness of the input image,

is the brightness range of the input image,

is the brightness of the gamma-corrected image,

represents the brightness range of the corrected image, respectively, and each of R, G, and B is independently corrected.

In improving the edge, it is necessary to make a small improvement in a part with a sharp edge, and to increase the improvement in a part with a weak edge. The edge enhancement method applied in this embodiment is as shown in Equation (2).

는 지역 평균을 나타내고,

는 지역 표준편차를 나타낸다.in Equation 2

represents the regional mean,

is the regional standard deviation.

An image whose image quality is improved as a result of gamma correction and edge enhancement in the image processing unit 200 is input to the object detection unit 300 .

The object detection unit 300 extracts features of the image through convolutional neural network-based filtering and downsampling on the image (input image) received from the image processing unit 200 , and extracts features from the extracted image. Information such as the presence/absence of an object, the location of the object, the size of the object, and the moving direction of the object can be detected.

The display unit 400 displays the object detection result detected by the object detection unit 300 , for example, FIG. 6 .

4 and 5 are diagrams for schematically explaining a detailed configuration of an object detection unit in the object detection apparatus of FIG. 2 .

Among the existing object detection devices, YOLOv3 shows that the detection performance compared to the amount of computation is significantly superior to that of other methods. On the other hand, YOLOv2, which requires less computation than YOLOv3, has much lower performance than YOLOv3 in detecting small objects such as humans, but approaches YOLOv3 in performance when detecting large objects such as buses.

In this embodiment, the detector is specialized for recognizing only the vehicle as an object in the image acquired through the fisheye lens cameras (100_1 to 100_N) installed on the ceiling of the parking lot. Consider YOLOv2 without YOLOv2. The features extracted from Darknet-19 are not as rich as those extracted from darknet-53, but it is sufficient to detect only the vehicle as an object.

When a vehicle on the lower parking surface is photographed with a fisheye lens camera (100_1 to 100_N) installed on the ceiling of a parking lot, the vehicle placed in the center of the image is large in size, but the vehicle placed in the outskirts of the image is small in size. In order to detect vehicles of various sizes, it is necessary to detect objects on multiple scales as in YOLOv3. Therefore, the detector proposed in this embodiment has a structure for detecting objects at multiple scales based on the darknet-19 convolutional neural network.

In this embodiment, the fisheye lens cameras 100_1 to 100_N are installed on the ceiling of the parking lot, and the object (for example, a driving vehicle) moving under the fisheye lens cameras 100_1 to 100_N is photographed and the moving direction of the object. information can be obtained. Since the existing methods use images obtained by photographing the front/back/left/right of the object next to the object, it is not possible to obtain the moving direction information of the object from the image as in the present embodiment.

4 and 5 , the object detection unit 300 may include a feature extractor 310 , a multi-preprocessor 320 , a multi-classifier 330 , and an NMS processor 340 .

The feature extractor 310 may extract features from the input image through filtering and downsampling. The feature extraction unit 310 extracts image features of an object such as, for example, an outer edge of a vehicle as an object, an outer corner of a vehicle, etc. , height, direction, and width can be estimated.

As shown in FIG. 5 , the feature extracting unit 310 applies the known darknet-19, and consists of continuous 3×3 and 1×1 convolutions and includes 19 convolutional layers that perform filtering and May include 5 max pooling layers that perform image downsampling, apply batch normalization to the input of each convolutional layer, and Leaky ReLU (corrective linear unit) as an activation function , a rectified linear unit) is applied.

In addition, the 6th-8th convolutional layer set out of 19 convolutional layers is named the after next higher layer, and the 9th-13th convolutional layer set is named the next higher layer (the next higher layer, the next lower resolution). layer), and the 14-19th convolutional layer set may be named the highest layer (the lowest resolution layer). The input image of the feature extraction unit 310 is composed of 416x416 cells, the second-order layer consists of 52x52 cells, the second-order layer consists of 26x26 cells, and the uppermost layer consists of 13x13 cells.

The first convolutional layer of the feature extractor 310 is convolutional with respect to an input image having a resolution of 416×416 using 32 filters of 3×3 size at 1-pixel intervals. The features output from the first convolutional layer are processed by the first maximal pooling layer containing filters of size 2Х2 applied at 2-pixel intervals after batch normalization and Leaky ReLU, downsampled to a size of 208Х208, and then the second It is input to the convolutional layer.

The second convolutional layer is convolved at 1-pixel intervals using 64 filters of 3×3 size with respect to an image having a resolution of 208×208 down-sampled by the first maximum pooling layer. The features output from the second convolutional layer are processed by a second maximal pooling layer containing filters of size 2Х2 applied at 2-pixel intervals after batch normalization and Leaky ReLU, downsampled to a size of 104Х104, and then the third It is input to the convolutional layer.

The third convolutional layer is convolved at 1-pixel intervals using 128 filters of 3×3 size for an image with a resolution of 104×104 down-sampled by the second maximum pooling layer. The features output from the third convolutional layer are input to the fourth convolutional layer after undergoing batch normalization and Leaky ReLU. Through this process, the next upper layer consists of 52×52 cells, the second layer consists of 26×26 cells, and the uppermost layer consists of 13×13 cells.

The multi-preprocessor 320 merges the features of the next-order layer and the uppermost layer of the feature extraction section 310 and outputs them to the multi-classifier 330 , and merges the features of the second-order layer and the next layer to the multi-classifier 330 . do.

The multi-classifier 330 receives the merged features from the multi-preprocessor 320 and displays the detected objects as a bounding box, the center coordinates of the bounding box (x,y), and the area of the object (w) , the height of the object (h), the reliability (C) of the object included in the bounding box, the moving direction (θ) of the object, and the probability (Pi) of the object existing in the cell being of a specific class are estimated.

In this embodiment, the multiple preprocessor 320 includes a first preprocessor 321 and a second preprocessor 322 , and the multiple classifier 330 includes a first classifier 331 and a second classifier ( 332). Also, the output of the first preprocessor 321 is input to the first classifier 331, and the first classifier 331 obtains 6 object detection results from each of the 13×13 cells in the uppermost layer as in YOLOv2. Dog information, that is, x, y, w, h, C, θ, Pi can be detected. The output of the second preprocessor 322 is input to the second classifier 332, and the second classifier 332 provides six pieces of information, that is, x, y, w, h, C, θ, and Pi can be detected.

Referring to FIG. 5 , the first preprocessing unit 321 includes 1024 features of the corresponding cell output from the 19th convolutional layer of the uppermost layer and 256 features extracted from 4 cells of the second layer corresponding to the uppermost cell (4×64). ) are merged and output to the first classification unit 331 . The first classification unit 331 passes the 1280 features merged by the first preprocessor 321 through 1024 1×1×1280 synthetic product filters, and the center coordinates (x, y) of the bounding box from 1024 features , the width (w) of the object, the height (h) of the object, the reliability (C) of the object included in the bounding box, the moving direction (θ) of the object, and the probability (Pi) of the object existing in the cell are estimated to be of a specific class .

The second pre-processing unit 322 extracts 1024 features of the corresponding cell output from the 13th convolutional layer of the second-order layer and 256 (4×64) features extracted from 4 cells of the second-order layer corresponding to the second-order cell. They are merged and output to the second classification unit 332 . The second classification unit 332 passes the 1280 features merged by the second pre-processor 322 through 1024 1×1×1280 synthetic product filters, and from 1024 features obtained by the central coordinates (x, y) of the bounding box. , the width (w) of the object, the height (h) of the object, the reliability (C) of the object included in the bounding box, the moving direction (θ) of the object, and the probability (Pi) of the object existing in the cell are estimated to be of a specific class .

The non-maximum suppression (NMS) processing unit 340 performs an overlapping portion (eg, one car) from the class probability Pi that the object exists in the bounding box detected by the first classification unit 331 and the second classification unit 332 . (as in the case where multiple bounding boxes are drawn on The NMS processing unit 340 uses a method of leaving the current pixel as it is if it is the maximum value compared with the surrounding pixels based on the current pixel, and removing it if it is not (non-maximum). As shown in FIG. 6 , the result of removing the overlapping portion by the non-maximum suppression (NMS) processing unit 340 is output to the display unit 400 .

In YOLOv3, objects are detected with a classifier in the top layer, the next layer, and the next layer. The object detector of each layer up-samples the features extracted from the top layer, merges them, and passes the convolution filter to select the features to be used in the classifier extract Since the features of the top layer are features extracted from a wide area, the background outside the limited vehicle area may affect the feature values. Of course, it is possible to select only the necessary parts from the features extracted from the uppermost layer in the learning process, but in this embodiment, the features obtained from the uppermost layer were not used in the classifier of the next higher layer because it was judged that there were more disadvantages than benefits. The validity of this approach in the proposed method is verified through experiments.

In addition, since the size of the vehicle as an object to be detected is greater than or equal to a certain size, in the present embodiment, unlike YOLOv3, the object is detected only in two floors, the uppermost floor and the uppermost floor.

In the convolutional neural network, the function used in YOLOv2 is applied as a loss function to be minimized during training, and Equation 3 is shown.

는 i번째 셀의 j번째 테두리 상자에 포함된 객체의 신뢰도를 나타내고,

는 i번째 셀에 존재할 객체가 특정 클래스일 확률을 나타내고,

는 객체의 진행방향을 나타낸다.

는 i번째 셀의 j번째 테두리 상자에 객체가 존재하면 1, 아니면 0 값을 가진다.

는 i번째 셀에 객체의 중심이 놓여지면 1, 아니면 0 값을 가진다. 본 실시 예에서는 총 객체의 학습 데이터에서 객체를 포함하는 셀의 개수와 배경만 포함하는 셀의 개수 비를 바탕으로

값을 설정한다.In Equation 3, x, y, w, h represent the center coordinates (x, y), width (w), and height (h) of the bounding box containing the object,

represents the reliability of the object included in the j-th bounding box of the i-th cell,

represents the probability that the object that exists in the i-th cell is of a specific class,

indicates the moving direction of the object.

has a value of 1 if there is an object in the j-th bounding box of the i-th cell, otherwise 0.

has a value of 1 if the center of the object is placed in the i-th cell, otherwise 0. In this embodiment, based on the ratio of the number of cells including the object to the number of cells including only the background in the learning data of the total object

Set the value.

7 is a flowchart illustrating an object detection method according to an embodiment of the present invention. In the following description, descriptions of parts overlapping with those of FIGS. 1 to 6 will be omitted.

Referring to FIG. 7 , in step S710 , the object detection apparatus 1 captures an image through N fisheye lens cameras (eg, 16) installed on the ceiling of an indoor parking lot.

In step S720, the object detection apparatus 1 performs image quality improvement by applying gamma correction and edge enhancement to the image captured by the fisheye lens camera.

In step S730, the object detection apparatus 1 extracts image features by applying a deep learning-based convolutional neural network model to an image of which image quality is improved by photographing through N fisheye lens cameras. In the present embodiment, the object detection apparatus 1 consists of continuous 3×3 and 1×1 convolution and includes 19 convolutional layers for filtering and 5 maximum pooling for image downsampling. Image features can be extracted by applying darknet-19 including a max pooling layer.

In step S740, the object detection apparatus 1 extracts 1024 features of the corresponding cell output from the 19th convolutional layer of the uppermost layer and 256 (4×64) features extracted from 4 cells of the second layer corresponding to the uppermost cell. 1, 1024 features of the cell output from the thirteenth convolutional layer of the next higher layer, and 256 (4×64) features extracted from 4 cells of the next higher layer corresponding to the next higher layer A merged second preprocessing result is generated.

In step S750, the object detection apparatus 1 performs the first pre-processing result, that is, the central coordinates (x, y) of the bounding box from 1024 features obtained by passing 1280 merged features through 1024 1×1×1280 convolutional product filters. ), the width of the object (w), the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object that exists in the cell is of a specific class (Pi), and the moving direction (θ) of the object are estimated. The center coordinates of the bounding box (x, y) from 1024 features obtained by generating a first classification result that A method for estimating the width (w) of the object, the height (h) of the object, the reliability of the object included in the bounding box (C), the probability that the object that exists in the cell is of a specific class (Pi), and the moving direction (θ) of the object 2 Generate classification results.

In operation S760 , the object detection apparatus 1 performs non-maximum suppression (NMS) processing of removing a portion where multiple bounding boxes overlap with the object detected from the first classification result and the second classification result.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium includes a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and a ROM. , RAM, flash memory, and the like, hardware devices specially configured to store and execute program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and used by those skilled in the computer software field. Examples of the computer program may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (especially in the claims), the use of the term "above" and similar referential terms may be used in both the singular and the plural. In addition, when a range is described in the present invention, each individual value constituting the range is described in the detailed description of the invention as including the invention to which individual values belonging to the range are applied (unless there is a description to the contrary). same as

The steps constituting the method according to the present invention may be performed in an appropriate order, unless there is an explicit order or description to the contrary. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and unless defined by the claims, the scope of the present invention is limited by the examples or exemplary terminology. it's not going to be In addition, those skilled in the art will recognize that various modifications, combinations, and changes may be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the spirit of the present invention is not limited to the scope of the scope of the present invention. will be said to belong to

1: object detection device
100: video recording unit
200: image processing unit
300: object detection unit
400: display unit

Claims (11)

a feature extraction unit for extracting features of the image by applying a deep learning-based convolutional neural network model to an image taken through a plurality of fisheye lens cameras provided on the ceiling of the parking lot;
a multiple pre-processing unit merging the features extracted from at least one of the uppermost layer, the next-level layer, and the next-level layer of the feature extraction unit; and
By receiving the merged features from the multi-preprocessor, one or more objects are displayed as a bounding box, and the center coordinate of the bounding box, the width of the object, the height of the object, the reliability of the object included in the bounding box, and the center of the cell A multi-classifier for estimating a probability (Pi) that one or more objects represented by the bounding box present is a specific class, and a moving direction of the object;
The feature extraction unit includes 19 convolutional layers for filtering and 5 maximal pooling layers for image downsampling, including continuous 3×3 and 1×1 convolutions, and the 19 convolutional layers The 6th-8th convolutional layer set means the next-order layer, the 9th-13th convolutional layer set means the next-order layer, and the 14th-19th convolutional layer set means the top layer.
The input image of the feature extraction unit consists of 416×416 cells, the second-order layer consists of 52×52 cells, the second-order layer consists of 26×26 cells, the uppermost layer consists of 13×13 cells, and the bounding box consists of the multiple The object detection apparatus according to claim 1, wherein the multi-classification unit displays the objects detected in the features of the uppermost layer, the next-level layer, and the next-order layer merged from the preprocessor as a box with a border. delete According to claim 1, wherein the multi-preprocessor,
Among the feature extraction unit including 19 convolutional layers and 5 maximum pooling layers, 1024 features of the corresponding cell output from the 19th convolutional layer of the uppermost layer, and 4 cells of the next layer corresponding to the uppermost cell a first preprocessor that merges 256 extracted features; and
a second pre-processing unit that merges 1024 features of the corresponding cell output from the 13th convolutional layer of the second-order layer among the feature extraction unit and 256 features extracted from 4 cells of the second-order layer corresponding to the second-order cell; Including, object detection device. The method of claim 3, wherein the multi-classifying unit,
The center coordinates (x,y) of the bounding box, the area (w) of the object, the center coordinates (x,y) of the bounding box, the A first classifier for estimating the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) of the object existing in the cell being of a specific class, and the moving direction (θ) of the object ; and
The center coordinates (x,y) of the bounding box, the area (w) of the object, the center coordinates (x,y) of the bounding box, the A second classification unit for estimating the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) that the object exists in the cell is a specific class, and the moving direction (θ) of the object ; Containing, object detection device. 5. The method of claim 4,
and a non-maximum suppression (NMS) processing unit that removes a portion where multiple bounding boxes overlap the object from the estimation result of the first classifier and the estimation result of the second classifier. extracting the features of the image by applying a deep learning-based convolutional neural network model to an image photographed through a plurality of fisheye lens cameras provided on the ceiling of the parking lot by the feature extraction unit;
merging the features extracted from at least one of the uppermost layer, the next higher layer, and the next higher layer by the multiple preprocessing unit; and
The multi-classifier receives the merged features from the multi-preprocessor and displays one or more objects as a bounding box, and the central coordinates of the bounding box, the width of the object, the height of the object, and the object included in the bounding box. estimating a probability (Pi) that one or more objects represented by the bounding box with the reliability of the cell and the center of the cell are of a specific class, and the moving direction of the object;
The feature extraction unit includes 19 convolutional layers for filtering and 5 maximal pooling layers for image downsampling, including continuous 3×3 and 1×1 convolutions, and the 19 convolutional layers The 6th-8th convolutional layer set means the next-order layer, the 9th-13th convolutional layer set means the next-order layer, and the 14th-19th convolutional layer set means the top layer.
The input image of the feature extraction unit consists of 416×416 cells, the second-order layer consists of 52×52 cells, the second-order layer consists of 26×26 cells, the uppermost layer consists of 13×13 cells, and the bounding box consists of the multiple The object detection method, characterized in that the multi-classification unit displays the objects detected from the features of the uppermost layer, the next-level layer, and the next-level layer merged from the preprocessor as a box with a border. delete The method of claim 6, wherein the merging comprises:
1024 features of the corresponding cell output from the 19th convolutional layer of the uppermost layer among the feature extraction unit including 19 convolutional layers and 5 maximal pooling layers by the first preprocessor, and corresponding to the uppermost cell merging 256 features extracted from 4 cells of the second layer; and
By the second pre-processing unit, 1024 features of the corresponding cell output from the 13th convolutional layer of the next-order layer among the feature extraction unit and 256 features extracted from the 4 cells of the next-order layer corresponding to the second-order cell are extracted. Including; merging; object detection method. The method of claim 8, wherein the estimating comprises:
By the first classification unit, the center coordinates (x,y) of the bounding box, the object The width (w) of the object, the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) of the object existing in the cell being a specific class, and the moving direction (θ) of the object estimating; and
By the second classification unit, the central coordinates (x,y) of the bounding box, the object The width (w) of the object, the height (h) of the object, the reliability (C) of the object included in the bounding box, the probability (Pi) of the object existing in the cell being a specific class, and the moving direction (θ) of the object Including; estimating the object detection method. 10. The method of claim 9,
removing, by a non-maximum suppression (NMS) processing unit, a portion in which multiple bounding boxes overlap the object from the estimation result of the first classification unit and the estimation result of the second classification unit; A computer program stored in the computer-readable recording medium for executing the method of any one of claims 6 and 8 to 10 using a computer.

Images