KR20200119369A - Apparatus and method for detecting object - Google Patents

Abstract

The present invention relates to an apparatus and a method for detecting an object in real time based on deep learning which are specialized in extracting information of a vehicle as an object to detect a parked vehicle in an image acquired through a fisheye lens camera installed on the ceiling of a parking lot and track a running vehicle. According to an embodiment of the present invention, the apparatus for detecting an object comprises: a feature extraction unit to apply a deep learning-based convolutional neural network model to an image photographed through a plurality of fisheye lens cameras provided on the ceiling of a parking lot to extract features of the image; a multi-preprocessing unit to merge features extracted from one or more among the highest layer, second-highest layer, and third-highest layer of the feature extraction unit; and a multi-classification unit to receive the merged features from the multi-preprocessing unit to represent one or more objects with edge boxes, and estimate the center coordinates of the edge boxes, the widths of the objects, the heights of the objects, the reliability of objects included in the edge boxes, the probability that the objects to exist in a cell are a specific class, and the moving direction of the objects.

Description

Object detection apparatus and method {APPARATUS AND METHOD FOR DETECTING OBJECT}

The present invention relates to a deep learning-based real-time object detection apparatus and method specialized in extracting vehicle information as an object in order to detect a parked vehicle from an image acquired through a fisheye lens camera installed on the ceiling of a parking lot and to track a driving vehicle. About.

Object recognition technology based on a convolution neural network needs to extract rich features for various object recognition when trying to recognize various objects in various environments. In order to extract rich features, the number of convolution layers must be very large, and the number of filters used in each convolution layer must also be large. This has a problem that requires a large amount of computation.

The above-described background technology is technical information possessed by the inventors for derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily known to be publicly known prior to filing the present invention.

Korean Patent Publication No. 2018-0136720

The present invention has been conceived to solve the above-described problems and/or limitations, and an object of the present invention according to an aspect is a vehicle that detects and drives a parked vehicle from an image acquired through a fisheye lens camera installed on the ceiling of a parking lot In order to track the data, a deep learning-based real-time object detection technology specialized in extracting vehicle information as an object is developed to achieve a high detection rate with a small amount of computation.

An object detection apparatus according to an embodiment of the present invention extracts features of the image by applying a deep learning-based convolutional neural network model to an image captured through a plurality of fisheye lens cameras provided on the ceiling of a parking lot. Extraction unit; A multiplex preprocessing unit for merging features extracted from at least one of an uppermost layer, a second upper layer, and a second upper layer of the feature extraction unit; And receiving the merged features from the multi-preprocessor to display one or more objects as a bounding box, and to the center coordinate of the bounding box, the width of the object, the height of the object, the reliability of the object included in the bounding box, and the cell. And a multiple classification unit that estimates a probability that the object to exist is a specific class and a moving direction of the object.

The feature extraction unit comprises 19 convolution layers that perform filtering by consisting of 3×3 and 1×1 convolutions continuously for the image output from the fisheye lens camera, and a 2×2 filter. It may include darknet-19 including five maximum pooling layers that perform down-sampling of an image.

The multi-preprocessor includes 1024 features of a corresponding cell output from a 19th convolutional layer of an uppermost layer among the feature extraction units including 19 convolutional layers and five maximum pooling layers, and a second upper layer corresponding to the uppermost cell. A first preprocessor for merging the 256 features extracted from 4 cells of And a second preprocessing unit for merging 1024 features of a corresponding cell output from a 13th convolutional layer of the next upper layer among the feature extracting units and 256 features extracted from four cells of the next higher layer corresponding to the next higher layer cell. It may include.

The multi-classification unit may include a central coordinate (x,y) of the bounding box and an area of the object from 1024 features obtained by passing 1280 features merged by the first preprocessor through 1024 1x1x1280 composite product filters. (w), the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object to be present in the cell is of a specific class (Pi), and the moving direction of the object (θ) A first classification unit; And the central coordinate (x,y) of the bounding box, the area of the object (w) from 1024 features obtained by passing 1280 features merged by the second preprocessor through 1024 1x1x1280 composite product filters, A second classification for estimating the height (h) of the object, the reliability of the object included in the bounding box (C), the probability that the object to exist in the cell is of a specific class (Pi), and the moving direction (θ) of the object May include;

The apparatus may further include a non-maximum suppression (NMS) processing unit that removes a portion where several frame boxes overlap the object from the estimation result of the first classification unit and the estimation result of the second classification unit.

In an object detection method according to an embodiment of the present invention, a deep learning-based convolutional neural network model is applied to an image captured through a plurality of fisheye lens cameras provided on a ceiling of a parking lot by a feature extraction unit to provide the image. Extracting features of; Merging the features extracted from at least one of the uppermost layer, the upper secondary layer, and the upper secondary layer of the feature extraction section by a multiple preprocessor; And receiving the features merged from the multiple preprocessing unit by the multi-classification unit, and representing one or more objects as a bounding box, and the center coordinate of the bounding box, the width of the object, the height of the object, and included in the bounding box. And estimating a reliability of an object, a probability that the object to exist in a cell is a specific class, and a moving direction of the object.

The step of extracting the features includes 19 convolution layers that perform filtering consisting of 3x3 and 1x1 convolutions continuously on the image output from the fisheye camera, and a 2x2 filter. And extracting the features using darknet-19 including five max pooling layers for down-sampling the image.

The merging may include, by a first preprocessor, 1024 features of a corresponding cell output from the 19th convolutional layer of the uppermost layer among the feature extractors including 19 convolutional layers and 5 maximum pooling layers, Merging the features of 256 features extracted from 4 cells of the next upper layer corresponding to the uppermost cell; And 1024 features of the corresponding cell output from the 13th convolutional layer of the next upper layer among the feature extracting unit and 256 features extracted from 4 cells of the next upper layer corresponding to the next upper layer cell by the second preprocessor. It may include; merging.

In the estimating step, the center coordinate (x) of the bounding box from 1024 features obtained by passing 1280 features merged by the first preprocessor through 1024 1x1x1280 composite product filters by a first classification unit (x ,y), the area of the object (w), the height of the object (h), the reliability of the object included in the bounding box (C), and the probability that the object exists in the cell is of a specific class (Pi) and the object Estimating the traveling direction (θ) of; And a center coordinate (x,y) of the bounding box from 1024 features obtained by passing 1280 features merged by the second preprocessor through 1024 1x1x1280 composite product filters by a second classification unit, The width (w) of the object, the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object to exist in the cell is of a specific class (Pi), and the moving direction of the object (θ ) Estimating; may include.

The method may further include removing, by a non-maximum suppression (NMS) processing unit, a portion where several bounding boxes overlap the object from the estimation result of the first classification unit and the estimation result of the second classification unit. have.

In addition to this, another method for implementing the present invention, another system, and a computer program for executing the method may be further provided.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to embodiments, a deep learning-based real-time object detection technology specialized for recognizing a vehicle as an object from an image acquired through a fisheye lens camera installed on the ceiling of a parking lot can be developed, thereby achieving a high detection rate with a small amount of computation. In particular, it is possible to achieve similar detection performance with about 70% of the computational amount of YOLOv3, which has superior performance compared to the amount of computation among previously announced detection devices.

In addition, it is possible to increase the reliability of vehicle tracking in the parking lot by recognizing the direction of the vehicle.

The effects of the present invention are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram showing the performance versus the amount of computation of various object detectors based on a convolutional neural network.
2 is a diagram schematically illustrating a configuration of an object detection apparatus according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating an image quality improvement performed by an image processing unit among the object detection apparatus of FIG. 2.
4 and 5 are diagrams schematically illustrating a detailed configuration of an object detection unit in the object detection apparatus of FIG. 2.
6 is a diagram illustrating an object detection result in the object detection apparatus of FIG. 2.
7 is a flowchart illustrating an object detection method according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to embodiments described in detail together with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all transformations, equivalents, or substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention, and to fully inform a person of ordinary skill in the art to which the present invention belongs. In describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

The terms used in the present application are used only to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance. Terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, identical or corresponding components are assigned the same reference numbers, and redundant descriptions thereof are omitted. I will do it.

With the development of deep learning learning techniques and the ability to collect large amounts of image data, the technology to recognize objects in images has achieved high recognition performance. Among deep learning-based object detection methods, the Faster R-CNN method proposed by Ren et al. is a method that shows fast detection speed and high performance. This method extracts features using a pyramidal convolutional neural network, and determines whether an object exists in the anchor box while moving k anchor boxes from the image. At this time, the existence of the object is determined by features extracted from the neural network in the anchor box area. Finally, the size and position of the region including the object, the reliability of the object existing in the region, and the probability that the object is a feature type are calculated. However, this method has a problem that a large amount of computation is required to implement in real time.

Rendman et al., instead of finding the area of the object while moving the anchor box, a Bro bounding box in each cell of the top layer (resolution: 7×7) of the convolutional neural network, the location and size of the object area, and the reliability that the bounding box contains the object. In addition, the probability that the included object is of a specific type was calculated as a fully connected network. This method is called YOLO (you only look once) because it performs object detection only once in fixed 7×7×b bounding boxes instead of moving the anchor box. This method requires very little computation and can implement real-time object detection. However, there is a problem in that the detection performance of small objects is low.

Liu et al. proposed a method that can effectively detect even small objects. This method is called a single-shot detector (SSD) because it handles the location and size of an object, and the probability of each type of object at once in the bounding box. This method implements a multiscale detector that detects objects in various layers from the high resolution layer to the lowest resolution layer of the convolutional neural network. This method shows a high detection rate with a computational amount similar to that of YOLO.

Redmon et al. constructed a convolutional neural network for feature detection with 19 convolution layers and 5 max pooling layers consisting of consecutive 3×3 and 1×1 convolutions. The resolution of was increased to 13×13, and this neural network was named darknet-19. Instead of a fully connected network, a classifier for object detection, YOLOv2, composed of a convolutional neural network (CNN) was proposed. This method achieves better detection performance than SSD with less computation than YOLO.

He et al. proposed a convolutional neural network composed of very deep layers capable of extracting rich features required for detection of thousands of objects, which is called ResNet. This neural network is a structure capable of stably learning very deep layers, and shows high performance in object detection and recognition. However, as the layer deepens, there is a problem that the amount of computation increases.

Lin et al. found a problem in which the number of object samples is very small compared to the number of background samples in learning a neural network in the method of detecting an object using an anchor box, and thus the performance of the learned detector is degraded, and it is supplemented to learn. We presented a way to do it. This is called RetinaNet.

Redman et al. used 53 convolutional layers in the form of a mixture of darknet-19 and ResNet to extract more rich features, and this is called darknet-53. darknet-53 uses more computation than darknet-19 and less computation than ResNet. Redman et al. extracted features with darknet-53 and created a classifier for object detection in three layers: the top layer, the second layer, and the second layer to detect objects of various sizes, and this method is called YOLOv3.

1 is a diagram showing the performance versus the amount of computation of various object detectors based on a convolutional neural network according to the prior art. Referring to FIG. 1, it is shown that YOLOv3 is superior to other methods in detecting performance compared to the amount of computation.

This deep learning-based object recognition technology has a limitation in that its performance decreases when attempting to recognize various objects in various environments. However, if the object to be recognized is specialized, the viewpoint and type of the camera that photographs the object, and the surrounding environment are specialized, object detection with a high detection rate is possible. In this embodiment, a deep learning-based real-time object detection technology specialized for detecting a vehicle as an object from an image acquired through a fisheye lens camera installed on the ceiling of an underground parking lot is proposed.

2 is a diagram schematically illustrating a configuration of an object detection apparatus according to an embodiment of the present invention. Referring to FIG. 2, the object detection apparatus 1 may include an image capture unit 100, an image processing unit 200, an object detection unit 300, and a display unit 400.

The image capture unit 100 may capture an image through N fisheye lens cameras 100_1 to 100_N (eg, 16) installed on the ceiling of an indoor parking lot. In the present embodiment, each of the fisheye lens cameras 100_1 to 100_N may capture an image by condensing incident light within a viewing angle range of 150 degrees or more and converting it into an electrical signal.

The image processing unit 200 may improve quality of an image captured by the image capturing unit 100. Even if lighting is evenly arranged inside the indoor parking lot, the lighting of the vehicle parked in the corner of the building is dark, so it is difficult to recognize the outline of the rear of the object as shown in Fig. 3(a). When an image including an object is input to the convolutional neural network-based object detection unit 300 as an input, the edge of the object contour is highlighted when visualizing the output value in the intermediate layer of the neural network. In other words, in order to improve the object detection rate, it is necessary to improve the image quality to make the outline of the object clear. Of course, when the object detector is learned, a data augmentation method that changes the intensity and hue of the input image is used, but generally, if the range of change of the object to be detected is large, the performance of object detection is degraded.

In this embodiment, the image processing unit 200 applies gamma correction and edge enhancement to improve the quality of the input image. Gamma correction is shown in Equation 1.

은 입력 영상의 밝기,

는 입력 영상의 밝기 범위,

는 감마 보정된 영상의 밝기,

는 보정된 영상의 밝기 범위를 각각 나타내며, R,G,B 각각 독립적으로 보정한다.In Equation 1

Is the brightness of the input image,

Is the brightness range of the input image,

Is the brightness of the gamma-corrected image,

Represents the brightness range of the corrected image, and R, G, and B are each independently corrected.

In improving the edge, it is necessary to improve less in areas with clear edges and more in areas with weak edges. The edge enhancement method applied in this embodiment is shown in Equation 2.

는 지역 평균을 나타내고,

는 지역 표준편차를 나타낸다.In Equation 2

Represents the regional average,

Represents the local standard deviation.

An image whose image quality is improved as a result of gamma correction and edge enhancement in the image processing unit 200 is input to the object detection unit 300.

The object detection unit 300 extracts features of the image through filtering and down-sampling based on a convolutional neural network on the image (input image) received from the image processing unit 200, and extracts features from the extracted image. Information such as the presence/absence of an object, the location of the object, the size of the object, and the moving direction of the object can be detected.

The display unit 400 displays an object detection result, for example, FIG. 6 detected by the object detection unit 300.

4 and 5 are diagrams schematically illustrating a detailed configuration of an object detection unit in the object detection apparatus of FIG. 2.

Among the existing object detection devices, YOLOv3 shows superior performance compared to other methods in terms of computational amount. On the other hand, YOLOv2, which has a lower computational load than YOLOv3, has much lower performance than YOLOv3 in detecting small objects such as humans, but when detecting large objects such as a bus, the performance is close to YOLOv3.

In this embodiment, as a detector specialized for recognizing only a vehicle as an object in an image acquired through a fisheye lens camera (100_1 to 100_N) installed on the ceiling of a parking lot, there is a large difference in the detection performance of a large object with less computation than YOLOv3. Consider YOLOv2, which is missing. The features extracted from Darknet-19 are not as rich as those extracted from darknet-53, but are sufficient to detect only vehicles as objects.

When a vehicle on the lower parking surface is photographed with a fisheye lens camera 100_1 to 100_N installed on the ceiling of a parking lot, the vehicle placed in the center of the image is large, but the vehicle placed outside the image is small. In order to detect vehicles of various sizes, it is necessary to detect objects with multiple scales as in YOLOv3. Therefore, the detector proposed in this embodiment has a structure that detects objects at multiple scales based on the darknet-19 convolutional neural network.

In the present embodiment, the fisheye lens cameras 100_1 to 100_N are installed on the ceiling of a parking lot, and photograph an object (for example, a driving vehicle) moving under the fisheye lens cameras 100_1 to 100_N, and the moving direction of the object Information can be obtained. Existing methods use an image photographing the front/back/left/right of the object next to the object, so it is not possible to obtain information about the moving direction of the object from the image as in the present embodiment.

4 and 5, the object detection unit 300 may include a feature extraction unit 310, a multiple preprocessor 320, a multiple classification unit 330, and an NMS processing unit 340.

The feature extractor 310 may extract features of an image through filtering and down-sampling of the input image. The feature extraction unit 310 extracts image features of an object such as, for example, an outer edge of a vehicle as an object, an outer corner of the vehicle, and the like, and the position of the vehicle in the classification unit 300 to be described later using these image features. , Height, direction, and area values can be estimated.

As disclosed in FIG. 5, the feature extraction unit 310 applies the previously known darknet-19, and consists of consecutive 3×3 and 1×1 convolutions to perform filtering, and 19 convolution layers. It can include 5 max pooling layers that perform down-sampling of the image, apply batch normalization to the input of each convolutional layer, and use Leaky ReLU (correction linear unit) as an activation function. , rectified linear unit) is applied.

In addition, among the 19 convolutional layers, the 6-8th convolutional layer set is named the after next higher layer, and the 9-13th convolutional layer set is the next higher layer, the next lower resolution. layer), and the 14-19th convolutional layer set may be named the highest layer, the lowest resolution layer. The input image of the feature extraction unit 310 is composed of 416 × 416 cells, the second upper layer is composed of 52 × 52 cells, the second upper layer is composed of 26 × 26 cells, and the top layer is composed of 13 × 13 cells.

The first convolutional layer of the feature extraction unit 310 is convolved with respect to an input image having a resolution of 416×416 at 1 pixel intervals using 32 filters having a size of 3×3. The features output from the first convolutional layer are processed by the first maximum pooling layer including filters of size 2Х2 applied at 2 pixel intervals after batch normalization and Leaky ReLU, and downsampled to a size of 208Х208, and then the second It is input as a convolutional layer.

The second convolutional layer is convolved at 1 pixel intervals using 64 filters having a size of 3×3 for an image having a resolution of 208×208 down-sampled by the first maximum pooling layer. The features output from the second convolutional layer are processed by the second maximum pooling layer that includes filters of 2Х2 size applied at 2 pixel intervals after batch normalization and Leaky ReLU, and downsampled to a size of 104Х104, and then the third. It is input as a convolutional layer.

The third convolutional layer is convolved with a resolution of 104×104 down-sampled by the second maximum pooling layer at 1-pixel intervals using 128 filters of 3×3 size. Features output from the third convolutional layer are input to the fourth convolutional layer after batch normalization and Leaky ReLU. Through this process, the second upper layer is composed of 52 × 52 cells, the second upper layer is composed of 26 × 26 cells, and the top layer is composed of 13 × 13 cells.

The multiple preprocessor 320 merges the features of the uppermost layer and the uppermost layer of the feature extraction unit 310 and outputs them to the multi-classification unit 330, and merges the features of the higher-order and second-order layers and inputs them to the multi-classification unit 330. do.

The multiple classification unit 330 receives the merged features from the multiple preprocessor 320 and displays the detected objects as a bounding box, the center coordinates of the bounding box (x,y), and the width of the object (w). , Estimate the height of the object (h), the reliability of the object included in the bounding box (C), the moving direction of the object (θ), and the probability that the object present in the cell is of a specific class (Pi).

In this embodiment, the multiple pre-processing unit 320 includes a first pre-processing unit 321 and a second pre-processing unit 322, and the multi-classifying unit 330 is a first classification unit 331 and a second classification unit ( 332). In addition, the output of the first pre-processing unit 321 is input to the first classification unit 331, and the first classification unit 331 is used as an object detection result from each of 13×13 cells in the uppermost layer as in YOLOv2. Dog information, that is, x, y, w, h, C, θ, Pi can be detected. The output of the second preprocessor 322 is input to the second classification unit 332, and the second classification unit 332 provides 6 pieces of information, that is, x, as object detection results from each of 26×26 cells of the next upper layer. y, w, h, C, θ, Pi can be detected.

Referring to FIG. 5, the first preprocessor 321 includes 1024 features of a corresponding cell output from the 19th convolutional layer of the top layer and 256 (4×64) features extracted from 4 cells of the next upper layer corresponding to the topmost cell. ) Features are merged and output to the first classification unit 331. The first classification unit 331 is the center coordinate (x,y) of the bounding box from 1024 features obtained by passing 1280 features merged by the first preprocessor 321 through 1024 1×1×1280 composite product filters. , The area of the object (w), the height of the object (h), the reliability of the object included in the bounding box (C), the direction of the object (θ), and the probability that the object in the cell is of a specific class (Pi) is estimated. .

The second preprocessor 322 performs 1024 features of the corresponding cell output from the 13th convolutional layer of the next upper layer, and 256 (4 x 64) features extracted from 4 cells of the next upper layer corresponding to the next upper layer cell. They are merged and output to the second classification unit 332. The second classification unit 332 is the central coordinate (x,y) of the bounding box from 1024 features obtained by passing 1280 features merged by the second preprocessor 322 through 1024 1×1×1280 composite product filters. , The area of the object (w), the height of the object (h), the reliability of the object included in the bounding box (C), the direction of the object (θ), and the probability that the object in the cell is of a specific class (Pi) is estimated. .

The non-maximum suppression (NMS) processing unit 340 overlaps from the class probability Pi that the object exists in the bounding box detected by the first classification unit 331 and the second classification unit 332 (for example, one car It is used to remove (same as when multiple bounding boxes are drawn). The NMS processing unit 340 uses a method of leaving the current pixel as it is when it is the maximum value when compared with the surrounding pixels, and removing it when it is not (non-maximum). The result of removing the overlapping portion by the non-maximum suppression (NMS) processing unit 340 is output to the display unit 400 as shown in FIG. 6.

In YOLOv3, objects are detected by classifiers in the top layer, the top layer, and the top layer, respectively, and the object detector of each layer up-samps the features extracted from the top layer, merges them, and then passes through a convolution filter to determine the features to be used in the classifier. Extract. Since the features of the top layer are features extracted from a wide area, a background outside the limited vehicle area may affect the feature value. Of course, it is possible to select only a necessary part of the features extracted from the top layer in the learning process, but in this embodiment, it was determined that the loss was more than the benefit, and the features obtained from the top layer were not used in the classifier of the next layer. In the proposed method, the validity of this approach is verified through experiments.

In addition, since the size of the vehicle as the object to be detected is more than a certain size, in the present embodiment, unlike YOLOv3, the object is detected only in two floors of the top floor and the top floor.

The convolutional neural network applied the function used in YOLOv2 as a loss function to be minimized during training, and is shown in Equation 3.

는 i번째 셀의 j번째 테두리 상자에 포함된 객체의 신뢰도를 나타내고,

는 i번째 셀에 존재할 객체가 특정 클래스일 확률을 나타내고,

는 객체의 진행방향을 나타낸다.

는 i번째 셀의 j번째 테두리 상자에 객체가 존재하면 1, 아니면 0 값을 가진다.

는 i번째 셀에 객체의 중심이 놓여지면 1, 아니면 0 값을 가진다. 본 실시 예에서는 총 객체의 학습 데이터에서 객체를 포함하는 셀의 개수와 배경만 포함하는 셀의 개수 비를 바탕으로

값을 설정한다.In Equation 3, x,y,w,h represent the center coordinates (x,y), area (w), and height (h) of the bounding box including the object,

Represents the reliability of the object included in the j-th bounding box of the i-th cell,

Represents the probability that the object present in the i-th cell is of a specific class,

Indicates the direction of the object.

Has a value of 1 if the object exists in the j-th bounding box of the i-th cell, or 0 otherwise.

Is 1 if the center of the object is placed in the i-th cell, otherwise it has a value of 0. In this embodiment, based on the ratio of the number of cells including the object and the number of cells including only the background in the training data of the total object

Set the value.

7 is a flowchart illustrating an object detection method according to an embodiment of the present invention. In the following description, portions overlapping with the descriptions of FIGS. 1 to 6 will be omitted.

Referring to FIG. 7, in step S710, the object detection device 1 captures an image through N fisheye cameras (eg, 16) installed on the ceiling of an indoor parking lot.

In step S720, the object detection apparatus 1 performs image quality improvement by applying gamma correction and edge enhancement to an image captured through a fisheye lens camera.

In step S730, the object detection apparatus 1 extracts features of the image by applying a convolutional neural network model based on deep learning to an image with improved image quality by photographing through N fisheye lens cameras. In this embodiment, the object detection apparatus 1 consists of consecutive 3×3 and 1×1 convolutions, and 19 convolution layers for filtering and 5 maximum pooling for down-sampling an image. A feature of an image can be extracted by applying darknet-19 including a max pooling layer.

In step S740, the object detection device 1 includes 1024 features of the corresponding cell output from the 19th convolutional layer of the top layer and 256 (4×64) features extracted from 4 cells of the next upper layer corresponding to the topmost cell. The first preprocessing result of merging and 1024 features of the corresponding cell output from the 13th convolutional layer of the next upper layer, and 256 (4×64) features extracted from the 4 cells of the next upper layer corresponding to the next upper layer cell The merged second preprocessing result is generated.

In step S750, the object detection apparatus 1 uses the first preprocessing result, that is, the merged 1280 features, which are obtained by passing 1024 1x1x1280 composite product filters, and the center coordinates (x,y ), the area of the object (w), the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object to exist in the cell is of a specific class (Pi), and the direction of the object (θ) The first classification result is generated, and the second preprocessing result, that is, the center coordinates (x,y) of the bounding box from 1024 features obtained by passing 1024 1×1×1280 composite product filters for the merged 1280 features, The object width (w), the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object to exist in the cell is of a specific class (Pi), and the direction of the object (θ) 2 Generate classification results.

In step S760, the object detection apparatus 1 performs a non-maximum suppression (NMS) process in which a portion where several frame boxes overlap an object detected from the first classification result and the second classification result is removed.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium is a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM. A hardware device specially configured to store and execute program instructions, such as, RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of the computer program may include not only machine language codes produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (especially in the claims), the use of the term "above" and a similar reference term may correspond to both the singular and the plural. In addition, when a range is described in the present invention, the invention to which an individual value falling within the range is applied (unless otherwise stated), and each individual value constituting the range is described in the detailed description of the invention. Same as

If there is no explicit order or contradictory description of the steps constituting the method according to the present invention, the steps may be performed in an appropriate order. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or illustrative terms (for example, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is limited by the above examples or illustrative terms unless limited by the claims. It does not become. In addition, those skilled in the art can recognize that various modifications, combinations, and changes may be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention is limited to the above-described embodiments and should not be determined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

1: object detection device
100: video recording unit
200: image processing unit
300: object detection unit
400: display unit

Claims (11)

A feature extraction unit for extracting features of the image by applying a deep learning-based convolutional neural network model to an image captured through a plurality of fisheye lens cameras provided on the ceiling of a parking lot;
A multiplex preprocessing unit for merging features extracted from at least one of an uppermost layer, a second upper layer, and a second upper layer of the feature extracting unit; And
Receiving the merged features from the multiple preprocessors, one or more objects are displayed as a bounding box, and the center coordinate of the bounding box, the width of the object, the height of the object, the reliability of the object included in the bounding box, and the existence of the cell Including a multiple classification unit for estimating a probability of the object being a specific class and a moving direction of the object. The method of claim 1, wherein the feature extraction unit,
The image output from the fisheye lens camera is composed of 19 convolution layers that perform filtering consisting of 3×3 and 1×1 convolutions, and a 2×2 filter to perform down-sampling of the image. An object detection apparatus comprising darknet-19 including five maximum pooling layers to perform. The method of claim 1, wherein the multiple preprocessing unit,
Of the feature extraction unit including 19 convolutional layers and 5 maximum pooling layers, 1024 features of the corresponding cell output from the 19th convolutional layer of the uppermost layer and 4 cells of the next upper layer corresponding to the uppermost layer are extracted. A first preprocessor for merging 256 features; And
A second preprocessing unit for merging 1024 features of a corresponding cell output from the 13th convolutional layer of the next upper layer among the feature extracting units and 256 features extracted from 4 cells of the next higher layer corresponding to the next upper layer cell; Containing, object detection device. The method of claim 3, wherein the multiple classification unit,
From 1024 features obtained by passing the 1280 features merged by the first preprocessor through 1024 1x1x1280 composite product filters, the center coordinate (x,y) of the bounding box, the area of the object (w), the A first classification unit for estimating the height (h) of the object, the reliability of the object included in the bounding box (C), the probability that the object present in the cell is of a specific class (Pi), and the moving direction (θ) of the object ; And
From 1024 features obtained by passing the 1280 features merged by the second preprocessor through 1024 1x1x1280 composite product filters, the center coordinate (x,y) of the bounding box, the area of the object (w), the A second classification unit for estimating the height (h) of the object, the reliability of the object included in the bounding box (C), the probability that the object to exist in the cell is of a specific class (Pi), and the moving direction (θ) of the object Containing; object detection device. The method of claim 4,
The object detection apparatus further comprises a non-maximum suppression (NMS) processing unit that removes a portion where several bounding boxes overlap the object from the estimation result of the first classification unit and the estimation result of the second classification unit. Extracting a feature of the image by applying a deep learning-based convolutional neural network model to an image captured by a plurality of fisheye lens cameras provided on a ceiling of a parking lot by a feature extraction unit;
Merging the features extracted from at least one of the uppermost layer, the upper secondary layer, and the upper secondary layer of the feature extraction section by a multiple preprocessor; And
The multiple classification unit receives the merged features from the multiple preprocessor and displays one or more objects as a bounding box, and the center coordinate of the bounding box, the width of the object, the height of the object, and the object included in the bounding box Including, estimating the reliability of the object and the probability that the object to be present in the cell is of a specific class, and a moving direction of the object. The method of claim 6, wherein the extracting the feature comprises:
The image output from the fisheye lens camera is composed of 19 convolution layers that perform filtering consisting of 3×3 and 1×1 convolutions, and a 2×2 filter to perform down-sampling of the image. Containing, object detection method comprising; extracting the feature using darknet-19 including five maximum pooling layers to perform. The method of claim 6, wherein the merging step,
By the first preprocessor, 1024 features of the cell output from the 19th convolutional layer of the uppermost layer among the feature extracting units including 19 convolutional layers and 5 maximum pooling layers, and corresponding to the uppermost cell. Merging the features of 256 features extracted from 4 cells of the next upper layer; And
The second preprocessor merges 1024 features of the corresponding cell output from the 13th convolutional layer of the next upper layer among the feature extraction units and 256 features extracted from 4 cells of the next upper layer corresponding to the next upper layer cell. Including; object detection method. The method of claim 8, wherein the estimating step,
By the first classification unit, the center coordinate (x,y) of the bounding box from 1024 features obtained by passing 1280 features merged by the first preprocessor through 1024 1×1×1280 composite product filters, and the object The area (w) of the object, the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object present in the cell is of a specific class (Pi), and the direction of the object (θ) Estimating; And
By the second classification unit, the center coordinate (x,y) of the bounding box from 1024 features obtained by passing 1280 features merged by the second preprocessor through 1024 1x1x1280 composite product filters, and the object The area (w) of the object, the height of the object (h), the reliability of the object included in the bounding box (C), the probability that the object present in the cell is of a specific class (Pi), and the direction of the object (θ) Estimating; containing, object detection method. The method of claim 9,
The method further comprising: removing, by a non-maximum suppression (NMS) processing unit, a portion where several bounding boxes overlap the object from the estimation result of the first classification unit and the estimation result of the second classification unit. A computer program stored in the computer-readable recording medium to execute the method of any one of claims 6 to 10 using a computer.

Images