KR20210081852A - Apparatus and method for training object detection model - Google Patents

Abstract

Disclosed are a device and method for training an object detection model. The device for training the object detection model according to one embodiment comprises: an image acquisition part that acquires a first image and a second image for a specific space generated using different image sensors; and a model training part that generates fusion information by exchanging some of the information in a feature map generated from the first image and some of the information in a feature map generated from the second image with each other, and trains an object detection model to detect one or more objects existing in the specific space based on the fusion information. Therefore, the present invention is capable of effectively training a model that detects objects.

Description

Object detection model training apparatus and method {APPARATUS AND METHOD FOR TRAINING OBJECT DETECTION MODEL}

Disclosed embodiments relate to techniques for training an object detection model.

With the development of big data processing technology, object detection technology is being treated as important in a wide variety of fields. Recently, object detection using deep learning techniques, rather than object detection using traditional machine learning techniques, has been mainly studied.

However, as object detection is mainly used in security and traffic fields, it has become an important goal to accurately detect objects not only during the daytime but also at night. For this purpose, Charge-Coupled Device (CCD) cameras and infrared (IR) Various methodologies have been proposed to fuse and use the data obtained from the camera.

However, the previously presented methodologies attempt to converge data based solely on concatenation in a part of the network structure, so there is a problem in that CCD camera data and IR camera data cannot be properly fused.

Republic of Korea Patent Publication No. 10-2019-0119261 (published on October 22, 2019)

The disclosed embodiments are for effectively training a model for detecting an object.

An object detection model learning apparatus according to an embodiment includes an image acquisition unit that acquires a first image and a second image for a specific space generated using different image sensors, and some of information in a feature map generated from the first image and model learning for generating fusion information by exchanging some of the information in the feature map generated from the second image, and training an object detection model to detect one or more objects existing in the specific space based on the fusion information includes wealth.

The object detection model includes a plurality of feature extraction layers having a plurality of scales, a first network to which the first image is input, and a second network to which the second image is input, and has the same structure as the first network. may include two networks, wherein the model learning unit outputs at least one of the feature maps output from each of the plurality of feature extraction layers included in the first network and each of the plurality of feature extraction layers included in the second network By arbitrarily matching at least one of the feature maps to be used, some information in each matched feature map may be exchanged.

The object detection model may further include a detection unit that generates one or more object detection boxes corresponding to the one or more objects based on the fusion information and calculates a confidence score for each of the one or more object detection boxes.

Each of the first network and the second network may further include a plurality of concatenation layers and one or more down-scale layers, and the plurality of feature extraction layers are continuous for each scale. , wherein the concatenated layer may be arranged at a rear end of the plurality of feature extraction layers for each scale, and the downscale layer may be a contiguous layer other than a contiguous layer of a minimum scale among the plurality of contiguous layers. It may be disposed at the rear end of each.

The detection unit is based on a result of concatenating a plurality of feature maps output from a plurality of concatenated layers included in the first network and a plurality of feature maps output from a plurality of concatenated layers included in the second network for each scale. to generate the one or more object detection boxes, and calculate the reliability score based on the one or more object detection boxes and a preset ground-truth box.

The detection unit may use a Non-Maximum Suppression (NMS) algorithm based on the confidence score for each of the one or more object detection boxes to remove the overlapping object detection box among the one or more object detection boxes. have.

The model learning unit may include a first feature map among the feature maps output from each of the plurality of feature extraction layers included in the first network and the second one of the feature maps output from each of the plurality of feature extraction layers included in the second network. A second feature map having the same scale as the first feature map may be matched.

The model learning unit may exchange element values of at least some pixels among pixels in the first feature map with element values of pixels at positions corresponding to the at least some pixels among pixels in the second feature map.

When the at least some pixels and the pixels at the corresponding positions are determined, the model learning unit may exchange element values of the at least some pixels with the element values of the pixels at the corresponding positions based on a preset probability.

A method for learning an object detection model according to an embodiment includes: acquiring a first image and a second image for a specific space generated using different image sensors; some of information in a feature map generated from the first image; generating fusion information by exchanging some of the information in the feature map generated from the second image, and training an object detection model to detect one or more objects existing in the specific space based on the fusion information include

The object detection model may include a first network including a plurality of feature extraction layers having a plurality of scales and a second network configured with the same structure as the first network, and generating the fusion information includes: , inputting the first image to the first network, inputting the second image to the second network, and at least one of a feature map output from each of a plurality of feature extraction layers included in the first network and arbitrarily matching at least one of the feature maps output from each of the plurality of feature extraction layers included in the second network to exchange some information in each matched feature map.

The object detection model may further include a detection unit that generates one or more object detection boxes corresponding to the one or more objects based on the fusion information and calculates a confidence score for each of the one or more object detection boxes.

The first network and the second network may further include a plurality of concatenated layers and one or more downscale layers, and the plurality of feature extraction layers may be continuously disposed for each scale, and the concatenated layer may include: Each of the scales may be disposed at a rear end of the plurality of feature extraction layers, and the downscale layer may be disposed at a rear end of each of the remaining contiguous layers except for a minimum scale contiguous layer among the plurality of contiguous layers.

The detection unit is based on a result of concatenating a plurality of feature maps output from a plurality of concatenated layers included in the first network and a plurality of feature maps output from a plurality of concatenated layers included in the second network for each scale. to generate the one or more object detection boxes, and calculate the confidence score based on the one or more object detection boxes and a preset correct answer box.

The detection unit may remove a duplicate object detection box from among the one or more object detection boxes by using a non-maximum value suppression algorithm based on the confidence score for each of the one or more object detection boxes.

The exchanging may include: a first feature map among feature maps output from each of the plurality of feature extraction layers included in the first network and among the feature maps output from each of the plurality of feature extraction layers included in the second network A second feature map having the same scale as the first feature map may be matched.

The exchanging may be performed by exchanging element values of at least some pixels among pixels in the first feature map with element values of pixels at positions corresponding to the at least some pixels among pixels in the second feature map.

The exchanging may be performed based on a preset probability when the at least some pixels and the pixels at the corresponding positions are determined.

According to the disclosed embodiments, the performance of the object detection model may be improved by fusing information of two images generated using different image sensors.

Also, according to the disclosed embodiments, by fusing image information without adding a hyper-parameter, additional memory or operation may not be required when detecting an object.

1 is a block diagram illustrating an apparatus for learning an object detection model according to an embodiment;
2 is a block diagram illustrating an apparatus for executing an object detection model according to an embodiment;
3 is a block diagram illustrating an object detection model according to an embodiment;
4 is a diagram for describing in detail an object detection model according to an embodiment;
5 is a flowchart illustrating a method for learning an object detection model according to an embodiment;
6 is a flowchart illustrating a method of generating fusion information according to an embodiment;
7 is a flowchart illustrating a method of executing an object detection model according to an embodiment;
8 is a diagram for comparing a conventional object detection result and an object detection result according to an embodiment;
9 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, specific embodiments will be described with reference to the drawings. DETAILED DESCRIPTION The following detailed description is provided to provide a comprehensive understanding of the methods, devices, and/or systems described herein. However, this is merely an example and the disclosed embodiments are not limited thereto.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the disclosed embodiments, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions in the disclosed embodiments, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing the embodiments only, and should in no way be limiting. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or a combination thereof, one or more other than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, acts, elements, or any part or combination thereof.

Hereinafter, 'object detection' refers to a computer vision technology that detects an object included in an image by displaying a box surrounding the object included in the image on the image. In this case, the 'object' may narrowly mean a person included in the image, and broadly may mean a car, person, animal, plant, etc. included in the image, but is not limited thereto.

1 is a block diagram illustrating an apparatus 100 for learning an object detection model according to an embodiment. Referring to FIG. 1 , the apparatus 100 for learning an object detection model according to an embodiment includes an image acquiring unit 110 and a model learning unit 120 .

The image acquisition unit 110 acquires a first image and a second image for a specific space generated using different image sensors.

Hereinafter, the 'image sensor' may be a means for directly photographing a specific space or a means for converting a pre-photographed image into a specific format.

For example, the 'image sensor' may be a charge-coupled device (CCD) camera, an infrared (IR) camera, a gray scale image converter, or a radiometric temperature image converter. may be, but is not necessarily limited thereto.

According to an embodiment, when the first image is an image generated by the CCD camera, the second image may be an image generated by the IR camera.

According to another embodiment, when the first image is a grayscale image, the second image may be a radiometric temperature image.

The model learning unit 120 generates fusion information by exchanging some of the information in the feature map generated from the first image and part of the information in the feature map generated from the second image with each other, and based on the fusion information, a specific space The object detection model 300 is trained to detect one or more objects present in .

According to an embodiment, when the object detection model 300 includes the first network 310 and the second network 320 , the model learning unit 120 includes a plurality of features included in the first network 310 . At least one of the feature maps output from each extraction layer and at least one of the feature maps output from each of the plurality of feature extraction layers included in the second network 320 are arbitrarily matched to exchange some information in each matched feature map. can

According to an embodiment, the model learning unit 120 includes a first feature map and a plurality of feature maps included in the second network 320 among the feature maps output from each of the plurality of feature extraction layers included in the first network 310 . Among the feature maps output from each feature extraction layer, a second feature map having the same scale as the first feature map may be matched.

Subsequently, the model learning unit 120 performs fusion by exchanging element values of at least some pixels among pixels in the first feature map and element values of pixels at positions corresponding to at least some pixels above among pixels in the second feature map. can do.

In this case, when at least some pixels in the first feature map for which element values are to be exchanged and pixels at corresponding positions in the second feature map are determined, the model learning unit 120 determines at least one of the pixels in the first feature map based on a preset probability. An element value of some pixel may be exchanged with an element value of a pixel at a corresponding position in the second feature map.

In more detail, when a pair of pixels to be exchanged is determined, the model learning unit 120 may finally exchange element values within the pair of pixels only when a preset probability condition is satisfied.

For example, the model learning unit 120 may set the probability that the exchange is performed in a range of 30% or more and 50% or less.

According to an embodiment, the model learner 120 may perform an exchange between element values within a pair of pixels according to Equation 1 below.

[Equation 1]

는 스케일 s에서의 i번째 특징 지도의 x, y 위치에 존재하는 픽셀의 원소 값,

은 스케일 s'에서의 i'번째 특징 지도의 x, y 위치에 존재하는 픽셀의 원소 값 및

은 값 교환을 위한 임시 변수를 나타낸다.In this case, s is a variable designating the scale of the first network 310 , s' is a variable designating the scale of the second network 320 corresponding to s, and x is a variable representing the horizontal coordinates of pixels on the feature map , y is a variable indicating the vertical coordinates of pixels on the feature map, i is the index value of the feature map sequentially increasing by 1 in the scale corresponding to s, i' is the random index of the feature map in the scale corresponding to s' value,

is the element value of the pixel at the x, y position of the i-th feature map at scale s,

is the element value of the pixel at the x, y position of the i'-th feature map on the scale s', and

represents a temporary variable for value exchange.

For example, referring to FIG. 4 , s has any one of 1, 2, and 3, which designates any one of three stages of classification according to the scale of a plurality of convolution layers. be the value Also, i is a variable having a value of 3 by increasing from 1 to 1 when the index values of three convolutional layers in which element values can be exchanged within one scale are 1, 2, and 3, respectively, i ' is a variable having an arbitrary integer value between 1 and 3.

2 is a block diagram illustrating an object detection model execution apparatus 200 according to an embodiment. Referring to FIG. 2 , the apparatus 200 for executing an object detection model according to an embodiment includes an image acquisition unit 210 and a model execution unit 220 .

The image acquisition unit 210 acquires a first image and a second image for a specific space generated using different image sensors.

The model executor 220 detects one or more objects existing in a specific space based on the first image and the second image using the object detection model 300 previously learned by the object detection model learning apparatus 100 . do.

Specifically, the model executor 220 uses the object detection model 300 to display an object detection box on one or more objects existing in the image with either the first image or the second image as a background. create an image

In this case, the feature map output from the feature extraction layer in the first network 310 and the feature extraction layer in the second network 320 are outputted from the model execution unit 220 using the pre-trained object detection model 300 . There is no information exchange between the feature maps that become available. That is, the exchange operation described with reference to FIG. 1 is performed only in the process of learning the object detection model 300 .

According to an embodiment, the model executor 220 may preset one of the first image and the second image to be the background of the output image.

For example, when the first image is an image photographed by a CCD camera and the second image is an image photographed by an IR camera, the model executor 220 considers visibility and determines that the first image is an output image. You can set it to be the background.

3 is a block diagram illustrating an object detection model 300 according to an exemplary embodiment.

Referring to FIG. 3 , the object detection model 300 according to an embodiment includes a first network 310 , a second network 320 , and a detection unit 330 .

The first network 310 includes a plurality of feature extraction layers having a plurality of scales and receives a first image.

According to an embodiment, the first network 310 may include a convolutional neural network (CNN) structure.

Specifically, the first network 310 may include a single-shot multibox detector (SSD) structure.

For example, the first network 310 may have a form in which a part of the Resnet structure and a part of the SSD structure are combined, but is not limited thereto, and a VGGnet structure may be used instead of the Resnet structure.

According to an embodiment, the plurality of feature extraction layers may be continuously disposed for each scale.

Specifically, each feature extraction layer may be a convolutional layer that outputs a feature map. In this case, the stride value of each feature extraction layer may be 1.

According to an embodiment, the first network 310 may further include a plurality of concatenation layers and one or more down-scale layers.

In this case, the contiguous layer may be respectively disposed at the rear end of the plurality of feature extraction layers for each scale.

Specifically, each concatenated layer may be a convolutional layer that outputs a feature map, and the feature map output from each concatenated layer may be concatenated by the detector 330 for each scale. In this case, the stride value of each contiguous layer may be the same as the stride value of each feature extraction layer.

In addition, the downscale layer may be disposed at a rear end of a plurality of contiguous layers except for the contiguous layer of the minimum scale.

Specifically, each downscale layer may be a convolutional layer that outputs a feature map, and in this case, the stride value of each downscale layer is greater than the stride value of the feature extraction layer of the same scale. For example, when the stride value of the feature map extraction layer is 1, the stride value of the downscale layer of the same scale may be 2 .

The second network 320 has the same structure as the first network 310 and receives a second image.

The detection unit 330 may generate one or more object detection boxes corresponding to one or more objects based on the fusion information and calculate a confidence score for each of the one or more object detection boxes.

According to an embodiment, the detector 330 includes a plurality of feature maps output from a plurality of concatenated layers included in the first network 310 and a plurality of feature maps output from a plurality of concatenated layers included in the second network 320 . One or more object detection boxes are generated based on each result of concatenating the feature map for each scale, and a reliability score can be calculated based on the generated one or more object detection boxes and a preset correct answer (ground-truth, GT) box. .

Specifically, the detector 330 generates a fusion feature map by concatenating the feature map output from the first network 310 and the feature map output from the second network 320 for each scale, and within the generated fusion feature map An object detection box may be generated based on element values of pixels.

According to an embodiment, the detection unit 330 may calculate an Intersection Over Union (IOU) between the generated object detection box and a preset correct answer box to distinguish a positive sample from a negative sample.

According to an embodiment, the detection unit 330 uses a Non-Maximum Suppression (NMS) algorithm based on a confidence score for each of one or more object detection boxes to detect duplicate objects among one or more object detection boxes. can be removed.

Specifically, the detector 330 sorts the object detection boxes in descending order of confidence score, calculates an IoU value between the object detection box having the highest confidence score and the remaining boxes, and removes the remaining boxes having an IoU value of 0.5 or more. do. Thereafter, the detector 330 repeats the same process between the object detection box having the second highest reliability score among the remaining object detection boxes and the remaining boxes. Thereafter, the detection unit 330 repeatedly performs the above-described process.

4 is a diagram for explaining in detail the object detection model 300 according to an embodiment.

Referring to FIG. 4 , a grayscale image and a radiation analysis temperature image for a specific space generated by different image sensors are input to two networks of the same structure arranged in parallel, respectively. In this case, the network structure in which the plurality of layers are arranged at the top represents the first network 310 , and the same network structure at the bottom represents the second network 320 .

According to an embodiment, the images input to the two networks may be images randomly selected from a pre-prepared image data set, or images processed by different image sensors, respectively.

Both networks having the same structure include a part of the Resnet structure previously learned in the previous stage. Subsequently, the two networks include a plurality of convolutional layers arranged for each scale of three stages.

Specifically, the plurality of convolutional layers are classified into a feature extraction layer, a concatenated layer, and a downscale layer. For each scale, the layer disposed in the front part is the feature extraction layer, the layer disposed in the middle part is the concatenation layer, and the last A layer disposed at the end of each of the first and second scales except for the scale is the downscale layer.

The exchange between pixels in the feature map output from each feature extraction layer is represented by an arrow in the center of the drawing, and the entangled arrows in a net shape indicate that the index of each feature map is randomly selected. For convenience, the configuration in which the exchange between pixels in the feature map is performed is called a Partially Random-Wired (PRW).

Referring to the dotted line box on the right side showing the function of the detector 330, the feature maps output from each concatenated layer are concatenated for each scale, and based on the concatenated result, the detector 330 is an object detection box and reliability generate a score. Thereafter, the detector 330 applies the NMS algorithm based on the object detection box and the confidence score.

5 is a flowchart illustrating a method for learning an object detection model according to an embodiment. The method shown in FIG. 5 may be performed, for example, by the above-described object detection model training apparatus 100 .

Referring to FIG. 5 , first, the object detection model training apparatus 100 acquires a first image and a second image for a specific space generated using different image sensors ( S510 ).

Thereafter, the object detection model learning apparatus 100 generates fusion information by exchanging some of the information in the feature map generated from the first image and some of the information in the feature map generated from the second image ( 520 ).

Thereafter, the object detection model training apparatus 100 trains the object detection model 300 to detect one or more objects existing in a specific space based on the fusion information ( 530 ).

In the illustrated flowchart, the method is described by dividing the method into a plurality of steps, but at least some of the steps are performed in a reversed order, are performed together in combination with other steps, are omitted, are performed separately, or are not shown. One or more steps may be added and performed.

6 is a flowchart illustrating a method of generating fusion information according to an embodiment. The method shown in FIG. 6 may be performed, for example, by the above-described object detection model training apparatus 100 .

Referring to FIG. 6 , first, the object detection model training apparatus 100 inputs a first image to the first network 310 ( 610 ).

Thereafter, the object detection model training apparatus 100 inputs a second image to the second network 320 ( 620 ).

Thereafter, the object detection model training apparatus 100 performs at least one of the feature maps output from each of the plurality of feature extraction layers included in the first network 310 and each of the plurality of feature extraction layers included in the second network 320 . At least one of the feature maps output from is arbitrarily matched to exchange some information in each matched feature map ( 630 ).

According to an embodiment, when performing step 630 , the object detection model training apparatus 100 includes a first feature map and a second network among feature maps output from each of a plurality of feature extraction layers included in the first network 310 . Among the feature maps output from each of the plurality of feature extraction layers included in 320 , a second feature map having the same scale as the first feature map may be matched.

According to an embodiment, operation 630 may be performed by exchanging element values of at least some of the pixels in the first feature map with element values of pixels at positions corresponding to at least some of the pixels in the second feature map. .

According to an embodiment, operation 630 may be performed based on a preset probability when at least some of the above-described pixels and pixels at corresponding positions are determined.

In the illustrated flowchart, the method is described by dividing the method into a plurality of steps, but at least some of the steps are performed in a reversed order, are performed together in combination with other steps, are omitted, are performed separately, or are not shown. One or more steps may be added and performed.

7 is a flowchart illustrating a method of executing an object detection model according to an exemplary embodiment. The method shown in FIG. 7 may be performed, for example, by the above-described object detection model execution apparatus 200 .

Referring to FIG. 7 , first, the object detection model execution apparatus 200 acquires a first image and a second image for a specific space generated using different image sensors ( 710 ).

Then, the object detection model execution apparatus 200 uses the object detection model 300 previously learned by the object detection model learning apparatus 100, based on the first image and the second image. One or more objects are detected (720).

In the illustrated flowchart, the method is described by dividing the method into a plurality of steps, but at least some of the steps are performed in a reversed order, are performed together in combination with other steps, are omitted, are performed separately, or are not shown. One or more steps may be added and performed.

8 is a diagram for comparing a conventional object detection result and an object detection result according to an exemplary embodiment. Specifically, FIGS. 8 (a) and (c) are diagrams showing an object detection result according to a conventional object detection method, and FIGS. 8 (b) and (d) are the above-described object detection model learning method It is a diagram showing an object detection result (baseline) by the object detection model 300 learned by .

Referring to (a) to (d) of FIG. 8 , the first and second rows are images of daytime roadside images, and as a result of object detection by the pre-learned object detection model 300 , detection is performed using the conventional method. It can be seen that the failed object (person) was additionally detected. In addition, the third row and the last row are images of the roadside at night, and as a result of object detection by the pre-learned object detection model 300, an object (person) that was not detected through the conventional method was additionally detected even at night. can see.

9 is a block diagram illustrating and describing a computing environment 10 including a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, the computing device 12 may be the object detection model training device 100 . Also, the computing device 12 may be the object detection model execution device 200 .

Computing device 12 includes at least one processor 14 , computer readable storage medium 16 , and communication bus 18 . The processor 14 may cause the computing device 12 to operate in accordance with the exemplary embodiments discussed above. For example, the processor 14 may execute one or more programs stored in the computer-readable storage medium 16 . The one or more programs may include one or more computer-executable instructions that, when executed by the processor 14, configure the computing device 12 to perform operations in accordance with the exemplary embodiment. can be

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer readable storage medium 16 includes a set of instructions executable by the processor 14 . In one embodiment, computer-readable storage medium 16 includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, other forms of storage medium accessed by computing device 12 and capable of storing desired information, or a suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12 , including processor 14 and computer readable storage medium 16 .

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . The input/output interface 22 and the network communication interface 26 are coupled to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output device 24 may include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or imaging devices. input devices, and/or output devices such as display devices, printers, speakers and/or network cards. The exemplary input/output device 24 may be included in the computing device 12 as a component constituting the computing device 12 , and may be connected to the computing device 12 as a separate device distinct from the computing device 12 . may be

Meanwhile, an embodiment of the present invention may include a program for performing the methods described in this specification on a computer, and a computer-readable recording medium including the program. The computer-readable recording medium may include program instructions, local data files, local data structures, and the like alone or in combination. The medium may be specially designed and configured for the present invention, or may be commonly used in the field of computer software. Examples of computer-readable recording media include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and program instructions specially configured to store and execute program instructions such as ROMs, RAMs, flash memories, and the like. Hardware devices are included. Examples of the program may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although representative embodiments of the present invention have been described in detail above, those of ordinary skill in the art will understand that various modifications are possible without departing from the scope of the present invention with respect to the above-described embodiments. . Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the following claims as well as the claims and equivalents.

10: Computing Environment
12: computing device
14: Processor
16: computer readable storage medium
18: communication bus
20: Program
22: input/output interface
24: input/output device
26: network communication interface
100: object detection model training device
110: image acquisition unit
120: model training unit
200: object detection model execution device
210: image acquisition unit
220: model execution unit
300: object detection model
310: first network
320: second network
330: detection unit

Claims (18)

an image acquisition unit configured to acquire a first image and a second image for a specific space generated using different image sensors; and
The fusion information is generated by exchanging part of information in the feature map generated from the first image and part of the information in the feature map generated from the second image, and based on the fusion information, one existing in the specific space An object detection model learning apparatus comprising a model learning unit for learning an object detection model to detect an abnormal object.
The method according to claim 1,
The object detection model is
a first network including a plurality of feature extraction layers having a plurality of scales and to which the first image is input; and
It has the same structure as the first network and includes a second network to which the second image is input,
The model learning unit randomly selects at least one of a feature map output from each of a plurality of feature extraction layers included in the first network and at least one of a feature map output from each of a plurality of feature extraction layers included in the second network. An object detection model learning apparatus that matches and exchanges some information in each matched feature map.
3. The method according to claim 2,
The object detection model may further include a detector configured to generate one or more object detection boxes corresponding to the one or more objects based on the fusion information and calculate a confidence score for each of the one or more object detection boxes. learning device.
4. The method according to claim 3,
Each of the first network and the second network further comprises a plurality of concatenation layers and one or more down-scale layers,
The plurality of feature extraction layers are successively arranged for each scale,
The contiguous layer is disposed at a rear end of the plurality of feature extraction layers for each scale,
The downscale layer is disposed at a rear end of each of the remaining contiguous layers except for the minimum scale concatenated layer among the plurality of concatenated layers.
5. The method according to claim 4,
The detection unit is based on a result of concatenating a plurality of feature maps output from a plurality of concatenated layers included in the first network and a plurality of feature maps output from a plurality of concatenated layers included in the second network for each scale. to generate the one or more object detection boxes, and calculate the confidence score based on the one or more object detection boxes and a preset ground-truth box.
5. The method according to claim 4,
The detection unit removes the overlapping object detection box among the one or more object detection boxes using a Non-Maximum Suppression (NMS) algorithm based on the confidence score for each of the one or more object detection boxes. Object detection model training device.
3. The method according to claim 2,
The model learning unit,
A first feature map among the feature maps output from each of the plurality of feature extraction layers included in the first network and the same as the first feature map among the feature maps output from each of the plurality of feature extraction layers included in the second network An apparatus for learning an object detection model that matches a second feature map having a scale.
8. The method of claim 7,
The model learning unit,
and exchanging element values of at least some pixels among the pixels in the first feature map with element values of pixels at positions corresponding to the at least some pixels among pixels in the second feature map.
9. The method of claim 8,
The model learning unit,
When the at least some pixels and the pixels at the corresponding positions are determined, the element values of the at least some pixels and the element values of the pixels at the corresponding positions are exchanged based on a preset probability.
acquiring a first image and a second image for a specific space generated using different image sensors;
generating fusion information by exchanging part of information in the feature map generated from the first image and part of information in the feature map generated from the second image; and
and training an object detection model to detect one or more objects existing in the specific space based on the fusion information.
11. The method of claim 10,
The object detection model is
a first network comprising a plurality of feature extraction layers having a plurality of scales; and
A second network configured with the same structure as the first network,
The step of generating the fusion information comprises:
inputting the first image to the first network;
inputting the second image to the second network; and
Each matched by randomly matching at least one of the feature maps output from each of the plurality of feature extraction layers included in the first network and at least one of the feature maps output from each of the plurality of feature extraction layers included in the second network A method of learning an object detection model, comprising exchanging some information in a feature map.
12. The method of claim 11,
The object detection model may further include a detector configured to generate one or more object detection boxes corresponding to the one or more objects based on the fusion information and calculate a confidence score for each of the one or more object detection boxes. How to learn.
13. The method of claim 12,
The first network and the second network further include a plurality of concatenated layers and one or more downscale layers,
The plurality of feature extraction layers are successively arranged for each scale,
The contiguous layer is disposed at a rear end of the plurality of feature extraction layers for each scale,
The downscale layer is disposed at a rear end of each of the remaining contiguous layers except for the minimum scale concatenated layer among the plurality of concatenated layers.
14. The method of claim 13,
The detection unit is based on a result of concatenating a plurality of feature maps output from a plurality of concatenated layers included in the first network and a plurality of feature maps output from a plurality of concatenated layers included in the second network for each scale. to generate the one or more object detection boxes, and calculate the confidence score based on the one or more object detection boxes and a preset correct answer box.
14. The method of claim 13,
The detection unit, the object detection model learning method, using a non-maximum value suppression algorithm based on the confidence score for each of the one or more object detection boxes to remove the overlapping object detection box among the one or more object detection boxes.
12. The method of claim 11,
The exchanging step is
A first feature map among the feature maps output from each of the plurality of feature extraction layers included in the first network and the same as the first feature map among the feature maps output from each of the plurality of feature extraction layers included in the second network A method for learning an object detection model, for matching a second feature map having a scale.
17. The method of claim 16,
The exchanging step is
and exchanging element values of at least some of the pixels in the first feature map with element values of pixels at positions corresponding to the at least some of the pixels in the second feature map.
18. The method of claim 17,
The exchanging step is
When the at least some pixels and the pixels at the corresponding positions are determined, the object detection model learning method is performed based on a preset probability.

Images