KR102274913B1 - Apparatus for Bounding Box Redundancy Removal and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a bounding box redundancy removal apparatus and a driving method thereof. A bounding box redundancy removal apparatus according to an embodiment of the present invention may include a data receiving part for receiving object data for objects detected from the video frame of a photographed image; and a bounding box redundancy removal part for setting a bounding box around the detected objects using the received object data, and performing redundancy removal of the bounding box set in the same object based on a redundancy range automatically set based on an artificial neural network. The bounding box redundancy removal apparatus automatically sets an appropriate redundancy range for each bounding box based on an artificial neural network without the need to set a redundancy threshold directly in advance, and is more suitable for parallel processing.

Description

Apparatus for Bounding Box Redundancy Removal and Driving Method Thereof

The present invention relates to an apparatus for deduplication of a bounding box and a method of driving the apparatus, and more particularly, an appropriate deduplication range for each bounding box is automatically determined based on an artificial neural network without, for example, directly setting a redundancy threshold in advance. It is set and relates to a bounding box deduplication apparatus that provides a deduplication method more suitable for parallel processing, and a method of driving the apparatus.

In general, an algorithm called Greedy Non-Maximum Suppression (Greedy NMS) is used to remove duplicates of bounding boxes (or bounding boxes) during object detection. Greedy NMS works by removing the overlapping bounding box over a preset overlapping threshold (IoU: Intersection over Union Threshold) based on the bounding box with the highest Objectiveness Score among the bounding boxes.

However, such Greedy NMS is cumbersome to find and set an appropriate redundancy threshold in advance to obtain an accurate result, and also has a problem in that it is not suitable for parallel processing because deduplication is sequentially performed for each bounding box.

Also, in the conventional Greedy NMS, since the IoU threshold for determining the degree of redundancy is applied to all boxes at once regardless of the degree of deduplication required for each box, there is a problem in that the accuracy of deduplication of the bounding box is also reduced. This may cause, for example, a problem in which a bounding box that should not be removed is removed.

Korean Patent Publication No. 10-1469099 (2014.11.28) Korean Patent Publication No. 10-1982942 (2019.05.21) Korean Patent Publication No. 10-2217003 (2021.02.10)

Website https://cool24151.tistory.com/36 Website https://blog.naver.com/dr_moms/221649538493

In an embodiment of the present invention, for example, an appropriate deduplication range is automatically set for each bounding box based on an artificial neural network without the need to directly set a duplication threshold in advance, and a boundary providing a deduplication method more suitable for parallel processing An object of the present invention is to provide a box deduplication device and a method of driving the device.

A bounding box deduplication apparatus according to an embodiment of the present invention includes a data receiver configured to receive object data for objects detected from a video frame of a captured image, and the object detected using the received object data. Deduplication of a bounding box that sets a bounding box around the perimeters and performs deduplication of the bounding box set in the same object based on a deduplication range automatically set based on an artificial neural network includes wealth.

The bounding box deduplication unit may automatically set the deduplication range for different objects, respectively.

The bounding box deduplication unit may perform deduplication of the bounding box in a parallel processing method for the different objects.

The bounding box deduplication unit calculates a maximum score using a confidence score for the bounding box included in the object data, and learns the calculated maximum score based on the artificial neural network based on the learning result. You can set the deduplication range automatically.

The bounding box deduplication unit generates binary bit-type deduplication mask data for each IoU based on the bounding box, the reliability score of the bounding box, and a plurality of deduplication degrees (IoU), The maximum score may be calculated by weighted summation of the generated mask data by reflecting a weight according to the characteristic of the object.

The bounding box deduplication unit, after obtaining a matrix according to the number of bounding boxes, binarizes the bounding box into adjacent boxes and non-adjacent boxes according to the IoU threshold, and for the box with the highest score among the adjacent boxes The deduplication mask data may be generated by outputting 1 or outputting 0 otherwise.

In addition, the method of driving a bounding box deduplication apparatus according to an embodiment of the present invention includes the steps of: receiving, by a data receiving unit, object data for objects detected from a video frame of a photographed image; and a bounding box deduplication unit, the above Setting a bounding box around the detected objects using the received object data, and performing deduplication of the bounding box set in the same object based on a deduplication range automatically set based on an artificial neural network. include

The performing of the deduplication may include automatically setting the deduplication ranges for different types of objects, respectively.

In the performing of the deduplication, the deduplication of the bounding box may be performed in a parallel processing method for the different types of objects.

The performing of the deduplication may include calculating a maximum score using a confidence score for the bounding box included in the object data, and learning the calculated maximum score based on the learning result based on the artificial neural network. It may include the step of automatically setting the deduplication range.

The performing of the deduplication includes: generating deduplication mask data in the form of binary bits for each IoU based on the bounding box, a reliability score of the bounding box, and a plurality of deduplication degrees (IoU); and calculating the maximum score by weighted summing the generated mask data by reflecting a weight according to the characteristic of the object.

The generating of the deduplication mask for each IoU includes: after obtaining a matrix according to the number of the bounding boxes, binarizing the bounding box into adjacent boxes and non-adjacent boxes according to an IoU threshold; It may include outputting 1 for a box having a high score, otherwise outputting 0 to generate the deduplication mask data.

According to an embodiment of the present invention, an appropriate deduplication range may be automatically set for each bounding box through, for example, an artificial neural network module for deduplication.

In addition, the embodiment of the present invention may provide higher detection performance in a general method for evaluating a detection result according to automatic setting of a deduplication range.

Furthermore, the embodiment of the present invention will be able to process the deduplication process for all bounding boxes in parallel.

1 is a diagram illustrating an image processing system according to an embodiment of the present invention;
2 is a block diagram illustrating a detailed structure of the image processing apparatus of FIG. 1;
3 is a block diagram illustrating another detailed structure of the image processing apparatus of FIG. 1;
4 is a view for explaining the detailed structure and operation of the LMM of FIG. 3;
5 is a view for explaining the operating principle of the mask generator of FIG. 4;
6 is a block diagram illustrating another detailed structure of the image processing apparatus of FIG. 1, and
7 is a flowchart illustrating a driving process of a bounding box deduplication device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating an image processing system according to an embodiment of the present invention.

As shown in FIG. 1 , the image processing system 90 according to the embodiment of the present invention includes some or all of the photographing apparatus 100 , the communication network 110 , the image processing apparatus 120 , and the third-party apparatus 130 . includes

Here, “including some or all” means that some components such as the third-party device 130 are omitted to configure the image processing system 90 , or a part of the components constituting the image processing device 120 or It means that all of them can be configured by being integrated into a network device (eg, a wireless switching device, etc.) constituting the communication network 110, and it will be described as including all in order to help a sufficient understanding of the invention.

The photographing apparatus 100 includes, for example, a general CCTV (Closed Circuit Television) camera or an IP (Internet Protocol) camera as a surveillance camera. In addition, the photographing apparatus 100 may include a Pan-Tilt-Zoom (PTZ) camera capable of pan, tilt, and zoom operations as well as a fixed camera. The photographing apparatus 100 may be installed in various places to provide photographed images for social safety, crime prevention, suicide issue, and public surveillance. For example, the photographing device 100 may be installed in a public place such as a subway or bus stop to monitor various situations such as incidents and accidents, and may also monitor acts of violence occurring on a railing of a bridge or in a remote place. Furthermore, it will be possible to monitor through CCTV installed in daycare centers and the like.

The communication network 110 includes both wired and wireless communication networks. For example, a wired/wireless Internet network may be used or interlocked as the communication network 110 . Here, the wired network includes an Internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), Wibro network, etc. is meant to include Of course, the communication network 110 according to the embodiment of the present invention is not limited thereto, and may be used, for example, in a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 110 is a wired communication network, the access point in the communication network 110 can connect to a switching center of a telephone company, etc., but in the case of a wireless communication network, it accesses the SGSN or GGSN (Gateway GPRS Support Node) operated by the communication company to transmit data. or by accessing various repeaters such as a Base Transmissive Station (BTS), NodeB, and e-NodeB to process data.

The communication network 110 may include an access point (AP). Here, the access point includes a small base station such as a femto or pico base station that is often installed in a building. Femto or pico base stations are classified according to the maximum number of access to the imaging device 100, etc. in the classification of small base stations. Of course, the access point may include a short-distance communication module for performing short-distance communication such as Zigbee and Wi-Fi with the photographing device 100 and the like. The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-range communication may be performed in various standards such as Bluetooth, Zigbee, infrared, radio frequency (RF) such as ultra high frequency (UHF) and very high frequency (VHF), and ultra wideband communication (UWB) in addition to Wi-Fi. Accordingly, the access point may extract the location of the data packet, designate the best communication path for the extracted location, and transmit the data packet to the next device, for example, the image processing apparatus 120 along the designated communication path. The access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

The image processing apparatus 120 may be named a (artificial neural network-based) bounding box deduplication apparatus or include a bounding box deduplication apparatus according to an embodiment of the present invention. The image processing apparatus 120 may serve as an image analysis apparatus for analyzing the captured image of the photographing apparatus 100 , a control apparatus for performing control using the analysis result, and the like. The image processing apparatus 120 may operate as a server, for example, detecting objects based on an artificial neural network, such as a deep convolutional neural network (DCNN), matching a bounding box to the detected object, and removing a bounding box overlapping with an object of the same type. and so on. For example, the image processing apparatus 120 detects pre-learned types of objects by using object detection DCNN from a video frame of a captured image. The object detection result (or object data) includes the bounding box coordinate value of the detected object and the object type (or object class) value. The operation of the object detection DCNN to be used may be processed in a GPU, DSP, VPU, NPU, etc. capable of large-scale parallel operation instead of a conventional CPU for real-time processing. Various types of object detection DCNN models have been proposed so far, but examples of representative object detection DCNN models include Fast/Faster R-CNN, SSD, and YOLO series.

In addition, the image processing apparatus 120 collects the acquired (or generated) object detection result (eg, bounding box information of the detection object, etc.) and the object tracking result of the previous frame (eg, bounding box information of the existing tracking object, etc.) Received as input and based on bounding box matching, we get the object tracking result in the current frame. By receiving the object tracking result up to the previous frame and the current video frame data as inputs, it is possible to obtain the object tracking result in the current frame based on template matching, for example. Of course, the image processing apparatus 120 may match the bounding box with respect to the object extracted from the video frame, and may perform deduplication of the bounding box in order to track the object.

More specifically, the image processing apparatus 120 according to the embodiment of the present invention may be equipped with an artificial neural network module for deduplication of the bounding box. It is possible to provide a more suitable deduplication method for parallel processing by eliminating the hassle of doing so and by automatically setting an appropriate deduplication range in each bounding box. In other words, since the deduplication range can be set automatically through artificial intelligence operation, for example, deduplication in the bounding box for object A and deduplication in the bounding box for object B are simultaneously possible. Parallel processing may be possible in the deduplication of In addition, since the redundancy threshold is automatically set by artificial intelligence, such as deep learning, it is possible to increase the accuracy of the degree of deduplication. In other words, since the existing IoU threshold is applied to all boxes at once regardless of the degree of deduplication required for each box, boxes that should not be removed may occur, so this problem can be improved. More details on deduplication of bounding boxes will be discussed later.

The third-party device 130 may include a server operated by a public office such as a police station, and a server of a company that provides other content images. For example, a control device operated by a local government may be a third-party device 130 . Of course, the control device is preferably the image processing device 120 according to the embodiment of the present invention. However, the embodiment of the present invention will not be particularly limited to any one. Since the third-party device 130 can be used for various purposes, it may be understood as a server or computing device of a company that provides content, that is, a video image.

FIG. 2 is a block diagram illustrating a detailed structure of the image processing apparatus of FIG. 1 .

As shown in FIG. 2 , the image processing apparatus 120 of FIG. 1 according to an embodiment of the present invention includes a communication interface unit 200 , a control unit 210 , a boundary box processing unit (or a boundary box deduplication device) 220 . ) and some or all of the storage unit 230 .

Here, “including some or all” means that some components such as the storage unit 230 are omitted to configure the image processing apparatus 120 or some components such as the bounding box processing unit 220 are included in the control unit 210 . It means that it can be configured by being integrated with other components, such as, and will be described as including all in order to help a sufficient understanding of the invention.

The communication interface unit 200 communicates with the photographing device 100 and the third-party device 130 via the communication network 110 of FIG. 1 , respectively. In the process of performing communication, the communication interface unit 200 may perform operations such as modulation/demodulation, encoding/decoding, muxing/demuxing, and scaling for converting resolution, which are obvious to those skilled in the art, so further description will be omitted. do.

The communication interface unit 200 may transmit, for example, a photographed image received from the photographing device 100 to the control unit 210 , when there is a request from the third-party device 130 as a manager device according to the request of the control unit 210 . An analysis result of the captured image may be provided.

The control unit 210 is in charge of overall control operations of the communication interface unit 200, the bounding box processing unit 220, and the storage unit 230 of FIG. 2 constituting the image processing apparatus 120 of FIG. 1 . The control unit 210 may include a CPU, an MPU, a GPU, and the like. The control unit 210 temporarily stores the captured image provided from the communication interface unit 200 in the storage unit 230 , and then calls it and provides it to the bounding box processing unit 220 . Also, the control unit 210 may systematically store the image analysis result of the specified format processed by the bounding box processing unit 220 in the DB 120a of FIG. 1 .

Above all, the control unit 210 sets (or matches) a bounding box with respect to objects extracted from a video frame of a captured image according to an embodiment of the present invention, and, of course, prior to this, an operation for extracting an object based on an intra-neural network can be performed, and a bounding box can be set for objects representing different objects to be extracted, respectively. For example, since 60 still images are implemented in a video frame per second, the bounding box may be overlapped with respect to the same object. Therefore, it is preferable that the overlapping bounding box be removed for efficient control, for example. To this end, the control unit 210 may operate in conjunction with the bounding box processing unit 220 .

The bounding box processing unit 220 may be referred to as a bounding box deduplication apparatus according to an embodiment of the present invention, and furthermore, the image processing apparatus 120 of FIG. 1 may be a bounding box deduplication apparatus. Therefore, the extent to which the range of the bounding box deduplication device is set will not be particularly limited to any one form. For example, the bounding box processing unit 220 may be involved only in the operation of processing the bounding box, and the control unit 210 of FIG. 2 may be in charge of image analysis or object detection. However, object detection and DCNN detection are already known, and in the embodiment of the present invention, the description will be focused on how the bounding box processing unit 220 differs from the existing bounding box processing operation.

If the degree of deduplication is not accurately set in removing the bounding box, a useful bounding box may be removed. That is, the economic box that should not be removed may be removed. Therefore, the bounding box processing unit 220 according to an embodiment of the present invention does not generate such an error or, in order to correct a previously generated error, the degree of deduplication can be freely determined through learning such as deep learning through an artificial neural network, that is, artificial intelligence. may be set, and in the process, automatic setting or automatic change of a preset degree of deduplication may be performed.

In addition, the bounding box processing unit 220 simultaneously performs the deduplication of the bounding box on different objects or types of objects, so that parallel processing of deduplication may be possible. That is, deduplication of bounding box for object A and deduplication of bounding box for object B can be performed simultaneously. As a result, the processing speed of image processing is increased that much.

The storage 230 may store and output various types of data processed under the control of the controller 210 . For example, when the lookup table (LUT) is utilized by the bounding box processing unit 220 , the storage unit 230 may store the LUT data in the storage unit 230 and then output the LUT data.

In addition to the above, the communication interface unit 200, the control unit 210, the bounding box processing unit 220 and the storage unit 230 of FIG. 2 according to an embodiment of the present invention may perform various operations, and other detailed information has been sufficiently explained above, so I will substitute those contents.

On the other hand, as another embodiment of the present invention, the control unit 210 may include a CPU and a memory, and may be formed as a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), an instruction interpreter and a registry, and the memory may include a RAM. The control circuit performs a control operation, the operation unit performs an operation operation of binary bit information, and the instruction interpreter performs an operation of converting a high-level language into a machine language and a machine language into a high-level language, including an interpreter or a compiler. , the registry may be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the image processing apparatus 120, a program stored in the bounding box processing unit 220 is copied, loaded into a memory, that is, a RAM, and then executed to speed up the data operation processing speed. can increase

FIG. 3 is a block diagram illustrating another detailed structure of the image processing apparatus of FIG. 1 , FIG. 4 is a diagram for explaining the detailed structure and operation of the LMM of FIG. 3 , and FIG. 5 illustrates the operating principle of the mask generator of FIG. 4 FIG. 6 is a block diagram illustrating another detailed structure of the image processing apparatus of FIG. 1 .

3, the image processing apparatus 120 of FIG. 1 or the bounding box processing unit 220 of FIG. 2 according to an embodiment of the present invention is an LLM (Local Maximum Module) (part) ( 300) may be included. Referring to FIG. 3 , the network unit 290 may be understood to mean the remaining components except for the LLM unit 300 in FIG. 3 . For example, the network unit 290 may be the control unit 210 of FIG. 2 , or an object detector as a SW module constituting the control unit 210 , or an object as a SW module configured in the bounding box processing unit 220 . It can also mean a detector. Therefore, in the embodiment of the present invention, the network unit 290 will not be specifically limited to any specific object.

The LLM unit 300 receives the box, score, and feature (feature) output from the network unit 290 to obtain a local Maximum score (hereinafter referred to as LM score). to output In this case, the deduplicated box has a low LM score.

The LLM unit 300 performs two operations: generating a mask and calculating an LM score using the generated mask. To this end, the LLM unit 300 may include a mask generator 400 as shown in FIG. 4 . Of course, the mask generator 400 may be configured by a S/W module, a H/W module, or a combination thereof. First, the mask generator 400 receives an IoU, a box, and a score, and generates a deduplication mask, that is, mask data for each IoU. It can also be seen that a mask corresponding to each IoU is generated. Here, IoU means the degree or threshold of the degree of deduplication. Also, the mask data generation may refer to, for example, a method of calculating a matrix according to a masking principle. In the next step, the LLM unit 300 calculates an LM score by calculating the weight of the mask from the input features and performing a weighted summation thereof. In general, an artificial neural network performs an operation consisting of a weighted sum and a non-linear function, and the lower part of FIG. 4 shows a weight calculation process.

For example, when there are 4 boxes (for example, in the case of Mask 0.5 based on IoU 0.5), the LLM unit 300 obtains the IoU matrix as shown in FIG. 5 and then the adjacent boxes according to the IoU threshold are Binarize to 1 and non-boxes to 0. Here, if the corresponding box has the highest score among adjacent boxes, that is, if it is LM (Local Maximum), 1 is output for that box, and 0 is output if it is not LM. So, "1010" is output as mask information (or data) for IoU 0.5. According to this principle, "1110" is output as mask information for IoU 0.9.

6 shows an LMM with a loss function and a MoG (Mixture of Gaussian). MoG density estimation is used to learn the LM score. The number of components of this MoG is the number of boxes. Therefore, the parameter mixing coefficient and standard deviation of MoG from the feature, that is, the feature are additionally required. ie used. The mixing coefficient is the weight of each Gaussian, and the sum of all the mixing coefficients is 1, obtained through softmax. Softmax refers to a function that outputs, as a result, the probability that the classification for each class is the correct answer in a multi-classification problem, for example. Since the standard deviation must be greater than zero, it is obtained through the softplus function. The SoftPlus function is a modified version of Relu. That is, it is a modification of the relu function that relieves the moment when the relu becomes 0. The loss function can be expressed as <Equation 1>.

(Where F: the probability density function of multi-variate Gaussian distribution, πk: mixing coefficient of the k-th component among K components, gk: local maximum score, αk: α controls the balance between the two loss terms → 1/ K* 0.2 ~ 0.5)

The first term of the loss function is an LM score applied to NLL (Negative Log Likelihood) of MoG, and is learned to express the distribution of boxes. The second condition is the l1 (or L1) penalty term of the LM score. Through this, only necessary Gaussian is used, and the LM score is learned so that non-LM boxes are first removed by LLM (Local Maximum Module).

In conclusion, the local max module (LLM) of FIGS. 3 and 4 according to an embodiment of the present invention generates a deduplication mask for each IoU based on a box, a score, and an IoU through the mask generator 400, and features LM score is calculated by calculating the weight of the mask based on the LM score. In addition, by performing learning on the calculated LM score, non-LM boxes are removed first, resulting in increased accuracy of bounding box deduplication, and an appropriate deduplication range can be automatically set for each bounding box. In addition, it becomes possible to process the deduplication process for all bounding boxes in parallel.

Meanwhile, the LLM units 300 and 300 ′ shown in FIGS. 3 and 6 are, for example, S/W modules, a data receiving unit for receiving object data related to a detection object from the network unit 290 , and a data receiving unit It may include a bounding box deduplication unit that performs deduplication of a bounding box set in the same object based on a deduplication range automatically set based on an artificial neural network using the object data provided in . For example, since the LLM unit 300 corresponds to a S/W module, the data receiving unit may mean an interface unit that is a data path, and the bounding box deduplication unit may include a manager and the like. However, the embodiment of the present invention will not be particularly limited to this configuration. In other words, when the control unit 210 of FIG. 2 stores the program for deduplication of the bounding box according to the embodiment of the present invention in a memory configured as one-chip with the CPU, and then executes it, the operation target is the control unit 210 Alternatively, it may be a CPU, which is a processor, or the like.

7 is a flowchart illustrating a driving process of a bounding box deduplication device according to an embodiment of the present invention.

For convenience of explanation, referring to FIG. 7 together with FIGS. 3 and 6 , as a bounding box deduplication device according to an embodiment of the present invention, the LLM units 300 and 300 ′ provide information about objects detected from a video frame of a captured image. Receive object data (S700). Here, the object data includes information about the object of the video frame, and may include a bounding box, a reliability score of the bounding box, and furthermore, a degree of deduplication (IoU).

In addition, the LLM unit (300, 300') sets a bounding box around the (pre)detected objects using the received object data, and based on the artificial neural network, for example, based on artificial intelligence learning, automatically set overlapping Deduplication of the bounding box set in the same object is performed based on the removal range (S710).

For example, if a video frame is implemented at 60 frames per second as a screen, the same object appears repeatedly in multiple frames unless it disappears from the screen. Therefore, in the case of image processing that sets a bounding box, the bounding box Deduplication may be required. Therefore, when objects are close to each other during deduplication, unnecessary removal of the bounding box may occur, and the manual setting of the degree of deduplication as in the past also causes unnecessary deduplication, thereby lowering the accuracy of deduplication of the bounding box. can do it Therefore, in an embodiment of the present invention, the accuracy of deduplication is increased by automatically setting the deduplication range accurately based on an artificial neural network such as deep learning of artificial intelligence, or by periodically changing the deduplication range according to the image processing state. In addition, since parallel processing of the deduplication operation for different objects is possible, the operation processing speed may be made faster.

In addition to the above, the bounding box deduplication apparatus of FIG. 7 can perform various operations, and since other details have been sufficiently described above, they will be replaced with the contents.

On the other hand, even though it has been described that all components constituting the embodiment of the present invention are combined or operated as one, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, although all of the components may be implemented as one independent hardware, some or all of the components are selectively combined to perform some or all functions of the combined components in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media, read and executed by the computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, memory, etc. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: photographing device 110: communication network
120: image processing device 130: third-party device
200: communication interface unit 210: control unit
220: bounding box processing unit 230: storage unit
290, 290': Network unit 300, 300': LLM unit
400: mask generator

Claims (12)

a data receiver configured to receive object data for objects detected from a video frame of a photographed image; and
A bounding box is set around the detected objects using the received object data, and the same object is set based on a deduplication range automatically set based on an artificial neural network. Bounding box deduplication unit for performing deduplication of the bounding box; including,
The bounding box deduplication unit calculates a maximum score using a confidence score for the bounding box included in the object data, and learns the calculated maximum score based on the artificial neural network based on the learning result. Automatically set the deduplication range,
The bounding box deduplication unit generates binary bit-type deduplication mask data for each IoU based on the bounding box, the reliability score of the bounding box, and a plurality of deduplication degrees (IoU), A bounding box deduplication apparatus for calculating the maximum score by performing weighted summation by reflecting a weight according to the characteristics of the object to the generated mask data. According to claim 1,
The bounding box deduplication unit is configured to automatically set the deduplication range for different objects, respectively. 3. The method of claim 2,
The bounding box deduplication unit, bounding box deduplication device for performing deduplication of the bounding box in a parallel processing method for the different objects. delete delete According to claim 1,
The bounding box deduplication unit, after obtaining a matrix according to the number of bounding boxes, binarizes the bounding box into adjacent boxes and non-adjacent boxes according to the IoU threshold, and for the box with the highest score among the adjacent boxes A bounding box deduplication apparatus for generating the deduplication mask data by outputting 1, otherwise outputting 0. receiving, by the data receiving unit, object data for objects detected from a video frame of a captured image; and
The bounding box deduplication unit sets a bounding box around the detected objects using the received object data, and the bounding box is set in the same object based on the automatically set deduplication range based on an artificial neural network. Including; performing deduplication;
The step of performing the deduplication includes:
calculating a maximum score using a confidence score for the bounding box included in the object data;
automatically setting a deduplication range based on the artificial neural network based on a learning result by learning the calculated maximum score;
generating deduplication mask data in the form of binary bits for each IoU based on the bounding box, a reliability score of the bounding box, and a plurality of deduplication degrees (IoU); and
calculating the maximum score by weighted summing the generated mask data by reflecting a weight according to the characteristics of the object;
A method of driving a bounding box deduplication device comprising a. 8. The method of claim 7,
The step of performing the deduplication includes:
A method of driving a bounding box deduplication apparatus for automatically setting the deduplication range for different objects, respectively. 9. The method of claim 8,
The step of performing the deduplication includes:
A driving method of a bounding box deduplication apparatus for performing deduplication of the bounding box in a parallel processing manner for the different objects. delete delete 8. The method of claim 7,
The step of generating a deduplication mask for each IoU comprises:
binarizing the bounding box into adjacent boxes and non-adjacent boxes according to an IoU threshold after obtaining a matrix according to the number of bounding boxes; and
Outputting 1 for the box having the highest score among the adjacent boxes, otherwise outputting 0 to generate the deduplication mask data;
A method of driving a bounding box deduplication device comprising a.

Images