KR102462733B1 - Robust Multi-Object Detection Apparatus and Method Using Siamese Network - Google Patents

Abstract

A robust multi-object detection apparatus and method utilizing a Siamese network are provided. A multi-object detection apparatus according to an embodiment includes: an object detection network configured to generate object detection data by detecting at least one object from an image input through a machine learning model based on a convolutional neural network; an object tracking network for generating object tracking data by tracking the presence and location of an object corresponding to the template based on a template including a feature of the object detection network and information on an object to be tracked; a Siamese network including the template, matching a feature provided in a layer of the object detection network and the template as a pair and providing it as an input value of the object tracking network; and a result comparison unit that compares the object detection data with the object tracking data, outputs an object detection result, and updates a template included in the Siamese network.

Description

Robust Multi-Object Detection Apparatus and Method Using Siamese Network

The present invention relates to a robust multi-object detection apparatus and method, and more particularly, to a robust multi-object detection apparatus and method utilizing a Siamese network.

Deep learning (deep learning) is a machine that attempts high-level abstraction (summary of features, core contents, or functions in large amounts of data or complex data) through a combination of several nonlinear transformation methods. Machine learning can be defined as a set of algorithms, and is a branch of machine learning that teaches computers how to think of humans.

Deep learning-based methods, especially object detection using Convolution Neural Networks (CNNs), have made great strides recently with the Fourth Industrial Revolution. In the current object detection technology, a one-stage detection method capable of quickly detecting an object is preferred rather than a two-stage detection method of the R-CNN series having high accuracy but a complicated pipeline. In object detection technology, detection speed and accuracy are opposite factors, and research that can supplement the accuracy of the one-stage detection method is more required.

The present invention is to solve the above-mentioned problems, and in order to compensate for the shortcomings of the one-stage detection method at the level where real-time detection is possible and to increase the object detection performance, a more robust multi-object detection apparatus utilizing a Siamese network and methods.

A multi-object detection apparatus according to an embodiment of the present invention includes: an object detection network for generating object detection data by detecting at least one object from an image input through a convolutional neural network-based machine learning model; an object tracking network for generating object tracking data by tracking the presence and location of an object corresponding to the template based on a template including a feature of the object detection network and information on an object to be tracked; a Siamese network including the template, matching a feature provided in a layer of the object detection network and the template as a pair and providing it as an input value of the object tracking network; and a result comparison unit that compares the object detection data with the object tracking data, outputs an object detection result, and updates a template included in the Siamese network.

A multi-object detection method according to another embodiment of the present invention includes generating object detection data by detecting at least one object in an image input through a machine learning model based on a convolutional neural network; The presence or absence of an object corresponding to the template based on an input value matched as a pair of a template including a feature of the convolutional neural network-based machine learning model and information on an object to be tracked; and generating object tracking data by tracking the location; and comparing the object detection data and the object tracking data to output an object detection result and updating the template.

A computer program according to another embodiment of the present invention is stored in a medium in combination with hardware to execute the above-described multi-object detection method.

The multi-object detection apparatus according to an embodiment of the present invention may detect a plurality of objects included in an input image, track the detected object, and supplement and correct the detected object information. Accordingly, when an object is detected in successive images, a situation in which an object that was detected in a previous frame is not detected can be compensated, and thus the accuracy of multi-object detection is improved. That is, a more robust multi-object detection apparatus may be provided.

1 is a block diagram illustrating a configuration of a multi-object detection apparatus according to an embodiment of the present invention.
2 is an exemplary diagram for explaining a final object detection process performed by the result comparison unit according to an embodiment of the present invention.
3 is a flowchart of a multi-object detection method according to another embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0016] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. Various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. In addition, the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the present invention is limited only by the appended claims along with all equivalents as claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions in several respects.

The terms used in this specification have been selected as currently widely used general terms as possible while considering their functions, but may vary depending on the intention or custom of a person skilled in the art or the emergence of new technology. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in the description of the corresponding specification. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the terms and the contents of the entire specification, rather than the names of simple terms.

1 is a block diagram illustrating a configuration of a multi-object detection apparatus according to an embodiment of the present invention.

The multi-object detection apparatus 10 may detect a plurality of objects included in the input image. Here, the plurality of objects include different objects (people, vehicles, ships, traffic lights, etc.) have. Here, an area in which an object is located is defined as a bounding box, and a type of each object may be displayed around the bounding box. Also, the multi-object detection apparatus 10 may detect a plurality of objects included in the input image, track the detected objects, and perform supplementation and correction on the detected object information. In addition, the image input to the multi-object detection apparatus 10 may be a moving image, and multi-object detection may be performed on the input moving image as an image segmented in units of frames.

Referring to FIG. 1 , a multi-object detection apparatus 10 according to an embodiment of the present invention includes an object detection network 100 , an object tracking network 110 , a Siamese network 120 , and a result comparison unit 130 . do.

The multi-object detection apparatus according to the embodiments and each network or unit constituting the same may have aspects that are entirely hardware, or partially hardware and partially software. For example, each component of a multi-object detection apparatus is intended to refer to a combination of hardware and software driven by the hardware. The hardware may be a data processing device including a central processing unit (CPU) or another processor. In addition, software driven by hardware may refer to a running process, an object, an executable file, a thread of execution, a program, and the like. For example, the object detection network 100 may refer to a combination of hardware and software for receiving an image and analyzing it to detect an object.

The object detection network 100 may generate object detection data by detecting at least one object from an input image through a machine learning model based on a convolutional neural network.

CNN is a type of deep neural network (DNN), and is a neural network composed of one or several convolutional layers, a pooling layer, and fully connected layers. CNN has a structure suitable for learning two-dimensional data, and can be trained through a backpropagation algorithm. In particular, CNN is one of the representative models of deep neural networks that are widely used in various application fields such as object classification in images and object detection.

Such CNNs are a type of machine learning in which a model directly extracts features of an image. CNNs are particularly useful for finding patterns to recognize objects, faces, and scenes in images, learn directly from data, use patterns to classify images, and eliminate the need to manually extract features. Depending on the application, you can build a CNN from scratch or use a pre-trained model with a dataset. Because these CNNs learn features directly, there is no need to manually extract features, high-level recognition results can be checked, and the advantages of re-learning and using CNNs for new recognition tasks based on the existing network are advantages. have.

How CNNs work, tens or hundreds of layers can each learn to detect different features of an image. Filters are used, which are applied at different resolutions to each training image, and the output of the filter is used as an input for the next layer. Filters can start with very simple features, such as brightness and edges, and evolve into more complex features that are unique to an object. CNNs, like other neural networks, consist of an input layer, an output layer, and several hidden layers between the two layers. Each layer performs calculations that change data in order to learn the characteristics of only the corresponding data. The most frequently used layers are convolution, activation/ReLU, and pooling.

Convolution passes the input image through a set of convolution filters that activate specific features in each image. A rectified linear unit (ReLU) maps negative values to zero and holds positive values, enabling faster and more effective learning. At this time, since it is transferred to the next layer after activation, this process is called activation. Pooling performs nonlinear downsampling and simplifies the output by reducing the number of parameters the network has to learn. This operation is repeated for tens or hundreds of layers, each layer learns how to detect multiple features.

The architecture of a CNN learns features from multiple layers and then moves on to classification. The second to last layer is a fully connected layer that outputs a K-dimensional vector. Here, K is the number of classes that the network can predict. This vector contains the probabilities for each class of images being classified. The final layer of the CNN's architecture uses a classification layer such as softmax to provide the classification output.

CNNs learn hundreds, thousands, and sometimes millions of images. When large amounts of data and complex network architectures are used, the application of GPUs can significantly reduce the processing time for training the model. CNNs can be trained from scratch, or transfer learning can be performed using pre-trained models. Which method you use will depend on the resources available and the type of program you want to create.

To train a network from scratch, the designer needs to define the number of layers and filters and other tunable parameters. Training a model from scratch can require large amounts of data and millions of samples to produce accurate results.

The object detection network 100 according to an embodiment of the present invention may be configured as a one-stage detection network suitable for real-time object detection among these convolutional neural network-based machine learning models. Here, the one-stage detection network refers to a detection network in which regional proposal and classification are simultaneously performed. That is, the task of finding the object bounding box and the task of classifying the object at the final output end of the network may be simultaneously performed. Illustratively, the object detection network 100 may be configured as a one-stage detection network such as a You Only Look Once (Yolo) network or CenterNet, but is not limited thereto.

The object detection data generated by the object detection network 100 includes a bounding box corresponding to a class and a region of an object detected in the input image. Here, the type of object may represent the type of the bounding box by being composed of different creation for each bounding box, but is not limited thereto. In some embodiments, the type of object may be configured to indicate the name of an object included in the bounding box in the form of a caption outside the bounding box.

As described above, the object detection network 100 is composed of a plurality of layers (layers), and each layer generates a feature. Here, the feature generated in any one of the plurality of layers included in the object detection network 100 may be provided to the Siamese network 120 .

The Siamese network 120 corresponds to a network that inputs input data to the object tracking network 110 , and may include features and templates of layers of the object detection network 100 . The Siamese network 120 may match the feature provided in the layer of the object detection network 100 and the template stored in the Siamese network 120 as a pair, and the matched feature-template pair to the object tracking network 110 . It can be provided as input. Here, the template is defined as data including information of an object to be tracked in the object tracking network 110 , and the object tracking network 110 may have an object tracking sub-network configured to correspond to each template. In addition, the template stored in the Siamese network 120 may be updated according to the object detection result according to the current frame in a state finally determined in the previous frame. This update process will be described later in more detail. The Siamese network 120 may provide an input value to the object tracking network 110 by matching a feature generated by analyzing the image corresponding to the current frame with the template information determined in the previous frame by the object detection network 100 .

The object tracking network 110 may perform object tracking corresponding to the template input from the Siamese network 120 . The object tracking network 110 may include at least one object tracking subnetwork that performs object tracking in response to a template input from the Siamese network 120 . The object tracking subnetwork may be configured to correspond to the number of templates. Each object tracking sub-network compares the features generated by the object detection network 100 with the template, and analyzes the existence of an object corresponding to the template (Classification) and the boundary box of the object corresponding to the template ( Regression) can be performed individually. Each object tracking subnetwork calculates a probability value for the existence of an object through classification of the object corresponding to the template, and the object through regression of the bounding box of the object corresponding to the template. of bounding box information can be calculated. Each object tracking sub-network may be composed of a Region Proposal Network (RPN), and the existence position of the object can be estimated by tracking the boundary box and existence of the object corresponding to the template input from the feature corresponding to the image. . The object tracking network 110 may generate object tracking data obtained by estimating the location of objects corresponding to each template by collecting the analyzed data and the estimated location of the object in each object tracking sub-network.

The result comparison unit 130 may compare the object detection data generated by the object detection network 100 with the object tracking data generated by the object tracking network 110 to output an object detection result. The result comparator 130 compares and analyzes the object detection data generated by the object detection network 100 and the object tracking data generated by the object tracking network 110 to finally determine the object of the image corresponding to the current frame. . That is, the object detection data generated by the object detection network 100 can be supplemented and corrected for the object detected through the object tracking data generated by using a template generated based on the object information determined in the previous frame. have. The result comparison unit 130 may finally output the object detection result through comparative analysis. Also, the result comparison unit 130 may provide the finally determined information on the object of the image corresponding to the current frame to the Siamese network 120 to be utilized for tracking the object of the image corresponding to the next frame. That is, the template included in the Siamese network 120 may be updated with object information about the image of the current frame.

2 is an exemplary diagram for explaining a final object detection process performed by the result comparison unit according to an embodiment of the present invention.

)와 객체 추적 데이터에 포함된 경계 박스(

)가 동일한 클래스 정보(종류 정보)를 가지고 있는 지를 판단할 수 있다. 또한, 결과 비교부(130)는 객체 검출 데이터에 포함된 경계 박스(

)와 객체 추적 데이터에 포함된 경계 박스(

)의 유사도를 하기 수학식 1과 같은 L2 norm에 따라 판별하게 된다.Referring to FIG. 2 , as described above, object detection data (M bounding boxes) and object tracking data (N bounding boxes) for the image of the current frame are provided by the object detection network 100 and the object tracking network 110 , respectively. can be created in The result comparison unit 130 may determine whether an object included in the object detection data (M bounding boxes) and an object included in the object tracking data (N bounding boxes) correspond to the same object. The result comparison unit 130 includes a bounding box (

) and the bounding box (

) can determine whether they have the same class information (type information). In addition, the result comparison unit 130 includes a bounding box (

) and the bounding box (

) is determined according to the L2 norm as in Equation 1 below.

[Equation 1]

(Here, x and y are spatial coordinates of the bounding box, w and h are the width and height of the bounding box, respectively, and l is a predetermined value and may be determined by learning or empirically.)

)와 객체 추적 데이터에 포함된 경계 박스(

)과 동일한 경계 박스인 것으로 판별이 되면, 하기 수학식 2를 통해 객체 추적 데이터에 포함된 경계 박스(

)의 값을 갱신할 수 있다.The result comparison unit 130 performs the above-described process in the bounding box (

) and the bounding box (

) and the same bounding box, the bounding box (

) can be updated.

[Equation 2]

) 중 객체 검출 데이터에 포함된 경계 박스(

)와 매칭되는 박스(객체)가 없는 경우, 이는 종래 존재하던 객체가 이미지 상에 사라졌다는 것을 의미할 수 있다. 다만, 객체 추적의 오류에 따라 이미지 상에 존재하는 객체를 검출하지 못하는 오차가 발생할 수도 있는 바, 결과 비교부(130)는 해당 박스(객체)를 바로 삭제하지 않고 객체 유무 임계값(k)와 실패 계수 임계값(c)을 사용하여 상기 매칭되지 않는 객체의 미검출 및 삭제 여부를 결정할 수 있다. 구체적으로, 결과 비교부(130)는 객체 추적 데이터에 포함되어 제공된 상기 객체의 유무에 대한 확률 값이 객체 유무 임계값(k)을 초과하는 지 여부를 판단하며, 초과하지 못하는 경우를 계수(count)하게 된다. 결과 비교부(130)는 상기 객체의 유무에 대한 확률 값이 객체 유무 임계값(k) 이하인 지를 연속하는 프레임 이미지에 따라 계속 카운트하게 되며, 카운트되는 값이 실패 계수 임계값(c)을 초과하면, 해당 템플릿을 삭제하도록 결정하게 된다.In addition, the result comparison unit 130 is a bounding box (

) of the bounding box (

) and there is no matching box (object), this may mean that the existing object has disappeared from the image. However, an error in detecting an object existing on an image may occur due to an error in object tracking, and the result comparison unit 130 does not immediately delete the box (object), but calculates the object presence threshold (k) and The failure coefficient threshold value c may be used to determine whether the non-matching object is not detected and deleted. Specifically, the result comparison unit 130 determines whether or not the probability value for the existence of the object provided by being included in the object tracking data exceeds the object existence threshold value k, and counts the case where it does not exceed ) will do. The result comparison unit 130 continues to count whether the probability value for the existence of the object is equal to or less than the object existence threshold value (k) or not according to successive frame images, and when the counted value exceeds the failure coefficient threshold value (c) , decides to delete the template.

)와 매칭되는 박스 중 객체 추적 데이터에 포함된 경계 박스(

)가 없는 경우, 이는 새로운 객체가 검출되었다는 의미할 수 있다. 결과 비교부(130)는 새롭게 검출된 객체를 새로운 템플릿으로 추가하는 것을 결정하게 된다.In addition, the result comparison unit 130 includes a bounding box (

) and the bounding box (

), this may mean that a new object has been detected. The result comparison unit 130 determines to add the newly detected object as a new template.

Information on the detection object finally determined according to the comparison result of the result comparison unit 130 may be output together with the image. Also, the result comparison unit 130 transmits the template in the currently determined state to the Siamese network 120 so that the template stored in the Siamese network 120 is updated to the latest state.

The multi-object detection apparatus 10 according to an embodiment of the present invention detects a plurality of objects included in an input image, respectively, and supplements and corrects the detected object information based on the information tracking the detected object. can be done Accordingly, when an object is detected in successive images, a situation in which an object that was detected in a previous frame is not detected can be compensated, and thus the accuracy of multi-object detection is improved. That is, a more robust multi-object detection apparatus may be provided.

Hereinafter, a multi-object detection method according to another embodiment of the present invention will be described.

3 is a flowchart of a multi-object detection method according to another embodiment of the present invention. The multi-object detection method according to another embodiment of the present invention may be a method performed by the above-described multi-object detection apparatus 10, and FIGS. 1 and 2 may be referred to for description.

Referring to FIG. 3 , a multi-object detection method according to another embodiment of the present invention generates object detection data by detecting at least one object in an image input through a machine learning model based on a convolutional neural network. step (S100); The presence and location of an object corresponding to the template is tracked based on an input value in which a template including the characteristics of the convolutional neural network-based machine learning model and information on the object to be tracked is matched as a pair. to generate object tracking data (S110); and comparing the object detection data with the object tracking data, outputting an object detection result, and updating the template (S120).

First, object detection data is generated by detecting at least one object in an input image (S100).

The object detection network 100 may generate object detection data by detecting at least one object from an input image through a machine learning model based on a convolutional neural network. The object detection network 100 according to an embodiment of the present invention may be configured as a one-stage detection network suitable for real-time object detection among these convolutional neural network-based machine learning models. Here, the one-stage detection network refers to a detection network in which regional proposal and classification are simultaneously performed. That is, the task of finding the object bounding box and the task of classifying the object at the final output end of the network may be simultaneously performed. Illustratively, the object detection network 100 may be configured as a one-stage detection network such as a You Only Look Once (Yolo) network or CenterNet, but is not limited thereto. The object detection data generated by the object detection network 100 includes a bounding box corresponding to a class and a region of an object detected in the input image. Here, the type of object may represent the type of the bounding box by being composed of different creation for each bounding box, but is not limited thereto. In some embodiments, the type of object may be configured to indicate the name of an object included in the bounding box in the form of a caption outside the bounding box. As described above, the object detection network 100 is composed of a plurality of layers, each layer generating a feature. Here, the feature generated in any one of the plurality of layers included in the object detection network 100 may be provided to the Siamese network 120 .

Next, the object tracking data is generated by tracking the presence and location of an object corresponding to the template based on an input value matched as a pair of a template including the characteristics of the object detection network and information on the object to be tracked. (S110).

The Siamese network 120 corresponds to a network that inputs input data to the object tracking network 110 , and may include features and templates of layers of the object detection network 100 . The Siamese network 120 may match the feature provided in the layer of the object detection network 100 and the template stored in the Siamese network 120 as a pair, and the matched feature-template pair to the object tracking network 110 . It can be provided as input.

The object tracking network 110 may perform object tracking corresponding to the template input from the Siamese network 120 . The object tracking network 110 may include at least one object tracking subnetwork that performs object tracking in response to a template input from the Siamese network 120 . The object tracking subnetwork may be configured to correspond to the number of templates. Each object tracking sub-network compares the features generated by the object detection network 100 with the template, and analyzes the existence of an object corresponding to the template (Classification) and the boundary box of the object corresponding to the template ( Regression) can be performed individually. Each object tracking subnetwork calculates a probability value for the existence of an object through classification of the object corresponding to the template, and the object through regression of the bounding box of the object corresponding to the template. of bounding box information can be calculated. Each object tracking sub-network may be composed of a Region Proposal Network (RPN), and the existence position of the object can be estimated by tracking the boundary box and existence of the object corresponding to the template input from the feature corresponding to the image. . The object tracking network 110 may generate object tracking data obtained by estimating the location of objects corresponding to each template by collecting the analyzed data and the estimated location of the object in each object tracking sub-network.

Next, the object detection data is compared with the object tracking data, an object detection result is output, and the template is updated (S120).

)와 객체 추적 데이터에 포함된 경계 박스(

)의 유사도를 하기 수학식 1과 같은 L2 norm에 따라 판별하게 된다.The result comparison unit 130 may compare the object detection data generated by the object detection network 100 with the object tracking data generated by the object tracking network 110 to output an object detection result. The result comparison unit 130 includes a bounding box (

) and the bounding box (

) is determined according to the L2 norm as in Equation 1 below.

[Equation 1]

(Here, x and y are spatial coordinates of the bounding box, w and h are the width and height of the bounding box, respectively, and l is a predetermined value and may be determined by learning or empirically.)

)와 객체 추적 데이터에 포함된 경계 박스(

)과 동일 경계 박스인 것으로 판별이 되면, 하기 수학식 2를 통해 객체 추적 데이터에 포함된 경계 박스(

)의 값을 갱신할 수 있다. The result comparison unit 130 performs the above-described process in the bounding box (

) and the bounding box (

) and the same bounding box, the bounding box (

) can be updated.

[Equation 2]

) 중 객체 검출 데이터에 포함된 경계 박스(

)와 매칭되는 박스(객체)가 없는 경우, 이는 종래 존재하던 객체가 이미지 상에 사라졌다는 것을 의미할 수 있다. 다만, 객체 추적의 오류에 따라 이미지 상에 존재하는 객체를 검출하지 못하는 오차가 발생할 수도 있는 바, 결과 비교부(130)는 해당 박스(객체)를 바로 삭제하지 않고 객체 유무 임계값(k)와 실패 계수 임계값(c)을 사용하여 상기 매칭되지 않는 객체의 미검출 및 삭제 여부를 결정할 수 있다. 구체적으로, 결과 비교부(130)는 객체 추적 데이터에 포함되어 제공된 상기 객체의 유무에 대한 확률 값이 객체 유무 임계값(k)을 초과하는 지 여부를 판단하며, 초과하지 못하는 경우를 계수(count)하게 된다. 결과 비교부(130)는 상기 객체의 유무에 대한 확률 값이 객체 유무 임계값(k) 이하인 지를 연속하는 프레임 이미지에 따라 계속 카운트하게 되며, 카운트되는 값이 실패 계수 임계값(c)을 초과하면, 해당 객체가 이미지 상에서 사라진 것으로 결정하게 된다. 즉, 대응하는 템플릿을 삭제하도록 결정할 수 있다.In addition, the result comparison unit 130 is a bounding box (

) of the bounding box (

) and there is no matching box (object), this may mean that the existing object has disappeared from the image. However, an error in detecting an object existing on an image may occur due to an error in object tracking, and the result comparison unit 130 does not immediately delete the box (object), but calculates the object presence threshold (k) and The failure coefficient threshold value c may be used to determine whether the non-matching object is not detected and deleted. Specifically, the result comparison unit 130 determines whether or not the probability value for the existence of the object provided by being included in the object tracking data exceeds the object existence threshold value k, and counts the case where it does not exceed ) will do. The result comparison unit 130 continues to count whether the probability value for the existence of the object is equal to or less than the object existence threshold value (k) or not according to successive frame images, and when the counted value exceeds the failure coefficient threshold value (c) , it is decided that the object has disappeared from the image. That is, it may be decided to delete the corresponding template.

) 중 객체 추적 데이터에 포함된 경계 박스(

)와 매칭되는 박스가 없는 경우, 이는 새로운 객체가 검출되었다는 의미할 수 있다. 즉, 결과 비교부(130)는 객체 검출 데이터에 포함된 경계 박스(

) 중 객체 추적 데이터에 포함된 경계 박스(

)와 매칭되는 박스가 없는 경우, 새로운 객체가 이미지 상에 검출된 것으로 결정하게 된다. 즉, 대응하는 템플릿을 생성하도록 결정할 수 있다. 결과 비교부(130)의 비교 결과에 따라 최종적으로 결정된 검출 객체에 대한 정보가 이미지와 함께 출력될 수 있다. 또한, 결과 비교부(130)는 현재 결정된 상태의 템플릿을 샴 네트워크(120)에 전달하여 샴 네트워크(120)에 저장된 템플릿이 최신 상태로 갱신되도록 한다.In addition, the result comparison unit 130 includes a bounding box (

) of the bounding box (

) and there is no matching box, it may mean that a new object has been detected. That is, the result comparison unit 130 is a bounding box (

) of the bounding box (

), it is determined that a new object has been detected on the image. That is, it may decide to generate a corresponding template. Information on the detection object finally determined according to the comparison result of the result comparison unit 130 may be output together with the image. Also, the result comparison unit 130 transmits the template in the currently determined state to the Siamese network 120 so that the template stored in the Siamese network 120 is updated to the latest state.

The operation by the multi-object detection method according to the embodiments described above may be at least partially implemented as a computer program and recorded in a computer-readable recording medium. A computer-readable recording medium in which a program for implementing an operation by the multi-object detection method according to the embodiments is recorded and includes all types of recording devices in which computer-readable data is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium may be distributed in network-connected computer systems, and the computer-readable code may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment may be easily understood by those skilled in the art to which the present embodiment belongs.

Although described above with reference to the embodiments, the present invention should not be construed as being limited by these embodiments or drawings, and those skilled in the art will appreciate the spirit and scope of the present invention described in the claims below. It will be understood that various modifications and variations of the present invention can be made without departing from the scope of the present invention.

10: multi-object detection device
100: object detection network
110: object tracking network
120: Siamese Network
130: result comparison unit

Claims (13)

an object detection network for generating object detection data by detecting at least one object from an image input through a convolutional neural network-based machine learning model;
An object tracking sub configured to generate object tracking data by tracking the presence and location of an object corresponding to the template based on a template including a feature of the object detection network and information of an object to be tracked, and corresponding to the template an object tracking network comprising a network;
a Siamese network including the template, matching a feature provided in a layer of the object detection network and the template as a pair and providing it as an input value of the object tracking network; and
Comprising a result comparison unit that compares the object detection data and the object tracking data, outputs an object detection result, and updates a template included in the Siamese network,
The object tracking sub-network calculates a probability value for the existence of an object through classification of the object corresponding to the template, and the object through regression of the bounding box of the object corresponding to the template. Multi-object detection apparatus, characterized in that generating the object tracking data by calculating the bounding box information of. The method of claim 1,
The input image is a moving picture, and the multi-object detection apparatus, characterized in that detecting multiple objects from the image segmented in units of frames. 3. The method of claim 2,
The object detection data is a multi-object detection apparatus, characterized in that it includes a bounding box corresponding to the type (class) of the object detected in the input image and the area of the object. delete The method of claim 1,
The result comparison unit includes a bounding box (

) and the bounding box (

) is determined through Equation 1 below,
The bounding box included in the object detection data (

) and the bounding box (

) is the same bounding box, the bounding box (

), a multi-object detection device, characterized in that it updates the value of .

[Equation 1]

(Here, x and y are spatial coordinates of the bounding box, w and h are the width and height of the bounding box, respectively, and l is a predetermined value for learning, which may be determined by learning or empirically.)

[Equation 2]

6. The method of claim 5,
The result comparison unit is a bounding box (

) of the bounding box (

), if there is no box matching the box, whether the probability value for the presence or absence of the object corresponding to the box is equal to or less than the object presence threshold (k) continues to be counted according to successive frame images, and the counted value is the failure coefficient threshold If (c) is exceeded, it is determined that the object has disappeared from the image,
The result comparison unit includes a bounding box (

) of the bounding box (

) and there is no matching box, it is determined that a new object has been detected. generating object detection data by detecting at least one object from an image input through a convolutional neural network-based machine learning model;
The presence or absence of an object corresponding to the template based on an input value in which a template including a feature of the convolutional neural network-based machine learning model and information of an object to be tracked is matched as a pair; and generating object tracking data by tracking the location; and
Comparing the object detection data and the object tracking data, outputting an object detection result and updating the template,
The step of generating the object tracking data,
It is performed in the object tracking subnetwork configured to correspond to the template,
The object tracking sub-network calculates a probability value for the existence of an object through classification of the object corresponding to the template, and the object through regression of the bounding box of the object corresponding to the template. Multi-object detection method, characterized in that generating the object tracking data by calculating the bounding box information of. 8. The method of claim 7,
The input image is a moving picture, and the method for detecting multiple objects is characterized in that multiple objects are detected from the image segmented in units of frames. 9. The method of claim 8,
The object detection data is a multi-object detection method, characterized in that it includes a bounding box corresponding to the type of the object detected in the input image (class) and the area of the object. delete 8. The method of claim 7,
Outputting the object detection result and updating the template includes:
The bounding box included in the object detection data (

) and the bounding box (

) is determined through Equation 1 below,
The bounding box included in the object detection data (

) and the bounding box (

) is the same bounding box, the bounding box (

), which includes updating the value of .

[Equation 1]

(Here, x and y are spatial coordinates of the bounding box, w and h are the width and height of the bounding box, respectively, and l is a predetermined value and may be determined by learning or empirically.)

[Equation 2]

12. The method of claim 11,
Outputting the object detection result and updating the template includes:
The bounding box included in the object tracking data (

) of the bounding box (

), if there is no box matching the box, whether the probability value for the presence or absence of the object corresponding to the box is equal to or less than the object presence threshold (k) continues to be counted according to successive frame images, and the counted value is the failure coefficient threshold If (c) is exceeded, it is determined that the object has disappeared from the image,
The bounding box included in the object detection data (

) of the bounding box (

) and there is no matching box, determining that a new object has been detected. A computer program stored on a medium in combination with hardware to execute the multi-object detection method according to one of claims 7 to 9, 11 and 12.

Images