KR20200015301A - Object detection apparatus for dynamic detection module selection and method thereof - Google Patents

Abstract

Disclosed are an apparatus for detecting an object for selecting a dynamic detection module and a method thereof. According to an embodiment of the present invention, the apparatus comprises: a storage unit for storing a plurality of detection techniques for object detection, a plurality of detection modules formed by combining a plurality of detection models, and information on each of the detection modules; a state measurement unit for measuring the length of a buffer indicating the amount of data loaded in a buffer at a current time point (t) when the object detection is requested; a detection module selecting unit for dynamically selecting an optimal detection module at a next time (t+1) among the detection modules on the basis of a preset detection module selection period, the length of the buffer, and an input rate per second and an output rate per second of the current time point (t); and an object detection unit for detecting an object from data on the basis of the optimal detection module.

Description

Object detection apparatus for dynamic detection module selection and method

The present invention relates to an object detecting apparatus and method for selecting a dynamic detection module, and more particularly, to an object detecting apparatus for selecting a dynamic detection module for dynamically selecting an optimal detection module based on a detection module selection period or a length of a buffer and It is about a method.

An object detection system that detects an object in an image in real time is largely composed of two parts, a data acquirer and a data processor. Sometimes data acquirer and data processor are connected through data transmission media and distributed independently. In this case, it is mainly designed by using a buffer as in FIG. 1 to reduce transmission delay or synchronization load. In this case, since the time constraint may be violated due to the queuing delay caused by the length of data stored in the buffer, the stability of the buffer must be guaranteed when using the buffer in a real-time system.

Most real-time object detection systems, on the other hand, are designed in a static manner in which the object detection model selected first once is used until the end of system operation. However, when operating an actual system, there may be various instabilities such as a decrease in the output rate per second due to the performance degradation of the device, or a change in the input rate per second due to the change in the number of data acquisition. Makes it difficult to choose the optimal model. When using this static method, it is necessary to select a model that can guarantee stability considering the occurrence of instability. To this end, it is necessary to predict the degree of occurrence and impact of instability during the entire operation period, but it is not easy to predict it in advance, and the longer the operation period of the planned system becomes more difficult. Even if it can be accurately predicted, it is necessary to select a model with a high processing speed in consideration of stability when the performance of a low frequency system is severely degraded. Therefore, the lower the frequency of occurrence, the less unwanted performance waste occurs. Therefore, the static model selection technique solved the problem mainly by developing a new model with faster GPU parallelism or faster processing speed than model selection.

In addition, in a static real-time object detection system that uses a fixed object detection model throughout the system operation, it prepares for instability that threatens the system's time constraints such as an increase in input rate per second or performance degradation of a device. Therefore, we use a detection technique or neural network model that has a faster processing speed than the optimal processing speed in a stable situation. However, in the object detection, there is a trade-off between the processing speed and the classification accuracy of the model to be used. Therefore, there is a disadvantage in that the performance of the system is generated while the instability does not occur.

Accordingly, there is a demand for development of real-time object detection technology capable of maintaining the stability of the system even in the real-time situation and maximizing the average time performance.

In this regard, there is a Japanese Patent Application Laid-Open No. 2005-141601 entitled "Model Selection Calculation Device, Dynamic Model Selection Device, Dynamic Model Selection Method and Program".

The technical problem to be solved by the present invention is to provide an object detection apparatus and method for selecting a dynamic detection module that can maintain the maximum performance of the average time while maintaining the stability of the system even in the situation that changes in real time when real-time object detection will be.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

Object detection apparatus for selecting a dynamic detection module according to an embodiment of the present invention for solving the technical problem, a plurality of detection formed by combining a plurality of detection techniques and a plurality of detection models for object detection (object detection) Module, a storage unit storing information on each detection module, a state measuring unit measuring a length of a buffer representing the amount of data loaded into the buffer at the current time t when an object detection is requested, a preset detection module Based on the selection period, the length of the buffer, the input rate per second of the current time t of each detection module and the output rate per second, the optimal time of the next time t + 1 among the plurality of detection modules A detection module selection unit for dynamically selecting a detection module, the object detection unit for detecting an object in the data based on the optimum detection module.

The storage unit may store information about each detection module including at least one of an input rate per second, an expected accuracy, and a processing time coefficient at a current time t of each detection module.

The detection module selector may select an optimal detection module based on Lyapunov optimization.

The detection module selecting unit may further include a throughput predictor that estimates a throughput per second for a next time point t + 1 based on the length of the buffer and an input rate per second, and a buffer based on the predicted throughput per second and a time constraint. A buffer limit length calculator for calculating a limit length, a weight calculator for calculating a reward term weight constant based on the throughput per second and the limit length of the buffer, a throughput per second calculated for each detection module, and a buffer limit Based on the length and the compensation weight, it may include an optimum detection module selection unit for selecting an optimal detection model that satisfies a predetermined requirement from the plurality of detection modules.

In addition, the detection module selector may select the optimal detection module m (t + 1) of the next time point t + 1 using the following equation.

[Equation]

은 탐지모듈 m을 사용할 경우의 기대 정확도,

은 시점 t+1에 대한 초당 입력률(input rate),

은 탐지모듈 m을 사용할 경우의 시점 t+1에 대한 예상 초당 처리율(output rate),

는 현재 버퍼의 길이를 의미함.Where m is an element of the detection module set, M is a detection module set, L is a time constraint (sec), V is a compensation weight,

Is the expected accuracy when using detection module m,

Is the input rate per second for time t + 1,

Is the expected output rate per second for time t + 1 when using detection module m,

Is the length of the current buffer.

In the method for detecting an object by the object detecting apparatus according to another embodiment of the present invention for solving the technical problem, when the object detection is requested, it means the amount of data loaded in the buffer at the current time (t) Measuring a length of a buffer, based on a preset detection module selection period, a length of the buffer, an input rate per second and an output rate per second at a current time t of each detection module, and a plurality of detections Dynamically selecting an optimal detection module at a next time point t + 1 among the modules, and detecting an object in data based on the optimal detection module.

According to the present invention, a detection for detecting an object according to a situation that changes in real time even when an instability situation that threatens time constraints such as an increase in input rate or deterioration of a device occurs as well as an instability situation does not occur. By dynamically choosing models, you can maximize time-averaged performance while maintaining system stability.

Effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a view for explaining a conventional object detection system.
2 is a view for explaining an object detection system according to an embodiment of the present invention.
3 is an exemplary view for explaining a detection module according to an embodiment of the present invention.
4 is a view for explaining a detection module selection unit shown in FIG.
5 is a flowchart illustrating a method of selecting a dynamic optimal detection module according to an embodiment of the present invention.
6 is a diagram illustrating a water algorithm for selecting an optimal detection module according to an embodiment of the present invention.
7 to 11 are diagrams for explaining the performance of the dynamic optimal detection module selection method according to an embodiment of the present invention as an experimental result.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the drawings, similar reference numerals are used for similar elements.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is said to be "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that another component may be present in the middle. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a view for explaining an object detection system according to an embodiment of the present invention, Figure 3 is an exemplary view for explaining a detection module according to an embodiment of the present invention, Figure 4 is a detection module shown in FIG. It is a figure for demonstrating a selection part.

Referring to FIG. 2, an object detection system according to an embodiment of the present invention includes an image acquisition device 100, a buffer 200, and an object detection device 300.

The image capturing apparatus 100 is an apparatus for receiving and sampling an image, for example, a camera apparatus such as CCTV.

The image capturing apparatus 100 samples data to be analyzed from an input (or photographed) image and stores the sampled data in the buffer 200. For example, when the camera captures an image, one minute of the image may be stored in the buffer 200, and after a four minute rest period, the image may be stored in the buffer 200 again. Alternatively, the image may be extracted once every 5 seconds and stored in the buffer 200.

The object detecting apparatus 300 loads data from the buffer 200, selects an optimal detection module to be used according to the current system state, and performs object detection using the selected detection module. Here, the detection module means a unit combining elements such as a detection technique, a detection model (neural network model), the number of proposals, etc. The detection technique is an object detection such as R-CNN, Faster R-CNN, R-FCN, Multibox, SSD, etc. The detection model may include various neural network models such as ResNet, InceptionV2, and MobileNet. Therefore, a plurality of detection modules may be formed in a manner in which various detection models are matched with each detection technique.

For example, as shown in FIG. 3, the detection technique R-CNN is matched with various detection models such as ResNet, InceptionV2, MobileNet, and the like, and various detection modules such as detection module 1, detection module 2, and detection module 3 are formed.

The processing time and accuracy of the deep learning based object detection system are dependent on the detection technique or the detection model (neural network model) used. In other words, not only which detection method is used such as R-CNN, Faster R-CNN, R-FCN, Multibox, SSD, etc., but also the detection model used by ResNet, InceptionV2, MobileNet even when the same detection method is used. The processing time and accuracy may also vary depending on the set values such as) or the number of proposals. Therefore, in the present invention, a unit combining elements such as a detection technique, a detection model (neural network model), and the number of proposals will be described as a 'detection module'.

The object detecting apparatus 300 detects an unstable situation that threatens time constraints such as an increase in input rate or deterioration of the device, as well as a detection to be used according to a changing situation in case an unstable situation does not occur. Modules can be selected dynamically. That is, in the object detection using neural network model, there is a trade-off between the processing speed of the detection module and the classification accuracy. Therefore, the system wastes performance while the instability does not occur. Accordingly, the object detecting apparatus 300 must dynamically select a detection module capable of maximizing the time average performance while maintaining the stability of the system even in a situation that changes in real time.

To this end, the object detection apparatus 300 should select a detection module that satisfies the detection module selection criteria and requirements for stability and performance-accuracy trade off problem. Here, the detection model selection criteria and requirements may include guaranteeing stability, maximizing performance, robustness , low time complexity, and the like.

Guaranteeing Stability means selecting a detection module that ensures the stability of the system so that it does not violate any time constraint under any circumstances. Therefore, the detection module selected should have the highest accuracy among detection modules having a processing speed that can guarantee the stability of the system. At this time, the stable state means that the throughput is higher than the input rate and the length of the buffer does not increase. Factors that cause unstable conditions, such as 'unstable' states that buffers are exhausted due to lower throughput than input rates, and increased throughput or reduced throughput due to poor device performance, are considered 'unstable'. If this instability persists, there is a risk of violating time constraints due to queuing delays. Therefore, the object detecting apparatus 300 should be able to select a detection module that ensures the stability of the system so as not to violate the time constraint under any circumstances.

Maximizing performance means that the processing time for a single piece of data must not violate time constraints. If the output rate is not lower than the input rate, the buffer will no longer be exhausted and stability is guaranteed. However, if you use a detection module that has a higher throughput than necessary, it means that you use a detection module that has a lower performance than necessary. Therefore, in order to maintain the stability of the system, it is necessary to select a detection module that maximizes the time average performance of the target system. Here, the time average performance means 'the number of processed data per unit time * the accuracy of the model used to process each data' and is calculated by Equation 1 below.

[Equation 1]

Robustness means choosing a robust detection module against false choices or noise. That is, even if the system loses stability due to a prediction error, the detection module should be selected in consideration of restoring stability before the time constraint is violated due to the queuing delay.

Low time complexity means the time complexity of the detection module selection algorithm itself used must be low.

As described above, the object detecting apparatus 300 may select a detection model having the highest accuracy as the optimum detection model among candidate detection models having a processing speed capable of guaranteeing stability of the system.

On the other hand, when using a first-in, first-out buffer system, the time constraints of the real-time system at any point in time may be represented by a limit of the buffer size. For example, consider a system that has a time constraint of 10 seconds and can process 1 image per second. In this case, if the buffer is completely empty, the newly requested image can be processed immediately, so if the transmission delay is ignored, the processing is completed in one second. If five data are accumulated in the buffer, processing of newly requested image has to wait until the previous five processes are completed, so it takes 5 seconds of queuing delay and takes 6 seconds of processing time. Still, time constraints are being met. However, if 10 data are accumulated in the buffer, the processing time takes 11 seconds and the time constraint is not satisfied. Thus, in this system, a time constraint of 10 seconds can be represented by the limit size of 10 buffers.

As such, the problem of time constraints can be thought of as a buffer stability problem that determines the limit size of the buffer and prevents the buffer size from exceeding the limit size. Therefore, the object detecting apparatus 300 may select an optimal detection module based on Lyapunov optimization.

In detail, the object detecting apparatus 300 uses an algorithm combining Lapunov optimization to find an optimal detection module, and maximizes the Lapunov drift, which is an equation for the state of the buffer and the objective function, to be applied to the next object detection. Find the best detection module. For this purpose, the relationship between the state of the buffer to be optimized in Rapunov and the objective function can be defined as in Equation 2 below.

[Equation 2]

Where V is a constant to indicate the trade-off between the minimization of the objective equation and the stability of the queue, and O (x) is the objective equation to be minimized according to the detection module x selected at time t. . Q (t) is the occupied state of the current buffer, and in (x) and out (x) are the ratio of input data and throughput in the detection module x.

Meanwhile, the object detecting apparatus 300 may optimize the next time t + 1 among the plurality of detection modules based on the state of the buffer, the input rate per second at the current time t, and the output rate per second. The detection module can be selected dynamically. Therefore, the object detecting apparatus 300 may select an optimal detection module using Equation 3 below by applying the Lyapunov optimization technique.

[Equation 3]

는 Lyapunov drift의 reward term으로 성능에 대한 보상함수, V는 reward term의 가중치 상수,

은 탐지모듈 m을 사용할 경우의 기대 정확도,

은 시점 t+1에 대한 예상 초당 입력률(input rate),

은 탐지모듈 m을 사용할 경우 시점 t+1에 대한 예상 초당 처리율(output rate),

는 현재 버퍼의 길이,

는 버퍼의 한계 길이,

시점 t에 데이터를 처리하는데 소요된 총 시간과 현재 시점 t에서 사용된 최적 탐지모듈 M에 의하여 소요된 시간을 각각 의미,

은 탐지모듈 m의 처리시간 계수를 의미할 수 있다. Where m (t + 1) is the optimal detection module to be selected at the next point in time (t + 1), t is the current point in time, t + 1 is the next point in the future, m is an element of the detection module set, and M is a detection module set , L is the time constraint (sec),

Is the reward term for Lyapunov drift, and V is the weight constant for reward term,

Is the expected accuracy when using detection module m,

Is the expected input rate per second for time t + 1,

Is the expected output rate per second for time t + 1 when using detection module m,

Is the length of the current buffer,

Is the limit length of the buffer,

Means the total time spent processing data at time t and the time spent by the optimal detection module M used at current time t,

May mean a processing time coefficient of the detection module m.

의 조건을 만족하지 않는 탐지모듈 m은 선택 대상에 포함시키지 않는다. 또한 현재 처리중인 데이터의 처리시간과 큐잉 딜레이를 합친 시간이 총 처리시간이기 때문에, Q(t)는 '현재 버퍼의 길이 + 1'로 계산될 수 있다. The object detecting apparatus 300 may select an optimal detection module to be used at a next time point using Equation 3. In this case, there is no reason to use a detection module that is expected to require a processing time longer than the constraint L when processing one piece of data.

The detection module m that does not satisfy the condition is not included in the selection. In addition, since the combined processing time of the data currently being processed and the queuing delay are the total processing time, Q (t) may be calculated as 'the current buffer length + 1'.

는 Lyapunov drift에서의 reward term의 역할을 하며, 탐지모듈 m을 선택했을 경우의 기대 시 평균 성능을 의미한다. V는 reward term의 가중치 상수로, 시 제약 조건에 따라 달라진다.

은 탐지모듈 m을 사용했을 때의 기대 시 평균 성능인 '기대 정확도 * 예상 처리율'을 의미한다.

는 탐지모듈 m의 기대 정확도를 의미하며 사전에 미리 계산된 탐지모듈 별 평균 정확도일 수 있다. 시스템의 전체 시 평균 성능은 초당 처리율에 비례하여 높아지지만, 초당 입력율을 넘는 초당 처리율은 의미가 없으므로 둘 중 최소값을 사용하여 계산한다. 수학식 3에서

은 Lyapunov drift에서의 penalty term의 역할을 한다. Penalty term은 기능적으로는 불안정 요소 또는 예측오류로 인하여 버퍼의 크기가 증가(또는 소진)했을 경우 그 정도에 비례하여 처리속도가 느린 탐지모듈에 대하여 penalty를 부가함으로써 처리속도에 대한 가중치를 증가시킨다. 이를 통해 처리속도가 더 빠른 탐지모듈을 선택하도록 하고 버퍼의 상태 및 안정성을 회복시킨다. 예상 입력률

는 모든 후보 탐지모듈 m에 대하여 동일한 값을 가지는 상수로서 취급할 수 있기 때문에, m(t+1)을 구하는 계산과정에 영향을 미치지 않을 수 있다. 따라서, 수학식 3은 아래 수학식 4와 같이 간소화될 수 있다.

Is the reward term for the Lyapunov drift and represents the expected average performance when the detection module m is selected. V is the weight constant for the reward term, depending on the time constraint.

`` Expected Accuracy * Estimated Throughput '', which is the expected average performance when using detection module m.

Denotes the expected accuracy of the detection module m and may be a previously calculated average accuracy of each detection module. The overall performance of the system is increased in proportion to the throughput per second, but the throughput per second over the input per second is meaningless, so it is calculated using the minimum of the two. In equation (3)

Acts as a penalty term in the Lyapunov drift. The penalty term increases the weight for processing speed by adding a penalty to the slower detection module in proportion to the extent of buffer size increase (or exhaustion) due to functional instability or prediction error. This allows you to choose a faster detection module and restore the buffer's state and stability. Expected input rate

Since may be treated as a constant having the same value for all candidate detection modules m, it may not affect the calculation process for obtaining m (t + 1). Therefore, Equation 3 may be simplified as shown in Equation 4 below.

[Equation 4]

In conclusion, the object detecting apparatus 300 may select an optimal object detecting module using Equation 4 and detect an object using the selected detecting module.

The object detecting apparatus 300 includes a storage 310, a loading unit 320, a state measuring unit 330, a detection module selecting unit 340, and an object detecting unit 350.

The storage 310 stores a plurality of detection modules formed by combining a plurality of detection techniques for object detection and a plurality of detection models, and information on each detection module. Here, the information on the detection module may include information at the current time t and preset information, such as an input rate per second, an expected accuracy, and a processing time coefficient at the current time t of each detection module.

The loading unit 320 loads data from the buffer 200.

When the object detection is requested, the state measuring unit 330 measures the length Q (t) of the buffer 200, which indicates the amount of data loaded into the buffer at the current time t. Since the combined processing time of the data currently being processed and the queuing delay is the total processing time, the state measuring unit may calculate the length Q (t) of the buffer as (the length of the current buffer +1).

The detection module selector 340 may be configured to include a plurality of detection modules selected in the storage unit 310 based on a preset detection module selection period, a buffer length, an input rate per second and an output rate per second at a current time t. Dynamically selects the optimal detection module at the next time point (t + 1) from the detection modules of.

The detection module selector 340 includes a throughput predictor 342, a buffer limit length calculator 344, a weight calculator 346, and an optimum detection module selector 348 as shown in FIG. 4. .

과 탐지모듈 m에 대한 초당 처리율

를 예측한다.The throughput predictor 342 predicts the throughput per second for the next time t + 1 based on the length of the buffer at the current time t and the input rate per second. That is, the throughput predicting unit 342 inputs the input rate per second at the next time point.

Throughput per second for detection module m

Predict.

Using static model selection techniques requires predicting changes over the lifetime of the system, and the predictions must be sophisticated because they are not hardened against prediction errors. In contrast, in the present invention, when selecting a detection module at the next point of time, the instability due to the change of the operation period of the system is considered together, so even if a prediction error occurs, only a slight penalty in terms of time average performance occurs, and stability is still guaranteed. It is hard to error. This advantage simplifies the problem of predicting the input rate / output rate to a relatively simple measure of the result of the previous time point t, and reduces the amount of time complexity.

In the present invention, the state of the system changes greatly at the moment of instability, but after a certain change, the change range is small, because the shorter the detection module selection cycle, the more the state of the next time converges to the state of the previous time. The result of the measurement at t, the previous point in time, can be used to some extent as a predicted value at time t + 1. In fact, the input rate per second is caused by external reasons such as the addition of a data acquisition device rather than instability due to changes in system performance. Therefore, the input rate measured during the previous time point t can be used as a predicted value of the input rate per second at the time t + 1. Similarly, the prediction rate per second for each detection module at t + 1 is used as it is. However, despite requiring the throughput measurement value at time t for all candidate detection modules, there is a problem that the only measurable throughput is the throughput for only one detection module selected and used at time t. For example, if there are nine candidate detection modules, measuring the throughput of all candidate detection modules requires at least nine times as much processing time as using the fastest detection module. This is impossible due to the nature of the system. Therefore, we use the method of simulating how much processing time was spent when another detection module was used at time t calculated based on the measurement processing time value of the detection module that was used as the optimum detection module at the previous time. The most obvious way to do this is to precompute the throughput per second for each model based on a number of situations in the system, but this is not practical.

Therefore, instead of calculating the processing time value of each detection module before the system operation, the processing time ratio is calculated and some processing time is required when another candidate detection module is used at time t using Equation 5 below. It can be predicted.

[Equation 5]

은 이전 시점 t에서 최적으로 선택된 탐지모듈 M에 의하여 소요된 시간, fm은 후보 탐지모듈의 처리시간 계수, f_M은 시점 t에 선택된 최적 탐지모듈 M의 처리시간 계수를 의미한다. 처리시간 계수는 사전에 미리 계산된 탐지모듈 별 처리시간을 기준 탐지모듈의 처리시간으로 나눈 비율을 의미한다. Where P _total (t) is the total processing time measured at time t,

Is the time spent by the detection module M optimally selected at the previous time t, fm is the processing time coefficient of the candidate detection module, and f _M is the processing time coefficient of the optimal detection module M selected at the time t. The processing time coefficient refers to a ratio obtained by dividing the processing time for each detection module previously calculated by the processing time of the reference detection module.

In order to predict the output rate for each detection module in this way, there must be a premise that their ratio must be maintained at a certain rate even as the system changes.

The buffer limit length calculator 344 calculates the limit length of the buffer based on the throughput per second and the time constraints predicted by the throughput predictor 342.

The limit length Qmax of the buffer represents a time constraint of the system and can be calculated using Equation 6 below.

[Equation 6]

)Qmax = L / (1 /

)

)가 5인 탐지모듈의 경우 Qmax는 50이 된다. 이처럼 Qmax는 탐지모듈의 처리속도에 따라 결정되는데, 시스템의 성능이 동적으로 변하기 때문에 Qmax 역시 이에 맞추어 탐지모듈 선택 주기마다 새롭게 계산해야 한다. For example, you have a 10 second time constraint (1 /

For a detection module with 5), Qmax is 50. As such, Qmax is determined by the processing speed of the detection module. Since the performance of the system changes dynamically, Qmax must be newly calculated at every detection module selection cycle accordingly.

On the other hand, since the problem of measurement error caused by simplifying the prediction problem is corrected by reducing the detection module selection period, using a detection module selection period that is too long may run away from optimal performance. Therefore, the shorter the detection module selection cycle, the closer the expected performance will be. However, depending on the system, the load on changing the detection module to be used for the entire system may be high. In this case, if the detection module selection cycle is too short, even if the detection module selection algorithm itself has a low time complexity, the detection module change over. Actual implementation issues such as heads can occur and the consistency of detection module usage can be compromised. Therefore, it must be taken into account in consideration of the nature of the system.

Also, if the detection module selection cycle is too long, the system may become extremely unstable at the moment and the system's time constraints may be violated before selecting a new detection module in the next selection cycle. The detection module can be selected even if the state has changed significantly. For example, if 50% of the limit length of the buffer is suddenly exhausted, detection module selection can be made regardless of the detection module selection cycle.

The weight calculator 346 calculates a reward term weight constant based on the throughput per second and the limit length of the buffer. The reward weight V is a weight constant that controls how much more or less reward will be considered compared to the penalty. If V is large, the penalty term becomes less important, and therefore, the buffer term is insensitive to exhaustion, or delay. On the contrary, if V is small, the penalty term is important and therefore sensitive to delay. Therefore, the tighter the time constraint, that is, the lower the time limit, the shorter the limit length of the buffer. This means that V has a function of determining the limit length of the buffer. As the buffer is exhausted by the penalty term, detection modules with faster processing speed will be selected, and ensuring the stability of the buffer will ensure that the highest throughput detection module is selected before the buffer reaches its limit length. Has the same meaning as Therefore, V may be set to satisfy Equation 7 for all candidate detection modules m.

[Equation 7]

와

는 각각 처리율순으로 오름차순 정렬했을 때, 처리율이 가장 높은 탐지모듈을 사용했을 경우의 정확도와 처리율을 의미한다.here,

Wow

Is the accuracy and throughput when the detection module with the highest throughput is used when sorted in ascending order.

Satisfying equation (7), the larger the V is tolerate a larger delay, so the available efficiency of the buffer increases. However, the time constraints are just as guaranteed. Under ideal circumstances, the larger the V, the better, but in practice there may be unstable conditions such as measurement errors in input and output rates or extreme performance degradation that cannot be handled by even the highest detection modules. Setting it to a value of ~ 70% can help to ensure stability. This acts as if the system thinks that it has a lower time limit than the given time limit, so it can be slightly lower than the average time performance in the ideal case, but the wasteful intervals are extremely low. It's only when it happens so there's no big difference.

When the throughput per second, the limit length of the buffer, and the compensation weight are calculated for each detection module, the optimum detection module selector 348 determines a plurality of detections based on the throughput per second, the limit length of the buffer, and the compensation weight calculated for each detection module. Among the modules, the optimal detection module at the next time point (t + 1) is dynamically selected. That is, the optimum detection module selection unit 348 may dynamically select an optimal detection module at a next time point using Equation 4.

은 버퍼의 길이가 Q이고 탐지모듈 m을 사용했을 때의 Lyapunov drift식의 계산 값을 의미하고, 아래 수학식 8과 같이 표현할 수 있다. The algorithm for selecting the optimum detection module by the object detecting apparatus 300 configured as described above may be as shown in FIG. 6. In the capital algorithm

Is the calculated value of the Lyapunov drift equation when the buffer length is Q and the detection module m is used, and can be expressed as Equation 8 below.

[Equation 8]

The object detecting apparatus 300 configured as described above may change in real time even when an instability situation that threatens time constraints such as an increase in an input rate or a decrease in performance of the device occurs, as well as when an instability situation does not occur. Therefore, by dynamically selecting the detection model for object detection, it is possible to maximize the time average performance while maintaining the stability of the system.

In FIG. 1, the image capturing apparatus 100, the buffer 200, and the object detecting apparatus 300 are implemented as separate apparatuses, but they may be implemented as a single apparatus.

5 is a flowchart illustrating a method of selecting a dynamic optimal detection module according to an embodiment of the present invention.

Referring to FIG. 5, when an object detection is requested, the object detecting apparatus measures a length of a buffer representing an amount of data loaded into the buffer at the current time t (S510).

및 기대 정확도

, 현재 버퍼의 길이

_, reward term의 가중치 V를 이용한 수학식 4의 값을 각 탐지모듈별로 산출하고, 산출된 값이 최대가 되는 탐지모듈을 최적 탐지모듈로 선택할 수 있다. Then, the object detecting apparatus is based on a preset detection module selection period, a buffer length, an input rate per second of the current time t, and an output rate per second, and then a next time point t of the plurality of detection modules. The optimal detection module of +1) is dynamically selected (S520). That is, the object detecting apparatus selects detection modules satisfying the time constraint. Then, the object detecting apparatus estimates the throughput per second for the next time point t + 1 of each selected detection module.

And expected accuracy

The length of the current buffer

_{For example,} the value of Equation 4 using the weight V of the reward term may be calculated for each detection module, and a detection module having the maximum calculated value may be selected as an optimal detection module.

Thereafter, the object detecting apparatus detects the object from the loaded data by using the optimum detecting module (S530).

7 to 11 are diagrams for explaining the performance of the dynamic optimal detection module selection method according to an embodiment of the present invention as an experimental result.

 The COCO object detection dataset, which is a Tensorflow real-time object detection framework API, was used for experiment-based performance evaluation, and the dynamic detection module adaptation algorithm of FIG. 6 was added. Also, in order to reproduce the instability, the scenario was set up by changing the input rate with time and stressing the GPU. Table 1 shows the specific settings of the scenario, and FIG. 7 is a graph of the change over time by visualizing it.

TABLE 1

The contextual meaning of each interval in the scenario of Table 1 is as follows:

A 0 to 20 second interval represents the normal state where no instability occurs. In the 20 to 40 second period, there is a severe instability situation where the input rate increases from 5 to 13. In the 40-60 second period, the input rate drops back to 2, indicating a very relaxed system. In the 60 to 80 seconds, the input rate is increased to 8 again, indicating a situation where some instability occurs. If the 0 ~ 80 second interval simulates the instability due to the input rate, the subsequent interval simulates the instability caused by the performance change of the device, which is reproduced by GPU stress. In the 80 to 100 seconds, the device is temporarily deteriorated due to some reason. In the 100 ~ 120 second period, the device recovers from the degradation and returns to the normal state. Subsequent intervals show a case where the device is restored to a normal state again.

Candidate detection module to be used for experiment is Tensorflow as shown in Table 2. Use the detection module provided by API . At this time, the detection modules are sorted in ascending order of throughput and called detection modules 1 to 5.

TABLE 2

A static model selection techniques to be used for comparisons with dynamic selection is the fastest, taking into account only the stability of the processing speed, Detect module 5] ssd _ mobilenet _ v1 ' static fast as opposed accuracy using only are high but slower processing time' [ Detection module 3] Consider static slow using only faster_rcnn_inception_v2 ' . The static opt simulates the ideal case of perfectly predicting the performance change of all scenarios and systems in order to provide a criterion for optimality. In this scenario, [detection module 4] ssd_inception_v2 is the most ideal. The dynamic detection module selection method proposed in the present invention compares the case of a system with a 0.5 second time constraint with that of a system with a harder 0.3 second time constraint.

8 illustrates a change in a model selected when each comparison object is used as the scenario progresses. The x-axis represents the time stamp when each data is processed, and the y-axis represents the model used to process the data. In this case, the closer the detection module is to the origin, the slower the processing time and the higher the accuracy. In the case of using the static model selection method, each model will not change from the first selected model. If the model is selected dynamically, it will select the optimal model for the situation of the system at each point in time.

9 shows the stability of the system as the scenario progresses. The stability of a system is represented by a change in delay, which is the time it takes for data to be requested and processed. The dotted lines represent the time constraints of 0.6 and 0.3 seconds, respectively, and if the delay is higher than this, it means that the constraint is violated at that time, which means that stability is not guaranteed.

10 illustrates a change in time average performance of the system as the scenario progresses. The hourly average performance is calculated as 'total sum of total mAP of the model used for processing each data / total elapsed time'. For example, if 10 images were processed using a model of mAP 30 for 10 seconds and 3 images were processed using a model of mAP 50, the time average performance of ((30 * 10 + 50 * 3) / 10 = 45 ' To have. In this case, the mAP means the average value of the AP, which is a widely used evaluation method for measuring accuracy by comprehensively considering various factors in the object detection field, and thus, the detection performance of the model. In the present invention, since measuring mAP is a separate problem, the experiment did not newly measure it during system operation, and used a bench mark measured in advance to map the model.

In addition, in the case of the static fast model, as shown in the red solid line of FIG. 9, since the processing speed was given priority, almost no delay occurred even in the unstable elements of the scenario, and thus, the overall stability was shown. However, in the case where the performance degradation of the system does not occur or the input rate is not high (0-20, 40-60, 140-180) as in the red solid line of FIG. 10, the performance is faster than necessary. Overall, the city's average performance was wasted. On the contrary, in the case of static slow, which gives priority to the purple color accuracy, the delay occurs very much from 20 seconds after the first instability situation, although the accuracy is high but the stability is low. In addition, it can be confirmed that high accuracy does not necessarily mean high time average performance because the total number of processed data is reduced even if the time constraint is ignored. The ideal optimal model considering the average time performance shown by green line shows the average time performance of about 30 ~ 40% higher than the static fast model and satisfies the 0.6 second time constraint. However, as can be seen in the interval 20 to 40, when the time constraint becomes more rapid and changes to 0.3 seconds, the static opt cannot guarantee the stability of the system and thus cannot be optimal.

On the other hand, in case of using dynamic model adaptation, the stability was maintained by selecting the appropriate model even in the heavy load section, and the performance was almost similar to the average performance of the ideal static selection method. Theoretically, even though the optimal model is chosen, it shows lower time-average performance than static opt, because the proposed technique avoids possible instability itself, and does not consider models that are expected to be lower than the input rate even if the throughput is slightly lower. to be. This can be near theoretically optimal as the model is diversified or the selection cycle is shorter. In particular, when using static opt, although the instability is resolved before the constraint is violated in the above scenario, there is a problem that there is a possibility of violating if the instability persists longer.

If you have a 0.3 second time constraint, you will choose a faster model to fit a harder time constraint even under the same instability, as in the 40-60 second interval. Although the average performance of the city is slightly indistinguishable, we can see that it is still stable. These experimental results show that the present invention is useful for optimizing the time-average performance while ensuring the stability of the system even when applied to actual applications under various conditions.

On the other hand, embodiments of the present invention may be implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Examples of program instructions such as magneto-optical, ROM, RAM, flash memory, etc. may be executed by a computer using an interpreter as well as machine code such as produced by a compiler. Contains high-level language codes. The hardware device described above may be configured to operate as one or more software modules to perform the operations of one embodiment of the present invention, and vice versa.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the appended claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

100: image acquisition device
200: buffer
300: object detection device
310: storage unit
320: loading unit
330: state measuring unit
340: detection module selection unit
342: throughput prediction unit
344: buffer limit length calculation unit
346: weight calculation unit
348: Optimal detection module selection unit
350: object detector

Claims (6)

A plurality of detection modules formed by combining a plurality of detection techniques for object detection and a plurality of detection models, and a storage unit for storing information on each detection module;
When the object detection is requested, the state measuring unit for measuring the length of the buffer means the amount of data loaded into the buffer at the current time (t);
Based on a preset detection module selection period, a length of the buffer, an input rate per second and an output rate per second of a current time t of each detection module, a next time point t + among the plurality of detection modules Detection module selection unit for dynamically selecting the optimal detection module of 1); And
An object detector for detecting an object from data based on the optimum detection module
Including, the object detecting apparatus for selecting a dynamic detection module. The method of claim 1,
The storage unit stores the information on each detection module including at least one of the input rate per second, expected accuracy, processing time coefficient at the current time (t) of each detection module for detecting the dynamic detection module, characterized in that . The method of claim 1,
The detection module selection unit,
Object detection apparatus for selecting a dynamic detection module, characterized in that for selecting the optimal detection module based on the Lyapunov optimization. The method of claim 3,
The detection module selection unit,
A throughput predicting unit for predicting a throughput per second for a next time point t + 1 based on the length of the buffer and an input rate per second;
A buffer limit length calculator for calculating a limit length of a buffer based on the predicted throughput per second and a time constraint;
A weight calculator configured to calculate a reward term weight constant based on the throughput per second and the limit length of the buffer; And
And an optimum detection module selection unit for selecting an optimum detection model satisfying a predetermined requirement among the plurality of detection modules based on the throughput per second, the buffer limit length, and the compensation weight calculated for each detection module. Object detection device for dynamic detection module selection. The method of claim 4, wherein
The object detecting apparatus for selecting a dynamic detection module, wherein the optimum detection module selecting unit selects an optimal detection module m (t + 1) at a next time point t + 1 using the following equation.
[Equation]

Where m is an element of the detection module set, M is a detection module set, L is a time constraint (sec), V is a compensation weight,

Is the expected accuracy when using detection module m,

Is the input rate per second for time t + 1,

Is the expected output rate per second for time t + 1 when using detection module m,

Is the length of the current buffer. In the method for the object detection device for selecting a dynamic detection module,
If the object detection is requested, measuring the length of the buffer, which indicates the amount of data loaded into the buffer at the current time t;
Based on a preset detection module selection period, a length of the buffer, an input rate per second and an output rate per second of the current time t of each detection module, a next time point (t + 1) among the plurality of detection modules Dynamically selecting an optimal detection module of; And
Detecting an object from data based on the optimal detection module
Including a method for selecting a dynamic detection module.

Images