KR102380573B1 - Object detection apparatus and method to minimize end-to-end delay and advanced driver assistance device using the same - Google Patents

Abstract

The present invention relates to a device and a method for detecting an object to minimize an end-to-end delay, and an advanced driver assistance device using the same. The device for detecting an object comprises: a first thread processing unit taking a camera image during every first time section of each cycle; a second thread processing unit inferring the taken camera image during every second time section directly adjacent to the first time section of each cycle; and a third thread processing unit detecting whether the inferred camera image exists during every first time section of each cycle and displaying the inferred camera image when the inferred camera image exists.

Description

OBJECT DETECTION APPARATUS AND METHOD TO MINIMIZE END-TO-END DELAY AND ADVANCED DRIVER ASSISTANCE DEVICE USING THE SAME

The present invention relates to a real-time object detection technology, and more particularly, an object detection apparatus and method for minimizing an end-to-end delay that can reduce the time from the appearance of an object to recognizing the object through an object detector, and It relates to a state-of-the-art driver assistance system using this.

To realize safe autonomous driving, the vehicle's camera-based object detection system must detect dangerous obstacles on the road as quickly as possible to reduce the risk of collision. Therefore, there is a need to thoroughly analyze and optimize the delay between the appearance and detection of an object.

Since the most recent object detectors are based on deep neural networks (DNNs), many studies have focused on developing lightweight neural networks or hardware-accelerated techniques to reduce DNN inference delays.

However, there is relatively little interest in end-to-end delay analysis and optimization of real-time object detection systems, including inference delay as well as other pre- and post-processing delays such as image resizing or result display.

Korean Patent Publication No. 10-2009-0062881 (June 17, 2009)

An embodiment of the present invention provides an object detection apparatus and method for minimizing the end-to-end delay that can reduce the time from the appearance of an object to recognizing the object through an object detector, and a state-of-the-art driver assistance device using the same want to

An embodiment of the present invention is to provide an object detection apparatus and method for minimizing the end-to-end delay that can optimize the end-to-end delay time through a total of three steps through analysis of the system internal structure, and an advanced driver assistance device using the same do.

In embodiments, an apparatus for detecting an object for minimizing an end-to-end delay may include: a first thread processing unit that fetches a camera image during a first time period of each cycle; a second thread processing unit for inferring the fetched camera image for every second time section immediately adjacent to the first time section of each cycle; and a third thread processing unit configured to detect whether the inferred camera image exists during the first time period of each period, and if so, to display the inferred camera image.

The first thread processing unit may perform on-demand capture by directly receiving the camera image from the camera in the first time period without a separate memory buffer for storing the camera image.

The first thread processing unit may be performed based on a central processing unit (CPU) and may receive the camera image by directly accessing the camera during the first time period.

The first thread processing unit may receive the camera image from a start point of each cycle determined according to the normal operation of the camera and store the received camera image in a contention-free image buffer between threads.

The second thread processing unit may perform image processing on the fetched camera image at a point in time of the second time period as soon as the first time period ends.

The second thread processing unit is performed based on a GPU (Graphic Processing Unit), and in the second time period, object recognition on the fetched camera image can be performed without competing for a contention-free image buffer between the first thread processing unit and a thread. there is.

The second thread processing unit may complete the object recognition according to a deadline corresponding to a time point of the first time section next to each cycle through a neural network-based stage detector.

The third thread processing unit may display the inferred camera image during the first time period of each period in synchronization with a time period for fetching the camera image.

The third thread processing unit is performed based on a CPU (Central Processing Unit) and may highlight an object in the inferred camera image to be overlaid.

The third thread processing unit may analyze the object based on metadata about the object before the display, and highlight differently according to the type of the object.

The apparatus may further include a control unit that synchronizes each period and provides a time point of each of the first and second time intervals in each period.

In embodiments, the advanced driver assistance device may include: a first thread processing unit that fetches a camera image during a first time period of each cycle; a second thread processing unit for inferring the fetched camera image for every second time section immediately adjacent to the first time section of each cycle; and a third thread processing unit configured to detect whether the inferred camera image exists during the first time period of each period, and if so, to display the inferred camera image; and a controller for controlling the vehicle by determining the urgency of the vehicle while driving based on the inferred camera image.

When an error occurs at each time point of the first and second time sections in each cycle, the control unit passes the inference of the imported camera image to reset the synchronization of each cycle to determine the first time section of the first time section You can reset the time point.

The disclosed technology may have the following effects. However, this does not mean that a specific embodiment should include all of the following effects or only the following effects, so the scope of the disclosed technology should not be understood as being limited thereby.

The object detection apparatus and method for minimizing the end-to-end delay according to an embodiment of the present invention, and a state-of-the-art driver assistance device using the same can reduce the time from the appearance of an object to recognizing the object through the object detector. .

The object detection apparatus and method for minimizing the end-to-end delay according to an embodiment of the present invention, and the advanced driver assistance device using the same can optimize the end-to-end delay time through a total of three steps through the analysis of the internal structure of the system.

1 is a diagram illustrating a hardware-software architecture of an object detection system.
2 is a diagram illustrating the overall flow of an object detection system and associated delays.
3 is a diagram for explaining USB bandwidth reservation.
4 is a diagram for explaining transmission delay.
5 is a diagram illustrating a multi-threaded pipeline architecture.
6 and 7 are diagrams for explaining an end-to-end delay optimization method according to the present invention.
8 is a diagram for explaining and comparing execution times according to the presence or absence of contention for shared memory bandwidth.

Since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiment described in the text. That is, since the embodiment may have various changes and may have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, it should not be understood that the scope of the present invention is limited thereby.

On the other hand, the meaning of the terms described in the present application should be understood as follows.

Terms such as “first” and “second” are for distinguishing one component from another, and the scope of rights should not be limited by these terms. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected to the other component, but it should be understood that other components may exist in between. On the other hand, when it is mentioned that a certain element is "directly connected" to another element, it should be understood that the other element does not exist in the middle. Meanwhile, other expressions describing the relationship between elements, that is, "between" and "between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The singular expression is to be understood as including the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" or "have" refer to the embodied feature, number, step, action, component, part or these It is intended to indicate that a combination exists, and it should be understood that it does not preclude the possibility of the existence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

In each step, identification numbers (eg, a, b, c, etc.) are used for convenience of description, and identification numbers do not describe the order of each step, and each step clearly indicates a specific order in context. Unless otherwise specified, it may occur in a different order from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be embodied as computer-readable codes on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. . Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium is distributed in a computer system connected to a network, so that the computer-readable code can be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless otherwise defined. Terms defined in the dictionary should be interpreted as being consistent with the meaning of the context of the related art, and cannot be interpreted as having an ideal or excessively formal meaning unless explicitly defined in the present application.

1 is a diagram illustrating a hardware-software architecture of an object detection system.

Referring to FIG. 1 , the object detection system 100 may be implemented as a hardware-software architecture including a camera subsystem 110 and a detector subsystem 130 .

Camera subsystem 110 may include a single camera coupled directly to detector subsystem 130 using a Universal Serial Bus (USB) connector. The detector subsystem 130 may be implemented as a CPU-GPU heterogeneous computing platform.

In this case, the Linux operating system may be used to execute a darknet neural network framework based on a Pthread library and CUDA framework for managing CPU threads and GPU kernels, respectively.

Darknet frameworks can use pre-trained YOLO v2, v3 and v4 DNNs. Also, a V4L2 (Video For Linux Two)-based USB camera driver can be used as an interface between the two subsystems.

The detector subsystem 130 may display a bounding box around the detected object using the OpenCV library to visualize the object detection result.

2 is a diagram illustrating the overall flow of an object detection system and associated delays.

Referring to Figure 2, the object detection system 100 creates a camera subsystem 110, a detector subsystem 130, and a V4L2 camera driver queue (hereafter queue) for buffering images with associated delays. can be included, and the end-to-end flow of data between each configuration can be checked.

The object detection system 100 can be modeled as a single server queue of size Q. In this case, the distribution A of the image arrival interval and the distribution S of the object detection service interval in the queue may be utilized. Here, the service interval may be defined as the time between two consecutive requests to retrieve an image from the queue (ie, the start of an object detection cycle).

The object detection system 100 includes (i) Capture, (ii) Transfer, (iii) Queue, (iv) Fetch, (v) Inference, and (vi) ) Delay may occur depending on the six components of the display. That is, the corresponding components may determine the end-to-end delay of the object detection system 110 . Also, delay components can be defined independently, but some can be correlated to affect end-to-end delay.

- Camera Subsystem Analysis

The image may be captured by the camera at a frame rate of F, and the cycle time C of the camera between captures may correspond to 1/F. Thus, a newly emerged object can wait up to C until the next capture moment. However, since the object is associated with the service interval of the detector subsystem 130 , a longer delay may occur before being captured, i.e., a capture delay denoted by d _capt .

Meanwhile, the amount of data in the image frame may be determined by the resolution (X×Y) and the pixel format. Non-compressing pixel formats such as YUYV and RGBA may have a fixed bits-per-pixel (bpp) P. On the other hand, compressed pixel formats such as H264 and MJPEG may have different frame sizes depending on image content. Herein, the description will be made assuming an uncompressed pixel format. That is, assuming a fixed bpp, the size of the image frame represented by I may be calculated as I=(X·Y·P)/8 bytes.

Each captured image may be transferred to the detector subsystem 130 via a USB connection in an isochronous transfer mode that provides a known delay time according to a bandwidth reservation. On the other hand, the camera driver can interact with the camera to determine the available bandwidth. In this case, the bandwidth may be expressed as the number of bytes B usable during each USB microframe with a length of U=125 μs. The USB driver can use a buffer to group M microframes into one USB Request Block (URB) (ie, unit processing). 3 may represent an example of bandwidth reservation measured in an experimental platform using the YUYV pixel format with P=16.

In addition, considering the transmission delay d _tran corresponding to the time between image capture and queue entry, the required number of microframes can be calculated from the given I, B, and U and expressed as Equation 1 below.

[Equation 1]

Here, j is (i) waiting for the start of the next microframe after capture (worst case: U), and (ii) waiting for the current URB to be grouped (worst case: (M-1) U) by It corresponds to the generated jitter. In particular, the latter is expressed as URB buffering jitter. Consequently, j can have values in the range 0 ≤ j < M U. A constant (constant) 2 indicates an additional microframe including protocol overhead (overhead).

4 may represent a transmission delay in consideration of the corresponding jitter. Since the camera driver checks for image arrivals at that interval, the transmission delay can be determined as a multiple of the URB buffering period for M microframes. M·U = 32·125 μs = 4 ms may correspond to all rows in FIG. 3 , but may be different for each camera.

Therefore, A can be a discrete probability distribution having (usually) two values a ^min and a ^max , and the probability that the image arrival interval a is equal to one of these values can be defined as the following Equation 2, respectively.

[Equation 2]

p ₁ = Pr{a=a ^min }, p ₂ = Pr{a=a ^max }

이고, a^max =

에 해당한다. 즉, 각각 C의 바닥(floor)과 천장(ceiling)에 가장 가까운 M·U의 배수에 해당할 수 있다. 또한, a(E(a))의 기대값은 C로 근사될 수 있다. 즉, a^minp₁ + a^maxp₂

C에 해당할 수 있다. p₁ + p₂ = 1이므로 확률 분포는 주어진 C, M 및 U로부터 도출될 수 있다.where a ^min =

, and a ^max =

corresponds to That is, it may correspond to a multiple of M·U closest to the floor and ceiling of C, respectively. Also, the expected value of a(E(a)) can be approximated to C. That is, a ^min p ₁ + a ^max p ₂

It may correspond to C. Since p ₁ + p ₂ = 1, a probability distribution can be derived from given C, M and U.

When an image frame arrives at the head of the queue, it can be accepted or dropped by the queue. If image arrival is faster than retrieval, some drops may be unavoidable. In practice, a queuing decision can be made before the image has fully arrived. A corresponding determination may be made when the first URB buffering period including the image frame ends, and the time at which the determination is made may be expressed as a queuing decision point as shown in FIG. 4 . An arbitrary time interval exists between the actual image capture and the wait decision, which may correspond to the maximum URB buffering period (M·U). Also, it may be expressed as queuing decision jitter.

Taking into account the capture delay d _capt of an object, which is defined as the elapsed time between the physical appearance of the object and the successful capture, a successful capture would mean that the object was queued after the capture was sent. can A failed capture may be when the queue does not have enough space to accommodate the frame associated with the object and is dropped by the queue. Thus, d _capt may be associated with the service interval distribution S of the detector subsystem 130 . At the beginning of each object detection cycle, image frames are retrieved from the queue and space can be created for new image frames. The queuing decision point can play an important role in enqueuing frames. The frame can be accepted by the queue only when a waiting decision point occurs after that point.

The best case might be when the object appears just before the moment of capture (i.e. before a new capture cycle begins), d _capt = 0, and there is enough space in the queue to accommodate the image frame associated with the object. Conversely, the worst case is when an object appears immediately after the moment of capture and the queue is just full and the image just captured and all images captured subsequently are dropped by the queue until the queue becomes available again. may be applicable. In the worst case, d _capt can be expressed using the maximum number of continuously dropped image frames within the longest object detection service interval denoted by s ^max . Considering the influence of the maximum queuing decision jitter (M·U), the number of queuing decision points in s ^max can be calculated and the corresponding number can be predicted. The time interval determined by the number of waiting determination points may be shorter than the interval determined by the same number of capture points up to M·U. Therefore, in order to account for the maximum atmospheric determination jitter, s ^max can be summed with M·U when divided by C to obtain d _capt , which can determine the range of d _capt as in Equation 3 below.

[Equation 3]

와

로 표현되는 최소 및 최대 카메라 서브시스템 지연(camera subsystem delay)을 획득할 수 있다.By summing Equations 1 and 3, respectively, as in Equation 4 below

Wow

It is possible to obtain the minimum and maximum camera subsystem delay represented by .

[Equation 4]

- Detector Subsystem Analysis

The detector subsystem 130 may consist of three modules: (i) fetch, (ii) inference, and (iii) display. The service interval distribution S of the detector subsystem 130 can be identified after analyzing and profiling the three modules in the Nvidia Jetson AGX Xavier. Also, it does not assume other background workloads.

(i) Fetch: The fetch module can retrieve image frames from the queue. However, since V4L2 supports the streaming I/O method in which only the pointer to the buffer is exchanged between the driver and the application program, the actual memory copy for data transfer may not be performed. Resizing the input image to match the resolution of the DNN input layer can take most of the execution time. Therefore, the runtime e _fetch of the fetch module can be expressed as in Equation 5 below.

[Equation 5]

e _fetch ~ D _fetch (r _in , r _out )

Here, D _fetch (·) may correspond to a probability distribution that is a function of an input resolution (r _in ) and an output resolution (r _out ) for size adjustment. The delay of the import module expressed by d _fetch is defined as the elapsed time between release (start of the import thread) and complete (end of image resizing), and is expressed as Equation 6 below. can be

[Equation 6]

d _fetch = e _fetch + b _fetch

Here, b _fetch corresponds to a blocking time factor. Since no other background workload is assumed, the preemption delay may not be applied in d _fetch . Regarding b _fetch , a blocking system call may occur in the fetch module when an image is retrieved from the queue. More specifically, the fetch module may block until a new image arrives if it tries to fetch an image while the queue is empty.

(ii) Inference: The inference module can perform forward propagation in neural networks, which can be decomposed into CPU and GPU parts. In the CPU part, asynchronous GPU operations can be simply called through CUDA function calls, which can include data copy between CPU and GPU memory and GPU kernel functions for neural networks. Afterwards, GPU operations can be queued by being inserted into a CUDA stream, which is a FIFO (first-in-first-out) queue maintained by the CUDA runtime. A CPU thread can block until asynchronous GPU operations are complete by explicitly calling the CUDA synchronization function. Also, the last GPU operation may copy the detection result to CPU memory. For example, to perform single forward propagation, YOLOv3 may request 4 copy operations and 322 GPU kernel operations (including iterations). The execution time of the inference module may depend on the resolution of the input to the neural network r _nn . Accordingly, the CPU execution time e_infer^CPU and the GPU execution time e_infer^GPU of the inference module may be expressed as in Equation 7 below by using a random variable.

[Equation 7]

및

는 CPU 및 GPU 실행 시간들 각각에 대한 확률 분포에 해당한다. CPU 및 GPU 실행 시간들은 각각 r_nn에 따라 달라질 수 있다. 표기의 단순화를 위해 추론 모듈의 (통합된) 실행 시간 e_infer는 e_infer =

+

로 정의된다.From here,

and

corresponds to the probability distribution for each of the CPU and GPU execution times. CPU and GPU execution times may vary depending on r _nn , respectively. For simplicity of notation, the (integrated) execution time of the inference module e _infer is e _infer =

+

is defined as

Since there is no additional blocking factor, the delay d _infer of the inference module can be expressed as Equation 8 below.

[Equation 8]

d _infer = e _infer ~ D _infer (r _nn )

Here, D _infer (·) can be predicted as the convolution of D_infer^CPU and D_infer^GPU as shown in Equation 9 below.

[Equation 9]

(iii) Display: The display module can visualize the detection result by displaying an image with a bounding box around each detected object. To this end, the display module may perform the following steps. First, raw detection results having various confidence levels may be filtered with a given confidence threshold. Next, irrelevant duplicates may be joined (or removed) by a non-maximal suppression algorithm. Also, the display module may display a rectangle around each detected object. Accordingly, the execution time of the display step represented by e _disp may vary depending on the number of detected objects n _obj , and may be expressed as in Equation 10 below.

[Equation 10]

e _disp ~ D _disp (n _obj )

Here, the number of raw detections and the final number of detected objects may exhibit a very high positive correlation. Therefore, instead of using two variables when defining the probability distribution, only the final number of detected objects can be used. Also, there may be a blocking factor b _disp that is generated by calling an OpenCV function that renders an image on the screen. Accordingly, the delay d _disp of the display module may be expressed as in Equation 11 below.

[Equation 11]

d _disp = e _disp + b _disp

Here, for the analysis of the worst case, it may be assumed that the maximum number of detectable objects n _obj = 30.

- Multithreaded pipeline architecture

In Darknet, the above three modules can be configured in a multi-threaded pipeline architecture to utilize multi-core processors using a fork-join model. That is, there can be three parallel threads, one for each module: (i) a fetch thread, (ii) an inference thread, and (iii) a display thread. When the object detection cycle starts, the three threads branch and then join at the end of the loop so that the next cycle can start. 5 shows a timeline of a multi-threaded pipeline architecture. In general, for the i-th object detection cycle, while the (i-1)-th image frame is processed by the neural network, the i-th image frame is obtained and the (i-2)-th image can be displayed on the screen. Since the speculation is longest in FIG. 5 , the fetch and display threads may wait until the speculation thread completes at the end of each object detection cycle. Accordingly, the object detection service interval, generally expressed as s, can be expressed as in Equation 12 below.

[Equation 12]

s = max({d _fetch , d _infer , d _disp })

In this case, it may correspond to an object detection period depending on the situation. With respect to s, d _infer can be significantly longer than d _fetch and d _disp . Accordingly, the service interval distribution S can be assumed as in Equation 13 below.

[Equation 13]

S = D _infer (·)

However, in general, the service interval distribution S of the detector subsystem 130 according to Equation 12 may be determined by a possible combination of values of three random variables {d _fetch , d _infer , d _disp }. That is, the distribution can be defined as the set of the maximum of these three values for each of all possible combinations. Assuming that d _fetch = A, d _infer = B, and d _disp = C, the corresponding probability can correspond to the product of three probabilities. That is, it may correspond to Pr{d _fetch = A} × Pr{d _infer = B} × Pr{d _disp = C}. The final S can be calculated by summing all probabilities for a given service interval. The delay of the detector subsystem 130 expressed as d _detector based on S can be expressed as in Equation 14 below.

[Equation 14]

d _detector = 2s + d _disp

As shown in FIG. 5 , it may correspond to two object detection cycles and a delay time of the display thread. Then, the minimum and maximum detector subsystem delays can be calculated as in Equation 15 below.

[Equation 15]

- Queue Delay

Based on the image arrival interval distribution A and the object detection service interval distribution S, we can analyze the queue delay d _queue , which is defined as the time each image frame spends in the queue. Darknet can use OpenCV VideoCapture library for queue management with default queue size Q = 4. A full queue cannot accommodate additional received image frames. The following three cases need to be considered for d _queue analysis.

(i) Case 1: min(A) > max(S). In this case, the object detection service interval may always be shorter (faster) than the image arrival interval. Then, from the perspective of the fetching thread, whenever it tries to retrieve an image frame, the queue is empty, so the fetching thread may block until the image frame arrives. Therefore, it can be expressed as Equation 16 below.

[Equation 16]

d _queue = 0

In this case, b _fetch can eventually be extended until d _fetch = b _fetch + e _fetch = a. Here, a corresponds to the image arrival interval. Consequently, the object detection cycle time can be adjusted by the image arrival interval regardless of d _infer and d _disp .

(ii) Case 2: max(A) < min(S). In this case, the image arrival interval is always shorter (faster) than the object detection service interval, so each time the fetching thread tries to retrieve an image frame, the queue can fill up. Therefore, it can always correspond to b _fetch = 0. In the case of queuing delay, Q object detection cycles (i.e. fetches) may be required after an image enters the end of the queue until it is dequeued. Thus, in the worst case, approximately Q·s ^max may be needed. More precisely, the maximum queue delay can be calculated as in Equation 17 below.

[Equation 17]

의 값은 다음의 수학식 18과 같이 산출될 수 있다.The subtracted portion may be the minimum time elapsed between the start of the object detection cycle (ie, freeing up space to accommodate the newly captured image) and the actual arrival of the image frame. The maximum atmospheric decision jitter (M·U) needs to be taken into account because the amount of image frames can be transmitted after capture, even before the object detection cycle begins. best case

The value of can be calculated as in Equation 18 below.

[Equation 18]

This case can occur when an object appears immediately past the moment of capture and has been successfully captured at the moment of capture. Then a delay of up to C can be taken into account in the capture delay instead of the queue delay. In this case, it can be assumed that the queuing decision jitter is zero to calculate the minimum queue delay.

= 0이거나 최악의 경우 사례 2에서와 같이 수학식 17 만큼 증가할 수 있다.(iii) Case 3: All other conditions. In this case, the queue state may be non-deterministic. A long-term average can only be estimated if the means of the two distributions are equal. Also, since the queue length is limited within [0, Q], the queue state at a specific moment cannot be predicted. Therefore, the queue delay is the best case as in case 1

= 0 or, in the worst case, may increase by Equation 17 as in Case 2.

Hereinafter, an approach for minimizing the end-to-end delay in the object detection system 130 will be described. More specifically, about three technologies: (i) On-demand Capture, (ii) Zero-slack Pipeline, and (iii) Contention-free Pipeline. Explain.

6 and 7 are diagrams for explaining an end-to-end delay optimization method according to the present invention, and FIG. 8 is a diagram for comparing execution times according to the presence or absence of contention for shared memory bandwidth.

6 and 7 may show how these techniques progressively reduce the end-to-end delay from the original (vanilla) architecture to the final architecture. The effectiveness of the optimization method may depend on the relative execution time of the three threads. For convenience of explanation, it is assumed that d _infer is much longer than d _fetch and d _disp as shown in Fig. (a). In this case, the object detection cycle time s is dominated by d _infer , which may be the most common case in embedded systems.

A. On-demand Capture

In Equations 17 and 18, it is possible to reduce the queue delay by reducing the queue size Q (especially for case 2 and partially case 3). Even in case 1, reducing Q already works as if there were no queues, so there are no negative side-effects. This means that you can completely remove the queue to receive image frames only when needed. This is called the on-demand capture method.

In the initialization phase, a fixed queue size Q is requested from the driver with the ioctl command (REQBUFS). The Q empty buffers, i.e. image holders, can then be queued using the ioctl command (QBUF). After that, the import loop can start. It first checks for the existence of an available image frame by making a selection system call. Upon returning from the select system call, you can use the ioctl command (DQBUF) to dequeue buffers containing images and use the ioctl command (QBUF) to requeue empty buffers. Immediately start the next loop.

Figure (a) shows a vanilla architecture with queues, while figure (b) shows an on-demand capture architecture. Comparing the two shows that the capture on demand architecture no longer has queuing delays. However, blocking time b _fetch may occur when making a selection system call.

More specifically, the minimum and maximum blocking times may be determined as follows.

[Equation 19]

It is the same as the part subtracted from Equations 17 and 18, which may correspond to the waiting time from the start of the object detection cycle to arrival at the queue even if a new image frame to be used later is not blocked, respectively. Conversely, in an on-demand capture architecture, if the fetch thread is blocked and unblocked for the same amount of time as the latency, the arriving image frames can be fed directly to the fetch thread without queuing delays.

In Figure (b), the queuing delay can be completely eliminated by the on-demand capture method described above. in other words,

[Equation 20]

d _queue = 0

As a side effect, in the figure d _camera and d _fetch are somewhat overlapped by b _fetch . This is because the fetch thread is initially blocked waiting for the image to arrive for b _fetch . Therefore, when calculating the detector subsystem delay, the overlapped time (b _fetch ) already calculated in d _camera must be subtracted from the original delay equation in Equation 15.

[Equation 21]

B. Zero-slack Pipeline

In Figure (b), a significant difference in idle time can be observed between the end of the fetch and the end of the object detection cycle due to the different lengths of the three pipeline stages (eg d _fetch , d _infer and d _disp ). If this idle time can be eliminated, the end-to-end latency can be further reduced. To this end, the concept of margin and offset of the import thread is explained as follows.

-fetch thread slack(δfetch): the time interval between the end of the fetch thread and the end of the object detection cycle

-Fetch thread offset(θfetch): Each time the fetch thread is executed, it can be artificially and uniformly delayed.

Taking the offset fetch into account, d _fetch can be increased as follows.

[Equation 22]

d _fetch = θ _fetch + e _fetch + b _fetch

Then slack can be computed as

[Equation 23]

δ _fetch = s - d _fetch

Here, s represents the object detection cycle time.

Therefore, as shown in Figure (c), θ _fetch can be increased until δ _fetch reaches 0 (zero-slack). This will eventually eliminate the idle time interval. Then the detector subsystem delay can be further reduced from Equation (21) as

[Equation 24]

However, if too many θ _fetches are applied, the object detection cycle itself may increase. Therefore, it is necessary to find the optimal θ _fetch that minimizes possible end-to-end delay without increasing s. To this end, θ _fetch can be conservatively determined as follows.

[Equation 25]

s_min)에 거의 영향을 주지 않는다. 보수적인 결정으로 인해 모든 유휴 시간 차이를 완전히 제거할 수는 없다. 그러나, 적절한 θ_fetch를 사용하는 제로 슬랙 파이프 라인은 종단 간 지연을 어느 정도 줄여줄 수 있다.The increased d _fetch is the minimum object detection cycle time s ^min (i.e. max(d _fetch )

s _min ) has little effect. Conservative decisions cannot completely eliminate all idle time differences. However, a zero-slack pipeline with proper θ _fetch can reduce the end-to-end delay to some extent.

C. Contention-free Pipeline

When multiple threads run on shared memory multi-core processors, significant memory bandwidth contention can occur between concurrent threads. Moreover, the effect can be even worse when CPU threads run from the integrated GPU to the GPU kernel. In order to quantify the effect of memory contention, FIG. 8 shows the distribution of execution time according to the presence or absence of contention. Contention-free execution time is measured in vanilla architectures, whereas contention-free execution time can be measured during sequential execution of the object detection phase. In particular, the figure (b) of FIG. 8 shows that the execution time (about 28 ms) is greatly increased when the GPU is executed in the inference thread. In contrast, Fig. 8 (a) and Fig. (c) show a relatively small effect. It is clear from the figure that the inference thread suffers the most from memory bandwidth contention issues. Thus, while isolating the inference threads to minimize contention, the fetch and display threads can run concurrently as in the original architecture. Figure (d) shows the new zero-fault pipeline architecture.

로 변경될 수 있다.In Fig. (d), the object detection cycle time is defined as

can be changed to

[Equation 26]

로의 증가는 미미할 수 있다. 그런 다음 검출기 서브시스템(130) 지연이 최종적으로 다음과 같이 감소할 수 있다.However, since d _infer is also reduced by contention-free execution, in s

The increase in furnace may be negligible. Then the detector subsystem 130 delay can finally be reduced as follows.

[Equation 27]

The delay element of the object detection system 100 according to the present invention is composed of a camera subsystem, a queue, and a detector subsystem. The delay by the camera subsystem means the time until the object appears and the image frame with the object arrives in the queue, and the delay by the detector subsystem is the time until the image fetched from the queue is passed through Darknet YOLO and the object on the image is recognized. means time. Finally, the delay by the queue means the time the image supplied from the camera subsystem stays in the queue until consumed by the detector subsystem.

As a result of analyzing the internal structure of the system, it was confirmed that there are delays due to unnecessary queues, delays due to imbalance in execution time between Darknet YOLO internal stages, and delays due to hardware resource contention. It is possible to optimize the end-to-end delay time through the slack pipeline structure, III. Contention-free pipeline structure).

First, when an image is needed in Darknet YOLO, it can be modified by requesting and receiving the image from the camera driver, thereby eliminating unnecessary delay caused by the queue. Second, the idle time caused by the imbalance in the execution time can be eliminated by executing the stage with a short execution time later. Lastly, if the stage using GPU and the stage using CPU among Darknet YOLO internal stages are executed in parallel, memory bandwidth interference will adversely affect the execution time of the stage using GPU. Stages that use the CPU to remove can be run in parallel. In other words, the stage using the GPU is protected and executed alone.

On the other hand, each optimization method according to the present invention was verified using Darknet YOLO (v1, v2, v3) on the Jetson AGX Xavier platform, and it was confirmed that the average end-to-end delay time based on v3 was reduced by 76% compared to the Original. Additionally, the optimization method according to the present invention may not affect the object recognition accuracy because only the system structure is changed without changing the neural network structure.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

100: object detection system
110: camera subsystem
130: detector subsystem

Claims (14)

a first thread processing unit that fetches a camera image during a first time period of each cycle;
a second thread processing unit for inferring the fetched camera image for every second time section immediately adjacent to the first time section of each cycle; and
a third thread processing unit that detects whether the inferred camera image exists during the first time period of each period, and if so, displays the inferred camera image;
The first thread processing unit to minimize the end-to-end delay, characterized in that the on-demand capture to receive the camera image directly from the camera in the first time period without a separate memory buffer for storing the camera image object detection device.
delete The method of claim 1, wherein the first thread processing unit
An object detection apparatus for minimizing end-to-end delay, which is performed based on a CPU (Central Processing Unit) and directly accesses the camera during the first time period to receive the camera image.
The method of claim 3, wherein the first thread processing unit
The object detection apparatus for minimizing end-to-end delay, characterized in that the camera image is received from the start time of each cycle determined according to the normal operation of the camera and stored in a contention-free image buffer between threads.
The method of claim 1, wherein the second thread processing unit
The object detection apparatus for minimizing the end-to-end delay, characterized in that as soon as the first time period ends, the image processing on the imported camera image is performed at the time point of the second time period.
5. The method of claim 4, wherein the second thread processing unit
End-to-end delay, which is performed based on GPU (Graphic Processing Unit) and performs object recognition on the fetched camera image without competing with the contention-free image buffer between the first thread processing unit and the thread in the second time period object detection device to minimize
The method of claim 6, wherein the second thread processing unit
The object detection apparatus for minimizing end-to-end delay, characterized in that through a neural network-based stage detector, the object recognition is completed according to a deadline corresponding to the time point of the first time section next to each cycle.
The method of claim 1 , wherein the third thread processing unit
The object detection apparatus for minimizing end-to-end delay, characterized in that the inferred camera image is displayed during the first time period of each period in synchronization with a time period for obtaining the camera image.
The method of claim 8, wherein the third thread processing unit
An object detection apparatus for minimizing end-to-end delay, characterized in that it is performed based on a CPU (Central Processing Unit) and highlights an object in the inferred camera image to be overlaid.
10. The method of claim 9, wherein the third thread processing unit
The object detection apparatus for minimizing the end-to-end delay, characterized in that before the display, the object is analyzed based on the metadata about the object and highlighted according to the type of the object.
According to claim 1,
The apparatus for minimizing the end-to-end delay according to claim 1, further comprising: a control unit that synchronizes each period and provides a time point of each of the first and second time sections in each period.
retrieving a camera image during a first time period of each cycle;
inferring the imported camera image for every second time section immediately adjacent to the first time section of each period; and
detecting whether the inferred camera image exists every first time interval of the period, and if so, displaying the inferred camera image;
The step of retrieving the camera image includes performing on-demand capture of directly receiving the camera image from the camera in the first time period without a separate memory buffer for storing the camera image. An object detection method to minimize inter-delay.
a first thread processing unit that fetches a camera image during a first time period of each cycle;
a second thread processing unit for inferring the fetched camera image for every second time section immediately adjacent to the first time section of each cycle; and
a third thread processing unit that detects whether the inferred camera image exists during the first time period of each period, and if so, displays the inferred camera image; and
A control unit for controlling the vehicle by determining the urgency of the vehicle while driving based on the inferred camera image,
and the first thread processing unit performs on-demand capture by directly receiving the camera image from the camera in the first time period without a separate memory buffer for storing the camera image.
14. The method of claim 13, wherein the control unit
If an error occurs at each time point of the first and second time sections in each cycle, the inference of the imported camera image is passed to reset the synchronization of each cycle to reset the time point of the first time section Advanced driver assistance systems, characterized in that

Images