KR20220063544A - Apparatus for predicting traffic line of box-level multiple object using only position information of box-level multiple object - Google Patents

Abstract

An apparatus for predicting traffic lines of multiple box-level objects using only positional information of the multiple objects according to a preferred embodiment of the present invention can quickly predict movements of the multiple objects via light operation by receiving only box-level positional information of the multiple objects as input through a long short-term memory (LSTM) network structure and predicting future traffic lines of the multiple objects at a box level. In addition, the apparatus can predict the movements of multiple objects even in situations where the objects whose movements are to be predicted are hidden by other surroundings or where the objects overlap each other by predicting the positions of the multiple objects using only the box-level positional information of the multiple objects.

Description

Apparatus for predicting traffic line of box-level multiple object using only position information of box-level multiple object

The present invention relates to a box-level multi-object motion prediction apparatus using only box-level location information of multiple objects, and more particularly, to an apparatus for predicting the future location of multiple objects based on past locations of multi-objects. .

Conventional techniques for performing multiple object tracking (MOT) or trajectory prediction for analyzing the motion of multiple objects require a lot of execution time. This is because motion is analyzed for each pixel-level by considering not only the relationship between various objects to be analyzed, but also the relationship between the background and objects except for the object. The task of analyzing the motion of these multi-objects is usually used in devices for accident prevention in the autonomous movement of mechanical devices such as autonomous vehicles or robots, and the closer to real time, the more natural and faster the motion of the mechanical device.

Looking at the recent changes in the autonomous movement of mechanical devices, we first look around us through the camera. It then predicts changes in its surroundings through algorithms within the device. If a problem situation such as a collision can occur in the predicted change, it will give an operation command to the mechanical device in a direction that does not occur as much as possible, and the mechanical device will move according to this command. At this time, most cameras looking around are fast enough to shoot 60 frames per second. Since most of the operation commands are also driven by electrical signals, the movement of mechanical devices can be quickly changed in the event of an accident. The part that makes the movement of the mechanical device unnatural and slows the movement is the operation of the algorithm that predicts the change in the surroundings. Therefore, if the operation of the algorithm itself becomes lighter, the responsiveness to the situation of the mechanical device can be increased.

In addition, since the prior art analyzes the motion of multiple objects based on images, it is difficult to follow the existing objects as they are in a situation where the objects are hidden.

An object of the present invention is to receive only box-level position information of multiple objects as input through a long short term memory (LSTM) structure, which is often used for future prediction of time series data, and box the future multi-object movement. - To provide a box-level multi-object movement line prediction apparatus using only box-level position information of multi-objects that predicts by level.

Other objects not specified in the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

The box-level multi-object movement line prediction apparatus using only box-level position information of multiple objects according to a preferred embodiment of the present invention for achieving the above object, using only box-level position information for multiple objects of training data , a learning unit for learning a multi-object position prediction model consisting of a long-term memory neural network (LSTM); and input box-level position information of multiple objects for a preset number of consecutive past frames to the multi-object position prediction model, and to a preset number of consecutive future frames output from the multi-object position prediction model. and a prediction unit for predicting the future movement of multiple objects based on box-level position information of multiple objects.

Here, the box-level object location information may include coordinate values of four corners of a bounding box surrounding the object.

Here, with respect to the learning video included in the learning data, the learning unit fills a matrix of the form of a preset maximum number of objects x 8 by using the box-level position information of the multiple objects with respect to the frame, A matrix for a frame may be obtained by padding the remaining part with 0, and the multi-object position prediction model may be learned using the obtained matrix.

Here, the predictor, for each of a preset number of consecutive past frames, fills the position of the multiple objects in a matrix of the form of a preset maximum number of objects x 8 by using box-level position information of the multiple objects for the past frame, , obtain a matrix for a past frame by padding the rest of the matrix with 0, input a plurality of matrices corresponding to each of a preset number of consecutive past frames into the multi-object position prediction model, and the multi-object position By using a plurality of matrices output from the prediction model, box-level position information of multiple objects for a preset number of consecutive future frames may be obtained.

According to the box-level multi-object movement prediction apparatus using only box-level position information of multiple objects according to a preferred embodiment of the present invention, the box-level positions of multiple objects through a long short term memory (LSTM) structure By receiving only information as input and predicting the future multi-object movement at box-level, it is possible to quickly predict the movement of multiple objects through light calculations.

In addition, by predicting the positions of multiple objects using only the box-level position information of multiple objects, the motion of multiple objects can be predicted even in a situation in which the object to be motion predicted is obscured by other environments or overlaps with each other.

Effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating an apparatus for predicting a box-level multi-object motion using only box-level position information of multiple objects according to a preferred embodiment of the present invention.
Figure 2 is a view for explaining the position of a box-level object according to a preferred embodiment of the present invention, Figure 2 (a) is a method showing the position of the conventional bounding box, Figure 2 (b) is the present invention This is a method of indicating the position of the bounding box according to
3 is a diagram for explaining an example of a multi-object position prediction model according to a preferred embodiment of the present invention.
4 is a diagram for explaining the performance of an apparatus for predicting a box-level multi-object motion using only box-level position information of multiple objects according to a preferred embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments published below, but may be implemented in various different forms, and only these embodiments make the publication of the present invention complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly defined in particular.

In the present specification, terms such as “first” and “second” are for distinguishing one component from other components, and the scope of rights should not be limited by these terms. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

In the present specification, identification symbols (eg, a, b, c, etc.) in each step are used for convenience of description, and identification symbols do not describe the order of each step, and each step is clearly Unless a specific order is specified, the order may differ from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

In this specification, expressions such as “have”, “may have”, “include” or “may include” indicate the existence of a corresponding feature (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In addition, the term '~ unit' as used herein means software or a hardware component such as a field-programmable gate array (FPGA) or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. '~' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, '~' refers to components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data structures and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'.

Hereinafter, a preferred embodiment of a box-level multi-object movement line prediction apparatus using only box-level location information of multi-objects according to the present invention will be described in detail with reference to the accompanying drawings.

First, an apparatus for predicting a box-level multi-object movement line using only box-level position information of multiple objects according to a preferred embodiment of the present invention will be described with reference to FIGS. 1 to 3 .

1 is a block diagram illustrating an apparatus for predicting a box-level multi-object movement using only box-level position information of multiple objects according to a preferred embodiment of the present invention, and FIG. 2 is a box-level according to a preferred embodiment of the present invention. As a diagram for explaining the position of a level object, Fig. 2 (a) is a method showing the position of a conventional bounding box, and Fig. 2 (b) is a method showing the position of the bounding box according to the present invention, Fig. 3 is a diagram for explaining an example of a multi-object position prediction model according to a preferred embodiment of the present invention.

Referring to FIG. 1 , an apparatus 100 for predicting a box-level multi-object movement using only box-level position information of multiple objects according to a preferred embodiment of the present invention includes a long- and short-term memory neural network (long and short-term memory) that is frequently used for future prediction of time series data. Through the short term memory (LSTM) structure, only the box-level position information of multiple objects is received as input and the future multi-object movement is predicted at the box-level.

Here, the box-level object position information may be composed of coordinate values of four corners of a bounding box surrounding the object. That is, the present invention, as shown in (a) of FIG. 2, is information consisting of the center coordinates (x, y), height (h) and width (w) of the bounding box surrounding the object "(x, y)" , w, h)" is not used as the box-level object position, and as shown in FIG. 2 (b), the (x, y) coordinate values of the four corners of the bounding box surrounding the object are sequential The information "(x ₁ , y ₁ , x ₂ , y ₂ , x ₃ , y ₃ , x ₄ , y ₄ )" is used as the box-level object position. Accordingly, it can be used to detect not only an axially aligned bounding box (Axis-Aligned Bounding Box, AABB) but also an oriented bounding box (OBB).

To this end, the box-level multi-object movement prediction apparatus 100 may include a learning unit 110 and a prediction unit 130 .

The learning unit 110 learns a multi-object location prediction model made of a long-short-term memory neural network (LSTM) using only box-level location information for multi-objects of the training data.

At this time, the learning unit 110 fills a matrix of the form of a preset maximum number of objects x 8 by using the box-level position information of the multiple objects for the frame with respect to the learning video included in the training data, and the matrix It is possible to obtain a matrix for a frame by padding the remaining part of ? with 0, and learn a multi-object position prediction model using the obtained matrix. Here, the preset maximum number of objects has a value greater than the maximum number of objects that can be detected in units of frames among frames of a moving picture, and may be, for example, 1,000.

For example, since the number of objects per frame in the learning video is not constant, the maximum number of objects per frame (eg, 1,000, etc.) is set in advance, and the preset maximum number of objects using box-level position information of multiple objects for the frame is used. After filling the corresponding positions of the detected multiple objects in the object count x 8 matrix, the remaining part of the matrix, that is, the (preset maximum number of objects - the number of detected multiple objects) x 8 matrix is padded with zeros. Finally, a matrix of the maximum number of objects x8 preset in the multi-object position prediction model is entered as an input vector, and an output vector is also a matrix of the preset maximum number of objects x8.

The prediction unit 130 predicts the future positions of multiple objects based on the input video through the multi-object position prediction model learned through the learning unit 110 .

That is, the prediction unit 130 may input the box-level position information of multiple objects for a preset number (eg, three, etc.) of consecutive past frames to the multi-object position prediction model. The conventional multi-object position prediction method uses a part of a frame of a moving picture and box information (or patches cut from an original image using box information) for each frame as an input vector of a long-term memory neural network (LSTM). On the other hand, the present invention uses box-level position information for multiple objects instead of images as input vectors.

And, the prediction unit 130 is based on the box-level position information of multiple objects for a preset number (eg, three, etc.) of consecutive future frames output from the multi-object position prediction model, the future of the multi-object can predict the route of

In this case, for each of the preset number of consecutive past frames, the prediction unit 130 determines the positions of the multiple objects using box-level position information of the multiple objects for the past frame in a preset maximum number of objects x 8 matrix. to obtain a matrix for the past frame by padding the rest of the matrix with 0, and input a plurality of matrices corresponding to each of a preset number of consecutive past frames into the multi-object position prediction model, and the multi-object position By using a plurality of matrices output from the prediction model, box-level position information of multiple objects for a preset number of consecutive future frames may be obtained. Here, the preset maximum number of objects has a value greater than the maximum number of objects that can be detected in units of frames among frames of a moving picture, and may be, for example, 1,000.

For example, as shown in FIG. 3 , the multi-object position prediction model according to the present invention uses box-level position information for multiple objects instead of images as input vectors and output vectors. The multi-object position prediction model receives box-level position information of multiple objects for three consecutive frames in the past, and predicts box-level position information of multiple objects for three consecutive frames thereafter. That is, since the number of objects per frame in the video is not constant, the maximum number of objects per frame (eg, 1,000, etc.) is preset and the maximum object preset using box-level position information of multiple objects for the frame. After filling the corresponding positions of the detected multi-objects in the number x 8 matrix, the remaining part of the matrix, that is, the (preset maximum number of objects - the number of detected multi-objects) x 8 matrix is padded with zeros. Finally, a matrix of the maximum number of objects x8 preset in the multi-object position prediction model is entered as an input vector, and an output vector is also a matrix of the preset maximum number of objects x8.

Then, the performance of the box-level multi-object movement line prediction apparatus using only the box-level position information of the multi-objects according to a preferred embodiment of the present invention will be described with reference to FIG. 4 .

4 is a diagram for explaining the performance of an apparatus for predicting a box-level multi-object motion using only box-level position information of multiple objects according to a preferred embodiment of the present invention.

In order to test the performance of the box-level multi-object motion prediction apparatus using only the box-level location information of multiple objects according to the present invention, the MOT16 data set (Milan, A., et al. al. "MOT16: A benchmark for multi-object tracking. arXiv 2016." arXiv preprint arXiv:1603.00831 9.) was used for the experiment.

The upper three columns shown in FIG. 4 are images expressing all the box information of multiple objects for a frame that entered the multi-object position prediction model as an input, and the lower three columns shown in FIG. 4 are output from the multi-object position prediction model. It is an image expressing all the box information of multiple objects per frame. Both top and bottom are expressed in chronological order from left to right.

Although the correct answer image overlaps the predicted box location a lot and is not shown in FIG. 4, it was confirmed that the predicted box location according to the present invention matches the correct answer location well. In addition, the data expressed the data set with good visibility among many data sets. In the case of other images, the box could not be displayed as a drawing because there were too many objects, but even in this case, the position of the box showed satisfactory accuracy and It was confirmed that steady accuracy was shown even in the obscured situation.

In addition, [Table 1] below measures the time it takes to predict the next three frames when the motion of multiple objects is predicted for three frames thereafter while controlling the number of frames used as input.

Method (number of input frames) Time required (ms) 2 0.037 Three 0.040 4 pieces 0.054 5 pieces 0.076

We usually talk about 30 fps as real-time motion. In other words, for real-time prediction, calculations must be made within at least 33ms for three frames. The values in [Table 1] correspond to approximately 1/1000 of this value. Therefore, according to the present invention, even if the camera input is received at a higher speed, it can be processed. In the present invention, although the accuracy may be somewhat lower than that of the conventional methods that directly use an image, in that it can respond more instantaneously, it can play a larger role in practically applying the prediction technology.

Even though all the components constituting the embodiment of the present invention described above are described as being combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, all of the components may be implemented as one independent hardware, but a part or all of each component is selectively combined to perform some or all of the functions of the combined hardware in one or a plurality of hardware program modules It may be implemented as a computer program having In addition, such a computer program is stored in a computer readable media such as a USB memory, a CD disk, a flash memory, etc., read and executed by a computer, thereby implementing an embodiment of the present invention. The recording medium of the computer program may include a magnetic recording medium, an optical recording medium, and the like.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains may make various modifications, changes and substitutions within the scope without departing from the essential characteristics of the present invention. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are for explaining, not limiting, the technical spirit of the present invention, and the scope of the technical spirit of the present invention is not limited by these embodiments and the accompanying drawings . The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: box-level multi-object movement prediction device,
110: learning department;
130: prediction unit

Claims (4)

a learning unit for learning a multi-object location prediction model made of a long-term memory neural network (LSTM) using only box-level location information for multi-objects of the training data; and
Input the box-level position information of multiple objects for a preset number of consecutive past frames into the multi-object position prediction model, and for a preset number of consecutive future frames output from the multi-object position prediction model a prediction unit for predicting a future movement of multiple objects based on box-level location information of multiple objects;
A box-level multi-object movement line prediction apparatus using only box-level position information of multiple objects including In claim 1,
The box-level object location information,
Consists of the coordinate values of the four corners of the bounding box surrounding the object,
A box-level multi-object motion prediction apparatus using only the box-level position information of multiple objects. In claim 2,
The learning unit,
With respect to the training video included in the training data, the positions of multiple objects are filled in a matrix of a preset maximum number of objects x 8 using box-level position information of multiple objects with respect to a frame, and the rest of the matrix is padded with 0 to obtain a matrix for the frame, and to learn the multi-object position prediction model using the obtained matrix,
A box-level multi-object motion prediction apparatus using only the box-level position information of multiple objects. In claim 2,
The prediction unit,
For each of a preset number of consecutive past frames, the positions of multiple objects are filled in a matrix of the preset maximum number of objects x 8 using box-level position information of multiple objects for the past frame, and the rest of the matrix is Obtaining a matrix for the past frame by padding with 0, inputting a plurality of matrices corresponding to each of a preset number of consecutive past frames to the multi-object location prediction model, and outputting a plurality of matrices from the multi-object location prediction model Obtaining box-level position information of multiple objects for a preset number of consecutive future frames using a matrix of
A box-level multi-object motion prediction apparatus using only the box-level position information of multiple objects.

Images