KR102082254B1 - a vehicle recognizing system - Google Patents

Abstract

A vehicle recognition system is disclosed. The vehicle recognition system according to an embodiment of the present invention includes an image input unit for receiving a traffic monitoring image, a control unit for recognizing a vehicle from an image received by the image input unit, and an output unit for outputting a result recognized by the control unit. The controller extracts a feature from the input image, the feature extractor generates a bounding box based on the extracted feature, creates and corrects a bounding box, and generates an image in which three sides of the bounding box are projected onto one surface. And a modeling unit configured to recognize a vehicle through a vehicle recognition model by using the projection surface generator and an image generated by the projection surface generator as an input.

Description

Vehicle recognizing system

The present invention relates to a vehicle recognition system. Specifically, the present invention relates to a vehicle recognition system for generating a bounding box for a detected vehicle and using the same as an input value to a deep learning based estimation model to recognize a vehicle.

Recently, traffic cameras are rapidly increasing in many large cities for the purpose of traffic safety and monitoring. Thus, the development of an intelligent traffic surveillance system based on computer vision work for the automatic monitoring of numerous cameras. Among the methods recently adopted in the surveillance camera market, for license plate recognition, vehicle detection and traffic measurement, automated systems are built that operate on their own without human supervision.

In addition, the vehicle subcategory recognition method should be embedded in an intelligent traffic monitoring system. When tracking a speeding or escaping vehicle, it is possible to track the suspect's vehicle using the vehicle model if the license plate is not known. The vehicle manufacturer can also check the frequency of passage for a particular model. Therefore, there is a need for a concrete recognition of a vehicle model in various fields. However, until now, the method of recognizing a vehicle model has been mainly focused on the size of the vehicle (small, medium, large) or the front of the vehicle.

Zeiler, M. D., & Fergus, R. (2014, September). Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818-833). Springer International Publishing. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097 ?? 1105. Curran Associates, Inc., 2012. Dubsk ㅱ, Mark ㅹ ta, Adam Herout, and Jakub Sochor. "Automatic Camera Calibration for Traffic Understanding." BMVC. 2014.

The vehicle recognition system according to an exemplary embodiment of the present invention is for recognizing a vehicle using a boundary box generated based on feature information.

In more detail, the present invention is to recognize information on sub-categories (for example, vehicle model, model, and manufacturer) that are not recognized by the existing vehicle recognition system.

The vehicle recognition system according to an exemplary embodiment of the present disclosure includes an image input unit for receiving a traffic monitoring image, a controller for recognizing a vehicle from an image received by the image input unit, and an output unit for outputting a result recognized by the controller. The controller extracts a feature from the input image, the feature extractor generates a bounding box based on the extracted feature, creates and corrects a bounding box, and generates an image in which three sides of the bounding box are projected onto one surface. And a modeling unit configured to recognize a vehicle through a vehicle recognition model by using the projection surface generator and an image generated by the projection surface generator as an input.

According to an embodiment of the present invention, the vehicle recognition system may recognize even lower layer information of the vehicle.

In addition, the vehicle recognition system according to an embodiment of the present invention is economical by using an existing image as it is, without using additional measurement equipment to recognize lower layer information.

1 illustrates a step of generating an average feature map.
2 shows three test examples with rectangular curtains. The heat map extraction shows the mean of the response map as the curtain is used for each sample.
3 shows an important part of the vehicle extracted.
4 is a flowchart illustrating a vehicle recognition algorithm according to an embodiment of the present invention.
5 is a block diagram of a vehicle recognition system according to an exemplary embodiment.
6 shows an example of a bounding box and a projected image.

Hereinafter, with reference to the drawings will be described in detail a specific embodiment of the present invention. However, the spirit of the present invention is not limited to the following embodiments, and those skilled in the art who understand the spirit of the present invention can easily add, change, delete, or add other components that fall within the scope of the same spirit. It may be proposed, but this will also be included within the scope of the present invention.

In the accompanying drawings, in order to easily express the spirit of the present invention, in describing the overall structure, minute parts may not be specifically described, and in describing the minute parts, the overall structure may not be specifically reflected. In addition, even if the specific parts, such as an installation position, are different, when the action is the same, the same name is given, and the convenience of understanding can be improved. In addition, when there exists a plurality of the same structure, only one structure is demonstrated, and the same description is applied about another structure, and the description is abbreviate | omitted.

As described above, in order to solve the existing problem, in the present invention, the layout of the interior structure to be restored is estimated, and the 3D restoration accuracy is improved by using the same. That is, in the present invention, the interior is restrained by restraining the interior structure so that the floor and the ceiling are parallel and perpendicular to the wall.

Compared with the general monitoring method, the vehicle subcategory recognition method has more limitations than other methods due to the unique characteristics of the vehicle that cannot be found in other objects. First, there is no big difference in the design style of the vehicle between different classes of classes such as SUVs, trucks and sedans. Obviously, despite the fact that each vehicle is a different manufacturer or a different year, it is not at all a big difference at first glance, so it is common to be difficult to recognize the vehicle. Second, despite being the same vehicle, the vehicle makes a very different appearance depending on posture and angle. The shape of the front and side of the vehicle is very different, making it difficult to identify the unique features of each vehicle from different perspectives.

In spite of the unique characteristics of the vehicle, which is a great burden for recognition, there has been an invention of recognizing subcategories of the vehicle. However, this existing invention is not a perfect vehicle recognition method due to many limitations in recognition. One typical constraint is that a subcategory of a vehicle is recognized only by the front of the vehicle. Of course, it is true that the recognition performance is good only by the front of the vehicle. However, the exterior of a car in a surveillance environment is not always the front, and sometimes the side, back and top are sensed. In addition, the existing invention was not able to recognize the sub-category that can identify the model name and the manufacturer of the vehicle, it was able to recognize only the upper category, such as SUV, truck, sedan, bus.

In the present invention, as a Convolution Neural Network (CNN) -based model, we propose a vehicle recognition model that extracts and learns the features of each part of the vehicle in order to obtain more powerful performance than before.

Specifically, a description will be given of the visualization of the feature map for understanding the convolutional layer, and a method of automatically finding the parts required for vehicle recognition through visualization analysis. This is done by visualizing the activation during the forward pass of the layer.

1 illustrates a step of generating an average feature map.

(b) means the average of the feature map of each layer, (c) shows the average of the layers collected in (b).

The present invention is based on visualization with the strongest activation function (prior art 1).

According to an embodiment of the present invention, the front image of the vehicle, the front and side by side images of the 224 * 224 cropped image are simultaneously configured and put into the Krizhevsky (prior art 2) structure. In the present invention, after completing all the learning, the feature map generated in the network is checked.

Nine of the most responsive features of each layer of the Krizhevsky structure are obtained. As the layer deepens, the feature map that reacts becomes more rare and localized throughout the learning process.

At the same time, images of feature maps that represent rare but reactive neurons are the most important part of cognition. Thus, the image activated by the last five pooling layers is the minimum and certain area required for ultimate recognition.

In the last five layers, the average response pixels of all feature maps are calculated. Here, f _l ⁱ is referred to as the i th feature map of the fifth pulling layer of the l th image. Then, to form the last response field, the feature map of the entire image, called a heat map, is averaged as shown in Equation 1 below.

The network used in the present invention has five convolutional layers, totaling five. These are averaged as in Equation 2.

Where F is the heatmap. Therefore, an average heat map as shown in Fig. 1C is obtained.

2 shows three test examples with rectangular curtains. The heat map extraction shows the mean of the response map as the curtain is used for each sample.

FIG. 2 visualizes and shows which parts are most important as the screen is used at various positions of the vehicle and the learning progresses. The average feature map for each test image shows the most active area of the vehicle classification. As shown in Figure 4, it can be seen that the position of the head light, tail light, and the center emblem of the vehicle is the most important part in vehicle recognition.

The last example of Figure 2 shows a unique case. The test images were sorted according to the front of the vehicle, and the final heatmap result was focused on the tail lights. Although the responsiveness of the headlights and emblems is less responsive than the visible images, the headlights, tail lights, and emblems can be seen as the most important parts in vehicle recognition through the convolutional layer. Therefore, in the present invention, the light and the emblem of the vehicle are selected as the main recognition features in order to maximize the recognition rate and distinguish the vehicle sub-categories such as the make and model.

In the following, a method for finding vehicles required for classification based on an algorithm for finding a 3D bounding box (prior art 3) and finding a characteristic part of the vehicle in a surveillance environment will be described. Assuming that the vehicle runs on a straight path in most traffic environments, the algorithm of the present invention is generally easy and stable to apply to real traffic surveillance video.

Unlike other objects, the present invention builds a classification model that takes into account the specific shape of the vehicle in order to develop a more accurate and reliable model. Specifically, the bounding box may be found to identify front, side, and top surfaces of each side of the bounding box.

The first step of the algorithm according to an embodiment of the present invention is to determine three vanishing points that can be identified through the flow of the vehicle, and then perform camera calibration. The first vanishing point is the direction to the feature point that is tracked and identified by the KLT tracking program. The second vanishing point is in a direction perpendicular to the dividing line of the road. The third vanishing point is the outward direction of the two vanishing points. By normalizing these vanishing points, it is possible to detect a rectangular parallelepiped vehicle.

In the second step, we find the junctions of the three vanishing points and build a cuboid box for the vehicle. Each section is represented by three sections: front / back (F), side (S), and top (U). Each part consists of the matrix P as follows.

The position of each cross section can be found through the P matrix. And projection images can be obtained in two dimensions for each cross section.

3 shows an important part of the vehicle extracted.

In the manner as described above, the divided parts of the front / rear, side, and upward of the vehicle can be obtained. However, as shown in FIG. 3, when the F section and the S section of the vehicle are simultaneously viewed, the front face is not maintained as it is in the section where the front face is extracted.

In addition, the F section obtained from three vanishing points may not include all the headlights. However, as previously seen, the vehicle headlights have been recognized as an important part in vehicle recognition. However, there may be a case where the headlight / taillight area is not correctly recognized only by the boundary box described above. Therefore, the following describes a method for efficiently obtaining the headlight / taillight area.

In order to fully view the front of the vehicle, it is recognized as a side bounding box of the vehicle, but the entire front can be obtained by acquiring an area belonging to the front part of the vehicle. Assuming that the vehicle is facing forward, calculate the angle as to how far the vehicle is. To this end, first, horizontal lines w1 and w2, which are parallel to the image, are defined. Here, w1 is in contact with the lowest point among the four points constituting the vehicle front rectangle (p6 in FIG. 3), and w2 is the highest point among the four points constituting the vehicle front rectangle (in FIG. 3). contact with p2).

The point at which any one of two lines in the vehicle traveling direction included in the rectangle constituting the side of the bounding box meets either w1 or w2 is referred to as b1. And b2 is a point where a line perpendicular to the horizontal line starting from b1 and a line which does not meet the horizontal line among the vehicle traveling direction lines of the side rectangles meet. As a result, the following matrix P is defined.

The matrix P gives an adjusted front that includes all the important parts of the vehicle. The corrected front pixel coordinate T can be expressed as follows.

T = (p2 b1 p5 b2)

Here, p1, b1, p5, and b2 form a rectangle. As shown in FIG. 3, a corrected value may be obtained through the matrix P for the front quadrangle consisting of p2, p3, p5, and p6.

4 is a flowchart illustrating a vehicle recognition algorithm according to an embodiment of the present invention.

Vehicle recognition system according to an embodiment of the present invention receives a traffic monitoring image (S101). The traffic monitoring image input here is an image collected by a general traffic monitoring system. For example, the traffic monitoring image may be a highway traffic image.

The vehicle recognition system detects a vehicle from the traffic image (S103). The vehicle recognition system may detect a vehicle in a traffic image, and a method of detecting a vehicle in the image is based on a known algorithm.

The vehicle recognition system generates feature information of an image through an image analysis algorithm (S105). The vehicle recognition system extracts three vectors from the image through an image analysis algorithm. In detail, the vehicle recognition system obtains a motion vector with respect to the moving direction of the vehicle, and obtains a vector perpendicular to the moving direction of the vehicle. And the vehicle recognition system finds the cross product vector between the motion vector and the vector perpendicular to the motion vector. The algorithm for generating specific feature information has been described above with reference to FIG. 3, and thus detailed description thereof will be omitted.

The vehicle recognition system generates a bounding box based on the feature information (S107). Specifically, the vehicle recognition system combines the three vectors obtained in step S105 to generate a rectangular box-shaped bounding box. In this case, the bounding box may be composed of three surfaces: a front surface (or a rear surface opposite to the front surface) including a front image of the vehicle, a side surface including a side image of the vehicle, and an upper surface including an upper image of the vehicle. For example, the front image of the vehicle may be viewed as an image including a headlight in general common sense, the side image of the vehicle may be viewed as an image including a door, and the upper image of the vehicle may be an image including a ceiling of the vehicle. can see.

The vehicle recognition system acquires a single image by projecting three surfaces of the generated bounding box (S109). Each side of the bounding box generally represents the front, side, and top of the vehicle, although there may be cases where the front, including the front, does not fully cover the front of the vehicle. Accordingly, the vehicle recognition system according to an exemplary embodiment of the present invention corrects the plane constituting the bounding box and projects the corrected plane to obtain a single image in which all three planes are represented.

In a specific embodiment, the vehicle recognition system corrects the traveling direction surface (front or rear of the boundary box) of the three surfaces. A detailed correction method has been described with reference to FIG. 3, and thus description thereof will be omitted.

The vehicle recognition system projects three faces including the corrected face to obtain an image in which the three faces are represented by one face.

The vehicle recognition system inputs the acquired image into the CNN model (S111). The CNN model used here is a model implemented by a deeper method using a single image composed of three sides of a bounding box as input data, and is a model composed of a large amount of vehicle model data.

The vehicle recognition system searches for the vehicle model through the CNN model, and outputs the searched result (S113).

5 is a block diagram of a vehicle recognition system according to an exemplary embodiment.

The vehicle recognition system 1 according to an exemplary embodiment may include an image input unit 10, a control unit 20, a storage unit 30, and an output unit 40. Here, the vehicle recognition system 1 may be a single independent device, a configuration of a device, a combination of a plurality of independent devices, a program, or a device in which a program is installed and executable. .

The image input unit 10 receives a vehicle surveillance image photographed by a photographing device such as a CCTV. Here, the image input unit 10 may be an interface connected to an external device, and may be a component including an image decoder. For example, the image input unit 10 may include a USB terminal and a decoder capable of decoding the compressed image according to a specific compression scheme. In addition, the image input unit 10 may further include an image photographing device such as a camera.

The controller 20 performs an overall operation related to vehicle recognition. The controller 20 may be a hardware processor. The control unit 20 may include a feature extractor 21, a boundary box generation and correction unit 22, a projection plane generator 23, and a modeling unit 24.

The feature extractor 21 detects a vehicle from the vehicle surveillance image acquired from the image input unit 10 and extracts a feature. The features extracted by the feature extractor 21 are three vectors as described above, and the detailed method of extracting the three vectors is described above.

The bounding box generating and correcting unit 22 generates a rectangular parallelepiped bounding box based on the three vectors extracted by the feature extracting unit 21. Then, the three sides of the bounding box need to be corrected. Detailed operation of the bounding box generation and correction has been described above, and thus detailed description thereof will be omitted. An example of generating the bounding box may refer to FIG. 6A.

The projection plane generator 23 projects one surface of the bounding box to generate one image. The projection surface is an image in which each surface of the vehicle for identifying the vehicle is combined into one surface. For example, the projection surface may refer to FIG. 6B.

The modeling unit 24 inputs the image generated by the projection surface generation unit 23 to the vehicle recognition model. As the estimation model used herein, the CNN may be used, and the estimation model may be an estimation model configured by repetitively learning a plurality of data. If the image input from the projection plane generator 23 cannot recognize the pre-configured model, the modeling unit 24 may generate a new model through learning using the new image as an input.

The storage unit 30 stores image data. The storage unit 30 may be an internal memory, an external storage device, or a server, and any medium capable of storing other data may be used. In detail, the storage unit 30 may store an image generated by the projection surface generator 23, and may also store image data for recognizing an existing vehicle. The data stored in the storage unit 30 may be used by the modeling unit 24 to construct and update the estimation model.

The output unit 40 outputs the vehicle recognition result performed by the control unit 20. The output unit 40 may be, for example, an output device such as a display device or a printer. The output unit 40 outputs a vehicle recognition result. The vehicle information output here may be at least one of a model name, a manufacturer, and a year of the vehicle. The existing vehicle recognition system recognizes a class related to the vehicle body, but the present invention can recognize and output relatively lower layer vehicle information using a bounding box, a projection surface, and a deep learning based estimation model.

6 shows an example of a bounding box and a projected image.

(a) shows an example of three sides that constitute the bounding box and the bounding box generated based on the feature information (three-direction vector).

(b) shows an example of the result of projecting three sides of the bounding box into one image.

The present invention described above can be embodied as computer readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like. This also includes those implemented in the form of carrier waves (eg, transmission over the Internet).

Accordingly, the above detailed description should not be construed as limiting in all aspects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (8)

Receiving a traffic monitoring image;
Detecting a vehicle from an input image;
Extracting feature information based on the detected vehicle from the input image;
Generating a bounding box based on the extracted feature information;
Generating an image projecting three sides of the bounding box onto one side; And
Inputting the projected image into a vehicle recognition model;
Generating an image projecting the three sides of the bounding box onto one side
Generating a plurality of horizontal lines in contact with points constituting a front or rear quadrangle among surfaces constituting the bounding box;
Deriving a point where the horizontal line meets a rectangle constituting the side of the plane constituting the bounding box, and correcting a front or rear rectangle of the plane constituting the bounding box;
Vehicle Recognition Method. The method of claim 1,
The feature information includes a plurality of vectors,
The plurality of vectors includes a first vector relating to a vehicle traveling direction, a second vector perpendicular to the first vector, and a third vector external to the first vector and the second vector.
Vehicle Recognition Method. The method of claim 2,
Generating an image projecting the three sides of the bounding box onto one side
Compensating the rear surface including the front or rear image including the front image of the surface constituting the bounding box, and generating an image projecting the corrected surface
Vehicle Recognition Method. The method of claim 1,
The vehicle recognition model is a convolution neural network
Vehicle Recognition Method. Image input unit for receiving a traffic monitoring image;
A controller for recognizing a vehicle from an image received by the image input unit; And
An output unit for outputting a result recognized by the controller,
The control unit may include a feature extraction unit for extracting a feature from an input image, a boundary box generation and correction unit for generating and correcting a boundary box based on the extracted feature, and an image in which three sides of the boundary box are projected onto one surface And a modeling unit configured to recognize a vehicle through a vehicle recognition model by inputting an image generated by the projection surface generator and an image generated by the projection surface generator.
The bounding box generation and correction unit
Create a plurality of horizontal lines in contact with the points constituting the front or rear rectangle of the surface constituting the bounding box,
Deriving a point where the horizontal line meets the square forming the side of the plane constituting the bounding box, to correct the front or rear square of the plane constituting the bounding box
Vehicle recognition system. The method of claim 5, wherein
Further comprising an output unit for outputting a vehicle recognition result
Vehicle recognition system. The method of claim 5, wherein
The apparatus may further include a storage unit configured to store the image generated by the projection plane generator and to provide data for constructing a model to the modeling unit.
Vehicle recognition system. The method of claim 5, wherein
The extracted feature includes a plurality of vectors,
The plurality of vectors includes a first vector relating to a vehicle traveling direction, a second vector perpendicular to the first vector, and a third vector external to the first vector and the second vector.
Vehicle recognition system.

Images