CN102880863B - Method for positioning license number and face of driver on basis of deformable part model - Google Patents

Abstract

The invention relates to a method for locating a license plate and a driver's face based on a deformable component model, and belongs to the field of object detection. The method uses a deformable part model to model the vehicle in the front view, takes the license plate and the driver's face as the parts in the model, and obtains the parameters in the model through training; based on the established model, accurate license plate positioning and driving Driver's face location, and vehicle model recognition based on the relative positional relationship between the license plate and the driver's face. The invention can make full use of the position information between the license plate and the driver's face, can accurately locate the license plate and the driver's face, and obtain vehicle model information.

Description

A license plate and driver's face location method based on a deformable component model

technical field

The invention belongs to the technical field of object detection, in particular to the combined application of license plate positioning technology and face detection technology.

Background technique

With the continuous development of society, the management of modern traffic is becoming more and more complex and burdensome. In order to accurately and efficiently monitor traffic conditions and save costs, many intelligent traffic monitoring technologies have been widely used. Among them, the automatic license plate recognition system is a relatively mature technology and is applicable to various traffic scenarios. The license plate recognition system is divided into three parts: license plate location, character segmentation and character recognition. Among them, the license plate positioning technology can already obtain a relatively high accuracy rate, but it has also reached a bottleneck. If other information in the scene cannot be used, the accuracy of license plate location can hardly be further improved.

With the development of imaging technology, high-definition cameras have been widely used. High-definition cameras can obtain clear images of moving vehicles and drivers. At present, there are not many studies on face positioning in such scenarios, and since the driver's face is generally behind the front windshield, it is easily affected by light and viewing angles, so it is still impossible to obtain good positioning results.

Therefore, if the driver's face information can be effectively combined with the vehicle's license plate information, the accuracy of license plate location can be further improved. At the same time, since the relative positions of the license plate and the driver's face are relatively fixed, the positioning result of the driver's face can also be improved through the positioning result of the license plate. In addition, the vehicle model can also be identified by the relative position of the license plate and the driver's face.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art, to locate the driver's face and license plate in the image obtained by the high-definition camera, and then to identify the vehicle type, so as to provide more effective information for subsequent traffic monitoring tasks.

The purpose of the present invention is achieved through the following technical solutions.

A method for locating a license plate and a driver's face based on a deformable component model, the specific implementation steps are as follows:

Step 1: Establish a deformable part model of the frontal vehicle

The license plate and the driver's face are used as components to establish a deformable part model of the frontal vehicle, and the positional relationship between the license plate and the driver's face is obtained through the training data as model parameters;

The deformable part model M of the vehicle in the front view is defined as follows

M＝{part _plate , part _face , pos _{plate, face} } (1)

Among them, part _plate represents the license plate part in the model, part _face represents the face part in the model, pos _{plate, face} = {p _{plate, face} , d _{plate, face} } represents the positional relationship between the license plate and the driver's face . Among them, p _{plate, face} represents the spatial position relationship between the license plate and the driver's face. For different countries and regions, because the location of the driver is different, the location relationship is also different. For mainland China, it is necessary to meet Among them, plate.x, face.x represent the x coordinates of the license plate and the face, and plate.y, face.y represent the y coordinates of the license plate and the face, that is, the license plate is at the lower left of the face. d _{plate, face} represents the distance between the license plate and the face, and

d _plate,face ∈N _i (μ _i ,δ _i ),i∈{big,middle,samll} (2)

N(μ,δ) represents a Gaussian model with mean μ and variance δ.

By counting the distance between the marked license plate and the human face, the corresponding distance of each type of vehicle is obtained.

The mean and variance of the Gaussian model, that is, the deformable part model of the vehicle is obtained.

Step 2: Carry out rough positioning of the license plate to obtain the candidate area of the license plate and the corresponding credibility.

Currently, there are various license plate location methods that can obtain license plate candidate areas and corresponding credibility. The present invention adopts a license plate location method based on paired morphological operators to obtain license plate candidate areas and corresponding credibility.

Suppose S _m×n is a structural element with size m×n and all values are 1, the local neighborhood of a certain pixel is determined by S _m×n . I represents a grayscale image, and Denote the corrosion and dilation operations in mathematical morphology, respectively, and the morphological operations used in the following definitions:

Closing operation: $I\·S_{m\×\mathrm{no}}=(I\⊕S_{m\×\mathrm{no}})\⊗S_{m\×\mathrm{no}}$

Open operation:

High hat operation: $\mathrm{I\Δ}{S}_{m\×\mathrm{no}}=I-(I\&Center\; Dot;S_{m\×\mathrm{no}})$

Black hat operation:

The top-hat operation (top-hat) can extract the locally brighter area by making the difference between the source image and the open operation image; the black hat transformation (bot-hat) can extract the locally darker area by making the difference between the closed operation image and the source image area. Due to the significant contrast between the license plate background brightness and character brightness, these two operations can separate the character and background areas of the license plate, suppress the background, and eliminate uneven illumination. Continental license plates have two types, dark characters on a bright background and bright characters on a dark background. Using only a single morphological operation (high-hat transformation or black-hat transformation) cannot simultaneously extract character regions for license plate location. We explicitly combine character information and license plate background information through pairwise morphological operations, and are able to detect both types of license plates under a unified framework.

Taking the license plate with bright characters on the dark background as an example, in order to extract the character area, it can be binarized by high-hat operation, as shown in Figure 2.

Now consider the background area of the license plate with dark background and bright characters. If the linear structure element S _1×n in the horizontal direction is selected, the background of the license plate can be divided into three parts, which are the background part between characters (red) and the background part inside characters (green). and other background parts (blue), as shown in Fig. 3(b).

Under the effect of S _1×n , the blue area is a local non-significant change area (the brightness of the pixel set in the neighborhood of the linear structural element where the pixel is located is consistent), and the red area and green area are local significant changes. If the black hat transformation and binarization are performed on the license plate with bright characters on the dark background, the background corresponding to the blue area will be filtered out while the background corresponding to the green area and red area will be retained, as shown in Figure 3(c). Classify the background of different regions (Figure 3(d)), and only consider the part of the background between characters (Figure 3(e)), it can be found that the background area of the part between characters meets the characteristics of license plate characters with consistent height and uniform distribution. We call the region corresponding to the background between characters the pseudo-character region.

Paired morphological operations on the license plate with bright characters on the dark background can extract the actual characters and pseudo characters respectively, and they all meet the characteristics of the license plate characters being highly consistent and evenly distributed, as shown in Figure 3(e). Therefore, their union can be used for license plate location. Similarly, performing pairwise morphological operations on license plates with bright background and dark characters can extract pseudo characters and actual characters respectively. The paired morphological operator method effectively solves the limitation of the single morphological operator method - the foreground-background collocation of the license plate needs to be known in advance. The paired operator is used to extract the actual character area and the pseudo-character area respectively, effectively combining the license plate foreground information and the license plate background information to jointly represent the license plate, and can process the two types of license plates in a unified manner. The character area extraction process is as follows:

1) Perform the following morphological operations on the grayscale image I respectively, I ₁ =I△S _1×n , I ₂ =I▽S _1×n

2) Use the Otsu method to binarize _I1 and _I2 respectively to obtain corresponding binary images _I3 and _I4 ;

3) mark the connected components of I ₃ and I ₄ respectively, and obtain two sets of connected components C ^top and C ^bot ;

4) Merge the real characters and dummy characters, and use the area to filter the characters through the prior size of the license plate to obtain the final character set: C=C ^top ∪C ^bot

After obtaining the character set, the center point of each character can be further obtained. According to the characteristics that the characters on the license plate are distributed on a straight line, we locate the license plate by judging whether these center points are on the same straight line. Usually the variation range of the license plate size is known in the scene, according to this prior knowledge, we set the detection window to be twice the width and height of the maximum license plate. Move the detection window in the image to locate the license plate. The step size of each movement in the x direction is 1/2 of the width of the detection window, and the step size of each movement in the y direction is 1/2 of the height of the detection window. This ensures that the license plate will appear in at least one window detector and can significantly reduce the number of candidate regions. In each detection window, it is judged whether the center points of the obtained characters are on the same straight line. If the number of center points on the same straight line is greater than a given threshold, the area corresponding to these center points is considered to be a candidate license plate area. At the same time, the confidence of the candidate region can be calculated as

confidence _plate =|num_c-num_thres| (3)

Among them, num_c is the number of characters obtained, and num_thres is the set threshold.

Step 3: Carry out rough positioning of the driver's face, and obtain the candidate area of the face and the corresponding reliability.

Currently, there are multiple face monitoring methods to obtain the candidate areas of the driver's face and the corresponding reliability. The present invention adopts the face detection method based on AdaBoost to obtain the candidate areas of the driver's face and the corresponding reliability.

The AdaBoost-based face detection method constructs a weak classifier based on rectangular features, and then uses the AdaBoost method to select a small number of key features, and weights and sums the corresponding weak classifiers to construct a strong classifier, which is used as the final The classifier is used for face detection. Among them, each rectangle feature is composed of 2-3 rectangles, as shown in Figure 4, and its value is the sum of the pixel values in the white rectangle minus the sum of the pixel values in the black rectangle.

Each weak classifier consists of a rectangular feature, and the training process of the strong classifier is as follows:

(1) Given training data (x ₁ , y ₁ ),...(x _n , y _n ), where y _i =0 represents negative samples, y _i =1 represents positive samples, and n is the number of training samples .

(2) Initialize weights, when y _i =0 When y _i =1 m and l are the number of negative samples and positive samples respectively.

(3) Corresponding to t=1,...,T:

A. Normalized weights

B. For each feature j, train a weak classifier h _j , if the error of the classifier is

C. Choose the classifier h _t with the smallest error

D. Update weights where e _i = 0 if _xi is correctly classified, otherwise

e _i =1, ${\mathrm{\β}}_t=\frac{{\mathrm{\ϵ}}_t}{1-{\mathrm{\ϵ}}_t}$

(4) Then the final strong classifier is

$h\left(x\right)=\left\{\begin{array}{cc}1& {\mathrm{\Σ}}_{t=1}^T{\mathrm{\α}}_t{h}_t\left(x\right)\&Greater\; Equal;\frac{1}{2}{\mathrm{\Σ}}_{t=1}^T{\mathrm{\α}}_t\\ 0& \mathrm{otherwise}\end{array}\right.,$ in ${\mathrm{\α}}_t=\log \frac{1}{{\mathrm{\β}}_t}$

Since the region obtained by rough positioning can be filtered out through the subsequent model based on deformable parts, when we use a strong classifier for face detection, the goal is to minimize the missed detection rate and allow a certain false detection rate. The detection rule is set as follows: when the detected area passes the first K (K<T) weak classifiers, the area is considered to be a face candidate area, and at the same time continue to scan using the K+1 to T weak classifiers, and record The number of weak classifiers it passes is taken as the credibility of the candidate region. now that

confidence _face = num _{passed_classifier} (4)

Step 4: Fine positioning of the license plate and driver's face based on the deformable part model

Step two provides possible license plate candidate areas in the image, and step three provides possible driver face candidate areas in the image. Then we use the deformable part model of the vehicle obtained in step 1 to perform fine positioning of the license plate and driver's face.

Let L={l _plate , l _face } be a realization of the model M in the image. Among them, l _plate represents the position of the license plate in the image, and l _face represents the position of the face in the license plate. Let m(l _plate ) represent the reliability of the license plate position on l _plate , which can be calculated by formula (3), m(l _face ) represents the credibility of the face position on l _face , which can be calculated by formula (4). m(l _plate ,l _face ) indicates the degree of conformity between the positional relationship between the license plate and the face and the model, and

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

in is the distance between the license plate candidate area and the face candidate area, N _i (μ _i , δ _i ) is the Gaussian model obtained through training in step 1, i∈{big,middle,samll}.

According to the deformable part model, the license plate and the driver's face are finely positioned, and L ^* is found so that

${\mathrm{<span\; class="notranslate">\; L</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; \{</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; platter</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

The subsequent area of the license plate can be extracted through step 2, and the candidate area of the face can be extracted through step 3. We sort the candidate areas according to the credibility of the candidate areas. Let the sorted license plate candidate areas be l _{plate_1} ,...l _{plate_n} , n is the number of license plate candidate areas, the sorted driver face candidate areas are l _{face_1} ,...l _{face_m} , m is the driver's face Candidate area. Then the process of locating the license plate and the driver’s face through the deformable part model is as follows: corresponding to j=1,...,m, calculate scoring value Record the maximum scoring value and its corresponding license plate and driver's face position as the final result of license plate and driver's face positioning.

Step 5: Vehicle model recognition based on the relative positional relationship between the license plate and the driver's face

After finding the best license plate position l ^* _plate and the driver's face position l ^* _face through step 4, calculate

${\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

i∈{big,middle,samll}, the final vehicle type recognition result can be obtained, that is, whether the vehicle is a large vehicle, a medium-sized vehicle or a small vehicle.

The main content of the present invention is: use the deformable component model to model the vehicle in the front view, use the license plate and the driver's face as the components in the vehicle model, and perform the positioning of the license plate and the driver's face according to the model, which can improve the accuracy of the license plate. The accuracy of positioning and the accuracy of driver face positioning. At the same time, vehicle type recognition can be performed according to the relative positional relationship between the license plate and the driver's face.

Advantages of the invention

Compared with the prior art, the present invention has the following advantages:

(1) Widely used high-definition cameras are used to obtain images without additional cost input.

(2) Make full use of various information in the scene to improve the positioning accuracy of the license plate and driver's face.

(3) According to the relative positional relationship between the license plate and the driver's face, the vehicle type recognition can be realized, and the application prospect is broad.

Description of drawings

Figure 1 is a flow chart of license plate and driver's face location based on the deformable component model;

Fig. 2 is a schematic diagram of bright characters extracted by high-hat operation; wherein, (a) original image, (b) high-hat image, (c) high-hat binary image;

Figure 3 is a schematic diagram of pseudo-character extraction; where, (a) original image, (b) schematic diagram of background division, (c) black hat binary image, (d) background classification, (e) pseudo-character area, (f) actual Characters and Pseudocharacters

Figure 4 is a rectangular feature.

Detailed ways

The process flow of the license plate and driver's face positioning method based on the deformable component model proposed by the present invention is shown in Figure 1, and the specific implementation steps are as follows:

Step 1: Establish a deformable part model of the frontal vehicle

Deformable Part-Based Model (Deformable Part-Based Model) is a popular model for object detection in images in recent years, and it is currently one of the best object detection algorithms. The deformable part model represents the object by describing the positional relationship between various parts. In the vehicle model of the front view we built, there are two parts: the license plate and the driver's face. After determining the components in the model, the training of the model is to obtain the positional relationship between the license plate and the driver's face through the training data.

Our training data includes three categories of large cars, medium-sized cars and small cars, and the position of the license plate and the position of the driver's face in each image in each category are manually annotated. A Gaussian mixture model is used to represent the positional relationship between the license plate and the driver's face. For each type of vehicle, there is a Gaussian model to represent the positional relationship between the license plate and the driver's face.

The deformable part model M of the vehicle in the front view is defined as follows

M＝{part _plate , part _face , pos _{plate, face} } (1)

Among them, part _plate represents the license plate part in the model, part _face represents the face part in the model, pos _{plate, face} = {p _{plate, face} , d _{plate, face} } represents the positional relationship between the license plate and the driver's face . Among them, p _{plate, face} represents the spatial position relationship between the license plate and the driver's face. For different countries and regions, because the location of the driver is different, the location relationship is also different. For mainland China, it is necessary to meet Among them, plate.x, face.x represent the x coordinates of the license plate and the face, and plate.y, face.y represent the y coordinates of the license plate and the face, that is, the license plate is at the lower left of the face.

d _{plate, face} represents the distance between the license plate and the face, and

d _plate,face ∈N _i (μ _i ,δ _i ),i∈{big,middle,samll} (2)

N(μ,δ) represents a Gaussian model with mean μ and variance δ.

We obtain the mean and variance of the Gaussian model corresponding to each type of vehicle by counting the distance between the marked license plate and the human face, that is, the deformable part model of the vehicle.

Step 2: Use the license plate location method based on paired morphological operators to perform rough positioning of the license plate, and obtain the candidate areas of the license plate and the corresponding credibility.

A license plate location method based on paired morphological operators is used for rough location of the license plate. Suppose S _m×n is a structural element with size m×n and all values are 1, the local neighborhood of a certain pixel is determined by S _m×n . I represents a grayscale image, and Denote the corrosion and dilation operations in mathematical morphology, respectively, and the morphological operations used in the following definitions:

Closing operation: $I\&CenterDot;S_{m\&times;\mathrm{no}}=(I\&CirclePlus;S_{m\&times;\mathrm{no}})\&CircleTimes;S_{m\&times;\mathrm{no}}$

Open operation:

High hat operation: $\mathrm{I\&Delta;}{S}_{m\&times;\mathrm{no}}=I-(I\&Center\; Dot;S_{m\&times;\mathrm{no}})$

Black hat operation:

The top-hat operation (top-hat) can extract the locally brighter area by making the difference between the source image and the open operation image; the black hat transformation (bot-hat) can extract the locally darker area by making the difference between the closed operation image and the source image area. Due to the significant contrast between the license plate background brightness and character brightness, these two operations can separate the character and background areas of the license plate, suppress the background, and eliminate uneven illumination. Continental license plates have two types, dark characters on a bright background and bright characters on a dark background. Using only a single morphological operation (high-hat transformation or black-hat transformation) cannot simultaneously extract character regions for license plate location. We explicitly combine character information and license plate background information through pairwise morphological operations, and are able to detect both types of license plates under a unified framework.

Taking the license plate with bright characters on the dark background as an example, in order to extract the character area, it can be binarized by high-hat operation, as shown in Figure 2.

Now consider the background area of the license plate with dark background and bright characters. If the linear structure element S _1×n in the horizontal direction is selected, the background of the license plate can be divided into three parts, which are the background part between characters (red) and the background part inside characters (green). and other background parts (blue), as shown in Fig. 3(b).

Under the effect of S _1×n , the blue area is a local non-significant change area (the brightness of the pixel set in the neighborhood of the linear structural element where the pixel is located is consistent), and the red area and green area are local significant changes. If the black hat transformation and binarization are performed on the license plate with bright characters on the dark background, the background corresponding to the blue area will be filtered out while the background corresponding to the green area and red area will be retained, as shown in Figure 3(c). Classify the background of different regions (Figure 3(d)), and only consider the part of the background between characters (Figure 3(e)), it can be found that the background area of the part between characters meets the characteristics of license plate characters with consistent height and uniform distribution. We call the region corresponding to the background between characters the pseudo-character region.

Paired morphological operations on the license plate with bright characters on the dark background can extract actual characters and pseudo characters respectively, and they all meet the characteristics of license plate characters with consistent height and uniform distribution, as shown in Figure 3(e). Therefore, their union can be used for license plate location. Similarly, performing pairwise morphological operations on license plates with bright background and dark characters can extract pseudo characters and actual characters respectively. The paired morphological operator method effectively solves the limitation of the single morphological operator method - the foreground-background collocation of the license plate needs to be known in advance. The paired operator is used to extract the actual character area and the pseudo-character area respectively, effectively combining the license plate foreground information and the license plate background information to jointly represent the license plate, and can process the two types of license plates in a unified manner. The character area extraction process is as follows:

1) Perform the following morphological operations on the grayscale image I respectively, I ₁ =I△S _1×n , I ₂ =I▽S _1×n

2) Use the Otsu method to binarize _I1 and _I2 respectively to obtain corresponding binary images _I3 and _I4 ;

3) mark the connected components of I ₃ and I ₄ respectively, and obtain two sets of connected components C ^top and C ^bot ;

4) Merge the real characters and dummy characters, and use the area to filter the characters through the prior size of the license plate to obtain the final character set: C=C ^top ∪C ^bot

After obtaining the character set, the center point of each character can be further obtained. According to the characteristics that the characters on the license plate are distributed on a straight line, we locate the license plate by judging whether these center points are on the same straight line. Usually the variation range of the license plate size is known in the scene, according to this prior knowledge, we set the detection window to be twice the width and height of the maximum license plate. Move the detection window in the image to locate the license plate. The step size of each movement in the x direction is 1/2 of the width of the detection window, and the step size of each movement in the y direction is 1/2 of the height of the detection window. This ensures that the license plate will appear in at least one window detector and can significantly reduce the number of candidate regions. In each detection window, it is judged whether the center points of the obtained characters are on the same straight line. If the number of center points on the same straight line is greater than a given threshold, the area corresponding to these center points is considered to be a candidate license plate area. At the same time, the confidence of the candidate region can be calculated as

confidence _plate =|num_c-num_thres| (3)

Among them, num_c is the number of characters obtained, and num_thres is the set threshold.

Step 3: Use the AdaBoost-based face detection method to roughly locate the driver's face, and obtain the candidate areas of the face and the corresponding reliability.

We use an AdaBoost-based face detection method for coarse localization of the driver's face. This method constructs a weak classifier based on rectangular features, and then uses the AdaBoost method to select a small number of key features, and weights and sums the corresponding weak classifiers to construct a strong classifier, which is used as the final classifier for the face detection. Among them, each rectangle feature is composed of 2-3 rectangles, as shown in Figure 4, and its value is the sum of the pixel values in the white rectangle minus the sum of the pixel values in the black rectangle.

Each weak classifier consists of a rectangular feature, and the training process of the strong classifier is as follows:

(1) Given training data (x ₁ , y ₁ ),...(x _n , y _n ), where y _i =0 represents negative samples, y _i =1 represents positive samples, and n is the number of training samples .

(2) Initialize weights, when y _i =0 When y _i =1 m and l are the number of negative samples and positive samples respectively.

(3) Corresponding to t=1,...,T:

A. Normalized weights $w_{t,i}\&LeftArrow;\frac{w_{t,i}}{{\mathrm{\&Sigma;}}_{j=1}^{\mathrm{no}}{w}_{t,j}}$

B. For each feature j, train a weak classifier h _j , if the error of the classifier is

C. Choose the classifier h _t with the smallest error

D. Update weights where e _i = 0 if _xi is correctly classified, otherwise e _i = 1, ${\mathrm{\&beta;}}_t=\frac{{\mathrm{\&epsiv;}}_t}{1-{\mathrm{\&epsiv;}}_t}$

(4) Then the final strong classifier is

$h\left(x\right)=\left\{\begin{array}{cc}1& {\mathrm{\&Sigma;}}_{t=1}^T{\mathrm{\&alpha;}}_t{h}_t\left(x\right)\&Greater\; Equal;\frac{1}{2}{\mathrm{\&Sigma;}}_{t=1}^T{\mathrm{\&alpha;}}_t\\ 0& \mathrm{otherwise}\end{array}\right.,$ in ${\mathrm{\&alpha;}}_t=\log \frac{1}{{\mathrm{\&beta;}}_t}$

Since the region obtained by rough positioning can be filtered out through the subsequent model based on deformable parts, when we use a strong classifier for face detection, the goal is to minimize the missed detection rate and allow a certain false detection rate. We set the detection rule as follows: when the detected area passes through the first K (K<T) weak classifiers, it is considered that the area is a face candidate area, and at the same time continue to use the K+1th to Tth weak classifiers for scanning, and Record the number of weak classifiers it passes as the credibility of the candidate region. now that

confidence _face = num _{passed_classifier} (4)

Step 4: License plate and driver face location based on deformable parts model

Step two provides possible license plate candidate areas in the image, and step three provides possible driver face candidate areas in the image. Then we use the deformable part model of the vehicle obtained in step 1 to perform fine positioning of the license plate and driver's face.

Let L={l _plate , l _face } be a realization of the model M in the image. Among them, l _plate represents the position of the license plate in the image, and l _face represents the position of the face in the license plate. Let m(l _plate ) represent the reliability of the license plate position on l _plate , which can be calculated by formula (3), m(l _face ) represents the credibility of the face position on l _face , which can be calculated by formula (4). m(l _plate ,l _face ) indicates the degree of conformity between the positional relationship between the license plate and the face and the model, and

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

in is the distance between the license plate candidate area and the face candidate area, N _i (μ _i , δ _i ) is the Gaussian model obtained through training in step 1, i∈{big,middle,samll}.

According to the deformable part model, the license plate and the driver's face are finely positioned, and L ^* is found so that

${\mathrm{<span\; class="notranslate">\; L</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; \{</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; platter</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; l</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

The subsequent area of the license plate can be extracted through step 2, and the candidate area of the face can be extracted through step 3. We sort the candidate areas according to the credibility of the candidate areas. Let the sorted license plate candidate areas be l _{plate_1} ,...l _{plate_n} , n is the number of license plate candidate areas, the sorted driver face candidate areas are l _{face_1} ,...l _{face_m} , m is the driver's face Candidate area. Then the process of locating the license plate and the driver’s face through the deformable part model is as follows: corresponding to j=1,...,m, calculate scoring value Record the maximum scoring value and its corresponding license plate and driver's face position as the final result of license plate and driver's face positioning.

Step 5: Vehicle model recognition based on the relative positional relationship between the license plate and the driver's face

After finding the best license plate position l ^* _plate and the driver's face position l ^* _face through step 4, calculate

${\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; plate</span>}}<span\; class="notranslate">\; ,</span>{{\mathrm{<span\; class="notranslate">\; l</span>}}^{<span\; class="notranslate">\; *</span>}}_{\mathrm{<span\; class="notranslate">\; the\; face</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

i∈{big,middle,samll}, the final vehicle type recognition result can be obtained, that is, whether the vehicle is a large vehicle, a medium-sized vehicle or a small vehicle.

Claims (2)

1. based on car plate and the driver's Face detection method of deformable part model, it is characterized in that, a kind of car plate based on deformable part model and driver's Face detection method, specific implementation step is as follows:

Step one: the deformable part model setting up front vehicle

Using car plate and driver's face as parts, set up the deformable part model of front vehicle, by training data, obtain the position relationship between car plate and driver's face, as model parameter;

The deformable part model M of the vehicle in front view is defined as follows

M＝{part _plate,part _face,pos _plate,face} (1)

Wherein, part _platerepresent the car plate parts in model, part _facerepresent the face component in model, represent the position relationship between car plate and driver's face; Wherein p _{plate, face}represent the spatial relation of car plate and driver's face; For country variant and area, because the position at driver place is different, this position relationship is also different; For Continental Area, plate.x < face.x need be met, plate.y < face.y, wherein plate.x, face.x represent the x coordinate of car plate and face, plate.y, face.y represents the y coordinate of car plate and face, and namely car plate is in the lower left of face; And

d _plate,face∈N _i(μ _i,δ _i),i∈{big,middle,samll} (2)

D _{plate, face}represent the distance between car plate and face;

N (μ, δ) represents that average is μ, and variance is the Gauss model of δ;

Distance between the car plate marked by statistics and face, is obtained all referring to and variance of the Gauss model corresponding to vehicle of each type, has namely arrived the deformable part model of vehicle;

Step 2: adopt the license plate locating method based on paired morphological operator to carry out the coarse positioning of car plate, obtains the candidate region of car plate and corresponding confidence level

Step 3: adopt the method for detecting human face based on AdaBoost to carry out the coarse positioning of driver's face, obtains the candidate region of face and corresponding confidence level

Step 4: the face candidate region that the license plate candidate area that the deformable part model of front vehicle set up based on step one, step 2 obtain and confidence level and step 3 obtain and confidence level carry out the meticulous location of car plate and driver's face

If L={l _plate, l _faceit is a model M realization in the picture; Wherein l _platerepresent car plate position in the picture, l _facerepresent the position of face in car plate; If m is (l _plate) represent that car plate position is at l _plateconfidence level, m (l _face) represent that face location is at l _faceconfidence level; M (l _plate, l _face) represent the degree of conformity of position relationship between car plate and face and model, and

$m(l_{\mathrm{plate}},l_{\mathrm{face}})=\arg \underset{i}{\max }P(d_{l_{\mathrm{plate}},l_{\mathrm{face}}}\&Element;N_i({\mathrm{\&mu;}}_i,{\mathrm{\&delta;}}_i))---\left(3\right)$

Wherein for the distance between license plate candidate area and face candidate region, N _i(μ _i, δ _i) train the Gauss model obtained, i ∈ { big, middle, samll} for passing through in step one;

Carry out car plate and the meticulous location of driver's face according to deformable part model, both find L ^*make

$L^{*}=\{{l^{*}}_{\mathrm{plate}},{l^{*}}_{\mathrm{face}}\}=\arg \underset{L}{\max }(m\left(l_{\mathrm{plate}}\right)+m\left(l_{\mathrm{face}}\right)+m(l_{\mathrm{plate}},l_{\mathrm{face}}))---\left(4\right)$

Car plate subsequent sections can be extracted by step 2, face candidate region can be extracted by step 3, according to the confidence level of candidate region, be sorted in candidate region; If the license plate candidate area after sequence is l _{plate_1 ...}l _{plate_n}, n is the number of license plate candidate area, and the driver's face candidate region after sequence is l _{face_1 ...}l _{face_m}, m is driver's face candidate region; The process of then carrying out car plate and driver's Face detection by deformable part model is: corresponding i=1 ..., n, j=1 ..., m, calculates marking value m (l _{plate_i})+m (l _{face_j})+m (l _{plate_i}, l _{face_j}), record the car plate of maximum marking value and correspondence thereof and the driver's face location result as final car plate and driver's Face detection;

Step 5: vehicle cab recognition is carried out in the position of the car plate obtained based on step 4 and driver's face

Best car plate position l is found by step 4 ^* _plateand driver face location l ^* _faceafter, by calculating

$i^{*}=\arg \underset{i}{\max }P(d({l^{*}}_{\mathrm{plate}},{l^{*}}_{\mathrm{face}})\&Element;N_i({\mathrm{\&mu;}}_i,{\mathrm{\&delta;}}_i))---\left(5\right)$

{ big, middle, samll} can obtain final vehicle cab recognition result to i ∈.

2. a kind of car plate based on deformable part model according to claim 1 and driver's Face detection method, is characterized in that, adopts the license plate locating method based on paired morphological operator to carry out the coarse positioning of car plate.