CN111145184A - Towing process safety early warning method based on multi-scale feature fusion - Google Patents
Towing process safety early warning method based on multi-scale feature fusion Download PDFInfo
- Publication number
- CN111145184A CN111145184A CN201910506741.XA CN201910506741A CN111145184A CN 111145184 A CN111145184 A CN 111145184A CN 201910506741 A CN201910506741 A CN 201910506741A CN 111145184 A CN111145184 A CN 111145184A
- Authority
- CN
- China
- Prior art keywords
- image
- license plate
- region
- scale
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Abstract
The invention discloses a towing process safety early warning method based on multi-scale feature fusion. The invention designs a towing process safety early warning method based on multi-scale feature fusion by combining the characteristics of towing and transporting of a light rescue vehicle and the characteristics of a towed vehicle monitoring image, and can effectively monitor the longitudinal shaking amplitude of a towed vehicle; in addition, the safety early warning method in the towing process has high reliability and strong anti-interference capability; the present invention takes advantage of various features of the license plate region, such as: the license plate is positioned by color features, rectangular features and the like, the anti-interference capability and the robustness are strong, and the vertical shaking amplitude is calculated by adopting a multi-scale feature fusion-based method, so that the accuracy is high; the invention has good real-time performance.
Description
Technical Field
The invention belongs to the field of safety early warning of rescue wreckers, and particularly relates to a towing process safety early warning method based on multi-scale feature fusion.
Background
With the rapid development of economy, the quantity of automobiles in China reaches 2.4 hundred million by 2018, which is increased by 2285 thousands by 10.51% compared with 2017. The explosive growth of the automobile holding capacity causes great pressure to road traffic in China, so that traffic accidents frequently occur, the road safety is seriously influenced, and the automobile holding capacity becomes an important factor for restricting the social and economic development and causing air pollution. Under the background, road traffic emergency rescue, as an important component of the construction of national public safety capability, has gradually become one of the important guarantees for maintaining the safety of people's lives and properties. In the road rescue equipment, the light road rescue equipment has high flexibility and strong functionality, and can enter and exit the traffic environment with small space and large gradient, thereby being widely applied. In the dragging and transporting process of the light rescue vehicle, the towed vehicle is positioned at the tail of the rescue vehicle, the visual angle of the rearview mirror of the rescue vehicle is limited, the towed vehicle cannot be comprehensively monitored, and potential safety hazards are easily generated. In a complex traffic environment with crowded traffic and changeable road conditions (such as bumpy roads), the towed vehicle is likely to shake up and down violently due to overlarge road bumpiness, so that the towed vehicle is separated from the rescue vehicle, urban traffic jam is easily caused, and even secondary traffic accidents are caused. Due to the potential safety hazard, the monitoring of the longitudinal shaking amplitude of the trailer is very important. In addition, the intelligent level of present light-duty rescue wrecker is not high, does not utilize current scientific and technological means, monitors by the vertical range of rocking of trailer. The visual sensor is widely applied to the fields of safety monitoring and the like due to the advantages of low cost, small size, convenience in installation and the like, so that the monitoring of the longitudinal shaking amplitude of the trailer based on the visual sensor becomes one of effective means for solving the problems. In the dragging process of the light rescue vehicle, the trailer needs to be detected and tracked in a plurality of frames of images during the monitoring of the longitudinal shaking amplitude of the trailer, and the method belongs to the field of target detection under a dynamic background.
The detection of the moving object under the dynamic background means that a platform where the camera is located can continuously move in a translation or rotation manner, so that the whole shooting scene and the camera have certain relative motion, and the motion form of the object to be detected in the scene becomes the vector synthesis problem of the self motion and the background motion. The method for detecting the moving target under the dynamic background is mainly divided into three methods: template matching, optical flow, and background motion compensation. The template matching method has small calculation amount, but is difficult to find a uniform template to be suitable for various conditions, so the robustness is poor; the optical flow method is easily influenced by illumination change, has poor environmental adaptability and large calculation amount; the common background motion compensation method is divided into a block matching method and a feature point matching method, wherein a feature point is an important local feature in computer vision and has strong stability under the conditions of image illumination change, visual angle change and noise interference, so that the feature point matching method has strong robustness and anti-noise capability.
Due to the potential safety hazard, the monitoring of the longitudinal shaking amplitude of the trailer is very important. In addition, the intelligent level of the existing light rescue wrecker is not high, and the existing scientific and technical means are not utilized to monitor the longitudinal shaking amplitude of the trailer. The visual sensor is widely applied to the fields of safety monitoring and the like due to the advantages of low cost, small volume, convenience in installation and the like, so that the monitoring of the longitudinal shaking amplitude of the trailer based on the visual sensor becomes one of effective means for solving the problems.
Disclosure of Invention
Aiming at the problems, the invention provides a towing process safety early warning method based on multi-scale feature fusion, which realizes effective monitoring of the shaking amplitude of a towed vehicle in the towing and transporting process, thereby achieving the purpose of safety monitoring of the towed vehicle.
The technical scheme adopted by the invention is as follows: a towing process safety early warning method based on multi-scale feature fusion comprises the following operation steps:
1.1, acquiring an image I of a region behind the trailer0Making reproduction to obtain colour image I0'; post-image I0' reduction to the size of the original
Obtaining an image I1(ii) a Then to I1Intercepting to obtain a license plate region-of-interest image I2The interception range is: x in the pixel coordinate system1The value range is (w)1/5,3×w1/5)，y1The value range is (h)1/3,2×h1/3) where x1、y1Is an image I2Middle image point abscissa and ordinate, w1、h1Is an image I1Width and height;
1.2, taking the license plate region as the large-scale feature of the current frame image, and carrying out image I on the license plate region of interest2Carrying out large-scale feature extraction operation to obtain position information of large-scale features, wherein the position information of the large-scale features is license plate region image I3And the width w of the license plate regionp0High h, hp0And coordinate information, which comprises the following specific steps:
1.2.1, image segmentation based on color features:
firstly, according to a color space conversion formula, a color image I is converted2Converting from red, green and blue space to HSL space; then traversing pixel points of the license plate interested image in the HSL space, setting the gray value of the pixel points meeting the blue threshold range to be 255, and otherwise, setting the gray value to be 0, obtaining the image after color segmentation, wherein the threshold range of the blue pixel points is as follows:
h∈[hmin,hmax],s∈[smin,smax],l∈[lmin,lmax](1)，
wherein h, s and l respectively represent the hue, saturation and brightness of the pixel point, hmax、hminIs the high and low threshold values of the hue component, smax、sminHigh and low thresholds for the saturation component,/max、lminHigh and low thresholds for the luminance component;
1.2.2, morphological treatment:
carrying out corrosion operation on the color-segmented image by adopting rectangular structural elements with the size of 3 multiplied by 3 to remove small noise areas; then, expansion operation is carried out by adopting rectangular structural elements with the size of 3 multiplied by 5, holes among characters are filled, license plate areas are communicated, and T license plates with different sizes and shapes are obtainedCommunicating region C oftT1, 2,3.. T; wherein t represents a connected region CtThe sequence number of (1), T is the total number of the region;
1.2.3, license plate positioning:
counting the maximum x of the abscissa of each connected regionmaxtMinimum value xmintWith the maximum value y of the ordinatemaxtMinimum value ymint(ii) a Secondly, the aspect ratio of each connected region is calculated according to the formula (2), which is as follows:
in the formula, rtIs a connected region CtAspect ratio of (2), xmaxt、xmint、ymaxtAnd ymintAre respectively a communication region CtMaximum and minimum values on the abscissa and maximum and minimum values on the ordinate, and t represents a connected region CtT is 1,2,3.. T, which is the total number of connected regions;
then, screening was performed according to the condition (3): if the length-width ratio meets the condition (3), reserving the corresponding connected region, and otherwise, rejecting the connected region; the screening conditions were as follows:
rmin≤rt≤rmax(3)，
wherein r istIs a connected region CtAspect ratio of rmin、rmaxLow and high thresholds representing aspect ratios, respectively;
if the number of the screened connected areas is 0, the license plate positioning is failed, and the step (1.1) is returned; otherwise, the area of the screened connected region is calculated, and the region with the largest area, namely the license plate region, is selected, so that the image I is obtained2Vertex coordinates (x) of upper left corner of middle license plate areap2，yp2) Width wp2And a height hp2(ii) a Then according to the geometric relation, the license plate area is mapped to the original image I0Obtaining the position information of the large-scale feature, namely the vertex coordinate (x) at the upper left corner of the corresponding license plate areap0，yp0) Width wp0And a height hp0(ii) a The mapping relation is as the formula (4) Shown in the figure:
in the formula, w1、h1Are respectively an image I1Is the image scaling, k is the image scaling;
finally, according to the license plate position information, an original image I is processed0Intercepting to obtain license plate area image I on the original image3Intercepting a range: x is the number of0The value range is (x)p0，xp0+wp0)，y0The value range is (y)p0，yp0+hp0) (ii) a Wherein x0、y0Are respectively an image I0Horizontal and vertical coordinates (x) of the middle image pointp0，yp0) Is the top left corner vertex coordinate of the license plate region, wp0、hp0Is the width and height of the license plate region; if the current frame is the first frame image, the vertical coordinate y of the top left corner vertex of the license plate area of the first frame image is set0A value of yp0；
1.3, using the license plate region characteristic points as the small-scale characteristics of the current frame image, and carrying out comparison on the license plate region image I3Extracting small-scale features based on improved BRISK to obtain feature points P of license plate regionjAnd position information of the small-scale features, namely the vertical coordinates of the feature points of the license plate region, J is 1,2jJ is PjWherein BRISK is a binary robust invariant scale feature algorithm. The method comprises the following specific steps:
1.3.1, self-adaptive threshold calculation based on the central region of the license plate:
image I of license plate region3Graying is carried out, n maximum values and n minimum values in the gray value of the pixel point in the central region of the license plate are counted, and the range of the central region of the license plate is as follows:
in the formula, mupc、νpcRespectively a horizontal seat and a vertical seat in the central area of the license plateLabel, wp0、hp0The width and the height of the license plate region image are respectively; then, calculating a threshold value according to the gray value and the contrast of the image, wherein the formula is as follows:
in the formula: f. ofimax、fiminRepresenting the ith gray value of the image, α is a scale factor, epsilon0Is a threshold value;
1.3.2, constructing a BRISK scale space pyramid:
sampling the license plate region gray level image to form N outer layer images and N inner layer images, thereby establishing a BRISK scale space, namely a binary robust invariant scale feature algorithm scale space; the outer layer image is obtained by continuously and semi-sampling the license plate gray level image, the first inner layer image is obtained by down-sampling the gray level image by a sampling factor which is 1.5 times, and the other inner layer images are obtained by continuously and semi-sampling the first inner layer image;
1.3.3, detecting the characteristic points of the scale space:
firstly, extracting candidate characteristic points by adopting a FAST5-8 detection algorithm for an original image and adopting an improved FAST9-16 detection algorithm for each outer layer and each inner layer; secondly, carrying out non-maximum value suppression in a 3 x 3 stereo neighborhood so as to remove unstable candidate feature points; finally, performing sub-pixel correction and scale correction on each detected candidate feature point to obtain a feature point PjThe exact scale and coordinates, J1, 2.., J; the improved FAST9-16 detection algorithm comprises the following steps:
1.3.3.1: counting the number M of continuous pixels of which the gray value on the circumference meets the following conditions on the circumference with the pixel point to be detected as the center and the radius of 3 pixels; if M is larger than 9, the pixel point to be detected is a candidate feature point, otherwise, the substep (1.3.3.2) is carried out; the judgment conditions are as follows:
(Gm＞Gp+ε0)＜or＞(Gm＜Gp-ε0) (7)，
in the formula (I), the compound is shown in the specification,Gmis the gray value of the pixel points on the circumference, m represents the serial number of the pixel points on the circumference, and m is 1,2pIs the gray value of the pixel point to be detected, epsilon0Is an adaptive threshold;
1.3.3.2: calculating the gradient value of the pixel point to be detected according to the Gaussian operator, and if the following conditions are met, taking the pixel point to be detected as a candidate feature point; the judgment conditions are as follows:
wherein M is the number of pixels with continuous gray value mutation on the circumference, grad is the gradient value calculated by Gaussian operator, and ZmaxIs the maximum value of the gradient, ε1Is a threshold value, β is a weight coefficient;
1.3.4: if the current frame image is the first frame, retaining the characteristic points of the first frame image, and returning to the step (1.1); otherwise, continuing to execute the following steps;
1.4, for the characteristic point P of the license plate regionjPerforming KNN-based feature point matching operation, matching the current frame with the first frame of license plate feature points to obtain an effective feature point pair set, wherein J is 1,2jJ is PjThe total number of (c); the matching includes two processes: firstly, determining a candidate characteristic point pair set, belonging to rough matching, and then determining an effective characteristic point pair set aiming at the candidate characteristic point pair set, namely fine matching; the rough matching comprises the following specific steps:
1.4.1, initializing j ═ 1;
1.4.2, for the current frame image feature point PjSearching the nearest neighbor and next nearest neighbor feature points P corresponding to the first frame image through a KNN algorithmj' and PjAnd get the characteristic point PjAnd Pj′、Pj"have Hamming distances of d respectivelyj′、dj"; if it is
The characteristic point PjAnd point Pj' matching is successful, otherwise matching is failed, and rejectingCorresponding to the characteristic point pairs; wherein epsilon2A threshold value that is a ratio of hamming distances;
1.4.3, if J is less than J, increasing J by 1 and entering the step (1.4.2); otherwise, ending the coarse matching process to obtain a candidate characteristic point pair set;
the precise matching process for determining the effective characteristic point pairs comprises the following steps: eliminating mismatching point pairs by a random sampling consistency algorithm for the candidate characteristic point pair set obtained by rough matching, thereby obtaining an effective characteristic point pair set;
1.5, according to the coordinate information of the effective characteristic point pair set, the position information of the multi-scale characteristic is fused to obtain the longitudinal shaking amplitude of the trailer, so that safety early warning is carried out, and the specific steps are as follows:
1.5.1, calculating a longitudinal coordinate difference value of each pair of feature points according to the license plate feature point coordinates;
1.5.2 if the logarithm of the feature point pairs is less than the threshold value epsilon3Skipping the step, and entering the step (1.5.3); otherwise, screening out the maximum value of the difference value of the longitudinal coordinates of the feature point pair set, removing the maximum value, and then entering the step (1.5.3);
1.5.3, counting the average value of the longitudinal coordinate difference values to obtain small-scale feature position information, namely the longitudinal deviation d of the feature point1；
1.5.4, fusing the position information of the multi-scale features according to a formula to obtain the longitudinal shaking amplitude d of the trailer0；
d0＝d1+(y0-yp0) (9)，
Wherein d is1Is the longitudinal deviation, y, of the characteristic points of the license plate0、yp0Respectively are the vertical coordinates of the left upper point of the license plate area of the first frame image and the current frame image; if the amplitude d0Greater than a threshold value epsilon4If yes, alarming; and finally returning to the step (1.1).
The invention has the beneficial effects that: 1. aiming at the potential safety hazard that the towed vehicle is separated from the rescue vehicle due to overlarge road jolt degree in the towing and transporting process of the light rescue vehicle, the invention designs a towing process safety early warning method based on multi-scale feature fusion by combining the characteristics of the towing and transporting of the light rescue vehicle and the characteristics of monitored images of the towed vehicle, and can effectively monitor the longitudinal shaking amplitude of the towed vehicle; 2. the safety early warning method in the towing process has high reliability and strong anti-interference capability. The present invention takes advantage of various features of the license plate region, such as: the license plate is positioned by color features, rectangular features and the like, the anti-interference capability and the robustness are strong, and the vertical shaking amplitude is calculated by adopting a multi-scale feature fusion-based method, so that the accuracy is high; 3. has good real-time performance. The invention analyzes the monitored image characteristics of the trailer, firstly determines the region of interest of the license plate, greatly reduces the calculation amount of the subsequent steps and has good real-time performance.
Drawings
FIG. 1 is a general method flowchart of a towing process safety pre-warning method based on multi-scale feature fusion according to the present invention;
FIG. 2 is a schematic illustration of a tow transport of a light rescue vehicle;
FIG. 3 is a reduced exemplary image, wherein FIG. 3a is a top frame image and FIG. 3b is a non-top frame image;
FIG. 4 is a license plate region-of-interest image, wherein FIG. 4a is a first frame image, and FIG. 4b is a non-first frame image;
FIG. 5 is a color-segmented image, wherein FIG. 5a is a first frame image and FIG. 5b is a non-first frame image;
FIG. 6 is a morphologically processed image, wherein FIG. 6a is a first frame image and FIG. 6b is a non-first frame image;
FIG. 7 is an image of a license plate after positioning, wherein FIG. 7a is a first frame image and FIG. 7b is a non-first frame image;
fig. 8 is a KNN-based feature point matching result image;
FIGS. 9-11 are three images taken while normally transported by a trailer;
FIGS. 12-14 are three images taken with excessive longitudinal trailer oscillation;
fig. 15 is a graph of the matching result of the feature points of fig. 9 and fig. 10 after being processed by the safety precaution method;
fig. 16 is a graph of the matching result of the feature points of fig. 9 and fig. 11 after being processed by the safety precaution method;
fig. 17 is a diagram of a matching result of feature points after being processed by the security early warning method in fig. 12 and 13;
fig. 18 is a graph of the matching result of the feature points of fig. 12 and fig. 14 after being processed by the safety precaution method;
Detailed Description
In the process of towing and transporting the light rescue vehicle, particularly in a complex traffic environment with crowded traffic and changeable road conditions (such as bumpy roads), the towed vehicle is likely to shake violently up and down due to overlarge road bumpiness so as to be separated from the rescue vehicle, so that urban traffic jam is easily caused, and even secondary traffic accidents are caused; due to the potential safety hazard, the monitoring of the longitudinal shaking amplitude of the trailer is crucial; in addition, the intelligent level of the existing light rescue wrecker is not high, and the longitudinal shaking amplitude of the trailer is not monitored by utilizing the existing scientific and technical means; the visual sensor is widely applied to the fields of safety monitoring and the like due to the advantages of low cost, small size, convenience in installation and the like, so that the monitoring of the longitudinal shaking amplitude of the trailer based on the visual sensor becomes one of effective means for solving the problems. (ii) a In the dragging process of the light rescue vehicle, the trailer needs to be detected and tracked in a plurality of frames of images during the monitoring of the longitudinal shaking amplitude of the trailer, and the method belongs to the field of target detection under a dynamic background.
Compared with a common moving object detection scene under a dynamic background, the monitoring image of the towed vehicle has the following three characteristics:
1. in the monitoring video image, the area of the trailer image is larger than that of the background image; if the trailer is used as a moving target for detection, the algorithm is time-consuming and is easily influenced by environmental changes;
2. the license plate of the trailer has abundant characteristics; the trailer license plate has regular shape, obvious color characteristic and large size, is easy to detect and can be used as the large-scale characteristic of a monitoring image to be detected; in addition, the edge points of the license plate characters are dense, the size of the feature points of the license plate region is small and easy to extract, so the feature points of the license plate region can be detected as the small-size features of the monitoring image; therefore, the license plate is selected as the moving target to be detected;
3. during towing transportation, the range of motion of the towed vehicle is limited. Therefore, the invention preprocesses the monitoring image and extracts the region of interest, thereby further improving the real-time performance of the algorithm and reducing the noise interference.
Aiming at the potential safety hazard that a trailer is easy to separate from a rescue vehicle due to the jolt road surface in the process of towing and transporting a light rescue vehicle, the invention provides a towing process safety early warning method based on multi-scale feature fusion by combining the characteristics of a monitored image of the trailer, and the method has the advantages of high accuracy and good real-time performance; in addition, unless otherwise specified, the coordinate values of the image points referred to herein are all numerical values relative to the pixel coordinate system.
As shown in fig. 1, the early warning method includes five steps, which are specifically as follows:
1.1, determining the region of interest of the license plate
The vehicle-mounted camera is mounted on a folding arm at the tail of the light road rescue equipment, referring to fig. 2, the height from the ground is 40-60 cm, the pointed direction is parallel to the longitudinal axis of the rescue vehicle body, and the horizontal direction points to the rear; the size of the image collected by the camera is fixed to 960 multiplied by 540, the license plate is positioned after the image is reduced, and the algorithm real-time performance can be improved; the distance between the trailer and the rescue vehicle is fixed in the dragging and transporting process, and the positions of the license plates of different types of vehicles are not greatly different, so that the calculation amount of the subsequent steps can be reduced by extracting the region of interest of the license plate; the method comprises the following specific steps:
1.1.1, acquiring images of the area behind the trailer to obtain a color image I0；
1.1.2, to image I0Making reproduction to obtain colour image I0′；
1.1.3, image I0' reduction to original size
Obtaining an image I1Referring to fig. 3, k is a reduced scale and is a value within 4 to 6;
1.1.4, to image I1Cutting to obtain a vehicleRegion of interest image I of card2See, fig. 4; intercepting a range: x is the number of1The value range is (w)1/5，3×w1/5)，y1The value range is (h)1/3,2×h13); wherein x1、y1Are respectively an image I2Middle image point abscissa and ordinate, w1、h1Are respectively an image I1Width and height.
1.2, large-scale feature extraction:
at present, license plates of most of domestic social vehicles are in blue background and white characters, so the safety early warning method in the towing process provided by the invention mainly aims at the small-sized vehicles with the license plates in the blue background and white characters under the condition of sufficient light; domestic blue license plates have two typical characteristics: 1. the license plates are all white characters with blue bottoms, and when the illumination is good, the blue pixel points in the image in the area are more and densely distributed; 2. the size of the license plate has the national standard, the license plate is a rectangle with the length of 44 cm and the width of 14 cm, and the length-width ratio is about 3;
according to the characteristics, the invention designs a large-scale characteristic extraction method by taking the number plate of the trailer as the large-scale characteristic of the current frame image: for license plate interested area image I2Firstly, color segmentation is carried out, and blue pixel points in the image are reserved; secondly, performing morphological processing on the image after color segmentation to communicate license plate regions to obtain communicated regions with different shapes and sizes; then, screening is carried out according to the rectangular characteristic and the area characteristic of the license plate, a license plate area is screened out, and according to the geometric relationship, an original image I is subjected to0The license plate is positioned in the middle to obtain a license plate area image I3And width w of license plate regionp0High h, hp0And location information; the method comprises the following specific steps:
1.2.1, image segmentation based on color features: firstly, according to a color space conversion formula, a color image I is converted2Converting from a red, green and blue space to a hue-saturation-luminance space (HSL space); traversing pixel points of the license plate interested image in the HSL space, setting the gray value of the pixel points meeting the blue threshold range to be 255, and otherwise, setting the gray value to be 0, thereby obtaining the image after color segmentation, and referring to FIG. 5; the threshold range of the blue pixel point is as follows:
h∈[hmin,hmax],s∈[smin,smax],l∈[lmin,lmax](1)，
wherein h, s and l respectively represent the hue, saturation and brightness of the pixel points; h ismax、hminIs the high and low threshold values of the hue component, hmaxTaking a value within 235-255, hminTaking values within 90-110; smax、sminHigh and low thresholds, s, of the saturation componentmaxThe value is within 0.8-1, sminTaking a value within 0.15-0.25; lmax、lminIs the high and low threshold of the luminance component,/maxTaking the value within 0.5-0.7, lminTaking a value within 0.05-0.15;
1.2.2, morphological treatment: carrying out corrosion operation on the color-segmented image by adopting rectangular structural elements with the size of 3 multiplied by 3 to remove small noise areas; then, expansion operation is carried out by adopting rectangular structural elements with the size of 3 multiplied by 5, holes among characters are filled, license plate areas are communicated, and T communicated areas C with different sizes and shapes are obtainedtT1, 2,3.. T; wherein t represents a connected region CtThe sequence number of (1), T is the total number of the region; the morphologically processed image is shown in fig. 6;
1.2.3, license plate positioning: counting the maximum x of the abscissa of each connected regionmaxtMinimum value xmintWith the maximum value y of the ordinatemaxtMinimum value ymint(ii) a The aspect ratio of each connected region is then calculated according to equation (2), which is as follows:
in the formula, rtIs a connected region CtAspect ratio of (2), xmaxt、xmint、ymaxtAnd ymintAre respectively a communication region CtMaximum and minimum values on the abscissa and maximum and minimum values on the ordinate, and t represents a connected region CtT is 1,2,3.. T, which is the total number of connected regions;
then, screening was performed according to the condition (3): if the length-width ratio meets the condition (3), reserving the corresponding connected region, and otherwise, rejecting the connected region; the screening conditions were as follows:
rmin≤rt≤rmax(3)，
wherein r istIs a connected region CtAspect ratio of rmin、rmaxRespectively representing low and high thresholds of aspect ratio, rminTaking the value r within 2-2.5maxThe value is within 3.5-4.
If the number of the screened connected regions is 0, the license plate positioning is failed, and the step 1 is returned (namely '1, the region of interest of the license plate') is determined; otherwise, the area of the screened connected region is calculated, and the region with the largest area, namely the license plate region, is selected, so that the image I is obtained2Vertex coordinates (x) of upper left corner of middle license plate areap2，yp2) Width wp2And a height hp2(ii) a Then according to the geometric relation, the license plate area is mapped to the original image I0Obtaining the position information of the large-scale feature, namely the vertex coordinate (x) at the upper left corner of the corresponding license plate areap0，yp0) Width wp0And a height hp0(ii) a The mapping relation is shown in formula (4);
in the formula, w1、h1Are respectively an image I1Is the image scaling, k is the image scaling;
finally, according to the license plate position information, the original image I is processed0Intercepting to obtain license plate area image I on the original image3The interception range is: x is the number of0The value range is (x)p0，xp0+wp0)，y0The value range is (y)p0，yp0+hp0) (ii) a Wherein x is0、y0Are respectively an image I0Abscissa and ordinate of the middle image point, (x)p0，yp0) Is the top left corner vertex coordinate of the license plate region, wp0、hp0Width and height of the license plate region respectively; if the current frame isSetting the top point ordinate y of the left upper corner of the license plate region of the first frame image as the first frame image0A value of yp0(ii) a The license plate image is intercepted, and the figure 7 is shown;
1.3, extracting small-scale features based on improved BRISK:
in the large-scale feature extraction process, a morphological method is adopted, the approximate range of the license plate region can be obtained, but the result is not very accurate and needs to be further refined; the feature point matching method is widely applied in the field of moving target detection and is commonly used for accurate tracking of targets; the BRISK algorithm is a feature extraction algorithm based on a scale space theory, and is widely applied to computer vision because of good real-time performance, rotation invariance, scale invariance and noise robustness; therefore, the invention takes the license plate region characteristic points as the small-scale characteristics of the current frame image, and based on the BRISK algorithm, the longitudinal shaking amplitude of the trailer is accurately calculated;
the BRISK algorithm has two defects in the process of detecting candidate feature points: 1, a threshold value used for screening feature points in a BRISK algorithm is a fixed value, so that the environment adaptability of the algorithm is poor; 2. the algorithm only takes the relation between the pixel points on the circumference and the central point as the detection basis of the candidate characteristic points, and does not consider the information of the pixel points inside the circumference, thereby causing the phenomenon of 'missing detection'. Aiming at the defects, the invention designs a small-scale feature extraction method based on improved BRISK;
the method comprises the steps of firstly calculating an adaptive threshold value epsilon based on the gray value distribution condition of a central region of a license plate0(ii) a Secondly, constructing a BRISK scale space pyramid; then, carrying out scale space feature point detection based on an improved FAST9-16 algorithm to obtain a feature point PjAnd position information of the small-scale feature, i.e., the ordinate of the feature point, J1, 2jJ is PjThe total number of (c). The method comprises the following specific steps:
1.3.1, self-adaptive threshold calculation based on the central region of the license plate: firstly, a license plate region image I3Graying is carried out, and n maximum values and n minimum values in the gray value of the pixel point in the central region of the license plate are countedThe value n is 8-10; the range of the central area of the license plate is as follows:
in the formula, mupc、νpcRespectively are the horizontal and vertical coordinates, w, of the central region of the license platep0、hp0The width and the height of the license plate region image are respectively; then, calculating a threshold value according to the gray value and the contrast of the image, wherein the formula is as follows:
in the formula: f. ofimax、fiminRespectively representing the maximum and minimum ith gray values in the image, wherein n is the maximum number and is a value within 8-10, α is a proportionality coefficient and is a value of 0.15-0.3, and epsilon0Is an adaptive threshold;
1.3.2, constructing a BRISK scale space pyramid: sampling the license plate region gray level image to form N outer layer images and N inner layer images so as to establish a BRISK scale space, wherein the value range of N is 4-6; the outer layer image is obtained by continuously and semi-sampling the license plate gray level image, the first inner layer image is obtained by down-sampling the gray level image by a sampling factor which is 1.5 times, and the other inner layer images are obtained by continuously and semi-sampling the first inner layer image.
1.3.3, detecting the characteristic points of the scale space: firstly, extracting candidate characteristic points by adopting a FAST5-8 detection algorithm for an original image and adopting an improved FAST9-16 detection algorithm for each outer layer and each inner layer; secondly, carrying out non-maximum value suppression in a 3 x 3 stereo neighborhood so as to remove unstable candidate feature points; finally, sub-pixel correction and scale correction are carried out on each detected maximum value to obtain a characteristic point PjThe exact coordinates and dimensions, J, 1, 2. The improved FAST9-16 detection algorithm comprises the following steps:
1.3.3.1, counting the number M of continuous pixels with the gray value meeting the following conditions on the circumference with the pixel to be detected as the center and the radius of 3 pixels; if M is larger than 9, the pixel point to be detected is a candidate feature point; otherwise, go to step (1.3.3.2); the judgment conditions are as follows:
(Gm＞Gp+ε0)＜or＞(Gm＜Gp-ε0) (7)，
in the formula, GmIs the gray value of the pixel points on the circumference, m represents the serial number of the pixel points on the circumference, and m is 1,2pIs the gray value of the pixel point to be detected, epsilon0Is an adaptive threshold;
1.3.3.2: calculating the gradient value of the pixel point to be detected according to the Gaussian operator, and if the following conditions are met, taking the pixel point to be detected as a candidate feature point; the judgment conditions are as follows:
wherein M is the number of pixels with continuous gray value mutation on the circumference, grad is the gradient value calculated by Gaussian operator, and ZmaxIs the maximum value of the gradient, and the value is 250-350 epsilon1Is a threshold value, the value range is 0.6-0.8, β is a weight coefficient, the value is 0.4-0.5.
1.3.4: if the current frame image is the first frame, retaining the characteristic points of the first frame image, and returning to (1.1) (namely '1.1, determining the region of interest of the license plate'); otherwise, continuing to execute the following steps;
two supplementary explanations are made for the improved BRISK algorithm: 1. compared with the whole image, the license plate area is small, is positioned in the center of the image and is relatively uniformly illuminated, so that the distribution condition of the gray value of the license plate center area is representative, and the threshold value epsilon0The method has a relation with the gray value distribution condition of the license plate area, only carries out statistics on the gray value of the pixel points in the center area of the license plate, then calculates the self-adaptive threshold value, and improves the real-time property while ensuring the effectiveness of the algorithm; 2. the gradient value of the pixel point can reflect the relation between the pixel point and surrounding neighborhood points, so that the gradient value is used as an influence factor and added into the judgment of candidate feature points, the number of the feature points can be increased, and the stability of the algorithm is improved;
1.4, KNN-based feature point matching:
the BRISK algorithm calculates the minimum Hamming distance of two groups of binary features one by one, namely, the matching point pairs between images can be obtained, but the matching method is low in speed and has many wrong matching point pairs, so that the algorithm precision and the operation efficiency are influenced; at present, a plurality of feature matching algorithms exist, and the more classical algorithm is a K-nearest neighbor search method (KNN), a fast approximate nearest neighbor search algorithm (fastlibrary for approximate nearest neighbor, FLANN), and the like; because the KNN algorithm has the advantages of simplicity and high efficiency, the invention designs a KNN-based feature matching method;
for the characteristic point P of the license plate regionjCarrying out KNN-based feature point matching operation, and matching the current frame with the first frame of license plate feature points to obtain an effective feature point pair set; the matching includes two processes: firstly, determining a candidate characteristic point pair set, belonging to rough matching, and then determining an effective characteristic point pair set aiming at the candidate characteristic point pair set, namely fine matching; wherein J1, 2, J denotes a feature point PjJ is PjThe total number of (c); the rough matching comprises the following specific steps:
1.4.1: initializing j to 1;
1.4.2: for the current frame image feature point PjSearching the nearest neighbor and next nearest neighbor feature points P corresponding to the first frame image through a KNN algorithmj' and PjAnd get the characteristic point PjAnd Pj′、Pj"have Hamming distances of d respectivelyj′、dj"; if it is
The characteristic point PjAnd point Pj' matching is successful; otherwise, the matching fails, and corresponding characteristic point pairs are removed; wherein epsilon2The threshold value of the Hamming distance ratio is 0.66-0.93;
1.4.3: if J is less than J, increasing J by 1 and entering (1.4.2); otherwise, ending the coarse matching process to obtain a candidate characteristic point pair set;
for the determined candidate characteristic point pair sets, in order to further eliminate the interference of mismatching, validity judgment, namely fine matching, needs to be carried out on the candidate characteristic point pair sets, and the specific steps are as follows: eliminating mismatching point pairs by a Random Sample Consensus (RANSAC) algorithm on the candidate characteristic point pair set obtained by rough matching, thereby obtaining an effective characteristic point pair set; the feature point matching result is shown in fig. 8;
1.5, calculating the longitudinal shaking amplitude of multi-scale feature fusion:
after the effective characteristic point pair set is determined, the position information of the multi-scale characteristics can be fused to obtain the longitudinal shaking amplitude of the trailer, so that safety early warning is carried out; because some mismatching point pairs may exist in the characteristic point pair set, further filtering processing is needed when the longitudinal shaking amplitude of the trailer is calculated; in order to ensure that the calculation process has stronger anti-interference capability and good real-time performance, the method calculates the longitudinal shaking amplitude of the trailer based on mean value filtering, and comprises the following specific steps of:
1.5.1: calculating a longitudinal coordinate difference value of each pair of feature points according to the license plate feature point coordinates;
1.5.2: if the logarithm of the feature point pairs is less than the threshold value epsilon3Skipping the step, and entering the step (1.5.3); otherwise, screening out the maximum value of the difference value of the longitudinal coordinates of the feature point pair set, removing the maximum value, and then entering the step (1.5.3); threshold value epsilon3Taking a value within 6-10;
1.5.3: counting the average value of the longitudinal coordinate difference values to obtain small-scale feature position information, namely the longitudinal deviation d of the feature point1；
1.5.4: fusing the position information of the multi-scale features according to a formula to obtain the longitudinal shaking amplitude d of the trailer0；
d0＝d1+(y0-yp0) (9)，
Wherein d is1Is the longitudinal deviation, y, of the characteristic points of the license plate0、yp0Respectively are the vertical coordinates of the top left corner of the license plate area of the first frame image and the current frame image; if the amplitude d0Greater than a threshold value epsilon4If yes, alarming; finally, returning to (1.1) (namely '1.1, determining the region of interest of the license plate'); wherein the amplitude thresholdValue epsilon4The value is within 60-100.
In order to test the actual effect of the towing process safety early warning method based on multi-scale feature fusion, the towing and transporting experiment of the light rescue vehicle is carried out. The experimental basic conditions are as follows:
purpose of the experiment: the real-time performance and the accuracy of the towing process safety early warning method based on the multi-scale feature fusion are tested.
The experimental system consists of: the experimental system consists of a software safety early warning program and hardware equipment. The towing process safety early warning program is a towing process safety early warning method based on multi-scale feature fusion, and the visual studio2013 is used as a development tool; the main hardware devices include: the computer (CPU is Inter (R) core (TM) i3-6100UCPU (2.3Hz), the memory is 4GB), Haokangwei infrared network camera, pickup type light rescue vehicle, vehicle-mounted power inverter and other related equipment.
The experimental environment is as follows: the towing process safety early warning experiment based on multi-scale feature fusion is carried out in a common road environment, a light rescue vehicle tows and transports a small automobile in the experiment process, a vehicle-mounted camera is mounted on a folding arm at the tail of light road rescue equipment, the height from the ground is 40-60 cm, the pointed direction is parallel to the longitudinal axis of the body of the rescue vehicle, the vehicle-mounted camera points to the rear horizontally, image acquisition is carried out on the area where the small automobile is located, and then the acquired image is processed through an algorithm, so that the accuracy and the real-time performance of the towing process safety early warning method are tested.
The experimental results are as follows: in order to verify the feasibility of the safety early warning method provided by the invention, two groups of videos in the actual towing process are selected for analysis, and three images are selected from each group of videos for experiment. Reference is made to the description accompanying fig. 9 to 14, wherein fig. 9 to 11 are images of normal hauling processes, and fig. 12 to 14 are images of processes in which the trailer is swayed longitudinally to a greater extent. The safety early warning method provided by the invention is used for testing, algorithm evaluation is carried out from two aspects of algorithm operation time and result accuracy, as can be seen from table 1, the time consumption of the large-scale feature extraction algorithm designed herein is less, the average time is about 3.3ms, the total operation average time of the algorithm is 219ms, and as the towing and transporting speed of the light rescue vehicle is low, the safety early warning method based on the multi-scale feature fusion in the towing process basically meets the real-time requirement, as can be seen from table 2, the method provided by the invention has high accuracy.
Experiments show that the towed vehicle early warning method provided by the invention has a good early warning effect, can meet the requirement of monitoring the longitudinal shaking amplitude of the towed vehicle in the towing and transporting process of the light rescue vehicle, and is particularly characterized in that the average time for processing a frame of image by ① is about 200ms, the towing and transporting speed of the light rescue vehicle is slow, so that the real-time requirement is basically met, and the average absolute error of ② for monitoring the longitudinal shaking amplitude of the towed vehicle is 2.25 pixels, so that the accuracy is high.
TABLE 1 safety precaution method time analysis(s)
| Experimental images | FIG. 9+ FIG. 10 | FIG. 9+ FIG. 11 | FIG. 12+ FIG. 13 | FIG. 12+ FIG. 14 |
| Large scale feature extraction | 0.0036 | 0.003 | 0.0032 | 0.0033 |
| Small scale feature extraction | 0.1474 | 0.0954 | 0.1092 | 0.114 |
| Coarse matching | 0.025 | 0.0162 | 0.013 | 0.0142 |
| Precision matching | 0.021 | 0.0906 | 0.0047 | 0.0199 |
| Total run time | 0.2476 | 0.2433 | 0.1759 | 0.2084 |
TABLE 2 safety precaution method accuracy analysis (number of pixels)
| Experimental images | FIG. 9+ FIG. 10 | FIG. 9+ FIG. 11 | FIG. 12+ FIG. 13 | FIG. 12+ FIG. 14 |
| Position deviation of characteristic point | 29 | 49 | 35 | 93 |
| True deviation | 32 | 46 | 33 | 94 |
| Absolute error | 3 | 3 | 2 | 1 |
Claims (5)
1. A towing process safety early warning method based on multi-scale feature fusion is characterized by comprising the following steps: the operation steps are as follows:
1.1, acquiring an image I of a region behind the trailer0Making reproduction to obtain colour image I0'; post-image I0' reduction to the size of the original
Obtaining an image I1(ii) a Then to I1Intercepting to obtain a license plate region-of-interest image I2The interception range is: x in the pixel coordinate system1The value range is (w)1/5,3×w1/5)，y1The value range is (h)1/3,2×h1/3) where x1、y1Is an image I2Middle image point abscissa and ordinate, w1、h1Is an image I1Width and height;
1.2, taking the license plate region as the large-scale feature of the current frame image, and carrying out image I on the license plate region of interest2And performing large-scale feature extraction operation to obtain position information of the large-scale features, wherein the method specifically comprises the following steps:
1.2.1, image segmentation based on color features:
firstly, according to a color space conversion formula, a color image I is converted2Converting from red, green and blue space to HSL space; then traversing pixel points of the license plate interested image in the HSL space, setting the gray value of the pixel points meeting the blue threshold range to be 255, and otherwise, setting the gray value to be 0, obtaining the image after color segmentation, wherein the threshold range of the blue pixel points is as follows:
h∈[hmin,hmax],s∈[smin,smax],l∈[lmin,lmax](1)，
wherein h, s and l respectively represent the hue, saturation and brightness of the pixel point, hmax、hminIs the high and low threshold values of the hue component, smax、sminHigh and low thresholds for the saturation component,/max、lminHigh and low thresholds for the luminance component;
1.2.2, morphological treatment:
carrying out corrosion operation on the color-segmented image by adopting rectangular structural elements with the size of 3 multiplied by 3 to remove small noise areas; then, expansion operation is carried out by adopting rectangular structural elements with the size of 3 multiplied by 5, holes among characters are filled, license plate areas are communicated, and T communicated areas C with different sizes and shapes are obtainedtT1, 2,3.. T; wherein t represents a connected region CtThe sequence number of (1), T is the total number of the region;
1.2.3, license plate positioning:
counting the maximum x of the abscissa of each connected regionmaxtMinimum value xmintWith the maximum value y of the ordinatemaxtMinimum value ymint(ii) a Then calculate each connectivity according to equation (2)The aspect ratio of the region is as follows:
in the formula, rtIs a connected region CtAspect ratio of (2), xmaxt、xmint、ymaxtAnd ymintAre respectively a communication region CtMaximum and minimum values on the abscissa and maximum and minimum values on the ordinate, and t represents a connected region CtT is 1,2,3.. T, which is the total number of connected regions;
then, screening was performed according to the condition (3): if the length-width ratio meets the condition (3), reserving the corresponding connected region, and otherwise, rejecting the connected region; the screening conditions were as follows:
rmin≤rt≤rmax(3)，
wherein r istIs a connected region CtAspect ratio of rmin、rmaxLow and high thresholds representing aspect ratios, respectively;
if the number of the screened connected areas is 0, the license plate positioning is failed, and the step (1.1) is returned; otherwise, the area of the screened connected region is calculated, and the region with the largest area, namely the license plate region, is selected, so that the image I is obtained2Vertex coordinates (x) of upper left corner of middle license plate areap2,yp2) Width wp2And a height hp2(ii) a Then according to the geometric relation, the license plate area is mapped to the original image I0Obtaining the position information of the large-scale feature, namely the vertex coordinate (x) at the upper left corner of the corresponding license plate areap0,yp0) Width wp0And a height hp0(ii) a The mapping relationship is shown in formula (4):
in the formula, w1、h1Are respectively an image I1Is the image scaling, k is the image scaling;
finally, according to the license plate position information, an original image I is processed0Intercepting to obtain license plate area image I on the original image3Intercepting a range: x is the number of0The value range is (x)p0,xp0+wp0)，y0The value range is (y)p0,yp0+hp0) (ii) a Wherein x0、y0Are respectively an image I0Horizontal and vertical coordinates (x) of the middle image pointp0,yp0) Is the top left corner vertex coordinate of the license plate region, wp0、hp0Is the width and height of the license plate region; if the current frame is the first frame image, the vertical coordinate y of the top left corner vertex of the license plate area of the first frame image is set0A value of yp0；
1.3, using the license plate region characteristic points as the small-scale characteristics of the current frame image, and carrying out comparison on the license plate region image I3Extracting small-scale features based on improved BRISK to obtain feature points P of license plate regionjAnd position information of the small-scale features, which comprises the following specific steps:
1.3.1, self-adaptive threshold calculation based on the central region of the license plate:
image I of license plate region3Graying is carried out, n maximum values and n minimum values in the gray value of the pixel point in the central region of the license plate are counted, and the range of the central region of the license plate is as follows:
in the formula, mupc、νpcRespectively are the horizontal and vertical coordinates, w, of the central region of the license platep0、hp0The width and the height of the license plate region image are respectively; then, calculating a threshold value according to the gray value and the contrast of the image, wherein the formula is as follows:
in the formula:
representing the maximum and minimum ith gray scale values in the image, α being the scaleCoefficient of epsilon0Is a threshold value;
1.3.2, constructing a BRISK scale space pyramid:
sampling the license plate region gray level image to form N outer layer images and N inner layer images, thereby establishing a BRISK scale space; the outer layer image is obtained by continuously and semi-sampling the license plate gray level image, the first inner layer image is obtained by down-sampling the gray level image by a sampling factor which is 1.5 times, and the other inner layer images are obtained by continuously and semi-sampling the first inner layer image;
1.3.3, detecting the characteristic points of the scale space:
firstly, extracting candidate characteristic points by adopting a FAST5-8 detection algorithm for an original image and adopting an improved FAST9-16 detection algorithm for each outer layer and each inner layer; secondly, carrying out non-maximum value suppression in a 3 x 3 stereo neighborhood so as to remove unstable candidate feature points; finally, performing sub-pixel correction and scale correction on each detected candidate feature point to obtain a feature point PjThe exact scale and coordinates, J1, 2.., J; the improved FAST9-16 detection algorithm comprises the following steps:
1.3.3.1: counting the number M of continuous pixels of which the gray value on the circumference meets the following conditions on the circumference with the pixel point to be detected as the center and the radius of 3 pixels; if M is larger than 9, the pixel point to be detected is a candidate feature point, otherwise, the substep (1.3.3.2) is carried out; the judgment conditions are as follows:
(Gm＞Gp+ε0)＜or＞(Gm＜Gp-ε0) (7)，
in the formula, GmIs the gray value of the pixel points on the circumference, m represents the serial number of the pixel points on the circumference, and m is 1,2pIs the gray value of the pixel point to be detected, epsilon0Is an adaptive threshold;
1.3.3.2: calculating the gradient value of the pixel point to be detected according to the Gaussian operator, and if the following conditions are met, taking the pixel point to be detected as a candidate feature point; the judgment conditions are as follows:
wherein M is the number of pixels with continuous gray value mutation on the circumference, grad is the gradient value calculated by Gaussian operator, and ZmaxIs the maximum value of the gradient, ε1Is a threshold value, β is a weight coefficient;
1.3.4: if the current frame image is the first frame, retaining the characteristic points of the first frame image, and returning to the step (1.1); otherwise, the following steps are continuously executed.
1.4, for the characteristic point P of the license plate regionjPerforming KNN-based feature point matching operation, matching the current frame with the first frame of license plate feature points to obtain an effective feature point pair set, wherein J is 1,2jJ is PjThe total number of (c); the matching includes two processes: firstly, determining a candidate characteristic point pair set, belonging to rough matching, and then determining an effective characteristic point pair set aiming at the candidate characteristic point pair set, namely fine matching; the rough matching comprises the following specific steps:
1.4.1, initializing j ═ 1;
1.4.2, for the current frame image feature point PjSearching the nearest neighbor and next nearest neighbor feature points P corresponding to the first frame image through a KNN algorithmj' and PjAnd get the characteristic point PjAnd Pj′、Pj"have Hamming distances of d respectivelyj′、dj"; if it is
The characteristic point PjAnd point PjMatching is successful, otherwise, matching is failed, and corresponding characteristic point pairs are removed; wherein epsilon2A threshold value that is a ratio of hamming distances;
1.4.3, if J is less than J, increasing J by 1 and entering the step (1.4.2); otherwise, ending the coarse matching process to obtain a candidate characteristic point pair set;
the precise matching process for determining the effective characteristic point pairs comprises the following steps: and eliminating mismatching point pairs by a random sampling consistency algorithm for the candidate characteristic point pair set obtained by rough matching, thereby obtaining an effective characteristic point pair set.
1.5, according to the coordinate information of the effective characteristic point pair set, the position information of the multi-scale characteristic is fused to obtain the longitudinal shaking amplitude of the trailer, so that safety early warning is carried out, and the specific steps are as follows:
1.5.1, calculating a longitudinal coordinate difference value of each pair of feature points according to the license plate feature point coordinates;
1.5.2 if the logarithm of the feature point pairs is less than the threshold value epsilon3Skipping the step, and entering the step (1.5.3); otherwise, screening out the maximum value of the difference value of the longitudinal coordinates of the feature point pair set, removing the maximum value, and then entering the step (1.5.3);
1.5.3, counting the average value of the longitudinal coordinate difference values to obtain small-scale feature position information, namely the longitudinal deviation d of the feature point1；
1.5.4, fusing the position information of the multi-scale features according to a formula to obtain the longitudinal shaking amplitude d of the trailer0；
d0＝d1+(y0-yp0) (9)，
Wherein d is1Is the longitudinal deviation, y, of the characteristic points of the license plate0、yp0Respectively are the vertical coordinates of the left upper point of the license plate area of the first frame image and the current frame image; if the amplitude d0Greater than a threshold value epsilon4If yes, alarming; and finally returning to the step (1.1).
2. The towing process safety early warning method based on multi-scale feature fusion as claimed in claim 1, wherein the position information of the large-scale features is a license plate region image I3And the width w of the license plate regionp0High h, hp0And coordinate information.
3. The towing process safety early warning method based on multi-scale feature fusion according to claim 1, wherein the position information of the small-scale features is the ordinate of the feature point of the license plate region, J is 1,2jJ is PjThe total number of (c).
4. The towing process safety precaution method based on multi-scale feature fusion of claim 1, characterized in that the HSL space is hue-saturation-brightness space.
5. The towing process safety early warning method based on multi-scale feature fusion as claimed in claim 1, wherein the BRISK algorithm is a binary robust invariant scale feature algorithm, and is an algorithm for feature point extraction in the field of digital images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910506741.XA CN111145184B (en) | 2019-06-12 | 2019-06-12 | Dragging process safety early warning method based on multi-scale feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910506741.XA CN111145184B (en) | 2019-06-12 | 2019-06-12 | Dragging process safety early warning method based on multi-scale feature fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111145184A true CN111145184A (en) | 2020-05-12 |
| CN111145184B CN111145184B (en) | 2022-07-29 |
Family
ID=70516727
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910506741.XA Active CN111145184B (en) | 2019-06-12 | 2019-06-12 | Dragging process safety early warning method based on multi-scale feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111145184B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145797A (en) * | 2018-08-13 | 2019-01-04 | 东南大学 | Light-duty roadside assistance equipment direct bearing based on vehicle bottom shadow character positioning licence plate, which drags, leads abductive approach |
| CN109285169A (en) * | 2018-08-13 | 2019-01-29 | 东南大学 | A kind of roadside assistance equipment side coil based on wheel identification drags and leads abductive approach |
| CN109376733A (en) * | 2018-08-13 | 2019-02-22 | 东南大学 | A kind of roadside assistance equipment direct bearing based on License Plate drags and leads abductive approach |
-
2019
- 2019-06-12 CN CN201910506741.XA patent/CN111145184B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145797A (en) * | 2018-08-13 | 2019-01-04 | 东南大学 | Light-duty roadside assistance equipment direct bearing based on vehicle bottom shadow character positioning licence plate, which drags, leads abductive approach |
| CN109285169A (en) * | 2018-08-13 | 2019-01-29 | 东南大学 | A kind of roadside assistance equipment side coil based on wheel identification drags and leads abductive approach |
| CN109376733A (en) * | 2018-08-13 | 2019-02-22 | 东南大学 | A kind of roadside assistance equipment direct bearing based on License Plate drags and leads abductive approach |
Non-Patent Citations (1)
| Title |
|---|
| 徐奔等: "基于BRISK的实时视频抖动检测算法", 《计算机工程与设计》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111145184B (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109816024B (en) | Real-time vehicle logo detection method based on multi-scale feature fusion and DCNN | |
| CN110487562B (en) | Driveway keeping capacity detection system and method for unmanned driving | |
| CN104778444B (en) | The appearance features analysis method of vehicle image under road scene | |
| CN103077384B (en) | A kind of method and system of vehicle-logo location identification | |
| CN105206109B (en) | A kind of vehicle greasy weather identification early warning system and method based on infrared CCD | |
| CN111274843B (en) | Truck overload monitoring method and system based on monitoring video | |
| CN106127107A (en) | The model recognizing method that multi-channel video information based on license board information and vehicle's contour merges | |
| CN110414355A (en) | The right bit sky parking stall of view-based access control model and parking stall line detecting method during parking | |
| CN103020948A (en) | Night image characteristic extraction method in intelligent vehicle-mounted anti-collision pre-warning system | |
| CN105678287B (en) | A kind of method for detecting lane lines based on ridge measurement | |
| Zhang et al. | Vehicle recognition algorithm based on Haar-like features and improved Adaboost classifier | |
| CN105139011B (en) | A kind of vehicle identification method and device based on mark object image | |
| CN202134079U (en) | Unmanned vehicle lane marker line identification and alarm device | |
| CN107808392A (en) | The automatic method for tracking and positioning of safety check vehicle and system of open scene | |
| CN106407951A (en) | Monocular vision-based nighttime front vehicle detection method | |
| CN112699267B (en) | Vehicle type recognition method | |
| CN111915583A (en) | Vehicle and pedestrian detection method based on vehicle-mounted thermal infrared imager in complex scene | |
| CN114299002A (en) | Intelligent detection system and method for abnormal road surface throwing behavior | |
| CN109145797B (en) | Square induction method for road rescue equipment based on vehicle bottom shadow positioning license plate | |
| CN108268866B (en) | Vehicle detection method and system | |
| CN112052768A (en) | Urban illegal parking detection method and device based on unmanned aerial vehicle and storage medium | |
| CN112666573A (en) | Detection method for retaining wall and barrier behind mine unloading area vehicle | |
| CN111145184B (en) | Dragging process safety early warning method based on multi-scale feature fusion | |
| CN109086671B (en) | Night lane marking line video detection method suitable for unmanned driving | |
| Li-Sheng et al. | Preceding vehicle detection based on multi-characteristics fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |