CN115984658B - Multi-sensor fusion vehicle window recognition method, system and readable storage medium - Google Patents
Multi-sensor fusion vehicle window recognition method, system and readable storage medium Download PDFInfo
- Publication number
- CN115984658B CN115984658B CN202310066606.4A CN202310066606A CN115984658B CN 115984658 B CN115984658 B CN 115984658B CN 202310066606 A CN202310066606 A CN 202310066606A CN 115984658 B CN115984658 B CN 115984658B
- Authority
- CN
- China
- Prior art keywords
- feature
- point cloud
- data
- illumination
- vehicle window
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The application provides a multi-sensor fusion vehicle window recognition method, a system and a readable storage medium, which are used for carrying out illumination compensation on rgb image data according to illumination data and extracting image characteristics; collecting temperature sensor data and a laser radar point cloud array; compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point; carrying out pixel-level fusion on the image characteristics and the point cloud characteristics; and carrying out vehicle window identification according to the fused features. And (3) carrying out illumination data compensation and fusion on the original rgb image by using a dynamic SSR algorithm, enhancing the image, reducing the influence of severe illumination on the vehicle window identification, collecting temperature sensor data, carrying out temperature compensation and fusion on a Lei Dadian cloud array based on a tested laser radar distance temperature compensation system, and reducing the influence of inaccurate point cloud data caused by temperature on the vehicle window identification.
Description
Technical Field
The application relates to the field of vehicle window recognition, in particular to a multi-sensor fusion vehicle window recognition method, a system and a readable storage medium.
Background
In the prior art, the vehicle window is identified, the bright/dark/area of the rgb image is easy to be excessively bright and dark under the severe illumination environment, the original rgb image is directly used for feature extraction under the severe illumination, the subsequent vehicle window detection is easy to have poor effect, meanwhile, under the condition of low temperature, the point cloud data is often invalid or inaccurate in position depth data deviating from a central area, the condition of small volume is generated during volume measurement, the noise of the edge area is increased under the condition of high temperature, the phenomenon that the internal height is outwards expanded is sometimes reflected, and thus the acquired initial data is inaccurate, and the vehicle window is identified.
The above problems are currently in need of solution.
Disclosure of Invention
The application aims to provide a multi-sensor fusion vehicle window identification method, a multi-sensor fusion vehicle window identification system and a readable storage medium.
In order to solve the technical problems, the application provides a multi-sensor fusion vehicle window identification method, which comprises the following steps:
collecting illumination data and rgb image data;
performing illumination compensation on the rgb image data according to the illumination data, and extracting image features;
collecting temperature sensor data and a laser radar point cloud array;
compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point;
carrying out pixel-level fusion on the image characteristics and the point cloud characteristics;
and carrying out vehicle window identification according to the fused features.
Further, the method for performing illumination compensation on the rgb image data according to the illumination data, namely dynamically performing illumination compensation on the rgb image by using Single Scale Retinex algorithm based on the illumination data, has the following expression:
wherein I (x, y) is an original image, R (x, y) is a reflection component, L (x, y) is an illumination component, I represents an I-th color channel, x represents convolution, and G (x, y) is a gaussian surround function;
the formula of G (x, y) is as follows:and lambda satisfies:
∫∫G(x,y)dxdy＝1；
where σ is the scale parameter of the gaussian surround, which is dynamically adjusted based on the illumination component Li,for the adjustment factor, lx is the illuminance unit:
r (x, y) is obtained by converting the logarithmic domain into the real domain after R (x, y) is obtained, and then linear stretching treatment is carried out to obtain an output image, wherein the final linear stretching formula is as follows:
wherein ssred_img is image data obtained by dynamically performing illumination compensation on the rgb image through a SingleScaleRetinex algorithm.
Further, the step of extracting the image features includes:
inputting the compensated image data into a trained ResNeXtCNN network to obtain a featuremap;
and setting the featuremap to obtain a candidate ROI, sending the candidate ROI into an RPN network to perform classification filtering part ROI, performing ROIAlign operation, and corresponding the rgb image data with the pixels of the featuremap to obtain a feature map of the candidate region ROI.
Further, the step of compensating the laser radar point cloud array according to the temperature sensor data and extracting the point cloud features point by point includes:
collecting distance data measured by laser radar at different temperatures
The distance data is used to construct a distance temperature variation relationship,
generating a relation between compensation time and temperature according to the relation between the light speed and the distance;
finally, searching a corresponding compensation distance in a spline interpolation table according to the current temperature of the laser radar, and calculating the compensation distance of each depth data in the point cloud array, thereby completing the compensation of the point cloud array;
and inputting the compensated point cloud array into a point cloud feature extraction network to extract features, and obtaining point cloud features.
Further, the step of identifying the vehicle window according to the fused features includes:
sending the pixel-level fusion features into the FC layer to obtain feature vectors feature pre For feature vector feature pre Performing Sigmoid operation to perform confidence judgment;
frame regression algorithm using RCNN, input feature pre Processing to obtain an offset function, and further performing offset adjustment on the original frame;
performing bilinear interpolation scaling on the pixel-level fusion features meeting the threshold requirement to the size of an ROI alignment feature map, and then sending the feature map into an FCN (fuzzy c-means) to generate window masks, wherein the size of each mask is as follows: m is m;
and performing size operation and background filling on the window mask by using the original Propos al and the image size to obtain the mask corresponding to the original image size, performing mask and frame regression on the original image, and simultaneously displaying the confidence judgment result of the Sigmoid branch on the corresponding region to finish the window recognition.
Further, the offset function is:
where P represents the original Proposal, phi (P) represents the input eigenvector, d * (P) represents a predictive offset value, W represents a learned parameter;
offset adjustment of the original frame includes translation and size scaling:
translation (Δm, Δn) is expressed as:
Δx＝P w d M (P),ΔN＝P h d N (P)；
size scaling(s) w ,s h ) Expressed as:
s w ＝P w d w (P),s h ＝P h d h (P)。
further, the step of performing pixel-level fusion on the image features and the point cloud features includes:
the color characteristics of each pixel point and the depth characteristics of the corresponding point cloud point are spliced on a channel to obtain a group of fused features, and the process is as follows:
fator 1 ＝f cat (pcloud,pcolor)
fator 2 ＝f cat (pcloud 1 ,pcolor 1 )
fused_feature＝f link (fator 1 ,fator 2 )
wherein pc feature 、rgb feature Respectively extracting point cloud characteristics and image characteristics, f cat The operation of the splice is indicated and,is the relu activation function, (-) represents the 1*1 convolution operation, f link Representing a splice operation on a channel;
sending the fused feature into mlp, and obtaining the globalfeature by utilizing the averagefeature
feature global ＝f avg_pooling (σ(fused_feature))
f avg_pooling For global average pooling operations, σ is mlp processing operations;
splicing the globalfeature and the fused feature on a channel to obtain a pixel-level fusion feature, namely a dense fused feature:
feature dense ＝f link (fused feature ,feature global )
feature dense i.e., densefusedfeature.
The application also provides a multi-sensor fusion vehicle window recognition system, which comprises:
a first acquisition module adapted to acquire illumination data and rgb image data
The first processing module is suitable for carrying out illumination compensation on the rgb image data according to the illumination data and extracting image characteristics;
the second acquisition module is suitable for acquiring temperature sensor data and a laser radar point cloud array;
the second processing module is suitable for compensating the laser radar point cloud array according to the temperature sensor data and extracting point cloud characteristics point by point;
the fusion module is suitable for carrying out pixel-level fusion on the image characteristics and the point cloud characteristics;
and the identification module is suitable for carrying out vehicle window identification according to the fused features.
The application also provides a computer readable storage medium, wherein at least one instruction is stored in the computer readable storage medium, and the instruction realizes the multi-sensor fusion vehicle window identification method when being executed by a processor.
The application also provides an electronic device, which comprises a memory and a processor; at least one instruction is stored in the memory; and the processor loads and executes the at least one instruction to realize the multi-sensor fusion vehicle window identification method.
The application has the beneficial effects that the application provides a multi-sensor fusion vehicle window identification method, a system and a readable storage medium, wherein the multi-sensor fusion vehicle window identification method comprises the steps of collecting illumination data and rgb image data; performing illumination compensation on the rgb image data according to the illumination data, and extracting image features; collecting temperature sensor data and a laser radar point cloud array; compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point; carrying out pixel-level fusion on the image characteristics and the point cloud characteristics; and carrying out vehicle window identification according to the fused features. And (3) carrying out illumination data compensation and fusion on the original rgb image by using a dynamic SSR algorithm, enhancing the image, reducing the influence of severe illumination on the vehicle window identification, collecting temperature sensor data, carrying out temperature compensation and fusion on a Lei Dadian cloud array based on a tested laser radar distance temperature compensation system, and reducing the influence of inaccurate point cloud data caused by temperature on the vehicle window identification.
Drawings
The application will be further described with reference to the drawings and examples.
Fig. 1 is a flowchart of a method for identifying a multi-sensor fusion vehicle window according to an embodiment of the present application.
Fig. 2 is a schematic block diagram of a multi-sensor fusion vehicle window recognition system according to an embodiment of the present application.
Fig. 3 is a partial schematic block diagram of an electronic device provided by an embodiment of the present application.
Detailed Description
The application will now be described in further detail with reference to the accompanying drawings. The drawings are simplified schematic representations which merely illustrate the basic structure of the application and therefore show only the structures which are relevant to the application.
Example 1
Referring to fig. 1-3, an embodiment of the present application provides a multi-sensor fusion vehicle window recognition method, which uses a dynamic SSR algorithm to compensate and fuse illumination data of an original rgb image, enhance the image and reduce the influence of severe illumination on vehicle window recognition, collects temperature sensor data, and performs temperature compensation and fusion on a Lei Dadian cloud array based on a tested laser radar distance temperature compensation system to reduce the influence of inaccurate point cloud data caused by temperature on vehicle window recognition.
Specifically, the multi-sensor fusion vehicle window recognition method includes:
s110: illumination data and rgb image data are acquired.
Specifically, illumination data acquisition is performed through illumination sensor data, and rgb image data acquisition is performed through a 2D camera.
S120: and carrying out illumination compensation on the rgb image data according to the illumination data, and extracting image characteristics.
Specifically, the method for performing illumination compensation on the rgb image data according to the illumination data, namely, dynamically performing illumination compensation on the rgb image by using a singlescaleRetinex algorithm based on the illumination data, has the following expression:
wherein I (x, y) is an original image, R (x, y) is a reflection component, L (x, y) is an illumination component, I represents an I-th color channel, x represents convolution, and G (x, y) is a gaussian surround function;
the formula of G (x, y) is as follows:and lambda satisfies:
∫∫G(x,y)dxdy＝1；
where σ is the scale parameter of the gaussian surround, which is dynamically adjusted based on the illumination component Li,for the adjustment factor, lx is the illuminance unit:
r (x, y) is obtained by converting the logarithmic domain into the real domain after R (x, y) is obtained, and then linear stretching treatment is carried out to obtain an output image, wherein the final linear stretching formula is as follows:
wherein ssred_img is image data obtained by dynamically performing illumination compensation on the rgb image through Single Scale Retinex algorithm.
The step of extracting the image features comprises the following steps:
inputting the compensated image data into a trained ResNeXt CNN network to obtain feature map;
the feature map is set to obtain candidate ROIs, the candidate ROIs are sent to an RPN network to carry out classification filtering part ROIs, then ROI alignment operation is carried out, and the rgb image data and the pixels of the feature map are correspondingly arranged to obtain a feature map of the candidate region ROIs.
Specifically, in order to obtain a feature map of a fixed size, ROI alignment uses bilinear interpolation, and the pixel values for a virtual point in the original image are determined in common by using four actually existing pixel values around the virtual point:
φ(x,y)＝(x-x 1 )(y-y 1 )φ(x 2 ,y 2 )+(x-x 2 )(y-y 1 )φ(x 1 ,y 2 )+(x-x 1 )(y-y 2 )φ(x 2 ,y 1 )+(x-x 1 )(y-y 1 )φ(x 2 ,y 2 )
phi () represents the pixel value corresponding to the coordinate point, (x) 1 ,y 2 )、(x 1 ,y 1 )、(x 2 ,y 2 )、(x 2 ,y 1 ) The coordinates of the virtual point are respectively the upper left, lower left, upper right and lower right coordinates, and (x, y) is the coordinate corresponding to the obtained virtual point.
After ROIAlign, taking the characteristic of scaling to original pictures after bilinear interpolation, the structure is as follows:
[H,W,dim]
dim is the dimension, H, W artwork ROI region size scalar.
S130: and acquiring temperature sensor data and a laser radar point cloud array.
S140: and compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point.
In the present embodiment, S140 includes the steps of:
s141: and acquiring distance data measured by the laser radar at different temperatures.
S142: and constructing a distance temperature change relation by using the distance data.
Specifically, the constructed distance temperature change curve equation is:
s＝7e -5 f 3 -0.0123f 2 -6.1153f+792.59
and s is the distance, f is the temperature, the compensation time corresponding to the current temperature of the radar is calculated, and the time compensation of the radar is completed by performing time compensation on the shutter signal.
S143: and generating a relation between the compensation time and the temperature according to the relation between the light speed and the distance.
Specifically, the initial point temperature of the laser radar reaching the standard is set to be F 0 Substituting the s into the distance temperature change curve equation to obtain the corresponding s 0 ；
The equation of the curve of the compensation distance deltas as a function of the temperature f is:
Δs＝7e -5 f 3 -0.0123f 2 -6.1153f+792.59-s 0
according to the speed of light distance formula s=0.5×c×Δt, wherein Δt is the sum of the laser emission and return time of the lidar, the function relation equation of the compensation time Δt and the temperature f is obtained as follows:
Δt＝＝[7E-05f 3 -0.0123f 2 -6.1153f+(792.59-s 0 )]/599.584916。
on the basis of time domain compensation, collecting the distances measured by the laser radar at each temperature to form a second group of data of the distance along with the temperature change; and performing cubic spline interpolation on the second group of data to generate a spline interpolation table, and constructing the relationship between the temperature and the compensation distance.
The interpolation function is:
res＝interp1(f,s,xq,'spline')
f is the acquired temperature, s is the distance of the corresponding temperature point, xq is the interpolation interval, and the corresponding compensation distance is searched in a spline interpolation table according to the current temperature of the laser radar, so that the compensation distance of the depth data is calculated. Wherein, first find the calibration initial temperature F 0 And the corresponding compensation distance is next, and the final compensation distance is the compensation distance minus the initial value of the compensation distance.
S144: and finally, searching a corresponding compensation distance in a spline interpolation table according to the current temperature of the laser radar, and calculating the compensation distance of each depth data in the point cloud array, thereby completing the compensation of the point cloud array.
The final compensated distance vector is expressed as:
Δs end ＝query(F)-query(F 0 )
where F is the current temperature, query () represents a lookup operation, Δs end Is a compensation distance vector.
Compensating the input point cloud data:
point_cloud i ＝(a i ,b i ,c i )+Δs end
point_closed is a point cloud array, i represents the ith point of the point cloud array, and a, b and c are coordinate values of each axis of xyz of the point cloud point in a point cloud three-dimensional coordinate system.
S145: and inputting the compensated point cloud array into a point cloud feature extraction network to perform feature extraction to obtain point cloud features, wherein the network takes a PCTR unit (point cloud converter unit) as a core.
Specifically, step S145 includes the steps of:
(1) FPS (furthest point sampling) is carried out on the point cloud, local coordinates and local features of the point cloud are extracted by combining a K approach method, and the local coordinates and the local features are respectively sent into a local feature extraction unit, a local PTCR unit and a local jump connection unit to extract local high-dimensional features in different feature subspaces.
(2) Feature linking and fusion
Three different local features are added through a matrix, spliced with global features in feature dimensions, and feature fusion is carried out through a layer of nonlinear convolution, wherein the formula is as follows:
wherein f lout 、f skip Is the output of the local feature extraction unit and the local jump connection unit, f lo_pctr And f gl_pctr Is the output of the local PCTR unit and the global PCTR unit, ψ fushion Is a single-layer non-linear convolution,and connecting the characteristic channels in proportion through the expansion dimension, and splicing in the dimension of the characteristic channels.
(5) In the decoding stage, the point-by-point cloud characteristics are finally obtained through up-sampling and jump connection of reverse interpolation, and the specific operation of reverse interpolation is as follows:
wherein X represents the point in the up-sampled point cloud feature set, X i Representing an existing point cloud feature set f sa Points omega in (2) i And (3) performing table weighting operation, wherein C represents the number of point clouds.
After reverse interpolation, if the transmitted characteristics lack local information, local jump connection is carried out, and finally point-by-point cloud characteristics are obtained, and the structure is expressed as
C*(d+F)
Wherein C represents the point cloud number, d represents the space coordinate dimension, and F represents the feature dimension.
S150: carrying out pixel-level fusion on the image characteristics and the point cloud characteristics;
specifically, the color features of each pixel point and the depth features of the corresponding point cloud point are spliced on the channel to obtain a group of fused features, and the process is as follows:
fator 1 ＝f cat (pcloud,pcolor)
fator 2 ＝f cat (pcloud 1 ,pcolor 1 )
fused_feature＝f link (fator 1 ,fator 2 )
wherein pc feature 、rgb feature Respectively extracting point cloud characteristics and image characteristics, f cat The operation of the splice is indicated and,is the relu activation function, (-) represents the 1*1 convolution operation, f link Representing a splice operation on a channel;
sending the filled feature into mlp, and obtaining global feature by using average feature
feature global ＝f avg_pooling (σ(fused_feature))
f avg_pooling For global average pooling operations, σ is mlp processing operations;
splicing the global feature and the fused feature on a channel to obtain a pixel-level fusion feature, namely dense fused feature:
feature dense ＝f link (fused feature ,feature global )
feature dense i.e. dense fused feature.
S160: carrying out vehicle window identification according to the fused characteristics;
step S160 includes the steps of:
sending the pixel-level fusion features into the FC layer to obtain feature vectors featyre pre ；
For feature vector featyre pre Performing Sigmoid operation to perform confidence judgment;
pre_truth i ＝Sigmoid(feature pre )。
frame regression algorithm using RCNN, input feature pre And (3) processing to obtain an offset function, and further performing offset adjustment on the original frame. d, d M ,d N ,d w ,d h Wherein (M, N, w, h) respectively represents the coordinates of the central point of the window and the width and height of the window, and further the original frame is subjected to offset adjustment.
Wherein the offset function is:
where P represents the original Proposal, phi (P) represents the input eigenvector, d * (P) represents a predictive offset value, W represents a learned parameter;
offset adjustment of the original frame includes translation and size scaling:
translation (Δm, Δn) is expressed as:
ΔM＝P w d M (P,ΔN＝P h d N (P)；
size scaling(s) w ,s h ) Expressed as:
s w ＝P w d w (P,s h ＝P h d h (P)。
performing bilinear interpolation scaling on the pixel-level fusion features meeting the threshold requirement to the size of an ROI alignment feature map, and then sending the feature map into an FCN (fuzzy c-means) to generate window masks, wherein the size of each mask is as follows: m is m;
and performing size operation and background filling on the window mask by using the original Propos al and the image size to obtain the mask corresponding to the original image size, performing mask and frame regression on the original image, and simultaneously displaying the confidence judgment result of the Sigmoid branch on the corresponding region to finish the window recognition.
Example 2
The embodiment provides a multi-sensor fusion vehicle window recognition system. The multi-sensor fusion vehicle window recognition system includes:
the first acquisition module is suitable for acquiring illumination data and rgb image data. In this embodiment, the first acquisition module is adapted to implement step S110 in embodiment 1.
The first processing module is suitable for carrying out illumination compensation on the rgb image data according to the illumination data and extracting image features. In this embodiment, the first processing module is adapted to implement step S120 in embodiment 1.
And the second acquisition module is suitable for acquiring temperature sensor data and the laser radar point cloud array. In this embodiment, the second acquisition module is adapted to implement step S130 in embodiment 1.
The second processing module is suitable for compensating the laser radar point cloud array according to the temperature sensor data and extracting point cloud characteristics point by point. In this embodiment, the first acquisition module is adapted to implement step S110 in embodiment 1. In this embodiment, the second processing module is adapted to implement step S140 in embodiment 1.
And the fusion module is suitable for carrying out pixel-level fusion on the image characteristics and the point cloud characteristics. In this embodiment, the fusion module is adapted to implement step S150 in embodiment 1.
And the identification module is suitable for carrying out vehicle window identification according to the fused features. In this embodiment, the identification module is adapted to implement step S160 in embodiment 1.
Example 3
The present embodiment provides a computer-readable storage medium having at least one instruction stored therein, which when executed by a processor implements the multi-sensor fusion vehicle window recognition method provided in embodiment 1.
The multi-sensor fusion car window recognition method comprises the steps of carrying out illumination compensation on rgb image data according to illumination data, and extracting image features; collecting temperature sensor data and a laser radar point cloud array; compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point; carrying out pixel-level fusion on the image characteristics and the point cloud characteristics; and carrying out vehicle window identification according to the fused features. And (3) carrying out illumination data compensation and fusion on the original rgb image by using a dynamic SSR algorithm, enhancing the image, reducing the influence of severe illumination on the vehicle window identification, collecting temperature sensor data, carrying out temperature compensation and fusion on a Lei Dadian cloud array based on a tested laser radar distance temperature compensation system, and reducing the influence of inaccurate point cloud data caused by temperature on the vehicle window identification.
Example 4
Referring to fig. 3, the present embodiment provides an electronic device, including: a memory 502 and a processor 501; at least one program instruction is stored in the memory 502; the processor 501, by loading and executing the at least one program instruction, implements the multi-sensor fusion window recognition method as provided in embodiment 1.
The memory 502 and the processor 501 are connected by a bus, which may include any number of interconnected buses and bridges, which connect together the various circuits of the one or more processors 501 and the memory 502. The bus may also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or may be a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 501 is transmitted over a wireless medium via an antenna, which further receives the data and transmits the data to the processor 501.
The processor 501 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 502 may be used to store data used by processor 501 in performing operations.
In summary, the application provides a method, a system and a readable storage medium for identifying a multi-sensor fusion vehicle window, wherein the method for identifying the multi-sensor fusion vehicle window comprises the steps of collecting illumination data and rgb image data; performing illumination compensation on the rgb image data according to the illumination data, and extracting image features; collecting temperature sensor data and a laser radar point cloud array; compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point; carrying out pixel-level fusion on the image characteristics and the point cloud characteristics; and carrying out vehicle window identification according to the fused features. And (3) carrying out illumination data compensation and fusion on the original rgb image by using a dynamic SSR algorithm, enhancing the image, reducing the influence of severe illumination on the vehicle window identification, collecting temperature sensor data, carrying out temperature compensation and fusion on a Lei Dadian cloud array based on a tested laser radar distance temperature compensation system, and reducing the influence of inaccurate point cloud data caused by temperature on the vehicle window identification.
The components (components not illustrating the specific structure) selected in the present application are common standard components or components known to those skilled in the art, and the structures and principles thereof are known to those skilled in the art through technical manuals or through routine experimental methods. Moreover, the software program related to the application is the prior art, and the application does not relate to any improvement on the software program.
In the description of embodiments of the present application, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art.
In the description of the present application, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
With the above-described preferred embodiments according to the present application as an illustration, the above-described descriptions can be used by persons skilled in the relevant art to make various changes and modifications without departing from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the description, but must be determined according to the scope of claims.
Claims (9)
1. The multi-sensor fusion vehicle window identification method is characterized by comprising the following steps of:
collecting illumination data and rgb image data;
performing illumination compensation on the rgb image data according to the illumination data, and extracting image features;
collecting temperature sensor data and a laser radar point cloud array;
compensating the laser radar point cloud array according to the temperature sensor data, and extracting point cloud characteristics point by point;
carrying out pixel-level fusion on the image characteristics and the point cloud characteristics;
carrying out vehicle window identification according to the fused characteristics;
the method for carrying out illumination compensation on the rgb image data according to the illumination data is characterized in that the Single Scale Retinex algorithm is used for carrying out illumination compensation on the rgb image on the basis of the illumination data, and the expression is as follows:
wherein I (x, y) is an original image, R (x, y) is a reflection component, L (x, y) is an illumination component, I represents an I-th color channel, x represents convolution, and G (x, y) is a gaussian surround function;
the formula of G (x, y) is as follows:and lambda satisfies: c G (x, y) dxdy=1;
wherein σ is Gao SihuanThe scale parameter of the windings, which is dynamically adjusted based on the illumination component Li,for the adjustment factor, lx is the illuminance unit:
r (x, y) is obtained by converting the logarithmic domain into the real domain after R (x, y) is obtained, and then linear stretching treatment is carried out to obtain an output image, wherein the final linear stretching formula is as follows:
wherein ssred_img is image data obtained by dynamically performing illumination compensation on the rgb image through Single Scale Retinex algorithm.
2. A method of multi-sensor fusion vehicle window identification as defined in claim 1, wherein,
the step of extracting the image features comprises the following steps:
inputting the compensated image data into a trained ResNeXt CNN network to obtain feature map;
the feature map is set to obtain candidate ROIs, the candidate ROIs are sent to an RPN network to carry out classification filtering part ROIs, then ROI alignment operation is carried out, and the rgb image data and the pixels of the feature map are correspondingly arranged to obtain a feature map of the candidate region ROIs.
3. A method of multi-sensor fusion vehicle window identification as defined in claim 1, wherein,
the step of compensating the laser radar point cloud array according to the temperature sensor data and extracting the point cloud characteristics point by point comprises the following steps:
collecting distance data measured by a laser radar at different temperatures;
constructing a distance temperature change relation by using the distance data;
generating a relation between compensation time and temperature according to the relation between the light speed and the distance;
finally, searching a corresponding compensation distance in a spline interpolation table according to the current temperature of the laser radar, and calculating the compensation distance of each depth data in the point cloud array, thereby completing the compensation of the point cloud array;
and inputting the compensated point cloud array into a point cloud feature extraction network to extract features, and obtaining point cloud features.
4. A method of multi-sensor fusion vehicle window identification as defined in claim 1, wherein,
the step of identifying the vehicle window according to the fused features comprises the following steps:
sending the pixel-level fusion features into the FC layer to obtain feature vectors feature pre For feature vector feature pre Performing Sigmoid operation to perform confidence judgment;
frame regression algorithm using RCNN, input feature pre Processing to obtain an offset function, and further performing offset adjustment on the original frame;
performing bilinear interpolation scaling on the pixel-level fusion features meeting the threshold requirement to the size of an ROI alignment feature map, and then sending the feature map into an FCN (fuzzy c-means) to generate window masks, wherein the size of each mask is as follows: m is m;
and performing size operation and background filling on the window mask by using the original Propos al and the image size to obtain the mask corresponding to the original image size, performing mask and frame regression on the original image, and simultaneously displaying the confidence judgment result of the Sigmoid branch on the corresponding region to finish the window recognition.
5. A method of multi-sensor fusion window identification as defined in claim 4, wherein,
the offset function is:
where P represents the original Proposal, phi (P) represents the input eigenvector, d * (P) represents a predictive offset value, W represents a learned parameter;
offset adjustment of the original frame includes translation and size scaling:
translation (Δx, Δy) is expressed as:
Δx＝P w d x (P)，Δy＝P h d y (P)；
size scaling(s) w ，s h ) Expressed as:
s w ＝P w d w (P)，s h ＝P h d h (P)。
6. a method of multi-sensor fusion vehicle window identification as defined in claim 1, wherein,
the step of performing pixel-level fusion on the image features and the point cloud features comprises the following steps:
the color characteristics of each pixel point and the depth characteristics of the corresponding point cloud point are spliced on a channel to obtain a group of fused features, and the process is as follows:
fator 1 ＝f cat (pcloud，pcolor)
fator 2 ＝f cat (pcloud 1 ，pcolor 1 )
fused_feature＝f link (fator 1 ，fator 2 )
wherein pc feature 、rgb feature Respectively extracting point cloud characteristics and image characteristics, f cat The operation of the splice is indicated and,is the relu activation function, (-) represents the 1*1 convolution operation, f link Representing a splice operation on a channel;
sending the filled feature into mlp, and obtaining global feature by using average feature
feature global ＝fav g_pooling (σ(fused_feature))
f avg_pooling For global average pooling operations, σ is mlp processing operations;
splicing the global feature and the fused feature on a channel to obtain a pixel-level fusion feature, namely dense fused feature:
feature dense ＝f link (fused feature ，feature global )
feature dense i.e. dense fused feature.
7. A multiple sensor fusion vehicle window identification system, comprising:
a first acquisition module adapted to acquire illumination data and rgb image data
The first processing module is suitable for carrying out illumination compensation on the rgb image data according to the illumination data and extracting image characteristics; the method for carrying out illumination compensation on the rgb image data according to the illumination data is characterized in that the Single Scale Retinex algorithm is used for carrying out illumination compensation on the rgb image on the basis of the illumination data, and the expression is as follows:
wherein I (x, y) is an original image, R (x, y) is a reflection component, L (x, y) is an illumination component, I represents an I-th color channel, x represents convolution, and G (x, y) is a gaussian surround function;
the formula of G (x, y) is as follows:and lambda satisfies: c G (x, y) dxdy=1;
where σ is the scale parameter of the gaussian surround, which is dynamically adjusted based on the illumination component Li,for the adjustment factor, lx is the illuminance unit:
r (x, y) is obtained by converting the logarithmic domain into the real domain after R (x, y) is obtained, and then linear stretching treatment is carried out to obtain an output image, wherein the final linear stretching formula is as follows:
wherein ssred_img is image data obtained by dynamically carrying out illumination compensation on the rgb image through Single Scale Retinex algorithm;
the second acquisition module is suitable for acquiring temperature sensor data and a laser radar point cloud array;
the second processing module is suitable for compensating the laser radar point cloud array according to the temperature sensor data and extracting point cloud characteristics point by point;
the fusion module is suitable for carrying out pixel-level fusion on the image characteristics and the point cloud characteristics;
and the identification module is suitable for carrying out vehicle window identification according to the fused features.
8. A computer readable storage medium having stored therein at least one instruction, wherein the instructions when executed by a processor implement the multi-sensor fusion vehicle window identification method of any one of claims 1 to 6.
9. An electronic device comprising a memory and a processor; at least one instruction is stored in the memory; the processor, by loading and executing the at least one instruction, implements the multi-sensor fusion vehicle window identification method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310066606.4A CN115984658B (en) | 2023-02-06 | 2023-02-06 | Multi-sensor fusion vehicle window recognition method, system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310066606.4A CN115984658B (en) | 2023-02-06 | 2023-02-06 | Multi-sensor fusion vehicle window recognition method, system and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115984658A CN115984658A (en) | 2023-04-18 |
| CN115984658B true CN115984658B (en) | 2023-10-20 |
Family
ID=85959698
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310066606.4A Active CN115984658B (en) | 2023-02-06 | 2023-02-06 | Multi-sensor fusion vehicle window recognition method, system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115984658B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118224993B (en) * | 2024-05-24 | 2024-07-30 | 杭州鲁尔物联科技有限公司 | Building structure deformation monitoring method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111860695A (en) * | 2020-08-03 | 2020-10-30 | 上海高德威智能交通系统有限公司 | Data fusion and target detection method, device and equipment |
| CN112363145A (en) * | 2020-11-09 | 2021-02-12 | 浙江光珀智能科技有限公司 | Vehicle-mounted laser radar temperature compensation system and method |
| CN112836734A (en) * | 2021-01-27 | 2021-05-25 | 深圳市华汉伟业科技有限公司 | Heterogeneous data fusion method and device and storage medium |
| CN113538569A (en) * | 2021-08-11 | 2021-10-22 | 广东工业大学 | Weak texture object pose estimation method and system |
| CN113850744A (en) * | 2021-08-26 | 2021-12-28 | 辽宁工程技术大学 | Image enhancement algorithm based on self-adaptive Retinex and wavelet fusion |
| CN115063329A (en) * | 2022-06-10 | 2022-09-16 | 中国人民解放军国防科技大学 | Visible light and infrared image fusion enhancement method and system under low-illumination environment |
| CN115656984A (en) * | 2022-09-19 | 2023-01-31 | 深圳市欢创科技有限公司 | TOF point cloud processing method, point cloud optimization method, laser radar and robot |
-
2023
- 2023-02-06 CN CN202310066606.4A patent/CN115984658B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111860695A (en) * | 2020-08-03 | 2020-10-30 | 上海高德威智能交通系统有限公司 | Data fusion and target detection method, device and equipment |
| CN112363145A (en) * | 2020-11-09 | 2021-02-12 | 浙江光珀智能科技有限公司 | Vehicle-mounted laser radar temperature compensation system and method |
| CN112836734A (en) * | 2021-01-27 | 2021-05-25 | 深圳市华汉伟业科技有限公司 | Heterogeneous data fusion method and device and storage medium |
| CN113538569A (en) * | 2021-08-11 | 2021-10-22 | 广东工业大学 | Weak texture object pose estimation method and system |
| CN113850744A (en) * | 2021-08-26 | 2021-12-28 | 辽宁工程技术大学 | Image enhancement algorithm based on self-adaptive Retinex and wavelet fusion |
| CN115063329A (en) * | 2022-06-10 | 2022-09-16 | 中国人民解放军国防科技大学 | Visible light and infrared image fusion enhancement method and system under low-illumination environment |
| CN115656984A (en) * | 2022-09-19 | 2023-01-31 | 深圳市欢创科技有限公司 | TOF point cloud processing method, point cloud optimization method, laser radar and robot |
Non-Patent Citations (3)
| Title |
|---|
| Multi-modal Data Analysis and Fusion for Robust Object Detection in 2D/3D Sensing;Jonathan Schierl et.al;《2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR)》;第1-7页 * |
| 基于Retinex理论的图像与视频增强算法研究;张诗;《中国优秀硕士学位论文全文数据库 信息科技辑》(第02期);论文第13-14页 * |
| 面向无序抓取的工件识别与位姿估计;蔡恒铨;《中国优秀硕士学位论文全文数据库 信息科技辑》(第01期);论文第38-42页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115984658A (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109683699B (en) | Method and device for realizing augmented reality based on deep learning and mobile terminal | |
| US11176694B2 (en) | Method and apparatus for active depth sensing and calibration method thereof | |
| CN114022830A (en) | Target determination method and target determination device | |
| CN115984658B (en) | Multi-sensor fusion vehicle window recognition method, system and readable storage medium | |
| CN112562093B (en) | Object detection method, electronic medium, and computer storage medium | |
| CN115578709B (en) | Feature level cooperative perception fusion method and system for vehicle-road cooperation | |
| CN112614164A (en) | Image fusion method and device, image processing equipment and binocular system | |
| CN112991218B (en) | Image processing method, device, equipment and storage medium | |
| CN113743163A (en) | Traffic target recognition model training method, traffic target positioning method and device | |
| CN113095358A (en) | Image fusion method and system | |
| CN112907557A (en) | Road detection method, road detection device, computing equipment and storage medium | |
| CN113538538B (en) | Binocular image alignment method, electronic device, and computer-readable storage medium | |
| CN118038229A (en) | Target detection method, device, equipment and storage medium | |
| CN112013820B (en) | Real-time target detection method and device for deployment of airborne platform of unmanned aerial vehicle | |
| CN117423077A (en) | BEV perception model, construction method, device, equipment, vehicle and storage medium | |
| WO2024045942A1 (en) | Ambient information sensing method, apparatus, and system, computer device, and storage medium | |
| CN116704125B (en) | Mapping method, device, chip and module equipment based on three-dimensional point cloud | |
| CN108229271B (en) | Method and device for interpreting remote sensing image and electronic equipment | |
| CN117078767A (en) | Laser radar and camera calibration method and device, electronic equipment and storage medium | |
| CN115601275A (en) | Point cloud augmentation method and device, computer readable storage medium and terminal equipment | |
| CN116310326A (en) | Multi-mode point cloud segmentation method, system, equipment and storage medium | |
| KR20210001555A (en) | Method of processing infrared image | |
| CN115294002A (en) | Image fusion method, electronic device, unmanned aerial vehicle and storage medium | |
| CN113920455B (en) | Night video coloring method based on deep neural network | |
| CN112348823A (en) | Object-oriented high-resolution remote sensing image segmentation algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |