CN115412844A - Real-time alignment method for vehicle networking beams based on multi-mode information synaesthesia - Google Patents
Real-time alignment method for vehicle networking beams based on multi-mode information synaesthesia Download PDFInfo
- Publication number
- CN115412844A CN115412844A CN202211024842.1A CN202211024842A CN115412844A CN 115412844 A CN115412844 A CN 115412844A CN 202211024842 A CN202211024842 A CN 202211024842A CN 115412844 A CN115412844 A CN 115412844A
- Authority
- CN
- China
- Prior art keywords
- data
- vehicle
- time
- matrix
- prediction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 230000006855 networking Effects 0.000 title claims abstract description 25
- 239000011159 matrix material Substances 0.000 claims abstract description 73
- 238000004891 communication Methods 0.000 claims abstract description 58
- 238000013528 artificial neural network Methods 0.000 claims abstract description 28
- 238000000605 extraction Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 14
- 230000004927 fusion Effects 0.000 claims abstract description 6
- 238000007781 pre-processing Methods 0.000 claims description 25
- 238000003062 neural network model Methods 0.000 claims description 15
- 230000000007 visual effect Effects 0.000 claims description 12
- 230000008447 perception Effects 0.000 claims description 11
- 238000010606 normalization Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 5
- 230000009467 reduction Effects 0.000 claims description 4
- 238000005549 size reduction Methods 0.000 claims description 3
- 238000003860 storage Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 2
- 238000005520 cutting process Methods 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 125000004122 cyclic group Chemical group 0.000 claims 1
- 239000010410 layer Substances 0.000 description 37
- 210000002569 neuron Anatomy 0.000 description 18
- 238000005516 engineering process Methods 0.000 description 9
- 230000006399 behavior Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000009977 dual effect Effects 0.000 description 3
- 239000000969 carrier Substances 0.000 description 2
- 210000004205 output neuron Anatomy 0.000 description 2
- 238000002203 pretreatment Methods 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/023—Services making use of location information using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B7/00—Radio transmission systems, i.e. using radiation field
- H04B7/02—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas
- H04B7/04—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas
- H04B7/0413—MIMO systems
- H04B7/0426—Power distribution
- H04B7/043—Power distribution using best eigenmode, e.g. beam forming or beam steering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B7/00—Radio transmission systems, i.e. using radiation field
- H04B7/02—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas
- H04B7/04—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas
- H04B7/06—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas at the transmitting station
- H04B7/0613—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas at the transmitting station using simultaneous transmission
- H04B7/0615—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas at the transmitting station using simultaneous transmission of weighted versions of same signal
- H04B7/0619—Diversity systems; Multi-antenna system, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas at the transmitting station using simultaneous transmission of weighted versions of same signal using feedback from receiving side
- H04B7/0621—Feedback content
- H04B7/0626—Channel coefficients, e.g. channel state information [CSI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/029—Location-based management or tracking services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/30—Services specially adapted for particular environments, situations or purposes
- H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Power Engineering (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a vehicle networking wave beam real-time alignment method based on multi-mode information synaesthesia, which comprises the steps of designing a track prediction network model, inputting distance data obtained by processing RGB (red, green and blue) images and radar signals captured by road side units RSU (road side unit) in the vehicle networking and a Channel State Information (CSI) matrix obtained from a Sub-6GHz frequency band of a control channel as multi-mode information into the track prediction network model, and performing feature extraction and early fusion on the multi-mode information to predict a wave beam forming angle, improve the accuracy of vehicle future position prediction and realize the real-time alignment of vehicle networking wave beams. The method learns implicit characteristics about the future position of the vehicle through the neural network, and predicts the beam forming angle of the vehicle relative to the RSU at the future moment. The invention can better deal with the transverse random micro-movement behavior of the vehicle, realize the establishment of more stable communication link and improve the achievable millimeter wave communication rate.
Description
Technical Field
The invention belongs to the technical field of wireless communication, and relates to a millimeter wave (mmWave) beam real-time alignment technology in wireless communication of an internet of vehicles, in particular to an internet of vehicles millimeter wave beam real-time alignment method based on multi-mode perception and Channel State Information (CSI) synaesthesia (namely, multi-mode perception and Channel State Information are fused), wherein multi-mode Information features are extracted and fused by applying a deep learning method, a nonlinear relation between a vehicle future position and the multi-mode Information is learned, and the internet of vehicles millimeter wave beam real-time alignment of stable and reliable wireless communication between a Road Side Unit (RSU) and a high-speed vehicle is realized.
Background
With the rapid development of the automobile industry, the internet of vehicles is one of the most important technologies for realizing intelligent travel and intelligent traffic as the most important component in the future intelligent traffic system. Meanwhile, with the large-scale commercial use of 5G, millimeter waves are regarded as a key technology for meeting various high-performance communication requirements of the Internet of vehicles due to the advantages of large bandwidth and low time delay. In order to guarantee the communication service requirements of various applications of the internet of vehicles and improve the overall safety and the user experience quality of wireless communication of the internet of vehicles, the vehicles need to guarantee high-quality wireless connection with a wireless communication network at any time. Among the technologies related to high-speed mobility management for vehicles in the millimeter wave internet of vehicles, real-time alignment of millimeter wave beams is a precondition for ensuring stable connection of vehicles to a communication network.
How to achieve beam alignment between a high-speed vehicle and an RSU is one of the core technologies for vehicle mobility management in the internet of vehicles, and is a key guarantee to ensure that the vehicle can be stably connected to a communication network with high quality. The modes for realizing the millimeter wave beam real-time alignment between the transmitting side and the receiving side mainly comprise beam training, beam tracking and beam forming prediction. Traditionally, alignment of narrow beams between millimeter wave transceivers is accomplished by beam training. In the beam training, the transmitting end needs to transmit a pilot signal to the global angle so as to find the beam forming direction with the strongest signal-to-noise ratio and determine the beam forming angle. However, this can cause significant communication overhead and latency, making it difficult to apply to internet of vehicles communications. In order to improve the defect, the millimeter wave beam tracking technology greatly reduces the space range to be searched for beam training by utilizing the time correlation of the beam angle change between the transmitting side and the receiving side at the adjacent moment, for example, the invention patent with the publication number of CN 112738764B proposes a broadband millimeter wave beam tracking method based on vehicle track cognition, and utilizes the motion characteristic of the vehicle to assist the millimeter wave beam tracking. Beam tracking still requires the pilot signal to be transmitted before each communication link is established, resulting in a large communication overhead. The beam forming prediction technology directly carries out beam forming at a prediction angle by predicting the future position of the vehicle in advance, and has low communication overhead and time delay. However, the stability of the communication link established in this manner and the achievable communication rate are greatly affected by the accuracy of the prediction algorithm.
Currently, the beamforming prediction of the internet of vehicles is mostly based on an extended kalman filter algorithm, and is given by a simple vehicle motion state evolution model and a measurement value obtained by radar equipment, so that the accuracy is low and the application scene is limited. With the increasing variety and the gradually enhanced performance of sensing devices equipped on intelligent vehicles and RSUs, the auxiliary role of multi-mode sensing information on the internet-of-vehicles communication system is gradually emphasized and researched. Different from the electromagnetic environment characteristics reflected by CSI, the multi-modal perception information contains visual space characteristics with finer granularity and wider visual field, and has the capability of better predicting the future position of the vehicle. How to select a proper mode to extract and fuse vehicle position features of multi-modal perception information and CSI in the Internet of vehicles so as to assist in predicting a future beamforming angle of a vehicle is a key direction of current research.
Disclosure of Invention
The invention provides a real-time alignment technology of a vehicle networking wave beam based on multi-mode information synaesthesia, which can better deal with the transverse random micro-movement behavior of a vehicle on the basis of ensuring the high precision of a vehicle networking wave beam forming angle predicted value, realize more stable communication link establishment and improve the achievable millimeter wave communication rate of the vehicle networking.
In the invention, multi-mode information joint perception Road end multi-mode perception information is fused with CSI, RGB images captured by Road Side Units (RSUs) in the internet of vehicles, distance data obtained after radar signal processing and CSI matrixes obtained on a control channel Sub-6GHz frequency band are used as multi-mode information input, and a track prediction network model containing different types of neural network components is designed to extract and early fuse the characteristics of the multi-mode information, so that the accuracy of vehicle future position prediction is improved, and the accuracy of predicted beam forming angles at the next moment is further ensured. In addition, multi-mode information on the passing track of the vehicle is constructed into a time sequence form at each prediction moment, and then the track prediction network model is used for extracting and learning time sequence features, so that the robustness of coping with the transverse random micro-movement behavior of the vehicle is further improved.
The technical scheme of the invention is as follows:
a real-time alignment method of a vehicle networking beam based on multi-mode information synaesthesia comprises the steps of designing a track prediction network model, inputting multi-mode information by utilizing distance data obtained after RGB images and radar signals captured by road side units RSUs in the vehicle networking are processed and a CSI matrix obtained from a control channel Sub-6GHz frequency band, extracting characteristics of the multi-mode information and performing early fusion on the multi-mode information, predicting a beam forming angle, and improving the accuracy of real-time alignment of the vehicle networking beam and prediction of the future position of a vehicle; the method comprises the following steps:
1) Acquiring original multi-modal data: before each data block between the RSU and the vehicle starts to be transmitted, namely at the moment of predicting each beam forming angle (in the invention, the current moment of predicting the beam forming angle is defined as the nth time block or nth time slot in periodic prediction by taking the time block as a unit), the RGB image and the vehicle distance data of a traffic system are obtained through sensing equipment, and a CSI matrix of a control channel frequency band is obtained through communication equipment; the CSI matrix is calculated by the RSU through channel estimation. The raw multimodal data includes: the channel state information comprises RGB images, vehicle distance data and a CSI matrix of a control channel frequency band.
During specific implementation, a Road Side Unit (RSU) is provided with various sensing devices (an RGB camera and a radar device) and communication devices operating at two frequency bands of Sub-6GHz and mmWave, a vehicle is provided with communication devices operating at two frequency bands of Sub-6GHz and mmWave, an RGB image of a traffic system is obtained by shooting through the RGB camera, distance data of a target vehicle is obtained through the radar device, and a CSI matrix of the frequency band of Sub-6GHz is obtained through a signal processing device of the RSU on a control channel of the communication devices;
2) Preprocessing the original multi-modal data obtained in the step 1) to obtain a preprocessed RGB image, a vehicle distance matrix and a CSI angular domain characteristic matrix;
on the basis of the original multi-modal data acquired in the previous step, the data preprocessing module preprocesses the original multi-modal data: performing size reduction and data standardization on the RGB image, constructing distance data into a matrix form, performing data normalization, and performing angular domain feature extraction on a CSI matrix;
3) Constructing to obtain time series multi-modal data; the time series multi-modal data comprises RGB images, a vehicle distance matrix and a CSI angular domain characteristic matrix;
after an antenna of the RSU obtains a CSI matrix by receiving signals sent by vehicles and estimating, a storage unit of the RSU at each beamforming angle prediction moment stores and stacks multi-modal sensing data and CSI at the current and previous prediction moments to construct time series multi-modal data;
4) Constructing a track prediction neural network model, inputting time series multi-modal data into the network model, extracting visual space characteristics, electromagnetic space characteristics and time sequence characteristics, carrying out early fusion, and predicting to obtain the vehicle position coordinates and the motion angle (namely the beam forming angle) of the (n + 2) th time slot;
inputting the time series multi-modal data into a track prediction neural network to extract visual space characteristics, electromagnetic space characteristics, time sequence characteristics and early fusion, predicting to obtain a vehicle position coordinate after a time slot, and further obtaining an angle of a vehicle relative to an RSU (remote terminal unit) in the (n + 2) th time slot, namely a beam forming angle of the (n + 2) th time slot;
5) In the next data block transmission, the RSU transmits the beamforming angle of the (n + 2) th slot obtained in step 4) to the vehicle;
the RSU transmits the beam forming angle of the (n + 2) th time slot predicted in the previous step to the vehicle through the data block of the (n + 1) th time slot, so that the vehicle knows the beam forming angle of the (n + 2) th time slot in advance, and the vehicle performs beam forming of the (n + 2) th time slot according to the angle;
6) In the (n + 2) th time slot, the RSU and the vehicle respectively carry out beam forming and alignment through the angle values obtained through prediction in advance, and a millimeter wave communication link is established for communication;
the RSU and the vehicle respectively carry out beam forming and alignment through angle values obtained through prediction in advance on the (n + 2) th time slot, and a millimeter wave communication link is established for communication;
7) At each prediction moment, the RSU executes the steps 1) to 6), thereby finishing the real-time alignment of the millimeter wave beam in the driving process of the vehicle and ensuring the stable connection of the vehicle to the wireless communication network;
through the steps, the beam forming prediction based on the multi-mode information synaesthesia is achieved, and real-time alignment of millimeter wave beams in the vehicle running process is completed.
When the device is specifically implemented, the invention also provides a vehicle networking beam real-time alignment device based on multi-mode information synaesthesia, which comprises an RSU and a vehicle; a perception module (comprising an RGB camera and radar equipment), a communication module (comprising Sub-6GHz and mmWave dual-frequency bands), an image preprocessing module, a distance data preprocessing module, an angular domain information extraction module and a track prediction neural network module are assembled on the RSU; a communication module (containing Sub-6GHz and mmWave dual bands) is assembled on a vehicle. Firstly, an RGB camera and radar equipment in a sensing module of a vehicle respectively acquire an RGB image and distance data in each time slot, and CSI data are acquired by a Sub-6GHz frequency channel in a communication module. And then, carrying out data preprocessing on the multi-modal original data by utilizing an image preprocessing module, a distance data preprocessing module and an angular domain information extraction module. And then, learning vehicle position characteristics and motion state evolution characteristics in the multi-modal data by using a trajectory prediction neural network module, and further completing prediction of the future position of the vehicle.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a vehicle networking wave beam real-time alignment technical scheme based on multi-modal information synaesthesia, which utilizes multi-modal perception information acquired by an RSU and CSI on a Sub-6GHz frequency band to learn implicit characteristics about the future position of a vehicle through a neural network, and predicts that the wave beam forming angle of the vehicle relative to the RSU at the future moment is closer to the reality. By adopting the technical scheme provided by the invention, the transverse random micro-movement behavior of the vehicle can be better responded on the basis of ensuring the high precision of the predicted value of the beam forming angle, the establishment of a more stable communication link is realized, and the achievable millimeter wave communication rate is improved.
The beam real-time alignment method based on multi-mode information synaesthesia provided by the invention has the following technical advantages:
establishing a track prediction neural network model, and learning the relation between the image and distance data and the future position of the vehicle from the image captured by the RSU and the distance data obtained after radar signal processing through the network model to ensure the reliability of the prediction of the future position of the moving vehicle;
secondly, angular domain characteristic information extraction is carried out on CSI data acquired by a Sub-6GHz frequency band on a control channel of the communication equipment, and further electromagnetic space characteristics of the communication system are extracted through a track prediction neural network model, so that prediction accuracy is improved;
thirdly, multi-mode information of the historical track of the passing vehicle is superposed at each prediction moment, and the motion trend characteristics of the vehicle are obtained by learning of a track prediction neural network model, so that the track prediction neural network model has good robustness on the transverse random micro-movement behavior of the vehicle;
and (IV) aiming at the multi-mode perception information and the data characteristics of CSI, extracting electromagnetic space characteristics, visual space characteristics and time sequence characteristics by adopting three networks with different structures, namely a full connection network, a residual neural network (ResNet-18) and a Gated Recurrent Unit (GRU), and early fusing different characteristics to further improve the vehicle position prediction precision.
Drawings
Fig. 1 is a block diagram of a beam real-time alignment apparatus provided in the embodiment of the present invention.
Fig. 2 is a flow chart of the beam real-time alignment algorithm provided by the present invention.
FIG. 3 is a block diagram of an RGB image pre-processing module according to an embodiment of the present invention.
Fig. 4 is a block diagram of an angular domain information extraction module of a CSI matrix according to an embodiment of the present invention.
FIG. 5 is a block flow diagram of a trajectory prediction neural network module, which is designed in accordance with an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Referring to fig. 1, the present invention utilizes a vehicle equipped with a communication module (including Sub-6GHz and mmWave dual bands), an RSU equipped with a sensing module (including RGB camera and radar device) and a communication module (including Sub-6GHz and mmWave dual bands), wherein electromagnetic spatial information of the Sub-6GHz band is acquired through the RSU and the communication device module on the vehicle. Visual space information of the vehicle position is obtained by a perception module on the RSU, and then data preprocessing is carried out on the multi-mode original data by an image preprocessing module, a distance data preprocessing module and an angular domain information extraction module. And then, learning vehicle position characteristics and motion state evolution characteristics in the multi-modal data by using a track prediction neural network module, and further completing the prediction of the future position of the vehicle.
The invention is provided with a beam real-time alignment device, which comprises a sensing module, a communication module, an image preprocessing module, an angular domain information extraction module and a track prediction neural network module on an RSU (remote subscriber Unit); the vehicle comprises a communication module, and the beam real-time alignment method provided by the invention is shown in fig. 2, and comprises the following specific steps:
s10: referring to fig. 1, the RSU is equipped with a sensing module, a communication module, an image preprocessing module, an angular domain information extraction module, and a trajectory prediction neural network module, and first, an RGB camera and a radar device in the sensing module respectively perform a process of predicting a trajectory at each time slotRGB image X and vehicle distance data D are acquired, CSI data H are acquired by a Sub-6GHz frequency channel in a communication module,
N t indicating the number of mmWave band transmitting antennas, N, of the RSU communication device s The number of OFDM sub-carriers adopted by the RSU communication equipment is represented and is stored in a storage unit of the RSU;
s20: referring to fig. 3, based on the RGB image data X acquired in the previous step, the image preprocessing module performs data preprocessing on the RGB image data X to obtain an image form X easy to extract features P ；
S30: based on the vehicle distance data D acquired by the radar in the step S10, the distance data is constructed into a matrix D' according to the position coordinates of the detection target vehicle, and then data normalization is carried out to obtain a form D which is easy for neural network processing and characteristic extraction P ；
S40: referring to fig. 4, based on the CSI obtained by the RSU in step S10 at Sub-6GHz band, the angular domain information extraction module performs data preprocessing on the CSI to obtain a form H that facilitates the neural network to extract angular domain features P ，
Where C represents the matrix complex field dimension.
S50: referring to fig. 5, inputting the preprocessed multi-modal data obtained in steps S20, S30, and S40 into a trajectory prediction neural network module, constructing time series multi-modal data, extracting and fusing a visual spatial feature, an electromagnetic spatial feature, and a time series feature by the trajectory prediction neural network module, learning a relationship between the features and a future position of the vehicle, predicting a beam forming angle of the RSU and the vehicle after a next time slot, and completing millimeter wave beam alignment;
in step S10: a sensing module of the RSU acquires RGB images and distance data of the traffic environment on each time slot, and a communication module acquires a CSI matrix (channel state information) of a Sub-6GHz frequency band, namely H, on a control channel;
in step S20: by usingThe image preprocessing module preprocesses the original RGB image data X, so as to obtain an image X with characteristics easy to extract P The pretreatment method includes the following steps S21 to S22:
s21: cutting the original RGB image with higher resolution to obtain an image X' with reduced size;
in specific implementation, an image X' with the resolution of 224X 224 is obtained according to a Z-shaped filling sequence of pixel values of an original RGB image from left to right and from top to bottom;
S22：X′ R ，X′ G ，X′ B respectively representing the pixel values of the three channels of the obtained images X' R, G and B. Three channels, X 'of the resulting image X' G ，X′ B Are respectively recorded as μ R 、μ G 、μ B And the standard deviation is respectively expressed as sigma R 、σ G 、σ B . Then to X' R ，X′ G ，X′ B The following data normalization operation was performed:
obtaining an image X with standardized data P 。
Representing the normalized R-channel pixel values of the data;
representing the normalized G-channel pixel values of the data;
representing the B channel pixel values after data normalization. In specific practice,. Mu. R 、μ G 、μ B Values of 0.306, 0.281, 0.251 can be selected. Sigma R 、σ G 、σ B The values are 0.016,0.0102 and 0.013. The mean value and the standard deviation adopted by the standardization of the three channels can be properly adjusted according to the track prediction network structure;
in step S30: distance data obtained after radar signal processingD, preprocessing, constructing the obtained target vehicle distance data into a matrix D 'according to the position coordinates of the target vehicle, and then carrying out data normalization on D', thereby obtaining a matrix form D of which the characteristics are easy to extract by the neural network P ；
Performing data normalization on the matrix D ', namely dividing all elements in the matrix D ' by the maximum value of the elements of the matrix D ', and further obtaining the distance matrix D after data normalization P ，
In step S40: preprocessing the CSI matrix H by utilizing an angular domain information extraction module so as to obtain a representation form H with stronger angular domain characteristics P The pretreatment method includes the following steps S41 to S43:
s4l: a codebook F with angular resolution B is predefined,
calculating to obtain an energy value matrix E, E = | F of each subcarrier at each angle T H P |，
FT denotes the transpose of F.
S42: based on the energy value matrix E obtained in the previous step,
e ij is the energy value of j sub-carriers at the angle i; accumulating the energy values of all the subcarriers in each angle to obtain an energy value dimension reduction matrix E ', E ' = (E ' ij ) B×1 ，
S43: based on the energy value dimension reduction matrix E' obtained in the previous step,
keeping the first five maximum energy values, setting the rest positions to zero, finishing the angular domain information extraction, obtaining a representation form with stronger angular domain characteristics and lower dimensionality, and recording the representation form as an angular domain characteristic matrix H P ；
In step S50: inputting the preprocessed multi-modal data into a trajectory prediction neural network model.
The track prediction neural network model constructed by the invention is used for extracting electromagnetic space characteristics, visual space characteristics and time sequence characteristics by adopting three networks with different structures, namely a full connection network, a residual neural network (ResNet-18) and a Gated Recurrent Unit (GRU), and early fusing the different characteristics.
In specific implementation, the track prediction neural network takes an RGB image, a distance matrix and an angular domain characteristic matrix which are preprocessed at the current prediction time and the last prediction time as input, time sequence data are constructed by stacking multi-mode data of the current prediction time and the last prediction time, visual space characteristics and electromagnetic space characteristics in the time sequence data and the RGB image at the current prediction time are learned by processing an image time sequence in the time sequence data and the RGB image at the current prediction time through a residual neural network ResNet-18, processing a distance matrix time sequence and a distance matrix at the current prediction time through ResNet-18 and processing an angular domain characteristic matrix time sequence and an angular domain characteristic matrix at the current prediction time through a full-connection network, all the characteristics at the current time are spliced and input into the full-connection network, and a longitudinal coordinate prediction value is output; and splicing the processed characteristics of the time sequence multi-modal data, inputting the characteristics into a GRU (generalized regression Unit) for processing to obtain time sequence characteristics, inputting the time sequence characteristics into a fully-connected network, and outputting to obtain an abscissa predicted value. And (3) the position of the vehicle after the next time slot is obtained through the track prediction neural network model, and then the beam forming angle is calculated by using an inverse trigonometric function, so that millimeter wave beam alignment is completed. This step includes the following steps S51 to S54:
s51: inputting the RGB image X preprocessed at the current moment P Distance matrix D P CSI matrix H after extraction of sum angular domain information P Stacking the constructed track prediction neural network model with the same type data of the previous time slot, adding time dimension, constructing time sequence multi-modal data, and recording the time sequence multi-modal data as T;
s52: time-series form RGB image in time-series multi-modal data T
And RGB image X at the current predicted time P Inputting into a ResNet-18 network, wherein the number of neurons in an output layer of the ResNet-18 network is 256; time-series distance matrix in multi-modal data T of time series
And a distance matrix D of the current predicted time P Inputting into a ResNet-18 network, wherein the number of neurons in an output layer of the ResNet-18 network is 256; mapping CSI angular domain feature matrices in time-series multi-modal data T
And CSI angular domain characteristic matrix H of current prediction time P Inputting into a fully-connected network, wherein the number of hidden layers of the fully-connected network is 2-4, the number of neurons of an output layer is 4-8, and in specific implementation, the number of hidden layers is 2, and the number of neurons of the output layer is 8. Then, the features obtained after the time sequence is processed in the above way are spliced to obtain the time sequence multi-modal features F 1 (ii) a All the features at the current moment are spliced to obtain the multi-modal feature F at the current moment 2 ；
S53: the multi-modal feature F of the time series obtained in the step S52 1 Inputting the time sequence characteristic F into a gate control circulation unit GRU, wherein the GRU is a single layer, the dimension value of a hidden layer is 16-32, and the value is 16 in specific implementation so as to obtain the time sequence characteristic F T . F is to be T Inputting into a fully-connected network for prediction, wherein the number of hidden layers is 2, the number of output neurons is 1, and outputting to obtain predicted value of abscissa of vehicle after next time slot
S54: the multi-modal characteristics F at the current moment obtained in the step S52 2 Inputting into a full-connection network for prediction, wherein the number of hidden layers is 3-4, the number of output neurons is 1, the number of hidden layers is 3 in specific implementation, and a predicted value of a vertical coordinate of a vehicle after the next time slot is obtained
Combined with that obtained in step S53
Calculating the beam forming angle of the vehicle relative to the RSU through an inverse trigonometric function
And finishing the real-time beam alignment.
In specific implementation, the number of neurons in each layer of the fully-connected network in steps S52 to S54 is different, where 64 neurons in the first layer of the fully-connected network in S52 are provided with a 0.2 dropout probability between the first layer and the second layer, 32 neurons in the second layer are provided with a 0.2 dropout probability between the second layer and the third layer, 16 neurons in the third layer, and 8 neurons in the fourth layer (i.e., output layer). In S53, a fully-connected network comprises 16 neurons in a first layer, a 0.1 dropout probability is set between the first layer and a second layer, 32 neurons in the second layer, a 0.1 dropout probability is set between the second layer and a third layer, 16 neurons in the third layer and 1 neuron in a fourth layer (namely an output layer). In S54, a fully-connected network comprises 520 neurons in the first layer, a 0.1 dropout probability is set between the first layer and the second layer, 256 neurons in the second layer, 128 neurons in the third layer, 64 neurons in the fourth layer and 1 neuron in the fifth layer (namely an output layer).
The invention provides a vehicle networking beam real-time alignment technology based on multi-mode information synaesthesia, which utilizes images acquired by an RSU (remote terminal Unit) and target distance information acquired after radar signal processing, and utilizes a neural network to extract visual space characteristics from the images, thereby ensuring the reliability of future position prediction; the angular domain information of the CSI data acquired on the Sub-6GHz frequency band is used for extraction optimization, so that the neural network is easy to extract the angular domain features rich in electromagnetic space, and the prediction precision is improved; and multi-mode information of the vehicle historical track is superposed at each moment to construct time sequence data, so that the neural network learns the vehicle motion trend, and the robustness of the neural network on the prediction of the vehicle transverse random micromotion behavior is improved. The RGB camera, the radar sensing equipment, the communication equipment of Sub-6GHz frequency band and mmWave frequency band are all common equipment on RSU and intelligent vehicle in the car networking, and the practical requirements of convenience in installation, flexibility and reliability in operation and low cost are met.
It is noted that the disclosed embodiments are intended to aid in further understanding of the invention, but those skilled in the art will appreciate that: various alternatives and modifications are possible without departing from the invention and scope of the appended claims. Therefore, the invention should not be limited to the embodiments disclosed, but the scope of the invention is defined by the appended claims.
Claims (9)
1. A real-time alignment method of vehicle networking beams based on multi-modal information synaesthesia comprises the steps of designing a track prediction network model, inputting distance data obtained by processing RGB images and radar signals captured by road side units RSUs in the vehicle networking and a Channel State Information (CSI) matrix obtained from a Sub-6GHz frequency band of a control channel as multi-modal information into the track prediction network model, extracting features and fusing early-stage the multi-modal information, predicting a beam forming angle, improving the accuracy of vehicle future position prediction and achieving real-time alignment of the vehicle networking beams; the method comprises the following steps:
1) Acquiring raw multimodal data:
in the nth time slot, RGB images and vehicle distance data of a traffic system are obtained through sensing equipment, and a CSI matrix of a control channel frequency band is obtained through communication equipment; the nth time slot is used for predicting the moment of each beam forming angle before each data block between the RSU and the vehicle starts to be transmitted;
2) Preprocessing the original multi-modal data obtained in the step 1) to obtain a preprocessed RGB image, a vehicle distance matrix and a CSI angular domain characteristic matrix;
on the basis of the original multi-modal data acquired in the previous step, the data preprocessing module preprocesses the original multi-modal data: performing size reduction and data standardization on the RGB image, constructing distance data into a matrix form, performing data normalization, and performing angular domain feature extraction on a CSI matrix;
3) Constructing to obtain time series multi-modal data;
the time series multi-modal data comprises RGB images, a vehicle distance matrix and a CSI angular domain characteristic matrix;
4) Constructing a track prediction neural network model, inputting time series multi-modal data and current moment multi-modal data into the track prediction neural network model, extracting visual space characteristics, electromagnetic space characteristics and time sequence characteristics, and performing early fusion; predicting the position coordinates of the vehicle after obtaining a time slot, and further obtaining the angle of the vehicle relative to the RSU at the n +2 time slot, namely the beam forming angle of the n +2 time slot; the trajectory prediction neural network model comprises a full-connection network, a residual error neural network and a gate control cycle unit network structure;
5) In the next data block transmission, the RSU transmits the beamforming angle of the n +2 time slot predicted in the step 4) to the vehicle through the data block of the n +1 time slot, so that the vehicle knows the beamforming angle of the n +2 time slot in advance, and the vehicle performs beamforming of the n +2 time slot according to the angle;
6) In the n +2 time slot, the RSU and the vehicle respectively carry out beam forming and alignment through a beam forming angle obtained through prediction in advance, and a millimeter wave communication link is established for communication;
7) At each prediction moment, the RSU executes the steps 1) to 6), thereby finishing the real-time alignment of the millimeter wave beam in the running process of the vehicle and ensuring the stable connection of the vehicle to the wireless communication network;
through the steps, the beam forming prediction based on multi-mode information synaesthesia is realized, and the real-time alignment of the millimeter wave beam in the vehicle driving process is completed through the fusion of the multi-mode perception information and the CSI.
2. The method for real-time alignment of the vehicle networking beam based on the multi-mode information synaesthesia as claimed in claim 1, wherein step 1) obtains original multi-mode data, specifically, a plurality of sensing devices and communication devices are equipped on the RSU, and communication devices are equipped on the vehicle; various sensing devices include RGB cameras and radar devices; the method comprises the steps that RGB images of a traffic system are obtained through shooting by an RGB camera, and distance data of a target vehicle are obtained through radar equipment; the communication equipment operates in Sub-6GHz and mmWave frequency bands; and acquiring the CSI matrix of the Sub-6GHz frequency band on a control channel of the communication equipment through a signal processing device of the RSU.
3. The method for real-time alignment of car networking beams based on multi-modal information awareness as claimed in claim 1, wherein the step 2) of preprocessing the original multi-modal data comprises: carrying out size reduction and data standardization on the RGB image; constructing the distance data into a matrix form, and performing data normalization; and performing angular domain feature extraction on the CSI matrix.
4. The method for real-time alignment of the car networking beam based on the multi-modal information synaesthesia as claimed in claim 1, wherein the step 3) constructs and generates time series multi-modal data, specifically: after the RSU obtains the CSI matrix by receiving signal estimation sent by a vehicle, the storage unit of the RSU at each beamforming angle prediction moment stores and stacks the multi-modal sensing data and the CSI matrix at the current and last prediction moments, and therefore time-series multi-modal data are constructed.
5. The vehicle networking beam real-time alignment method based on multi-mode information synaesthesia of claim 1, which is realized by arranging a beam real-time alignment device; the device comprises a sensing module, a communication module, an image preprocessing module, an angular domain information extraction module and a track prediction neural network module on an RSU (remote subscriber Unit); a communication module is included on the vehicle.
6. The method for real-time alignment of car networking beams based on multi-modal information awareness as claimed in claim 5,the method is characterized in that an image preprocessing module is utilized to preprocess original RGB image data X to obtain an image X easy to extract features P (ii) a The method comprises the following steps S21-S22:
s21: cutting the original RGB image with higher resolution to obtain an image X' with reduced size;
s22: prepared from X' R ,X′ G ,X′ B Respectively representing the pixel values of R, G and B channels of the obtained image X'; the mean value and the standard deviation adopted by the standardization of the three channels are determined according to the track prediction network structure; pixel values X 'of three channels of image X' R ,X′ G ,X′ B Are respectively recorded as μ R 、μ G 、μ B The standard deviation is respectively denoted as σ R 、σ G 、σ B (ii) a To X' R ,X′ G ,X′ B And (3) carrying out data standardization operation:
obtaining an image X with standardized data P ；
Representing the normalized R-channel pixel values of the data;
representing the normalized G-channel pixel values of the data;
representing the normalized B-channel pixel values of the data;
preprocessing distance data D obtained after radar signal processing, constructing the obtained target vehicle distance data into a matrix D 'according to the position coordinates of a target vehicle, and performing data normalization on the matrix D', namely dividing all elements in the matrix D 'by the maximum value of the elements of the matrix D', so as to obtain a matrix form D of which the characteristics are easy to extract by a neural network P 。
7. The method as claimed in claim 5, wherein the angular domain information extraction module is used to preprocess the CSI matrix H to obtain the representation form H with stronger angular domain characteristics P (ii) a Includes the following steps S41 to S43:
s41, a codebook F with the angular resolution of B is predefined,
calculating to obtain an energy value matrix E, E = | F of each subcarrier at each angle T H P |，
S42: on the basis of the matrix of energy values E,
e ij is the energy value of the sub-carrier wave j at the angle i; accumulating the energy values of all the subcarriers in each angle to obtain an energy value dimension reduction matrix E ', E ' = (E ' ij ) B×1 ，
S43, based on the obtained energy value dimension reduction matrix E', reserving the maximum set number of energy values, setting the rest positions to be zero, finishing the angular domain information extraction, and marking the obtained angular domain characteristics as an angular domain characteristic matrix H P 。
8. The vehicle networking beam real-time alignment method based on multi-mode information synaesthesia of claim 7, wherein preprocessed multi-mode data is input into a constructed trajectory prediction neural network model; the track prediction neural network takes the RGB images, distance matrixes and angular domain feature matrixes which are preprocessed at the current prediction time and the last prediction time as input, time sequence data are constructed by stacking multi-mode data of the current prediction time and the last prediction time, the image time sequence in the time sequence data and the RGB images at the current prediction time are processed by adopting a residual neural network ResNet-18, the distance matrix time sequence and the distance matrix at the current prediction time are processed by ResNet-18, the angular domain feature matrix time sequence and the angular domain feature matrix at the current prediction time are processed by a full-connection network, visual space features and electromagnetic space features are learned, all the features at the current time are spliced and input into the full-connection network, and a longitudinal coordinate prediction value is obtained by outputting; splicing the processed characteristics of the time sequence multi-modal data, inputting the characteristics into a GRU (generalized regression Unit) for processing to obtain time sequence characteristics, inputting the time sequence characteristics into a fully-connected network, and outputting to obtain a predicted value of the abscissa; and (4) calculating a beam forming angle according to the position of the vehicle after the next time slot, which is obtained by the track prediction neural network model, and finishing millimeter wave beam alignment.
9. The method for aligning in real time the vehicle networking beams based on multi-modal information synaesthesia as claimed in claim 8, which comprises the following procedures S51-S54:
s51: inputting the RGB image X preprocessed at the current moment P Distance matrix D P And CSI matrix H after angular domain information extraction P Stacking the built trajectory prediction neural network model with the same type data of the previous time slot, adding a time dimension, and constructing time sequence multi-modal data which are recorded as T;
s52: time-series form RGB image in time-series multi-modal data T
And RGB image X at the current predicted time P Inputting into a ResNet-18 network; time-series distance matrix in multi-modal data T
And a distance matrix D for the current predicted time P Inputting into a ResNet-18 network; mapping CSI angular domain feature matrices in time-series multi-modal data T
And CSI angular domain characteristic matrix H of current prediction time P Inputting into a fully connected network; splicing the characteristics obtained after the time sequence is processed to obtain time sequence multi-modal characteristics F 1 (ii) a All the features at the current moment are spliced to obtain the multi-modal features F at the current moment 2 ；
S53: the multi-modal feature F of the time series obtained in the step S52 1 Input into a gated cyclic unit GRU to obtain a timing characteristic F T (ii) a F is to be T Inputting into a full-connection network for prediction, and outputting to obtain a predicted value of the abscissa of the vehicle after the next time slot
S54: the multi-modal characteristics E at the current moment obtained in the step S52 2 Inputting the data into a full-connection network for prediction to obtain a predicted value of the ordinate of the vehicle after the next time slot
Obtained in connection with step S53
Calculating the beam forming angle of the vehicle relative to the RSU through an inverse trigonometric function
And finishing the real-time beam alignment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211024842.1A CN115412844B (en) | 2022-08-25 | 2022-08-25 | Multi-mode information alliance-based real-time alignment method for vehicle networking wave beams |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211024842.1A CN115412844B (en) | 2022-08-25 | 2022-08-25 | Multi-mode information alliance-based real-time alignment method for vehicle networking wave beams |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115412844A true CN115412844A (en) | 2022-11-29 |
| CN115412844B CN115412844B (en) | 2024-05-24 |
Family
ID=84161211
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211024842.1A Active CN115412844B (en) | 2022-08-25 | 2022-08-25 | Multi-mode information alliance-based real-time alignment method for vehicle networking wave beams |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115412844B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111787481A (en) * | 2020-06-17 | 2020-10-16 | 北京航空航天大学 | Road-vehicle coordination high-precision sensing method based on 5G |
| CN112738764A (en) * | 2020-12-28 | 2021-04-30 | 北京邮电大学 | Broadband millimeter wave beam tracking method based on vehicle motion track cognition |
| CN113260084A (en) * | 2021-05-18 | 2021-08-13 | 北京邮电大学 | Millimeter wave-based vehicle networking V2X communication link establishment method |
| CN114120288A (en) * | 2021-12-02 | 2022-03-01 | 北京航空航天大学合肥创新研究院(北京航空航天大学合肥研究生院) | Vehicle detection method based on millimeter wave radar and video fusion |
| US20220198806A1 (en) * | 2020-12-21 | 2022-06-23 | Beihang University | Target detection method based on fusion of prior positioning of millimeter-wave radar and visual feature |
| CN114706068A (en) * | 2022-02-24 | 2022-07-05 | 重庆邮电大学 | Road side unit cooperative target tracking system, method and storage medium |
| CN114844545A (en) * | 2022-05-05 | 2022-08-02 | 东南大学 | Communication beam selection method based on sub6GHz channel and partial millimeter wave pilot frequency |
-
2022
- 2022-08-25 CN CN202211024842.1A patent/CN115412844B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111787481A (en) * | 2020-06-17 | 2020-10-16 | 北京航空航天大学 | Road-vehicle coordination high-precision sensing method based on 5G |
| US20220198806A1 (en) * | 2020-12-21 | 2022-06-23 | Beihang University | Target detection method based on fusion of prior positioning of millimeter-wave radar and visual feature |
| CN112738764A (en) * | 2020-12-28 | 2021-04-30 | 北京邮电大学 | Broadband millimeter wave beam tracking method based on vehicle motion track cognition |
| CN113260084A (en) * | 2021-05-18 | 2021-08-13 | 北京邮电大学 | Millimeter wave-based vehicle networking V2X communication link establishment method |
| CN114120288A (en) * | 2021-12-02 | 2022-03-01 | 北京航空航天大学合肥创新研究院(北京航空航天大学合肥研究生院) | Vehicle detection method based on millimeter wave radar and video fusion |
| CN114706068A (en) * | 2022-02-24 | 2022-07-05 | 重庆邮电大学 | Road side unit cooperative target tracking system, method and storage medium |
| CN114844545A (en) * | 2022-05-05 | 2022-08-02 | 东南大学 | Communication beam selection method based on sub6GHz channel and partial millimeter wave pilot frequency |
Non-Patent Citations (2)
| Title |
|---|
| 段续庭等;: "深度学习在自动驾驶领域应用综述", 无人系统技术, no. 06, 31 December 2021 (2021-12-31) * |
| 胡延平等;: "毫米波雷达与视觉传感器信息融合的车辆跟踪", 中国机械工程, no. 18, 31 December 2021 (2021-12-31) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115412844B (en) | 2024-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105488534B (en) | Traffic scene deep analysis method, apparatus and system | |
| CN110569875B (en) | Deep neural network target detection method based on feature multiplexing | |
| US20210216077A1 (en) | Method, apparatus and computer storage medium for training trajectory planning model | |
| CN114972654B (en) | Three-dimensional target detection method based on road side point cloud completion | |
| CN113095277B (en) | Unmanned aerial vehicle aerial photography vehicle detection method based on target space distribution characteristics | |
| CN112489072B (en) | Vehicle-mounted video perception information transmission load optimization method and device | |
| CN114973411A (en) | Self-adaptive evaluation method, system, equipment and storage medium for attitude motion | |
| CN117058456A (en) | Visual target tracking method based on multiphase attention mechanism | |
| US11860627B2 (en) | Image processing apparatus, vehicle, control method for information processing apparatus, storage medium, information processing server, and information processing method for recognizing a target within a captured image | |
| CN115412844A (en) | Real-time alignment method for vehicle networking beams based on multi-mode information synaesthesia | |
| Ozer et al. | Offloading deep learning powered vision tasks from UAV to 5G edge server with denoising | |
| Liu et al. | HPL-ViT: A Unified Perception Framework for Heterogeneous Parallel LiDARs in V2V | |
| CN115577305B (en) | Unmanned aerial vehicle signal intelligent recognition method and device | |
| CN115965961A (en) | Local-to-global multi-modal fusion method, system, device and storage medium | |
| CN116013067A (en) | Vehicle data processing method, processor and server | |
| CN107831762A (en) | The path planning system and method for a kind of new-energy automobile | |
| Mukhtar | Machine learning enabled-localization in 5g and lte using image classification and deep learning | |
| Tian et al. | Multimodal Transformers for Wireless Communications: A Case Study in Beam Prediction | |
| Zhang et al. | Integrated Sensing and Communications towards Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF) | |
| Gupta et al. | Efficient mmWave Beam Selection using ViTs and GVEC: GPS-based Virtual Environment Capture | |
| Neema et al. | User spatial localization for vision aided beam tracking based millimeter wave systems using convolutional neural networks | |
| KR102552051B1 (en) | Vehicle risk prediction device and method therefor | |
| Rahman et al. | Dynamic Error-bounded Lossy Compression (EBLC) to Reduce the Bandwidth Requirement for Real-time Vision-based Pedestrian Safety Applications | |
| CN114219826B (en) | Ground target tracking method applied to aerial video | |
| CN116682039A (en) | Detection and tracking method for small unmanned aerial vehicle based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |