CN115390116A - Dynamic mapping method and device based on roadside image recognition and satellite image - Google Patents

Dynamic mapping method and device based on roadside image recognition and satellite image Download PDF

Info

Publication number
CN115390116A
CN115390116A CN202211103145.5A CN202211103145A CN115390116A CN 115390116 A CN115390116 A CN 115390116A CN 202211103145 A CN202211103145 A CN 202211103145A CN 115390116 A CN115390116 A CN 115390116A
Authority
CN
China
Prior art keywords
road
image
vehicle
background
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211103145.5A
Other languages
Chinese (zh)
Other versions
CN115390116B (en
Inventor
蒲虹旭
王映焓
王雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ziqing Zhixing Technology Beijing Co ltd
Original Assignee
Ziqing Zhixing Technology Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ziqing Zhixing Technology Beijing Co ltd filed Critical Ziqing Zhixing Technology Beijing Co ltd
Priority to CN202211103145.5A priority Critical patent/CN115390116B/en
Publication of CN115390116A publication Critical patent/CN115390116A/en
Application granted granted Critical
Publication of CN115390116B publication Critical patent/CN115390116B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
    • G01S19/38—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system
    • G01S19/39—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system the satellite radio beacon positioning system transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
    • G01S19/42—Determining position
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
    • G01S19/01—Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
    • G01S19/13—Receivers
    • G01S19/35—Constructional details or hardware or software details of the signal processing chain
    • G01S19/37—Hardware or software details of the signal processing chain
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention discloses a method and a device for dynamically establishing a map based on roadside image identification and satellite images, wherein the method comprises the following steps: identifying and tracking the vehicles and the pedestrians through a roadside camera of the road to obtain a vehicle moving track and a pedestrian moving track; superposing and recursing images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road; taking a satellite image which is shot in advance and corresponds to the road background as a second background image; calculating a conversion matrix between the image coordinates of the road side camera and the satellite image coordinates; determining a road area on the second background image according to the movement track of the vehicle, and combining the road area with the second background image to obtain a static map; and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on a static map to realize a dynamic map.

Description

Dynamic mapping method and device based on roadside image recognition and satellite image
Technical Field
The invention relates to the technical field of automatic driving, in particular to a dynamic mapping method and a dynamic mapping device based on roadside image recognition and satellite images.
Background
In real life, the cameras on the road side are everywhere visible. The camera at the road side is widely applied to target recognition, face recognition, traffic monitoring and the like. Besides the monitoring video of pure video, the target detection can be realized by training a neural network model, and the purposes of flow monitoring, individual monitoring and the like are achieved.
The maps currently used by the automatic driving vehicle comprise a high-precision map, a topological map, a grid map and other multi-layer maps, and different types of maps can be used according to different scenes and functions.
Currently, actual mapping is mostly needed for obtaining a map by an automatic driving vehicle. The actual mapping is mainly realized by collecting data through a mapping instrument and manually drawing; maps for use with autonomous vehicles may be mapped by collecting data with the autonomous vehicle. The map content includes information about road shape, topological relationships, layout around roads, static obstacles, and the like.
However, the current practical mapping scheme requires manual data acquisition in advance for mapping, which is complicated. The field surveying and mapping is needed to be carried out again every place, the reusability is poor, and the mass production cannot be realized. In addition, in actual surveying and mapping, only static obstacles at the time of surveying and mapping can be drawn, and subsequent dynamic obstacles cannot be updated.
Disclosure of Invention
It is an object of the present invention to provide a method and apparatus for dynamic mapping based on roadside image recognition and satellite images to overcome or at least alleviate at least one of the above-mentioned deficiencies of the prior art.
In order to achieve the above object, the present invention provides a dynamic mapping method based on roadside image recognition and satellite images, comprising:
identifying and tracking vehicles and pedestrians through a road side camera of a road to obtain a vehicle moving track and a pedestrian moving track;
performing superposition recursion on images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road;
taking a satellite image which is shot in advance and corresponds to the road background as a second background image;
selecting a feature point in a road, and calculating a conversion matrix between image coordinates of a roadside camera and satellite image coordinates according to the corresponding relation between the position of the feature point in a first background image and the position of the feature point in a second background image;
determining a road area on the second background image according to the movement track of the vehicle on the road, and combining the road area and the second background image to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on a static map to realize a dynamic map.
Preferably, the recognizing and tracking of the vehicles and pedestrians by the roadside cameras of the road includes:
carrying out target recognition training on vehicles and pedestrians by utilizing a neural network model and data in a database in advance, wherein each recognition result comprises two-dimensional frame information and category information, the two-dimensional frame information is used for indicating the position of a target, and the category information is used for indicating that the target is various vehicles or pedestrians; and after the target recognition training, the road side camera is used for carrying out target recognition and tracking on the vehicles and the pedestrians on the road, and obtaining the moving track of the vehicles and the moving track of the pedestrians.
Preferably, the superimposing and recursion of the images shot by the road-side camera, and the extraction of the same parts in a preset number of images as the first background map of the road, include:
superposing a preset number of images according to the following formula, and removing image parts which change before and after superposition to obtain a first background image:
Figure BDA0003840181640000021
wherein ,
Figure BDA0003840181640000022
the background map obtained for the t-th overlay,
Figure BDA0003840181640000023
background derived for t-1 st superpositionIn the figure, the figure shows that,
Figure BDA0003840181640000024
for the image taken on the t-th road side, n t And alpha is a preset empirical value. For example, α ≈ 0.01,n t ≥70。
Preferably, the determining the road area on the second background map according to the moving track of the vehicle includes:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is one element in the set;
for each vehicle movement trajectory in the set:
calculating pixel difference values of boundary pixels and external adjacent pixels in the vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track and the adjacent pixels to serve as a communicated area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and if the pixel difference values meet preset conditions, combining the adjacent pixels into the connected region;
and if the pixel difference value does not meet the preset condition, determining the boundary of the connected region according to the adjacent pixels.
Preferably, the method may further comprise: simultaneously calculating pixel difference values of a plurality of vehicle moving tracks and adjacent pixels;
and if the same pixel belongs to adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixel of the two connected regions meets a preset condition, combining the two connected regions.
Preferably, the calculating the transformation matrix between the image coordinates of the roadside camera and the image coordinates of the satellite includes:
and calculating to obtain a plurality of conversion matrixes according to the corresponding relation between the positions of the characteristic points in the first background image and the positions of the characteristic points in the second background image, so that the conversion matrix with the most matched characteristic point positions is used as a final conversion matrix.
The invention also provides a dynamic mapping device based on roadside image recognition and satellite images, which comprises:
the communication module is used for communicating with the roadside camera;
a processing module to:
identifying and tracking the vehicles and the pedestrians through a roadside camera of the road to obtain a vehicle moving track and a pedestrian moving track;
superposing and recursing images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road;
taking a satellite image which is shot in advance and corresponds to the road background as a second background image;
selecting feature points in a road, and calculating a conversion matrix between image coordinates of a road side camera and satellite image coordinates according to the corresponding relation between the positions of the feature points in the first background image and the positions of the feature points in the second background image;
determining a road area on the second background image according to the movement track of the vehicle on the road, and combining the road area and the second background image to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
Preferably, the processing module is configured to:
superposing a preset number of images according to the following formula, and removing image parts which change before and after superposition to obtain a first background image:
Figure BDA0003840181640000031
wherein ,
Figure BDA0003840181640000032
the background map obtained for the t-th overlay,
Figure BDA0003840181640000033
the background map obtained for the t-1 st overlay,
Figure BDA0003840181640000034
for the image taken on the t-th road side, n t And alpha is a predetermined empirical value. For example, α ≈ 0.01,n t ≥70。
Preferably, the processing module is configured to:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is one element in the set;
for each vehicle movement trajectory in the set:
calculating pixel difference values of boundary pixels and external adjacent pixels in the vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track and the adjacent pixels to serve as a communicated area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and if the pixel difference values meet preset conditions, combining the adjacent pixels into the connected region;
and if the pixel difference value does not meet the preset condition, determining the boundary of the connected region according to the adjacent pixels.
Preferably, the processing module is configured to:
simultaneously calculating pixel difference values of a plurality of vehicle moving tracks and adjacent pixels;
and if the same pixel belongs to adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixel of the two connected regions meets a preset condition, combining the two connected regions.
Due to the adoption of the technical scheme, the invention has the following advantages:
by adopting the scheme of the invention, the dynamic map can be made by directly utilizing the map and the satellite image provided by the road side monitoring equipment, actual surveying and mapping are not required, manual calibration is also not required, and the degree of automation is high. Compared with a satellite map, the automatic vehicle-driving method is more structured and can perform real-time dynamic updating, and by using the scheme provided by the invention, the automatic vehicle-driving method can perform advanced judgment preparation on the traffic condition of a monitored area under a complex traffic environment and optimize decision.
Drawings
Fig. 1 is a schematic flow chart of a dynamic mapping method based on roadside image recognition and satellite images provided by the present invention.
Fig. 2 is a schematic diagram of a target moving track obtained by roadside sensing in the example of the present invention.
FIG. 3 is a schematic diagram showing a comparison between roadside monitoring images and satellite images of the same background in an example of the present invention.
Fig. 4 is a schematic diagram of a dynamic map obtained in an example of the present invention.
Fig. 5 is a schematic structural diagram of a dynamic map building device based on roadside image recognition and satellite images provided by the invention.
Fig. 6 is a schematic view of an application scene of the dynamic mapping device based on roadside image recognition and satellite images provided by the invention.
Detailed Description
In the drawings, the same or similar reference numerals are used to designate the same or similar elements or elements having the same or similar functions. Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
In the description of the present invention, the terms "central", "longitudinal", "lateral", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience in describing the present invention and simplifying the description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed in a particular orientation, and be operated, and therefore, should not be construed as limiting the scope of the present invention.
In the case of conflict, the technical features in the embodiments and implementations of the present invention may be combined with each other, and are not limited to the embodiments or implementations in which the technical features are located.
The present invention will be further described with reference to the accompanying drawings and specific embodiments, it should be noted that the technical solutions and design principles of the present invention are described in detail in the following only by way of an optimized technical solution, but the scope of the present invention is not limited thereto.
The following terms are referred to herein, and their meanings are explained below for ease of understanding. It will be understood by those skilled in the art that the following terms may have other names, but any other names should be considered consistent with the terms set forth herein without departing from their meaning.
The invention provides a dynamic mapping method based on roadside image recognition and satellite images, which comprises the following steps of:
and S10, identifying and tracking the vehicle and the pedestrian through a road side camera of the road, and acquiring a vehicle moving track and a pedestrian moving track.
The method comprises the steps that a neural network model and data in a database are utilized in advance to conduct target recognition training of vehicles and pedestrians, each recognition result comprises two-dimensional frame information and category information, the two-dimensional frame information is used for indicating the position of a target, and the category information is used for indicating that the target is various vehicles or pedestrians. And then, the target is identified and tracked by applying to the road side to obtain the track. After the target recognition training, the roadside camera is used for carrying out target recognition and tracking on the vehicles and the pedestrians on the road, and a vehicle moving track and a pedestrian moving track are obtained.
And step S20, performing superposition recursion on the images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road.
The method comprises the following steps of superposing a preset number of images according to the following formula, and removing image parts which change before and after superposition to obtain a first background image:
Figure BDA0003840181640000051
wherein ,
Figure BDA0003840181640000052
the background map found for the t-th overlay,
Figure BDA0003840181640000053
the background map obtained for the t-1 st overlay,
Figure BDA0003840181640000054
for the image taken on the t-th road side, n t And alpha is a predetermined empirical value. For example, α ≈ 0.01,n t ≥70。
And step S30, taking the satellite image which is shot in advance and corresponds to the road background as a second background image.
The corresponding road backgrounds may be the same road background, or may include part of the same road background.
And S40, selecting the feature points in the road, and calculating a conversion matrix between the image coordinates of the road side camera and the satellite image coordinates according to the corresponding relation between the positions of the feature points in the first background image and the positions of the feature points in the second background image.
A plurality of conversion matrices can be obtained by calculation according to the correspondence between the positions of the feature points in the first background map and the positions of the feature points in the second background map, so that the conversion matrix with the most matched feature point positions serves as a final conversion matrix.
And S50, determining a road area on the second background image according to the movement track of the vehicle, and combining the road area and the second background image to obtain a static map.
Wherein, determining the road area on the second background map according to the vehicle movement track comprises: acquiring a set of vehicle moving tracks on a road; wherein each vehicle movement track is one element in the set; for each vehicle movement trajectory in the set: calculating pixel difference values of boundary pixels and external adjacent pixels in the vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track and the adjacent pixels to serve as a communicated area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and if the pixel difference values meet preset conditions, combining the adjacent pixels into the connected region; and if the pixel difference value does not meet the preset condition, determining the boundary of the connected region according to the adjacent pixels.
Wherein, the method also comprises: simultaneously calculating pixel difference values of a plurality of vehicle moving tracks and adjacent pixels; and if the same pixel belongs to adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixel of the two connected regions meets a preset condition, combining the two connected regions.
And S60, converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
The method for dynamically constructing a map based on roadside image recognition and satellite images provided by the invention is described in detail by an example.
The method provided by the invention can be divided into three parts: the road side sensing part, the image calibration part and the dynamic image construction part.
1. Roadside sensing section
Roadside perception is achieved through a roadside camera, the position of the roadside camera is generally located on the side of a road, the roadside camera is consistent with monitoring of a general road, the height of the roadside camera is generally 3-5 m from the ground, and the depth of the roadside camera is 3-5 m from the nearest observation point.
Before the roadside camera identifies the target, a neural network (such as yolov 4) model based on an anchor frame method can be used in advance, and a traffic road data set is used for detecting and training road users and pedestrians. Wherein, the road users include various vehicles.
The output result of the target recognition by the roadside camera is the two-dimensional frame information of all the targets in one frame image and the feature information such as the category thereof, and includes (x), for example b ,y b ,w b ,h b E, c) six items of information.
wherein ,(xb ,y b ) Representing the centre point, w, of the two-dimensional frame sought b Width, h, of the two-dimensional frame b Representing the height of the two-dimensional frame sought, e represents the confidence of the prediction, c represents the probability vector of the classification, i.e. representing the class of the object, whether it is a pedestrian or a vehicle of all kinds.
When two-dimensional frame information is obtained, coordinates of the upper-left corner point of the frame, which are expressed as (x), can be obtained c ,y c ) Further, the following relationship is used to obtain:
x b ＝σ(x o )+x c
y b ＝σ(y o )+y c
Figure BDA0003840181640000071
Figure BDA0003840181640000072
wherein, the sigma is a Sigmoid function and takes a value of [0,1 ]]An interval. (x) o ,y o ,w o ,h o ) Different classes have different model predicted compensation values for a preset model predicted compensation value.
Taking the six items of information as input, applying Kalman filtering, adopting a multi-target and multi-type tracking algorithm to track each target, and obtaining a moving track set of all target motions as follows:
Figure BDA0003840181640000073
Figure BDA0003840181640000074
as shown in fig. 2, v i And p i Respectively representing a certain vehicle or pedestrian,
Figure BDA0003840181640000075
and
Figure BDA0003840181640000076
represents the moving trajectory of the vehicle or pedestrian, and V and P represent sets of tracking IDs of the detected vehicle and pedestrian, respectively, in which each trajectory point includes the above-mentioned six items of information.
2. Image calibration part
The image calibration part comprises background extraction and image coordinate conversion relation establishment.
Background extraction: used as background roads and static obstacles in the drawn map. Occurring at an initial time, and subsequent updates may occur at intervals.
In this example, the road background in the monitored area is found by superimposing recursion on the images taken by the road-side camera, leaving the same portion in the images.
For example, the superposition relationship is as follows
Figure BDA0003840181640000077
wherein ,
Figure BDA0003840181640000078
background, n, found for the t-th superposition t And α is an empirical value, n t Approximately 70, i.e. 70 images are iterated to find the background, alpha is the weighting value of the new image, approximately 0.01,
Figure BDA0003840181640000079
the monitored image shot for the t time.
Under the condition that the position of the roadside camera is basically unchanged, the background such as a road structure and the like is fixed, and pixels of the background are also fixed, so that the sum of the added pixels of the background part is unchanged no matter what the value of factors (namely alpha and 1-alpha) multiplied before the t-th time and the t-1-th time is, the dynamic object pixels are refreshed along with the superposition of the images, and the image with only the background is obtained after the dynamic object pixels are superposed for a certain number of times.
Establishing an image coordinate conversion relation:
the invention relates to an image shot by a road side camera, namely a road side monitoring camera image
Figure BDA00038401816400000710
And satellite images
Figure BDA00038401816400000711
Fisheye view images may also be used
Figure BDA00038401816400000712
The coordinates of the fish-eye perspective image and the satellite perspective image are basically consistent (namely
Figure BDA00038401816400000713
)。
Conversion of surveillance camera images to satellite or fisheye images, conversion matrices available
Figure BDA0003840181640000081
Figure BDA0003840181640000082
Is shown, and the relationship is reversible, i.e.
Figure BDA0003840181640000083
wherein ,
Figure BDA0003840181640000084
representing coordinates in the satellite image or the fisheye image,
Figure BDA0003840181640000085
representing coordinates in the surveillance camera image.
The relation of G is equal to G = K [ R | T ], K is the camera internal reference matrix, and R and T are the rotation and translation matrices in the camera external reference matrix.
In this example, G is calculated by means of feature matching.
As shown in fig. 3, in the background-only monitoring image
Figure BDA0003840181640000086
And satellite images
Figure BDA0003840181640000087
Respectively extracting feature points and performing feature matching. The two images correspond to the same background, satellite image
Figure BDA0003840181640000088
It may be shot in advance or acquired in real time. The process comprises the following steps:
selecting characteristic points; the feature points may be empirical values, such as road boundaries, corners, etc. at easily determined locations.
Adjusting the illumination, the tone and the like of the two pictures to the same range, and then uniformly adjusting the two pictures to gray level images to obtain processed characteristic points;
and comparing and matching the Feature points of the two processed pictures, and obtaining the Feature matching points of the two pictures by utilizing an ASIFT (Affine and Scale Invariant Feature Transform) algorithm. In this example, a conversion matrix G may be calculated by using a Random Sample Consensus (Random Sample Consensus) algorithm through every four feature matching points, and then, whether other matching points match simultaneously under the matrix G is calculated one by one, and the conversion matrix G with the most feature matching points is selected as the final result estimation value G. The matching points may also be determined directly by visual selection, which is not limited herein.
After the conversion matrix G is determined, converting the image of the monitoring camera through the conversion matrix G, and checking whether the converted image can be overlapped with the satellite image; if so, the transformation matrix G is valid, otherwise the transformation matrix G is redetermined.
3. Dynamic mapping part
In this example, an SRG (Region Growing) algorithm is adopted, and a set M of all vehicle movement trajectories obtained by a sensing part is used V As an initial seed, the moving track of each vehicle is a seed, and the background-only monitoring image obtained from the previous part is used
Figure BDA0003840181640000089
As an initial extracted image, the mapping process comprises the following steps:
calculating the pixel difference between each seed x and the adjacent pixel; for example, the image is first converted into a gray scale image, the values of the three channels of the image are made to be the same, and then the value of one channel is taken to calculate the difference between the pixels. In a preferred implementation, the neighboring pixels may include 8 pixels adjacent to the periphery of a certain pixel.
If the pixel difference with a neighboring pixel is less than the threshold tau x Dividing the seed and the neighbor pixel into a set as a connected region; then, new neighbor pixels are searched for with the boundary pixels of the connected region, the pixel difference between the boundary pixels and the neighbor pixels is calculated, if still less than the threshold tau x Merging new neighbor pixels into the connected region, and continuously searching for the next neighbor pixel until the pixel difference between the boundary pixel and the neighbor pixels is not less than the threshold tau x Then the boundary of the connected region is determined from the neighboring pixels. For example, there are neighboring pixels as the boundary, or boundary pixels of a connected region as the boundary.
When a pixel is adjacent to two sets at the same time, the pixel difference can be calculated respectively, if both are less than the threshold tau x Then the two sets are merged, i.e. two connected regions are merged into one connected region.
And repeating the steps until no new connected region exists, and taking the connected region as a road region.
In this example, the road area may be refined, and the boundaries of different connected areas may be identified by different lines.
Through the steps, the static map can be obtained. The map can be automatically updated at intervals, namely, a background image is obtained by updating the image calibration part, and then an updated static map is obtained by calculating the background image.
After the static map is obtained, the information of the real-time dynamic obstacle can be added to obtain a dynamic map, as shown in fig. 4.
Six items of information (x) obtained by the roadside sensing section b ,y b ,w b ,h b And e, c), after multi-target and multi-type tracking is carried out, the position of an enclosing frame of a single target, the speed and the orientation angle under pixel coordinates can be calculated.
For example, the calculation of the bounding box position may include:
taking the bottom midpoint of the two-dimensional surrounding frame as a reference point, and calculating to obtain the coordinates of the bottom midpoint based on the position information obtained by the roadside sensing part:
Figure BDA0003840181640000091
converting the coordinate of the bottom midpoint into the bottom midpoint of the bounding box under the satellite image coordinate by using a conversion matrix G and a formula of a calibration part
Figure BDA0003840181640000092
Figure BDA0003840181640000093
The length and the width of the bounding box under the satellite image coordinate are respectively, the values can be empirical values, namely the empirical values of the objects are taken according to the types of the targets, and then the conversion is carried out according to the proportion value from meters to pixels on the satellite map to obtain the length and the width on the pixels. The orientation of the bounding box can be the actual orientation angle obtained by perception tracking.
By adopting the scheme of the invention, the automatic driving vehicle can directly utilize the map and the satellite image provided by the road side monitoring equipment to make the dynamic map without actual surveying and mapping and manual calibration, and the degree of automation is high. Compared with a satellite map, the method is more structured, can also carry out real-time dynamic updating, and can ensure that the automatic driving vehicle carries out advanced judgment preparation and optimized decision on the traffic condition of a monitored area under a complex traffic environment.
The invention provides a dynamic mapping device based on roadside image recognition and satellite images, as shown in fig. 5, comprising:
a communication module 51 for communicating with the roadside camera;
a processing module 52 configured to:
tracking vehicles and pedestrians through a road side camera of a road to obtain a vehicle moving track and a pedestrian moving track;
superposing and recursing images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road;
taking a satellite image which is shot in advance and corresponds to a road background as a second background image;
selecting a feature point in a road, and calculating a conversion matrix between image coordinates of a roadside camera and satellite image coordinates according to the corresponding relation between the position of the feature point in a first background image and the position of the feature point in a second background image;
determining a road area on the second background image according to the movement track of the vehicle, and combining the road area with the second background image to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
Wherein the processing module 52 is configured to:
superposing a preset number of images according to the following formula, and removing image parts which change before and after superposition to obtain a first background image:
Figure BDA0003840181640000101
wherein ,
Figure BDA0003840181640000102
the background map obtained for the t-th overlay,
Figure BDA0003840181640000103
the background map obtained for the t-1 st overlay,
Figure BDA0003840181640000104
for the image taken on the t-th road side, n t And alpha is a preset empirical value. For example, α ≈ 0.01,n t ≥70。
Wherein the processing module 52 is configured to:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is one element in the set;
for each vehicle movement trajectory in the set:
calculating pixel difference values of boundary pixels and external adjacent pixels in the vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track and the adjacent pixels to serve as a communicated area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and if the pixel difference values meet preset conditions, combining the adjacent pixels into the connected region;
and if the pixel difference value does not meet the preset condition, determining the boundary of the connected region according to the adjacent pixels.
Wherein the processing module 52 is configured to:
simultaneously calculating pixel difference values of a plurality of vehicle moving tracks and adjacent pixels;
and if the same pixel belongs to adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixel of the two connected regions meets a preset condition, combining the two connected regions.
Fig. 6 is a schematic view of an application scene of the dynamic mapping device based on roadside image recognition and satellite images provided by the invention. As shown in fig. 6, the scene includes a map calculation device 61, a roadside camera 62, and an autonomous vehicle 63.
The map computing device 61 includes a communication gateway device 611, a V2X (Vehicle-to-Vehicle wireless communication technology) device 612, and a processor 613.
The communication gateway device 611 is configured to interactively transmit information such as an image captured by a road-side camera with the road-side camera (or other road-side devices), and receive a satellite image. For example, the connection between the roadside apparatus and the communication gateway apparatus 611 may be realized by USB (Universal Serial Bus)/Ethernet (Ethernet).
The V2X device 612 is used to communicate with the autonomous vehicle.
Processor 613 comprises three modules: the target perception module is used for carrying out target detection on the image obtained by the camera, the image calibration module is used for solving a static map and a conversion matrix between the image obtained by the camera and a satellite image, and the dynamic mapping module is used for calculating a real-time dynamic map in a monitoring area.
And the road side camera 62 is connected with the computing unit through USB/Ethernet and is used for obtaining a real-time monitoring road image.
The autonomous vehicle 63 is connected to the map calculation device 61 by wire or wirelessly, and receives the dynamic map of the map calculation device 61.
The map calculation device 61 may also be integrated into the autonomous vehicle 63, so that the autonomous vehicle 63 can perform calculations to obtain a dynamic map.
By adopting the scheme of the invention, the automatic driving vehicle can directly utilize the map and the satellite image provided by the road side monitoring equipment to make the dynamic map without actual surveying and mapping and manual calibration, and the degree of automation is high. Compared with a satellite map, the automatic driving vehicle is more structured, real-time dynamic updating can be carried out, the automatic driving vehicle can be guaranteed to judge and prepare the traffic condition of a monitored area in advance under a complex traffic environment, and decision is optimized.
Finally, it should be pointed out that: the above examples are only for illustrating the technical solutions of the present invention, and are not limited thereto. Those of ordinary skill in the art will understand that: modifications can be made to the technical solutions described in the foregoing embodiments, or some technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A dynamic mapping method based on roadside image recognition and satellite images is characterized by comprising the following steps:
identifying and tracking vehicles and pedestrians through a road side camera of a road to obtain a vehicle moving track and a pedestrian moving track;
performing superposition recursion on images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road;
taking a satellite image which is shot in advance and corresponds to the road background as a second background image;
selecting a feature point in a road, and calculating a conversion matrix between image coordinates of a roadside camera and satellite image coordinates according to the corresponding relation between the position of the feature point in a first background image and the position of the feature point in a second background image;
determining a road area on the second background image according to the movement track of the vehicle on the road, and combining the road area and the second background image to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on a static map to realize a dynamic map.
2. The method of claim 1, wherein identifying and tracking vehicles and pedestrians by roadside cameras of the road comprises:
carrying out target recognition training on vehicles and pedestrians by utilizing a neural network model and data in a database in advance, wherein each recognition result comprises two-dimensional frame information and category information, the two-dimensional frame information is used for indicating the position of a target, and the category information is used for indicating that the target is various vehicles or pedestrians;
and after the target recognition training, the road side camera is used for carrying out target recognition and tracking on the vehicles and the pedestrians on the road, and obtaining the moving track of the vehicles and the moving track of the pedestrians.
3. The method of claim 1, wherein performing a superposition recursion on the images captured by the road-side camera, and extracting the same parts in a preset number of images as a first background map of the road comprises:
superposing a preset number of images according to the following formula, and removing image parts which change before and after superposition to obtain a first background image:
Figure FDA0003840181630000011
wherein ,
Figure FDA0003840181630000012
the background map obtained for the t-th overlay,
Figure FDA0003840181630000013
the background map obtained for the t-1 st overlay,
Figure FDA0003840181630000014
for the image taken on the t-th road side, n t And alpha is a preset empirical value.
4. The method of claim 1, wherein determining the road region on the second background map according to the vehicle movement track comprises:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is one element in the set;
for each vehicle movement trajectory in the set:
calculating pixel difference values of boundary pixels and external adjacent pixels in the vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track and the adjacent pixels to serve as a communicated area; calculating pixel difference values of boundary pixels of the connected region and pixels of the adjacent non-connected region, and if the pixel difference values meet preset conditions, combining the adjacent pixels into the connected region;
and if the pixel difference value does not meet the preset condition, determining the boundary of the connected region according to the adjacent pixels.
5. The method of claim 4, further comprising: simultaneously calculating pixel difference values of a plurality of vehicle moving tracks and adjacent pixels;
and if the same pixel belongs to adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixel of the two connected regions meets a preset condition, combining the two connected regions.
6. The method of claim 1, wherein calculating the transformation matrix between the roadside camera image coordinates and the satellite image coordinates comprises:
and calculating to obtain a plurality of conversion matrixes according to the corresponding relation between the positions of the characteristic points in the first background image and the positions of the characteristic points in the second background image, so that the conversion matrix with the most matched characteristic point positions is used as a final conversion matrix.
7. A dynamic mapping device based on roadside image recognition and satellite images is characterized by comprising:
the communication module is used for communicating with the roadside camera;
a processing module to:
identifying and tracking vehicles and pedestrians through a road side camera of a road to obtain a vehicle moving track and a pedestrian moving track;
superposing and recursing images shot by the road side camera, and extracting the same parts in a preset number of images to be used as a first background image of the road;
taking a satellite image which is shot in advance and corresponds to a road background as a second background image;
selecting feature points in a road, and calculating a conversion matrix between image coordinates of a road side camera and satellite image coordinates according to the corresponding relation between the positions of the feature points in the first background image and the positions of the feature points in the second background image;
determining a road area on the second background image according to the movement track of the vehicle on the road, and combining the road area with the second background image to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on a static map to realize a dynamic map.
8. The apparatus of claim 7, wherein the processing module is configured to:
superposing a preset number of images through the following formula, and removing image parts which change before and after superposition to obtain a first background image:
Figure FDA0003840181630000031
wherein ,
Figure FDA0003840181630000032
the background map obtained for the t-th overlay,
Figure FDA0003840181630000033
the background map obtained for the t-1 st overlay,
Figure FDA0003840181630000034
for the image taken on the t-th road side, n t And alpha is a predetermined empirical value.
9. The apparatus of claim 7, wherein the processing module is configured to:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is one element in the set;
for each vehicle movement trajectory in the set:
calculating pixel difference values of boundary pixels and external adjacent pixels in the vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track and the adjacent pixels to serve as a communicated area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and if the pixel difference values meet preset conditions, combining the adjacent pixels into the connected region;
and if the pixel difference value does not meet the preset condition, determining the boundary of the connected region according to the adjacent pixels.
10. The apparatus of claim 9, wherein the processing module is configured to:
simultaneously calculating pixel difference values of a plurality of vehicle moving tracks and adjacent pixels;
and if the same pixel belongs to adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixel of the two connected regions meets the preset condition, combining the two connected regions.
CN202211103145.5A 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image Active CN115390116B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211103145.5A CN115390116B (en) 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211103145.5A CN115390116B (en) 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image

Publications (2)

Publication Number Publication Date
CN115390116A true CN115390116A (en) 2022-11-25
CN115390116B CN115390116B (en) 2023-05-12

Family

ID=84125846

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211103145.5A Active CN115390116B (en) 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image

Country Status (1)

Country Link
CN (1) CN115390116B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101109647A (en) * 2007-08-22 2008-01-23 广东瑞图万方科技有限公司 Navigation apparatus
US20120230550A1 (en) * 2011-03-10 2012-09-13 Jacob Kraut Method and Apparatus for Generating a Map from Landmarks
CN107180215A (en) * 2017-05-31 2017-09-19 同济大学 Figure and high-precision locating method are built in parking lot based on warehouse compartment and Quick Response Code automatically
CN108288014A (en) * 2017-01-09 2018-07-17 北京四维图新科技股份有限公司 Intelligent road extracting method and device, extraction model construction method and hybrid navigation system
CN110717007A (en) * 2019-10-15 2020-01-21 财团法人车辆研究测试中心 Map data positioning system and method applying roadside feature identification
CN111391823A (en) * 2019-12-27 2020-07-10 湖北亿咖通科技有限公司 Multilayer map making method for automatic parking scene
CN112308904A (en) * 2019-07-29 2021-02-02 北京初速度科技有限公司 Vision-based drawing construction method and device and vehicle-mounted terminal
CN113156461A (en) * 2021-05-17 2021-07-23 紫清智行科技(北京)有限公司 Dynamic loading method and system for 2D laser radar SLAM map
CN113865581A (en) * 2021-09-16 2021-12-31 江苏大学 Closed scene positioning method based on multi-level map

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101109647A (en) * 2007-08-22 2008-01-23 广东瑞图万方科技有限公司 Navigation apparatus
US20120230550A1 (en) * 2011-03-10 2012-09-13 Jacob Kraut Method and Apparatus for Generating a Map from Landmarks
CN108288014A (en) * 2017-01-09 2018-07-17 北京四维图新科技股份有限公司 Intelligent road extracting method and device, extraction model construction method and hybrid navigation system
CN107180215A (en) * 2017-05-31 2017-09-19 同济大学 Figure and high-precision locating method are built in parking lot based on warehouse compartment and Quick Response Code automatically
CN112308904A (en) * 2019-07-29 2021-02-02 北京初速度科技有限公司 Vision-based drawing construction method and device and vehicle-mounted terminal
CN110717007A (en) * 2019-10-15 2020-01-21 财团法人车辆研究测试中心 Map data positioning system and method applying roadside feature identification
CN111391823A (en) * 2019-12-27 2020-07-10 湖北亿咖通科技有限公司 Multilayer map making method for automatic parking scene
CN113156461A (en) * 2021-05-17 2021-07-23 紫清智行科技(北京)有限公司 Dynamic loading method and system for 2D laser radar SLAM map
CN113865581A (en) * 2021-09-16 2021-12-31 江苏大学 Closed scene positioning method based on multi-level map

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
何剑涛: "面向动态环境的稠密地图构建与更新", 《中国优秀硕士学位论文全文数据库信息科技辑》 *
孙彬: "基于跨视角匹配的图像地理位置定位研究", 《中国优秀硕士学位论文全文数据库基础科学辑》 *

Also Published As

Publication number Publication date
CN115390116B (en) 2023-05-12

Similar Documents

Publication Publication Date Title
CN112184818B (en) Vision-based vehicle positioning method and parking lot management system applying same
CN111462200A (en) Cross-video pedestrian positioning and tracking method, system and equipment
CN107067794B (en) Indoor vehicle positioning and navigation system and method based on video image processing
CN111220993B (en) Target scene positioning method and device, computer equipment and storage medium
EP2798611B1 (en) Camera calibration using feature identification
CN107767400B (en) Remote sensing image sequence moving target detection method based on hierarchical significance analysis
CN109598794B (en) Construction method of three-dimensional GIS dynamic model
CN109099929B (en) Intelligent vehicle positioning device and method based on scene fingerprints
CN112801074B (en) Depth map estimation method based on traffic camera
CN114022830A (en) Target determination method and target determination device
CN108711172B (en) Unmanned aerial vehicle identification and positioning method based on fine-grained classification
CN112308913B (en) Vehicle positioning method and device based on vision and vehicle-mounted terminal
CN114120283A (en) Method for distinguishing unknown obstacles in road scene three-dimensional semantic segmentation
CN112800938B (en) Method and device for detecting occurrence of side rockfall of unmanned vehicle
CN109753945A (en) Target subject recognition methods, device, storage medium and electronic equipment
CN114325634A (en) Method for extracting passable area in high-robustness field environment based on laser radar
CN113256731A (en) Target detection method and device based on monocular vision
CN112884841B (en) Binocular vision positioning method based on semantic target
CN113689459B (en) Real-time tracking and mapping method based on GMM and YOLO under dynamic environment
CN114511592A (en) Personnel trajectory tracking method and system based on RGBD camera and BIM system
CN113971697A (en) Air-ground cooperative vehicle positioning and orienting method
CN117333846A (en) Detection method and system based on sensor fusion and incremental learning in severe weather
CN114170499A (en) Target detection method, tracking method, device, visual sensor and medium
CN115390116B (en) Dynamic mapping method and device based on road side image recognition and satellite image
CN111833384B (en) Method and device for rapidly registering visible light and infrared images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant