CN115390116B - Dynamic mapping method and device based on road side image recognition and satellite image - Google Patents

Dynamic mapping method and device based on road side image recognition and satellite image Download PDF

Info

Publication number
CN115390116B
CN115390116B CN202211103145.5A CN202211103145A CN115390116B CN 115390116 B CN115390116 B CN 115390116B CN 202211103145 A CN202211103145 A CN 202211103145A CN 115390116 B CN115390116 B CN 115390116B
Authority
CN
China
Prior art keywords
road
image
background
map
road side
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211103145.5A
Other languages
Chinese (zh)
Other versions
CN115390116A (en
Inventor
蒲虹旭
王映焓
王雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ziqing Zhixing Technology Beijing Co ltd
Original Assignee
Ziqing Zhixing Technology Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ziqing Zhixing Technology Beijing Co ltd filed Critical Ziqing Zhixing Technology Beijing Co ltd
Priority to CN202211103145.5A priority Critical patent/CN115390116B/en
Publication of CN115390116A publication Critical patent/CN115390116A/en
Application granted granted Critical
Publication of CN115390116B publication Critical patent/CN115390116B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
    • G01S19/38—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system
    • G01S19/39—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system the satellite radio beacon positioning system transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
    • G01S19/42—Determining position
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
    • G01S19/01—Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
    • G01S19/13—Receivers
    • G01S19/35—Constructional details or hardware or software details of the signal processing chain
    • G01S19/37—Hardware or software details of the signal processing chain
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention discloses a dynamic mapping method and a device based on road side image identification and satellite images, wherein the method comprises the following steps: identifying and tracking vehicles and pedestrians through road side cameras of a road to acquire vehicle movement tracks and pedestrian movement tracks; overlapping and recursing images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road; taking a satellite image corresponding to the road background shot in advance as a second background image; calculating a conversion matrix between the image coordinates of the road side camera and the satellite image coordinates; determining a road area on the second background map according to the vehicle movement track, and combining the road area with the second background map to obtain a static map; and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.

Description

Dynamic mapping method and device based on road side image recognition and satellite image
Technical Field
The invention relates to the technical field of automatic driving, in particular to a dynamic mapping method and device based on road side image recognition and satellite images.
Background
In real life, cameras on the road side are visible everywhere. Cameras on road sides are widely used for target recognition, face recognition, traffic monitoring and the like. Besides the monitoring video of the pure video, the target detection can be realized by training a neural network model, so that the purposes of flow monitoring, individual monitoring and the like are achieved.
The current maps of the automatic driving vehicle comprise a high-precision map, a topological map, a grid map and other multi-layer maps, and different forms of maps can be used according to different scenes and functions.
Currently, actual mapping is often required for an automatic driving vehicle to acquire a map. The actual mapping is mainly carried out by collecting data through a mapping instrument and then manually drawing; the map used by the automatic driving vehicle can be used for collecting data by the automatic driving vehicle, so that the map drawing can be used. The map content includes a map for road shape, topological relation, layout around the road, static obstacle, and the like.
However, the current practical mapping scheme requires manual data acquisition in advance for mapping, which is complicated. Each place needs to be mapped again in the field, has poor reusability and cannot be produced in batches. Moreover, actual mapping can only map static obstacles at the mapping moment, and subsequent dynamic obstacles cannot be updated.
Disclosure of Invention
It is an object of the present invention to provide a method and apparatus for dynamic mapping based on road side image recognition and satellite images that overcomes or at least alleviates at least one of the above-mentioned drawbacks of the prior art.
In order to achieve the above object, the present invention provides a dynamic mapping method based on road side image recognition and satellite image, comprising:
identifying and tracking vehicles and pedestrians through road side cameras of a road to acquire vehicle movement tracks and pedestrian movement tracks;
overlapping and recursing images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road;
taking a satellite image corresponding to the road background shot in advance as a second background image;
selecting characteristic points in a road, and calculating a conversion matrix between road side camera image coordinates and satellite image coordinates according to the corresponding relation between the positions of the characteristic points in a first background image and the positions of the characteristic points in a second background image;
determining a road area on the second background map according to the vehicle moving track on the road, and combining the road area with the second background map to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
Preferably, identifying and tracking vehicles and pedestrians through road side cameras of a road includes:
the method comprises the steps that target recognition training of vehicles and pedestrians is carried out by utilizing a neural network model and data in a database in advance, each recognition result comprises two-dimensional frame information and category information, wherein the two-dimensional frame information is used for indicating the position of a target, and the category information is used for indicating that the target is various vehicles or pedestrians; and after the target recognition training, a road side camera is used for carrying out target recognition and tracking on vehicles and pedestrians on the road, and a vehicle moving track and a pedestrian moving track are obtained.
Preferably, overlapping recursion is performed on images shot by a road side camera, and the same part in a preset number of images is extracted as a first background image of a road, including:
overlapping a preset number of images, and removing image parts which change before and after overlapping to obtain a first background image:
Figure BDA0003840181640000021
wherein ,
Figure BDA0003840181640000022
for the background map determined for the t-th overlay, < >>
Figure BDA0003840181640000023
The background map obtained for the t-1 st overlay,/->
Figure BDA0003840181640000024
For the image shot at the t th road side, n t And alpha is a preset empirical value. For example, α≡0.01, n t ≥70。
Preferably, determining the road area on the second background map according to the vehicle movement track includes:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is an element in the set;
for each vehicle movement trajectory in the collection:
calculating pixel difference values of boundary pixels and external adjacent pixels in a vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track with the adjacent pixels to serve as a communication area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and merging the adjacent pixels into the connected region if the pixel difference values meet preset conditions;
and if the pixel difference value does not meet the preset condition, determining the boundary of the communication area according to the adjacent pixels.
Preferably, the method may further comprise: simultaneously calculating pixel difference values of a plurality of vehicle movement tracks and adjacent pixels;
if the same pixel belongs to the adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixels of the two connected regions meets the preset condition, combining the two connected regions.
Preferably, calculating the transformation matrix between the roadside camera image coordinates and the satellite image coordinates includes:
and calculating a plurality of conversion matrixes according to the corresponding relation between the positions of the feature points in the first background image and the positions of the feature points in the second background image, so that the conversion matrix with the most feature points is matched as a final conversion matrix.
The invention also provides a dynamic image construction device based on road side image recognition and satellite images, which comprises:
the communication module is used for communicating with the road side camera;
a processing module for:
identifying and tracking vehicles and pedestrians through road side cameras of a road to acquire vehicle movement tracks and pedestrian movement tracks;
overlapping and recursing images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road;
taking a satellite image corresponding to the road background shot in advance as a second background image;
selecting characteristic points in a road, and calculating a conversion matrix between road side camera image coordinates and satellite image coordinates according to the corresponding relation between the positions of the characteristic points in a first background image and the positions of the characteristic points in a second background image;
determining a road area on the second background map according to the vehicle moving track on the road, and combining the road area with the second background map to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
Preferably, the processing module is configured to:
overlapping a preset number of images, and removing image parts which change before and after overlapping to obtain a first background image:
Figure BDA0003840181640000031
wherein ,
Figure BDA0003840181640000032
for the background map determined for the t-th overlay, < >>
Figure BDA0003840181640000033
The background map obtained for the t-1 st overlay,/->
Figure BDA0003840181640000034
For the image shot at the t th road side, n t And alpha is a preset empirical value. For example, α≡0.01, n t ≥70。
Preferably, the processing module is configured to:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is an element in the set;
for each vehicle movement trajectory in the collection:
calculating pixel difference values of boundary pixels and external adjacent pixels in a vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track with the adjacent pixels to serve as a communication area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and merging the adjacent pixels into the connected region if the pixel difference values meet preset conditions;
and if the pixel difference value does not meet the preset condition, determining the boundary of the communication area according to the adjacent pixels.
Preferably, the processing module is configured to:
simultaneously calculating pixel difference values of a plurality of vehicle movement tracks and adjacent pixels;
if the same pixel belongs to the adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixels of the two connected regions meets the preset condition, combining the two connected regions.
Due to the adoption of the technical scheme, the invention has the following advantages:
by adopting the scheme of the invention, the map and the satellite image provided by the road side monitoring equipment can be directly utilized to manufacture the dynamic map, the actual mapping is not needed, the manual calibration is not needed, and the automation degree is high. Compared with a satellite map, the method and the system have the advantages that the satellite map is more structured, real-time dynamic update can be performed, and the automatic driving vehicle can judge and prepare the traffic condition of the monitoring area in advance and optimize the decision under the complex traffic environment by utilizing the scheme provided by the invention.
Drawings
Fig. 1 is a flow chart of a dynamic mapping method based on road side image recognition and satellite image provided by the invention.
Fig. 2 is a schematic diagram of a target movement track perceived by a road side in an example of the present invention.
Fig. 3 is a schematic diagram comparing road side monitoring images and satellite images of the same background in an example of the invention.
Fig. 4 is a schematic diagram of a dynamic map obtained in an example of the invention.
Fig. 5 is a schematic structural diagram of a dynamic mapping device based on road side image recognition and satellite image provided by the invention.
Fig. 6 is a schematic view of an application scenario of the dynamic mapping device based on road side image recognition and satellite image provided by the invention.
Detailed Description
In the drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
In the description of the present invention, the terms "center", "longitudinal", "lateral", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate an orientation or a positional relationship based on that shown in the drawings, only for convenience of description and simplification of the description, and do not indicate or imply that the apparatus or element to be referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the scope of protection of the present invention.
In the case of no conflict, the technical features in the embodiments and the implementation modes of the present invention may be combined with each other, and are not limited to the embodiments or implementation modes where the technical features are located.
The invention will be further described with reference to the drawings and the specific embodiments, it being noted that the technical solution and the design principle of the invention will be described in detail with only one optimized technical solution, but the scope of the invention is not limited thereto.
The following terms are referred to herein, and for ease of understanding, the meaning thereof is described below. It will be understood by those skilled in the art that other names are possible for the following terms, but any other name should be construed to be consistent with the terms set forth herein without departing from their meaning.
The invention provides a dynamic mapping method based on road side image recognition and satellite images, which is shown in fig. 1 and comprises the following steps:
and S10, identifying and tracking the vehicle and the pedestrian through a road side camera of the road, and acquiring a vehicle movement track and a pedestrian movement track.
The method comprises the steps of carrying out target recognition training of vehicles and pedestrians by utilizing a neural network model and data in a database in advance, wherein each recognition result comprises two-dimensional frame information and category information, the two-dimensional frame information is used for indicating the position of the target, and the category information is used for indicating that the target is various vehicles or pedestrians. Then the method is applied to the road side to carry out target identification and tracking, and the track is obtained. Herein, after the target recognition training, a road side camera is used for carrying out target recognition and tracking on vehicles and pedestrians on a road, and a vehicle moving track and a pedestrian moving track are obtained.
And S20, performing superposition recursion on images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road.
The method comprises the steps of superposing a preset number of images by the following formula, removing image parts which change before and after superposition, and obtaining a first background image:
Figure BDA0003840181640000051
wherein ,
Figure BDA0003840181640000052
for the background map determined for the t-th overlay, < >>
Figure BDA0003840181640000053
The background map obtained for the t-1 st overlay,/->
Figure BDA0003840181640000054
For the image shot at the t th road side, n t And alpha is a preset empirical value. For example, α≡0.01, n t ≥70。
Step S30, taking a satellite image corresponding to the road background shot in advance as a second background image.
The corresponding road backgrounds may be the same road background, or may include a part of the same road background.
Step S40, selecting characteristic points in the road, and calculating a conversion matrix between the image coordinates of the road side camera and the satellite image coordinates according to the corresponding relation between the positions of the characteristic points in the first background image and the positions of the characteristic points in the second background image.
The plurality of transformation matrices can be calculated according to the corresponding relation between the position of the feature point in the first background image and the position of the feature point in the second background image, so that the transformation matrix with the most feature point matching is used as a final transformation matrix.
And S50, determining a road area on the second background map according to the movement track of the vehicle, and combining the road area with the second background map to obtain the static map.
Wherein determining the road area on the second background map according to the vehicle movement track includes: acquiring a set of vehicle movement tracks on a road; wherein each vehicle movement track is an element in the set; for each vehicle movement trajectory in the collection: calculating pixel difference values of boundary pixels and external adjacent pixels in a vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track with the adjacent pixels to serve as a communication area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and merging the adjacent pixels into the connected region if the pixel difference values meet preset conditions; and if the pixel difference value does not meet the preset condition, determining the boundary of the communication area according to the adjacent pixels.
Wherein the method further comprises: simultaneously calculating pixel difference values of a plurality of vehicle movement tracks and adjacent pixels; if the same pixel belongs to the adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixels of the two connected regions meets the preset condition, combining the two connected regions.
And step S60, converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
The dynamic mapping method based on road side image recognition and satellite images provided by the invention is described in detail below through an example.
The method provided by the invention can be divided into three parts: the system comprises a road side sensing part, an image calibration part and a dynamic image construction part.
1. Road side sensing part
The road side sensing is realized through the road side camera, the position of the road side camera is generally at the road side and is consistent with the monitoring of a general road, the height of the road side camera is generally 3-5 meters away from the ground, and the depth of the nearest point of observation is 3-5 meters away.
Before the target is identified by the road side camera, a neural network (such as yolov 4) model based on an anchor frame method can be utilized in advance, and a traffic road data set is utilized to carry out road user and pedestrian detection training. Among them, road users include various types of vehicles.
The output result of identifying the targets by the roadside camera is the two-dimensional frame information of all targets in one frame image and the characteristic information such as the category thereof, for example, the two-dimensional frame information comprises (x) b ,y b ,w b ,h b E, c) six information.
wherein ,(xb ,y b ) Representing the center point, w, of the two-dimensional frame b Represents the width of the two-dimensional frame, h b Representing the high of the two-dimensional box sought, e represents the confidence of the prediction, c represents the probability vector of the classification, i.e. whether the class of the object is a pedestrian or a vehicle of various types.
In acquiring two-dimensional frame information, the coordinates of the upper left corner of the frame, expressed as (x c ,y c ) Further, the following relationship is used to obtain:
x b ＝σ(x o )+x c
y b ＝σ(y o )+y c
Figure BDA0003840181640000071
Figure BDA0003840181640000072
wherein sigma is a Sigmoid function, and the value is 0,1]Interval. (x) o ,y o ,w o ,h o ) And predicting the compensation value for the preset model, wherein different categories have different model prediction compensation values.
The six items of information are used as input, a Kalman filtering is used, a multi-target multi-kind tracking algorithm is adopted to track each target, and a moving track set of all target movements can be obtained as follows:
Figure BDA0003840181640000073
Figure BDA0003840181640000074
as shown in FIG. 2, v i And p is as follows i Respectively, a certain vehicle or a pedestrian,
Figure BDA0003840181640000075
and->
Figure BDA0003840181640000076
Representing the movement track of the vehicle or the pedestrian, V and P represent the set of detected tracking IDs of the vehicle and the pedestrian, respectively, wherein each track point comprises the six pieces of information.
2. Image calibration part
The image calibration part comprises background extraction and image coordinate conversion relation establishment.
Background extraction: used as background roads and static obstacles in the map. Which occurs at an initial time and may be updated at a later time.
In this example, the road background in the monitored area is found by recursively superimposing images taken by the roadside cameras, leaving the same portions in the images.
For example, the superimposed relationship is as follows
Figure BDA0003840181640000077
wherein ,
Figure BDA0003840181640000078
for the background obtained by the t-th superposition, n t With alpha being an empirical value, n t Approximately 70, i.e. the background can be found by iterating 70 images, alpha being the weight of the new image, approximately 0.01 +.>
Figure BDA0003840181640000079
Is the monitored image photographed at the t time.
Under the condition that the position of the road side camera is basically unchanged, the background such as a road structure is fixed, the pixels of the road structure are fixed, so that the sum of the pixels of the background part is unchanged no matter what the values of factors (namely alpha and 1-alpha) multiplied by the t time and the t-1 time before, the dynamic object pixels can be refreshed along with the superposition of images, and the images with only the background are finally obtained after a certain times of superposition.
Establishing an image coordinate conversion relation:
the invention relates to an image shot by a road side camera, namely a road side monitoring camera image
Figure BDA00038401816400000710
And satellite image->
Figure BDA00038401816400000711
A fish eye view image +.>
Figure BDA00038401816400000712
The fish-eye view image is substantially identical to the satellite view image in terms of coordinates (i.e. +.>
Figure BDA00038401816400000713
)。
Conversion relation from monitoring camera image to satellite image or fish eye image, conversion matrix can be used
Figure BDA0003840181640000081
Figure BDA0003840181640000082
Is expressed and the relationship is reversible, i.e
Figure BDA0003840181640000083
wherein ,
Figure BDA0003840181640000084
representing coordinates in a satellite image or a fish eye image, < >>
Figure BDA0003840181640000085
Representing coordinates in the monitoring camera image.
The relationship of G is equal to g=k [ r|t ], K is the camera internal matrix, and R and T are the rotation and translation matrices in the camera external matrix.
In this example, G is calculated by means of feature matching.
As shown in fig. 3, in the case of a background-only monitoring image
Figure BDA0003840181640000086
And satellite image->
Figure BDA0003840181640000087
Respectively extracting characteristic points and performing characteristic matching. Two images correspond to the same background, satellite image->
Figure BDA0003840181640000088
Either pre-shot or real-time acquisition. The process comprises the following steps:
selecting characteristic points; the feature point may be an empirical value, such as a road boundary, corner, etc., where it is easily determined.
Regulating the illumination, the tone and the like of the two pictures to the same range, and regulating the illumination, the tone and the like to gray level images to obtain processed characteristic points;
and comparing and matching the feature points of the two processed pictures, and obtaining feature matching points of the two pictures by using an ASIFT (affine invariance matching, affine and Scale Invariant Feature Transform) algorithm. In this example, a RANSAC (random sample consensus ) algorithm may be used to calculate a transformation matrix G from every fourth feature matching point, and then calculate, one by one, whether other matching points match at the same time under the matrix G, and select the transformation matrix G that makes the feature matching point match the most as the final result estimate G. Direct determination of the matching points can also be selected by the naked eye, as this is not a limitation.
After determining the conversion matrix G, converting the image of the monitoring camera through the conversion matrix G, and checking whether the converted image can be overlapped with the satellite image or not; if so, the transformation matrix G is valid, otherwise the transformation matrix G is redetermined.
3. Dynamic map building part
In this example, the SRG (Seeded Region Growing, region growing) algorithm is adopted to collect M of all vehicle movement tracks obtained by the sensing part V As initial seed, wherein the moving track of each vehicle is a seed, and the monitoring image with only background obtained from the last part is adopted
Figure BDA0003840181640000089
As an initial extracted image, the mapping process includes:
calculating the pixel difference between each seed x and the neighbor pixel; for example, an image is first converted into a grayscale image such that the values of the three channels of the image are the same, and then the value of one channel is taken to calculate the pixel-to-pixel difference. In a preferred implementation, a neighbor pixel may include 8 pixels adjacent around a pixel.
If the pixel difference from one neighbor pixel is less than the threshold τ x Dividing the seed and the neighbor pixels into a set as a connected region; then, searching for new neighbor pixels with boundary pixels of the connected region, calculating pixel differences between the boundary pixels and the neighbor pixels, if still smaller than the threshold τ x Merging the new neighbor pixels into the connected region, and continuing to search the next neighbor pixel until the pixel difference between the boundary pixel and the neighbor pixel is not smaller than the threshold tau x The boundary of the connected region is determined from the neighbor pixel. For example, there are neighboring pixels as boundaries, or boundary pixels of connected regions as boundaries.
When a pixel is adjacent to two sets at the same time, the pixel difference values can be calculated respectively, if both are smaller than the threshold value τ x Two sets are then assembledAnd combining, namely combining the two communication areas into one communication area.
Repeating the steps until no new communication area exists, and taking the communication area as a road area.
In this example, the road area may be further refined, and different boundaries of the connected areas may be identified by different lines.
Through the steps, the static map can be obtained. The map can be automatically updated at intervals, namely, the background image is obtained by updating through the image calibration part, and then the updated static map is obtained by calculating through the part.
After the static map is obtained, the information of the real-time dynamic obstacle can be added to obtain the dynamic map, as shown in fig. 4.
Six pieces of information (x b ,y b ,w b ,h b And E, c), after multi-target multi-type tracking, the bounding box position, the speed and the orientation angle of a single target under pixel coordinates can be calculated.
For example, the bounding box position calculation process may include:
taking the bottom midpoint of the two-dimensional bounding box as a reference point, and calculating to obtain the coordinates of the bottom midpoint based on the position information obtained by the road side sensing part:
Figure BDA0003840181640000091
converting the coordinates of the bottom midpoint into the bottom midpoint of the bounding box under the satellite image coordinates by using a conversion matrix G and calibrating a partial formula
Figure BDA0003840181640000092
Figure BDA0003840181640000093
The length and width of the bounding box under the satellite image coordinates can be the empirical value, namely, the empirical value of the object is taken according to the category of the object, and then the object is obtained according to the satellite mapAnd converting the meter-to-pixel ratio value to obtain the length and width of the pixel. The orientation of the bounding box may then be the actual orientation angle obtained by the perceptual tracking.
By adopting the scheme of the invention, the automatic driving vehicle can directly use the map and satellite image provided by the road side monitoring equipment to manufacture the dynamic map, does not need actual mapping or manual calibration, and has high automation degree. Compared with a satellite map, the method has the advantages that the satellite map is more structured, real-time dynamic updating can be performed, the automatic driving vehicle can be guaranteed to judge and prepare the traffic condition of a monitoring area in advance under a complex traffic environment, and the decision is optimized.
The invention provides a dynamic mapping device based on road side image recognition and satellite images, as shown in fig. 5, comprising:
a communication module 51 for communicating with the roadside camera;
a processing module 52 for:
tracking vehicles and pedestrians through road side cameras of a road to acquire vehicle movement tracks and pedestrian movement tracks;
overlapping and recursing images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road;
taking a satellite image corresponding to the road background shot in advance as a second background image;
selecting characteristic points in a road, and calculating a conversion matrix between road side camera image coordinates and satellite image coordinates according to the corresponding relation between the positions of the characteristic points in a first background image and the positions of the characteristic points in a second background image;
determining a road area on the second background map according to the vehicle movement track, and combining the road area with the second background map to obtain a static map;
and converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on the static map to realize the dynamic map.
Wherein the processing module 52 is configured to:
overlapping a preset number of images, and removing image parts which change before and after overlapping to obtain a first background image:
Figure BDA0003840181640000101
wherein ,
Figure BDA0003840181640000102
for the background map determined for the t-th overlay, < >>
Figure BDA0003840181640000103
The background map obtained for the t-1 st overlay,/->
Figure BDA0003840181640000104
For the image shot at the t th road side, n t And alpha is a preset empirical value. For example, α≡0.01, n t ≥70。
Wherein the processing module 52 is configured to:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is an element in the set;
for each vehicle movement trajectory in the collection:
calculating pixel difference values of boundary pixels and external adjacent pixels in a vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track with the adjacent pixels to serve as a communication area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and merging the adjacent pixels into the connected region if the pixel difference values meet preset conditions;
and if the pixel difference value does not meet the preset condition, determining the boundary of the communication area according to the adjacent pixels.
Wherein the processing module 52 is configured to:
simultaneously calculating pixel difference values of a plurality of vehicle movement tracks and adjacent pixels;
if the same pixel belongs to the adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixels of the two connected regions meets the preset condition, combining the two connected regions.
Fig. 6 is a schematic view of an application scenario of the dynamic mapping device based on road side image recognition and satellite image provided by the invention. As shown in fig. 6, the scene includes a map calculation device 61, a roadside camera 62, and an autonomous vehicle 63.
The map calculation means 61 comprises a communication gateway device 611, a V2X (Vehicle-to-evaluation) device 612 and a processor 613.
The communication gateway device 611 is configured to exchange information such as an image captured by a roadside camera with the roadside camera (or other roadside devices), and receive a satellite image. For example, the connection between the roadside apparatus and the communication gateway apparatus 611 may be achieved by USB (Universal Serial Bus )/Ethernet (Ethernet).
The V2X device 612 is used to communicate with an autonomous vehicle.
The processor 613 includes three modules: the target sensing module is used for detecting targets of images obtained by the camera, the image calibration module is used for obtaining a static map and a conversion matrix between the images obtained by the camera and satellite images, and the dynamic mapping module is used for calculating a real-time dynamic map in the monitoring area.
The road side camera 62 is connected to the computing unit by USB/ethernet for obtaining real-time monitoring road images.
The autonomous vehicle 63 is connected to the map calculation device 61 by wire or wirelessly, and receives the dynamic map of the map calculation device 61.
The map calculation device 61 may also be integrated into the autonomous vehicle 63, so that the autonomous vehicle 63 can perform calculations to obtain a dynamic map.
By adopting the scheme of the invention, the automatic driving vehicle can directly use the map and satellite image provided by the road side monitoring equipment to manufacture the dynamic map, does not need actual mapping or manual calibration, and has high automation degree. Compared with a satellite map, the method has the advantages that the satellite map is more structured, real-time dynamic updating can be performed, the automatic driving vehicle can be guaranteed to judge and prepare the traffic condition of a monitoring area in advance under a complex traffic environment, and the decision is optimized.
Finally, it should be pointed out that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting. Those of ordinary skill in the art will appreciate that: the technical schemes described in the foregoing embodiments may be modified or some of the technical features may be replaced equivalently; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (8)

1. A dynamic mapping method based on road side image recognition and satellite images is characterized by comprising the following steps:
identifying and tracking vehicles and pedestrians through road side cameras of a road to acquire vehicle movement tracks and pedestrian movement tracks;
overlapping and recursing images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road;
taking a satellite image corresponding to the road background shot in advance as a second background image;
selecting characteristic points in a road, and calculating a conversion matrix between road side camera image coordinates and satellite image coordinates according to the corresponding relation between the positions of the characteristic points in a first background image and the positions of the characteristic points in a second background image;
determining a road area on the second background map according to the vehicle moving track on the road, and combining the road area with the second background map to obtain a static map;
converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on a static map to realize a dynamic map;
determining the road area on the second background map according to the vehicle movement track comprises:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is an element in the set;
for each vehicle movement trajectory in the collection:
calculating pixel difference values of boundary pixels and external adjacent pixels in a vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track with the adjacent pixels to serve as a communication area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and merging the adjacent pixels into the connected region if the pixel difference values meet preset conditions; and if the pixel difference value does not meet the preset condition, determining the boundary of the communication area according to the adjacent pixels.
2. The method of claim 1, wherein identifying and tracking vehicles and pedestrians through roadside cameras of a roadway comprises:
the method comprises the steps that target recognition training of vehicles and pedestrians is carried out by utilizing a neural network model and data in a database in advance, each recognition result comprises two-dimensional frame information and category information, wherein the two-dimensional frame information is used for indicating the position of a target, and the category information is used for indicating that the target is various vehicles or pedestrians;
and after the target recognition training, a road side camera is used for carrying out target recognition and tracking on vehicles and pedestrians on the road, and a vehicle moving track and a pedestrian moving track are obtained.
3. The method of claim 1, wherein performing a superposition recursion on images captured by the roadside camera to extract the same portion of the preset number of images as a first background map of the road comprises:
overlapping a preset number of images, and removing image parts which change before and after overlapping to obtain a first background image:
Figure FDA0004096680260000021
wherein ,
Figure FDA0004096680260000022
for the background map determined for the t-th overlay, < >>
Figure FDA0004096680260000023
The background map obtained for the t-1 st overlay,/->
Figure FDA0004096680260000024
For the image shot at the t th road side, n t And alpha is a preset empirical value.
4. The method as recited in claim 1, further comprising: simultaneously calculating pixel difference values of a plurality of vehicle movement tracks and adjacent pixels;
if the same pixel belongs to the adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixels of the two connected regions meets the preset condition, combining the two connected regions.
5. The method of claim 1, wherein calculating a transformation matrix between roadside camera image coordinates and satellite image coordinates comprises:
and calculating a plurality of conversion matrixes according to the corresponding relation between the positions of the feature points in the first background image and the positions of the feature points in the second background image, so that the conversion matrix with the most feature points is matched as a final conversion matrix.
6. The dynamic mapping device based on road side image recognition and satellite image is characterized by comprising:
the communication module is used for communicating with the road side camera;
a processing module for:
identifying and tracking vehicles and pedestrians through road side cameras of a road to acquire vehicle movement tracks and pedestrian movement tracks;
overlapping and recursing images shot by the road side cameras, and extracting the same parts in the images with the preset quantity as a first background image of the road;
taking a satellite image corresponding to the road background shot in advance as a second background image;
selecting characteristic points in a road, and calculating a conversion matrix between road side camera image coordinates and satellite image coordinates according to the corresponding relation between the positions of the characteristic points in a first background image and the positions of the characteristic points in a second background image;
determining a road area on the second background map according to the vehicle moving track on the road, and combining the road area with the second background map to obtain a static map;
converting the dynamic position information of the vehicle and the dynamic position information of the pedestrian monitored by the road side camera into satellite image coordinates, and displaying the dynamic positions of the vehicle and the pedestrian on a static map to realize a dynamic map;
wherein, processing module is used for:
acquiring a set of vehicle movement tracks; wherein each vehicle movement track is an element in the set;
for each vehicle movement trajectory in the collection:
calculating pixel difference values of boundary pixels and external adjacent pixels in a vehicle moving track, and if the pixel difference values meet preset conditions, communicating the vehicle moving track with the adjacent pixels to serve as a communication area; calculating pixel difference values of boundary pixels of the connected region and adjacent pixels of the non-connected region, and merging the adjacent pixels into the connected region if the pixel difference values meet preset conditions; and if the pixel difference value does not meet the preset condition, determining the boundary of the communication area according to the adjacent pixels.
7. The apparatus of claim 6, wherein the processing module is to:
overlapping a preset number of images, and removing image parts which change before and after overlapping to obtain a first background image:
Figure FDA0004096680260000031
wherein ,
Figure FDA0004096680260000032
for the background map determined for the t-th overlay, < >>
Figure FDA0004096680260000033
The background map obtained for the t-1 st overlay,/->
Figure FDA0004096680260000034
For the image shot at the t th road side, n t And alpha is a preset empirical value.
8. The apparatus of claim 6, wherein the processing module is to:
simultaneously calculating pixel difference values of a plurality of vehicle movement tracks and adjacent pixels;
if the same pixel belongs to the adjacent pixels of the two connected regions and the pixel difference between the same pixel and the boundary pixels of the two connected regions meets the preset condition, combining the two connected regions.
CN202211103145.5A 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image Active CN115390116B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211103145.5A CN115390116B (en) 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211103145.5A CN115390116B (en) 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image

Publications (2)

Publication Number Publication Date
CN115390116A CN115390116A (en) 2022-11-25
CN115390116B true CN115390116B (en) 2023-05-12

Family

ID=84125846

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211103145.5A Active CN115390116B (en) 2022-09-09 2022-09-09 Dynamic mapping method and device based on road side image recognition and satellite image

Country Status (1)

Country Link
CN (1) CN115390116B (en)

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101109647A (en) * 2007-08-22 2008-01-23 广东瑞图万方科技有限公司 Navigation apparatus
US20120230550A1 (en) * 2011-03-10 2012-09-13 Jacob Kraut Method and Apparatus for Generating a Map from Landmarks
CN108288014A (en) * 2017-01-09 2018-07-17 北京四维图新科技股份有限公司 Intelligent road extracting method and device, extraction model construction method and hybrid navigation system
CN107180215B (en) * 2017-05-31 2020-01-31 同济大学 Parking lot automatic mapping and high-precision positioning method based on library position and two-dimensional code
CN112308904A (en) * 2019-07-29 2021-02-02 北京初速度科技有限公司 Vision-based drawing construction method and device and vehicle-mounted terminal
CN110717007A (en) * 2019-10-15 2020-01-21 财团法人车辆研究测试中心 Map data positioning system and method applying roadside feature identification
CN111391823A (en) * 2019-12-27 2020-07-10 湖北亿咖通科技有限公司 Multilayer map making method for automatic parking scene
CN113156461B (en) * 2021-05-17 2021-11-05 紫清智行科技(北京)有限公司 Dynamic loading method and system for 2D laser radar SLAM map
CN113865581A (en) * 2021-09-16 2021-12-31 江苏大学 Closed scene positioning method based on multi-level map

Also Published As

Publication number Publication date
CN115390116A (en) 2022-11-25

Similar Documents

Publication Publication Date Title
CN111220993B (en) Target scene positioning method and device, computer equipment and storage medium
CN107767400B (en) Remote sensing image sequence moving target detection method based on hierarchical significance analysis
CN109598794B (en) Construction method of three-dimensional GIS dynamic model
CN109099929B (en) Intelligent vehicle positioning device and method based on scene fingerprints
JP5180126B2 (en) Road recognition device
CN114022830A (en) Target determination method and target determination device
US10996337B2 (en) Systems and methods for constructing a high-definition map based on landmarks
CN112308913B (en) Vehicle positioning method and device based on vision and vehicle-mounted terminal
CN115032651A (en) Target detection method based on fusion of laser radar and machine vision
CN114120283A (en) Method for distinguishing unknown obstacles in road scene three-dimensional semantic segmentation
CN112800938B (en) Method and device for detecting occurrence of side rockfall of unmanned vehicle
CN111723778B (en) Vehicle distance measuring system and method based on MobileNet-SSD
JP4762026B2 (en) Road sign database construction device
CN117333846A (en) Detection method and system based on sensor fusion and incremental learning in severe weather
CN115390116B (en) Dynamic mapping method and device based on road side image recognition and satellite image
CN112150448A (en) Image processing method, device and equipment and storage medium
CN114170499A (en) Target detection method, tracking method, device, visual sensor and medium
CN112017259A (en) Indoor positioning and image building method based on depth camera and thermal imager
CN115661453B (en) Tower crane object detection and segmentation method and system based on downward view camera
WO2020113425A1 (en) Systems and methods for constructing high-definition map
CN114742955A (en) Flood early warning method and device, electronic equipment and storage medium
CN114152955A (en) High-precision obstacle identification system based on SLAM technology
CN114972470A (en) Road surface environment obtaining method and system based on binocular vision
CN115131360A (en) Road pit detection method and device based on 3D point cloud semantic segmentation
CN111833384B (en) Method and device for rapidly registering visible light and infrared images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant