CN110377670B

CN110377670B - Method, device, medium and equipment for determining road element information

Info

Publication number: CN110377670B
Application number: CN201810322044.4A
Authority: CN
Inventors: 薛涛; 郭勤振; 汤蓉
Original assignee: Tencent Technology Shenzhen Co Ltd; Tencent Dadi Tongtu Beijing Technology Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd; Tencent Dadi Tongtu Beijing Technology Co Ltd
Priority date: 2018-04-11
Filing date: 2018-04-11
Publication date: 2021-11-26
Anticipated expiration: 2038-04-11
Also published as: CN110377670A

Abstract

The invention belongs to the technical field of electronic maps, and provides a method, a device, a medium and equipment for determining road element information, wherein in the technical scheme provided by the invention, a first type of road elements and first type of road element information of a road image and a second type of road elements and second type of road element information of the road image are obtained, and the road element information comprises the confidence coefficient of the road elements and the positions of the road elements in the road image; determining the same road elements between the first type of road elements and the second type of road elements; if the confidence of the same road element in the first-type road element information is greater than a first threshold and the confidence in the second-type road element information is less than a second threshold, the confidence of the same road element in the first-type road element information and the position of the same road element in the second-type road element information are determined as the road element information of the same road element. The invention can obtain more accurate road element information.

Description

Method, device, medium and equipment for determining road element information

Technical Field

The present application relates to the field of electronic map technologies, and in particular, to a method, an apparatus, a medium, and a device for determining road element information.

Background

With the development of road construction, the electronic map data needs to be updated in real time to keep the real-time performance and accuracy of the electronic map data. The electronic map data includes determination of road element information, and at present, the following ways are often adopted to determine the road element information:

acquiring images on a road at intervals of a set distance by using image acquisition equipment; and taking the collected images as input of a deep learning target detection algorithm to obtain road elements and road element information contained in each image output by the algorithm, wherein the road element information comprises confidence degrees of the road elements and positions of the road elements in the images.

At the present stage, the deep learning single target detection algorithm is mostly used for determining the road element information, however, the road element information obtained by the deep learning single target detection algorithm is not accurate enough, so how to improve the accuracy of the road element information is a problem to be considered.

Disclosure of Invention

The application provides a method, a device, a medium and equipment for determining road element information, which are used for solving the problem that the road element information obtained by utilizing a deep learning single target detection algorithm is not accurate enough in the prior art.

In one aspect, an embodiment of the present application provides a method for determining road element information, including:

acquiring first type road element information and first type road element information of the road image determined by a rapid detection algorithm and second type road element information of the road image determined by a precise detection algorithm aiming at each determined road image, wherein the road element information comprises the confidence coefficient of the road element and the position of the road element in the road image; determining road elements of the first type that are identical to road elements of the second type; for each identical road element, if the confidence of the identical road element in the first-type road element information is greater than a first threshold and the confidence of the identical road element in the second-type road element information is less than a second threshold, determining the confidence of the identical road element in the first-type road element information and the position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, wherein the first threshold is not less than the second threshold. In the embodiment of the application, when the road element determined by the rapid detection algorithm and the road element determined by the accurate detection algorithm are the same road element, the confidence of the same road element determined by the rapid detection algorithm and the position of the road element in the image determined by the accurate detection algorithm are used as the road element information of the same road element, so that more accurate road element information is obtained.

In one possible embodiment, the road image is determined in the following manner: according to the acquisition positions of the candidate road images, performing hierarchical clustering on the candidate road images to obtain first-level classification results for the candidate road images, wherein the actual distance between the candidate road images adjacent to the acquisition positions in each first-level classification is not greater than a preset distance; aiming at each obtained first-level classification, if the road elements between the candidate road images adjacent to the acquisition time in the first-level classification are determined to be similar, dividing the candidate road images adjacent to the acquisition time into the same classification to obtain a second-level classification result; and respectively selecting one candidate road image from each second-level classification as the road image. The implementation mode can greatly reduce the number of road images input into the accurate detection algorithm, and further improve the processing speed of the road elements determined by the accurate detection algorithm to a certain extent.

As a possible implementation, before determining the road image, the method further includes: using each collected original image as the input of the rapid detection algorithm; and determining the original image containing the road elements output by the rapid detection algorithm as a candidate road image. The embodiment can ensure that all candidate road images are original images containing road elements, so that when the road element information of the road images is determined, the road images without the road elements can be prevented from being processed, and the efficiency of determining the road element information is improved.

As a possible implementation manner, determining that the road elements between the candidate road images adjacent to each other in the collection time in the first-level classification are similar specifically includes: aiming at candidate road images adjacent to the collection time, respectively obtaining a first rectangular detection frame corresponding to a first road element of the candidate road image close to the collection time, and a second rectangular detection frame corresponding to a second road element in the candidate road image close to the collection time; and if the length-width ratio deviation of the first rectangular detection frame and the second rectangular detection frame is within a preset range, the first rectangular detection frame is smaller than the second rectangular detection frame, and the image characteristics of the image area corresponding to the first rectangular detection frame are similar to the image characteristics of the image area corresponding to the second rectangular detection frame, determining that the first road element is similar to the second road element. In this embodiment, the similar road elements are determined by combining the variation in the length-width ratio of the rectangular detection frames of different road elements, the size of the rectangular detection frame, and the image characteristics of the image area included in the rectangular detection frame, so that the determined similar road elements can be more accurately determined.

As a possible implementation manner, selecting one candidate road image from any second-level classification specifically includes: for each candidate road image in any second-level classification, if the rectangular detection frame corresponding to each road element of the candidate road image is completely contained in the candidate road image, determining that the candidate road image is a complete road image; and selecting a complete road image from the determined complete road images according to the image definition and the image size. This embodiment combines image integrity, image sharpness, and image size to select candidate road images from the second level of classification, thereby ensuring that relatively superior candidate road images are selected.

As a possible implementation manner, selecting one complete road image from the determined complete road images specifically includes: determining the weighted sum result of the image definition and the image size of each determined complete road image as the weight of the complete road image; and selecting one complete road image with the largest weight from the determined complete road images.

As a possible embodiment, the image sharpness is weighted less than the image size.

As a possible implementation, determining the same road element between the first road element and the second road element specifically includes: determining, for each road element of the first class of road elements, a road element of the second class of road elements having an overlap region with a position of the road element in the road image as an overlapping road element; and if the overlapping area of the road element and the overlapping road element is larger than the area threshold value, determining that the road element and the overlapping road element are the same road element.

As a possible implementation manner, the determining the second type road element and the second type road element information of the road image by using a precise detection algorithm specifically includes: intercepting the first type of road elements from the road image to obtain at least one screenshot corresponding to the road image, wherein the screenshot comprises at least one road element in the first type of road elements; taking each screenshot as the input of the accurate detection algorithm to obtain the road element and the road element information of the corresponding screenshot output by the accurate detection algorithm; for each screenshot, mapping the position of the road element in the screenshot in the road element information of the screenshot into the position of the road element in the road image; and taking the confidence coefficient of the road element in the road element information of each screenshot and the position of the corresponding road element in the road image as the second type road element information. According to the possible implementation mode, on one hand, the condition that each screenshot contains the road elements is guaranteed to avoid the condition that the screenshot of the input accurate detection algorithm does not contain the road elements, so that the processing load of the accurate detection algorithm is reduced to a certain degree, on the other hand, the screenshot is used as the input of the accurate detection algorithm, the size of the image of the input accurate detection algorithm is reduced, and the processing speed of the accurate detection algorithm for determining the road element information is improved to a certain degree.

As a possible implementation manner, the step of intercepting the first type of road element from the road image specifically includes: determining a first minimum bounding rectangle containing all road elements in the first type of road elements; if the length of the first minimum circumscribed rectangle is not greater than a length threshold value or the width of the first minimum circumscribed rectangle is not greater than a width threshold value, a screenshot including all road elements in the first type of road elements is intercepted from the road image, wherein the length of the screenshot is not less than the length threshold value, and the width of the screenshot is not less than the width threshold value.

As a possible implementation manner, if the length of the first minimum bounding rectangle is greater than the length threshold or the width of the first minimum bounding rectangle is greater than the width threshold, grouping the road elements in the first class of road elements to obtain a plurality of groups, wherein the length of a second minimum bounding rectangle containing all the road elements in each group is equal to the length threshold, and the width of the second minimum bounding rectangle is equal to the width threshold; for each grouping, a screenshot is taken from the road image that includes all of the road elements in the grouping.

In another aspect, an embodiment of the present application provides an apparatus for determining road element information, including:

the acquisition module is used for acquiring first road element information and first road element information of the road image determined by using a rapid detection algorithm and second road element information of the road image determined by using an accurate detection algorithm aiming at each determined road image, wherein the road element information comprises the confidence coefficient of the road element and the position of the road element in the road image; a first determining module for determining the same road elements between the first type of road elements and the second type of road elements; a second determining module, configured to determine, for each identical road element, if a confidence of the identical road element in the first-type road element information is greater than a first threshold and a confidence of the identical road element in the second-type road element information is less than a second threshold, the confidence of the identical road element in the first-type road element information and a position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, where the first threshold is not less than the second threshold.

As a possible implementation manner, the apparatus for determining road element information provided in an embodiment of the present application further includes: a third determining module for determining the road image in the following way: according to the acquisition positions of the candidate road images, performing hierarchical clustering on the candidate road images to obtain first-level classification results for the candidate road images, wherein the actual distance between the candidate road images adjacent to the acquisition positions in each first-level classification is not greater than a preset distance; aiming at each obtained first-level classification, if the road elements between the candidate road images adjacent to the acquisition time in the first-level classification are determined to be similar, dividing the candidate road images adjacent to the acquisition time into the same classification to obtain a second-level classification result; and respectively selecting one candidate road image from each second-level classification as the road image.

As a possible implementation manner, the apparatus for determining road element information provided in an embodiment of the present application further includes:

the fourth determining module is used for taking each collected original image as the input of the rapid detection algorithm before the third determining module determines the road image; and determining the original image containing the road elements output by the rapid detection algorithm as a candidate road image.

As a possible implementation manner, the third determining module is specifically configured to: aiming at candidate road images adjacent to the collection time, respectively obtaining a first rectangular detection frame corresponding to a first road element of the candidate road image close to the collection time, and a second rectangular detection frame corresponding to a second road element in the candidate road image close to the collection time; and if the length-width ratio deviation of the first rectangular detection frame and the second rectangular detection frame is within a preset range, the first rectangular detection frame is smaller than the second rectangular detection frame, and the image characteristics of the image area corresponding to the first rectangular detection frame are similar to the image characteristics of the image area corresponding to the second rectangular detection frame, determining that the first road element is similar to the second road element.

As a possible implementation, the third determining module is specifically configured to select one candidate road image from any of the second-level classifications according to the following manner: for each candidate road image in any second-level classification, if the rectangular detection frame corresponding to each road element of the candidate road image is completely contained in the candidate road image, determining that the candidate road image is a complete road image; and selecting a complete road image from the determined complete road images according to the image definition and the image size.

As a possible implementation manner, the third determining module is specifically configured to select one complete road image from the determined complete road images in the following manner: determining the weighted sum result of the image definition and the image size of each determined complete road image as the weight of the complete road image; and selecting one complete road image with the largest weight from the determined complete road images.

As a possible implementation manner, the first determining module is specifically configured to: determining, for each road element of the first class of road elements, a road element of the second class of road elements having an overlap region with a position of the road element in the road image as an overlapping road element; and if the overlapping area of the road element and the overlapping road element is larger than the area threshold value, determining that the road element and the overlapping road element are the same road element.

As a possible implementation manner, the apparatus for determining road element information provided in an embodiment of the present application further includes: the fifth determining module is used for determining the second type road elements and the second type road element information of the road image by using an accurate detection algorithm according to the following modes: intercepting the first type of road elements from the road image to obtain at least one screenshot corresponding to the road image, wherein the screenshot comprises at least one road element in the first type of road elements; taking each screenshot as the input of the accurate detection algorithm to obtain the road element and the road element information of the corresponding screenshot output by the accurate detection algorithm; for each screenshot, mapping the position of the road element in the screenshot in the road element information of the screenshot into the position of the road element in the road image; and taking the confidence coefficient of the road element in the road element information of each screenshot and the position of the corresponding road element in the road image as the second type road element information.

As a possible implementation manner, the fifth determining module is specifically configured to intercept the first type road element from the road image in the following manner: determining a first minimum bounding rectangle containing all road elements in the first type of road elements; if the length of the first minimum circumscribed rectangle is not greater than a length threshold value or the width of the first minimum circumscribed rectangle is not greater than a width threshold value, a screenshot including all road elements in the first type of road elements is intercepted from the road image, wherein the length of the screenshot is not less than the length threshold value, and the width of the screenshot is not less than the width threshold value.

As a possible implementation, the fifth determining module is further configured to: if the length of the first minimum bounding rectangle is larger than the length threshold or the width of the first minimum bounding rectangle is larger than the width threshold, grouping the road elements in the first type of road elements to obtain a plurality of groups, wherein the length of a second minimum bounding rectangle containing all the road elements in each group is equal to the length threshold, and the width of the second minimum bounding rectangle is equal to the width threshold; for each grouping, a screenshot is taken from the road image that includes all of the road elements in the grouping.

In yet another aspect, the present application provides a non-volatile computer storage medium, wherein the computer storage medium stores an executable program, and the executable program is executed by a processor to implement any of the method for determining road element information provided in the foregoing embodiments.

In another aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored on the memory, where the processor implements the steps of any one of the methods for determining road element information provided in the foregoing embodiments when executing the computer program.

The method, the device, the medium and the equipment for determining the road element information are provided by the embodiment of the application, wherein a rapid detection algorithm and an accurate detection algorithm belong to a deep learning multi-target detection algorithm, and the advantages of high confidence degree of the road element determined by the rapid detection algorithm and high position accuracy of the road element determined by the accurate detection algorithm in an image are considered.

Drawings

Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application;

fig. 2 is a schematic flowchart of a method for determining road element information according to an embodiment of the present disclosure;

fig. 3 is a schematic diagram of an overlapping area of two road elements in an image according to an embodiment of the present application;

FIG. 4 is a schematic flowchart of a method for determining a road image according to an embodiment of the present disclosure;

FIG. 5 is a flowchart illustrating a method for determining similarity of road elements between candidate road images according to an embodiment of the present disclosure;

FIG. 6 is a flowchart illustrating a method for selecting a candidate road image from a second-level classification according to an embodiment of the present application;

FIG. 7 is a schematic flowchart of a method for selecting a complete road image according to an embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating a method for determining a second type of road element and second type of road element information of a road image according to an embodiment of the present application;

fig. 9a is a schematic position diagram of a road element in a screenshot provided in the embodiment of the present application;

fig. 9b is a schematic position diagram of a screenshot provided in an embodiment of the present application in a road image;

fig. 9c is a schematic diagram of a position of a road element in a road image in the screenshot provided in the embodiment of the present application;

FIG. 10 is a flowchart illustrating a method for capturing a first type of road element from a road image according to an embodiment of the present application;

fig. 11 is a flowchart illustrating a method for grouping road elements according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of an apparatus for determining road element information according to an embodiment of the present application;

fig. 13 is a schematic hardware configuration diagram of a computer device for determining road element information according to an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Reference herein to "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

In this document, it is to be understood that any number of elements in the figures are provided by way of illustration and not limitation, and any nomenclature is used for differentiation only and not in any limiting sense. For convenience of understanding, terms referred to in the embodiments of the present application are explained below.

Road elements: the information in the road for identification may include traffic lights, traffic signs, landmark lines, zebra crossings, and the like.

Road element information: the confidence level of the road element is used for representing the credibility of the determined road element, for example, the confidence level of a traffic light is the credibility of the determined road element being the traffic light; the position of the road element in the image is the position of the rectangular detection frame corresponding to the determined road element in the image.

And (3) a quick detection algorithm: the method belongs to a deep learning multi-target Detection algorithm, can detect a plurality of targets in an image and determine the confidence coefficient of each target and the position of each target in the image, has high Detection speed, ensures that the confidence coefficient of the determined target is reliable, but has poor accuracy of the position of the determined target in the image, and commonly used rapid Detection algorithms comprise yolo v1 (yolo algorithm of a first version), yolo v2 (yolo algorithm of a second version) and the like, wherein yolo is an abbreviation of You Only Look Once, the yolo algorithm is a Unified Real-Time target Detection algorithm (Unifield, Real-Time Object Detection), yolo v2 is better, faster and stronger than yolo v1, and when the rapid Detection algorithm is applied to the embodiment of the invention, a road element is a target to be detected.

And (3) accurate detection algorithm: the method belongs to a deep learning multi-target detection algorithm, can detect a plurality of targets in an image and determine the confidence coefficient of each target and the position of each target in the image, the determined position of each target in the image has high accuracy but low detection speed, and the commonly used precise detection algorithm comprises the following steps: R-CNN, Faster R-CNN, SSD, etc., wherein the English of R-CNN is all called as: the Chinese characters of Regions with connected neural network features are all called as the accurate target detection method based on the characteristics of the regional convolutional neural network, and the English characters of the Faster R-CNN are all called as: the Chinese language of fast Regions with connected neural networks features, fast R-CNN, is called: a rapid accurate target detection method based on the characteristics of a regional convolutional neural network is disclosed, wherein the SSD is called as follows: the Single Shot MultiBox Detector, SSD, is called in chinese: when the accurate detection algorithm of the single deep neural network multi-frame target detector is applied to the embodiment of the invention, the road elements are targets needing to be detected.

Image characteristics: the image itself features, such as a color histogram of the image or image texture features.

At present, road elements and road element information of an image are often determined by using a rapid detection algorithm or an accurate detection algorithm, and the inventor finds that the accuracy of the positions of the road elements in the image obtained by using the rapid detection algorithm is poor, and the confidence coefficient of the road elements determined by using the accurate detection algorithm is lower than the reliability of the confidence coefficient of the road elements determined by using the rapid detection algorithm, which all cause the problem of inaccurate road element information, and the map data obtained by the inaccurate road element information is directly unreliable, so an improved scheme for determining the road element information is needed to obtain the accurate road element information.

To this end, an embodiment of the present application provides a method for determining road element information, which may include: acquiring first type road element information and first type road element information of the road image determined by a rapid detection algorithm and second type road element information of the road image determined by a precise detection algorithm aiming at each determined road image, wherein the road element information comprises the confidence coefficient of the road element and the position of the road element in the road image; determining the same road elements between the first type of road elements and the second type of road elements; for each identical road element, if the confidence of the identical road element in the first-type road element information is greater than a first threshold value and the confidence of the identical road element in the second-type road element information is less than a second threshold value, determining the confidence of the identical road element in the first-type road element information and the position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, wherein the first threshold value is not less than the second threshold value.

In the method for determining road element information provided by the embodiment of the application, considering the advantages that the confidence of the road element determined by using the rapid detection algorithm is reliable and the position accuracy of the road element determined by using the accurate detection algorithm in the image is high, when the road element determined by using the rapid detection algorithm and the road element determined by using the accurate detection algorithm are the same road element, the confidence of the same road element determined by using the rapid detection algorithm and the position of the road element determined by using the accurate detection algorithm in the image are used as the road element information of the same road element, so that the more accurate road element information is obtained.

The following describes a scheme for determining road element information according to an embodiment of the present application with reference to an application scenario provided in fig. 1.

As shown in fig. 1, the system comprises a database server 101 and a computing server 102. The database server 101 stores therein first-type road elements and first-type road element information determined by a fast detection algorithm, and second-type road elements and second-type road element information determined by a precise detection algorithm. The calculation server 102 acquires a first type road element and first type road element information of a road image, and a second type road element and second type road element information of the road image from the database server 101; determining the same road elements between the first type of road elements and the second type of road elements; for each identical road element, if the confidence of the identical road element in the first-type road element information is greater than a first threshold value and the confidence of the identical road element in the second-type road element information is less than a second threshold value, determining the confidence of the identical road element in the first-type road element information and the position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, wherein the first threshold value is not less than the second threshold value. The calculation server 102 may transmit the determined road element information of the same road element to the client terminal 103 for display, and the related art person may view the determined road element information of the same road element through the client terminal 103.

In fig. 1, the database server 101 and the computing server 102 may communicate via a communication network such as a local area network, a wide area network, or a mobile internet, and the computing server 102 and the client 103 may communicate via a communication network such as a local area network, a wide area network, or a mobile internet. The database server 101, the server apparatus 102, and the client 103 may be portable devices (e.g., mobile phones, tablet computers, notebook computers, etc.) or Personal Computers (PCs).

As another application scenario, the database server and the computing server in fig. 1 may be the same computing device, and the computing device is a computing device including a display screen, and at this time, a technician may view the road element information of the determined road element through the display screen of the computing device, that is, the application scenario may not include a separate client, but integrate the function of the client for displaying the road element information on the computing device.

It should be noted that the above-mentioned application scenarios are merely illustrated for the convenience of understanding the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. Rather, the embodiments of the present application may be applied to any applicable scenario.

The following describes a method for determining road element information according to an embodiment of the present application with reference to fig. 2.

As shown in fig. 2, a method for determining road element information according to an embodiment of the present application may include the following steps:

step 201, for each determined road image, acquiring a first type of road element and first type of road element information of the road image determined by a rapid detection algorithm, and acquiring a second type of road element and second type of road element information of the road image determined by a precise detection algorithm, wherein the road element information includes a confidence coefficient of the road element and a position of the road element in the road image.

In the specific implementation, the road elements and the road element information of the road image are determined in advance by using a fast detection algorithm and a precise detection algorithm, wherein the road elements of the road image determined by using the fast detection algorithm are referred to as first-type road elements, the road element information of the road image determined by using the fast detection algorithm is referred to as first-type road element information, the road elements of the road image determined by using the precise detection algorithm are referred to as second-type road elements, and the road element information of the road image determined by using the precise detection algorithm is referred to as second-type road element information. The first type road element comprises at least one road element, and the first type road element information comprises the confidence coefficient of each road element in the first type road element and the position of each road element in the road image; the second type road element comprises at least one road element, and the second type road element information comprises the confidence coefficient of each road element in the second type road elements and the position of each road element in the road image.

The way of determining the first kind of road elements and the first kind of road element information of the road image by using the rapid detection algorithm is as follows: training a rapid detection model in advance; carrying out reduction processing on the road image; and performing non-maximum inhibition processing on the output of the rapid detection model to obtain first-class road elements and first-class road element information of the road image, wherein the step of performing reduction processing on the road image is to reduce the road image to the precision capable of ensuring recall of the plurality of road elements.

The way of determining the second type road elements of the road image and the second type road element information by using the precise detection algorithm is as follows: training an accurate detection model in advance; and taking the road image as the input of a pre-trained accurate detection model to obtain the output of the accurate detection model, and performing fusion and non-maximum inhibition processing on the output of the accurate detection model to obtain a second type road element and second type road element information of the road image.

It should be noted that the road image is an image captured at different positions of a preset area of the road by using a capturing device, for example, the capturing device is mounted on a capturing vehicle, and an image in front of the capturing vehicle is captured during the movement of the capturing vehicle.

Step 202, the same road elements between the first type of road elements and the second type of road elements are determined.

In specific implementation, the first-type road element and the second-type road element are road elements of the same road image determined by using different detection algorithms, and therefore, the possibility that the same road element exists between the first-type road element and the second-type road element is high, for example, if the road element of the road image includes a traffic light 1, the first-type road element includes a traffic light 1, and the second-type road element also includes a traffic light 1, the traffic light 1 is considered as the same road element between the first-type road element and the second-type road element.

In practical application, the image characteristics of each road element in the first type of road elements and the image characteristics of each road element in the second type of road elements can be determined, and two road elements with similar image characteristics are determined as the same road element, wherein one road element belongs to the first type of road elements and the other road element belongs to the second type of road elements. The same road elements between the first type road elements and the second type road elements may also be determined in other ways, which are not limited herein.

Alternatively, the same road elements between the first type of road element and the second type of road element are determined in the following manner:

determining, for each road element of the first class of road elements, a road element of the second class of road elements having an overlap region with a position of the road element in the road image as an overlapping road element; and if the overlapping area of the road element and the overlapping road element is larger than the area threshold value, determining that the road element and the overlapping road element are the same road element.

In a specific embodiment, a road element of the first type of road element is referred to as a road element a, a road element of the second type of road element is referred to as a road element B, and a road element B of the second type of road element having an overlapping area with a position of the road element a in the road image is specified for each road element a, and the overlapping road element belongs to the second type of road element. The overlapping area is the overlapping area of the rectangular detection frame corresponding to the road element A and the rectangular detection frame corresponding to the road element B in the road image. The region threshold may be equal to a set ratio of the area of the rectangular detection frame corresponding to the road element, such as 80% of the area of the rectangular detection frame corresponding to the road element, where the road element may be the first type road element or the second type road element.

In a specific implementation, when the overlapping area of the positions of the rectangular detection frames corresponding to two road elements in the same road image is greater than the area threshold, the two road elements are determined to be the same road element, wherein one road element (road element a) of the two road elements belongs to the first type of road element, and the other road element (road element B) belongs to the second type of road element. As shown in fig. 3, an overlapping area of the position of the road element a in the first type road element in the road image and the position of the road element B in the second type road element in the image is an area C, and at this time, if the area C is greater than an area threshold, it is determined that the road element a and the road element B are the same road element, for example, both the road element a and the road element B are traffic light 1, otherwise, it is determined that the road element a and the road element B are different road elements, for example, the road element a is traffic light 1, and the road element B is a sign 1.

Step 203, for each identical road element, if the confidence of the identical road element in the first-type road element information is greater than a first threshold and the confidence of the identical road element in the second-type road element information is less than a second threshold, determining the confidence of the identical road element in the first-type road element information and the position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, wherein the first threshold is not less than the second threshold.

In a specific implementation, after the same road element between the first-type road element and the second-type road element is determined, for each same road element, if the confidence of the same road element in the first-type road element information is greater than a first threshold and the confidence of the same road element in the second-type road element is less than a second threshold, the confidence of the same road element in the first-type road element information and the position of the same road element in the second-type road element information in the road image are determined as the road element information of the same road element.

Specifically, for a road element of the second type of road element whose confidence of the road element is not less than the second threshold, the road element information of the road element is determined to be the road element information of the road element of the second type of road element information. For a road element different between the first-type road element and the second-type road element, if the different road element belongs to the first-type road element and the confidence level of the different road element in the first-type road element information is not greater than the first threshold, the confidence level of the different road element in the first-type road element information and the position of the road element in the road image are still used as the road element information of the different road element.

According to the method for determining the road element information, the rapid detection algorithm and the accurate detection algorithm belong to a deep learning multi-target detection algorithm, the advantages that the confidence degree of the road element determined by the rapid detection algorithm is reliable and the position accuracy of the road element determined by the accurate detection algorithm in an image is high are considered, when the road element determined by the rapid detection algorithm and the road element determined by the accurate detection algorithm are the same road element, the confidence degree of the same road element determined by the rapid detection algorithm and the position of the road element determined by the accurate detection algorithm in the image are used as the road element information of the same road element, and therefore accurate road element information is obtained.

In practical applications, a classifier is trained according to sample data to determine unknown parameters in the classifier, the confidence of the road element a and the position of the road element a in the image, the confidence of the road element B and the position of the road element B in the image are used as input parameters, the classifier after the unknown parameters are determined is input, if the output result of the classifier is yes, the road element a and the road element B are considered to be the same road element, and the confidence of the road element a and the position of the road element B in the road image are determined as the road element information of the same road element. Wherein the road element a belongs to a first class of road elements and the road element B belongs to a second class of road elements. The position of the road element in the image is used as an input parameter, and specifically, the position of four vertexes in the rectangular detection frame corresponding to the road element in the image is used as an input parameter. Further, the input parameters may also comprise the type of road element, i.e. road element a belongs to a first type of road element and road element B belongs to a second type of road element. The process of training the classifier according to the sample data to determine the unknown parameters in the classifier can be referred to in the prior art, which is not described herein again.

As a possible implementation, the road image may be determined according to the content provided in fig. 4:

step 401, according to the collection position of each candidate road image, performing hierarchical clustering on the candidate road images to obtain a first-level classification result for the candidate road images, wherein the actual distance between the candidate road images adjacent to the collection position in each first-level classification is not greater than a preset distance.

In a specific implementation, the candidate road image may be an original image acquired by the acquisition device, or may also be an image screened from the original image acquired by the acquisition device, for example, a blurred image and a backlit image are deleted from the original image, and the remaining original image is used as the candidate road image.

In this step, the collection position of each candidate road image is specifically a geographical position where the collection device is located when the corresponding candidate road image is collected, the geographical position may be extracted from recorded GPS (Global Positioning System) information of the collection device when the corresponding candidate road image is collected, and the geographical position may specifically include longitude and latitude.

In the step, the candidate road images are clustered by using a hierarchical clustering algorithm, specifically, the candidate road images can be clustered by using an agglomerated hierarchical clustering algorithm, and the process of clustering by using the hierarchical clustering algorithm can refer to the prior art and is not detailed here.

After the candidate road images are hierarchically clustered, the obtained first-level classification result comprises at least one first-level classification, the actual distance between the candidate road images adjacent to the acquisition position in each first-level classification is not greater than a preset distance, the preset distance can be set according to an actual application scene, for example, the preset distance can be set to be 30 meters, 50 meters and the like, and the setting is not limited herein.

The candidate road images in each first-level classification are sorted according to the collection positions, wherein the actual distance between the collection positions of the candidate road images in the first specified order and the collection positions of the candidate road images in the second specified order is assumed to be a first actual distance; the actual distance between the acquisition positions of the candidate road images in the second designated sequence and the acquisition positions of the candidate road images in the third designated sequence is a second actual distance, and the first actual distance is smaller than the second actual distance; if other candidate road images do not exist between the candidate road image in the first specified sequence and the candidate road image in the second specified sequence, the candidate road image in the first specified sequence and the candidate road image in the second specified sequence are considered as candidate road images with adjacent collection positions; accordingly, if there is no other candidate road image between the candidate road image in the second specified order and the candidate road image in the third specified order, the candidate road image in the second order and the candidate road image in the third specified order are considered as candidate road images whose collection positions are adjacent. The sequence corresponding to the second designated sequence is positioned between the sequence corresponding to the first designated sequence and the sequence corresponding to the third execution sequence.

For example, the first-level classification includes a candidate road image 1, a candidate road image 2, a candidate road image 3, and a candidate road image 4, where the results of sorting according to the collection positions are the candidate road image 2, the candidate road image 1, the candidate road image 3, and the candidate road image 4, then, in the first-level classification, the candidate road image 2 is adjacent to the collection position of the candidate road image 1, the candidate road image 1 is adjacent to the collection position of the candidate road image 3, and the candidate road image 3 is adjacent to the collection position of the candidate road image 4, where an actual distance between the collection positions of any two adjacent candidate road images is not greater than a preset distance. The actual distance between two candidate road images with non-adjacent collection positions is greater than the distance between two candidate road images with adjacent collection time, for example, the actual distance between the collection position of the candidate road image 2 and the collection position of the candidate road image 1 is less than the actual distance between the collection position of the candidate road image 1 and the collection position of the candidate road image 3; the actual distance between the acquisition position of the candidate road image 1 and the acquisition position of the candidate road image 3 is smaller than the actual distance between the acquisition position of the candidate road image 3 and the acquisition position of the candidate road image 4.

Step 402, aiming at each obtained first-level classification, if the road elements between the candidate road images adjacent to the acquisition time in the first-level classification are determined to be similar, the candidate road images adjacent to the acquisition time are divided into the same classification, and a second-level classification result is obtained.

In specific implementation, after the first-level classification result is obtained, the candidate road images in each first-level classification are further classified to obtain a second-level classification result. Wherein the candidate road images in the first classification are re-classified according to the similarity of road elements between the candidate road images adjacent to each other in the acquisition time. Specifically, when the candidate road images in the first-level classification are classified again, the following steps may be executed in a loop according to the sequence of the collection time from early to late until the candidate road image with the latest collection time in the first-level classification is classified:

determining whether road elements between the candidate road image in the current order and the candidate road image in the next order are similar; if yes, dividing the candidate road image in the current sequence and the candidate road image in the next sequence into the same classification; taking the candidate road image in the next order as the candidate road image in the current order and performing a step of determining whether or not the road elements between the candidate road image in the current order and the candidate road image in the next order are similar; if not, the candidate road image of the current sequence and the candidate road image of the next sequence are divided into different classifications, the candidate road image of the next sequence is taken as the candidate road image of the current sequence, and the step of determining whether the road elements between the candidate road image of the current sequence and the candidate road image of the next sequence are similar is executed.

The candidate road images with adjacent acquisition times can be classified into the same category when the road elements of the same type are determined to be similar between the candidate road images with adjacent acquisition times, or any road element is determined to be similar between the candidate road images with adjacent acquisition times, or the number of the road elements with similar between the candidate road images with adjacent acquisition times is determined to reach a certain number. The two road elements may be determined to be similar when the similarity of the image features of the image areas corresponding to the two road elements is greater than a certain value, or may be determined to be similar in other manners, which is not limited herein.

Optionally, before step 402 is executed, candidate road images of an ascending road and candidate road images of a descending road in the same road are further screened out, specifically, the candidate road images whose acquisition time interval in the first-level classification is within a preset time period may be used as the first group, for example, all the candidate road images in the first group may be considered as candidate road images on the ascending road. And aiming at the obtained candidate road images in the first group of each first-level classification, if the road elements between the candidate road images adjacent to the collection time in the group are determined to be similar, the candidate road images adjacent to the collection time are divided into the same classification, and a second-level classification result is obtained.

The candidate road images outside the first group in the first-level classification are classified into a second group, for example, all the candidate road images in the second group are considered as candidate road images on a down road, and the candidate road images in the second group are subjected to the second-level classification, or, because the acquisition time interval between the candidate road images in the second group may not be within a preset time period, the candidate road images which are not classified into the first group can be regarded as unsatisfactory images and discarded.

Step 403, selecting a candidate road image from each second-level classification as a road image.

In a specific implementation, the obtained second-level classification result includes at least one second-level classification, each second-level classification includes at least one candidate road image, and one candidate road image is selected from each second-level classification as a road image, so as to obtain the determined road images, where the number of the determined road images is the same as the number of the second-level classifications. A candidate road image may be randomly selected from each second-level classification, or a road image may be selected from the second-level classification according to other selection rules, which are not limited herein. In practical applications, a plurality of candidate road images may be selected from each second-level classification according to practical application scenarios, for example, two candidate road images may be selected from each second-level classification as road images.

In the embodiment, the candidate road images are classified at the acquisition positions to obtain first-level classification results, then the first-level classification results are subjected to second-level classification according to the similarity of the road elements, at the moment, the similarity between the candidate road images in each second-level classification can be considered to be high, and one image is selected from each second-level classification to serve as the road image, so that the number of road images input with an accurate detection algorithm can be greatly reduced, and the processing speed of the road elements is determined by the accurate detection algorithm to a certain extent.

Fig. 4 is only one possible embodiment of determining a road image, and the real-time manner of determining the road image is not limited in the embodiment of the present application, and the road image may also be determined in the following manner:

the first method is as follows: and according to the acquisition position of each candidate road image, performing hierarchical clustering on the candidate road images to obtain a classification result aiming at the candidate road images, and selecting at least one candidate road image from each classification as a road image.

The second method comprises the following steps: if the road elements between the candidate road images adjacent to the acquisition time in each candidate road image are determined to be similar, the candidate road images adjacent to the acquisition time are divided into the same classification, the classification result for the candidate road images is obtained, and at least one candidate road image is selected from each classification to be used as the road image.

As a possible implementation manner, before determining the road image, the method for determining the road element information provided in the embodiment of the present application further includes the following steps:

using each collected original image as the input of a rapid detection algorithm; and determining the original image containing the road elements output by the rapid detection algorithm as a candidate road image.

In the specific implementation process, the image acquired by the acquisition equipment may not contain the road elements, and in order to avoid using the image not containing the road elements as the road image, the original image containing the road elements can be determined by using a rapid detection algorithm, so that all candidate road images can be ensured to be the original images containing the road elements, and thus, when the road element information of the road image is determined, the road image not containing the road elements can be prevented from being processed, and the efficiency of determining the road element information is improved.

As one possible implementation, the road element similarity between the candidate road images in the first-level classification that are collected temporally adjacent may be determined according to the content provided in fig. 5:

step 501, for candidate road images adjacent to each other in acquisition time, respectively acquiring a first rectangular detection frame corresponding to a first road element of the candidate road image closer to the acquisition time, and a second rectangular detection frame corresponding to a second road element of the candidate road image closer to the acquisition time.

In a specific implementation, the candidate road images with adjacent collection time comprise two candidate road images, wherein the collection time of one candidate road image is earlier than that of the other candidate road image. The first road element is any road element in the candidate road image with the earlier acquisition time, and the second road element is any road element in the candidate road image with the later acquisition time.

The position of the first rectangular detection frame in the road image is the position of the first road element in the road image, and the position of the second rectangular detection frame in the road image is the position of the second road element in the road image.

Step 502, if the length-width ratio deviation of the first rectangular detection frame and the second rectangular detection frame is within a preset range, the first rectangular detection frame is smaller than the second rectangular detection frame, and the image feature of the image area corresponding to the first rectangular detection frame is similar to the image feature of the image area corresponding to the second rectangular detection frame, it is determined that the first road element is similar to the second road element.

Specifically, the deviation between the length-width ratio of the first rectangular detection frame and the length-width ratio of the second rectangular detection frame is determined, for example, the absolute value of the difference between the length-width ratio of the first rectangular detection frame and the length-width ratio of the second rectangular detection frame is taken as the deviation between the two, or the ratio of the length-width ratio of the first rectangular detection frame and the length-width ratio of the second rectangular detection frame is taken as the deviation between the two. The size of the preset range may be set according to an actual application scenario, and is not limited herein.

In specific implementation, when the area of the first rectangular detection frame is smaller than that of the second rectangular detection frame, it is determined that the first rectangular detection frame is smaller than the second rectangular detection frame. If the acquired candidate road image comprises road elements with high similarity, the acquisition equipment acquires the candidate road image according to a principle of distance and proximity, wherein the acquisition time of the acquired image is earlier as the distance from the road elements is farther, so that the rectangular detection frame corresponding to the road element in the candidate road image with the earlier acquisition time is smaller than the rectangular detection frame corresponding to the road element in the candidate road image with the later acquisition time.

In a specific implementation, the rectangular detection frame corresponding to the road element is displayed on the candidate road image, and the image area corresponding to the rectangular detection frame is specifically the image area on the candidate road image included in the rectangular detection frame. In the embodiment of the application, the image features of the rectangular detection frames are all the same type of image features, such as all the color histograms of images or all the texture features of images.

This possible embodiment is intended to illustrate the process of determining that two road elements between different candidate road images are similar. In this possible embodiment, similar road elements are determined by combining the variation in the length-width ratio of the rectangular detection frames of different road elements, the size of the rectangular detection frame, and the image characteristics of the image area included in the rectangular detection frame, so that the determined similar road elements can be determined with higher accuracy.

Of course, the content provided in fig. 5 is only one possible embodiment for determining whether the road elements are similar, and it may also be determined whether the road elements are similar by combining at least two items of the deviation of the length-width ratio of the rectangular detection frames of different road elements, the size of the rectangular detection frame, and the image characteristics of the image area included by the rectangular detection frame, for example, for candidate road images with adjacent acquisition times, a first rectangular detection frame corresponding to a first road element of a candidate road image with an acquisition time earlier is obtained, and a second rectangular detection frame corresponding to a second road element of a candidate road image with a later acquisition time is obtained; if the deviation of the length-width ratio of the first rectangular detection frame and the second rectangular detection frame is within a preset range and the first rectangular detection frame is smaller than the second rectangular detection frame, determining that the first road element is similar to the second road element; for example, aiming at candidate road images adjacent to the collection time, a first rectangular detection frame corresponding to a first road element of the candidate road image close to the collection time is respectively obtained, and a second rectangular detection frame corresponding to a second road element in the candidate road image close to the collection time is respectively obtained; and if the deviation of the length-width ratio of the first rectangular detection frame and the second rectangular detection frame is within the preset range and the image characteristics of the image area corresponding to the first rectangular detection frame are similar to the image characteristics of the image area corresponding to the second rectangular detection frame, determining that the first road element is similar to the second road element.

As a possible implementation, a candidate road image may be selected from any of the second-level classifications, as provided in fig. 6:

step 601, for each candidate road image in any second-level classification, if the rectangular detection frame corresponding to each road element of the candidate road image is completely contained in the candidate road image, determining that the candidate road image is a complete road image.

In order to ensure the integrity of the image in this step, when the rectangular detection frames corresponding to the road elements in the candidate road image are completely included in the candidate road image, the candidate road image is determined to be a complete road image.

Step 602, selecting a complete road image from the determined complete road images according to the image definition and the image size.

In specific implementation, after the integrity of the image is ensured, a complete road image is selected from the determined complete road images according to the definition and the size of the image. Specifically, a complete road image with a definition greater than a definition threshold and an image size greater than a size threshold may be selected from the determined complete road images.

In this possible embodiment, the candidate road image is selected from the second-level classification in combination with the image integrity, the image definition and the image size, so that it is ensured that a relatively better candidate road image is selected.

Fig. 6 is only one possible implementation of selecting the candidate road image from the second-level classification, and in practical applications, the candidate road image may be selected from the second-level classification by considering at least one of image integrity, image sharpness, and image size, for example, a candidate road image with better sharpness is selected from the second-level classification.

As a possible implementation, according to the content provided in fig. 7, it may be implemented to select one complete road image from the determined complete road images according to the image definition and the image size:

step 701, determining a weighted sum result of image definition and image size of each determined complete road image as a weight of the complete road image.

Step 702, selecting a complete road image with the largest weight from the determined complete road images.

In specific implementation, the image definition and the image size weight are respectively set, and the weighted sum result of the image definition and the image size of the complete road image is used as the weight of the complete road image. Selecting a complete road image with the largest weight from the determined complete road images; and under the condition that the whole road image with the maximum weight comprises a plurality of whole road images, randomly selecting one whole road image from the plurality of whole road images with the maximum weight. In practical application, the complete road image with the weight larger than the weight threshold value can be determined from the determined complete road images, and one complete road image is selected from the complete road images with the weight larger than the weight threshold value. Optionally, the weight of the image definition is smaller than the weight of the image size, which may indicate that the priority of the image size is higher than the priority of the image definition, for example, a complete road image with an image size larger than a size threshold is preferably selected from the complete road images.

As a possible implementation, the second type road element and the second type road element information of the road image can be determined by using a precise detection algorithm according to the content provided in fig. 8:

step 801, intercepting a first type of road element from the road image to obtain at least one screenshot corresponding to the road image, where the screenshot includes at least one road element in the first type of road element.

In a specific implementation, the first-class road elements are cut from the road image to obtain the corresponding screenshot of the road image, and specifically, each road element in the first-class road elements can be cut from the road image to obtain the screenshot of each road element, where in this case, the number of the road elements in the first road elements is the same as the number of the screenshots. Of course, a plurality of road elements in the first type of road element may be captured in the same screenshot, which is not limited herein. Optionally, the screenshot is an image block with a set size, for example, the set size may be m × n, m and n are the number of pixels, and the image block corresponding to the screenshot is larger than the image block corresponding to any one road element in the road image.

And step 802, taking each screenshot as the input of the accurate detection algorithm, and obtaining the road element and the road element information of the corresponding screenshot output by the accurate detection algorithm.

In specific implementation, each screenshot corresponding to the road image is used as the input of the accurate detection algorithm, and the road element information of each screenshot output by the accurate detection algorithm are obtained.

Step 803, for each screenshot, mapping the position of the road element in the screenshot in the road element information to the position of the road element in the road image.

In a specific implementation, the position of the road element in the road element information in the screenshot is the position of the road element in the screenshot, in this step, the position of the road element in the screenshot needs to be mapped to the position of the road element in the road image to which the screenshot belongs, and the specific mapping manner may be: recording the positions of all screenshots corresponding to the road image in the road image after intercepting the first type of road elements from the road image, and mapping the screenshots to the road image according to the positions of the screenshots in the road image; the road element is mapped to a position in the road image according to the position of the road element in the screenshot. Such as: as shown in fig. 9a, a schematic diagram of a position of a road element in the screenshot, as shown in fig. 9b, a schematic diagram of a position of the screenshot in a road image, as shown in fig. 9c, a schematic diagram of a position of a road element in the screenshot in the road image, wherein an origin of a coordinate system in fig. 9a is a vertex at the upper left corner of the screenshot, and horizontal and vertical coordinate axes are two edges connected to the vertex at the upper left corner of the screenshot respectively; in fig. 9b and 9c, the origin of the coordinate system is the vertex of the top left corner of the road image, the abscissa and ordinate axes are two edges connected to the vertex of the top left corner of the road image, and the positions of the road elements in the screenshot are the positions of the four vertices of the rectangular detection frame corresponding to the road element in the screenshot, which are respectively: (a1, b1), (a2, b1), (a1, b2), (a2, b2), the positions of the four vertices of the screenshot mapped to the road image according to the position of the screenshot in the road image are respectively: (c1, d1), (c2, d1), (c1, d2), and (c2, d2), the positions of the four vertices of the road element in the road image are: (a1+ c1, b1+ d1), (a2+ c1, b1+ d1), (a1+ c1, b2+ d1), (a2+ c1, b2+ d 1).

And step 804, taking the road elements of each screenshot as second-class road elements of the road image, and taking the confidence degrees of the road elements in the road element information of each screenshot and the positions of the corresponding road elements in the road image as second-class road element information.

In specific implementation, the road elements of each screenshot are taken as the second type road elements of the corresponding road image, and the confidence of the road elements in the road element information of each screenshot and the positions of the mapped road elements in the road image are taken as the second type road element information.

According to the possible implementation mode, the first type of road elements in the road image are subjected to screenshot, the obtained screenshot is used as the input of the accurate detection algorithm, on one hand, the fact that each screenshot contains the road elements is guaranteed, the situation that the screenshot of the input accurate detection algorithm does not contain the road elements is avoided, the processing burden of the accurate detection algorithm is reduced to a certain extent, on the other hand, the screenshot is used as the input of the accurate detection algorithm, the size of the image of the input accurate detection algorithm is reduced, and the processing speed of the accurate detection algorithm for determining the road element information is improved to a certain extent.

As a possible implementation, according to the content provided in fig. 10, the first type road elements are cut out from the road image:

step 1001, a first minimum bounding rectangle containing all road elements of the first class of road elements is determined.

In specific implementation, for all road elements in the first type of road elements, a minimum bounding rectangle capable of simultaneously containing all the road elements is determined as a first minimum bounding rectangle.

Step 1002 compares the length of the first minimum bounding rectangle to a length threshold, and compares the width of the first minimum bounding rectangle to a width threshold.

In specific implementation, the size of the length threshold and the width threshold is not limited, and the length threshold may be smaller than the width threshold, and optionally, the length threshold is not smaller than the width threshold, and the size of the length threshold and the size of the width threshold may be set according to an actual application scenario, which is not limited here.

Step 1003, if the length of the first minimum circumscribed rectangle is not greater than the length threshold or the width of the first minimum circumscribed rectangle is not greater than the width threshold, capturing a screenshot including all road elements in the first type of road elements from the road image, wherein the length of the screenshot is not less than the length threshold, and the width of the screenshot is not less than the width threshold.

When the first minimum bounding rectangle is smaller than or equal to the length threshold, or the frame of the first minimum bounding rectangle is smaller than or equal to the width threshold, a screenshot including each road element in the first type of road elements is captured from the road image. The length of the screenshot is greater than or equal to a length threshold and the width of the screenshot is greater than or equal to a width threshold.

And 1004, if the length of the first minimum bounding rectangle is larger than the length threshold or the width of the first minimum bounding rectangle is larger than the width threshold, grouping the road elements in the first class of road elements to obtain a plurality of groups, wherein the length of a second minimum bounding rectangle containing all the road elements in each group is equal to the length threshold, and the width of the second minimum bounding rectangle is equal to the width threshold.

In specific implementation, the road elements in the first type of road elements are grouped under the condition that the length of the first minimum circumscribed rectangle is larger than the length threshold value, or the width of the first minimum circumscribed rectangle is larger than the width threshold value. For each resulting grouping, at least one road element of the first class of road elements is included in the grouping, and the length of a second minimum bounding rectangle that includes all road elements in the grouping is equal to the length threshold, and the width of the second minimum bounding rectangle is equal to the width threshold.

It should be noted that, in the embodiment of the present invention, the second minimum bounding rectangle is a rectangle having a length equal to the length threshold and a width equal to the width threshold.

Step 1005, for each group, intercepting a screenshot including all road elements in the group from the road image.

In specific implementation, the corresponding screenshots of each group are intercepted from the road image, and the screenshots intercepted in the step only comprise all road elements in one group, namely, the quantity of the screenshots intercepted from the road image in the step is the same as the quantity of the groups. In this step, the length of the screenshot is greater than or equal to the length threshold, and the width of the screenshot is greater than or equal to the width threshold.

In this step, one group corresponds to one screenshot, and if the same road element exists in the screenshots corresponding to different groups, it is determined that only part of the screenshots of the same road element are included, and the color of the part of the same road element in the determined screenshots is set to be a pure color, for example, white. In the different screenshots, the same road element may not be completely included, for example, screenshot 1 includes a road element a, and screenshot 2 also includes a road element a, then only one screenshot in screenshot 1 or screenshot 2 completely includes the road element a, and the other screenshot only includes a part of the road element a, that is, only a part of the road element a may be displayed in the other screenshot.

Optionally, the length of the screenshot related in the embodiment of the present application is equal to a product of the length threshold and the length deviation, and the width of the screenshot is equal to a product of the width threshold and the width deviation. The screenshot may be captured from the road image with the center of the rectangle as the center, the product of the length threshold and the length deviation as the long side, and the product of the width threshold and the width deviation as the wide side, where the rectangle is specifically a first minimum bounding rectangle for step 1003, and the rectangle is specifically a second minimum bounding rectangle for step 1004.

In practice, the length deviation and the width deviation can be determined as follows:

the first method is as follows: determining sample road elements of a plurality of road images by utilizing a rapid detection algorithm in advance; counting the length and width of a theoretical rectangular detection frame corresponding to each sample road element, and counting the length and width of a real rectangular detection frame corresponding to each sample road element; determining the sum of the lengths of the theoretical rectangular detection frames corresponding to the sample road elements as a first sum, and determining the sum of the lengths of the real rectangular detection frames corresponding to the sample road elements as a second sum; taking the absolute value of the difference value between the first sum value and the second sum value, and taking the set multiple of the absolute value as the length deviation; determining the width sum of the theoretical rectangular detection frame corresponding to each sample road element as a third sum, and determining the width sum of the real rectangular detection frame corresponding to each sample road element as a fourth sum; and taking the absolute value of the difference value between the third sum and the fourth sum, and taking the set multiple of the absolute value as the width deviation. The setting multiple can be set according to an actual application scenario, for example, the setting multiple is 2, or the setting multiple is 3.

The second method comprises the following steps: determining sample road elements of a plurality of road images by utilizing a rapid detection algorithm in advance; counting the length and width of a theoretical rectangular detection frame corresponding to each sample road element, and counting the length and width of a real rectangular detection frame corresponding to each sample road element; determining the difference value between the length of a theoretical rectangular detection frame and the length of a real rectangular detection frame of each sample road element as a first difference value; taking the absolute value of the first difference as the length error of the sample road element, and determining the difference value between the width of the theoretical rectangular detection frame and the width of the real rectangular detection frame of the sample road element as a second difference value; taking the absolute value of the second difference as the width error of the sample road element; the maximum length error among the length errors of the respective sample road elements is taken as the length deviation, and the maximum width error among the width errors of the respective sample road elements is taken as the width deviation.

It should be noted that, in the case that the external rectangle is a square, the length of the external rectangle is equal to the width of the external rectangle, the length and the width of the external rectangle in the embodiment of the present application are only used for distinguishing different sides of the rectangle, the length of the external rectangle may be less than the width of the external rectangle, and the length of the external rectangle may also be greater than or equal to the width of the external rectangle; in the embodiment of the application, the length and the width of the screenshot are only used for distinguishing different sides of the screenshot, the length of the screenshot may be smaller than the width of the screenshot, and the length of the screenshot may also be greater than or equal to the width of the screenshot, wherein the length and the width of the screenshot are equal when the screenshot is a square, and optionally, the sides of the screenshot are parallel to the sides of the road image.

As one possible real-time approach, road elements in the first category of road elements may be grouped according to the content provided in fig. 11:

step 1101, determining whether the first-class road elements include road elements which are not grouped, if yes, executing step 1102, otherwise, executing step 1107.

In step 1102, the road element at the upper left position of the road image among the ungrouped road elements is specified as the reference road element.

In a specific implementation, a road element at the leftmost position in the road image is identified as a road element at the upper left position in the road image for the road elements not yet grouped in the first type of road elements.

Step 1103, aligning the upper left corner of the rectangular detection frame corresponding to the reference road element with the upper left corner of the second minimum circumscribed rectangle.

In step 1104, if the second minimum bounding rectangle completely contains the rectangle detection frame corresponding to the reference road element, it is searched whether the second minimum bounding rectangle completely contains the rectangle detection frames corresponding to other ungrouped road elements, if yes, step 1105 is executed, otherwise, step 1106 is executed.

In a specific implementation, it may be searched whether the second minimum bounding rectangle completely contains other ungrouped road elements according to the order from left to right and from top to bottom of the positions of the road elements in the image.

In step 1105, the road elements completely included in the second minimum bounding rectangle are divided into the same group, and the process returns to step 1101.

In specific implementation, the same mark can be marked on the road elements in the same group, and different marks can be marked on the road elements in different groups, so that the groups to which the road elements belong can be distinguished.

In step 1106, the reference road elements are grouped, and the process returns to step 1101.

In step 1107, grouping of road elements in the first category of road elements is ended.

Of course, the content provided in fig. 11 is only one possible implementation manner, and in practical applications, the road elements in the first category of road elements may also be grouped by using a clustering algorithm, as long as it is ensured that the obtained road elements in each group can be completely contained in the second minimum bounding rectangle, and all the road elements in the first category of road elements are grouped.

The following describes an apparatus for specifying road element information according to an embodiment of the present application with reference to fig. 12.

As shown in fig. 12, a schematic structural diagram of an apparatus for determining road element information according to an embodiment of the present application includes:

an obtaining module 1201, configured to obtain, for each determined road image, a first type of road element and first type of road element information of the road image determined by using a fast detection algorithm, and obtain a second type of road element and second type of road element information of the road image determined by using a precise detection algorithm, where the road element information includes a confidence of the road element and a position of the road element in the road image;

a first determining module 1202 for determining the same road elements between the first type of road elements and the second type of road elements;

a second determining module 1203, configured to determine, for each identical road element, if the confidence level of the identical road element in the first type of road element information is greater than a first threshold and the confidence level of the identical road element in the second type of road element information is smaller than a second threshold, the confidence level of the identical road element in the first type of road element information and the position of the identical road element in the second type of road element information in the road image as the road element information of the identical road element, where the first threshold is not smaller than the second threshold.

a third determining module 1204, configured to determine the road image in the following manner:

according to the acquisition positions of the candidate road images, performing hierarchical clustering on the candidate road images to obtain first-level classification results for the candidate road images, wherein the actual distance between the candidate road images adjacent to the acquisition positions in each first-level classification is not greater than a preset distance;

aiming at each obtained first-level classification, if the road elements between the candidate road images adjacent to the acquisition time in the first-level classification are determined to be similar, dividing the candidate road images adjacent to the acquisition time into the same classification to obtain a second-level classification result;

and respectively selecting one candidate road image from each second-level classification as the road image.

a fourth determining module 1205, configured to use each acquired original image as an input of the fast detection algorithm before the third determining module 1204 determines the road image; and determining the original image containing the road elements output by the rapid detection algorithm as a candidate road image.

As a possible implementation manner, the third determining module 1204 is specifically configured to:

aiming at candidate road images adjacent to the collection time, respectively obtaining a first rectangular detection frame corresponding to a first road element of the candidate road image close to the collection time, and a second rectangular detection frame corresponding to a second road element in the candidate road image close to the collection time;

and if the length-width ratio deviation of the first rectangular detection frame and the second rectangular detection frame is within a preset range, the first rectangular detection frame is smaller than the second rectangular detection frame, and the image characteristics of the image area corresponding to the first rectangular detection frame are similar to the image characteristics of the image area corresponding to the second rectangular detection frame, determining that the first road element is similar to the second road element.

As a possible implementation, the third determining module 1204 is specifically configured to select a candidate road image from any of the second-level classifications in the following manner:

for each candidate road image in any second-level classification, if the rectangular detection frame corresponding to each road element of the candidate road image is completely contained in the candidate road image, determining that the candidate road image is a complete road image;

and selecting a complete road image from the determined complete road images according to the image definition and the image size.

As a possible implementation manner, the third determining module 1204 is specifically configured to select one complete road image from the determined complete road images in the following manner:

determining the weighted sum result of the image definition and the image size of each determined complete road image as the weight of the complete road image;

and selecting one complete road image with the largest weight from the determined complete road images.

As a possible implementation manner, the first determining module 1202 is specifically configured to:

determining, for each road element of the first class of road elements, a road element of the second class of road elements having an overlap region with a position of the road element in the road image as an overlapping road element;

and if the overlapping area of the road element and the overlapping road element is larger than the area threshold value, determining that the road element and the overlapping road element are the same road element.

a fifth determining module 1206, configured to determine the second type road element and the second type road element information of the road image by using the precise detection algorithm according to the following manner:

intercepting the first type of road elements from the road image to obtain at least one screenshot corresponding to the road image, wherein the screenshot comprises at least one road element in the first type of road elements;

taking each screenshot as the input of the accurate detection algorithm to obtain the road element and the road element information of the corresponding screenshot output by the accurate detection algorithm;

for each screenshot, mapping the position of the road element in the screenshot in the road element information of the screenshot into the position of the road element in the road image;

and taking the confidence coefficient of the road element in the road element information of each screenshot and the position of the corresponding road element in the road image as the second type road element information.

As a possible implementation, the fifth determining module 1206 is specifically configured to intercept the first type road element from the road image according to the following manner:

determining a first minimum bounding rectangle containing all road elements in the first type of road elements;

if the length of the first minimum circumscribed rectangle is not greater than a length threshold value or the width of the first minimum circumscribed rectangle is not greater than a width threshold value, a screenshot including all road elements in the first type of road elements is intercepted from the road image, wherein the length of the screenshot is not less than the length threshold value, and the width of the screenshot is not less than the width threshold value.

As a possible implementation, the fifth determining module 1206 is further configured to:

if the length of the first minimum bounding rectangle is larger than the length threshold or the width of the first minimum bounding rectangle is larger than the width threshold, grouping the road elements in the first type of road elements to obtain a plurality of groups, wherein the length of a second minimum bounding rectangle containing all the road elements in each group is equal to the length threshold, and the width of the second minimum bounding rectangle is equal to the width threshold;

for each grouping, a screenshot is taken from the road image that includes all of the road elements in the grouping.

Having described the method and apparatus for determining road element information provided by the embodiments of the present application, a non-volatile computer storage medium provided by the embodiments of the present application is described below.

Embodiments of the present application provide a non-volatile computer storage medium having stored thereon an executable program that is executed by a processor to perform the steps of implementing any of the methods of determining road element information provided in the above embodiments.

Having described the method, apparatus, and medium for determining road element information provided by the embodiments of the present application, a computer device provided by the embodiments of the present application is described below.

The embodiment of the present application further provides a computer device, which includes a memory, a processor and a computer program stored on the memory, and when the processor executes the computer program, the processor implements the steps of any one of the above-mentioned methods for determining road element information.

An embodiment of the present application further provides a computer device, configured to execute the method for determining road element information in the foregoing embodiment, as shown in fig. 13, which is a schematic diagram of a hardware structure of the computer device in the implementation of the present application, where the computer device may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, and the like. In particular, the computer device may comprise a memory 1301, a processor 1302 and a computer program stored on the memory, the processor when executing the program implementing the steps of any of the methods of determining road element information in the above embodiments. Memory 1301 may include Read Only Memory (ROM) and Random Access Memory (RAM), among other things, and provides processor 1302 with program instructions and data stored in memory 1301.

Further, the computer device described in the embodiment of the present application may further include an input device 1303, an output device 1304, and the like. The input device 1303 may include a keyboard, a mouse, a touch screen, and the like; the output device 1304 may include a Display device such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), a touch screen, or the like. The memory 1301, the processor 1302, the input device 1303 and the output device 1304 may be connected by a bus or other means, and fig. 13 illustrates an example of a connection by a bus.

The processor 1302 calls the program instructions stored in the memory 1301 and executes the method of determining road element information provided in the above-described embodiment according to the obtained program instructions.

By utilizing the method, the device, the medium and the equipment for determining the road element information, provided by the embodiment of the application, the following beneficial effects are achieved:

the fast detection algorithm and the accurate detection algorithm both belong to a deep learning multi-target detection algorithm, and the advantages of the reliable confidence degree of the road element determined by the fast detection algorithm and the high position accuracy of the road element determined by the accurate detection algorithm in the image are considered.

It should be noted that although several modules of the apparatus for determining road element information are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the modules described above may be embodied in one module according to embodiments of the application. Conversely, the features and functions of one module described above may be further divided into embodiments by a plurality of modules.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method of determining road element information, comprising:

acquiring first type road element information and first type road element information of the road image determined by a rapid detection algorithm and second type road element information of the road image determined by a precise detection algorithm aiming at each determined road image, wherein the road element information comprises the confidence coefficient of the road element and the position of the road element in the road image;

determining road elements of the first type that are identical to road elements of the second type;

for each identical road element, if the confidence of the identical road element in the first-type road element information is greater than a first threshold and the confidence of the identical road element in the second-type road element information is less than a second threshold, determining the confidence of the identical road element in the first-type road element information and the position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, wherein the first threshold is not less than the second threshold.

2. The method according to claim 1, characterized in that the road image is determined in the following way:

3. The method of claim 2, wherein prior to determining the road image, further comprising:

using each collected original image as the input of the rapid detection algorithm;

and determining the original image containing the road elements output by the rapid detection algorithm as a candidate road image.

4. The method of claim 2, wherein determining road element similarity between temporally adjacent candidate road images captured in the first-level classification comprises:

5. The method of claim 2, wherein selecting a candidate road image from any of the second-level classifications comprises:

6. The method according to claim 5, wherein selecting a complete road image from the determined complete road images comprises:

7. The method of claim 6, wherein the image sharpness is weighted less than the image size.

8. The method according to claim 1, wherein determining the same road element between the first type of road element and the second type of road element comprises:

9. The method of claim 1, wherein determining the second type road element and the second type road element information of the road image by using a precise detection algorithm specifically comprises:

10. The method according to claim 9, wherein the step of extracting the first type of road element from the road image comprises:

11. The method of claim 10, further comprising:

12. An apparatus for determining road element information, comprising:

the acquisition module is used for acquiring first road element information and first road element information of the road image determined by using a rapid detection algorithm and second road element information of the road image determined by using an accurate detection algorithm aiming at each determined road image, wherein the road element information comprises the confidence coefficient of the road element and the position of the road element in the road image;

a first determining module for determining the same road elements between the first type of road elements and the second type of road elements;

a second determining module, configured to determine, for each identical road element, if a confidence of the identical road element in the first-type road element information is greater than a first threshold and a confidence of the identical road element in the second-type road element information is less than a second threshold, the confidence of the identical road element in the first-type road element information and a position of the identical road element in the second-type road element information in the road image as the road element information of the identical road element, where the first threshold is not less than the second threshold.

13. A non-transitory computer storage medium storing an executable program for execution by a processor to perform the steps of the method of any one of claims 1 to 11.

14. A computer device comprising a memory, a processor and a computer program stored on the memory, the processor implementing the steps of the method of any one of claims 1 to 11 when executing the program.