WO2023179031A1 - 一种图像处理方法、装置、电子设备、存储介质及计算机程序产品 - Google Patents

一种图像处理方法、装置、电子设备、存储介质及计算机程序产品 Download PDF

Info

Publication number
WO2023179031A1
WO2023179031A1 PCT/CN2022/129070 CN2022129070W WO2023179031A1 WO 2023179031 A1 WO2023179031 A1 WO 2023179031A1 CN 2022129070 W CN2022129070 W CN 2022129070W WO 2023179031 A1 WO2023179031 A1 WO 2023179031A1
Authority
WO
WIPO (PCT)
Prior art keywords
category
image
traffic
confidence
frame
Prior art date
Application number
PCT/CN2022/129070
Other languages
English (en)
French (fr)
Inventor
王筱涵
林培文
冨手要
Original Assignee
商汤集团有限公司
本田技研工业株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 商汤集团有限公司, 本田技研工业株式会社 filed Critical 商汤集团有限公司
Publication of WO2023179031A1 publication Critical patent/WO2023179031A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches

Definitions

  • the present disclosure relates to, but is not limited to, the technical field of image processing, and specifically relates to an image processing method, device, electronic equipment, storage medium, and computer program product.
  • the classification and recognition of traffic objects can be achieved through multi-classification task models.
  • traffic objects such as traffic signs
  • multi-classification task models because there are many traffic objects and are easily affected by long distances or object occlusion, there is a problem of low recognition accuracy.
  • embodiments of the present disclosure provide an image processing method, device, electronic device, storage medium and computer program product.
  • Embodiments of the present disclosure provide an image processing method, which includes:
  • correction information for categories whose confidence levels do not meet preset conditions is determined.
  • An embodiment of the present disclosure also provides an image processing device, which includes: an acquisition unit, a first determination unit, a second determination unit, and a third determination unit; wherein,
  • the acquisition unit is configured to acquire the video stream collected by the image acquisition device installed on the traveling equipment;
  • the first determining unit is configured to determine multiple frames of images containing specific traffic objects from the video stream;
  • the second determination unit is configured to determine the confidence of the category of the specific traffic object in each frame of the multi-frame image
  • the third determination unit is configured to determine, based on a comparison result of confidence levels of categories of specific traffic objects in the multi-frame images, correction information for categories whose confidence levels do not meet a preset condition.
  • Embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the image processing method described in the embodiments of the present disclosure are implemented.
  • An embodiment of the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the program, the image described in the embodiment of the present disclosure is realized. Processing method steps.
  • Embodiments of the present disclosure provide a computer program product.
  • the computer program product includes a non-transitory computer-readable storage medium storing a computer program.
  • the embodiments of the present disclosure are implemented. Some or all steps in an image processing method.
  • Embodiments of the present disclosure provide image processing methods, devices, electronic equipment, storage media, and computer program products.
  • the method includes: obtaining a video stream collected by an image acquisition device installed on a traveling device, and determining from the video stream a specific Multi-frame images of traffic objects; determining the category of a specific traffic object in each frame of the multi-frame image and the confidence of the category; based on the confidence of the category of the specific traffic object in the multi-frame image Compare the results to determine the correction information for categories whose confidence does not meet the preset conditions.
  • the categories whose confidence levels do not meet the preset conditions are corrected, that is, the high reliability of the specific traffic objects is used to correct the categories.
  • Classification results and correction of low-reliability classification results can, on the one hand, improve the classification accuracy of traffic objects in the image, and on the other hand, provide a reliable basis for downstream decision-making control to facilitate subsequent real-time control.
  • Figure 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present disclosure
  • Figure 3 is a schematic diagram of classification results in an image processing method provided by an embodiment of the present disclosure.
  • Figure 4 is a schematic diagram of the reliability of classification results in an image processing method provided by an embodiment of the present disclosure
  • Figure 5 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram of the hardware composition of an electronic device provided by an embodiment of the present disclosure.
  • the identification of traffic objects is mainly achieved through a single-layer multi-classifier. Since there are many categories of traffic objects and they are easily affected by long distances, object occlusion, etc., there are problems such as difficulty in labeling and inability to achieve accurate classification.
  • Figure 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present disclosure
  • the 21st frame (Frame21) image cannot recognize the content of the traffic sign due to the distance
  • the 51st frame (Frame51) Because the image is blocked by tree trunks, the traffic sign is easily recognized as a speed limit of 30
  • the 55th frame (Frame55) image is also blocked by a tree trunk, and the traffic sign is easily recognized as a speed limit of 60
  • the traffic The sign can be correctly read as speed limit 50.
  • the electronic device corrects the categories whose confidence does not meet the preset conditions based on the comparison results of the confidence levels of the categories of specific traffic objects in the multi-frame images, that is, using the high reliability classification of specific traffic objects.
  • correcting the low-reliability classification results of specific traffic objects can, on the one hand, improve the classification accuracy of traffic objects in the image, and on the other hand, provide a reliable basis for downstream decision-making control.
  • the traffic object may be any object on the road, and may include, for example, at least one of traffic signs, road signs, traffic participants, and traffic lights.
  • the terms “comprising”, “comprises” or any other variations thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements not only includes the explicitly stated elements, but also other elements not expressly listed, or elements inherent to the implementation of the method or apparatus. Without further limitation, an element defined by the statement “comprises a" does not exclude the presence of other related elements (such as steps in the method or devices) in the method or device including the element.
  • a unit in the device for example, a unit may be part of a circuit, part of a processor, part of a program or software, etc.).
  • the image processing method provided by the embodiment of the present disclosure includes a series of steps, but the image processing method provided by the embodiment of the present disclosure is not limited to the recorded steps.
  • the image processing device provided by the embodiment of the present disclosure includes a series of modules, but the device provided by the embodiment of the present disclosure is not limited to include the explicitly recorded modules, and may also include modules that need to be set up to obtain relevant information or perform processing based on the information.
  • a and/or B can mean: A exists alone, A and B exist simultaneously, and they exist alone. B these three situations.
  • at least one herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C, which can mean including from A, Any one or more elements selected from the set composed of B and C.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present disclosure; as shown in Figure 2, the method includes:
  • Step 101 Obtain the video stream collected by the image acquisition device installed on the traveling equipment, and determine multiple frames of images containing specific traffic objects from the video stream;
  • Step 102 Determine the category of the specific traffic object in each frame of the multi-frame image and the confidence of the category;
  • Step 103 Based on the comparison results of the confidence levels of the categories of specific traffic objects in the multi-frame images, determine the correction information of the categories whose confidence levels do not meet the preset conditions.
  • the image processing method in this embodiment is applied to electronic devices, which may be vehicle-mounted devices, cloud platforms, or other computer devices.
  • the vehicle-mounted device may be a thin client, a thick client, a microprocessor-based system, a small computer system, etc. installed on the driving device
  • the cloud platform may be a distribution including a small computer system or a large computer system. Cloud computing technology environment and so on.
  • the traveling equipment may be, for example, various vehicles running on the road. In the following embodiments, the traveling equipment is a vehicle as an example for description.
  • the vehicle-mounted equipment can communicate with the vehicle's sensors, positioning devices, etc., and the vehicle-mounted equipment can obtain the data collected by the vehicle's sensors and the geographical location information reported by the positioning device through the communication connection.
  • the sensor of the vehicle may be at least one of millimeter wave radar, lidar, camera and other equipment;
  • the positioning device may be a device for providing positioning services based on at least one of the following positioning systems: Global Positioning System (GPS) , Global Positioning System), Beidou Satellite Navigation System or Galileo Satellite Navigation System.
  • GPS Global Positioning System
  • GPS Global Positioning System
  • Beidou Satellite Navigation System Beidou Satellite Navigation System
  • Galileo Satellite Navigation System Galileo Satellite Navigation System
  • the vehicle-mounted device can be an Advanced Driving Assistant System (ADAS).
  • ADAS is set on the vehicle.
  • the ADAS can obtain the vehicle's real-time location information from the vehicle's positioning device, and/or the ADAS can obtain the vehicle's real-time location information from the vehicle's positioning device.
  • Image data, radar data, etc. representing information about the vehicle's surrounding environment are obtained from the vehicle's sensors.
  • ADAS can send vehicle driving data including the vehicle's real-time location information to the cloud platform.
  • the cloud platform can receive the vehicle's real-time location information and/or image data and radar data representing the vehicle's surrounding environment information. etc.
  • the video stream is obtained through an image acquisition device (i.e., the above-mentioned sensor, such as a camera) installed on the driving device.
  • the image acquisition device collects road images or environment images around the driving device in real time as the driving device moves, that is,
  • the video stream may be a continuous image of the surrounding environment or scene continuously collected by the traveling device while in a traveling state.
  • the electronic device can identify each frame of image in the video stream through a classification network, determine that each frame of image includes a specific traffic object, and determine the category of the specific traffic object.
  • the video stream can be used as the input data of a classification network.
  • Features are extracted from each frame of the image in the video stream through the classification network.
  • Specific traffic objects in the image are determined based on the extracted features, and the location of the specific traffic object in the image is determined.
  • the first area in and determine the category of a specific traffic object.
  • the category of the specific traffic object may be one category among multiple traffic object categories.
  • each traffic object is divided into multiple categories in advance, and each category may contain one or more traffic objects.
  • the category to which a specific traffic object belongs may be one of the above-mentioned pre-divided categories.
  • determining the confidence of the category of the specific traffic object in each of the multiple frame images may include: determining the confidence of the category of the specific traffic object in each of the multiple frame images. Template image of the category, calculate the similarity between the specific traffic object and the template image, and determine the confidence of the category of the specific traffic object based on the similarity.
  • template images of various categories are stored in the electronic device. After determining the category of the specific traffic object, the electronic device compares the image of the first area where the specific traffic object is located with the template image of the corresponding category, calculates the similarity between the specific traffic object and the template image of the corresponding category, and calculates The similarity can be used as the confidence of the category of a specific traffic object.
  • the electronic device determines the correction information of the categories whose confidence levels do not meet the preset conditions, that is, the confidence levels do not meet the preset conditions. Categories of conditions are corrected. Wherein, for example, if the confidence does not meet the preset condition, the confidence may be less than the preset threshold.
  • determining the correction information of the category whose confidence level does not meet the preset condition includes: responding to the The confidence of the category of the specific traffic object in the first image satisfies the preset condition, and the confidence of the category of the specific traffic object in the second image of the multi-frame image does not satisfy the preset condition,
  • the category of the specific traffic object in the second image is set to the category of the specific traffic object in the first image.
  • the order of the first image and the second image is not limited, that is, the first image is after the second image, or the first image is before the second image.
  • the electronic device can use the high-reliability classification results for a specific traffic object to correct the low-reliability classification results for the specific traffic object, thereby realizing the correction of categories whose confidence does not meet the preset conditions, and can Provide sufficient basis for downstream modules (such as control module, decision-making module, etc.) to facilitate subsequent real-time control.
  • downstream modules such as control module, decision-making module, etc.
  • determining multiple frames of images containing specific traffic objects from the video stream includes: determining the position of the traffic object in the image containing the traffic object in the video stream. A region; for each first region, determine a second region within the first region, the second region being smaller than the first region; based on the information of each second region, from the image containing traffic objects Images containing traffic objects of a first category are selected as multi-frame images containing specific traffic objects, and the first category is a category to which the specific traffic object belongs.
  • the electronic device determines the first area of the traffic object in the image in each frame of the video stream, that is, obtains the detection frame (such as a rectangular frame) of the traffic object in the image, and the detection frame is in the image.
  • the area is also the first area; and then the second area is determined within the first area in each image included in the video stream.
  • the second area is the central area of the first area
  • the information of the second area is the information of the central area of the feature map of the first area.
  • the second area may be determined by: reducing the length and width of the first area (ie, the detection frame of the traffic object) in equal proportions, and the reduced area is used as the second area.
  • the second area can also be called the central area of the first area.
  • the second area may also be determined by: reducing the length and width of the first area in non-proportional proportions according to the degree of obstruction of the traffic object, and then reducing the length and width according to the degree of obstruction of the traffic object. The position moves the center point to obtain the second area.
  • the first area where the traffic object is located (that is, the detection of the traffic object) can be The length and width of the frame) are unequally reduced.
  • the reduced area can be moved to the right (the moved area is also within the first area), thereby obtaining the second area, so as to retain as many characteristics of traffic objects as possible in the second area and reduce the characteristics of obstructions.
  • the electronic device filters images containing traffic objects of the first category from images containing traffic objects, and treats multiple frames of images containing traffic objects of the first category as images containing specific traffic objects.
  • Multi-frame images of traffic objects the first category is the category to which the specific traffic object belongs.
  • selecting images containing traffic objects of the first category from the images containing traffic objects based on the information of each second area includes: characterizing each second area respectively. Extract, determine the first similarity of the pixels of each second area based on the extracted features; locate the second area whose location information satisfies the first preset condition and the first similarity satisfies the second preset condition.
  • the image is determined to be an image containing a traffic object of the first category.
  • the first category is the category to which the specific traffic object belongs.
  • the position information of the second area in the image containing traffic objects satisfies the first preset condition, and may be the second area in any two adjacent frames of the image in the image containing traffic objects.
  • the distance between the position information that is, the difference between the positions of the second area in the image where it is located, is less than the first threshold.
  • the distance may be a distance in a specified coordinate system (such as a pixel coordinate system, an image coordinate system, etc.).
  • the first similarity meets the second preset condition, which may mean that the first similarity is greater than or equal to the second threshold.
  • the second image is a frame of image after the first image; in one example, the second image may be The second image may be a frame after the first image. In another example, the second image may also be an image several frames after the first image.
  • the first image may be the 21st frame image
  • the second image may be the 51st frame image, the 55th frame image, or the 60th frame image.
  • the electronic device performs traffic object recognition on the first image and the second image respectively, and determines the first area of the traffic object in the first image and the first area of the traffic object in the second image.
  • the tracking of the identified traffic object in the first image is performed in a second area smaller than the first area where the traffic object is located.
  • the traffic object will usually be occluded. edges, so a second region is used for tracking, which is robust to occlusions.
  • the length and width of the first area are reduced in equal proportions, and the reduced area is used as the second area.
  • the second area may also be called the central area of the first area. In some implementations, it may be determined based on the pixel points of the second area in the first image and the pixel points of the second area in the second image whether the traffic objects in the first image and the second image are Traffic objects of the first category.
  • feature extraction processing can be performed on the pixels in the second area in the first image
  • feature extraction processing can be performed on the pixels in the second area in the second image
  • both are calculated based on the respectively extracted features.
  • the degree of similarity between them herein referred to as the first similarity
  • the first position information may be the coordinates of the center point of the second area in the first image
  • the second position information may be the coordinates of the center point of the second area in the second image.
  • the first position information and the second position information satisfy a first preset condition, which may be the distance between the first position information and the second position information (for example, the first image
  • the distance between the center point coordinates of the second area in and the center point coordinates of the second area in the second image) is less than the first threshold.
  • the distance may be a distance in a specified coordinate system (such as a pixel coordinate system, an image coordinate system, etc.). For example, in the pixel coordinate system, determine the first center point coordinates corresponding to the first position information, determine the second center point coordinates corresponding to the second position information, and make the difference between the first center point coordinates and the second center point coordinates to obtain the above The distance between the first position information and the second position information.
  • the frame interval of image collection is extremely small, if a traffic object of the first category is included in different frame images, the position of the traffic object in different frame images is also similar.
  • the first similarity meets the second preset condition, which may be that the first similarity is greater than or equal to the second threshold; in the case that the first similarity is greater than or equal to the second threshold, the first similarity may be greater than or equal to the second threshold.
  • feature extraction can be performed on the second area in each image respectively, and the first similarity of the pixels in the second area in each image is determined based on the extracted features; when the first similarity satisfies the third Under the condition of two preset conditions, it can be determined that the traffic objects in each image are of the same category (such as the first category). In this case, the traffic objects in each image can be determined to be of the same category (such as the first category), but may not be the same traffic object. In another case, based on the above determination of the first similarity, the position information of the second area in each image satisfies the first preset condition, and the first similarity satisfies the second preset condition.
  • the traffic objects in each image are of the same category (such as the first category).
  • the traffic objects in each image are of the same category (such as the first category) and are the same traffic object.
  • the first category of traffic objects described in this embodiment is not limited to the fact that the traffic objects in at least two frames of images belong to the same category, but may also include the case where the traffic objects in at least two frames of images are the same traffic object.
  • the following solution can be used to determine that the traffic objects in at least two frames of images are the same traffic objects: for the traffic objects identified in each frame of the image, their corresponding unique identifiers (IDs) can be assigned, so that they can be identified in different frames.
  • IDs unique identifiers
  • the same traffic object is identified between the images; in response to the situation that the traffic object in the multiple frame images is of the same category (such as the first category), the first category will be assigned to the traffic object in the multiple frame image.
  • Identity association can be used to determine that the traffic objects in at least two frames of images are the same traffic objects: for the traffic objects identified in each frame of the image, their corresponding unique identifiers (IDs) can be assigned, so that they can be identified in different frames.
  • the same traffic object is identified between the images; in response to the situation that the traffic object in the multiple frame images is of the same category (such as the first category), the first category will be assigned to the traffic object in the multiple frame image.
  • Identity association can be used to determine that
  • the traffic object (object 1) is assigned a third An identification; determining that the traffic object corresponding to the second area in the second image (for example, marked as object 2) and the traffic object corresponding to the second area in the first image (object 1) belong to the same category (such as first category), the first identifier assigned to object 1 can be associated with object 2, that is, both object 1 and object 2 are associated with the first identifier, as traffic of the same category (such as the first category) object.
  • the second identification is replaced by the first identification, that is, both object 1 and object 2 are associated with the first identification, as traffic objects of the same category (such as the first category).
  • determining the category of a specific traffic object in each of the multiple frame images and the confidence of the category includes: determining each of the multiple frame images.
  • the first category is the category to which the specific traffic object belongs.
  • determining the correction information of the category whose confidence does not meet the preset condition includes: based on all the categories of the multi-frame image.
  • the comparison result of the confidence of the subdivided category of the traffic object of the first category with the highest confidence is used to determine the correction information of the subdivided category of the traffic object of the first category whose maximum confidence does not meet the preset condition.
  • the electronic device first determines the first category of traffic objects (i.e., rough classification category), and then determines the subdivision categories of the traffic objects in the first category, that is, first performs a coarse-grained classification of the traffic objects, and then classifies the traffic objects. Objects are classified fine-grained within this coarse-grained classification.
  • the electronic device can identify each frame image in the video stream through the first layer network and determine that each frame image includes traffic objects of the same category (first category), that is, detect the traffic objects in each frame image. , and determine that the detected traffic objects belong to the same category (i.e., the first category).
  • the video stream can be used as the input data of the first-layer network.
  • Features are extracted from each frame image in the video stream through the first-layer network, and the traffic objects in each frame image are determined based on the extracted features.
  • the first area of the traffic object in each frame of image determines the category of the traffic object (such as a rough classification category), that is, the detection frame of the traffic object in each frame of image and the category to which the traffic object belongs (coarse classification category) are output; and then Multiple frames of images of traffic objects belonging to the same category (here noted as the first category) are determined.
  • the multi-frame images may be continuous or discontinuous frame images in the video stream.
  • the video stream includes 100 frames of images, and the determined multi-frame images containing traffic objects of the first category may be the 10th to 50th frames among them, or may also be the 5th and 50th frames among the 100 frames of images.
  • the 15th frame, the 25th frame, the 35th frame, the 45th frame, etc. are not limited in this embodiment.
  • the category to which the specific traffic object belongs is one category among multiple traffic object categories. It can be understood that the first layer network is obtained by pre-training based on traffic object classification. Through the processing of the image by the first layer network, it can be obtained whether the traffic object in the image belongs to the pre-labeled traffic object classification and which kind of traffic it belongs to. Object classification.
  • traffic signs can be divided into multiple first categories 41 in advance, such as speed signs, sidewalk signs, warning signs, stop signs, etc.; assuming that the traffic objects in each frame of the image in the video stream are identified, that is, Multiple frames of images containing traffic objects of the "Speed Class Identification" category can be filtered out.
  • traffic signs can be classified according to their functions or functions. In other embodiments, other classification methods may also be used, which is not limited in this embodiment.
  • the first category in each frame of the multiple frame images is determined through the second layer network.
  • the traffic objects are subdivided and processed to obtain the subdivision category of the first category of traffic objects in each frame image and the confidence level of the subdivision category.
  • the second layer network may be a classification network corresponding to the category to which the traffic object belongs.
  • the number of second-layer networks can correspond to the number of categories to which traffic objects belong, that is, each category of traffic objects can correspond to a second-layer network, and the corresponding traffic is pre-marked in each second-layer network.
  • the subdivision category with the highest confidence of the traffic object and the confidence level of the subdivision category can be obtained, such as the confidence level of the traffic object
  • the largest subdivision category can be the 70km/h speed mark.
  • the second layer network may also correspond to the categories to which multiple traffic objects belong.
  • the second-layer network can be used to identify subdivision categories in "One-way signs", "Turn type signs" and "Lane type signs”.
  • the second layer network may include multiple branch networks used for classification processing, and each branch network may be used to identify subdivision categories corresponding to the categories to which one or more types of traffic objects belong. For example, after identifying the category to which the traffic object belongs (such as the first category), the electronic device cuts out the sub-image corresponding to the first area where the traffic object is located, and inputs the sub-image into the branch network corresponding to the first category. , used to identify the subdivision categories under the first category.
  • the first category (rough classification category) of the traffic object is first determined, and then the subdivision category of the traffic object in the first category is determined, that is, the traffic object is first classified into a coarse-grained classification, and then the traffic object is classified into this coarse-grained classification.
  • Fine-grained classification can improve the classification accuracy of traffic objects in images (such as traffic signs, road signs, etc.); especially in the current situation where traffic objects with many categories are identified through a single-layer multi-classifier, it can improve the classification accuracy due to the categories. There are many problems that are difficult to label and cannot be accurately classified.
  • determining the subdivision category with the highest confidence of the first category of traffic objects in each of the multi-frame images and the confidence of the subdivision category includes: : Determine the second similarity between the traffic objects of the first category in each frame of the multi-frame image and the template images of each second category, each second category being a subdivision category of the first category; based on The second similarity determines the subdivided category with the highest confidence of the first category of traffic objects in each frame of the image and the confidence of the subdivided category.
  • template images of each second category are stored in the electronic device.
  • the electronic device compares the image (which may be a feature map of the area where the traffic object is located) with the template images of each second category, and determines the traffic object of the first category in each frame of image.
  • the similarity between template images of each second category (herein referred to as the second similarity).
  • the maximum second similarity can also be used as the confidence of the subdivision category (such as the second category) of the traffic object, or the confidence of the subdivision category (such as the second category) of the traffic object can also be based on the largest second similarity.
  • the second similarity is calculated.
  • the traffic object in the 60th frame (Frame60) image is compared with the template images of each second category, and it is determined that its second similarity with the "50km/h speed sign" is 100 %;
  • the traffic objects in the 55th frame (Frame55) image are compared with the template images of each second category, and the second similarity with the "60km/h speed sign" is determined to be 50%;
  • Frame51 (Frame51) The traffic objects in the image are compared with the template images of each second category, and it is determined that the second similarity between them and the "30km/h speed sign" is 40%; the traffic objects in the 21st frame (Frame21) image are consistent with each second category.
  • the maximum confidence corresponding to the second category to which the traffic object belongs in the multi-frame image is greater than or equal to the third threshold, it is determined that the confidence (ie, reliability) of the second category to which the traffic object belongs is high; Correspondingly, when the maximum confidence corresponding to the second category to which the traffic object in the multi-frame image belongs is less than the third threshold, it is determined that the confidence (ie reliability) of the second category to which the traffic object belongs is low.
  • the maximum confidence level corresponding to the second category of the traffic object in the 60th frame (Frame60) image is 100%, which is regarded as high reliability; while in the 21st frame (Frame21) image, the traffic object belongs to The maximum confidence corresponding to the second category is 80%.
  • the maximum confidence corresponding to the second category the traffic object belongs to is 40%.
  • the maximum confidence level corresponding to a category is 50%, and all of them can be regarded as low reliability.
  • the above third threshold can be determined based on actual conditions.
  • the third threshold can be determined based on the maximum confidence corresponding to each frame image. For example, in 100 frames of images, the images with the maximum confidence of 100% account for 80%, the images with the maximum confidence of 80% account for 15%, and the images with the maximum confidence of 50% account for 5%, then you can If the classification result of the image is considered to be relatively reliable, the value of the third threshold can be set higher, for example, to 90% or even 95%. Correspondingly, if the overall maximum confidence calculation result in the image is not too high, the value of the third threshold can be set smaller. This is not limited in this embodiment.
  • the electronic device determines, based on the comparison result of the confidence of the subdivision category with the largest confidence of the first category of traffic objects in the multi-frame images, that the maximum confidence does not meet the preset condition.
  • the correction information of the subdivision categories of traffic objects of the category is corrected to correct the subdivision categories of the first category of traffic objects whose maximum confidence does not meet the preset conditions, thereby improving the classification accuracy of traffic objects in the video stream and providing downstream services. Provide a reliable basis for decision-making control.
  • the correction information of the subdivided categories of the first category of traffic objects under preset conditions includes: the confidence of the subdivided category with the highest confidence level for the first category of traffic objects in the first image among the multi-frame images.
  • the third preset condition will be
  • the subdivision category with the highest confidence level for the traffic objects of the first category in the two images is set to the same category as the subdivision category with the greatest confidence level for the traffic objects of the first category in the first image.
  • the confidence level satisfying the third preset condition may be that the confidence level is greater than or equal to the fourth threshold.
  • the order of the first image and the second image in the multi-frame images is not limited, that is, the first image is after the second image, or the first image is after the second image. Two images before.
  • the second image is a frame of image after the first image. This embodiment is suitable for real-time detection of images.
  • the confidence level satisfies the third preset condition, it may indicate a high degree of reliability or high reliability; correspondingly, if the confidence level does not meet the third preset condition, it may indicate a low degree of reliability or low reliability.
  • the largest second category i.e., subdivision category
  • the highly reliable subdivision result i.e. the largest subdivision category
  • the second category replaces the low-reliability subdivision result in the second image (ie, the largest second category).
  • Table 1 shows the classification representation before correction
  • Table 2 shows the classification representation after correction.
  • Table 1 shows the classification representation before correction
  • Table 2 shows the classification representation after correction.
  • Table 1 shows the classification representation before correction
  • Table 2 shows the classification representation after correction.
  • Table 1 shows the classification representation before correction
  • Table 2 shows the classification representation after correction.
  • Table 1 shows the classification representation before correction
  • Table 2 shows the classification representation after correction.
  • Table 1 shows the classification representation before correction
  • Table 2 shows the classification representation after correction.
  • Second category 50 Do Not Pass 30 50 Confidence high high Low Low high
  • This embodiment uses high-reliability subdivision results to correct low-reliability subdivision results (the subdivision category with the highest confidence) for the same traffic object, thereby realizing the first category whose maximum confidence does not meet the preset conditions. Correction of the subdivision categories of traffic objects can provide sufficient basis for downstream modules (such as control modules, decision-making modules, etc.) to facilitate subsequent real-time control.
  • the first image is a frame of image after the second image. This embodiment is suitable for scenarios where training data is classified incorrectly.
  • the above-mentioned images (including the first image and the second image) and the classification results are used to train the network model.
  • the 21st frame image (Frame21) is far away, and the identified subdivision result is low reliability (or confidence);
  • the classification result of frame 60 is at high reliability (or confidence).
  • the subsequent subdivision result of the first image with high reliability can be replaced with the previous subdivision result of the second image with low reliability.
  • Table 3 shows the classification representation before correction
  • Table 4 shows the classification representation after correction.
  • Table 3 through real-time detection and classification of five frames of images, four traffic objects with identification IDs of 1, 2, 3 and 4 were obtained (the traffic objects of the first two frames of images are the same traffic objects, so using same ID) and the first and second categories of each traffic object, as well as the corresponding confidence.
  • the four traffic objects are associated with the same identification ID as 1, as shown in Table 4.
  • the reliability of the subdivision classification results of the first few frames of images is low (the third preset condition is not met)
  • the reliability of the subdivision classification results of the same traffic objects in the fifth frame of images is If the reliability is high (the third preset condition is met), then the low-reliability subdivision results corresponding to the first few frames are replaced with high-reliability subdivision results.
  • the replaced subdivision results are: the first category is speed, The second classification is 50, and the confidence level is high.
  • Identification ID 1 1 2 3 4 First category prohibit speed speed speed speed
  • the correction information of the subdivided categories of the first category of traffic objects with preset conditions may also include: in response to the confidence of the subdivided category with the highest confidence of the first category of traffic objects in the multi-frame images. If the third preset condition is not met, a prompt message indicating that the classification result of the traffic object cannot be determined is output.
  • the confidence of the subdivision category with the highest confidence of the traffic object of the first category in each frame of image does not meet the third preset condition, that is, the confidence of the traffic object of the first category in each frame of image is not satisfied.
  • the confidence of the subdivision category with the highest confidence level of the traffic object is all less than the fourth threshold, that is, the reliability of the subdivision category with the greatest confidence level of the traffic object in the first category in each frame of image is low, so the traffic cannot be determined.
  • the subdivision category of the object and output a prompt message indicating that the classification result cannot be determined.
  • the traffic objects of the first two frames of images are the same traffic objects. objects, therefore using the same ID) and the first and second categories of each traffic object, as well as the corresponding confidence.
  • the four traffic objects are associated with the same identification ID as 1, as shown in Table 5. Assume that the fourth frame of image is currently collected.
  • the classification result of the second category corresponding to the same traffic object in the fourth frame of image is still If the confidence level is low (the third preset condition is not met), a prompt message indicating that the classification result cannot be determined is output.
  • the high confidence level can be output as the fifth frame of image is collected and it is detected that the classification result of the second category corresponding to the same traffic object in the fifth frame of image is high confidence (satisfying the third preset condition), then the high confidence level can be output.
  • the classification result of the confidence level is the subdivided classification result with a speed of 50km/h.
  • FIG. 5 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure; as shown in Figure 5, the device includes: an acquisition unit 21, a first determination unit 22, a second determination unit 23 and a third determination unit 24; among them,
  • the acquisition unit 21 is configured to acquire the video stream collected by the image acquisition device installed on the traveling equipment;
  • the first determining unit 22 is configured to determine multiple frames of images containing specific traffic objects from the video stream;
  • the second determination unit 23 is configured to determine the category of the specific traffic object in each frame of the multi-frame image and the confidence of the category;
  • the third determination unit 24 is configured to determine, based on the comparison results of confidence levels of categories of specific traffic objects in the multi-frame images, correction information for categories whose confidence levels do not meet preset conditions.
  • the first determining unit 22 is configured to determine the first area of the traffic object in the image containing the traffic object in the video stream; for each th A region, a second region is determined within the first region, the second region is smaller than the first region; based on the information of each second region, selecting images containing the first category from the images containing traffic objects For images of traffic objects, multiple frames of images containing traffic objects of a first category are used as multiple frame images containing specific traffic objects, and the first category is the category to which the specific traffic object belongs.
  • the second area is the central area of the first area
  • the information of the second area is the information of the central area of the feature map of the first area
  • the first determination unit 22 is configured to perform feature extraction on each of the second areas respectively, and determine the third value of the pixel points of each of the second areas based on the extracted features. A degree of similarity; determining the image in which the second area where the location information satisfies the first preset condition and the first similarity satisfies the second preset condition is located as an image containing traffic objects of the first category.
  • the second determination unit 23 is configured to determine the subdivision category with the highest confidence of the first category of traffic objects in each frame of the multi-frame image and Confidence of the subdivision category, the first category is the category to which the specific traffic object belongs.
  • the third determining unit 24 is configured to determine the confidence level based on the confidence of the subdivision category with the largest confidence level of the first category of traffic objects in the multi-frame images. The results are compared to determine the correction information of the subdivision categories of the first category of traffic objects whose maximum confidence does not meet the preset conditions.
  • the second determination unit 23 is configured to determine the difference between the traffic object of the first category and the template image of each second category in each frame of the multi-frame image.
  • the second similarity between each second category is a subdivision category of the first category; based on the second similarity, determine the subdivision category with the highest confidence of the traffic object of the first category in each frame image and all the subdivision categories. The confidence level of the segmented category.
  • the third determining unit 24 is configured to respond to the subdivision category with the highest confidence of the first category of traffic objects in the first image among the multi-frame images.
  • the confidence level satisfies the third preset condition, and the confidence level of the subdivision category with the highest confidence level of the traffic object of the first category in the second image among the multi-frame images does not meet the third preset condition, all the The subdivision category with the highest confidence level for the traffic objects of the first category in the second image is set to the same category as the subdivision category with the highest confidence level for the traffic objects of the first category in the first image.
  • the third determining unit 24 is configured to respond to the confidence of the subdivision category with the largest confidence of the first category of traffic objects in the multi-frame images. If the third preset condition is not met, a prompt message indicating that the classification result of the traffic object cannot be determined is output.
  • the acquisition unit 21, the first determination unit 22, the second determination unit 23 and the third determination unit 24 in the device can all be configured by a central processing unit (CPU, Central Processing Unit), Implemented by digital signal processor (DSP, Digital Signal Processor), microcontroller unit (MCU, Microcontroller Unit) or programmable gate array (FPGA, Field-Programmable Gate Array).
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • MCU Microcontroller Unit
  • FPGA Field-Programmable Gate Array
  • the image processing device provided in the above embodiment performs image processing
  • only the division of the above program modules is used as an example.
  • the above processing can be allocated to different program modules according to needs. That is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above.
  • the image processing device provided by the above embodiments and the image processing method embodiments belong to the same concept, and the implementation process can be referred to the method embodiments.
  • FIG. 6 is a schematic diagram of the hardware composition of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device includes a memory 32, a processor 31 and a storage device.
  • a computer program is stored on the memory 32 and can be run on the processor 31.
  • the processor 31 executes the program, the steps of the image processing method described in the embodiment of the present disclosure are implemented.
  • the electronic device may also include a user interface 33 and a network interface 34.
  • the user interface 33 may include a display, keyboard, mouse, trackball, click wheel, keys, buttons, touch pad or touch screen, etc.
  • bus system 35 various components in the electronic device are coupled together by a bus system 35 .
  • the bus system 35 is used to implement connection communication between these components.
  • the bus system 35 also includes a power bus, a control bus and a status signal bus.
  • the various buses are labeled bus system 35 in FIG. 6 .
  • the memory 32 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.
  • non-volatile memory can be read-only memory (ROM, Read Only Memory), programmable read-only memory (PROM, Programmable Read-Only Memory), erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory).
  • Volatile memory can be random access memory (RAM, Random Access Memory), which is used as an external cache.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • SSRAM Synchronous Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • SDRAM Synchronous Dynamic Random Access Memory
  • DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • ESDRAM enhanced Type Synchronous Dynamic Random Access Memory
  • SLDRAM Synchronous Link Dynamic Random Access Memory
  • DRRAM Direct Rambus Random Access Memory
  • the memory 32 described in embodiments of the present disclosure is intended to include, but is not limited to, these and any other suitable types of memory.
  • the methods disclosed in the above embodiments of the present disclosure can be applied to the processor 31 or implemented by the processor 31 .
  • the processor 31 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 31 .
  • the above-mentioned processor 31 may be a general processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the processor 31 can implement or execute the disclosed methods, steps and logical block diagrams in the embodiments of the present disclosure.
  • a general-purpose processor may be a microprocessor or any conventional processor, etc.
  • the steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software module may be located in a storage medium, which is located in the memory 32.
  • the processor 31 reads the information in the memory 32 and completes the steps of the foregoing method in combination with its hardware.
  • the electronic device may be configured by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs, Complex Programmable Logic Device), FPGA, general processor, controller, MCU, microprocessor (Microprocessor), or other electronic component implementation, used to execute the aforementioned method.
  • ASICs Application Specific Integrated Circuits
  • DSPs Digital Signal processors
  • PLDs Programmable Logic Devices
  • CPLDs Complex Programmable Logic Devices
  • FPGA general processor
  • controller MCU
  • Microprocessor microprocessor
  • the embodiment of the present disclosure also provides a computer-readable storage medium, such as a memory 32 including a computer program.
  • the above-mentioned computer program can be executed by the processor 31 of the electronic device to complete the steps of the aforementioned method.
  • the computer-readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one or any combination of the above memories.
  • the computer-readable storage medium provided by the embodiments of the present disclosure has a computer program stored thereon, and when the program is executed by the processor, the steps of the image processing method described in the embodiments of the present disclosure are implemented.
  • Embodiments of the present disclosure provide a computer program product.
  • the computer program product includes a non-transitory computer-readable storage medium storing a computer program.
  • the embodiments of the present disclosure are implemented. Some or all steps in an image processing method.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division.
  • the coupling, direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be electrical, mechanical, or other forms. of.
  • the units described above as separate components may or may not be physically separated.
  • the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present disclosure can be all integrated into one processing unit, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; the above-mentioned integration
  • the unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
  • the aforementioned program can be stored in a computer-readable storage medium.
  • the program When the program is executed, It includes the steps of the above method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
  • the above-mentioned integrated units of the present disclosure are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.
  • the computer software products are stored in a storage medium and include a number of instructions to A computer device (which may be a personal computer, a server, a network device, etc.) is caused to execute all or part of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage media include: mobile storage devices, ROM, RAM, magnetic disks or optical disks and other media that can store program codes.
  • Embodiments of the present disclosure disclose an image processing method, device, electronic equipment, storage medium and computer program product.
  • the method includes: obtaining a video stream collected by an image acquisition device installed on the driving equipment, determining multiple frames of images containing specific traffic objects from the video stream; determining the specific traffic in each frame of the multiple frame images.
  • the category of the object and the confidence of the category based on the comparison result of the confidence of the category of the specific traffic object in the multi-frame image, the correction information of the category whose confidence does not meet the preset condition is determined.
  • the categories whose confidence levels do not meet the preset conditions are corrected, that is, the highly reliable classification results of the specific traffic objects are used.
  • correcting low-reliability classification results can, on the one hand, improve the classification accuracy of traffic objects in the image, and on the other hand, it can provide a reliable basis for downstream decision-making control to facilitate subsequent real-time control.

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例公开了一种图像处理方法、装置、电子设备、存储介质及计算机程序产品。所述方法包括：获取行驶设备上安装的图像采集装置采集的视频流，从所述视频流中确定包含特定交通对象的多帧图像；确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述细分类别的置信度；基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。

Description

一种图像处理方法、装置、电子设备、存储介质及计算机程序产品
相关申请的交叉引用
本公开基于申请号为202210301591.0、申请日为2022年03月24日、申请名称为“一种图像处理方法、装置、电子设备和存储介质”的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本公开作为参考。
技术领域
本公开涉及但不限于图像处理技术领域，具体涉及一种图像处理方法、装置、电子设备、存储介质及计算机程序产品。
背景技术
相关技术中，可通过多分类任务模型实现交通对象(例如交通标识)的分类识别，但由于交通对象众多，且容易受到远距离或物体遮挡等影响，因此存在识别准确率不高的问题。
发明内容
有鉴于此，本公开实施例提供一种图像处理方法、装置、电子设备、存储介质及计算机程序产品。
本公开实施例的技术方案是这样实现的：
本公开实施例提供了一种图像处理方法，所述方法包括：
获取行驶设备上安装的图像采集装置采集的视频流，从所述视频流中确定包含特定交通对象的多帧图像；
确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度；
基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。
本公开实施例还提供了一种图像处理装置，所述装置包括：获取单元、第一确定单元、第二确定单元和第三确定单元；其中，
所述获取单元，被配置为获取行驶设备上安装的图像采集装置采集的视频流；
所述第一确定单元，被配置为从所述视频流中确定包含特定交通对象的多帧图像；
所述第二确定单元，被配置为确定所述多帧图像中的每帧图像中的特定交通对象的类别的置信度；
所述第三确定单元，被配置为基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。
本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本公开实施例所述图像处理方法的步骤。
本公开实施例还提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现本公开实施例所述图像处理方法的步骤。
本公开实施例提供一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现本公开实施例所述图像处理方法中的部分或全部步骤。
本公开实施例提供的图像处理方法、装置、电子设备、存储介质及计算机程序产品，所述方法包括：获取行驶设备上安装的图像采集装置采集的视频流，从所述视频流中确定包含特定交通对象的多帧图像；确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度；基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。采用本公开实施例的技术方案，根据多帧图像中的特定交通对象的类别的置信度的比较结果，对置信度不满足预设条件的类别进行校正，即利用特定交通对象的高可靠性的分类结果，对低可靠性的分类结果进行校正，一方面能够提高图像中的交通对象的分类准确度，另一方面能够为下游的决策控制提供可靠的依据，便于后续的实时控制。
附图说明
图1为本公开实施例提供的一种图像处理方法的应用场景示意图；
图2为本公开实施例提供的一种图像处理方法的流程示意图；
图3为本公开实施例提供的一种图像处理方法中的分类结果示意图；
图4为本公开实施例提供的一种图像处理方法中的分类结果的可靠性示意图；
图5为本公开实施例提供的一种图像处理装置的组成结构示意图；
图6为本公开实施例提供的一种电子设备的硬件组成结构示意图。
具体实施方式
下面结合附图及具体实施例对本公开实施例作进一步详细的说明。
相关技术中，交通对象的识别主要通过一个单层多分类器来实现，由于交通对象类别众多，且容易受到远距离、物体遮挡等影响，因此存在着难以标注、不能实现准确分类等问题。图1为本公开实施例提供的一种图像处理方法的应用场景示意图；如图1所示，第21帧(Frame21)图像由于距离过远导致不能识别交通标志牌内容；第51帧(Frame51)图像由于树干遮挡，交通标志牌易被识别为限速30；第55帧(Frame55)图像同样由于树干遮挡，交通标志牌易被识别为限速60；最终在第60帧(Frame60) 图像，交通标志牌可以被正确认为限速50。
本公开实施例中，电子设备根据多帧图像中的特定交通对象的类别的置信度的比较结果，对置信度不满足预设条件的类别进行校正，即利用特定交通对象的高可靠性的分类结果，对特定交通对象的低可靠性的分类结果进行校正，一方面能够提高图像中的交通对象的分类准确度，另一方面能够为下游的决策控制提供可靠的依据。
本公开各实施例中，交通对象可以是道路上的任意对象，例如可包括交通标志牌、道路标识、交通参与者和交通指示灯中的至少一种对象。
需要说明的是，在本公开实施例中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的方法或者装置不仅包括所明确记载的要素，而且还包括没有明确列出的其他要素，或者是还包括为实施方法或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个......”限定的要素，并不排除在包括该要素的方法或者装置中还存在另外的相关要素(例如方法中的步骤或者装置中的单元，例如的单元可以是部分电路、部分处理器、部分程序或软件等等)。
例如，本公开实施例提供的图像处理方法包含了一系列的步骤，但是本公开实施例提供的图像处理方法不限于所记载的步骤，同样地，本公开实施例提供的图像处理装置包括了一系列模块，但是本公开实施例提供的装置不限于包括所明确记载的模块，还可以包括为获取相关信息、或基于信息进行处理时所需要设置的模块。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
本公开实施例提供了一种图像处理方法。图2为本公开实施例提供的一种图像处理方法的流程示意图；如图2所示，所述方法包括：
步骤101：获取行驶设备上安装的图像采集装置采集的视频流，从所述视频流中确定包含特定交通对象的多帧图像；
步骤102：确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度；
步骤103：基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。
本实施例的图像处理方法应用于电子设备中，所述电子设备可以是车载设备，也可以是云平台或其他计算机设备。示例性的，车载设备可以是安装在行驶设备上的瘦客户机、厚客户机、基于微处理器的系统、小型计算机系统，等等，云平台可以是包括小型计算机系统或大型计算机系统的 分布式云计算技术环境等等。其中，所述行驶设备例如可以是道路上行驶的各种车辆，以下实施例中均以行驶设备为车辆为例进行说明。
本实施例中，车载设备可以与车辆的传感器、定位装置等通信连接，车载设备可以通过通信连接获取车辆的传感器采集的数据、以及定位装置上报的地理位置信息等。示例性的，车辆的传感器可以是毫米波雷达、激光雷达、摄像头等设备中的至少一种；定位装置可以是基于以下至少一种定位系统的用于提供定位服务的装置：全球定位系统(GPS，Global Positioning System)、北斗卫星导航系统或伽利略卫星导航系统。
在一个示例中，车载设备可以为高级辅助驾驶系统(ADAS，Advanced Driving Assistant System)，ADAS设置在车辆上，ADAS可以从车辆的定位装置中获取车辆的实时位置信息，和/或，ADAS可以从车辆的传感器中获得表示车辆周围环境信息的图像数据、雷达数据等等。其中，可选地，ADAS可以将包括车辆的实时位置信息的车辆行驶数据发送至云平台，如此，云平台可以接收到车辆的实时位置信息和/或表示车辆周围环境信息的图像数据、雷达数据等等。
本实施例中，通过设置在行驶设备上的图像采集装置(即上述传感器，如摄像头)获得视频流，图像采集装置伴随行驶设备的移动而实时采集行驶设备周围的道路图像或环境图像，也即所述视频流可以是行驶设备在行驶状态下连续采集的周围环境或场景得到的连续图像。
在一些可选实施例中，电子设备可通过分类网络识别所述视频流中的每帧图像，确定每帧图像中包括特定交通对象，以及特定交通对象的类别。示例性的，所述视频流可作为分类网络的输入数据，通过分类网络对视频流中的各帧图像进行特征提取，基于提取出的特征确定图像中的特定交通对象，确定特定交通对象在图像中的第一区域，以及确定特定交通对象的类别。其中，所述特定交通对象的类别可以是多个交通对象分类中的一个类别。例如预先将各交通对象划分为多个类别，每个类别中可包含一种或多种交通对象。特定交通对象所属的类别可以是上述预划分的一个类别。
在一些可选实施例中，确定所述多帧图像中的每帧图像中的特定交通对象的类别的置信度，可包括：确定所述多帧图像中的每帧图像中的特定交通对象的类别的模板图像，计算特定交通对象与模板图像之间的相似度，基于所述相似度确定特定交通对象的类别的置信度。
本实施例中，电子设备中存储有各个类别的模板图像。电子设备在确定特定交通对象的类别后，将特定交通对象所在的第一区域的图像与对应类别的模板图像进行比对，计算特定交通对象与对应类别的模板图像之间的相似度，计算出的相似度可作为特定交通对象的类别的置信度。
本实施例中，电子设备基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息，也即对置信度不满足预设条件的类别进行校正。其中，示例性的，置信度不满 足预设条件可以是置信度小于预设阈值。
可选地，所述基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息，包括：响应于所述多帧图像中的第一图像中的特定交通对象的类别的置信度满足预设条件，且所述多帧图像中的第二图像中的所述特定交通对象的类别的置信度不满足所述预设条件，将所述第二图像中的所述特定交通对象的类别设置为所述第一图像中的所述特定交通对象的类别。其中，所述第一图像和所述第二图像的先后顺序不限，即所述第一图像在所述第二图像之后，或者，所述第一图像在所述第二图像之前。
本实施例中，电子设备可利用针对特定交通对象的高可靠性的分类结果，校正该特定交通对象的低可靠性的分类结果，实现了对置信度不满足预设条件的类别进行校正，能够为下游模块(例如控制模块、决策模块等)提供充分的依据，便于后续的实时控制。
在本公开的一些可选实施例中，从所述视频流中确定包含特定交通对象的多帧图像，包括：确定所述视频流中的包含交通对象的图像中的交通对象在图像中的第一区域；针对每个第一区域，在所述第一区域内确定第二区域，所述第二区域小于所述第一区域；基于各个第二区域的信息，从所述包含交通对象的图像中挑选包含第一类别的交通对象的图像作为包含特定交通对象的多帧图像，所述第一类别为所述特定交通对象所属的类别。
本实施例中，电子设备确定视频流中的各帧图像中的交通对象在图像中的第一区域，也即获得交通对象在图像中的检测框(例如矩形框)，检测框在图像中的区域也即第一区域；进而在视频流包括的各图像中的第一区域内确定第二区域。
可选地，所述第二区域为所述第一区域的中心区域，所述第二区域的信息为所述第一区域的特征图的中心区域的信息。
作为一种示例，所述第二区域的确定方式可以是：将所述第一区域(即交通对象的检测框)的长度和宽度等比例的缩小，缩小后的区域作为第二区域，该第二区域也可称为所述第一区域的中心区域。作为另一种示例，所述第二区域的确定方式还可以是：按照交通对象的被遮挡程度将所述第一区域的长度和宽度进行非等比例的缩小，缩小后依据交通对象的被遮挡位置移动中心点，从而得到第二区域。以图1中的第51帧图像或第55帧图像为例，通过对交通对象进行检测，发现交通对象的左侧被遮挡，因此可针对该交通对象所在的第一区域(即交通对象的检测框)的长度和宽度进行非等比例的缩小，缩小后由于交通对象的左侧被遮挡，可将缩小后的区域向右移动(移动后的区域也在第一区域内)，从而得到第二区域，这样使得第二区域内尽可能多的保留交通对象的特征，减少遮挡物的特征。
本实施例中，电子设备基于图像中的第二区域的信息，从包含交通对象的图像中筛选包含第一类别的交通对象的图像，将多帧包含第一类别的 交通对象的图像作为包含特定交通对象的多帧图像，所述第一类别为所述特定交通对象所属的类别。
在一些可选实施例中，所述基于各个第二区域的信息，从所述包含交通对象的图像中挑选包含第一类别的交通对象的图像，包括：分别对所述各个第二区域进行特征提取，基于提取的特征确定所述各个第二区域的像素点的第一相似度；将位置信息满足第一预设条件、且所述第一相似度满足第二预设条件的第二区域所在的图像确定为包含第一类别的交通对象的图像。其中，第一类别为特定交通对象所属的类别。
其中，可选地，所述包含交通对象的图像中的第二区域的位置信息满足第一预设条件，可以是所述包含交通对象的图像中的任意相邻两帧图像中的第二区域的位置信息之间的距离，即第二区域在其所在图像中的位置之间的差异，小于第一阈值。其中，所述距离可以是在指定坐标系(例如像素坐标系、图像坐标系等)下的距离。
可选地，所述第一相似度满足第二预设条件，可以是指所述第一相似度大于或等于第二阈值。
示例性的，以所述包含交通对象的图像包括第一图像和第二图像为例，所述第二图像为所述第一图像后的一帧图像；一种示例中，第二图像可以是第一图像的后一帧图像，另一种示例中，第二图像也可以是第一图像后、相隔几帧的图像。例如图1中，第一图像可以是第21帧图像，第二图像可以是第51帧图像、第55帧图像或第60帧图像。
电子设备分别对第一图像和第二图像进行交通对象识别，确定交通对象在第一图像中的第一区域以及交通对象在第二图像中的第一区域。本示例中，针对识别出的第一图像中的交通对象进行追踪(tracking)，是采用小于交通对象所在的第一区域的第二区域进行追踪，考虑到遮挡的情况通常会遮挡到交通对象的边缘，因此采用第二区域进行追踪，对遮挡具有鲁棒性。
例如，将所述第一区域(即检测框)的长度和宽度等比例的缩小，缩小后的区域作为第二区域，该第二区域也可称为所述第一区域的中心区域。在一些实施方式中，可以基于第一图像中的第二区域的像素点和第二图像中的第二区域的像素点，确定所述第一图像和所述第二图像中的交通对象是否为第一类别的交通对象。
在一些实施方式中，可对第一图像中的第二区域的像素点进行特征提取处理，对第二图像中的第二区域的像素点进行特征提取处理，基于分别提取出的特征计算二者之间的相似程度(这里记为第一相似度)；在所述第一位置信息和所述第二位置信息满足第一预设条件以及所述第一相似度满足第二预设条件的情况下，确定所述第一图像和所述第二图像中对应于所述第二区域的交通对象为第一类别。其中，所述第一位置信息可以是第一图像中的第二区域的中心点的坐标，所述第二位置信息可以是第二图像中 的第二区域的中心点坐标。
其中，可选的，所述第一位置信息和所述第二位置信息满足第一预设条件，可以是所述第一位置信息和所述第二位置信息之间的距离(例如第一图像中的第二区域的中心点坐标与第二图像中的第二区域的中心点坐标之间的距离)小于第一阈值。其中，所述距离可以是在指定坐标系(例如像素坐标系、图像坐标系等)下的距离。例如在像素坐标系下，确定第一位置信息对应的第一中心点坐标，确定第二位置信息对应的第二中心点坐标，将第一中心点坐标和第二中心点坐标做差得到所述第一位置信息和所述第二位置信息之间的距离。实际应用中，由于图像采集的帧间隔极小，因此在不同的帧图像中，如果包括第一类别的交通对象，则该交通对象在不同的帧图像中的位置也是相近的。
可选的，所述第一相似度满足第二预设条件，可以是所述第一相似度大于或等于第二阈值；在所述第一相似度大于或等于第二阈值的情况下，可确定第一图像中第二区域的交通对象和第二图像中第二区域的交通对象为相同类别，即第一类别的对象(但可能不是相同的交通对象)；再结合上述第一位置信息和第二位置信息满足第一预设条件，可确定第一图像中对应于第二区域的交通对象和第二图像中对应于第二区域的交通对象为相同的交通对象。
在一种情况下，可分别对各图像中的第二区域进行特征提取，基于提取的特征确定各图像中的第二区域的像素点的第一相似度；在所述第一相似度满足第二预设条件的情况下，可确定各图像中的交通对象为相同类别(如第一类别)。这种情况可认定各图像中的交通对象为相同类别(如第一类别)，但可能不是同一个交通对象。在另一种情况下，在上述确定第一相似度的基础上，在各图像中的第二区域的位置信息满足第一预设条件，且所述第一相似度满足第二预设条件的情况下，可确定各图像中的交通对象为相同类别(如第一类别)。这种情况可认定各图像中的交通对象为相同类别(如第一类别)，并且是同一个交通对象。也就是说，本实施例中所述的第一类别的交通对象，不仅限于至少两帧图像中的交通对象所属类别相同，还可以包括至少两帧图像中的交通对象是相同交通对象的情况。
在实施时，确定至少两帧图像中的交通对象是相同交通对象可以采用以下方案：针对每帧图像中识别出的交通对象，均可分配其对应的唯一标识(ID)，以便于在不同帧图像之间识别出相同的交通对象；响应于多帧图像中的交通对象为相同类别(如第一类别)的情况，将为所述多帧图像中的所述交通对象分配的所述第一标识关联。
本实施例中，以多帧图像包括第一图像和第二图像为例，在识别出第一图像中的交通对象(例如记为对象1)后，为所述交通对象(对象1)分配第一标识；在确定第二图像中对应于所述第二区域的交通对象(例如记为对象2)与所述第一图像中对应于第二区域的交通对象(对象1)属于相 同类别(如第一类别)的情况下，则可将为对象1分配的第一标识与对象2进行关联，即对象1和对象2均关联第一标识，以此作为相同类别(如第一类别)的交通对象。或者，为对象1分配第一标识，为对象2分配第二标识；在确定对象1和对象2属于相同类别(如第一类别)的情况下，关联第一标识和第二标识，例如可将第二标识替换为第一标识，即对象1和对象2均关联第一标识，以此作为相同类别(如第一类别)的交通对象。
在本公开的一些可选实施例中，所述确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度，包括：确定所述多帧图像中的每帧图像中的第一类别的交通对象的置信度最大的细分类别及所述细分类别的置信度，所述第一类别为所述特定交通对象所属的类别。
相应的，所述基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息，包括：基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息。
本实施例中，电子设备先确定交通对象的第一类别(即粗分类别)，进而确定交通对象在第一类别中的细分类别，也即先对交通对象进行粗粒度分类，再对交通对象在该粗粒度分类中进行细粒度分类。
可选地，电子设备可通过第一层网络识别所述视频流中的各帧图像，确定各帧图像中包括相同类别(第一类别)的交通对象，也即检测各帧图像中的交通对象，并且确定检测到的交通对象属于相同类别(即第一类别)。示例性的，所述视频流可作为第一层网络的输入数据，通过第一层网络对视频流中的各帧图像进行特征提取，基于提取出的特征确定每帧图像中的交通对象，确定交通对象在每帧图像中的第一区域，确定交通对象的类别(如粗分类别)，也即输出每帧图像中的交通对象的检测框以及交通对象所属的类别(粗分类别)；进而从中确定属于相同类别(这里记为第一类别)的交通对象的多帧图像。其中，可选地，所述多帧图像可以是视频流中的连续的或不连续帧图像。例如，视频流包括100帧图像，确定的包含第一类别的交通对象的多帧图像可以是其中的第10帧到第50帧图像，或者也可以是从100帧图像中的第5帧、第15帧、第25帧、第35帧、第45帧图像等，本实施例中对此不做限定。
在一些可选实施例中，所述特定交通对象的所属类别(包括第一类别)为多个交通对象分类中的一个类别。可以理解，所述第一层网络是基于交通对象分类预先训练获得的，通过第一层网络对图像的处理，可以得到图像中的交通对象是否属于预标注的交通对象分类以及属于哪一种交通对象分类。
示例性的，以交通对象为交通标识(例如包括交通标志牌和道路标识)为例，由于交通标识类别众多，因此本实施例中，预先对各种交通标识进 行分类，例如图3中所示，交通标识可预先分为多个第一类别41，如速度类标识、人行道类标识、警告类标识、停止类标识等等；假设对视频流中的各帧图像中的交通对象进行识别，即可筛选出包含“速度类标识”类别的交通对象的多帧图像。实际应用中，可依据各交通标识的功能或作用进行分类。在其他实施例中，也可采用其他分类方式，本实施例对此不做限定。
在一些实施方式中，通过第一层网络确定出包括相同类别(即第一类别)的交通对象的多帧图像后，通过第二层网络对多帧图像中的每帧图像中的第一类别的交通对象进行细分类处理，得到每帧图像中的第一类别的交通对象的细分类别及该细分类别的置信度。
本实施例中，所述第二层网络可以是对应于交通对象所属类别的分类网络。可选地，第二层网络的数量可对应于交通对象所属类别的数量，也即每种交通对象所属类别均可对应一个第二层网络，每个第二层网络中均预先标注对应的交通对象所属类别中的细分类别。以图3所示的速度类标识为例，速度类标识可包括多种细分类别42，如80千米每小时(km/h)速度标识、40km/h速度标识、120km/速度标识、70km/h速度标识等等。则确定交通对象为速度类标识后，通过对应的第二层网络的分类处理，可得到该交通对象的置信度最大的细分类别及该细分类别的置信度，例如该交通对象的置信度最大的细分类别可以是70km/h速度标识。
在其他实施方式中，第二层网络也可对应于多个交通对象所属类别。以交通对象为交通标识为例，第二层网络可用于识别“单向(One)类标识”、“转向(Turn)类标识”和“车道(Lane)类标识”中的细分类别。又或者，第二层网络可包括多个用于分类处理的分支网络，每个分支网络可用于对一类或多类交通对象所属类别对应的细分类别进行识别。示例性的，电子设备在识别出交通对象所属类别(如第一类别)后，剪切出交通对象所在的第一区域对应的子图像，将该子图像输入至该第一类别对应的分支网络，以用于对第一类别下的细分类别的识别。
如此，先确定交通对象的第一类别(粗分类别)，进而确定交通对象在第一类别中的细分类别，也即先对交通对象进行粗粒度分类，再对交通对象在该粗粒度分类中进行细粒度分类，能够提高图像中的交通对象(例如交通标志牌、道路标识等)的分类精度；尤其在目前通过单层多分类器识别类别众多的交通对象的情况下，可以改善由于类别众多存在的难以标注、无法准确分类的问题。
在本公开的一些可选实施例中，所述确定所述多帧图像中的每帧图像中的第一类别的交通对象的置信度最大的细分类别及该细分类别的置信度，包括：确定所述多帧图像中的每帧图像中的第一类别的交通对象与各个第二类别的模板图像之间的第二相似度，各个第二类别为第一类别的细分类别；基于所述第二相似度确定所述每帧图像中第一类别的交通对象的置信 度最大的细分类别及该细分类别的置信度。
本实施例中，电子设备中存储有各个第二类别的模板图像。电子设备在确定交通对象的第一类别后，将图像(可以是交通对象所在区域的特征图)与各个第二类别的模板图像进行比对，确定每帧图像中的第一类别的交通对象与各个第二类别的模板图像之间的相似度(这里记为第二相似度)。其中，最大的第二相似度也可作为该交通对象的细分类别(如第二类别)的置信度，或者，交通对象的细分类别(如第二类别)的置信度也可根据最大的第二相似度计算得到。
示例性的，如图4所示，第60帧(Frame60)图像中的交通对象与各个第二类别的模板图像进行比对，确定其与“50km/h速度标识”的第二相似度为100％；第55帧(Frame55)图像中的交通对象与各个第二类别的模板图像进行比对，确定其与“60km/h速度标识”的第二相似度为50％；第51帧(Frame51)图像中的交通对象与各个第二类别的模板图像进行比对，确定其与“30km/h速度标识”的第二相似度为40％；第21帧(Frame21)图像中的交通对象与各个第二类别的模板图像进行比对，确定其与“禁止掉头标识”的第二相似度为80％。从各个第二相速度中确定置信度最大的细分类别及该细分类别的置信度，上述示例中，最大的置信度为100％，其对应的细分类别为“50km/h速度标识”。
示例性的，在多帧图像中的交通对象所属的第二类别对应的最大置信度大于或等于第三阈值的情况下，确定交通对象所属第二类别的置信度(即可靠性)为高；相应的，在所述多帧图像中的交通对象所属的第二类别对应的最大置信度小于所述第三阈值的情况下，确定交通对象所属第二类别的置信度(即可靠性)为低。如图4所示，第60帧(Frame60)图像中的交通对象所属第二类别对应的最大置信度为100％，其认定为高可靠性；而第21帧(Frame21)图像中，交通对象所属第二类别对应的最大置信度为80％，第51帧(Frame51)图像中，交通对象所属第二类别对应的最大置信度为40％，第55帧(Frame55)图像中，交通对象所属第二类别对应的最大置信度为50％，其均可认定为低可靠性。
其中，需要说明的是，上述第三阈值可依据实际情况确定。作为一种实施方式，可依据各帧图像对应的最大置信度确定第三阈值。例如100帧图像，其中最大置信度为100％的图像占比为80％，最大置信度为80％的图像占比为15％，最大置信度为50％的图像占比为5％，则可认为图像的分类结果较为可靠，则可将所述第三阈值的取值设定较高，例如设置为90％甚至95％。相应的，若图像中的最大置信度计算结果总体不太高，则可将所述第三阈值的取值设定的小一些。本实施例中对此不做限定。
本实施例中，电子设备基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息，从而对最大置信 度不满足预设条件的第一类别的交通对象的细分类别进行校正，进而提高视频流中的交通对象的分类精度，为下游的决策控制提供可靠的依据。
在本公开的一些可选实施例中，所述基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息，包括：响应于所述多帧图像中的第一图像中的第一类别的交通对象的置信度最大的细分类别的置信度满足第三预设条件，且所述多帧图像中的第二图像中的第一类别的交通对象的置信度最大的细分类别的置信度不满足第三预设条件，将所述第二图像中的第一类别的交通对象的置信度最大的细分类别设置为与所述第一图像中的第一类别的交通对象的置信度最大的细分类别相同的类别。
其中，示例性的，所述置信度满足第三预设条件可以是所述置信度大于或等于第四阈值。
本实施例中，所述多帧图像中的第一图像和第二图像的先后顺序不限，即所述第一图像在所述第二图像之后，或者，所述第一图像在所述第二图像之前。
在一些可选实施例中，所述第二图像为所述第一图像后的一帧图像。本实施例适用于对图像进行实时检测的场景。
示例性的，所述置信度满足第三预设条件，可以表明可靠程度高或可靠性高；相应的，所述置信度不满足第三预设条件，可以表明可靠程度低或可靠性低。针对相同类别(第一类别)的交通对象，在第一图像中、所述交通对象的最大的第二类别(即细分类别)的置信度高(满足第三预设条件)，且第二图像中所述交通对象的最大的第二类别(即细分类别)的置信度低(不满足第三预设条件)，则可采用第一图像中的高可靠性的细分类结果(即最大的第二类别)替换第二图像中的低可靠性的细分类结果(即最大的第二类别)。
上述实施例适用于对图像的实时检测分类过程，例如存在遮挡造成的在后的图像的分类结果不准确。例如，表1为校正前的分类示意，表2为校正后的分类示意。参照表1所示，通过对五帧图像的实时检测以及分类，得到标识ID分别为1、2、3和4的四个交通对象(其中前两帧图像的交通对象为相同交通对象，因此采用相同的ID)以及每个交通对象的第一分类和第二分类，以及对应的置信度。通过对交通对象进行跟踪确定该四个交通对象为相同交通对象，则将该四个交通对象关联相同的标识ID为1，并且采用高可靠性的细分类结果替换低可靠性的细分类结果，如表2所示，将低可靠性的细分类结果均被替换为高可靠性的细分类结果为：第一分类为速度，第二分类为50，置信度为高。
表1
标识ID 1 1 2 3 4
第一分类 速度 速度 禁止 速度 速度
第二分类 50 50 禁止通过 30 50
置信度 高 高 低 低 高
表2
标识ID 1 1 1 1 1
第一分类 速度 速度 速度 速度 速度
第二分类 50 50 50 50 50
置信度 高 高 - 高 高
本实施例通过利用高可靠性的细分类结果校正相同交通对象的低可靠性的细分类结果(置信度最大的细分类别)，实现了对最大置信度不满足预设条件的第一类别的交通对象的细分类别进行校正，能够为下游模块(例如控制模块、决策模块等)提供充分的依据，便于后续的实时控制。
在另一些可选实施例中，所述第一图像为所述第二图像后的一帧图像。本实施例适用于训练数据分类错误的场景。
本实施例中，上述图像(包括第一图像和第二图像)以及分类结果均是用于训练网络模型的。在存在利于距离远导致的分类错误的情况下，例如图1所示的场景，第21帧(Frame21)帧图像由于距离较远，识别出的细分类结果为低可靠性(或置信度)；而随着逐渐靠近交通对象，相应的采集到的图像越来愈清晰，细分类结果的可靠性也会变化，例如第60帧(Frame60)的分类结果处于高可靠性(或置信度)。则为了优化训练数据，可将在后的、处于高可靠性的第一图像的细分类结果替换在先的、处于低可靠性的第二图像的细分类结果。
例如，表3为校正前的分类示意，表4为校正后的分类示意。参照表3所示，通过对五帧图像的实时检测以及分类，得到标识ID分别为1、2、3和4的四个交通对象(其中前两帧图像的交通对象为相同交通对象，因此采用相同的ID)以及每个交通对象的第一分类和第二分类，以及对应的置信度。通过对交通对象进行跟踪确定该四个交通对象为相同交通对象，则将该四个交通对象关联相同的标识ID为1，参照表4所示。对于相同类别(第一类别)的交通对象，由于前几帧图像的细分类结果的可靠性低(不满足第三预设条件)，第五帧图像中该相同交通对象的细分类结果的可靠性高(满足第三预设条件)，则将前几帧对应的低可靠性的细分类结果均被替换为高可靠性的细分类结果，替换的细分类结果为：第一分类为速度，第二分类为50，置信度为高。
表3
标识ID 1 1 2 3 4
第一分类 禁止 速度 速度 速度 速度
第二分类 速度 30 30 60 50
置信度 低 低 低 低 高
表4
标识ID 1 1 1 1 1
第一分类 速度 速度 速度 速度 速度
第二分类 50 50 50 50 50
置信度 高 高 高 高 高
在本公开的一些可选实施例中，所述基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息，还可以包括：响应于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度均不满足第三预设条件，输出表示所述交通对象的分类结果不能确定的提示信息。
本实施例中，若多帧图像中、每帧图像中第一类别的交通对象的置信度最大的细分类别的置信度均不满足第三预设条件，即每帧图像中第一类别的交通对象的置信度最大的细分类别的置信度均小于第四阈值，也即每帧图像中第一类别的交通对象的置信度最大的细分类别的可靠性较低，则不能确定该交通对象的细分类别，输出表示分类结果不能确定的提示信息。
例如，参照前述表3的分类结果，通过对五帧图像的实时检测以及分类，得到标识ID分别为1、2、3和4的四个交通对象(其中前两帧图像的交通对象为相同交通对象，因此采用相同的ID)以及每个交通对象的第一分类和第二分类，以及对应的置信度。通过对交通对象进行跟踪确定该四个交通对象为相同交通对象，则将该四个交通对象关联相同的标识ID为1，参照表5所示。假定当前采集到第四帧图像，由于前三帧图像中相同交通对象对应的第二分类的分类结果均是低置信度，第四帧图像中相同交通对象对应的第二分类的分类结果依旧是低置信度(不满足第三预设条件)，则输出表示分类结果不能确定的提示信息。在一些实施方式中，随着采集到第五帧图像，检测到第五帧图像中相同交通对象对应的第二分类的分类结果是高置信度(满足第三预设条件)，则可输出高置信度的分类结果，即速度为50km/h的细分类结果。
表5
标识ID 1 1 2 3 4
第一分类 ？？ ？？ ？？ ？？ 速度
第二分类 ？？ ？？ ？？ ？？ 50
置信度 ？？ ？？ ？？ ？？ 高
基于上述方法实施例，本公开实施例还提供了一种图像处理装置。图5为本公开实施例提供的一种图像处理装置的组成结构示意图；如图5所示，所述装置包括：获取单元21、第一确定单元22、第二确定单元23和第三确定单元24；其中，
所述获取单元21，被配置为获取行驶设备上安装的图像采集装置采集的视频流；
所述第一确定单元22，被配置为从所述视频流中确定包含特定交通对象的多帧图像；
所述第二确定单元23，被配置为确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度；
所述第三确定单元24，被配置为基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。
在本公开的一些可选实施例中，所述第一确定单元22，被配置为确定所述视频流中的包含交通对象的图像中的交通对象在图像中的第一区域；针对每个第一区域，在所述第一区域内确定第二区域，所述第二区域小于所述第一区域；基于各个第二区域的信息，从所述包含交通对象的图像中挑选包含第一类别的交通对象的图像，将多帧包含第一类别的交通对象的图像作为包含特定交通对象的多帧图像，所述第一类别为所述特定交通对象所属的类别。
在本公开的一些可选实施例中，所述第二区域为所述第一区域的中心区域，所述第二区域的信息为所述第一区域的特征图的中心区域的信息。
在本公开的一些可选实施例中，所述第一确定单元22，被配置为分别对所述各个第二区域进行特征提取，基于提取的特征确定所述各个第二区域的像素点的第一相似度；将位置信息满足第一预设条件、且所述第一相似度满足第二预设条件的第二区域所在的图像确定为包含第一类别的交通对象的图像。
在本公开的一些可选实施例中，所述第二确定单元23，被配置为确定所述多帧图像中的每帧图像中的第一类别的交通对象的置信度最大的细分类别及所述细分类别的置信度，所述第一类别为所述特定交通对象所属的类别。
在本公开的一些可选实施例中，所述第三确定单元24，被配置为基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息。
在本公开的一些可选实施例中，所述第二确定单元23，被配置为确定所述多帧图像中的每帧图像中的第一类别的交通对象与各个第二类别的模板图像之间的第二相似度，各个第二类别为第一类别的细分类别；基于所 述第二相似度确定所述每帧图像中第一类别的交通对象的置信度最大的细分类别及所述细分类别的置信度。
在本公开的一些可选实施例中，所述第三确定单元24，被配置为响应于所述多帧图像中的第一图像中的第一类别的交通对象的置信度最大的细分类别的置信度满足第三预设条件，且所述多帧图像中的第二图像中的第一类别的交通对象的置信度最大的细分类别的置信度不满足第三预设条件，将所述第二图像中的第一类别的交通对象的置信度最大的细分类别设置为与所述第一图像中的第一类别的交通对象的置信度最大的细分类别相同的类别。
在本公开的一些可选实施例中，所述第三确定单元24，被配置为响应于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度均不满足第三预设条件，输出表示所述交通对象的分类结果不能确定的提示信息。
本公开实施例中，所述装置中的获取单元21、第一确定单元22、第二确定单元23和第三确定单元24，在实际应用中均可由中央处理器(CPU，Central Processing Unit)、数字信号处理器(DSP，Digital Signal Processor)、微控制单元(MCU，Microcontroller Unit)或可编程门阵列(FPGA，Field－Programmable Gate Array)实现。
需要说明的是：上述实施例提供的图像处理装置在进行图像处理时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将装置的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的图像处理装置与图像处理方法实施例属于同一构思，其实现过程可以参照方法实施例。
本公开实施例还提供了一种电子设备，图6为本公开实施例提供的一种电子设备的硬件组成结构示意图，如图6所示，所述电子设备包括存储器32、处理器31及存储在存储器32上并可在处理器31上运行的计算机程序，所述处理器31执行所述程序时实现本公开实施例所述图像处理方法的步骤。
可选地，所述电子设备还可包括用户接口33和网络接口34。其中，用户接口33可以包括显示器、键盘、鼠标、轨迹球、点击轮、按键、按钮、触感板或者触摸屏等。
可选地，电子设备中的各个组件通过总线系统35耦合在一起。可理解，总线系统35用于实现这些组件之间的连接通信。总线系统35除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图6中将各种总线都标为总线系统35。
可以理解，存储器32可以是易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储 器(ROM，Read Only Memory)、可编程只读存储器(PROM，Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM，Ferromagnetic Random Access Memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM，Compact Disc Read-Only Memory)；磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM，Random Access Memory)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM，Static Random Access Memory)、同步静态随机存取存储器(SSRAM，Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM，Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM，Synchronous Dynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM，Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM，Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM，SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM，Direct Rambus Random Access Memory)。本公开实施例描述的存储器32旨在包括但不限于这些和任意其它适合类型的存储器。
上述本公开实施例揭示的方法可以应用于处理器31中，或者由处理器31实现。处理器31可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器31中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器31可以是通用处理器、DSP，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。处理器31可以实现或者执行本公开实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本公开实施例所公开的方法的步骤，可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中，该存储介质位于存储器32，处理器31读取存储器32中的信息，结合其硬件完成前述方法的步骤。
在示例性实施例中，电子设备可以被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，Programmable Logic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable Logic Device)、FPGA、通用处理器、控制器、MCU、微处理器(Microprocessor)、或其他电子元件实现，用于执行前述方法。
在示例性实施例中，本公开实施例还提供了一种计算机可读存储介质，例如包括计算机程序的存储器32，上述计算机程序可由电子设备的处理器 31执行，以完成前述方法所述步骤。计算机可读存储介质可以是FRAM、ROM、PROM、EPROM、EEPROM、Flash Memory、磁表面存储器、光盘、或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备。
本公开实施例提供的计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本公开实施例所述的图像处理方法的步骤。
本公开实施例提供一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现本公开实施例所述图像处理方法中的部分或全部步骤。
本申请所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得到新的方法实施例。
本申请所提供的几个产品实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的产品实施例。
本申请所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
在本申请所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元，即可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。
另外，在本公开各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
或者，本公开上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。 基于这样的理解，本公开实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。
工业实用性
本公开实施例公开了一种图像处理方法、装置、电子设备、存储介质及计算机程序产品。所述方法包括：获取行驶设备上安装的图像采集装置采集的视频流，从所述视频流中确定包含特定交通对象的多帧图像；确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度；基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。在本公开实施例中，根据多帧图像中的特定交通对象的类别的置信度的比较结果，对置信度不满足预设条件的类别进行校正，即利用特定交通对象的高可靠性的分类结果，对低可靠性的分类结果进行校正，一方面能够提高图像中的交通对象的分类准确度，另一方面能够为下游的决策控制提供可靠的依据，便于后续的实时控制。

Claims (13)

  1. 一种图像处理方法，所述方法包括：
    获取行驶设备上安装的图像采集装置采集的视频流，从所述视频流中确定包含特定交通对象的多帧图像；
    确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度；
    基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。
  2. 根据权利要求1所述的方法，其中，从所述视频流中确定包含特定交通对象的多帧图像，包括：
    确定所述视频流中的包含交通对象的图像中的交通对象在图像中的第一区域；
    针对每个第一区域，在所述第一区域内确定第二区域，所述第二区域小于所述第一区域；
    基于各个第二区域的信息，从所述包含交通对象的图像中挑选包含第一类别的交通对象的图像作为包含特定交通对象的多帧图像，所述第一类别为所述特定交通对象所属的类别。
  3. 根据权利要求2所述的方法，其中，所述第二区域为所述第一区域的中心区域，所述第二区域的信息为所述第一区域的特征图的中心区域的信息。
  4. 根据权利要求2所述的方法，其中，所述基于各个第二区域的信息，从所述包含交通对象的图像中挑选包含第一类别的交通对象的图像，包括：
    分别对所述各个第二区域进行特征提取，基于提取的特征确定所述各个第二区域的像素点的第一相似度；
    将位置信息满足第一预设条件、且所述第一相似度满足第二预设条件的第二区域所在的图像确定为包含第一类别的交通对象的图像。
  5. 根据权利要求1所述的方法，其中，所述确定所述多帧图像中的每帧图像中的特定交通对象的类别及所述类别的置信度，包括：
    确定所述多帧图像中的每帧图像中的第一类别的交通对象的置信度最大的细分类别及所述细分类别的置信度，所述第一类别为所述特定交通对象所属的类别。
  6. 根据权利要求5所述的方法，其中，所述基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息，包括：基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息。
  7. 根据权利要求5所述的方法，其中，所述确定所述多帧图像中的每 帧图像中的第一类别的交通对象的置信度最大的细分类别及所述细分类别的置信度，包括：
    确定所述多帧图像中的每帧图像中的第一类别的交通对象与各个第二类别的模板图像之间的第二相似度，各个第二类别为第一类别的细分类别；
    基于所述第二相似度确定所述每帧图像中第一类别的交通对象的置信度最大的细分类别及所述细分类别的置信度。
  8. 根据权利要求6所述的方法，其中，所述基于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度的比较结果，确定最大置信度不满足预设条件的第一类别的交通对象的细分类别的校正信息，包括：
    响应于所述多帧图像中的第一图像中的第一类别的交通对象的置信度最大的细分类别的置信度满足第三预设条件，且所述多帧图像中的第二图像中的第一类别的交通对象的置信度最大的细分类别的置信度不满足第三预设条件，将所述第二图像中的第一类别的交通对象的置信度最大的细分类别设置为与所述第一图像中的第一类别的交通对象的置信度最大的细分类别相同的类别。
  9. 根据权利要求8所述的方法，其中，所述方法还包括：
    响应于所述多帧图像中的所述第一类别的交通对象的置信度最大的细分类别的置信度均不满足第三预设条件，输出表示所述交通对象的分类结果不能确定的提示信息。
  10. 一种图像处理装置，所述装置包括：获取单元、第一确定单元、第二确定单元和第三确定单元；其中，
    所述获取单元，被配置为获取行驶设备上安装的图像采集装置采集的视频流；
    所述第一确定单元，被配置为从所述视频流中确定包含特定交通对象的多帧图像；
    所述第二确定单元，被配置为确定所述多帧图像中的每帧图像中的特定交通对象的类别的置信度；
    所述第三确定单元，被配置为基于所述多帧图像中的特定交通对象的类别的置信度的比较结果，确定置信度不满足预设条件的类别的校正信息。
  11. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现权利要求1至9任一项所述方法的步骤。
  12. 一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1至9任一项所述方法的步骤。
  13. 一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现权利要求1至9任一项所述方法中的步骤。
PCT/CN2022/129070 2022-03-24 2022-11-01 一种图像处理方法、装置、电子设备、存储介质及计算机程序产品 WO2023179031A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210301591.0 2022-03-24
CN202210301591.0A CN114694066A (zh) 2022-03-24 2022-03-24 一种图像处理方法、装置、电子设备和存储介质

Publications (1)

Publication Number Publication Date
WO2023179031A1 true WO2023179031A1 (zh) 2023-09-28

Family

ID=82138801

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/129070 WO2023179031A1 (zh) 2022-03-24 2022-11-01 一种图像处理方法、装置、电子设备、存储介质及计算机程序产品

Country Status (2)

Country Link
CN (1) CN114694066A (zh)
WO (1) WO2023179031A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114694066A (zh) * 2022-03-24 2022-07-01 商汤集团有限公司 一种图像处理方法、装置、电子设备和存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170061241A1 (en) * 2015-08-31 2017-03-02 Sony Corporation Method and system to adaptively track objects
CN109740517A (zh) * 2018-12-29 2019-05-10 上海依图网络科技有限公司 一种确定待识别对象的方法及装置
CN111274426A (zh) * 2020-01-19 2020-06-12 深圳市商汤科技有限公司 类别标注方法及装置、电子设备和存储介质
CN113705406A (zh) * 2021-08-19 2021-11-26 上海商汤临港智能科技有限公司 交通指示信号的检测方法及相关装置、设备、介质
CN114694066A (zh) * 2022-03-24 2022-07-01 商汤集团有限公司 一种图像处理方法、装置、电子设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170061241A1 (en) * 2015-08-31 2017-03-02 Sony Corporation Method and system to adaptively track objects
CN109740517A (zh) * 2018-12-29 2019-05-10 上海依图网络科技有限公司 一种确定待识别对象的方法及装置
CN111274426A (zh) * 2020-01-19 2020-06-12 深圳市商汤科技有限公司 类别标注方法及装置、电子设备和存储介质
CN113705406A (zh) * 2021-08-19 2021-11-26 上海商汤临港智能科技有限公司 交通指示信号的检测方法及相关装置、设备、介质
CN114694066A (zh) * 2022-03-24 2022-07-01 商汤集团有限公司 一种图像处理方法、装置、电子设备和存储介质

Also Published As

Publication number Publication date
CN114694066A (zh) 2022-07-01

Similar Documents

Publication Publication Date Title
US11600006B2 (en) Deep neural network architecture for image segmentation
US8725394B2 (en) Multi-modal speed limit assistant
Rezaei et al. Robust vehicle detection and distance estimation under challenging lighting conditions
Du et al. Comprehensive and practical vision system for self-driving vehicle lane-level localization
US20200380859A1 (en) Method and system for determining driving assisting data
WO2021063006A1 (zh) 驾驶预警方法、装置、电子设备和计算机存储介质
CN110929655B (zh) 一种行驶过程中车道线识别方法、终端设备及存储介质
JP2022507077A (ja) 区画線属性検出方法、装置、電子機器及び可読記憶媒体
US20210174113A1 (en) Method for limiting object detection area in a mobile system equipped with a rotation sensor or a position sensor with an image sensor, and apparatus for performing the same
CN110008891B (zh) 一种行人检测定位方法、装置、车载计算设备及存储介质
WO2023179031A1 (zh) 一种图像处理方法、装置、电子设备、存储介质及计算机程序产品
US20170314936A1 (en) Method and apparatus for disambiguating probe points within an ambiguous probe region
Liu et al. Vehicle detection and ranging using two different focal length cameras
Tak et al. Development of AI‐Based Vehicle Detection and Tracking System for C‐ITS Application
US20230085455A1 (en) Vehicle condition estimation method, vehicle condition estimation device, and vehicle condition estimation program
CN114705121A (zh) 车辆位姿测量方法、装置及电子设备、存储介质
Sung et al. Real-time traffic light recognition on mobile devices with geometry-based filtering
US20200103918A1 (en) Method for detecting caller by autonomous vehicle
Ashraf et al. HVD-net: a hybrid vehicle detection network for vision-based vehicle tracking and speed estimation
CN113029185A (zh) 众包式高精度地图更新中道路标线变化检测方法及系统
CN113238209A (zh) 基于毫米波雷达的道路感知方法、系统、设备及存储介质
CN109740502B (zh) 道路质量检测方法及装置
WO2023179030A1 (zh) 一种道路边界检测方法、装置、电子设备、存储介质和计算机程序产品
CN111353481A (zh) 基于激光点云与视频图像的道路障碍物识别方法
Xiong et al. Fast and robust approaches for lane detection using multi‐camera fusion in complex scenes

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22933082

Country of ref document: EP

Kind code of ref document: A1