WO2022027873A1 - 基于多模态信息融合的车辆再识别方法及装置 - Google Patents

基于多模态信息融合的车辆再识别方法及装置 Download PDF

Info

Publication number
WO2022027873A1
WO2022027873A1 PCT/CN2020/131777 CN2020131777W WO2022027873A1 WO 2022027873 A1 WO2022027873 A1 WO 2022027873A1 CN 2020131777 W CN2020131777 W CN 2020131777W WO 2022027873 A1 WO2022027873 A1 WO 2022027873A1
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
queried
license plate
features
different vehicles
Prior art date
Application number
PCT/CN2020/131777
Other languages
English (en)
French (fr)
Inventor
闫军
刘艳洋
Original Assignee
智慧互通科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 智慧互通科技有限公司 filed Critical 智慧互通科技有限公司
Publication of WO2022027873A1 publication Critical patent/WO2022027873A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/625—License plates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/08—Detecting or categorising vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Definitions

  • the invention relates to the field of intelligent parking management, in particular to a vehicle re-identification method and device based on multimodal information fusion.
  • Vehicle re-identification is to find the same vehicle shot by different cameras, or the same vehicle shot by the same camera under different lighting and different viewing angles. Through vehicle re-identification technology, it is possible to automatically identify and lock the same vehicle across cameras. It plays a very important role in specific tasks such as urban traffic scheduling and illegal vehicle tracking, and is beneficial to the planning and development of intelligent transportation and smart cities.
  • the current common vehicle re-identification methods usually include the following types: One is to use sensors to solve the problem of vehicle re-identification. For example, vehicle re-identification is carried out through geomagnetic sensors, infrared radio frequency sensors, etc. However, this method requires high cost and complex installation environment requirements, which is not suitable for large-scale promotion and use.
  • the second method is to recognize the license plate to realize the tracking and positioning of the vehicle across the cameras. However, in many cases, the license plate number is not accurately recognized due to various reasons such as illumination, occlusion, and contamination, which leads to the re-identification of the vehicle. The error is large; the third method is to re-identify the vehicle based on the local appearance features of the vehicle. However, because the local appearance features of the vehicle reflect the vehicle in a one-sided manner, the progress of vehicle re-identification is low.
  • the present invention provides a vehicle re-identification method and device based on multi-modal information fusion, which can solve the problem that the existing re-identification method only uses the vehicle appearance or the license plate number to re-identify the vehicle, and the applied identification features It is a relatively simple problem with low recognition accuracy.
  • a vehicle re-identification method based on multimodal information fusion is characterized in that, the method includes:
  • a vehicle re-identification ranking table is generated according to the similarity or the joint probability of matching between different vehicles and the vehicle to be queried.
  • the method also includes:
  • the spatiotemporal matching probability between the vehicle and different vehicles is obtained.
  • the preset vehicle spatiotemporal transfer model is based on different vehicles.
  • the corresponding historical vehicle driving time and space data are respectively established.
  • the step of generating a vehicle re-identification ranking table according to the similarity between different vehicles and the vehicle to be queried or the joint probability of matching includes:
  • the vehicle re-identification ranking table is generated according to the joint probability or similarity of matching between different vehicles and the vehicle to be queried, and the spatiotemporal matching probability between the vehicles and different vehicles.
  • the step of extracting the image of the vehicle to be queried from the surveillance video includes:
  • the redundant background information in the vehicle target image is eliminated to obtain the vehicle image to be queried.
  • the global feature is the overall appearance feature of the vehicle, and the multiple local features include the appearance feature of the front of the vehicle, the appearance feature of the rear of the vehicle, and the appearance feature of the license plate area of the vehicle.
  • the step of obtaining the similarity between different vehicles and the vehicle's global features, multiple local features and license plate characters from a preset database includes:
  • vehicle front appearance characteristics e.g., vehicle rear appearance characteristics, vehicle license plate area appearance characteristics, and license plate character characteristics
  • construct a ternary loss function e.g., a ternary loss function and calculate the difference between different vehicles and the vehicle.
  • the feature distance is used to obtain the similarity between different vehicles and the vehicle.
  • the step of obtaining the joint probability of matching between different vehicles and the vehicle includes:
  • the present invention provides a vehicle re-identification device based on multimodal information fusion, the device comprising:
  • the extraction module is used to extract the image of the vehicle to be queried from the surveillance video
  • the extraction module is further configured to use the image of the vehicle to be queried as the input of the preset multi-branch monitoring mechanism network model, extract the global feature and multiple local features of the vehicle, and extract the vehicle's global features and multiple local features according to the license plate area in the multiple local features. extracting the license plate characters of the vehicle;
  • the acquisition module is used to acquire the similarity or the joint probability of matching between different vehicles and the global features of the vehicle, multiple local features and license plate characters respectively from a preset database, where different vehicles are stored in the preset database. respectively corresponding global features, multiple local features and license plate characters, and the preset database stores the global features, multiple local features and license plate characters corresponding to different vehicles respectively;
  • the generating module is configured to generate a vehicle re-identification ranking table according to the similarity or the joint probability of matching between different vehicles and the vehicle to be queried.
  • the apparatus further includes: a prediction module
  • the extraction module is further configured to extract the spatiotemporal information of the image of the vehicle to be queried from the surveillance video;
  • the prediction module is configured to predict the relative driving direction of the vehicle according to the joint probability of matching between different vehicles and the vehicle to be queried and the spatiotemporal information of the image of the vehicle to be queried;
  • the obtaining module is further configured to obtain the space-time matching probability between the vehicle and different vehicles according to the topological relationship between different cameras, the relative driving direction of the vehicle, and the preset vehicle space-time transfer model.
  • the vehicle spatiotemporal transfer model is established based on the historical vehicle driving spatiotemporal data corresponding to different vehicles.
  • the generating module is specifically configured to generate a vehicle re-identification ranking table according to the joint probability or similarity of matching between different vehicles and the vehicle to be queried, and the spatiotemporal matching probability between the vehicles and different vehicles.
  • the extraction module is specifically used to perform vehicle target detection on the image of the vehicle to be queried obtained from the surveillance video, and obtain the vehicle target image that is completely marked by the bounding box; and remove the redundancy in the vehicle target image.
  • the background information is obtained, and the image of the vehicle to be queried is obtained.
  • the global feature is the overall appearance feature of the vehicle, and the multiple local features include the appearance feature of the front of the vehicle, the appearance feature of the rear of the vehicle, and the appearance feature of the license plate area of the vehicle.
  • the acquisition module is specifically configured to construct a ternary loss function according to different vehicles and the overall appearance characteristics of the vehicles, vehicle front appearance characteristics, vehicle rear appearance characteristics, vehicle license plate area appearance characteristics, and license plate character characteristics. And calculate the characteristic distance between different vehicles and the vehicle, and obtain the similarity between the different vehicles and the vehicle respectively.
  • the invention provides a vehicle re-identification method and device based on multi-modal information fusion.
  • a preset multi-branch monitoring mechanism network model By incorporating a preset multi-branch monitoring mechanism network model, and by comparing the global features, multiple local features and license plate characters of the vehicle, the realization of The feature matching based on the fusion of global appearance features, local appearance features and license plate characters can effectively improve the accuracy of vehicle re-identification compared with the existing feature matching that only uses a single feature or license plate characters. It also integrates the time and geographic location information corresponding to the vehicle to form a multi-modal fusion probability feature, which can well distinguish the inter-class and intra-class differences between vehicles, and can adapt to different lighting and different camera models in real scenes. Identified application requirements.
  • FIG. 1 is a flowchart of a vehicle re-identification method based on multimodal information fusion provided by the present invention
  • FIG. 2 is a schematic diagram of a vehicle re-identification device based on multimodal information fusion provided by the present invention.
  • the present invention provides a parking event determination method based on image depth information, as shown in FIG. 1 , which specifically includes the following steps:
  • step 101 may specifically include: performing vehicle target detection on the image of the vehicle to be queried obtained from the surveillance video, and obtaining a vehicle target image completely marked by a bounding box; The remaining background information is obtained to obtain the image of the vehicle to be queried.
  • step 101 may specifically include: performing vehicle target detection on the image of the vehicle to be queried obtained from the surveillance video, and obtaining a vehicle target image completely marked by a bounding box; The remaining background information is obtained to obtain the image of the vehicle to be queried.
  • the interference of background noise is removed, and all information of the vehicle can be fully extracted, thereby ensuring the clarity of the extracted image, and further Improve vehicle re-identification accuracy.
  • the specific process of performing vehicle target detection on the picture of the vehicle to be queried obtained from the surveillance video may include the following steps: when a surveillance camera is capturing a video frame, simultaneously acquiring the geographic location information of the surveillance camera, and photographing the surveillance camera Time information for video frames.
  • a single-stage detection method is used as a target detection tool, and then vehicle target detection is performed on the vehicle image obtained from the source monitoring video stream, and the vehicle target completely marked by the bounding box is obtained as a vehicle instance segmentation source data, and Ignore pictures with a resolution less than 256*256, and mark the ID corresponding to each vehicle in the image after vehicle detection.
  • the geographic location includes but is not limited to country, city, latitude and longitude coordinates, and the like.
  • the time includes, but is not limited to, year, month, day, hour, minute, second, lunar calendar, solar calendar, and the like.
  • the single-stage detection methods include but are not limited to regression-type target detection methods such as YOLO and SSD.
  • the source vehicle image covers multiple images of the same vehicle from different viewing angles, different backgrounds, and different light intensities captured by real road traffic scene monitoring, wherein the content covered by the vehicle image includes: license plate appearance information, license plate character information , color, model, vehicle label information.
  • the steps of obtaining the vehicle target image completely marked by the bounding box, eliminating redundant background information in the vehicle target image, and obtaining the vehicle image to be queried may specifically include: using a polygon labeling tool to convert the vehicle image data.
  • the centralized vehicle target is cut out along the vehicle boundary to make a vehicle instance segmentation dataset.
  • Use the convolutional neural network to generate feature vectors, train the vehicle instance segmentation network model, and then perform preliminary instance segmentation on the vehicle image, and then perform data cleaning on the preliminary segmentation results, such as edge detection, hole filling, connected domain detection, and pixel area comparison. , improve the vehicle and instance segmentation effect, and obtain the segmented vehicle dataset.
  • the instance segmentation methods include but are not limited to instance segmentation methods such as MASK-RCNN and SOLO.
  • the convolutional neural network includes but is not limited to convolutional neural networks such as VGG, AlexNet, and ResNet.
  • the edge detection includes, but is not limited to, Gaussian filtering, Canny, Sobel and other edge detection methods.
  • the hole filling includes, but is not limited to, pixel filling methods such as opening operation, closing operation, and flood filling.
  • the global feature is the overall appearance feature of the vehicle, and the multiple local features include the appearance feature of the front of the vehicle, the appearance feature of the rear of the vehicle, and the appearance feature of the license plate area of the vehicle.
  • the preset multi-branch monitoring mechanism network model includes but is not limited to detection models such as SSD, YOLO, M2det, Fast-RCNN, etc.
  • the data set used by the detection model is the data set of the vehicle image after instance segmentation with background noise removed.
  • the license plate character recognition methods include but are not limited to CRNN+CTC, YOLOV3 and other character recognition methods, thereby ensuring the accuracy of license plate recognition.
  • the preset database stores global features, multiple local features, and license plate characters corresponding to different vehicles
  • the preset database stores global features, multiple local features, and license plate characters corresponding to different vehicles.
  • the step of acquiring the similarity between different vehicles and the vehicle's global features, multiple local features and license plate characters from the preset database includes: according to different vehicles and the overall appearance features of the vehicle , vehicle front appearance characteristics, vehicle rear appearance characteristics, vehicle license plate area appearance characteristics, and license plate character characteristics, construct a ternary loss function and calculate the feature distance between different vehicles and the vehicle, and obtain different vehicles and the vehicle respectively. similarity between.
  • the convolution layer is used to extract the initial features of each vehicle image
  • the multi-branch detection module mechanism is used to extract the four parts of local features, namely, the front appearance features, the rear appearance features, the appearance features of the license plate, and the character features of the license plate. Fusion, the four parts of local features and global features are fused to obtain a re-identification model.
  • loss function is a triple loss function:
  • f(P) and f(N) are images of vehicles with the same ID captured by different cameras
  • f(A) is an image of vehicles with different IDs.
  • the extraction of multiple local features and global features requires setting initial parameters of the convolutional neural network for extracting features from different regions.
  • the vehicle identity feature is output;
  • the feature fusion network adopts a 5-layer fully connected layer neural network, and the output after the first fully connected layer is taken as the fusion feature of the vehicle.
  • the feature extraction network includes but is not limited to feature extraction network models such as ResNet, VGG, and AlexNet.
  • the method of training the feature fusion network is as follows: the metric learning of the cross-entropy loss function and the triplet loss function is used to train the feature learning process of the network; the 5-layer fully connected layer neural network of the feature fusion network is trained using the loss function of the metric learning.
  • the intra-class distance of the same ID of the vehicle is reduced, the distance between the different ID classes of the vehicle is enlarged, and the robustness of the vehicle fusion feature is enhanced.
  • the license plate matching probability between the vehicles to be queried, and ⁇ is the confidence level of the license plate recognition.
  • the vehicle re-identification ranking table may be arranged in descending order of similarity or probability.
  • the spatiotemporal information of the vehicle may also be incorporated, and the specific method may include: extracting the spatiotemporal information of the image of the vehicle to be queried from the surveillance video; According to the joint probability of matching between the joint probability and the spatiotemporal information of the vehicle image to be queried, the relative driving direction of the vehicle is predicted; The space-time matching probability between the vehicle and different vehicles, and the preset vehicle space-time transfer model is established according to the historical vehicle driving space-time data corresponding to different vehicles respectively.
  • step 104 may specifically include: generating a vehicle re-identification ranking table according to the joint probability or similarity of matching between different vehicles and the vehicle to be queried, and the spatiotemporal matching probability between the vehicles and different vehicles.
  • the invention provides a vehicle re-identification method based on multi-modal information fusion.
  • a preset multi-branch monitoring mechanism network model By integrating a preset multi-branch monitoring mechanism network model, the global feature, multiple local features and license plate characters of the vehicle are compared respectively.
  • the feature matching that combines appearance features, local appearance features and license plate characters can effectively improve the accuracy of vehicle re-identification compared with the existing feature matching that only uses a single feature or license plate characters.
  • the time and geographic location information corresponding to the vehicle is integrated to form a multi-modal fusion probability feature, which can well distinguish the inter-class difference and intra-class difference between vehicles, and can adapt to different lighting and different camera models in real scenes. Application requirements.
  • an embodiment of the present invention provides a vehicle re-identification device based on multimodal information fusion.
  • the device includes: an extraction module 21 , which is used for extracting data from surveillance video extract the image of the vehicle to be queried;
  • the extraction module 21 is further configured to use the image of the vehicle to be queried as the input of the preset multi-branch monitoring mechanism network model, extract the global feature and multiple local features of the vehicle, and extract the vehicle's global features and multiple local features according to the license plate in the multiple local features. region to extract the license plate characters of the vehicle;
  • the obtaining module 22 is used to obtain the similarity or the joint probability of matching between different vehicles and the global features, multiple local features and license plate characters of the vehicle respectively from a preset database, and the preset database saves different Global features, multiple local features and license plate characters corresponding to the vehicles respectively, and the preset database stores the global features, multiple local features and license plate characters corresponding to different vehicles respectively;
  • the generating module 23 is configured to generate a vehicle re-identification ranking table according to the similarity or the joint probability of matching between different vehicles and the vehicle to be queried.
  • the apparatus further includes: a prediction module 24;
  • the extraction module 21 is further configured to extract the spatiotemporal information of the image of the vehicle to be queried from the surveillance video;
  • the prediction module 24 is configured to predict the relative driving direction of the vehicle according to the joint probability of matching between different vehicles and the vehicle to be queried and the spatiotemporal information of the image of the vehicle to be queried;
  • the obtaining module 22 is further configured to obtain the space-time matching probability between the vehicle and different vehicles according to the topological relationship between different cameras, the relative driving direction of the vehicle, and the preset vehicle space-time transfer model.
  • the vehicle spatiotemporal transfer model is established based on the historical vehicle driving spatiotemporal data corresponding to different vehicles.
  • the generating module 23 is specifically configured to generate a vehicle re-identification ranking table according to the joint probability or similarity of matching between different vehicles and the vehicle to be queried, and the spatiotemporal matching probability between the vehicles and different vehicles.
  • the extraction module 21 is specifically used to detect the vehicle target on the to-be-queried vehicle picture obtained from the surveillance video, and obtain the vehicle target image completely marked by the bounding box; The remaining background information is obtained to obtain the image of the vehicle to be queried.
  • the extraction module 21 by detecting the vehicle category of the original monitoring image and performing instance segmentation, the interference of background noise is removed, and all information of the vehicle can be fully extracted, thereby ensuring the clarity of the extracted image, and further Improve vehicle re-identification accuracy.
  • the global feature is the overall appearance feature of the vehicle, and the multiple local features include the appearance feature of the front of the vehicle, the appearance feature of the rear of the vehicle, and the appearance feature of the license plate area of the vehicle.
  • the acquisition module 22 is specifically configured to construct a ternary loss according to different vehicles and the overall appearance characteristics of the vehicles, the vehicle front appearance characteristics, the vehicle rear appearance characteristics, the vehicle license plate area appearance characteristics, and the license plate character characteristics. function and calculate the characteristic distance between different vehicles and the vehicle to obtain the similarity between the different vehicles and the vehicle.
  • the invention provides a vehicle re-identification device based on multi-modal information fusion.
  • a preset multi-branch monitoring mechanism network model By integrating a preset multi-branch monitoring mechanism network model, the global feature, multiple local features and license plate characters of the vehicle are compared respectively.
  • the feature matching that combines appearance features, local appearance features and license plate characters can effectively improve the accuracy of vehicle re-identification compared with the existing feature matching that only uses a single feature or license plate characters.
  • the time and geographic location information corresponding to the vehicle is integrated to form a multi-modal fusion probability feature, which can well distinguish the inter-class difference and intra-class difference between vehicles, and can adapt to different lighting and different camera models in real scenes. Application requirements.
  • a general-purpose processor may be a microprocessor, or alternatively, the general-purpose processor may be any conventional processor, controller, microcontroller, or state machine.
  • a processor may also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a digital signal processor core, or any other similar configuration. accomplish.
  • the steps of the method or algorithm described in the embodiments of the present invention may be directly embedded in hardware, a software module executed by a processor, or a combination of the two.
  • Software modules may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
  • a storage medium may be coupled to the processor such that the processor may read information from, and store information in, the storage medium.
  • the storage medium can also be integrated into the processor.
  • the processor and storage medium may be provided in the ASIC, and the ASIC may be provided in the user terminal. Alternatively, the processor and the storage medium may also be provided in different components in the user terminal.
  • the above functions described in the embodiments of the present invention may be implemented in hardware, software, firmware, or any combination of the three. If implemented in software, the functions may be stored on, or transmitted over, a computer-readable medium in the form of one or more instructions or code.
  • Computer-readable media includes computer storage media and communication media that facilitate the transfer of a computer program from one place to another. Storage media can be any available media that a general-purpose or special-purpose computer can access.
  • Such computer-readable media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other device that can be used to carry or store instructions or data structures and Other media in the form of program code that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor.
  • any connection is properly defined as a computer-readable medium, for example, if software is transmitted from a web site, server or other remote source over a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) Or transmitted by wireless means such as infrared, wireless, and microwave are also included in the definition of computer-readable media.
  • DSL digital subscriber line
  • the disks and disks include compact disks, laser disks, optical disks, DVDs, floppy disks and blu-ray disks. Disks usually reproduce data magnetically, while discs generally reproduce data optically with lasers. Combinations of the above can also be included in computer readable media.

Abstract

一种基于多模态信息融合的车辆再识别方法及装置，涉及智能停车管理领域，方法包括：从监控视频中提取待查询车辆图像（101）；将待查询车辆图像作为预置多分支监测机制网络模型的输入，提取车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取车辆的车牌字符（102）；从预置数据库中获取不同车辆分别与车辆的全局特征、多个局部特征和车牌字符之间的相似度或者匹配的联合概率（103）；根据不同车辆与待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表（104）。该方法可以有效提高车辆再识别的准确率，进一步地区分了车辆之间的类间差异和类内差异，能够适应真实场景下的各种应用需求。

Description

基于多模态信息融合的车辆再识别方法及装置 技术领域
本发明涉及智能停车管理领域，特别涉及一种基于多模态信息融合的车辆再识别方法及装置。
背景技术
车辆作为城市监控中的重要对象，在检测、跟踪、调度等大量与车辆相关的任务中引起了广泛关注。车辆再识别是找出不同摄像机所拍摄到的同一辆车，或同一摄像机在不同光照、不同视角下拍摄到的同一车辆。通过车辆再识别技术，可以实现跨镜头自动识别并锁定同一辆车，在城市交通调度、违法车辆追踪等具体任务中，起着十分重要的作用，有利于智能交通和智慧城市的规划与发展。
目前常见的车辆再识别方法通常包含如下几种类型：一种是利用传感器来解决车辆再识别的问题。例如，通过地磁传感器、红外射频传感器等进行车辆再识别，然而这种方法所需成本较高，安装环境要求复杂，不适用于大规模推广使用。第二种方法是通过车牌进行识别，实现车辆跨摄像头的追踪定位，但是在很多情况下，车牌由于光照、遮挡、污损等各种原因，导致车牌号码识别不准确，进而导致车辆再识别的误差较大；第三种方法是基于车辆局部外观特征进行车辆再识别，然而由于车辆局部外观特征对于车辆的反映情况较为片面，进而导致车辆再识别进度较低。
发明内容
为解决上述技术问题，本发明提供一种基于多模态信息融合的车辆再识别方法及装置，可以解决现有再识别方法仅通过车辆外观或者是车牌号码进行车辆再识别，所应用的识别特征较为单一，识别精度较低的问题。
为实现上述目的，一种基于多模态信息融合的车辆再识别方法，其特征在于，所述方法包括：
从监控视频中提取待查询车辆图像；
将所述待查询车辆图像作为预置多分支监测机制网络模型的输入，提取所述车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取所述车辆的车牌字符；
从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符 之间的相似度或者匹配的联合概率，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符；
根据不同车辆与所述待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表。
进一步地，所述方法还包括：
从监控视频中提取待查询车辆图像的时空信息；
根据不同车辆与待查询车辆之间匹配的联合概率和所述待查询车辆图像的时空信息预测所述车辆的相对行驶方向；
根据不同摄像头之间的拓扑关系、所述车辆相对行驶方向、以及预置车辆时空转移模型，得到所述车辆与不同车辆之间的时空匹配概率，所述预置车辆时空转移模型是根据不同车辆分别对应的历史车辆行驶时空数据建立的。
进一步地，所述根据不同车辆与待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表的步骤包括：
根据不同车辆与待查询车辆之间匹配的联合概率或者相似度，以及所述车辆与不同车辆之间的时空匹配概率生成车辆重识别排序表。
进一步地，从监控视频中提取待查询车辆图像的步骤包括：
从监控视频中获取的待查询车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标图像；
剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像。
进一步地，所述全局特征为所述车辆的整体外观特征，所述多个局部特征包括车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征。
进一步地，从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度的步骤包括：
根据不同车辆以及所述车辆的整体外观特征、车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、以及车牌字符特征，构造三元损失函数并计算不同车辆与所述车辆之间的特征距离，得到不同车辆分别与所述车辆之间的相似度。
进一步地，所述获取不同车辆与所述车辆匹配的联合概率的步骤包括：
基于贝叶斯概率模型，根据公式Pa＝PF×θ×Ptpo进行计算，其中，Pa为候选车辆与所述待查询车辆匹配的联合概率，PF为候选车辆与待查询车辆之间车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、车辆整体外观特征匹配的联合概率，Ptpo为 候选车辆与待查询车辆之间的车牌匹配概率，θ为车牌识别的置信度。
进一步地，本发明提供一种基于多模态信息融合的车辆再识别装置，所述装置包括：
提取模块，用于从监控视频中提取待查询车辆图像；
所述提取模块，还用于将所述待查询车辆图像作为预置多分支监测机制网络模型的输入，提取所述车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取所述车辆的车牌字符；
获取模块，用于从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度或者匹配的联合概率，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符；
生成模块，用于根据不同车辆与所述待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表。
进一步地，所述装置还包括：预测模块；
所述提取模块，还用于从监控视频中提取所述待查询车辆图像的时空信息；
所述预测模块，用于根据不同车辆与待查询车辆之间匹配的联合概率和所述待查询车辆图像的时空信息预测所述车辆的相对行驶方向；
所述获取模块，还用于根据不同摄像头之间的拓扑关系、所述车辆相对行驶方向、以及预置车辆时空转移模型，得到所述车辆与不同车辆之间的时空匹配概率，所述预置车辆时空转移模型是根据不同车辆分别对应的历史车辆行驶时空数据建立的。
进一步地，所述生成模块，具体用于根据不同车辆与待查询车辆之间匹配的联合概率或者相似度，以及所述车辆与不同车辆之间的时空匹配概率生成车辆重识别排序表。
进一步地，所述提取模块，具体用于从监控视频中获取的待查询车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标图像；剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像。
进一步地，所述全局特征为所述车辆的整体外观特征，所述多个局部特征包括车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征。
进一步地，所述获取模块，具体用于根据不同车辆以及所述车辆的整体外观特征、车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、以及车牌字符特征，构造三元损失函数并计算不同车辆与所述车辆之间的特征距离，得到不同车辆分别与所述车辆之间的相似度。
进一步地，所述获取模块，具体还用于基于贝叶斯概率模型，根据公式Pa＝ PF×θ×Ptpo进行计算，其中，Pa为候选车辆与所述待查询车辆匹配的联合概率，PF为候选车辆与待查询车辆之间车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、车辆整体外观特征匹配的联合概率，Ptpo为候选车辆与待查询车辆之间的车牌匹配概率，θ为车牌识别的置信度。
本发明提供的一种基于多模态信息融合的车辆再识别方法及装置，通过融入了预置多分支监测机制网络模型，通过分别比对车辆的全局特征、多个局部特征和车牌字符，实现基于全局外观特征、局部外观特征以及车牌字符相融合的特征匹配，进而与现有只是通过单一特征或者车牌字符进行特征匹配相比，可以有效提高车辆再识别的准确率；进一步地在模态上，还融入了车辆对应的时间与地理位置信息，形成多模态融合概率特征，很好的区分了车辆之间的类间差异和类内差异，能够适应真实场景下的不同光照、不同摄像头型号识别的应用需求。
通过以下参照附图对本公开的示例性实施例的详细描述，本公开的其它特征及其优点将会变得清楚。
附图说明
此处所说明的附图用来提供对本公开的进一步理解，构成本申请的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。在附图中：
图1是本发明提供的一种基于多模态信息融合的车辆再识别方法的流程图；
图2是本发明提供的一种基于多模态信息融合的车辆再识别装置的示意图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开的范围。
同时，应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不是按照实际的比例关系绘制的。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为授权说明书的一部分。
在这里示出和讨论的所有示例中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它示例可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
本发明提供一种基于图像深度信息的停车事件确定方法，如图1所示，具体包括如下步骤：
101、从监控视频中提取待查询车辆图像。
对于本发明实施例，步骤101具体可以包括：从监控视频中获取的待查询车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标图像；剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像。对于本发明实施例，通过将原始监控图像的车辆类别检出并进行实例分割，去除了背景噪声的干扰，可充分提取出车辆的全部信息，从而保证了提取的图像的清晰度，进而可以进一步提升车辆的再识别精度。
具体地，从监控视频中获取的待查询车辆图片上进行车辆目标检测的具体过程可以包括如下步骤：当某一监控摄像机在捕获视频帧时，同时获取该监控摄像头的地理位置信息，以及拍摄该视频帧的时间信息。同时，利用单阶段的检测方法作为目标检测工具，再从源监控视频流中获取的车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标，作为车辆实例分割源数据，并且忽略分辨率小于256*256的图片，并在车辆检测后图像中，标注每辆车对应的身份ID。其中，所述地理位置包括但不限于国家、城市、经纬度坐标等。所述时间包括但不限于年月日、时分秒、农历、阳历等。所述单阶段检测方法包括但不限于YOLO、SSD等回归式的目标检测方法。所述源车辆图片涵盖真实的道路交通场景监控捕捉的同一辆车的多幅不同视角、不同背景、不同光照强度下的图像，其中，车辆图像所涵盖的内容包括：车牌外观信息、车牌字符信息、颜色、车型、车辆标注信息。
进一步地，获取完整地被边界框标出的车辆目标图像，剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像的步骤具体可以包括：使用多边形标注工具将车辆图片数据集中的车辆目标，沿着车辆边界抠出，制成车辆实例分割数据集。使用卷积神经网络生成特征向量，训练车辆实例分割网络模型，然后将车辆图片进行初步的实例分割，之后对初步分割结果做数据清洗，如边缘检测、孔洞填充、连通域检测和像素面积比对，完善车辆和实例分割效果，获得分割后的车辆数据集。其中，所述实例分割方法包括但不限于MASK-RCNN、SOLO等实例分割方法。所述卷积神经网络包括但不限于VGG、AlexNet、ResNet等卷积神经网络。所述边缘检测包括但不限于高斯滤波、Canny、Sobel等边缘检测方法。所述孔洞填充包括但不限于开运算、闭运算、漫水填充等像素填充方法。
102、将所述待查询车辆图像作为预置多分支监测机制网络模型的输入，提取所述车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取所述车辆的车牌字符。
其中，所述全局特征为所述车辆的整体外观特征，所述多个局部特征包括车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征。所述预置多分支监测机制网络模型包括但不限于SSD、YOLO、M2det、Fast-RCNN等检测模型，检测模型所用的数据集为实例分割后的剔除背景噪声的车辆图像的数据集。所述车牌字符识别方法包括但不限于CRNN+CTC、YOLOV3等字符识别方法，进而保证车牌识别的准确性。
103、从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度或者匹配的联合概率。
其中，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符。
对于本发明实施例，从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度的步骤包括：根据不同车辆以及所述车辆的整体外观特征、车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、以及车牌字符特征，构造三元损失函数并计算不同车辆与所述车辆之间的特征距离，得到不同车辆分别与所述车辆之间的相似度。
具体地，利用卷积层分别提取每张车辆图像的初始特征，再利用多分支检测模块机制分别提取车头外观特征、车尾外观特征、车牌外观特征以及车牌字符特征这四部分局部特征，经过图像融合，将四部分局部特征与全局特征相融合得到再识别模型。
进一步地，所述损失函数为三元组损失函数为：
Figure PCTCN2020131777-appb-000001
其中，f(P)、f(N)为由不同摄像机拍摄的ID相同的车辆图像，f(A)为一张ID不同的车辆图像。所述提取多个局部特征和全局特征，需要设置提取不同区域特征的卷积神经网络的初始参数。具体地，经过车辆特征提取神经网络,输出车辆身份特征；特征融合网络采用5层全连接层神经网络，取其中第一层全连接层后的输出作为车辆的融合特征。所述特征提取网络包括但不限于ResNet、VGG、AlexNet等特征提取网路模型。训练特征融合网络的方法如下：采用了交叉熵损失函数和三元组损失函数的度量学习，训练所述网络的特征学习过程；在采用度量学习的损失函数训练特征融合网络5层全连接层神经网络的过程中，减小了车辆相同ID的类内距离，扩大了车辆不同ID类间距离，增强车辆融合特征的鲁棒性。
对于本发明实施例，所述获取不同车辆与所述车辆匹配的联合概率的步骤包括：基于 贝叶斯概率模型，根据公式Pa＝PF×θ×Ptpo进行计算，其中，Pa为候选车辆与所述待查询车辆匹配的联合概率，PF为候选车辆与待查询车辆之间车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、车辆整体外观特征匹配的联合概率，Ptpo为候选车辆与待查询车辆之间的车牌匹配概率，θ为车牌识别的置信度。
104、根据不同车辆与所述待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表。
其中，为了方便进行候选车辆筛选，所述车辆重识别排序表可以按照相似度或者概率的降序进行排列。
对于本发明实施例，为了进一步提升车辆再识别准确率，还可以融入车辆的时空信息，具体方法可以包括：从监控视频中提取所述待查询车辆图像的时空信息；根据不同车辆与待查询车辆之间匹配的联合概率和所述待查询车辆图像的时空信息预测所述车辆的相对行驶方向；根据不同摄像头之间的拓扑关系、所述车辆相对行驶方向、以及预置车辆时空转移模型，得到所述车辆与不同车辆之间的时空匹配概率，所述预置车辆时空转移模型是根据不同车辆分别对应的历史车辆行驶时空数据建立的。
此时，步骤104具体可以包括：根据不同车辆与待查询车辆之间匹配的联合概率或者相似度，以及所述车辆与不同车辆之间的时空匹配概率生成车辆重识别排序表。
本发明提供的一种基于多模态信息融合的车辆再识别方法，通过融入了预置多分支监测机制网络模型，通过分别比对车辆的全局特征、多个局部特征和车牌字符，实现基于全局外观特征、局部外观特征以及车牌字符相融合的特征匹配，进而与现有只是通过单一特征或者车牌字符进行特征匹配相比，可以有效提高车辆再识别的准确率；进一步地在模态上，还融入了车辆对应的时间与地理位置信息，形成多模态融合概率特征，很好的区分了车辆之间的类间差异和类内差异，能够适应真实场景下的不同光照、不同摄像头型号识别的应用需求。
作为图1所示方法的具体实现方式，本发明实施例提供一种基于多模态信息融合的车辆再识别装置，如图2所示，所述装置包括：提取模块21，用于从监控视频中提取待查询车辆图像；
所述提取模块21，还用于将所述待查询车辆图像作为预置多分支监测机制网络模型的输入，提取所述车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取所述车辆的车牌字符；
获取模块22，用于从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度或者匹配的联合概率，所述预置数据库中保存有不同车 辆分别对应的全局特征、多个局部特征和车牌字符，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符；
生成模块23，用于根据不同车辆与所述待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表。
进一步地，所述装置还包括：预测模块24；
所述提取模块21，还用于从监控视频中提取所述待查询车辆图像的时空信息；
所述预测模块24，用于根据不同车辆与待查询车辆之间匹配的联合概率和所述待查询车辆图像的时空信息预测所述车辆的相对行驶方向；
所述获取模块22，还用于根据不同摄像头之间的拓扑关系、所述车辆相对行驶方向、以及预置车辆时空转移模型，得到所述车辆与不同车辆之间的时空匹配概率，所述预置车辆时空转移模型是根据不同车辆分别对应的历史车辆行驶时空数据建立的。
进一步地，所述生成模块23，具体用于根据不同车辆与待查询车辆之间匹配的联合概率或者相似度，以及所述车辆与不同车辆之间的时空匹配概率生成车辆重识别排序表。
进一步地，所述提取模块21，具体用于从监控视频中获取的待查询车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标图像；剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像。对于本发明实施例，通过将原始监控图像的车辆类别检出并进行实例分割，去除了背景噪声的干扰，可充分提取出车辆的全部信息，从而保证了提取的图像的清晰度，进而可以进一步提升车辆的再识别精度。
进一步地，所述全局特征为所述车辆的整体外观特征，所述多个局部特征包括车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征。
进一步地，所述获取模块22，具体用于根据不同车辆以及所述车辆的整体外观特征、车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、以及车牌字符特征，构造三元损失函数并计算不同车辆与所述车辆之间的特征距离，得到不同车辆分别与所述车辆之间的相似度。
进一步地，所述获取模块22，具体还用于基于贝叶斯概率模型，根据公式Pa＝PF×θ×Ptpo进行计算，其中，Pa为候选车辆与所述待查询车辆匹配的联合概率，PF为候选车辆与待查询车辆之间车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、车辆整体外观特征匹配的联合概率，Ptpo为候选车辆与待查询车辆之间的车牌匹配概率，θ为车牌识别的置信度。
本发明提供的一种基于多模态信息融合的车辆再识别装置，通过融入了预置多分支监测机制网络模型，通过分别比对车辆的全局特征、多个局部特征和车牌字符，实现基于全 局外观特征、局部外观特征以及车牌字符相融合的特征匹配，进而与现有只是通过单一特征或者车牌字符进行特征匹配相比，可以有效提高车辆再识别的准确率；进一步地在模态上，还融入了车辆对应的时间与地理位置信息，形成多模态融合概率特征，很好的区分了车辆之间的类间差异和类内差异，能够适应真实场景下的不同光照、不同摄像头型号识别的应用需求。
应该明白，公开的过程中的步骤的特定顺序或层次是示例性方法的实例。基于设计偏好，应该理解，过程中的步骤的特定顺序或层次可以在不脱离本公开的保护范围的情况下得到重新安排。所附的方法权利要求以示例性的顺序给出了各种步骤的要素，并且不是要限于所述的特定顺序或层次。
在上述的详细描述中，各种特征一起组合在单个的实施方案中，以简化本公开。不应该将这种公开方法解释为反映了这样的意图，即，所要求保护的主题的实施方案需要比清楚地在每个权利要求中所陈述的特征更多的特征。相反，如所附的权利要求书所反映的那样，本发明处于比所公开的单个实施方案的全部特征少的状态。因此，所附的权利要求书特此清楚地被并入详细描述中，其中每项权利要求独自作为本发明单独的优选实施方案。
为使本领域内的任何技术人员能够实现或者使用本发明，上面对所公开实施例进行了描述。对于本领域技术人员来说；这些实施例的各种修改方式都是显而易见的，并且本文定义的一般原理也可以在不脱离本公开的精神和保护范围的基础上适用于其它实施例。因此，本公开并不限于本文给出的实施例，而是与本申请公开的原理和新颖性特征的最广范围相一致。
上文的描述包括一个或多个实施例的举例。当然，为了描述上述实施例而描述部件或方法的所有可能的结合是不可能的，但是本领域普通技术人员应该认识到，各个实施例可以做进一步的组合和排列。因此，本文中描述的实施例旨在涵盖落入所附权利要求书的保护范围内的所有这样的改变、修改和变型。此外，就说明书或权利要求书中使用的术语“包含”，该词的涵盖方式类似于术语“包括”，就如同“包括，”在权利要求中用作衔接词所解释的那样。此外，使用在权利要求书的说明书中的任何一个术语“或者”是要表示“非排它性的或者”。
本领域技术人员还可以了解到本发明实施例列出的各种说明性逻辑块(illustrative logical block)，单元，和步骤可以通过电子硬件、电脑软件，或两者的结合进行实现。为清楚展示硬件和软件的可替换性(interchangeability)，上述的各种说明性部件(illustrative components)，单元和步骤已经通用地描述了它们的功能。这样的功能 是通过硬件还是软件来实现取决于特定的应用和整个系统的设计要求。本领域技术人员可以对于每种特定的应用，可以使用各种方法实现所述的功能，但这种实现不应被理解为超出本发明实施例保护的范围。
本发明实施例中所描述的各种说明性的逻辑块，或单元都可以通过通用处理器，数字信号处理器，专用集成电路(ASIC)，现场可编程门阵列或其它可编程逻辑装置，离散门或晶体管逻辑，离散硬件部件，或上述任何组合的设计来实现或操作所描述的功能。通用处理器可以为微处理器，可选地，该通用处理器也可以为任何传统的处理器、控制器、微控制器或状态机。处理器也可以通过计算装置的组合来实现，例如数字信号处理器和微处理器，多个微处理器，一个或多个微处理器联合一个数字信号处理器核，或任何其它类似的配置来实现。
本发明实施例中所描述的方法或算法的步骤可以直接嵌入硬件、处理器执行的软件模块、或者这两者的结合。软件模块可以存储于RAM存储器、闪存、ROM存储器、EPROM存储器、EEPROM存储器、寄存器、硬盘、可移动磁盘、CD-ROM或本领域中其它任意形式的存储媒介中。示例性地，存储媒介可以与处理器连接，以使得处理器可以从存储媒介中读取信息，并可以向存储媒介存写信息。可选地，存储媒介还可以集成到处理器中。处理器和存储媒介可以设置于ASIC中，ASIC可以设置于用户终端中。可选地，处理器和存储媒介也可以设置于用户终端中的不同的部件中。
在一个或多个示例性的设计中，本发明实施例所描述的上述功能可以在硬件、软件、固件或这三者的任意组合来实现。如果在软件中实现，这些功能可以存储与电脑可读的媒介上，或以一个或多个指令或代码形式传输于电脑可读的媒介上。电脑可读媒介包括电脑存储媒介和便于使得让电脑程序从一个地方转移到其它地方的通信媒介。存储媒介可以是任何通用或特殊电脑可以接入访问的可用媒体。例如，这样的电脑可读媒体可以包括但不限于RAM、ROM、EEPROM、CD-ROM或其它光盘存储、磁盘存储或其它磁性存储装置，或其它任何可以用于承载或存储以指令或数据结构和其它可被通用或特殊电脑、或通用或特殊处理器读取形式的程序代码的媒介。此外，任何连接都可以被适当地定义为电脑可读媒介，例如，如果软件是从一个网站站点、服务器或其它远程资源通过一个同轴电缆、光纤电缆、双绞线、数字用户线(DSL)或以例如红外、无线和微波等无线方式传输的也被包含在所定义的电脑可读媒介中。所述的碟片(disk)和磁盘(disc)包括压缩磁盘、镭射盘、光盘、DVD、软盘和蓝光光盘，磁盘通常以磁性复制数据，而碟片通常以激光进行光学复制数据。上述的组合也可以包含在电脑可读媒介中。
以上所述的具体实施方式，对本发明的目的、技术方案和有益效果进行了进一步详细 说明，所应理解的是，以上所述仅为本发明的具体实施方式而已，并不用于限定本发明的保护范围，凡在本发明的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (14)

  1. 一种基于多模态信息融合的车辆再识别方法，其特征在于，所述方法包括：
    从监控视频中提取待查询车辆图像；
    将所述待查询车辆图像作为预置多分支监测机制网络模型的输入，提取所述车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取所述车辆的车牌字符；
    从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度或者匹配的联合概率，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符；
    根据不同车辆与所述待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表。
  2. 根据权利要求1所述的一种基于多模态信息融合的车辆再识别方法，其特征在于，所述方法还包括：
    从监控视频中提取所述待查询车辆图像的时空信息；
    根据不同车辆与待查询车辆之间匹配的联合概率和所述待查询车辆图像的时空信息预测所述车辆的相对行驶方向；
    根据不同摄像头之间的拓扑关系、所述车辆相对行驶方向、以及预置车辆时空转移模型，得到所述车辆与不同车辆之间的时空匹配概率，所述预置车辆时空转移模型是根据不同车辆分别对应的历史车辆行驶时空数据建立的。
  3. 根据权利要求2所述的一种基于多模态信息融合的车辆再识别方法，其特征在于，所述根据不同车辆与待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表的步骤包括：
    根据不同车辆与待查询车辆之间匹配的联合概率或者相似度，以及所述车辆与不同车辆之间的时空匹配概率生成车辆重识别排序表。
  4. 根据权利要求1所述的一种基于多模态信息融合的车辆再识别方法，其特征在于，从监控视频中提取待查询车辆图像的步骤包括：
    从监控视频中获取的待查询车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标图像；
    剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像。
  5. 根据权利要求1-4任一项所述的一种基于多模态信息融合的车辆再识别方法，其 特征在于，所述全局特征为所述车辆的整体外观特征，所述多个局部特征包括车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征。
  6. 根据权利有要求5所述的一种基于多模态信息融合的车辆再识别方法，其特征在于，从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度的步骤包括：
    根据不同车辆以及所述车辆的整体外观特征、车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、以及车牌字符特征，构造三元损失函数并计算不同车辆与所述车辆之间的特征距离，得到不同车辆分别与所述车辆之间的相似度。
  7. 根据权利有要求5所述的一种基于多模态信息融合的车辆再识别方法，其特征在于，所述获取不同车辆与所述车辆匹配的联合概率的步骤包括：
    基于贝叶斯概率模型，根据公式Pa＝PF×θ×Ptpo进行计算，其中，Pa为候选车辆与所述待查询车辆匹配的联合概率，PF为候选车辆与待查询车辆之间车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、车辆整体外观特征匹配的联合概率，Ptpo为候选车辆与待查询车辆之间的车牌匹配概率，θ为车牌识别的置信度。
  8. 一种基于多模态信息融合的车辆再识别装置，其特征在于，所述装置包括：
    提取模块，用于从监控视频中提取待查询车辆图像；
    所述提取模块，还用于将所述待查询车辆图像作为预置多分支监测机制网络模型的输入，提取所述车辆的全局特征和多个局部特征，并根据多个局部特征中的车牌区域提取所述车辆的车牌字符；
    获取模块，用于从预置数据库中获取不同车辆分别与所述车辆的全局特征、多个局部特征和车牌字符之间的相似度或者匹配的联合概率，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符，所述预置数据库中保存有不同车辆分别对应的全局特征、多个局部特征和车牌字符；
    生成模块，用于根据不同车辆与所述待查询车辆之间的相似度或者匹配的联合概率生成车辆重识别排序表。
  9. 根据权利要求8所述的一种基于多模态信息融合的车辆再识别装置，其特征在于，所述装置还包括：预测模块；
    所述提取模块，还用于从监控视频中提取所述待查询车辆图像的时空信息；
    所述预测模块，用于根据不同车辆与待查询车辆之间匹配的联合概率和所述待查询车辆图像的时空信息预测所述车辆的相对行驶方向；
    所述获取模块，还用于根据不同摄像头之间的拓扑关系、所述车辆相对行驶方向、以 及预置车辆时空转移模型，得到所述车辆与不同车辆之间的时空匹配概率，所述预置车辆时空转移模型是根据不同车辆分别对应的历史车辆行驶时空数据建立的。
  10. 根据权利要求9所述的一种基于多模态信息融合的车辆再识别装置，其特征在于，
    所述生成模块，具体用于根据不同车辆与待查询车辆之间匹配的联合概率或者相似度，以及所述车辆与不同车辆之间的时空匹配概率生成车辆重识别排序表。
  11. 根据权利要求8所述的一种基于多模态信息融合的车辆再识别装置，其特征在于，
    所述提取模块，具体用于从监控视频中获取的待查询车辆图片上进行车辆目标检测，获取完整地被边界框标出的车辆目标图像；剔除所述车辆目标图像中的冗余背景信息，得到所述待查询车辆图像。
  12. 根据权利要求8-11任一项所述的一种基于多模态信息融合的车辆再识别装置，其特征在于，所述全局特征为所述车辆的整体外观特征，所述多个局部特征包括车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征。
  13. 根据权利要求12所述的一种基于多模态信息融合的车辆再识别装置，其特征在于，
    所述获取模块，具体用于根据不同车辆以及所述车辆的整体外观特征、车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、以及车牌字符特征，构造三元损失函数并计算不同车辆与所述车辆之间的特征距离，得到不同车辆分别与所述车辆之间的相似度。
  14. 根据权利要求12所述的一种基于多模态信息融合的车辆再识别装置，其特征在于，
    所述获取模块，具体还用于基于贝叶斯概率模型，根据公式Pa＝PF×θ×Ptpo进行计算，其中，Pa为候选车辆与所述待查询车辆匹配的联合概率，PF为候选车辆与待查询车辆之间车辆车头外观特征、车辆车尾外观特征、车辆车牌区域外观特征、车辆整体外观特征匹配的联合概率，Ptpo为候选车辆与待查询车辆之间的车牌匹配概率，θ为车牌识别的置信度。
PCT/CN2020/131777 2020-08-05 2020-11-26 基于多模态信息融合的车辆再识别方法及装置 WO2022027873A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010769769.5 2020-08-05
CN202010769769.5A CN111931627A (zh) 2020-08-05 2020-08-05 一种基于多模态信息融合的车辆再识别方法及装置

Publications (1)

Publication Number Publication Date
WO2022027873A1 true WO2022027873A1 (zh) 2022-02-10

Family

ID=73307698

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/131777 WO2022027873A1 (zh) 2020-08-05 2020-11-26 基于多模态信息融合的车辆再识别方法及装置

Country Status (2)

Country Link
CN (1) CN111931627A (zh)
WO (1) WO2022027873A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113486886A (zh) * 2021-06-21 2021-10-08 华侨大学 一种自然场景下的车牌识别方法和装置
CN114530056A (zh) * 2022-02-15 2022-05-24 超级视线科技有限公司 一种基于定位信息和图像信息的停车管理方法及系统
CN114926990A (zh) * 2022-05-30 2022-08-19 深圳市双盈电子科技有限公司 一种智能化道闸车牌识别设备
CN115034257A (zh) * 2022-05-09 2022-09-09 西北工业大学 一种基于特征融合的跨模态信息目标识别方法及装置
CN116110222A (zh) * 2022-11-29 2023-05-12 东风商用车有限公司 基于大数据的车辆应用场景分析方法
CN116403171A (zh) * 2023-06-08 2023-07-07 松立控股集团股份有限公司 一种车辆重识别方法、系统及电子设备

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111931627A (zh) * 2020-08-05 2020-11-13 智慧互通科技有限公司 一种基于多模态信息融合的车辆再识别方法及装置
CN112990217A (zh) * 2021-03-24 2021-06-18 北京百度网讯科技有限公司 针对车辆的图像识别方法、装置、电子设备和介质
CN112949751B (zh) * 2021-03-25 2023-03-24 深圳市商汤科技有限公司 车辆图像聚类及轨迹还原方法
CN113128441B (zh) * 2021-04-28 2022-10-14 安徽大学 一种属性和状态引导的结构嵌入车辆重识别系统和方法
CN113569912A (zh) * 2021-06-28 2021-10-29 北京百度网讯科技有限公司 车辆识别方法、装置、电子设备及存储介质
CN113569911A (zh) * 2021-06-28 2021-10-29 北京百度网讯科技有限公司 车辆识别方法、装置、电子设备及存储介质
CN113408472B (zh) * 2021-07-06 2023-09-26 京东科技信息技术有限公司 目标重识别模型的训练方法、目标重识别方法及装置
CN113918679A (zh) * 2021-09-22 2022-01-11 三一汽车制造有限公司 一种知识问答方法、装置及工程机械
CN116311214B (zh) * 2023-05-22 2023-08-22 珠海亿智电子科技有限公司 车牌识别方法和装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107729818A (zh) * 2017-09-21 2018-02-23 北京航空航天大学 一种基于深度学习的多特征融合车辆重识别方法
US20180060684A1 (en) * 2016-08-31 2018-03-01 Beijing University Of Posts And Telecommunications Progressive vehicle searching method and device
CN108681707A (zh) * 2018-05-15 2018-10-19 桂林电子科技大学 基于全局和局部特征融合的大角度车型识别方法和系统
CN110795580A (zh) * 2019-10-23 2020-02-14 武汉理工大学 基于时空约束模型优化的车辆重识别方法
CN111931627A (zh) * 2020-08-05 2020-11-13 智慧互通科技有限公司 一种基于多模态信息融合的车辆再识别方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180060684A1 (en) * 2016-08-31 2018-03-01 Beijing University Of Posts And Telecommunications Progressive vehicle searching method and device
CN107729818A (zh) * 2017-09-21 2018-02-23 北京航空航天大学 一种基于深度学习的多特征融合车辆重识别方法
CN108681707A (zh) * 2018-05-15 2018-10-19 桂林电子科技大学 基于全局和局部特征融合的大角度车型识别方法和系统
CN110795580A (zh) * 2019-10-23 2020-02-14 武汉理工大学 基于时空约束模型优化的车辆重识别方法
CN111931627A (zh) * 2020-08-05 2020-11-13 智慧互通科技有限公司 一种基于多模态信息融合的车辆再识别方法及装置

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113486886A (zh) * 2021-06-21 2021-10-08 华侨大学 一种自然场景下的车牌识别方法和装置
CN113486886B (zh) * 2021-06-21 2023-06-23 华侨大学 一种自然场景下的车牌识别方法和装置
CN114530056A (zh) * 2022-02-15 2022-05-24 超级视线科技有限公司 一种基于定位信息和图像信息的停车管理方法及系统
CN114530056B (zh) * 2022-02-15 2023-05-02 超级视线科技有限公司 一种基于定位信息和图像信息的停车管理方法及系统
CN115034257A (zh) * 2022-05-09 2022-09-09 西北工业大学 一种基于特征融合的跨模态信息目标识别方法及装置
CN114926990A (zh) * 2022-05-30 2022-08-19 深圳市双盈电子科技有限公司 一种智能化道闸车牌识别设备
CN116110222A (zh) * 2022-11-29 2023-05-12 东风商用车有限公司 基于大数据的车辆应用场景分析方法
CN116403171A (zh) * 2023-06-08 2023-07-07 松立控股集团股份有限公司 一种车辆重识别方法、系统及电子设备
CN116403171B (zh) * 2023-06-08 2023-09-01 松立控股集团股份有限公司 一种车辆重识别方法、系统及电子设备

Also Published As

Publication number Publication date
CN111931627A (zh) 2020-11-13

Similar Documents

Publication Publication Date Title
WO2022027873A1 (zh) 基于多模态信息融合的车辆再识别方法及装置
Qu et al. RGBD salient object detection via deep fusion
Soilán et al. Traffic sign detection in MLS acquired point clouds for geometric and image-based semantic inventory
Garg et al. Don't look back: Robustifying place categorization for viewpoint-and condition-invariant place recognition
CN103208185B (zh) 一种基于车灯识别的夜间车辆检测方法及系统
Buch et al. A review of computer vision techniques for the analysis of urban traffic
US20200364467A1 (en) Method and device for detecting illegal parking, and electronic device
Mallikarjuna et al. Traffic data collection under mixed traffic conditions using video image processing
Abdullah et al. YOLO-based three-stage network for Bangla license plate recognition in Dhaka metropolitan city
Zhang et al. Semi-automatic road tracking by template matching and distance transformation in urban areas
CN111310662B (zh) 一种基于集成深度网络的火焰检测识别方法及系统
CN110348463B (zh) 用于识别车辆的方法和装置
CN115717894B (zh) 一种基于gps和普通导航地图的车辆高精度定位方法
CN110688902B (zh) 一种检测停车位内车辆区域的方法及装置
CN111652035B (zh) 一种基于ST-SSCA-Net的行人重识别方法及系统
Jelača et al. Vehicle matching in smart camera networks using image projection profiles at multiple instances
CN112634368A (zh) 场景目标的空间与或图模型生成方法、装置及电子设备
Saha et al. i LPR: An indian license plate recognition system
Bu et al. A UAV photography–based detection method for defective road marking
CN112699711A (zh) 车道线检测方法、装置、存储介质及电子设备
CN116704490B (zh) 车牌识别方法、装置和计算机设备
Shi et al. Dense semantic 3D map based long-term visual localization with hybrid features
Ashraf et al. HVD-net: a hybrid vehicle detection network for vision-based vehicle tracking and speed estimation
Chen et al. Vision-based traffic surveys in urban environments
CN109325487B (zh) 一种基于目标检测的全种类车牌识别方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20948774

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20948774

Country of ref document: EP

Kind code of ref document: A1