CN115578621B - Image recognition method based on multi-source data fusion - Google Patents
Image recognition method based on multi-source data fusion Download PDFInfo
- Publication number
- CN115578621B CN115578621B CN202211355233.4A CN202211355233A CN115578621B CN 115578621 B CN115578621 B CN 115578621B CN 202211355233 A CN202211355233 A CN 202211355233A CN 115578621 B CN115578621 B CN 115578621B
- Authority
- CN
- China
- Prior art keywords
- image
- fusion
- basic
- visible light
- image recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 76
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 7
- 238000007500 overflow downdraw method Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- -1 rain Substances 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image recognition method based on multi-source data fusion, which can realize multi-source image fusion of visible light, infrared and the like and is applied to image recognition tasks. Aiming at the problem that the single-mode data cannot provide enough information for the image recognition task, the image recognition method based on multi-source data fusion is provided, and the image recognition task can be effectively performed by fusing the multi-source data. Firstly, decomposing visible light and infrared images through scale decomposition to obtain a basic image of reinforced contour information; then, respectively carrying out sub-division on the visible light and infrared images; then, respectively carrying out weighted fusion on the subgraphs corresponding to the two images, and obtaining a basic fusion image from the fused subgraphs; in order to obtain the original detail information, the basic fusion image and the original image are subjected to weighted fusion to highlight the salient features of the image, so that a final fusion image is obtained; and finally, inputting the fused image into an image recognition network to perform an image recognition task.
Description
Technical Field
The invention relates to an image recognition method based on multi-source data fusion, and belongs to the field of pattern recognition.
Background
With the continued development of sensor technology, a variety of different types of data are continually being collected by researchers. The single-mode data with insufficient information quantity cannot meet the high requirement of each task on precision. The multi-mode fusion technology can fuse different data about the same target acquired from different sensors through a unified frame, so that sufficient information is provided for training of a model. Infrared and visible light fusion is an important and frequently occurring problem. In recent years, a plurality of fusion methods have been proposed to fuse an infrared image with a visible light image to obtain a fused image with more retained detailed information, which can be applied to various tasks such as object detection, image classification, segmentation, and the like.
The multi-modal fusion method can be broadly divided into three types: and (3) data fusion, decision fusion and feature fusion. The data fusion directly processes the data acquired by different sensors and then inputs the data into the model. The most common data fusion method based on the spatial domain and the transformation domain comprises a logic filtering method, a weighted average method, a pyramid decomposition method, a wavelet transformation method and the like. The data fusion can reserve more detail information, and the data utilization rate is increased.
The decision fusion is to extract the characteristics of the multi-mode data respectively, then to fuse and utilize the multi-mode characteristics in a targeted manner in a decision layer, and finally to make an optimal decision according to a certain criterion. Common decision-based fusion methods include expert systems, bayesian reasoning, voting methods, and the like.
The feature fusion firstly extracts the features of the image, and then the fused feature images are obtained through analysis, processing and integration, and the features with discriminant are extracted from the fused feature images, so that the detection precision is improved. Compared with the disadvantages that the data fusion processing is long in time consumption and easy to be affected by noise, strict registration is needed, and late fusion is relatively dependent on the feature extraction capability of the previous level, the feature fusion not only furthest reserves the effective identification information of multiple features participating in fusion, but also eliminates the redundant information of the correlation among the multiple features caused by subjective and objective factors to a great extent, thereby reserving effective target detection information and realizing considerable information compression.
Many multi-mode fusion methods have good effects in infrared and visible light image fusion tasks, but the loss of many detail areas is difficult to avoid during image fusion, so that the fusion quality is low, and the fusion picture is unfavorable for downstream tasks. This patent is therefore considered to be designed.
Disclosure of Invention
The invention aims to: in order to overcome the defects in the prior art, the invention provides an image recognition method based on multi-source data fusion, which can fuse a visible light image and an infrared image into an image with enhanced significance characteristics so as to facilitate the execution of an image recognition task.
The invention discloses an image recognition method based on multi-source data fusion, which comprises the following steps:
step 1, respectively carrying out scale decomposition on a visible light original image and an infrared light original image to obtain a visible light basic image and an infrared light basic image of enhanced contour information;
step 2, respectively carrying out sub-division on the visible light basic image and the infrared light basic image; weighting and fusing the corresponding subgraphs to obtain a basic fusion image;
step 3, carrying out weighted fusion on the basic fusion image and the visible light original image to obtain a final fusion image;
and step 4, inputting the fused images into an image recognition network to perform an image recognition task.
Further, in step 1, the visible light base image and the infrared light base image may be obtained by the following formula:
wherein ,XS Representing an original image, which is a visible original image or an infrared original image,
represents a visible light base image or an infrared light base image g x and gy Respectively horizontal and vertical gradient operators, and lambda is a weight factor.
Further, the step 2 specifically includes the following:
first, sub-division is performed on a visible light base image and an infrared light base image, respectively:
wherein ,
representing an nth sub-graph after dividing the basic image;
then, respectively carrying out weighted fusion on the subgraphs corresponding to the two images:
wherein ,
and />
The ith sub-picture, respectively representing visible and infrared light,/->
Subgraph fusion weights based on second order statistics and representing visible light base image, < ->
Sub-graph fusion weight of table infrared basic image based on second order statistics,/->
And the final output basic fusion image.
Further, in step 3, the basic fusion image and the original image are subjected to weighted fusion to highlight the salient features of the image, so as to obtain a final fusion image, which is specifically expressed as follows:
wherein ,
and XI Respectively a basic fusion image and an original visible light image, wherein mu is a significance fusion coefficient based on second order statistics, F k Is the final fused image.
The beneficial effects are that: the method comprises the steps of decomposing an original image to obtain a visible light basic image with rich distinguishing characteristics and an infrared light basic image with obvious contour characteristics, dividing the two basic images into subgraphs, fusing the subgraphs, obtaining local area characteristics of the two images to obtain a fused image with the obvious contour characteristics, carrying out weighted fusion on the basic fused image and the original image to obtain original detail information, and highlighting the obvious characteristics of the image to obtain a final fused image.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention.
Detailed Description
The invention is further explained below with reference to the drawings.
As shown in fig. 1, an image recognition method based on multi-source data fusion includes the following steps:
step 1, respectively carrying out scale decomposition on a visible light original image and an infrared light original image to obtain a visible light basic image and an infrared light basic image of the enhanced contour information;
visible light images can provide a richer discriminative feature, but for scenes with lower visibility, such as smoke, rain, fog, evening and night scenes, it is difficult to obtain the salient features of the object, while infrared images can provide salient features in these scenes. And decomposing the visible light and infrared images through scale decomposition to obtain a basic image of the enhanced contour information.
For each original image X S Base image
Can be obtained by solving the following formula:
wherein ,gx and gy Respectively horizontal and vertical gradient operators, wherein lambda is a weight factor; both the visible light base image and the infrared light base image context can be obtained by the above equation.
Step S2: firstly, respectively carrying out sub-division on visible light and infrared images; then respectively carrying out weighted fusion on the subgraphs corresponding to the two images; finally, obtaining a basic fusion image from the fused subgraphs;
the visible light image is difficult to obtain the salient contour features for scenes with low visibility, the infrared is sensitive to the heat source, and the basic images obtained by decomposing the two images are subjected to sub-image weighted fusion to obtain the basic fusion image with the salient contour features so as to fully utilize the local area features of the two images.
First, sub-division is performed on a visible light base image and an infrared light base image, respectively:
wherein ,
representing the nth sub-graph after dividing the base image
Then, respectively carrying out weighted fusion on the subgraphs corresponding to the two images:
wherein ,
and />
The ith sub-picture, respectively representing visible and infrared light,/->
Subgraph fusion weights based on second order statistics and representing visible light base image, < ->
Sub-graph fusion weight of table infrared basic image based on second order statistics,/->
And the final output basic fusion image.
Step S3: in order to obtain the original detail information, the basic fusion image and the visible light original image are subjected to weighted fusion to highlight the salient features of the image, so that a final fusion image is obtained;
and fusing visible light and infrared significance contour information by the basic fusion image obtained through sub-image weighted fusion to obtain a contour information reinforced fusion image. However, in the case of scale decomposition, a part of detail images are discarded, so that the original images need to be weighted and fused to supplement the detail parts.
Weighting and fusing the basic fusion image and the original image to highlight the salient features of the image, so as to obtain a final fusion image:
wherein ,
and XI Respectively a basic fusion image and a visible light original image, wherein mu is a significance fusion coefficient based on second order statistics, F k Is the final fused image.
Step S4: and inputting the final fusion image into an image recognition network to perform an image recognition task.
Claims (2)
1. An image recognition method based on multi-source data fusion is characterized by comprising the following steps:
step 1, respectively performing scale decomposition on a visible light original image and an infrared light original image to obtain a visible light basic image and an infrared light basic image, wherein the visible light basic image and the infrared light basic image can be obtained by the following formula:
wherein ,XS The original image is presented in a table,
represents a visible light base image or an infrared light base image g x and gy Respectively horizontal and vertical gradient operators, wherein lambda is a weight factor;
step 2, respectively carrying out sub-division on the visible light basic image and the infrared light basic image; the corresponding subgraphs are subjected to weighted fusion to obtain a basic fusion image, which comprises the following specific contents:
first, sub-division is performed on a visible light base image and an infrared light base image, respectively:
wherein ,
representing an nth sub-graph after dividing the basic image;
then, respectively carrying out weighted fusion on the subgraphs corresponding to the two images:
wherein ,
and />
The ith sub-picture, respectively representing visible and infrared light,/->
Subgraph fusion weight representing visible light base image based on second order statistics，/>
Sub-graph fusion weight of table infrared basic image based on second order statistics,/->
A basic fusion image which is finally output;
step 3, carrying out weighted fusion on the basic fusion image and the visible light original image to obtain a final fusion image;
and step 4, inputting the fused images into an image recognition network to perform an image recognition task.
2. The image recognition method based on multi-source data fusion according to claim 1, wherein step 3 specifically comprises:
wherein ,
and XI Respectively a basic fusion image and a visible light original image, wherein mu is a significance fusion coefficient based on second order statistics, F k Is the final fused image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211355233.4A CN115578621B (en) | 2022-11-01 | 2022-11-01 | Image recognition method based on multi-source data fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211355233.4A CN115578621B (en) | 2022-11-01 | 2022-11-01 | Image recognition method based on multi-source data fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115578621A CN115578621A (en) | 2023-01-06 |
| CN115578621B true CN115578621B (en) | 2023-06-20 |
Family
ID=84588360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211355233.4A Active CN115578621B (en) | 2022-11-01 | 2022-11-01 | Image recognition method based on multi-source data fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115578621B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116192049B (en) * | 2023-04-28 | 2023-06-30 | 佰聆数据股份有限公司 | Safety detection method and system applied to distributed photovoltaic project |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761724A (en) * | 2014-01-28 | 2014-04-30 | 中国石油大学(华东) | Visible light and infrared video fusion method based on surreal luminance contrast pass algorithm |
| KR102294630B1 (en) * | 2020-03-19 | 2021-08-26 | 한양대학교 산학협력단 | Apparatus and method for fusing images |
| CN114529487A (en) * | 2022-02-14 | 2022-05-24 | 安徽信息工程学院 | Multi-source image fusion method and system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9292910B2 (en) * | 2012-05-14 | 2016-03-22 | National Institute Of Japan Science And Technology Agency | Image processing apparatus, image processing method, program, printing medium, and recording medium |
| CN110766676B (en) * | 2019-10-24 | 2022-04-26 | 中国科学院长春光学精密机械与物理研究所 | Target detection method based on multi-source sensor fusion |
| CN112184604B (en) * | 2020-09-15 | 2024-02-20 | 杭州电子科技大学 | Color image enhancement method based on image fusion |
-
2022
- 2022-11-01 CN CN202211355233.4A patent/CN115578621B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761724A (en) * | 2014-01-28 | 2014-04-30 | 中国石油大学(华东) | Visible light and infrared video fusion method based on surreal luminance contrast pass algorithm |
| KR102294630B1 (en) * | 2020-03-19 | 2021-08-26 | 한양대학교 산학협력단 | Apparatus and method for fusing images |
| CN114529487A (en) * | 2022-02-14 | 2022-05-24 | 安徽信息工程学院 | Multi-source image fusion method and system |
Non-Patent Citations (1)
| Title |
|---|
| 基于提升小波的红外和可见光图像融合方法;赵高鹏;薄煜明;刘娣;;计算机工程与设计(07);正文第143-145页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115578621A (en) | 2023-01-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110956094B (en) | RGB-D multi-mode fusion personnel detection method based on asymmetric double-flow network | |
| CN106897670B (en) | Express violence sorting identification method based on computer vision | |
| CN110458844B (en) | Semantic segmentation method for low-illumination scene | |
| CN111553387B (en) | Personnel target detection method based on Yolov3 | |
| CN108446678B (en) | Dangerous driving behavior identification method based on skeletal features | |
| CN114241260B (en) | Open set target detection and identification method based on deep neural network | |
| CN110807422A (en) | Natural scene text detection method based on deep learning | |
| CN113420607A (en) | Multi-scale target detection and identification method for unmanned aerial vehicle | |
| CN111666843A (en) | Pedestrian re-identification method based on global feature and local feature splicing | |
| CN110807434A (en) | Pedestrian re-identification system and method based on combination of human body analysis and coarse and fine particle sizes | |
| CN103049751A (en) | Improved weighting region matching high-altitude video pedestrian recognizing method | |
| CN106529494A (en) | Human face recognition method based on multi-camera model | |
| CN110929593A (en) | Real-time significance pedestrian detection method based on detail distinguishing and distinguishing | |
| CN105718866A (en) | Visual target detection and identification method | |
| CN110119726A (en) | A kind of vehicle brand multi-angle recognition methods based on YOLOv3 model | |
| CN113505670B (en) | Remote sensing image weak supervision building extraction method based on multi-scale CAM and super-pixels | |
| CN113361466B (en) | Multispectral target detection method based on multi-mode cross guidance learning | |
| CN115578621B (en) | Image recognition method based on multi-source data fusion | |
| CN110059675A (en) | A kind of robot identifies road traffic law enforcement behavior and provides the method for standardization auxiliary | |
| CN114170511B (en) | CASCADE RCNN-based pavement crack disease identification method | |
| CN112287906B (en) | Template matching tracking method and system based on depth feature fusion | |
| CN114581486A (en) | Template updating target tracking algorithm based on full convolution twin network multilayer characteristics | |
| CN111539351A (en) | Multi-task cascaded face frame selection comparison method | |
| CN108509950A (en) | Railway contact line pillar number plate based on probability characteristics Weighted Fusion detects method of identification | |
| CN111667465A (en) | Metal hand basin defect detection method based on far infrared image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |