CN115015910B - Interactive perception identification method, device, terminal and medium for microwave and optical vision - Google Patents

Interactive perception identification method, device, terminal and medium for microwave and optical vision Download PDF

Info

Publication number
CN115015910B
CN115015910B CN202210600633.0A CN202210600633A CN115015910B CN 115015910 B CN115015910 B CN 115015910B CN 202210600633 A CN202210600633 A CN 202210600633A CN 115015910 B CN115015910 B CN 115015910B
Authority
CN
China
Prior art keywords
target
image
radar
optical
recognition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210600633.0A
Other languages
Chinese (zh)
Other versions
CN115015910A (en
Inventor
赵博
张磊
黄磊
梁承美
司璀琪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to CN202210600633.0A priority Critical patent/CN115015910B/en
Publication of CN115015910A publication Critical patent/CN115015910A/en
Application granted granted Critical
Publication of CN115015910B publication Critical patent/CN115015910B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/867—Combination of radar systems with cameras
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/89—Radar or analogous systems specially adapted for specific applications for mapping or imaging
    • G01S13/90—Radar or analogous systems specially adapted for specific applications for mapping or imaging using synthetic aperture techniques, e.g. synthetic aperture radar [SAR] techniques
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/89—Radar or analogous systems specially adapted for specific applications for mapping or imaging
    • G01S13/90—Radar or analogous systems specially adapted for specific applications for mapping or imaging using synthetic aperture techniques, e.g. synthetic aperture radar [SAR] techniques
    • G01S13/9021—SAR image post-processing techniques
    • G01S13/9027—Pattern recognition for feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Electromagnetism (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Radar Systems Or Details Thereof (AREA)

Abstract

The invention discloses a method, a device, a terminal and a medium for identifying interactive perception of microwave and optical vision, wherein the method comprises the following steps: shooting by a set depth camera to obtain a depth image and an optical image, and obtaining a radar SAR image by a radar; performing target identification on the optical image, obtaining target category information, and fusing a depth image to obtain target distance information; labeling the target class information into the radar SAR image; acquiring a plurality of marked radar SAR images, and constructing and training to obtain an SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR images; when the method is applied, the current environmental data are detected, the SAR image target recognition neural network model and the optical image target recognition are controlled to be adopted for interactive recognition according to the detected different environmental data, and the recognition result is output. The invention realizes the mutual interaction and fusion of the microwave radar and the optical vision, improves the stability of the system in the target recognition work and improves the accuracy of the whole target recognition.

Description

Interactive perception identification method, device, terminal and medium for microwave and optical vision
Technical Field
The present invention relates to the field of target recognition technologies, and in particular, to a method and apparatus for interactive perception recognition of microwave and optical vision, a terminal device, and a storage medium.
Background
For radar target recognition, acquiring target detail information is limited due to the self-performance limitations of the radar system. When the radar system wants to acquire more target information, more radar resources such as frequency spectrum, energy and the like are consumed, and the processing time is greatly increased. The method brings great constraint to radar real-time target recognition engineering; not only is time and labor consuming, but also the target recognition effect is not good.
Accordingly, there is a need for improvement and advancement in the art.
Disclosure of Invention
The invention aims at solving the technical problems of the prior art and provides an interactive perception identification method, device, terminal equipment and storage medium of microwave and optical vision.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
A method of interactive perception recognition of microwave and optical vision, wherein the method comprises:
shooting by a set depth camera to acquire a depth image and an optical image, and acquiring a radar SAR image by a radar;
performing target identification on the optical image, obtaining target category information, and fusing a depth image to obtain target distance information;
labeling the target class information into the radar SAR image;
acquiring a plurality of marked radar SAR images to construct a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
when the method is applied, the current environmental data are detected, the SAR image target recognition neural network model and the depth image target recognition are controlled to be adopted for interactive recognition according to the detected different environmental data, and a recognition result is output.
The method for identifying the interactive perception of the microwave and the optical vision comprises the steps of shooting by a set depth camera to obtain a depth image and an optical image, and obtaining a radar SAR image by a radar, wherein the steps comprise:
Controlling to start a depth camera to shoot an image, and acquiring a depth image and an optical image;
and controlling the radar to acquire data, and processing the acquired radar data through a radar imaging algorithm to obtain a radar SAR image.
The method for identifying the interactive perception of the microwave and the optical vision, wherein the steps of identifying the target of the optical image, obtaining target category information, fusing a depth image and obtaining target distance information comprise the following steps:
acquiring optical image information and a depth image through a depth camera;
performing real-time target identification on the optical image information acquired by the depth camera by using the depth neural network to acquire target class information C, and acquiring a region D, a coordinate range (x i ，y i )∈D；
And indexing the distance information of the target according to the depth image generated by the depth camera.
The method for identifying the interactive perception of the microwave and the optical vision, wherein the step of labeling the target category information into the radar SAR image comprises the following steps:
and fusing and labeling target category information acquired from the optical image and target distance information acquired from the depth image to the radar SAR image.
The method for identifying the interactive perception of the microwave and the optical vision, wherein the step of fusing and labeling the target category information acquired from the optical image and the target distance information acquired from the depth image to the radar SAR image comprises the following steps of:
In the labeling mapping process, y is the center coordinate of the region D where the target is located in the optical image r An azimuth dimension representative value as a depth image target; the coordinate system of the radar SAR image consists of azimuth dimension and distance dimension, and the image contains distance information and Doppler information. The coordinate system of the optical image is constituted by the abscissa and the ordinate, and each pixel is constituted by (x i ,y i ) A representation;
based on the object class information of the obtained optical image, and the distance indexed from the depth imageInformation d r Distance information d r As a distance representative value of the object in the optical image, an azimuth-distance dimensional coordinate system of the optical image is abstracted.
The method for identifying the interactive perception of the microwave and the optical vision, wherein the step of fusing and labeling the target category information acquired from the optical image and the target distance information acquired from the depth image to the radar SAR image further comprises the following steps:
averaging target azimuth information of the optical image and distance information in the depth image in the time of the target T; obtaining the average azimuth of the target in the time of TTarget average distance within target T time +.>
Segmenting the abstract azimuth-distance dimensional coordinate system of the optical image according to distance, and utilizing different scale factors k i Correcting the position-distance dimensional coordinate system of the abstracted optical image to map the position-distance dimensional coordinate system to the radar image;
calculating the average orientation of the target in the optical image within a unit time TThe average azimuth occupies the optical image width W 2 Ratio eta of (2);
by mean distance of target within T time of targetPosition dimension boundary [ L ] of extracted radar SAR image under distance i ,R i ]The method comprises the steps of carrying out a first treatment on the surface of the Width W of optical image occupied according to average azimuth 2 Calculating azimuth dimension coordinates pos of the target in the optical image in the radar SAR image according to the proportion eta of the target ori The method comprises the steps of carrying out a first treatment on the surface of the According to the target average distance within the target T time +.>Correcting and rounding the target to obtain a distance dimension coordinate pos of the target corresponding to the SAR image in the optical image dis According to azimuth dimension coordinates and distance dimension coordinates (pos ori ,pos dis ) Mapping the optical image target position into a radar SAR image;
by means of both azimuth and distance dimensions (pos) ori ,pos dis ) Taking r as an origin, searching a maximum connected domain Q in the region, wherein the maximum connected domain Q is a region of the optical image, corresponding to a target in the SAR image; marking the connected domain Q by a box, and marking the category information C obtained by the recognition of the deep learning method under the box; and the mapping of the optical image category information and the depth image distance information to the radar SAR picture is completed.
In the interactive perception recognition method of microwave and optical vision, when the method is applied, current environmental data is detected, and according to the detected different environmental data, the steps of controlling the adoption of SAR image target recognition neural network model and depth image target recognition for interactive recognition and outputting recognition results comprise the following steps:
when the method is applied, detecting current environmental data;
when the detected illumination environment and visibility environment in the current environmental data accord with a preset value, judging that the weather condition is good, controlling to take target identification of an optical image as a leading mode, taking radar SAR image target identification as auxiliary feedback, carrying out feedback labeling according to the SAR image target identification result, carrying out interactive identification, and outputting an identification result;
when the illumination environment and the visibility environment in the detected current environmental data do not accord with the preset value, controlling to take radar SAR image target identification as a leading, taking optical image target identification as an auxiliary reference, carrying out interactive identification, and outputting an identification result.
An interactive perception recognition device of microwave and optical vision, wherein the device comprises:
the image acquisition module is used for acquiring a depth image and an optical image through shooting of the set depth camera;
The radar image acquisition module is used for acquiring radar SAR images through a radar;
the identification module is used for carrying out target identification on the optical image, obtaining target category information, fusing the depth image and obtaining target distance information;
the annotation fusion module is used for annotating the target class information into the radar SAR image;
the SAR image target recognition neural network model training module is used for acquiring a plurality of marked radar SAR images and constructing a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
and the interactive application module is used for detecting current environmental data during application, controlling the adoption of the SAR image target recognition neural network model to carry out interactive recognition with the depth image target recognition according to the detected different environmental data, and outputting a recognition result.
The terminal equipment comprises a memory, a processor and a microwave and optical vision interactive perception recognition program which is stored in the memory and can run on the processor, wherein the processor realizes the steps of the microwave and optical vision interactive perception recognition method when executing the microwave and optical vision interactive perception recognition program.
A computer readable storage medium, wherein the computer readable storage medium stores a microwave and optical visual interactive perception recognition program, and when the microwave and optical visual interactive perception recognition program is executed by a processor, the steps of the microwave and optical visual interactive perception recognition method are implemented.
The beneficial effects are that: compared with the prior art, the invention provides an interactive perception recognition method of microwave and optical vision, which adopts the following steps: the method comprises the steps of imaging radar acquisition signals in real time, carrying out real-time target recognition on optical acquisition images, and designing a fusion strategy of microwave radar and optical vision on the basis of the real-time imaging. The invention takes target recognition of the depth camera as a dominant and radar target recognition as an auxiliary. The radar system assists to greatly reduce the false recognition rate of the optical camera. When the weather condition is bad, the radar imaging target recognition is taken as the leading, the optical camera target recognition is taken as the reference, and the influence of severe weather on the system work can be effectively reduced due to the all-weather working characteristics of the radar sensor. The mutual interaction and fusion of the microwave radar and the optical vision greatly improve the stability of the system in the target recognition work and improve the overall accuracy of target recognition.
Drawings
Fig. 1 is a flowchart of an interactive perception recognition method of microwave and optical vision according to an embodiment of the present invention.
Fig. 2 is a process flow diagram of an embodiment of a method for interactive perception recognition of microwave and optical vision according to an embodiment of the present invention.
Fig. 3 is a flow chart of a depth camera acquisition process of an interactive perception recognition method of microwave and optical vision according to an embodiment of the present invention.
Fig. 4 is a flowchart of radar data acquisition processing of the interactive perception recognition method of microwave and optical vision according to an embodiment of the present invention.
Fig. 5 is a schematic diagram showing information fusion between microwave radar information and a depth camera in the interactive perception recognition method of microwave and optical vision according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of a detection device with a depth camera and a radar detection module according to an embodiment of the present invention.
Fig. 7 is a schematic block diagram of an interactive perception recognition device for microwave and optical vision according to an embodiment of the present invention.
Fig. 8 is a schematic diagram of an internal structure of a terminal device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and more specific, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It should be noted that, if directional indications (such as up, down, left, right, front, and rear … …) are included in the embodiments of the present invention, the directional indications are merely used to explain the relative positional relationship, movement conditions, etc. between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indications are correspondingly changed.
In addition, if there is a description of "first", "second", etc. in the embodiments of the present invention, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
Along with the development of technology and the continuous improvement of living standard of people, the use of target recognition technology is becoming more and more popular.
For radar target recognition, acquiring target detail information is limited due to the self-performance limitations of the radar system. When the radar system wants to acquire more target information, more radar resources such as frequency spectrum, energy and the like are consumed, and the processing time is greatly increased. The method brings great constraint to radar real-time target recognition engineering; not only is time and labor consuming, but also the target recognition effect is not good.
For the optical visual target recognition, after the monitoring, learning and training, the recognition effect is good outdoors under the condition of good weather conditions, but the recognition effect of the optical camera is suddenly reduced when the optical camera is extremely low in extreme weather such as heavy rain, dense fog, sand dust and the like or when the vehicle is driven in a night non-illumination environment, and accurate environment perception cannot be provided.
In order to solve the above problems, an embodiment of the present invention provides a method for identifying interactive perception of microwave and optical vision, where the method adopts: shooting by a set depth camera to acquire a depth image and an optical image, and acquiring a radar SAR image by a radar; performing target identification on the optical image, obtaining target category information, and fusing a depth image to obtain target distance information; labeling the target class information into the radar SAR image; acquiring a plurality of marked radar SAR images to construct a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image; when the method is applied, the current environmental data are detected, the SAR image target recognition neural network model and the depth image target recognition are controlled to be adopted for interactive recognition according to the detected different environmental data, and a recognition result is output.
Aiming at the problems that the radar has very strong capability of penetrating smoke, dust, rain and fog and has the working characteristics of all weather and all day, but the imaging resolution is inferior to that of an optical sensor, the inventor researches a target recognition processing method based on the integration of radar imaging and an optical camera: the interactive perception recognition method of the microwave and the optical vision is provided, the radar acquisition signal is imaged in real time, the optical acquisition image is recognized in real time, and the fusion strategy of the microwave radar and the optical vision is designed on the basis. The interaction between the microwave radar and the optical vision not only well solves the problem of target recognition performance attenuation caused by factors such as imaging resolution, clutter interference and the like of radar images, but also improves the working performance of the optical camera in severe weather. The cross-modal real-time information fusion ensures that the radar and the camera are mutually matched in sensing, achieves the complementary effect, and effectively ensures the reliability of the system in target recognition work. The method is applied to the fields of target detection, automatic driving, tracking and positioning and the like, and improves the accuracy of target identification.
Exemplary method
First embodiment
As shown in fig. 1, the method for identifying interactive perception of microwave and optical vision provided in embodiment 1 of the present invention includes the following steps:
step S10, shooting by a set depth camera to acquire a depth image and an optical image, and acquiring a radar SAR image by a radar;
in the embodiment of the invention, two image acquisition modules can be installed on one device, as shown in fig. 6, for example, one depth camera is installed for shooting and acquiring a depth image and an optical image, and the other radar module is installed for acquiring a radar SAR image.
S20, carrying out target recognition on the optical image, obtaining target category information, and fusing a depth image to obtain target distance information;
in the embodiment of the present invention, it is necessary to perform target recognition on the optical image, obtain target class information, and fuse a depth image to obtain target distance information, specifically for example, as shown in fig. 3: acquiring optical image information and a depth image through a depth camera; in the invention, the optical image is an RGB image obtained by a depth camera; the depth image should be a depth image (containing distance information) acquired by the depth camera.
The invention also called a target recognition network is utilized to carry out real-time target recognition on the optical image information acquired by the depth camera by utilizing the depth neural network, and the target category information C and the region D of the optical image where the target is positioned are acquired, and the coordinate range (x i ，y i ) E, D; and indexing the distance information of the target according to the depth image generated by the depth camera.
As shown in fig. 5, the target category information C is category information identifying what the target is, such as a bicycle or a person. The region D and the coordinate range of the optical image where the object is located are, for example, regions such as the middle or left side of the specific region example of the image where the identified object is located, and the coordinate range may be specific coordinate values or the like.
In contrast, according to the depth image generated by the depth camera, the distance information of the index target can be exemplified as the target class of bicycle 1 with a distance of 5.02
S30, labeling the target class information into the radar SAR image;
in the embodiment of the invention, the target class information is marked in the radar SAR image; specifically, for example, target class information acquired from an optical image and target distance information acquired from a depth image are fused and annotated to a radar SAR image.
Specifically: in the labeling mapping process, the y of the center coordinate of the region D where the target is positioned in the optical image is calculated r An azimuth dimension representative value as a depth image target; the coordinate system of the radar SAR image consists of azimuth dimension and distance dimension, and the image contains distance information and Doppler information. The coordinate system of the optical image is constituted by the abscissa and the ordinate, and each pixel is constituted by (x i ,y i ) A representation;
from the object class information of the obtained optical image, and the distance information d indexed from the depth image r Distance information d r As a distance representative value of a target in the optical image, abstracting an azimuth-distance dimension coordinate system of the optical image;
then, averaging target azimuth information of the optical image in the target T time and distance information in the depth image; obtaining the average azimuth of the target in the time of TTarget average distance within target T time +.>
Segmenting the abstract azimuth-distance dimensional coordinate system of the optical image according to distance, and utilizing different scale factors k i Correcting the position-distance dimensional coordinate system of the abstracted optical image to map the position-distance dimensional coordinate system to the radar image;
calculating the average orientation of the target in the optical image within a unit time T The average azimuth occupies the optical image width W 2 Ratio eta of (2);
by mean distance of target within T time of targetPosition dimension boundary [ L ] of extracted radar SAR image under distance i ,R i ]The method comprises the steps of carrying out a first treatment on the surface of the Width W of optical image occupied according to average azimuth 2 Calculating azimuth dimension coordinates pos of the target in the optical image in the radar SAR image according to the proportion eta of the target ori The method comprises the steps of carrying out a first treatment on the surface of the According to the target average distance within the target T time +.>Correcting and rounding the target to obtain a distance dimension coordinate pos of the target corresponding to the SAR image in the optical image dis According to azimuth dimension coordinates and distance dimension coordinates (pos ori ,pos dis ) Mapping the optical image target position into a radar SAR image;
by means of both azimuth and distance dimensions (pos) ori ,pos dis ) Taking r as an origin, searching a maximum connected domain Q in the region, wherein the maximum connected domain Q is a region of the optical image, corresponding to a target in the SAR image; marking the connected domain Q by a box, and marking the category information C obtained by the recognition of the deep learning method under the box; and the mapping of the optical image category information and the depth image distance information to the radar SAR picture is completed.
In the present invention, the detailed process of the specific steps related to the fusion labeling is explained in detail in the following step S300.
S40, acquiring a plurality of marked radar SAR images to construct a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
According to the method, the radar target annotation graph with high annotation accuracy is obtained by effectively annotating the radar SAR image under the auxiliary annotation condition of the optical image information. Then, based on the automatic labeling mechanism, a radar SAR image target recognition database is created. Training and testing the SAR image target recognition database by using a deep learning method to finally obtain the SAR image target recognition network model. The network model obtained by the training mode is tested, so that the beneficial effect is obtained, and the capability of autonomous target identification of SAR images is improved.
And S50, detecting current environmental data during application, controlling to adopt the SAR image target recognition neural network model to interactively recognize the depth image target recognition according to the detected different environmental data, and outputting a recognition result.
In the embodiment of the invention, when the method is applied specifically, the current environmental data can be detected in real time;
when the detected illumination environment and visibility environment in the current environmental data accord with a preset value, for example, the visibility is considered to be good when the visibility is more than 15 km, the visual field is clear, and/or the light environment is 100Lux-600 Lux, the weather condition is judged to be good, the target recognition of an optical image is controlled to be dominant, the target recognition of a radar SAR image is controlled to be auxiliary feedback, feedback labeling is carried out according to the target recognition result of the SAR image, interactive recognition is carried out, and the recognition result is output;
When the detected illumination environment and visibility environment in the current environmental data do not accord with a preset value (namely, the illumination environment and the visibility environment in the current environmental data do not accord with the preset value), the current weather is considered to be under extreme weather such as heavy rain, dense fog, sand dust and the like, or when the vehicle runs under the dark environment at night, the outdoor visibility is extremely low, the radar SAR image target is controlled to be identified as a leading, the optical image target is identified as an auxiliary reference, the interactive identification is carried out, and the identification result is output.
Thus, the invention passes through the fusion strategy of the microwave radar and the optical vision. The interaction between the microwave radar and the optical vision not only well solves the problem of target recognition performance attenuation caused by factors such as imaging resolution, clutter interference and the like of radar images, but also improves the working performance of the optical camera in severe weather. The cross-modal real-time information fusion ensures that the radar and the camera are mutually matched in sensing, achieves the complementary effect, and effectively ensures the reliability of the system in target recognition work.
The invention is described in further detail below by means of specific application examples:
example two
As shown in fig. 2, i.e. when the invention is implemented, an edge radar-depth camera target recognition system can be firstly built for fusion of microwave radar and optical visual information; specifically, acquiring data through an edge-side microwave radar, then processing radar data, and performing edge-side real-time imaging; meanwhile, the edge-side depth camera collects an optical image and a depth image, performs target identification on the optical image, and acquires target category and depth information after fusion of depth image information; and then, carrying out data fusion through a data fusion model, carrying out target labeling on the radar image, filtering clutter, and automatically generating a radar target labeling diagram. And then acquiring and generating a plurality of radar target annotation graphs, constructing a radar SAR image recognition data set, inputting the data set into a radar image target recognition network for training, generating a trained SAR image target recognition neural network model, detecting current environmental data when the SAR image target recognition neural network model is applied, controlling the SAR image target recognition neural network model to interactively recognize with depth image target recognition according to the detected different environmental data, and outputting a recognition result. According to the invention, through the cross learning supervision of the optical vision and the microwave radar, the system can independently and independently operate at the edge end to perform the target identification of the interaction of the microwave radar and the optical vision, and the target identification of the system is improved.
Specifically, the embodiment of the invention provides an interactive perception recognition method of microwave and optical vision, which can be applied to the fields of target detection, automatic driving, tracking and positioning and the like, and improves the accuracy of target recognition. In a specific embodiment of the invention, the method comprises the following steps:
step S100, shooting and obtaining a depth image and an optical image through a set depth camera, and carrying out real-time target identification on the shot and obtained optical image through a depth neural network to obtain target category information C, a region D of the depth image where a target is located and a coordinate range; and indexing the target depth distance information in the depth image generated by the depth camera according to the range of the target coordinates.
Specifically, as shown in fig. 3, the implementation of this step includes: the depth camera is connected with the embedded platform and is carried on the embedded platform, and the depth camera is driven to work at the embedded end. Acquiring optical image information and depth image information (depth image) by a depth camera; then carrying out target recognition on the optical image information through a deep neural network, namely a target recognition network, so as to obtain target category information and coordinate information; and then, the depth image information is indexed by the coordinate information, a target depth value is obtained from the depth image, and the target depth information and the category information are returned. In the embodiment of the invention, the depth camera can simultaneously return the optical image information and the depth distance information.
As shown in fig. 3, the depth camera is driven to work at the embedded end, the depth neural network is applied to perform real-time object recognition on the content of the photographed optical image of the depth camera, the object type information C is obtained, the region D of the optical image where the object is located, and the coordinate range (x i ，y i ) E D. And indexing the distance information of the target according to the depth image generated by the depth camera. The way of indexing the target distance is:
wherein Depth (·) represents the index Depth value, mean (·) represents the average value, d min Representing the minimum value, d, of the radar SAR image in the distance dimension max Represents the maximum value of the radar SAR image in the distance dimension.
In the embodiment of the invention, the effect is considered to be poorer if the image data acquired by the radar just begins to directly do target recognition. Therefore, the method and the device firstly label the radar image by utilizing the target identification result of the camera. And the accuracy of radar target identification is improved. A complementary effect is achieved in the final object recognition system.
And step 200, controlling the radar to acquire data, acquiring radar data, and processing the acquired data through a radar imaging algorithm to acquire a synthetic aperture radar image.
The invention can use python language to drive the real-time operation of the radar at the embedded end. On the basis, the radar is controlled to acquire data, and radar data are acquired and transmitted to the processing board. And processing the acquired data through a radar imaging algorithm, such as a BP imaging algorithm and a RD imaging algorithm, so as to obtain a radar SAR image (synthetic aperture radar image).
As shown in fig. 4, the present invention uses python (computer programming language) to drive the real-time operation of the radar at the embedded end. On the basis, the radar is controlled to acquire data, radar echo data are acquired, and the acquired radar data are transmitted to the processing board. And processing the acquired radar data through a radar imaging algorithm, such as a BP imaging algorithm and a RD imaging algorithm, so as to obtain a radar SAR imaging map, namely obtaining a radar SAR (synthetic aperture radar) image.
Among them, python (computer programming language) is a software design language. No chinese is fully known, like the C language, java language.
And step S300, fusing and labeling target category information acquired from the optical image and target distance information acquired from the depth image to the radar SAR image.
In the invention, the microwave radar information and the information of the depth camera are fused. In the step, a fusion strategy is designed, and target class information acquired from an optical image and target distance information acquired from a depth image are fused and marked to a radar SAR image.
In step S310, the radar coordinate system is matched with the depth camera coordinate system and physically calibrated. And the starting and ending moments of the data collected by the radar and the depth camera are synchronized. The coordinate system of the radar SAR image consists of azimuth dimension and distance dimension, and the image contains distance information and Doppler information.
The coordinate system of the radar SAR image is different from the coordinate system of the optical picture in meaning; one dimension of the SAR image coordinate system is the distance dimension, and the other dimension is the Doppler dimension (also called the azimuth dimension). That is, in the radar SAR image, two kinds of information, range information and doppler information, are contained.
The coordinate system of the optical image is constituted by the abscissa and the ordinate, and each pixel is constituted by (x i ,y i ) In the mapping fusion process, the invention leads the y of the central coordinate of the region D where the target is positioned in the optical image r As the azimuth dimension representative value of the object in the optical image, the object category information of the optical image obtained in step 100, and the distance information d indexed from the depth image r The invention will be d r As a distance representative value of the object in the optical image. This abstracts the azimuth-distance dimensional coordinate system of the optical image. Wherein the method comprises the steps of
Step S320, due to the frame rate fps of the acquired optical image of the depth camera within the unit time T 2 ＞fps 1 The invention averages the target azimuth information of the optical image and the distance information in the depth image in the time T at the frame rate of radar SAR imaging. Obtaining the average azimuth of the target in the T timeTarget average distance within T time- >
Wherein delta y Compensation in azimuth dimension for optical image per unit time, delta d Mean (·) represents the average for the compensation of the optical image in the distance dimension per unit time.
Step S330, because the scope of action of the radar SAR image in the azimuth dimension increases with the distance, when mapping the azimuth-distance coordinate system of the optical image abstracted in step 310 to the radar image, the invention needs to perform segmentation processing according to the distance and utilizes different scale factors k i Correcting, and co-integrating the distance d of the radar SAR image in the azimuth dimension in the whole mapping min -d max Is divided into four sections s 1 -s 2 ,s 2 -s 3 ,s 3 -s 4 ,s 4 -s 5 . Among each segment map:
1) In the embodiment of the present invention, it is necessary to calculate the average azimuth of the target in the optical image in step S320 within the unit time TThe average azimuth occupies the optical image width W 2 Is defined as the ratio eta. (the dimension of the optical image acquired by the depth camera is W 2 *H 2 )
2) The horizontal view angle of the radar is larger than that of the camera, the azimuth dimension boundary of the radar SAR image is recorded in the common view angle range of the radar and the camera, and the azimuth dimension boundary is marked as [ L, R ], and the distance value corresponding to the azimuth boundary. In the case of a fixed distance, the azimuth boundary of the radar SAR image can be uniquely determined. The corresponding azimuth dimension boundary can be de-indexed according to distance.
3) Target average distance within target T time according to step 320Position dimension boundary [ L ] of extracted radar SAR image under distance i ,R i ]. According to the proportion eta, the azimuth dimension coordinate pos of the target in the optical image in the radar SAR image can be calculated or i. According to the mean distance of the target within T time +.>Correcting and rounding the image to obtain a distance dimension coordinate pos of a target corresponding to the optical image in the SAR image dis According to (pos) ori ,pos dis ) The optical image target location can be uniquely mapped into the radar SAR image.
(pos ori ,pos dis ) The mapping process may be expressed as follows:
wherein fix (·) represents a rounding, u represents the pixel value size represented by a unit distance in the SAR image distance dimension, k i To segment correction factor, W 2 For the width of the optical picture (the dimension of the optical image acquired by the depth camera is W 2 *H 2 )，H 1 For the height of the radar SAR image (the dimension of the generated radar SAR image is W 1 *H 1 )，L i Designating a left boundary of a dimension in the upper direction of a distance for SAR images, R i Specifying the right boundary, s, of the dimension of the direction over the distance for the SAR image 1 ,s 2 ,s 3 ,s 4 ,s 5 For radar range d min -d max Is defined in the block diagram.
Step S340, after obtaining (pos ori ,pos dis ) After the coordinates, mapping the coordinates to a radar picture, searching a maximum connected domain Q in the region by taking r as a radius when the coordinates are the origin, wherein the maximum connected domain is the region of the optical image, corresponding to the target in the SAR image. The connected domain Q is marked with a box, and the category information C identified by the deep learning method in step S100 is printed below. And the mapping of the optical image category information and the depth image distance information to the radar SAR picture is completed.
In this way, the radar SAR image target annotation map with higher accuracy is obtained under the supervision and annotation of the depth image and the optical image obtained by the depth camera through the steps.
And step S350, under the auxiliary labeling condition of the optical image information, the radar SAR image is effectively labeled, and a radar target labeling chart with high labeling accuracy is obtained. The invention is based on the automatic labeling mechanism to create a radar SAR image target recognition database. Training and testing the SAR image target recognition database by using a deep learning method to finally obtain the SAR image target recognition network model. The network model obtained by the training mode is tested, so that the beneficial effect is obtained, and the capability of autonomous target identification of SAR images is improved.
And step 400, interaction of microwave radar information and optical visual information. According to the method, the switching between the optical image target recognition and the radar SAR image target recognition is performed under the condition of different weather environments.
Specifically, the step S400 specifically includes:
in step S410, when the weather condition is good, the target recognition of the optical image is dominant, the target recognition of the radar SAR image is auxiliary feedback, and because the target recognition based on the SAR image has better effect in recognizing some specific targets, when the optical image does not recognize the related targets, the feedback labeling can be performed according to the result of the target recognition of the SAR image. The coordinate mapping between the two images is seen in step S330. The radar system assists to greatly reduce the false recognition rate of the optical camera. When the weather condition is poor, the radar SAR image target recognition is taken as the dominant, the optical image target recognition is taken as the auxiliary reference, and the radar sensor has all-weather working characteristics all the time, so that the influence of severe weather on the system work can be effectively reduced.
The mutual interactive feedback and fusion of the microwave radar and the optical vision greatly improve the stability of the system in the target recognition work.
In the automatic labeling mechanism of the radar SAR image target, the distance information in the depth image is needed to be used as the target distance information in the optical image. If the method for obtaining the distance information of the target in the optical image in other ways is used, the method should be in the scope of protection of the patent, for example, obtaining the distance information of the target in a binocular ranging way, directly solving the distance information in the optical image by using a deep learning method, and the like, so as to achieve the final automatic labeling of the radar SAR image target information.
From the above, the method of the embodiment of the invention carries out real-time imaging on the radar acquisition signal, carries out real-time target recognition on the optical acquisition image, and designs a fusion strategy of the microwave radar and the optical vision on the basis. The interaction between the microwave radar and the optical vision not only well solves the problem of target recognition performance attenuation caused by factors such as imaging resolution, clutter interference and the like of radar images, but also improves the working performance of the optical camera in severe weather. The cross-modal real-time information fusion ensures that the radar and the camera are mutually matched in sensing, achieves the complementary effect, and effectively ensures the reliability of the system in target recognition work.
Exemplary apparatus
As shown in fig. 7, an embodiment of the present invention provides an interactive perception recognition apparatus for microwave and optical vision, including:
an image acquisition module 610, configured to acquire a depth image and an optical image by capturing with a set depth camera;
a radar image acquisition module 620 for acquiring a radar SAR image by a radar;
the identifying module 630 is configured to identify a target of the optical image, obtain target category information, and fuse a depth image to obtain target distance information;
the labeling fusion module 640 is configured to label the target class information into the radar SAR image;
the SAR image target recognition neural network model training module 650 is used for acquiring a plurality of annotated radar SAR images and constructing a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
the interactive application module 660 is configured to detect current environmental data during application, control interactive recognition between the SAR image target recognition neural network model and the depth image target recognition according to the detected different environmental data, and output a recognition result, as described above.
Based on the above embodiment, the present invention further provides a terminal device, where the terminal device in the embodiment of the present invention may be an intelligent robot (as shown in fig. 6), and the schematic block diagram may be as shown in fig. 8. The terminal equipment comprises a processor, a memory, a network interface and a display screen which are connected through a system bus. Wherein the processor of the terminal device is adapted to provide computing and control capabilities. The memory of the terminal device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the terminal device is used for communicating with an external terminal through a network connection. The computer program is executed by the processor to implement an interactive perceptual recognition of microwave and optical vision. The display screen of the terminal device may be a liquid crystal display screen or an electronic ink display screen.
It will be appreciated by those skilled in the art that the schematic block diagram of fig. 8 is merely a block diagram of a portion of the structure related to the present invention, and does not constitute a limitation of the terminal device to which the present invention is applied, and that a specific terminal device may include more or less components than those shown, or may combine some components or have a different arrangement of components.
In one embodiment, there is provided a terminal device comprising a memory, a processor, and a microwave and optical visual interactive perception recognition program stored on the processor and executable on the processor, the process performing the steps of:
shooting by a set depth camera to acquire a depth image and an optical image, and acquiring a radar SAR image by a radar;
performing target identification on the optical image, obtaining target category information, and fusing a depth image to obtain target distance information;
labeling the target class information into the radar SAR image;
acquiring a plurality of marked radar SAR images to construct a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
when the method is applied, the current environmental data are detected, the SAR image target recognition neural network model and the depth image target recognition are controlled to be adopted for interactive recognition according to the detected different environmental data, and a recognition result is output.
The step of capturing a depth image and an optical image through the set depth camera, and capturing a radar SAR image through a radar comprises the following steps:
controlling to start the depth camera to shoot an image, acquiring a depth image and an optical image,
and controlling the radar to acquire data, and processing the acquired radar data through a radar imaging algorithm to obtain a radar SAR image.
The step of performing target recognition on the optical image to obtain target category information and fusing a depth image to obtain target distance information comprises the following steps:
acquiring optical image information and a depth image through a depth camera;
performing real-time target identification on the optical image information acquired by the depth camera by using the depth neural network to acquire target class information C, and acquiring a region D, a coordinate range (x i ，y i )∈D；
And indexing the distance information of the target according to the depth image generated by the depth camera.
The step of labeling the target class information into the radar SAR image comprises the following steps:
and fusing and labeling target category information acquired from the optical image and target distance information acquired from the depth image to the radar SAR image.
The step of fusing and labeling the target category information acquired from the optical image and the target distance information acquired from the depth image to the radar SAR image comprises the following steps:
in the labeling mapping process, y is the center coordinate of the region D where the target is located in the optical image r An azimuth dimension representative value as a depth image target; the coordinate system of the radar SAR image consists of azimuth dimension and distance dimension, and the image contains distance information and Doppler information. The coordinate system of the optical image is constituted by the abscissa and the ordinate, and each pixel is constituted by (x i ,y i ) A representation;
from the object class information of the obtained optical image, and the distance information d indexed from the depth image r Distance information d r As a distance representative value of the object in the optical image, an azimuth-distance dimensional coordinate system of the optical image is abstracted.
The step of fusing and labeling the target category information acquired from the optical image and the target distance information acquired from the depth image to the radar SAR image further comprises the following steps:
averaging target azimuth information of the optical image and distance information in the depth image in the time of the target T; obtaining the average azimuth of the target in the time of T Target average distance within target T time +.>
Segmenting the abstract azimuth-distance dimensional coordinate system of the optical image according to distance, and utilizing different scale factors k i Correcting the position-distance dimensional coordinate system of the abstracted optical image to map the position-distance dimensional coordinate system to the radar image;
calculating the average orientation of the target in the optical image within a unit time TThe average azimuth occupies the optical image width W 2 Ratio eta of (2); />
By mean distance of target within T time of targetPosition dimension boundary [ L ] of extracted radar SAR image under distance i ,R i ]The method comprises the steps of carrying out a first treatment on the surface of the Width W of optical image occupied according to average azimuth 2 Calculating azimuth dimension coordinates pos of the target in the optical image in the radar SAR image according to the proportion eta of the target or i; according to the target average distance within the target T time +.>Correcting and rounding the target to obtain a distance dimension coordinate pos of the target corresponding to the SAR image in the optical image dis According to azimuth dimension coordinates and distance dimension coordinates (pos ori ,pos dis ) Mapping the optical image target position into a radar SAR image;
by means of both azimuth and distance dimensions (pos) ori ,pos dis ) Taking r as an origin, searching a maximum connected domain Q in the region, wherein the maximum connected domain Q is a region of the optical image, corresponding to a target in the SAR image; marking the connected domain Q by a square frame, and identifying the obtained category information C by a deep learning method Labeling under the square frame; and the mapping of the optical image category information and the depth image distance information to the radar SAR picture is completed.
When the method is applied, current environmental data are detected, and according to the detected different environmental data, the steps of controlling the adoption of the SAR image target recognition neural network model to carry out interactive recognition with the depth image target recognition and outputting a recognition result comprise the following steps:
when the method is applied, detecting current environmental data;
when the detected illumination environment and visibility environment in the current environmental data accord with a preset value, judging that the weather condition is good, controlling to take target identification of an optical image as a leading mode, taking radar SAR image target identification as auxiliary feedback, carrying out feedback labeling according to the SAR image target identification result, carrying out interactive identification, and outputting an identification result;
when the detected illumination environment and visibility environment in the current environmental data do not accord with the preset value, controlling to take radar SAR image target identification as a leading part and optical image target identification as an auxiliary reference, carrying out interactive identification, and outputting an identification result, wherein the specific steps are as described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
In summary, the invention discloses a method, a device, a terminal device and a storage medium for identifying interactive perception of microwave and optical vision, wherein the method adopts: the method comprises the following steps: the method comprises the steps of imaging radar acquisition signals in real time, carrying out real-time target recognition on optical acquisition images, and designing a fusion strategy of microwave radar and optical vision on the basis of the real-time imaging. The invention takes target recognition of the depth camera as a dominant and radar target recognition as an auxiliary. The radar system assists to greatly reduce the false recognition rate of the optical camera. When the weather condition is bad, the radar imaging target recognition is taken as the leading, the optical camera target recognition is taken as the reference, and the influence of severe weather on the system work can be effectively reduced due to the all-weather working characteristics of the radar sensor. The mutual interaction and fusion of the microwave radar and the optical vision greatly improve the stability of the system in the target recognition work and improve the overall accuracy of target recognition.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A method for interactive perception recognition of microwave and optical vision, the method comprising:
shooting by a set depth camera to acquire a depth image and an optical image, and acquiring a radar SAR image by a radar;
performing target identification on the optical image, obtaining target category information, and fusing a depth image to obtain target distance information;
labeling the target class information into the radar SAR image;
acquiring a plurality of marked radar SAR images to construct a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
when the method is applied, current environmental data are detected, according to the detected different environmental data, interactive identification is controlled to be carried out by adopting an SAR image target identification neural network model and optical image target identification, and an identification result is output;
the step of annotating the target class information into the radar SAR image comprises:
And fusing and labeling target category information acquired from the optical image and target distance information acquired from the depth image to the radar SAR image.
2. The method of claim 1, wherein the steps of capturing a depth image and an optical image by a set depth camera, and capturing a radar SAR image by a radar comprise:
controlling to start the depth camera to shoot an image, acquiring a depth image and an optical image,
and controlling the radar to acquire data, and processing the acquired radar data through a radar imaging algorithm to obtain a radar SAR image.
3. The method for identifying the interactive perception of microwave and optical vision according to claim 1, wherein the step of identifying the object of the optical image, obtaining object category information, and fusing depth images to obtain object distance information comprises the steps of:
acquiring optical image information and a depth image through a depth camera;
using deep nervesThe network carries out real-time target recognition on the optical image information acquired by the depth camera, acquires target class information C, and the region D, coordinate range (x i ，y i )∈D；
And indexing the distance information of the target according to the depth image generated by the depth camera.
4. The method of claim 1, wherein the step of fusing the target class information obtained from the optical image and the target distance information obtained from the depth image to the radar SAR image comprises:
in the labeling mapping process, y is the center coordinate of the region D where the target is located in the optical image r An azimuth dimension representative value as a depth image target; the coordinate system of the radar SAR image consists of azimuth dimension and distance dimension, and the image contains distance information and Doppler information; the coordinate system of the optical image is constituted by the abscissa and the ordinate, and each pixel is constituted by (x i ,y i ) A representation;
from the object class information of the obtained optical image, and the distance information d indexed from the depth image r Distance information d r As a distance representative value of the object in the optical image, an azimuth-distance dimensional coordinate system of the optical image is abstracted.
5. The method of claim 4, wherein the step of fusing the target class information obtained from the optical image and the target distance information obtained from the depth image to the radar SAR image further comprises:
Averaging target azimuth information of the optical image and distance information in the depth image in the time of the target T; obtaining the average azimuth of the target in the time of TTarget average distance within target T time +.>
Segmenting the abstract azimuth-distance dimensional coordinate system of the optical image according to distance, and utilizing different scale factors k i Correcting the position-distance dimensional coordinate system of the abstracted optical image to map the position-distance dimensional coordinate system to the radar image;
calculating the average orientation of the target in the optical image within a unit time TAnd the ratio eta of the average azimuth to the width W2 of the optical image;
by mean distance of target within T time of targetPosition dimension boundary [ L ] of extracted radar SAR image under distance i ,R i ]The method comprises the steps of carrying out a first treatment on the surface of the Calculating the azimuth dimension coordinates pos of the target in the optical image in the radar SAR image according to the proportion eta of the average azimuth to the width W2 of the optical image ori The method comprises the steps of carrying out a first treatment on the surface of the According to the target average distance within the target T time +.>Correcting and rounding the target to obtain a distance dimension coordinate pos of the target corresponding to the SAR image in the optical image dis According to azimuth dimension coordinates and distance dimension coordinates (pos ori ,pos dis ) Mapping the optical image target position into a radar SAR image;
by means of both azimuth and distance dimensions (pos) ori ,pos dis ) Taking r as an origin, searching a maximum connected domain Q in the region, wherein the maximum connected domain Q is a region of the optical image, corresponding to a target in the SAR image; marking the connected domain Q by a box, and marking the category information C obtained by the recognition of the deep learning method under the box; and the mapping of the optical image category information and the depth image distance information to the radar SAR picture is completed.
6. The interactive perception recognition method of microwave and optical vision according to claim 1, wherein the steps of detecting current environmental data, controlling the interactive recognition of the target recognition neural network model and the depth image by using the SAR image target recognition neural network model according to the detected different environmental data, and outputting the recognition result comprise:
when the method is applied, detecting current environmental data;
when the detected illumination environment and visibility environment in the current environmental data accord with a preset value, judging that the weather condition is good, controlling to take target identification of an optical image as a leading mode, taking radar SAR image target identification as auxiliary feedback, carrying out feedback labeling according to the SAR image target identification result, carrying out interactive identification, and outputting an identification result;
when the illumination environment and the visibility environment in the detected current environmental data do not accord with the preset value, controlling to take radar SAR image target identification as a leading, taking optical image target identification as an auxiliary reference, carrying out interactive identification, and outputting an identification result.
7. An interactive perception recognition device for microwave and optical vision, characterized in that the device comprises:
the image acquisition module is used for acquiring a depth image and an optical image through shooting of the set depth camera;
the radar image acquisition module is used for acquiring radar SAR images through a radar;
the identification module is used for carrying out target identification on the optical image, obtaining target category information, fusing the depth image and obtaining target distance information;
the annotation fusion module is used for annotating the target class information into the radar SAR image; the step of annotating the target class information into the radar SAR image comprises: fusing and labeling target category information acquired from the optical image and target distance information acquired from the depth image to the radar SAR image;
the SAR image target recognition neural network model training module is used for acquiring a plurality of marked radar SAR images and constructing a radar SAR image target database; inputting a plurality of marked radar SAR image data in the radar SAR image target database data into a set deep neural network for training to obtain a trained SAR image target recognition neural network model capable of automatically carrying out target recognition on the radar SAR image;
And the interactive application module is used for detecting the current environmental data during application, controlling the SAR image target recognition neural network model to be used for carrying out interactive recognition with the optical image target recognition according to the detected different environmental data, and outputting a recognition result.
8. A terminal device, characterized in that it comprises a memory, a processor and a program for identifying the interactive perception of microwave and optical vision stored in the memory and capable of running on the processor, the processor implementing the steps of the method for identifying the interactive perception of microwave and optical vision according to any one of claims 1-6 when executing the program for identifying the interactive perception of microwave and optical vision.
9. A computer readable storage medium, wherein a microwave and optical visual interactive perception recognition program is stored on the computer readable storage medium, and when the microwave and optical visual interactive perception recognition program is executed by a processor, the steps of the microwave and optical visual interactive perception recognition method according to any one of claims 1-6 are implemented.
CN202210600633.0A 2022-05-30 2022-05-30 Interactive perception identification method, device, terminal and medium for microwave and optical vision Active CN115015910B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210600633.0A CN115015910B (en) 2022-05-30 2022-05-30 Interactive perception identification method, device, terminal and medium for microwave and optical vision

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210600633.0A CN115015910B (en) 2022-05-30 2022-05-30 Interactive perception identification method, device, terminal and medium for microwave and optical vision

Publications (2)

Publication Number Publication Date
CN115015910A CN115015910A (en) 2022-09-06
CN115015910B true CN115015910B (en) 2024-02-13

Family

ID=83071780

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210600633.0A Active CN115015910B (en) 2022-05-30 2022-05-30 Interactive perception identification method, device, terminal and medium for microwave and optical vision

Country Status (1)

Country Link
CN (1) CN115015910B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108960300A (en) * 2018-06-20 2018-12-07 北京工业大学 A kind of urban land use information analysis method based on deep neural network
CN109360190A (en) * 2018-09-21 2019-02-19 清华大学 Building based on image superpixel fusion damages detection method and device
KR102272411B1 (en) * 2020-08-12 2021-07-02 국방과학연구소 Method and apparatus for learning artificial nearal network to improve the target recognition of simulation-image database in sar image
CN113420838A (en) * 2021-08-20 2021-09-21 中国科学院空天信息创新研究院 SAR and optical image classification method based on multi-scale attention feature fusion

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108960300A (en) * 2018-06-20 2018-12-07 北京工业大学 A kind of urban land use information analysis method based on deep neural network
CN109360190A (en) * 2018-09-21 2019-02-19 清华大学 Building based on image superpixel fusion damages detection method and device
KR102272411B1 (en) * 2020-08-12 2021-07-02 국방과학연구소 Method and apparatus for learning artificial nearal network to improve the target recognition of simulation-image database in sar image
CN113420838A (en) * 2021-08-20 2021-09-21 中国科学院空天信息创新研究院 SAR and optical image classification method based on multi-scale attention feature fusion

Also Published As

Publication number Publication date
CN115015910A (en) 2022-09-06

Similar Documents

Publication Publication Date Title
CN109100741B (en) Target detection method based on 3D laser radar and image data
CN110163904B (en) Object labeling method, movement control method, device, equipment and storage medium
CN110084895B (en) Method and equipment for marking point cloud data
CN109931939B (en) Vehicle positioning method, device, equipment and computer readable storage medium
CN110598743A (en) Target object labeling method and device
CN108875730B (en) Deep learning sample collection method, device, equipment and storage medium
CN111753649B (en) Parking space detection method, device, computer equipment and storage medium
CN103049909B (en) A kind of be focus with car plate exposure method
CN111027381A (en) Method, device, equipment and storage medium for recognizing obstacle by monocular camera
CN107894189A (en) A kind of EOTS and its method for automatic tracking of target point automatic tracing
US20220292747A1 (en) Method and system for performing gtl with advanced sensor data and camera image
CN112115913B (en) Image processing method, device and equipment and storage medium
CN111967396A (en) Processing method, device and equipment for obstacle detection and storage medium
CN114120283A (en) Method for distinguishing unknown obstacles in road scene three-dimensional semantic segmentation
CN114299146A (en) Parking assisting method, device, computer equipment and computer readable storage medium
CN111832410A (en) Forward train detection method based on fusion of vision and laser radar
CN115187941A (en) Target detection positioning method, system, equipment and storage medium
CN115015910B (en) Interactive perception identification method, device, terminal and medium for microwave and optical vision
CN117496515A (en) Point cloud data labeling method, storage medium and electronic equipment
CN111256651B (en) Week vehicle distance measuring method and device based on monocular vehicle-mounted camera
CN115792912A (en) Method and system for sensing environment of unmanned surface vehicle based on fusion of vision and millimeter wave radar under weak observation condition
CN111738035A (en) Method, device and equipment for calculating yaw angle of vehicle
CN111626078A (en) Method and device for identifying lane line
CN115830140A (en) Offshore short-range photoelectric monitoring method, system, medium, equipment and terminal
CN115755072A (en) Special scene positioning method and system based on binocular structured light camera

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant