Disclosure of Invention
The invention solves the technical problem of how to improve the detection precision of the remote sensing target in the remote sensing image.
The invention provides a bimodal target detection method, which comprises the following steps:
obtaining bimodal target detection data according to the initial remote sensing image data, wherein the bimodal target detection data comprises visible light image data and non-visible light image data;
obtaining depth characteristics of a visible light part in the initial remote sensing image data according to the visible light image data;
obtaining an illumination distribution image of a non-visible light part in the initial remote sensing image data according to the non-visible light image data through a Phong model and an IEMBP model;
obtaining direct illumination characteristics of each pixel point in the illumination distribution image according to the illumination distribution image;
constructing indirect illumination characteristics of each pixel point in the illumination distribution image according to the direct illumination characteristics;
and performing target detection according to the depth characteristic of the visible light part, the direct illumination characteristic and the indirect illumination characteristic.
Optionally, the obtaining bimodal target detection data according to the initial remote sensing image data includes:
classifying the initial remote sensing image data according to a spectrum band;
taking an RGB wave band image in the initial remote sensing image data as the visible light image data;
and taking a near infrared band image, a radar image and a digital elevation image in the initial remote sensing image data as the non-visible light image data.
Optionally, the obtaining the depth feature of the visible light part in the initial remote sensing image data according to the visible light image data includes:
inputting the RGB band image into a depth convolution network;
and obtaining the depth characteristic of the visible light part in the initial remote sensing image data through the output of the depth convolution network.
Optionally, the obtaining, according to the non-visible light image data, an illumination distribution image of the non-visible light part in the initial remote sensing image data through a Phong model and an IEMBP model includes:
dividing the illumination intensity of each pixel point of the invisible light image data into specular reflection light intensity, scattering light intensity and environment light intensity through the Phong model and the IEMBP model;
obtaining the specular reflection light intensity according to the digital elevation image;
obtaining scattered light intensity according to the radar image;
and obtaining an illumination distribution image according to the specular reflection light intensity and the scattering light intensity of each pixel point of the invisible light image data.
Optionally, the obtaining the specular reflection light intensity according to the digital elevation image includes:
obtaining the three-dimensional coordinates of each pixel point in the digital elevation image according to the two-dimensional coordinates of each pixel point in the preset coordinate system and the pixel value of each pixel point in the digital elevation image;
obtaining a three-dimensional curved surface of the digital elevation image according to the three-dimensional coordinates of each pixel point in the digital elevation image;
and carrying out dot product on the normal vector of each point in the three-dimensional curved surface and a tangential plane perpendicular to the normal vector and a light source of the near infrared band image to obtain the specular reflection light intensity of each pixel point in the digital elevation image.
Optionally, the obtaining the scattered light intensity according to the radar image includes:
adjusting the numerical value of each pixel point in the radar image through a residual error network, and taking the adjusted numerical value as a scattering coefficient of each pixel point in the radar image;
and carrying out dot product according to the scattering coefficient and the light source of the near infrared band image to obtain the scattering light intensity of each pixel point in the radar image.
Optionally, the obtaining the direct illumination characteristic of each pixel point in the illumination distribution image according to the illumination distribution image includes:
obtaining the environment light intensity through self-adaptive adjustment of preset learnable super parameters;
and adding the specular reflection light intensity, the scattering light intensity and the environment light intensity in the illumination distribution image to obtain the direct illumination characteristic of each pixel point in the illumination distribution image.
Optionally, the constructing an indirect illumination feature of each pixel point in the illumination distribution image according to the direct illumination feature includes:
inputting the direct illumination characteristics into a graph network to obtain a weight matrix and an adjacent matrix of the direct illumination characteristics of each pixel point in the illumination distribution image;
and obtaining the indirect illumination characteristic of each pixel point in the illumination distribution image according to the weight matrix, the adjacent matrix and the activation function.
Optionally, the detecting the target according to the depth feature of the visible light part, the direct illumination feature and the indirect illumination feature includes:
performing dot product according to the direct illumination characteristic and the indirect illumination characteristic to obtain the integral illumination characteristic of the non-visible light part in the initial remote sensing image data;
obtaining object features in the initial remote sensing image data according to the product of the integral illumination features and the depth features;
and detecting the target according to the object characteristics.
The invention also provides a bimodal target detection system comprising a computer readable storage medium storing a computer program and a processor, the computer program implementing the bimodal target detection method as described above when read and run by the processor.
According to the bimodal target detection method and system, initial remote sensing image data are divided into visible light image data and non-visible light image data, the non-visible light image data are processed by utilizing a Phong model and an IEMBP model, direct illumination characteristics and indirect illumination characteristics of a non-visible light part in the initial remote sensing image data are extracted, illumination characteristics in the visible light image data are enhanced by utilizing the IEMBP model, noise caused by uneven illumination and shadow in the initial remote sensing image data is reduced, the problems of inaccurate target detection boundary and false detection are reduced, the influence of illumination conditions in an image on target detection is reduced, finally the direct illumination characteristics and the indirect illumination characteristics are fused with depth characteristics of a visible light part to be used for target detection, the image data of the visible light part in the initial remote sensing image data are enhanced by the direct illumination characteristics and the indirect illumination characteristics, and the detection accuracy of remote sensing targets in the remote sensing image is improved.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, the present invention provides a bimodal target detection method, including:
s1: and obtaining bimodal target detection data according to the initial remote sensing image data, wherein the bimodal target detection data comprises visible light image data and non-visible light image data.
Specifically, initial remote sensing image data can be obtained from unmanned aerial vehicles, satellites or other rocker devices, the initial remote sensing image data comprises multispectral detection data, classification is carried out according to the multispectral image data, and bimodal target detection data are obtained, wherein the bimodal target detection data comprise visible light image data and non-visible light image data, the visible light data are true color images, and the images comprise more shadows and areas with insufficient illumination.
S2: and obtaining the depth characteristics of the visible light part in the initial remote sensing image data according to the visible light image data.
Specifically, the depth features of the visible light part are acquired according to the visible light image data, the features can be extracted by using a pre-trained convolutional neural network model according to the visible light image data, and the depth features capture advanced semantic information in the image, such as edges, textures, shapes and the like, so that the tasks of identifying, classifying, detecting targets and the like of the image can be facilitated, and more accurate results can be obtained in the field of image analysis.
S3: and obtaining an illumination distribution image of the non-visible light part in the initial remote sensing image data according to the non-visible light image data through a Phong model and an IEMBP model.
Specifically, an illumination distribution image is obtained according to non-visible light image data, a Phong model and an IEMBP model are utilized, wherein the Phong model is a classical illumination model and is used for calculating the influence of illumination on the surface of an object, the influence comprises scattered light, specular reflection light and ambient light, and the Phong model can be utilized for simulating the illumination effect. On the basis, the illumination depth features in the image are further enhanced and extracted by applying an IEMBP model (Illumination Enhanced Module Based on Phong shader-an illumination enhancement model based on a Phong shader, so that an illumination distribution image of a non-visible light part in the initial remote sensing image data is obtained.
S4: and obtaining the direct illumination characteristic of each pixel point in the illumination distribution image according to the illumination distribution image.
Specifically, the direct illumination of each pixel point in the illumination distribution image includes specular reflection light, scattered light and ambient light, and a corresponding direct illumination characteristic is calculated according to each pixel point in the illumination distribution image.
S5: and constructing indirect illumination characteristics of each pixel point in the illumination distribution image according to the direct illumination characteristics.
Specifically, according to the direct illumination characteristic, an indirect illumination characteristic of each pixel point in the illumination distribution image is constructed. Considering the indirect illumination in illumination, that is, the light of one point can be refracted or scattered to another point, therefore, modeling is required to be performed on the indirect illumination of each pixel point according to the direct illumination characteristic of each pixel point, so as to obtain the indirect illumination characteristic of each pixel point.
S6: and performing target detection according to the depth characteristic of the visible light part, the direct illumination characteristic and the indirect illumination characteristic.
Specifically, in the initial remote sensing image data, due to the relationship of the sun direct angle, shadows exist in a photographed scene, and the shadows block other objects, so that the texture features of the objects displayed on the image are blurred or removed. The image data of the visible light part in the initial remote sensing image data is enhanced through the direct illumination characteristic and the indirect illumination characteristic, the influence of shadows caused by the direct solar angle is reduced, and the target detection is carried out on the basis.
According to the bimodal target detection method, initial remote sensing image data are divided into visible light image data and non-visible light image data, the non-visible light image data are processed by utilizing a Phong model and an IEMBP model, direct illumination characteristics and indirect illumination characteristics of a non-visible light part in the initial remote sensing image data are extracted, illumination characteristics in the visible light image data are enhanced by utilizing the IEMBP model, noise caused by uneven illumination and shadow in the initial remote sensing image data is reduced, the problems of inaccurate target detection boundary and false detection are reduced, the influence of illumination conditions in an image on target detection is reduced, finally the direct illumination characteristics and the indirect illumination characteristics are fused with depth characteristics of a visible light part to be used for target detection, the image data of the visible light part in the initial remote sensing image data are enhanced by the direct illumination characteristics and the indirect illumination characteristics, and the detection accuracy of the remote sensing target in the remote sensing image is improved.
In the embodiment of the present invention, the obtaining bimodal target detection data according to the initial remote sensing image data includes:
classifying the initial remote sensing image data according to a spectrum band;
taking an RGB wave band image in the initial remote sensing image data as the visible light image data;
and taking a near infrared band image, a radar image and a digital elevation image in the initial remote sensing image data as the non-visible light image data.
In this embodiment, in the given initial remote sensing image data, for the construction of the bimodal target detection data, the image data may be classified and extracted according to the difference of the spectral bands. As shown in fig. 2, red, green and blue band images in the initial remote sensing image data are taken as visible light image data, that is, RGB in fig. 2. The Near Infrared band image, the radar image, and the digital elevation image are used as non-visible light image data, and correspond to NIR (Near Infrared), SAR (Synthetic Aperture Radar-synthetic aperture radar), and DEM (Digital Elevation Model-digital elevation model) in FIG. 2, respectively. For visible light image data (RGB band images) and non-visible light image data (including near infrared, radar, digital elevation images), a corresponding object detection algorithm may be used to extract bimodal object detection data.
According to the bimodal target detection method, the initial remote sensing image data is classified according to the spectrum wave bands, the data is extracted according to the characteristics of visible light and non-visible light, and bimodal target detection data can be constructed. Information of different wave bands can be fully utilized when the target detection task is carried out, and performance and robustness of the target detection model under a complex environment can be improved.
In the embodiment of the present invention, the obtaining the depth feature of the visible light portion in the initial remote sensing image data according to the visible light image data includes:
inputting the RGB band image into a depth convolution network;
and obtaining the depth characteristic of the visible light part in the initial remote sensing image data through the output of the depth convolution network.
In this embodiment, as shown in fig. 2, an image of RGB bands is input into a depth convolution network to extract depth features from the image. After the RGB image is processed by the deep convolution network, the high-level semantic features of the image, including edges, textures, shapes and the like, can be obtained, and the analysis and understanding of the image are facilitated. The deep convolution network consists of a plurality of convolution layers and pooling layers, and low-level features of the image gradually evolve into high-level abstract features through multiple convolution and pooling operations, so that the task related to the image is solved.
According to the bimodal target detection method, the RGB wave band image is input into the depth convolution network, and the depth characteristic in the image can be obtained by reading and processing the output of the depth convolution network. These features have important roles in image recognition, classification and segmentation tasks, helping to improve the performance of image analysis.
In the embodiment of the present invention, the obtaining, by using the Phong model and the IEMBP model, the illumination distribution image of the non-visible light portion in the initial remote sensing image data according to the non-visible light image data includes:
dividing the illumination intensity of each pixel point of the invisible light image data into specular reflection light intensity, scattering light intensity and environment light intensity through the Phong model and the IEMBP model;
obtaining the specular reflection light intensity according to the digital elevation image;
obtaining scattered light intensity according to the radar image;
and obtaining an illumination distribution image according to the specular reflection light intensity and the scattering light intensity of each pixel point of the invisible light image data.
In this example, the illumination intensity is divided into three components by the Phong model: scattered light, ambient light, and specularly reflected light. The ambient light refers to uniform illumination formed by light rays emitted by the light source after multiple reflections in the space; scattered light refers to the diffusely reflected portion of the object surface to the light source; specular reflection refers to specular light produced by light striking a smooth surface. The specular reflection intensity is obtained from a digital elevation image that provides three-dimensional information of the earth's surface, which in the preferred embodiment of the invention, in conjunction with fig. 2, may simulate the bounce (specular reflection) of the sun upon irradiation of the earth, such information being important for identifying surface features such as rocks, bodies of water, etc., in accordance with the DEM. SAR is used to detect the geometry of the earth and the electromagnetic properties of certain materials. The scattered light intensity is determined by analyzing the intensity of the scattering effect of the radar wave due to the surface features in the radar image. The intensity of specular and scattered light can be estimated for each pixel in the image separately by the Phong model and the IEMBP model. And then combining the two illumination intensities to obtain an illumination distribution image comprehensively considering the terrain influence, the object surface characteristics and the radar scattering characteristics.
According to the bimodal target detection method, the response of the physical characteristics of the earth surface to illumination is extracted and analyzed from the invisible light image through the Phong model and the IEMBP model.
In an embodiment of the present invention, the obtaining the specular reflection light intensity according to the digital elevation image includes:
obtaining the three-dimensional coordinates of each pixel point in the digital elevation image according to the two-dimensional coordinates of each pixel point in the preset coordinate system and the pixel value of each pixel point in the digital elevation image;
obtaining a three-dimensional curved surface of the digital elevation image according to the three-dimensional coordinates of each pixel point in the digital elevation image;
and carrying out dot product on the normal vector of each point in the three-dimensional curved surface and a tangential plane perpendicular to the normal vector and a light source of the near infrared band image to obtain the specular reflection light intensity of each pixel point in the digital elevation image.
In the present embodiment, in order to calculate specular reflection light, first, two-dimensional coordinates of each pixel point in the DEM are acquired, for example:wherein->Representing the width of the image and h representing the height of the image. Changing each two-dimensional point into a three-dimensional point +.>Wherein->The pixel values representing the points, i.e. three-dimensional point data, i.e. the 3D points in fig. 2, can be obtained. Thus, the whole image is changed from a two-dimensional plane to a curved surface in a three-dimensional space. After the curved surface in the three-dimensional space is obtained, the normal vector of each point in the curved surface can be calculated. The normal vector of each point is perpendicular to the tangent plane of that point. The illumination intensity of the specular reflection light of the point can be calculated through dot product with the light source. Wherein, in conjunction with fig. 2, the light source of the near infrared band image (NIR) is a simulated light source through the near infrared band image, and then the difference between the simulated light source and the real illumination is corrected through a residual block.
According to the bimodal target detection method, the three-dimensional coordinates of each pixel point in the digital elevation image can be obtained through the two-dimensional coordinates and the pixel value of the pixel point, and then the three-dimensional curved surface of the digital elevation image is obtained. And then, carrying out dot product by utilizing the normal vector of each point in the curved surface and a tangential plane perpendicular to the normal vector and combining a light source, so that the specular reflection light intensity of each pixel point can be obtained. By analyzing the information of the digital elevation image and the light source, the rapid measurement and analysis of the specular reflection light intensity of the specific area of the earth surface can be realized, and the environment condition and the characteristics of the specific area of the earth surface can be known.
In an embodiment of the present invention, the obtaining the scattered light intensity according to the radar image includes:
adjusting the numerical value of each pixel point in the radar image through a residual error network, and taking the adjusted numerical value as a scattering coefficient of each pixel point in the radar image;
and carrying out dot product according to the scattering coefficient and the light source of the near infrared band image to obtain the scattering light intensity of each pixel point in the radar image.
In this embodiment, for the radar image, since the radar has higher penetration capability, if the weaker the numerical value in the SAR data is, the stronger the absorption capability of the object is, reflecting the physical property of the object, and having higher absorption rate for energy. Meaning that when the light irradiates the object, most of the light will be absorbed, the scattered light is weaker and the scattering coefficient is lower. The magnitude of the scattering coefficient is not completely accurately reflected in the radar image. The values in the radar image are adjusted using the residual network block. In the illumination model used in the method, the scattering intensity of the illumination in all directions of the hemisphere is considered to be uniform. The method assumes the adjusted radar image as the scattering coefficient of the object in the image. Referring to fig. 2, the light source term is used to perform dot product with the radar image after the adjustment of the residual block data, so as to obtain the scattered light intensity.
According to the bimodal target detection method, the residual error network block is utilized to adjust the numerical value in the radar image, so that a more accurate scattering coefficient is obtained. In the illumination model, it is assumed that the scattering intensity of the illumination in various directions of the hemisphere is uniform. The scattered light intensity of each pixel point can be obtained by dot product of the light source item and the radar image adjusted by the residual network.
In the embodiment of the present invention, the obtaining the direct illumination characteristic of each pixel point in the illumination distribution image according to the illumination distribution image includes:
obtaining the environment light intensity through self-adaptive adjustment of preset learnable super parameters;
and adding the specular reflection light intensity, the scattering light intensity and the environment light intensity in the illumination distribution image to obtain the direct illumination characteristic of each pixel point in the illumination distribution image.
In this embodiment, the ambient light generally refers to a light source with no specific directivity, which is from all directions of the surrounding environment, so that the ambient light is smooth and uniform for the entire scene, and when the ambient light intensity is acquired, the super-parameters need to be adaptively adjusted by an optimization algorithm. As shown in connection with fig. 2, the ambient light intensity is obtained by learning the superparameter and providing a convolution block. The direct illumination characteristic of each pixel point in the illumination distribution image can be obtained by adding, and in the embodiment of the present invention, the direct illumination characteristic of each pixel point can be calculated by a light intensity formula:
;
wherein,for specularly reflecting the light intensity>For scattering light intensity +.>Is the ambient light intensity.
According to the bimodal target detection method, the direct illumination characteristics of each pixel point in the illumination distribution image obtained by the method can reflect the real illumination condition more accurately.
In the embodiment of the present invention, the constructing an indirect illumination characteristic of each pixel point in the illumination distribution image according to the direct illumination characteristic includes:
inputting the direct illumination characteristics into a graph network to obtain a weight matrix and an adjacent matrix of the direct illumination characteristics of each pixel point in the illumination distribution image;
and obtaining the indirect illumination characteristic of each pixel point in the illumination distribution image according to the weight matrix, the adjacent matrix and the activation function.
In this embodiment, as shown in fig. 2, the direct illumination feature is input into a graph network to obtain a weight matrix and an adjacent matrix of the direct illumination feature of each pixel point in the illumination distribution image, and finally output through a pyramid network; wherein, based on the graph network, the calculation mode is as follows:
;
wherein,for the weight matrix, fine tuning by learning, +.>For the regularized adjacency matrix, +.>Is->The characteristics of the layers ultimately result in illumination characteristics that combine direct illumination and indirect illumination.
According to the bimodal target detection method, the direct illumination characteristics are input into the graph network, so that the weight matrix and the adjacent matrix of the direct illumination characteristics of each pixel point can be obtained through learning. The weights and the adjacency relations can help the model to understand the association between the pixel points, so that the propagation and influence of illumination in a scene can be simulated more truly, and the illumination distribution image is more realistic.
In an embodiment of the present invention, the performing object detection according to the depth feature of the visible light portion, the direct illumination feature, and the indirect illumination feature includes:
performing dot product according to the direct illumination characteristic and the indirect illumination characteristic to obtain the integral illumination characteristic of the non-visible light part in the initial remote sensing image data;
obtaining object features in the initial remote sensing image data according to the product of the integral illumination features and the depth features;
and detecting the target according to the object characteristics.
In this embodiment, the original image features are element multiplied with the illumination features.
;
Representing the characteristics of an object->Representing depth features>Representing the overall illumination characteristics. If no direct solar angle is affected, the characteristic of a certain point in the image is obvious and the value is large, but in actual situation, the shadow is covered during shooting, and the shadow place in the image is +.>Smaller, the original characteristic size can be reflected by the illumination characteristics constructed by the illumination model,/-degree>The method is relatively large, and features of the object in the original image can be amplified by a multiplication method, so that the texture features of the object are more obvious and are easier to distinguish.
According to the bimodal target detection method, the image data of the visible light part in the initial remote sensing image data is enhanced through the direct illumination characteristic and the indirect illumination characteristic, and the detection precision of the remote sensing target in the remote sensing image is improved.
The invention also provides a bimodal target detection system comprising a computer readable storage medium storing a computer program and a processor, the computer program implementing the bimodal target detection method as described above when read and run by the processor.
According to the bimodal target detection system, initial remote sensing image data are divided into visible light image data and non-visible light image data, the non-visible light image data are processed by utilizing a Phong model and an IEMBP model, direct illumination characteristics and indirect illumination characteristics of a non-visible light part in the initial remote sensing image data are extracted, illumination characteristics in the visible light image data are enhanced by utilizing the IEMBP model, noise caused by uneven illumination and shadow in the initial remote sensing image data is reduced, the problems of inaccurate target detection boundary and false detection are reduced, the influence of illumination conditions in an image on target detection is reduced, finally the direct illumination characteristics and the indirect illumination characteristics are fused with depth characteristics of a visible light part to be used for target detection, the image data of the visible light part in the initial remote sensing image data are enhanced by the direct illumination characteristics and the indirect illumination characteristics, and the detection accuracy of the remote sensing target in the remote sensing image is improved.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.