CN111028302B - Compressed object imaging method and system based on deep learning - Google Patents
Compressed object imaging method and system based on deep learning Download PDFInfo
- Publication number
- CN111028302B CN111028302B CN201911182458.2A CN201911182458A CN111028302B CN 111028302 B CN111028302 B CN 111028302B CN 201911182458 A CN201911182458 A CN 201911182458A CN 111028302 B CN111028302 B CN 111028302B
- Authority
- CN
- China
- Prior art keywords
- image
- compressed
- target object
- network
- deep learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/002—Image coding using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a compressed object imaging method and a compressed object imaging system based on deep learning. The invention breaks through the limitation of the traditional imaging technology that only images the whole shot scene, only extracts the needed specific target object, separates the specific target object from the background, filters other background objects, completes the optical compression imaging of the specific target object, is convenient for the real-time monitoring, identification and tracking of the target, simultaneously greatly saves the occupation and storage expenditure of the data to the network bandwidth in the transmission process, solves the storage and transmission problems of mass data, and improves the real-time performance of the system.
Description
Technical Field
The invention relates to the technical fields of optical imaging, computational imaging, target detection, identification and tracking, artificial intelligence application and the like, in particular to a compressed object imaging method and system based on deep learning.
Background
Under a complex physical scene, how to simulate a biological vision system, rapidly capture important image information, process and understand the image information, actively screen out an interested target object, and always be an enthusiastic research direction of students at home and abroad, thereby having important research value. Particularly, the method enters into the digitization era that a large amount of data flows are generated every day, the characteristics of high-speed calculation of a computer are fully utilized, important image information can be rapidly captured and corresponding processing and understanding can be carried out when the method faces a complex physical scene, and interested target objects are actively screened out, so that the method becomes a great challenge of a multimedia imaging technology under the big data era.
In recent years, a feature extraction method of a Convolutional Neural Network (CNN) based on deep learning has proved to be more accurate and effective, and is gradually becoming a mainstream method in the field of computer vision because it can more effectively simulate a human visual system to acquire feature information of a target object step by step. Therefore, experts in various disciplines begin to explore the cognitive mechanism of the natural vision of living beings in a dispute, and attempt to automatically screen and identify the target objects of interest of people by using a computer technology to simulate a biological vision system.
However, most of the current target recognition and detection methods adopt a mode of acquisition before processing, which causes that a system needs to analyze and process a large amount of useless information to influence the real-time performance of the system, and the acquired massive data can bring huge pressure to the hardware of the system and the bandwidth of the network in the storage and transmission processes. Obviously, this way is limited by the information transmission speed and the information processing speed, and becomes a bottleneck for restricting the development of the way. It is urgent how to obtain information of similar quality with a small amount of sampling on the premise of ensuring that the quality presented by the information is substantially unchanged.
Disclosure of Invention
Aiming at the problems in the background technology, a novel compressed object imaging method based on deep learning is provided. And acquiring a characteristic measurement matrix of a specific target object by utilizing a characteristic extraction technology with strong and effective depth network, embedding the characteristic measurement matrix into a single-pixel compression imaging system, carrying out target object compression measurement, and finally accurately recovering a target object image from compression measurement data through a specific same deep learning network. Under a complex scene, other unnecessary interference object information can be eliminated in the imaging process, and the active imaging of a specific target object is completed.
The invention relates to a compressed object imaging method based on deep learning, which comprises the following steps:
s1, acquiring a training sample image shot by a camera;
s2, removing background interference of the training sample image to obtain a label image only containing sample objects;
s3, inputting the training sample image and the label image into a deep learning network for training to obtain a characteristic measurement matrix of a target object and a target object reconstruction sub-network;
s4, projecting a shot scene image containing a target object on a spatial light modulator loaded with the characteristic measurement matrix to obtain compressed measurement data of the target object image;
s5, importing the compressed measurement data into the target object reconstruction sub-network to obtain compressed object imaging.
The deep learning is derived from the study of artificial neural networks. The multi-layer sensor with multiple hidden layers is a deep learning structure. Deep learning discovers a distributed feature representation of data by combining low-level features to form more abstract high-level representation attribute categories or features.
The invention breaks through the limitation of the traditional imaging technology that only the whole shot image is imaged, only the required specific target object is extracted and separated from the background, other background objects are filtered, the optical compression imaging of the specific target object is finished, the real-time monitoring, identification and tracking of the target are convenient, the occupation and storage expenditure of network bandwidth in the data transmission process are greatly saved, the storage and transmission problems of mass data are solved, the real-time performance of the system is improved, and the possibility is provided for the parallel processing of machine vision and artificial intelligence.
Specifically, a light source is arranged to irradiate on a photographed scene when photographing an image of the photographed scene containing the target object; the light source is parallel light generated by light emitted by the laser through the neutral density filter, the pinhole filter and the Fourier lens in sequence.
The neutral density filter is a light intensity adjusting device. The neutral density filter is used for adjusting the light intensity of the light path to reach the conditions required by the experiment.
The laser light passing through the pinhole filter can be converged into a very small point, so that the laser light can be used as a point light source close to ideal to generate spherical waves.
The Fourier lens is used for converting the point light source output by the pinhole filter into parallel light.
Further, the step of acquiring the training sample image shot by the camera comprises the following steps: placing a sample object in different natural environments, and shooting images at fixed positions through a camera; under each natural environment, the sample object is rotated from 0 degrees to 359 degrees, and every 30 degrees of rotation is shot in an off-white mode, so that 2000 training sample images with different angles are obtained in a cumulative mode.
Further, the label image is an image only containing a sample object obtained by performing a matting operation on the training sample image by using Photoshop and removing the background interference of all natural environments.
Further, the step of inputting the training sample image and the label image into a deep learning network to train to obtain a feature measurement matrix of the target object and a target object reconstruction sub-network comprises the following steps: and placing the training sample image into the input end of the deep learning network, placing the label image into the output end of the deep learning network, training by using the feature extraction technology of the deep learning network and an end-to-end training method, iterating and optimizing errors, taking the convolutional layer parameter of the first layer of the deep learning network as a feature measurement matrix after the network converges, and reconstructing the sub-network by taking other network layers as target objects.
The deep learning network is a convolutional neural network, is a feedforward neural network, and can respond to surrounding units by artificial neurons and can perform large-scale image processing. The convolutional neural network includes a convolutional layer and a pooling layer.
Further, the characteristic measurement matrix is a measurement matrix parameter required by the compressed sensing measurement of a specific target object obtained by a characteristic extraction technology of the deep learning network; the target object reconstruction sub-network comprises an up-sampling layer, a down-sampling layer and convolution operation layers with different convolution kernel sizes and numbers.
Further, the step of projecting a photographed scene image containing the target object onto a modulator loaded with the feature measurement matrix to obtain compressed measurement data of the target object image includes: and loading the characteristic measurement matrix on the DMD spatial light modulator, inputting a shot image containing a target object, and performing compressed sampling on the shot image to obtain compressed measurement data of the target object image.
Spatial light modulators are a class of devices that can load information on a one-or two-dimensional optical data field in order to effectively exploit the inherent speed, parallelism, and interconnection capabilities of light. For example, the light field amplitude is modulated, the phase is modulated by the refractive index, the polarization state is modulated by the rotation of the polarization plane, or the incoherent-coherent light is converted, so that certain information is written into the light wave, and the purpose of light wave modulation is achieved. It is often used as a building block or key device in systems such as real-time optical information processing, optical computing, and optical neural networks.
Compressed sensing, also known as compressed sampling, compressed sensing. As a new sampling theory, by developing the sparse characteristic of the signal, under the condition of being far smaller than the Nyquist sampling rate, the discrete sample of the signal is obtained by random sampling, and then the signal is perfectly reconstructed by a nonlinear reconstruction algorithm.
Further, the compressed measurement data is imported into the target object reconstruction sub-network to obtain compressed object imaging, and the method comprises the following steps: and inputting the compressed measurement data into a target object reconstruction sub-network, and outputting a reconstruction image result of the compressed object through expansion and splicing operation, different characteristic convolution operation, fusion operation, up-sampling operation, down-sampling operation and error iteration.
The convolution operation is to start from the upper left corner of the image, open a movable window with the same size as the template, multiply the window image with the template pixels, add them, and replace the pixel brightness value in the center of the window with the calculation result. Then, the active window is shifted one column to the right and the same operation is performed. And the like, a new image can be obtained from left to right and from top to bottom.
Upsampling is the collection of samples of an analog signal. The sampling is to convert the continuous signals in time and amplitude into discrete signals in time and amplitude under the action of sampling pulse. The sampling is also known as the discretization of the waveform. Downsampling, i.e. the extraction of a signal. In fact, both up-sampling and down-sampling are re-sampling of the digital signal, the re-sampling rate being greater than the original sampling rate at which the digital signal was originally obtained (e.g., sampled from an analog signal), and less than the original sampling rate.
The invention also provides a compressed object imaging system based on deep learning, comprising:
the device comprises a light source generation module, an image acquisition module and an image reconstruction module;
the light source generation module is used for acquiring a light source required by shooting of a camera;
the image generation module is used for projecting a shot scene image containing a target object on a modulator loaded with a characteristic measurement matrix to obtain compressed measurement data of the target object image;
the image acquisition module is used for acquiring compressed measurement data of the target object image through a converging lens and a photodiode;
the image reconstruction module is used for importing the compressed measurement data into a target object reconstruction sub-network to obtain compressed object imaging.
Further, the present invention provides a readable storage medium having stored thereon a control program, characterized in that: the control program, when executed by a processor, implements the compressed object imaging method based on deep learning as described in any one of the above.
Further, the present invention also provides a computer control system including a memory, a processor, and a control program stored in the memory and executable by the processor, characterized in that: the processor, when executing the control program, implements the compressed object imaging method based on deep learning as described in any one of the above.
In order that the invention may be more clearly understood, specific embodiments thereof will be described below with reference to the accompanying drawings.
Drawings
FIG. 1 is a block diagram of a deep learning network based on a packet full convolutional neural network and an experimental system diagram of a compressed object imaging method and system based on deep learning;
FIG. 2 is a flow diagram of a compressed object imaging method and system based on deep learning;
fig. 3 is a block diagram of a compressed object imaging method and system based on deep learning.
Detailed Description
Please refer to Part1 and Part3 of fig. 1, which are deep learning network structure diagrams based on a packet full convolutional neural network according to an embodiment of the present invention.
In the deep learning network, part1 is a characteristic measurement matrix Part of an image input and extraction target object, an arrow with CS Layer character pattern represents a convolution Layer designed according to a compressed sensing theory, and parameters of the convolution Layer are used as the characteristic measurement matrix in the compressed sensing theory after network training is completed, so that efficient measurement of a specific target object is realized. The size and the number of the filters used as the first convolution layer are manually adjusted according to the needs of the samples and the compression rate, and belong to super parameters. Part3 is the reconstruction sub-network of the target object and the output Part of the image, and the arrows with conv0, conv1, conv2, conv3, conv4, inconv, outconv words represent the convolution operation layers of the convolution neural network in deep learning, which are different in the size and number of convolution kernels. The filter size, number and number of channels of each convolution operation layer are determined. Arrows with up and down typeface represent up-sampling layer and down-sampling layer operations, respectively. The arrow with the reshape+concat type indicates that the feature map with the size of 16×16×1024 is adjusted and unfolded into a picture with the size of 512×512 according to a certain sequence, and the picture is taken as the primary reconstruction result of the image and sent to a subsequent reconstruction network. The arrow with the concat word indicates that feature maps of the convolved channels of each packet are fused. Except the first Layer CS Layer as the object-specific measurement matrix Layer, the rest of the network layers are all object-specific reconstruction sub-networks.
The deep learning network of this embodiment trains the feature measurement matrix of the target object and the reconstructed sub-network of the target object as follows:
s01: a gradation image f of 512×512×1, 512×512×1 is inputGray label image f 0 (target object), the grayscale image f is divided into 16×16 small blocks each having a size of 32×32. The compressed sampling rate is taken to be 103/1024=10%, and the size of the sampling matrix Φ is 103×1024.
The gray level image f is obtained by placing a sample object in different natural environments and shooting an image at a fixed position through a camera. Under the same natural environment, from 0 degrees to 359 degrees, every 30 degrees of rotation is shot in an off-white mode, and 2000 training sample images with different angles are obtained in total.
The gray label image f 0 The Photoshop CS6 is utilized to perform the matting operation on the gray image f, and the background interference of all scenes is removed to obtain a label image only containing specific sample objects.
S02: the first layer convolution layer filter size is set to 32×32, the convolution step size is set to 32×32, and the number of filters is 103, as opposed to considering one filter as a sample measurement. All parameters of the first layer convolution layer filter are the characteristic measurement matrix phi. After the input image f passes through the first convolution Layer CS Layer, the feature map obtained has a size of 16×16×103, abbreviated as T1. The feature map T1 stores 16×16×103 data points obtained by sampling 16×16 image blocks of 32×32 size of the original input image f at 10% sampling rate, i.e., compressed sampling data f 'of the target object' 0 。
In step S02, regarding the construction of the feature measurement matrix (sampling matrix), a conventional compressed sensing sampling matrix Φ may be simulated with a convolution layer according to the block compression concept in the Compressed Sensing (CS) theory. Assuming that the division size of the image block is KxK and the sampling rate is N/N, the measurement matrix Φ is N k ＝[n/N·K 2 ]Line K 2 A matrix of columns. Then the first convolution layer will be a layer having n K ＝[n/N·K 2 ]The convolution layer of the K multiplied by K filter sets the sliding step length of the filter (convolution kernel) on the input image to K multiplied by K, thus the image block compression can be completed. The filter parameters of the convolutional layer after the network training is completed can be constructed as oneIs provided.
S03: the filter size of the second convolution layer conv0 layer is set to 1×1×1024×103, 1024 is the number of filters, and 103 is the number of filter channels. After the feature map T1 in step S02 passes through the second convolution layer conv0 layer, the output feature map has a size of 16×16×1024, abbreviated as T2. At this time, the feature map T2 stores compressed sampling data f 'in step S02' 0 Is a result of the preliminary reconstruction of (a).
S04: the feature map T2 obtained in step S03 is input to the reshape+concat layer, and the unfolding and splicing operations are performed to obtain a feature map T3, where the size of the feature map T3 is 512×512×1, and is abbreviated as T5. Thus, the preliminary reconstruction of the image is completed.
The unwrapped splice parameters are matched to the filter parameters of the first convolutional layer.
S05: and inputting the preliminarily reconstructed image T5 into a group full convolution neural network to finish the fine reconstruction of the target object. The front half part of the group full convolution neural network consists of 4 different convolution layer channels, each convolution layer channel consists of a series of convolution and downsampling operations, different convolution channels respectively extract different characteristics of a target object, finally, the target object is fused through a concat layer, and after the fusion, the reconstruction result of the target object is finally output after two layers of downsampling and three times of upsampling Is the target image f 0 Is described.
The number of filters and the size parameters of the entire network of the fine reconstruction network are fixed.
S06: reconstructing the output target objectAnd a target image f 0 MSE error calculation is performed through deep learning and stackingThe generation optimization algorithm minimizes the error to accomplish accurate reconstruction of the compressed object.
In the network training process, training parameters of the network model are as follows:
s07: after the network training is finished, the model parameters of the whole deep learning network are saved.
Please refer to Part2 of fig. 1, which is an experimental system diagram of a compressed object imaging method and system based on deep learning according to an embodiment of the present invention.
P1: the light source generation module 21 generates parallel light by sequentially passing the light emitted from the laser 211 through the neutral density filter 212, the pinhole filter 213, and the fourier lens 214.
P2: the image generating module 22, after the parallel light passes through the photographed scene image 221 containing the target object, converges the object image onto the plane of the DMD spatial light modulator 224 loaded with the feature measurement matrix of the target object in Part1 through the lenticular lens 222, so as to complete the compressed sampling of the target object.
In step P2, the feature measurement matrix 223 of the target object, i.e., the images 1-M, are images of a specific measurement matrix obtained after convergence of the deep learning network training. The process of loading the feature measurement matrix Φ onto the DMD spatial light modulator 224 for sampling can be expressed as:
y i ＝Φx i ，
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the ith image block of division size K x K>Representing the result of compression sampling of the ith image block, n K The number of measurements, i e 1,2, m represents the maximum block into which the input image can be partitioned into K x K sizesA number.
P3: the image acquisition module 23 performs acquisition of target object image compression data through the condensing lens 232 and the photodiode 231.
P4: the image reconstruction module 24 utilizes the PC 24 to guide the acquired data into the input end of a reconstruction sub-network of the target object in Part3, so as to complete the imaging of the compressed object.
Referring to fig. 2, a flow chart of a compressed object imaging method and system based on deep learning according to an embodiment of the invention is shown.
Firstly, in a training stage, training sample images are input, 2000 pictures of a sample image library are input into a deep learning network for training, after the network converges, the convolutional layer parameters of a first layer are used as a characteristic measurement matrix of a target object, and the rest of network layers are used as reconstruction sub-networks of the target object; then in the experimental stage, through a compressed object imaging experimental system based on deep learning of Part2, a shot scene image containing a target object is projected on a DMD spatial light modulator loaded with a characteristic measurement matrix of the target object in Part1 to obtain compressed measurement data of the target object image. And importing the obtained compressed measurement data into a reconstruction sub-network of the Part3 target object, and finally completing imaging of the compressed object.
The embodiment of the invention designs a deep learning network based on a group full convolution neural network for compressed object imaging, which has two functions of training a characteristic measurement matrix of a target object and completing reconstruction (imaging) of the target object. The input end of the network is a sample library picture containing specific target object images, and the label image of the output end is a label image with only specific target object images and black background. The method utilizes the advantages of the deep network strong and effective feature extraction technology and the end-to-end training, and obtains the feature measurement matrix of the specific target object and the output of the reconstructed image through iterative training.
After the network training is completed, taking the parameters of the first layer of the convolution layer of the network as the characteristic measurement matrix of the target object, and loading the characteristic measurement matrix onto the DMD to complete the efficient compression measurement of the target object.
And importing the compressed measurement data into a reconstruction sub-network of the deep learning network to complete rapid reconstruction of the compressed measurement data, thereby realizing compressed object imaging.
Please refer to fig. 3, which is a block diagram of a compressed object imaging method and system based on deep learning according to an embodiment of the present invention.
The invention also provides a compressed object imaging system based on deep learning, comprising:
a light source generation module 21, an image generation module 22, an image acquisition module 23, and an image reconstruction module 24;
the light source generating module 21 is used for acquiring a light source required by camera shooting;
the image generation module 22 is configured to project a photographed scene image containing a target object onto a spatial light modulator DMD loaded with the feature measurement matrix to obtain compressed measurement data of the target object image;
the image acquisition module 23 is used for acquiring compressed measurement data of the target object image through a converging lens and a photodiode;
the image reconstruction module 24 is configured to import the compressed measurement data into the target object reconstruction sub-network to obtain compressed object imaging.
In fig. 3, the right half corresponds to a computer software architecture diagram of a compressed object imaging method based on deep learning, which includes the steps of image input 11, feature measurement matrix 12, object imaging (image reconstruction) 31 and image output 32. The left half part corresponds to an experimental structure diagram of the compressed object imaging method and system based on deep learning, wherein the experimental structure diagram comprises a light source generation module 21, an image generation module 22, an image acquisition module 23 and an image reconstruction module 24.
In fig. 3, two steps of the image input 11 and the feature measurement matrix 12 of the target object in the computer software flow of the compressed object imaging method based on deep learning correspond to the image generation module 22 in the experimental system structure diagram; the two steps of object imaging (image reconstruction) 31 and image output 32 correspond to the image reconstruction module 24 in the experimental system.
The core of the invention is to extract the characteristic of a specific target object by using an effective deep learning characteristic extraction method, and obtain a characteristic measurement matrix of the target object through training to form a characteristic filter of the specific object. The characteristic filter is different from other common filters (such as a low-pass filter and the like), can project characteristic information of a specific target object in a complex scene to a specific mathematical characteristic signal subspace, and the subspace can separate characteristic information of the specific target object from characteristic information of other interference objects, so that the specific target object in the complex scene can be screened out by the imaging system, and an active imaging process of the specific object is realized. The feature measurement matrix of the specific target object can be generated by using a strong and effective feature extraction method of the deep learning network.
According to the invention, only the required specific target object is extracted, the specific target object is separated from the background, and other background objects are filtered, so that other unnecessary interference object information can be eliminated in the imaging process under a complex scene, and active imaging of the specific target object is completed. Meanwhile, the occupation and storage expense of the data to the network bandwidth in the transmission process are saved, the storage and transmission problems of massive data are solved, the real-time performance of the system is improved, and the possibility is provided for parallel processing of machine vision and artificial intelligence.
Simulation experiments show that the method can well remove interference background in the image, only images a specific target object, has high precision, and verifies that the compressed object imaging method based on deep learning is feasible.
Compared with the prior art, the method breaks through the limitation of imaging before processing in the common imaging method, not only can realize the active identification imaging of the specific target object at the imaging end, but also avoids the process of analyzing and processing the global information of all objects in the scene, greatly reduces the processing amount of information and the storage and transmission of useless information, saves the occupation and storage expense of the data to the network bandwidth in the transmission process, and has good instantaneity. In addition, the invention can train under the condition of very few sample data, and can obtain accurate reconstruction results with a small number of measurement times. Thus, the consumption of funds, time and manpower in the data set acquisition process is greatly reduced, and the method has important and profound application prospects in the specific fields of medical treatment, military, remote sensing, navigation and the like.
Compared with the traditional imaging method, the method can break through the limit that the traditional imaging technology only images the whole shot scene, only extracts the specific target object of interest in the imaging process, separates the specific target object from the background, and filters other background objects. Therefore, the shot and collected picture only has the interested specific object which needs to be focused, and other background interference objects do not need to be considered or even need to be transmitted, so that the purposes of saving network bandwidth occupation and video storage space can be achieved, and the active imaging process of the target object is realized.
Secondly, compared with the method of adopting a machine learning method to complete the design of a measurement matrix and adopting a traditional TVAL3 method to complete the object imaging method based on compressed sensing in the reconstruction process, the method integrates sampling and reconstruction, and realizes the collaborative optimization of the two tasks of extracting the characteristic measurement matrix of the target object and reconstructing (imaging) the target object by means of the powerful characteristic learning capability of deep learning. Under a complex scene, object imaging can be rapidly and accurately realized, and the method has the advantages of high precision, good instantaneity and strong robustness. The method has important theoretical research significance and practical application value in the fields of video monitoring, virtual reality, man-machine interaction, autonomous navigation and the like.
When imaging is carried out in a complex scene, the trained model parameters are used as a characteristic filter of a specific target object, and the target object can be screened out from a photographed scene in the imaging process by utilizing an optical and electrical imaging method, so that other interference objects in the background are filtered out, and the active imaging of the specific target is realized.
The present invention is not limited to the above-described embodiments, and if various modifications or variations of the present invention are not departing from the spirit and scope of the present invention, the present invention also includes such modifications and variations provided they fall within the scope of the claims and the equivalents thereof.
Claims (8)
1. A compressed object imaging method based on deep learning, comprising:
acquiring a training sample image shot by a camera;
removing background interference of the training sample image to obtain a label image only containing sample objects;
inputting the training sample image and the label image into a deep learning network, wherein the deep learning network comprises a plurality of network layers, and the network layers comprise a first layer convolution layer, a second layer convolution layer and a group full convolution neural network;
inputting the training sample image into the first layer convolution layer, and carrying out block compression on the training sample image according to a preset compressed sensing sampling matrix to obtain a first feature map, wherein compressed sampling data are stored in the first feature map;
inputting the first characteristic diagram into a second layer convolution layer for reconstruction to obtain a second characteristic diagram, wherein the second characteristic diagram stores reconstructed compressed sampling data;
performing unfolding and splicing operation on the second feature map to obtain a third feature map, and inputting the third feature map into the group full convolution neural network to obtain a reconstruction result of the training sample image;
training the tag image as a target image according to a reconstruction result of the training sample image and the target image, iterating and optimizing errors, and after the network is converged, taking a compressed sensing sampling matrix of a first layer convolution layer of the deep learning network as a characteristic measurement matrix and taking other network layers as target object reconstruction sub-networks, wherein the characteristic measurement matrix is a measurement matrix parameter required by compressed sensing measurement of a specific target object; the target object reconstruction sub-network comprises an up-sampling layer, a down-sampling layer and convolution operation layers with different convolution kernel sizes and numbers;
projecting a photographed scene image containing a target object onto a spatial light modulator loaded with the characteristic measurement matrix to obtain compressed measurement data of the target object image;
and importing the compressed measurement data into a reconstruction sub-network of the target object to obtain compressed object imaging.
2. The compressed object imaging method based on deep learning according to claim 1, wherein: setting a light source to irradiate on a sample object shot scene when shooting the shot scene image containing the target object; the light source is parallel light generated by light emitted by the laser through the neutral density filter, the pinhole filter and the Fourier lens in sequence.
3. The compressed object imaging method based on deep learning of claim 1, wherein the step of acquiring training sample images photographed by a camera comprises: placing a sample object in different natural environments, and shooting images at fixed positions through a camera; under each natural environment, rotating the sample object from 0 degrees to 359 degrees, shooting once in an off-white mode every 30 degrees of rotation, and accumulating to obtain 2000 training sample images with different angles; the label image is an image which is obtained by performing a matting operation on the training sample image by using Photoshop and removing the background interference of all natural environments and only contains sample objects.
4. The compressed object imaging method based on deep learning according to claim 1, wherein: the step of obtaining compressed measurement data of the target object image by projecting the shot scene image containing the target object onto a spatial light modulator loaded with the feature measurement matrix comprises the following steps: and loading the characteristic measurement matrix on the DMD spatial light modulator, inputting a shot image containing a target object, and performing compressed sampling on the shot image to obtain compressed measurement data of the target object image.
5. The compressed object imaging method based on deep learning according to claim 1, wherein the step of importing the compressed measurement data into the target object reconstruction sub-network to obtain compressed object imaging includes: and inputting the compressed measurement data into a target object reconstruction sub-network, and outputting a reconstruction image result of the compressed object through expansion and splicing operation, different characteristic convolution operation, fusion operation, up-sampling operation, down-sampling operation and error iteration.
6. A compressed object imaging system based on deep learning, comprising:
the device comprises a light source generation module, an image acquisition module and an image reconstruction module;
the light source generation module is used for acquiring a light source required by shooting of a camera;
the image generation module is used for projecting a shot scene image containing a target object on a spatial light modulator loaded with a feature measurement matrix to obtain compressed measurement data of the target object image, wherein the feature measurement matrix is measurement matrix parameters required by compressed sensing measurement of a specific target object obtained by a feature extraction technology of a deep learning network, the deep learning network comprises a plurality of network layers, the network layers comprise a first layer convolution layer, a second layer convolution layer and a group full convolution neural network, and the training steps of the deep learning network are as follows: acquiring a training sample image shot by a camera; removing background interference of the training sample image to obtain a label image only containing sample objects;
inputting the training sample image into the first layer convolution layer, and carrying out block compression on the training sample image according to a preset compressed sensing sampling matrix to obtain a first feature map, wherein compressed sampling data are stored in the first feature map;
inputting the first characteristic diagram into a second layer convolution layer for reconstruction to obtain a second characteristic diagram, wherein the second characteristic diagram stores reconstructed compressed sampling data;
performing unfolding and splicing operation on the second feature map to obtain a third feature map, and inputting the third feature map into the group full convolution neural network to obtain a reconstruction result of the training sample image;
training the tag image as a target image according to a reconstruction result of the training sample image and the target image, iterating and optimizing errors, and after the network is converged, taking a compressed sensing sampling matrix of a first layer convolution layer of the deep learning network as a characteristic measurement matrix and taking other network layers as target object reconstruction sub-networks, wherein the characteristic measurement matrix is a measurement matrix parameter required by compressed sensing measurement of a specific target object; the target object reconstruction sub-network comprises an up-sampling layer, a down-sampling layer and convolution operation layers with different convolution kernel sizes and numbers;
the image acquisition module is used for acquiring compressed measurement data of the target object image through a converging lens and a photodiode;
the image reconstruction module is used for importing the compressed measurement data into a target object reconstruction sub-network to obtain compressed object imaging, wherein the target object reconstruction sub-network comprises an up-sampling layer, a down-sampling layer and convolution operation layers with different convolution kernel sizes and numbers.
7. A readable storage medium having a control program stored thereon, characterized in that: the control program, when executed by a processor, implements the compressed object imaging method based on deep learning as claimed in any one of claims 1 to 5.
8. A computer control system comprising a memory, a processor, and a control program stored in the memory and executable by the processor, characterized in that: the processor, when executing the control program, implements the compressed object imaging method based on deep learning as set forth in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911182458.2A CN111028302B (en) | 2019-11-27 | 2019-11-27 | Compressed object imaging method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911182458.2A CN111028302B (en) | 2019-11-27 | 2019-11-27 | Compressed object imaging method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111028302A CN111028302A (en) | 2020-04-17 |
| CN111028302B true CN111028302B (en) | 2023-07-25 |
Family
ID=70207182
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911182458.2A Active CN111028302B (en) | 2019-11-27 | 2019-11-27 | Compressed object imaging method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111028302B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111372062B (en) * | 2020-05-02 | 2021-04-20 | 北京花兰德科技咨询服务有限公司 | Artificial intelligence image communication system and recording method |
| CN111711511B (en) * | 2020-06-16 | 2021-07-13 | 电子科技大学 | Method for lossy compression of frequency domain data |
| CN113052925A (en) * | 2021-04-02 | 2021-06-29 | 广东工业大学 | Compressed sensing reconstruction method and system based on deep learning |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107121709A (en) * | 2017-06-01 | 2017-09-01 | 华南师范大学 | A kind of subject imaging system and its imaging method based on compressed sensing |
| CN109284761A (en) * | 2018-09-04 | 2019-01-29 | 苏州科达科技股份有限公司 | A kind of image characteristic extracting method, device, equipment and readable storage medium storing program for executing |
| CN110188774A (en) * | 2019-05-27 | 2019-08-30 | 昆明理工大学 | A kind of current vortex scan image classifying identification method based on deep learning |
| CN110288526A (en) * | 2019-06-14 | 2019-09-27 | 中国科学院光电技术研究所 | A kind of image reconstruction algorithm based on deep learning promotes the optimization method of single pixel camera imaging quality |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10685429B2 (en) * | 2017-02-22 | 2020-06-16 | Siemens Healthcare Gmbh | Denoising medical images by learning sparse image representations with a deep unfolding approach |
-
2019
- 2019-11-27 CN CN201911182458.2A patent/CN111028302B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107121709A (en) * | 2017-06-01 | 2017-09-01 | 华南师范大学 | A kind of subject imaging system and its imaging method based on compressed sensing |
| CN109284761A (en) * | 2018-09-04 | 2019-01-29 | 苏州科达科技股份有限公司 | A kind of image characteristic extracting method, device, equipment and readable storage medium storing program for executing |
| CN110188774A (en) * | 2019-05-27 | 2019-08-30 | 昆明理工大学 | A kind of current vortex scan image classifying identification method based on deep learning |
| CN110288526A (en) * | 2019-06-14 | 2019-09-27 | 中国科学院光电技术研究所 | A kind of image reconstruction algorithm based on deep learning promotes the optimization method of single pixel camera imaging quality |
Non-Patent Citations (1)
| Title |
|---|
| 基于分层卷积深度学习系统的植物叶片识别研究;张帅等;《北京林业大学学报》;20160915;第38卷(第09期);第108-114页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111028302A (en) | 2020-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Qiao et al. | Deep learning for video compressive sensing | |
| Bashir et al. | A comprehensive review of deep learning-based single image super-resolution | |
| Mei et al. | Progressive feature fusion network for realistic image dehazing | |
| Sonka et al. | Image processing, analysis and machine vision | |
| CN111028302B (en) | Compressed object imaging method and system based on deep learning | |
| Li et al. | Survey of single image super‐resolution reconstruction | |
| CN111340183B (en) | Deep learning-based scatter imaging device and method | |
| CN115358932B (en) | Multi-scale feature fusion face super-resolution reconstruction method and system | |
| Ruan et al. | Aifnet: All-in-focus image restoration network using a light field-based dataset | |
| Zeng et al. | Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh’s drawings | |
| CN112651911A (en) | High dynamic range imaging generation method based on polarization image | |
| Yang et al. | Deep networks with detail enhancement for infrared image super-resolution | |
| Li et al. | Effective data-driven technology for efficient vision-based outdoor industrial systems | |
| Zhang et al. | Iterative multi‐scale residual network for deblurring | |
| Maiwald | A window to the past through modern urban environments: Developing a photogrammetric workflow for the orientation parameter estimation of historical images | |
| CN114119428B (en) | Image deblurring method and device | |
| Yuan et al. | Unsupervised real image super-resolution via knowledge distillation network | |
| BR et al. | Machine Learning Application for Evidence Image Enhancement | |
| CN115205974A (en) | Gesture recognition method and related equipment | |
| Liu et al. | Recurrent context-aware multi-stage network for single image deraining | |
| Wang et al. | A simple and efficient reconstruction backbone for snapshot compressive imaging | |
| Wang et al. | MSF-Net: Multi-Scale Feedback Reconstruction for Guided Depth Map Super-Resolution | |
| Sharma et al. | Multilevel progressive recursive dilated networks with correlation filter (MPRDNCF) for image super-resolution | |
| CN116310959B (en) | Method and system for identifying low-quality camera picture in complex scene | |
| Aarizou et al. | Perceptual-based super-resolution reconstruction using image-specific degradation estimation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |