CN114900619A - Self-adaptive exposure driving camera shooting underwater image processing system - Google Patents
Self-adaptive exposure driving camera shooting underwater image processing system Download PDFInfo
- Publication number
- CN114900619A CN114900619A CN202210487677.7A CN202210487677A CN114900619A CN 114900619 A CN114900619 A CN 114900619A CN 202210487677 A CN202210487677 A CN 202210487677A CN 114900619 A CN114900619 A CN 114900619A
- Authority
- CN
- China
- Prior art keywords
- image
- module
- network
- action
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/30—Assessment of water resources
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a self-adaptive exposure driving camera shooting underwater image processing system, which comprises: a programmable camera and a computing processing device; the computing processing device includes: the device comprises a self-adaptive exposure module and a pre-training underwater image restoration module. The system encodes an image captured by a previous frame of the programmable camera and sends the encoded image into the self-adaptive exposure module, and parameters such as shutter speed, light sensitivity, aperture and the like of the programmable camera are comprehensively adjusted by processing the image of the previous frame; ensuring that the shot original image is beneficial to deep network extraction and characteristic utilization; and finally, the pre-training underwater image restoration module judges the quality of the execution action of the intelligent body and outputs the image after color correction and characteristic enhancement, so that a better effect is achieved, and the efficiency and accuracy of underwater operation can be improved.
Description
Technical Field
The invention relates to the technical field of submarine surveying and image processing, in particular to an underwater image processing system for shooting by a self-adaptive exposure driving camera.
Background
When the underwater robot executes tasks such as submarine surveying, marine organism diversity surveying, accident surveying, target detection, path planning and the like, images transmitted by an underwater camera have the problem of color distortion due to different light attenuation rates of seawater to different wavelengths in natural light. In addition, a large amount of suspended particles in the seawater affect light transmission, so that the contrast of a shot image is low, and high-value features are difficult to extract.
At present, in the aspect of underwater image color correction and feature enhancement, the previous research is focused on feature extraction and processing of the underwater image which is completely photographed. The physics-based method is to utilize a physical model of light propagation in water and to estimate the absorption amount of light with different wavelengths and the scattering effect of suspended particles in water by applying an imaging scattering model in water. However, the physical method requires strict mathematical calculation, requires high precision for each item of data, and the scattering coefficient estimation is closely related to the water quality of the water area corresponding to the image, and is difficult to apply to an image data set having large differences in depth, water quality, and the like.
Another approach to dewatering is to use an end-to-end deep dewatering network, usually a convolutional neural network is used to extract features, and a deep network training convolutional core is used to process underwater images. However, the existing method does not consider the influence of the distance between the shot object and the camera on the imaging result, namely the shot object far away from the camera is influenced by seawater to cause stronger color cast and fuzzy effect. In the existing method, one image is still regarded as a whole during training, and the influence effect of seawater in different areas of the same image is not considered, so that the image dewatering effect is limited.
However, both the physical method and the depth network method take a finished picture taken by a camera as an input, and do not consider the influence of the parameters of the camera on the taken image and the subsequent processing. The current pictures are automatically exposed and shot by the visual perception of human eyes, but the human eyes have good visual perception and do not represent that the deep neural network is easy to extract features from the network.
Therefore, how to adjust parameters during shooting and obtain an image with a depth network easy to extract features is a problem worthy of study.
Disclosure of Invention
The invention mainly aims to provide a self-adaptive exposure driving camera shooting underwater image processing system which can solve the technical problems, and in the shooting stage, various shooting parameters of a programmable camera are innovatively adjusted by using a reinforcement learning intelligent body of a self-adaptive exposure module, so that the shot original image is beneficial to deep network extraction and the utilization of characteristics; in the processing stage, a feature fusion attention network based on a double-layer optimization framework is adopted, and corresponding weight is given to each image block according to the influence degree of seawater in the image, so that the network is trained more efficiently, and a better effect is achieved.
In order to realize the purpose, the invention adopts the technical scheme that:
the embodiment of the invention provides a self-adaptive exposure driving camera shooting underwater image processing system, which comprises: a programmable camera and a computing processing device;
the computing processing device includes: the self-adaptive exposure module and the pre-training underwater image restoration module;
the programmable camera is used for obtaining a target image of an underwater scene and transmitting the target image to the self-adaptive exposure module and the pre-training underwater image restoration module; the device is also used for receiving a parameter adjusting instruction and updating shooting parameters;
the self-adaptive exposure module dynamically senses the shooting quality of the target image, generates an instruction for adjusting the shooting parameters of the programmable camera equipment in real time and transmits the instruction to the programmable camera;
the pre-training underwater image restoration module evaluates the target image in a training stage and awards the target image to the self-adaptive exposure module; in the application stage, the target image to be processed is optimized, and the underwater image subjected to color correction and characteristic enhancement is output.
Further, the programmable camera includes: a shooting module and a communication module;
the shooting module captures an external image by using an image sensor, and transmits the image to the self-adaptive exposure module and the pre-training underwater image restoration module through the communication module; receiving an instruction from the reinforcement learning agent, and changing shooting parameters; the shooting parameters include: sensitivity, shutter speed, and aperture.
Further, the self-adaptive exposure module adopts an action-value framework and a deep network formed by an action network and a value network; and coding an image output by the last frame of the programmable camera to be used as input, outputting an adjustment action of the camera shooting parameters through action network processing, and finishing the dynamic adjustment of the camera shooting parameters.
Further, the action network includes: the system comprises a first multilayer perceptron module, a first recurrent neural network module and an activation function module;
the received image and the track information containing the historical image and the camera action are processed through the first multi-layer perceptron module, the influence of the historical image and the camera action on the time axis is considered through the first recurrent neural network module, the processed information is transmitted to the first multi-layer perceptron module to extract relevant information, and finally the action network for generating the strategy is formed after activation of the activation function.
Further, the parameter updating process of the action network is as follows:
the method comprises the following steps: sampling a training track S from an experience replay pool, the training track S comprising: images shot at historical time and corresponding camera actions; the camera action is a corresponding shooting parameter;
step two: inputting the training track S as sample information into an action network before updating to obtain probability distribution of the action executed by the programmable camera, and selecting an execution action a;
step three: gradient of action network by using action a and value function Q (s, a) output by value network
Back propagation update action network parameter pi θ ；
In the above equation, s represents an image acquired at the previous time,
represents the gradient of the parameter theta of the action network, o represents the information received by the adaptive exposure module, pi θ I.e. representing the action network of the adaptive exposure module, and outputs a probability distribution relating to the action based on the input information o.
Further, the value network is used for assisting in training of an action network, and comprises: the second multilayer perceptron module and the second recurrent neural network module;
the received image and the track information containing the historical image and the camera action are processed through the second multi-layer perceptron module, the influence of the historical image and the camera action on the time axis is considered through the second recurrent neural network module, the processed information is transmitted to the second multi-layer perceptron module to extract relevant information, and finally value evaluation about the current state is output.
Further, during training, the value network parameter updating process is as follows:
the method comprises the following steps: sampling a track S from an experience playback pool to obtain an image S acquired in a pre-conversion state, an image S' acquired in a post-conversion state and a reward r given by the environment;
step two: inputting an image s obtained in a state before conversion and an image s ' obtained in a state after conversion into a value network to obtain value functions Q (s, a) and Q (s ', a ');
step three: receiving a reward r from the environment, calculating a loss function l c -updating the parameters of the value network, r + γ Q (s ', a') -Q (s, a); where gamma is the discount factor.
Further, the pre-training underwater image restoration module adopts a double-layer optimization framework; the upper network estimates the lost weight between different areas in the image, the lower network utilizes a plurality of features to fuse the attention network to express and extract the features, and finally the image which is subjected to color correction and feature enhancement is output.
Further, the pre-training underwater image restoration module performs pre-training by using the existing data set and awards the self-adaptive exposure module according to the score of the output image;
the network parameters are updated as follows:
the method comprises the following steps: the coded image is sent to a full connection layer on an upper layer network, and a weight distribution matrix is obtained;
step two: the coded image is sent to a feature fusion attention module group on a lower network to obtain a processed image;
step three: solving the L1 loss by using the processed image and the truth value image;
step four: the loss is weighted by the weight distribution matrix of the upper network and the parameters are updated.
Compared with the prior art, the invention has the following beneficial effects:
the invention provides a self-adaptive exposure driving camera shooting underwater image processing system, which is characterized in that an image captured by a previous frame of a programmable camera is coded and then sent into a self-adaptive exposure module, and parameters such as shutter speed, light sensitivity and aperture of the programmable camera are comprehensively adjusted by processing the previous frame of the image; ensuring that the shot original image is beneficial to deep network extraction and characteristic utilization; and finally, the pre-training underwater image restoration module judges the quality of the execution action of the intelligent body and outputs the image after color correction and characteristic enhancement, so that a better effect is achieved, and the efficiency and accuracy of underwater operation can be improved.
Drawings
FIG. 1 is a schematic diagram of a system for processing underwater images for adaptive exposure-driven camera photography according to an embodiment of the present invention;
FIG. 2 is a flow chart of action network parameter update during training according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating the updating of value network parameters during training according to an embodiment of the present invention;
fig. 4 is a framework diagram of a pre-training underwater image restoration module according to an embodiment of the present invention.
Detailed Description
In order to make the technical means, the creation characteristics, the achievement purposes and the effects of the invention easy to understand, the invention is further explained by combining the specific embodiments.
In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front", "rear", "both ends", "one end", "the other end", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it is to be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "disposed," "connected," and the like are to be construed broadly, such as "connected," which may be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
The self-adaptive exposure driving camera shooting underwater image processing system provided by the invention has the main application scenes of underwater robot image sensors, seabed task data set shooting, underwater target detection and the like which need to carry out image capturing and transmission under water, and can realize the process of acquiring underwater images intelligently and efficiently.
It includes: a programmable camera and a computing processing device; the computing processing device may be one or more of, on which an adaptive exposure module and a pre-trained underwater image restoration module are loaded, as shown in fig. 1:
the programmable camera is used for obtaining a target image of an underwater scene and transmitting the target image to the self-adaptive exposure module and the pre-training underwater image restoration module; the device is also used for receiving a parameter adjusting instruction and updating shooting parameters; the programmable camera has high autonomous controllability, provides a Python code interface, can quickly and efficiently adjust shooting parameters such as light sensitivity, shutter speed, aperture and the like, and stably and continuously outputs high-quality images.
The self-adaptive exposure module dynamically senses the shooting quality of the target image, generates an instruction for adjusting the shooting parameters of the programmable camera equipment in real time and transmits the instruction to the programmable camera; the module is a reinforcement learning intelligent agent based on an action-value (Actor-criticic) algorithm framework, and the main body is an action network and a value network which are composed of a deep network. The method comprises the steps that an image of an input intelligent agent is coded by a multilayer sensing machine and then sent to a full connection layer, on one hand, parameters of Gaussian distribution are generated for an action network, and random sampling is carried out in the Gaussian distribution to obtain adjustment of camera parameters; another aspect generates expected rewards for the value network.
The pre-training underwater image restoration module adopts a deep learning network based on a double-layer optimization framework, applies a feature fusion attention network to process and extract features, and provides rewards for the reinforcement learning intelligent agent according to a processing result. The method comprises the steps of evaluating a target image in a training stage and awarding a reward to a self-adaptive exposure module; in the application stage, the target image to be processed is optimized, and the underwater image subjected to color correction and characteristic enhancement is output.
The processing system intelligently adjusts the shooting parameters of the programmable camera through the dynamic perception of the self-adaptive exposure module on the shooting quality, finally processes images through an end-to-end image characteristic enhancement network based on a double-layer optimization framework, outputs underwater images subjected to color correction and characteristic enhancement, realizes a whole set of working flow from the setting of the shooting parameters to the output of the final images, and conveys high-quality data for underwater target detection, submarine tasks and the like.
In specific implementation, the adaptive exposure driving camera shooting underwater image processing system can be designed into an image shooting system with complete encapsulation, high computational power and low power consumption, mainly comprises a programmable camera, a reinforcement learning intelligent body based on an action-value (Actor-criticic) framework, a pre-training underwater image restoration module and the like, and can realize color correction and feature enhancement of underwater images. The concrete module composition (I), (II) and the function introduction are as follows:
a programmable camera: comprises a shooting module and a communication module. The shooting module captures an external image by using the image sensor, transmits the image to the reinforcement learning agent and the pre-training underwater image restoration module through the communication module, and receives an instruction from the reinforcement learning agent to change shooting parameters such as sensitivity, shutter speed and aperture. The image sensor has a parameter adjusting function, and the communication module is, for example, a WIFI module, 5G, bluetooth or near field communication NFC.
(ii) an adaptive exposure module based on action-value (Actor-critical) framework: the module adopts an action-value (Actor-criticic) framework, encodes an image output by a frame on a programmable camera module and then uses the encoded image as input, and outputs an adjustment action of camera shooting parameters (shutter speed, light sensitivity, aperture and the like) through strategy network processing to complete dynamic adjustment of the camera shooting parameters.
Reinforcement learning can be classified into a policy-based method and a value-based method according to the difference of update modes. Training a strategy network based on a strategy method, taking situation information received by an agent as input, outputting probability distribution of all executable actions of the agent, and then sampling and selecting the executed actions according to the probability distribution; a value-based method trains a value network, and similarly, situation information received by an agent is used as input to output a value for judging whether a certain action executed under the current state is good or bad. The action-value (Actor-criticic) algorithm combines the two methods to train an action network and a value network: the action network is developed by a strategy-based method, and can efficiently select proper actions in a continuous action space; the value network is developed by a value-based method, and can evaluate the advantages and disadvantages of the current strategy in real time and correct the current strategy in time. An action-value (Actor-criticic) algorithm integrates an action network for generating a strategy and a value network for evaluating the strategy, and meanwhile, the strategy network and the value network are updated to carry out complementary training, so that the updating speed can be higher than that of a traditional strategy-based method and a value-based method.
(1) An action network module: the system comprises a multilayer perceptron module, a recurrent neural network module, an activation function module and other components. The module processes observation information and track information containing historical observation and action received by an intelligent agent through a multilayer perceptron, considers the influence of the historical observation and action on a time axis through a recurrent neural network module, transmits the processed information to the multilayer perceptron module to extract related information, and finally forms an action network for generating a strategy after activation of an activation function.
The track information includes: historical observations and actions; the historical observation refers to an image captured at a previous time; the action means the action output by the agent in reinforcement learning, and the action is the adjustment made to the camera in the past; (e.g., before 1 second, a certain image is read in, and the sensitivity of the camera is +100) the trajectory information is a series of image-motion sequences, such as < image 1, sensitivity +100> < image 2, shutter speed 1/10> … …, etc., which is a common way to consider information on time sequences in reinforcement learning.
As shown in fig. 2, during training, the action network parameter updating process is as follows:
the method comprises the following steps: sampling a training trajectory S from an experience replay pool, the training trajectory S comprising: images shot at historical time and corresponding camera actions; the camera movement serves as a corresponding shooting parameter. A large number of training samples are required in reinforcement learning, and therefore, using agents to interact with the environment results in data that can be used for training. According to a small segment in the trace as an example: comprises s t a t ,r t ,s t+1 The meaning is as follows: at time t, the environmental state is s t In this context, the agent takes action a t Change of environment to s after taking action t+1 Giving the agent a reward r at the same time t The same applies at time t + 1. Constructed in this mannerA long list, called a trace, represents a record of the entire process of an agent's complete interaction with the environment once to the end.
Step two: and inputting the training track S as sample information into the action network before updating to obtain the probability distribution of the executed action of the intelligent agent, and selecting the executed action a.
Step three: gradient of action network by using action a and value function Q (s, a) output by value network
Back propagation update action network parameter pi θ 。
In the above equation, s represents an image acquired at the previous time,
representing the gradient of the parameter theta of the action network, and o representing the information received by the agent, namely the shot image, which can be obtained from the historical track S; pi θ I.e. representing the action network of the adaptive exposure module, and outputs a probability distribution relating to the action based on the input information o.
(2) Value network: the module comprises a multilayer perceptron module and a recurrent neural network module. The module processes observation information received by an intelligent agent and track information containing historical observation and action through a multilayer perceptron module, considers the influence of the historical observation and the action on a time axis through a recurrent neural network module, transmits the processed information to the multilayer perceptron module to extract related information, and finally outputs value evaluation about the current state. The value network is used for assisting the training of the action network, and only plays a role in the training stage of the intelligent agent and does not play a role in application.
As shown in fig. 3, during training, the updating process of the value network parameters is as follows:
the method comprises the following steps: a track S is sampled from the empirical playback pool, and an image S taken in the pre-transition state, an image S' taken in the post-transition state, and a reward r given by the environment are obtained. For example, at time t, the image acquired by the environment state is s t Intelligent, intelligenceThe body taking action in this environment a t Images obtained by taking action with environmental changes to s t+1 Giving the agent a reward r at the same time t The same applies at time t + 1.
Step two: and inputting the image s acquired in the state before the conversion and the image s ' acquired in the state after the conversion into the value network to obtain the value functions Q (s, a) and Q (s ', a ').
Step three: receiving a reward r from the environment, calculating a loss function
l c -updating a parameter of the value network, r + γ Q (s ', a') -Q (s, a), where γ is the discount factor. Gamma is a hyper-parameter in reinforcement learning, is 0 to 1 in size, and represents that the reward obtained after the agent performs an action is diluted over time. For example, if reward 1 is acquired at time t after action a is taken at time t, then the reward may be considered to be completely related to a; then, at time t +1, prize 2 is acquired, and the prize portion is associated with a, and thus multiplied by the discount factor γ, representing the partial effect.
When the method is applied, the action network receives the information at the moment and outputs the adjustment of parameters such as camera light sensitivity, shutter speed, aperture and the like through the depth network, so that the dynamic adjustment of camera shooting is realized; the value network is only used for assisting the action network training and does not play a role in application.
The pre-training underwater image restoration module: and (3) estimating the lost weight among different regions by using a double-layer optimization frame by using an upper layer network, representing and extracting the characteristics by using a plurality of characteristic fusion attention networks (FFA-Net) by using a lower layer network, and finally outputting an image subjected to color correction and characteristic enhancement. The network utilizes the existing open source data set (containing a large number of underwater scene images with different colors and different environments) to pre-train, and awards the reinforcement learning intelligent agent according to the scores of the output images.
The Feature Fusion Attention Network (FFA-Net) is used as a method with excellent processing capability in the image enhancement field, a Feature Attention module (FA) is combined with a Local Residual Learning module (LRL), features of a plurality of Local Residual Learning modules are organically fused by the Feature Attention module and are transmitted to a global Residual Learning module, and Feature enhancement of an image is realized.
After the image is encoded by the multi-layer perceptron module, as shown in fig. 4, the image is sent to an upper layer network and a lower layer network respectively. The upper network further sends the coded image to a full-connection layer, extracts characteristic information to distinguish the light and heavy degrees of different parts of the image, which are influenced by seawater, and obtains a weight distribution matrix aiming at the image; and the lower layer network further sends the coded image to six feature fusion attention modules (FFA modules) to perform feature extraction and processing on the image. The network parameter updating process comprises the following steps:
the method comprises the following steps: the coded image is sent to a full connection layer on an upper layer network, and a weight distribution matrix is obtained;
step two: the coded image is sent to a feature fusion attention module group on a lower layer network to obtain a processed image;
step three: solving the L1 loss by using the processed image and the true value image;
step four: the loss is weighted by the weight distribution matrix of the upper network and the parameters are updated.
And processing the image by using the trained double-layer optimized end-to-end image enhancement network, and giving a reward to the self-adaptive exposure module according to the quality of the image processing (for example, the reward is obtained with good effect and is +1, and the reward is obtained with poor effect and is-1) during training.
According to the underwater image processing system for the self-adaptive exposure driving camera to photograph, provided by the invention, the photographing parameters of the programmable camera are dynamically adjusted by using the reinforcement learning intelligent body of the self-adaptive exposure module according to the image input obtained by each frame of the system, so that the characteristic enhancement of an original image at an image acquisition end is realized, the effects of performing color correction and characteristic enhancement on the image by an end-to-end depth network are further improved, and the efficiency and the accuracy of underwater operation are improved.
The invention adopts a reinforcement learning intelligent agent based on an action-value (Actor-criticic) algorithm to control various shooting parameters such as shutter speed, light sensitivity, aperture and the like of the programmable camera, and can shoot images which are more beneficial to end-to-end depth network extraction and utilize characteristics through training of an action network and a value network. In addition, when working scenes influencing illumination conditions, such as obvious difference of seawater turbidity, great change of underwater working depth and the like, occur in practical application, the intelligent body can rapidly and dynamically adjust various shooting parameters of the camera according to shot images, the anti-interference capability is stronger, and the robustness of a shooting system is improved.
For example, taking underwater biodiversity exploration as an example:
in the process of exploring marine organisms by using the underwater camera, due to the reasons of complex water body turbidity, large ocean depth change, different light absorption rates of different wave bands by seawater, and the like, the returned image has the characteristics of color distortion, low contrast ratio and difficult feature extraction, and the better effect cannot be obtained by directly applying the conventional example segmentation algorithm. The specific implementation steps are as follows:
1. the processing system of the invention is pre-trained on the existing data set to obtain the network parameters of the adaptive exposure module and the pre-trained underwater image restoration module.
2. The programmable camera enters the submarine environment, and the shooting module captures the current image and sends the current image to the self-adaptive exposure module and the pre-training underwater image restoration module.
3. The self-adaptive exposure module outputs an adjustment instruction for relevant parameters of the programmable camera according to the captured current image, and the quality of pictures shot by the shooting module is improved.
4. And the pre-training underwater image restoration module processes the image captured by the camera module and outputs the image subjected to color correction and characteristic enhancement.
5. Species in the image are identified using existing example segmentation algorithms.
6. And iterating the processes 2-5 for multiple times until the underwater biodiversity exploration task is completed.
The foregoing shows and describes the general principles and broad features of the present invention and advantages thereof. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (9)
1. An adaptive exposure driving camera photography underwater image processing system, comprising: a programmable camera and a computing processing device;
the computing processing device includes: the self-adaptive exposure module and the pre-training underwater image restoration module;
the programmable camera is used for obtaining a target image of an underwater scene and transmitting the target image to the self-adaptive exposure module and the pre-training underwater image restoration module; the device is also used for receiving a parameter adjusting instruction and updating shooting parameters;
the self-adaptive exposure module dynamically senses the shooting quality of the target image, generates an instruction for adjusting the shooting parameters of the programmable camera equipment in real time and transmits the instruction to the programmable camera;
the pre-training underwater image restoration module evaluates the target image in a training stage and awards the target image to the self-adaptive exposure module; in the application stage, the target image to be processed is optimized, and the underwater image subjected to color correction and characteristic enhancement is output.
2. An adaptive exposure-driven camera photography underwater image processing system according to claim 1, wherein the programmable camera comprises: a shooting module and a communication module;
the shooting module captures an external image by using an image sensor, and transmits the image to the self-adaptive exposure module and the pre-training underwater image restoration module through the communication module; receiving an instruction from the self-adaptive exposure module and changing shooting parameters; the shooting parameters include: sensitivity, shutter speed, and aperture.
3. The system of claim 2, wherein the adaptive exposure module employs a motion-value framework, a depth network consisting of a motion network and a value network; and coding an image output by the last frame of the programmable camera to be used as input, outputting an adjustment action of the camera shooting parameters through action network processing, and finishing the dynamic adjustment of the camera shooting parameters.
4. The adaptive exposure-driven camera photography underwater image processing system of claim 3, wherein the action network comprises: the system comprises a first multilayer perceptron module, a first recurrent neural network module and an activation function module;
the received image and the track information containing the historical image and the camera action are processed through the first multi-layer perceptron module, the influence of the historical image and the camera action on the time axis is considered through the first recurrent neural network module, the processed information is transmitted to the first multi-layer perceptron module to extract relevant information, and finally the action network for generating the strategy is formed after activation of the activation function.
5. The system for processing the underwater image for the adaptive exposure driving camera photography according to claim 4, wherein the parameter updating process of the action network comprises the following steps:
the method comprises the following steps: sampling a training track S from an experience replay pool, the training track S comprising: images shot at historical time and corresponding camera actions; the camera action is a corresponding shooting parameter;
step two: inputting the training track S as sample information into an action network before updating to obtain probability distribution of the action executed by the programmable camera, and selecting an execution action a;
step three: ladder-solving on action network by utilizing action a and value function Q (s, a) output by value networkDegree of rotation
Back propagation update action network parameter pi θ ；
In the above equation, s represents an image acquired at the previous time,
represents the gradient of the parameter theta of the action network, o represents the information received by the adaptive exposure module, pi θ I.e. representing the action network of the adaptive exposure module, and outputs a probability distribution relating to the action based on the input information o.
6. The adaptive exposure driven camera photography underwater image processing system of claim 4, wherein the value network is used to assist in training of a motion network, comprising: the second multilayer perceptron module and the second recurrent neural network module;
the received image and the track information containing the historical image and the camera action are processed through the second multi-layer perceptron module, the influence of the historical image and the camera action on the time axis is considered through the second recurrent neural network module, the processed information is transmitted to the second multi-layer perceptron module to extract relevant information, and finally value evaluation about the current state is output.
7. The system of claim 6, wherein the value network parameter update procedure during training is as follows:
the method comprises the following steps: sampling a track S from an experience playback pool to obtain an image S acquired in a state before transition, an image S' acquired in a state after transition and a reward r given by the environment;
step two: inputting an image s obtained in a state before conversion and an image s ' obtained in a state after conversion into a value network to obtain value functions Q (s, a) and Q (s ', a ');
step three: receiving information from the environmentReward r, calculating a loss function l c -updating the parameters of the value network, r + γ Q (s ', a') -Q (s, a); where gamma is the discount factor.
8. The adaptive exposure-driven camera photography underwater image processing system according to claim 5, wherein the pre-training underwater image restoration module employs a two-layer optimization framework; the upper network estimates the lost weight between different areas in the image, the lower network utilizes a plurality of features to fuse the attention network to express and extract the features, and finally the image which is subjected to color correction and feature enhancement is output.
9. The adaptive exposure-driven camera photography underwater image processing system according to claim 8, wherein the pre-training underwater image restoration module is pre-trained using an existing data set and awards the adaptive exposure module according to scores of output images;
the network parameters are updated as follows:
the method comprises the following steps: the coded image is sent to a full connection layer on an upper layer network, and a weight distribution matrix is obtained;
step two: the coded image is sent to a feature fusion attention module group on a lower layer network to obtain a processed image;
step three: solving the L1 loss by using the processed image and the truth value image;
step four: the loss is weighted by the weight distribution matrix of the upper network and the parameters are updated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210487677.7A CN114900619B (en) | 2022-05-06 | 2022-05-06 | Self-adaptive exposure driving camera shooting underwater image processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210487677.7A CN114900619B (en) | 2022-05-06 | 2022-05-06 | Self-adaptive exposure driving camera shooting underwater image processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114900619A true CN114900619A (en) | 2022-08-12 |
| CN114900619B CN114900619B (en) | 2023-05-05 |
Family
ID=82720250
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210487677.7A Active CN114900619B (en) | 2022-05-06 | 2022-05-06 | Self-adaptive exposure driving camera shooting underwater image processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114900619B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111079305A (en) * | 2019-12-27 | 2020-04-28 | 南京航空航天大学 | Different-strategy multi-agent reinforcement learning cooperation method based on lambda-reward |
| CN111137292A (en) * | 2018-11-01 | 2020-05-12 | 通用汽车环球科技运作有限责任公司 | Spatial and temporal attention based deep reinforcement learning for hierarchical lane change strategies for controlling autonomous vehicles |
| CN111246091A (en) * | 2020-01-16 | 2020-06-05 | 北京迈格威科技有限公司 | Dynamic automatic exposure control method and device and electronic equipment |
| CN111429433A (en) * | 2020-03-25 | 2020-07-17 | 北京工业大学 | Multi-exposure image fusion method based on attention generation countermeasure network |
| CN111708355A (en) * | 2020-06-19 | 2020-09-25 | 中国人民解放军国防科技大学 | Multi-unmanned aerial vehicle action decision method and device based on reinforcement learning |
| CN112907464A (en) * | 2021-02-01 | 2021-06-04 | 涂可致 | Underwater thermal disturbance image restoration method |
| CN113269698A (en) * | 2021-05-21 | 2021-08-17 | 中国矿业大学 | Low-exposure vein image enhancement method based on Actor-Critic model |
| CN113284064A (en) * | 2021-05-24 | 2021-08-20 | 西安理工大学 | Cross-scale context low-illumination image enhancement method based on attention mechanism |
| CN113313267A (en) * | 2021-06-28 | 2021-08-27 | 浙江大学 | Multi-agent reinforcement learning method based on value decomposition and attention mechanism |
| CN113359704A (en) * | 2021-05-13 | 2021-09-07 | 浙江工业大学 | Self-adaptive SAC-PID method suitable for complex unknown environment |
| CN113392935A (en) * | 2021-07-09 | 2021-09-14 | 浙江工业大学 | Multi-agent deep reinforcement learning strategy optimization method based on attention mechanism |
| CN113780152A (en) * | 2021-09-07 | 2021-12-10 | 北京航空航天大学 | Remote sensing image ship small target detection method based on target perception |
| CN113936219A (en) * | 2021-10-29 | 2022-01-14 | 北京航空航天大学 | Hyperspectral image band selection method based on reinforcement learning |
| CN114125216A (en) * | 2021-10-27 | 2022-03-01 | 中国科学院软件研究所 | Imaging system and imaging method for software defined satellite |
| CN114187203A (en) * | 2021-12-09 | 2022-03-15 | 南京林业大学 | Attention-optimized deep codec defogging generation countermeasure network |
| CN114399431A (en) * | 2021-12-06 | 2022-04-26 | 北京理工大学 | Dim light image enhancement method based on attention mechanism |
-
2022
- 2022-05-06 CN CN202210487677.7A patent/CN114900619B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111137292A (en) * | 2018-11-01 | 2020-05-12 | 通用汽车环球科技运作有限责任公司 | Spatial and temporal attention based deep reinforcement learning for hierarchical lane change strategies for controlling autonomous vehicles |
| CN111079305A (en) * | 2019-12-27 | 2020-04-28 | 南京航空航天大学 | Different-strategy multi-agent reinforcement learning cooperation method based on lambda-reward |
| CN111246091A (en) * | 2020-01-16 | 2020-06-05 | 北京迈格威科技有限公司 | Dynamic automatic exposure control method and device and electronic equipment |
| CN111429433A (en) * | 2020-03-25 | 2020-07-17 | 北京工业大学 | Multi-exposure image fusion method based on attention generation countermeasure network |
| CN111708355A (en) * | 2020-06-19 | 2020-09-25 | 中国人民解放军国防科技大学 | Multi-unmanned aerial vehicle action decision method and device based on reinforcement learning |
| CN112907464A (en) * | 2021-02-01 | 2021-06-04 | 涂可致 | Underwater thermal disturbance image restoration method |
| CN113359704A (en) * | 2021-05-13 | 2021-09-07 | 浙江工业大学 | Self-adaptive SAC-PID method suitable for complex unknown environment |
| CN113269698A (en) * | 2021-05-21 | 2021-08-17 | 中国矿业大学 | Low-exposure vein image enhancement method based on Actor-Critic model |
| CN113284064A (en) * | 2021-05-24 | 2021-08-20 | 西安理工大学 | Cross-scale context low-illumination image enhancement method based on attention mechanism |
| CN113313267A (en) * | 2021-06-28 | 2021-08-27 | 浙江大学 | Multi-agent reinforcement learning method based on value decomposition and attention mechanism |
| CN113392935A (en) * | 2021-07-09 | 2021-09-14 | 浙江工业大学 | Multi-agent deep reinforcement learning strategy optimization method based on attention mechanism |
| CN113780152A (en) * | 2021-09-07 | 2021-12-10 | 北京航空航天大学 | Remote sensing image ship small target detection method based on target perception |
| CN114125216A (en) * | 2021-10-27 | 2022-03-01 | 中国科学院软件研究所 | Imaging system and imaging method for software defined satellite |
| CN113936219A (en) * | 2021-10-29 | 2022-01-14 | 北京航空航天大学 | Hyperspectral image band selection method based on reinforcement learning |
| CN114399431A (en) * | 2021-12-06 | 2022-04-26 | 北京理工大学 | Dim light image enhancement method based on attention mechanism |
| CN114187203A (en) * | 2021-12-09 | 2022-03-15 | 南京林业大学 | Attention-optimized deep codec defogging generation countermeasure network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114900619B (en) | 2023-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115442515B (en) | Image processing method and apparatus | |
| CN111402146B (en) | Image processing method and image processing apparatus | |
| CN108604369B (en) | Method, device and equipment for removing image noise and convolutional neural network | |
| Jiang et al. | Underwater image enhancement with lightweight cascaded network | |
| CN109993707B (en) | Image denoising method and device | |
| Hu et al. | Underwater image restoration based on convolutional neural network | |
| CN111062880A (en) | Underwater image real-time enhancement method based on condition generation countermeasure network | |
| CN111161178A (en) | Single low-light image enhancement method based on generation type countermeasure network | |
| CN112188088B (en) | Underwater self-photographing system | |
| CN111652817B (en) | Underwater image sharpening method based on human eye visual perception mechanism | |
| Sun et al. | Underwater image enhancement with reinforcement learning | |
| CN113658040A (en) | Face super-resolution method based on prior information and attention fusion mechanism | |
| CN117351216B (en) | Image self-adaptive denoising method based on supervised deep learning | |
| Saleh et al. | Adaptive uncertainty distribution in deep learning for unsupervised underwater image enhancement | |
| CN113538702B (en) | Ocean cultivation area underwater scene panorama generation method | |
| CN110276831A (en) | Constructing method and device, equipment, the computer readable storage medium of threedimensional model | |
| CN115035010A (en) | Underwater image enhancement method based on convolutional network guided model mapping | |
| CN114004766A (en) | Underwater image enhancement method, system and equipment | |
| JP2022067858A (en) | Learned model and data processor | |
| CN109671031A (en) | A kind of multispectral image inversion method based on residual error study convolutional neural networks | |
| Han et al. | UIEGAN: Adversarial learning-based photorealistic image enhancement for intelligent underwater environment perception | |
| Li et al. | Human perceptual quality driven underwater image enhancement framework | |
| CN117351542A (en) | Facial expression recognition method and system | |
| Saleem et al. | A non-reference evaluation of underwater image enhancement methods using a new underwater image dataset | |
| JP2021033571A (en) | Information processor, control method and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |