CN113256541A - Method for removing water mist from drilling platform monitoring picture by machine learning - Google Patents

Method for removing water mist from drilling platform monitoring picture by machine learning Download PDF

Info

Publication number
CN113256541A
CN113256541A CN202110803908.6A CN202110803908A CN113256541A CN 113256541 A CN113256541 A CN 113256541A CN 202110803908 A CN202110803908 A CN 202110803908A CN 113256541 A CN113256541 A CN 113256541A
Authority
CN
China
Prior art keywords
model
neural network
training
video
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110803908.6A
Other languages
Chinese (zh)
Other versions
CN113256541B (en
Inventor
刘云川
郑光胜
郑侃
蒋俊之
黄进凯
杨正川
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baolong Hongrui Technology Co.,Ltd.
Chongqing Hongbao Technology Co.,Ltd.
Sichuan hongbaorunye Engineering Technology Co.,Ltd.
Original Assignee
Beijing Baolong Hongrui Technology Co ltd
Chongqing Qingyun Petroleum Engineering Technology Co ltd
Sichuan Hongbaorunye Engineering Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baolong Hongrui Technology Co ltd, Chongqing Qingyun Petroleum Engineering Technology Co ltd, Sichuan Hongbaorunye Engineering Technology Co ltd filed Critical Beijing Baolong Hongrui Technology Co ltd
Priority to CN202110803908.6A priority Critical patent/CN113256541B/en
Publication of CN113256541A publication Critical patent/CN113256541A/en
Application granted granted Critical
Publication of CN113256541B publication Critical patent/CN113256541B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention belongs to the field of combination of image processing and machine learning in specific industries, and particularly relates to a method for removing water mist from a drilling platform monitoring picture by utilizing machine learning. In the method, a video sample of a monitoring picture of the drilling platform is collected through a camera according to a preset frame rate, frames are extracted from the video sample, continuous 8 images are taken from a video sequence containing fog to form a video frame sequence, a prepared training set is divided into a part A and a part B, wherein the part A accounts for 30% of the training set, the rest part is the part B, a model is trained and updated, and water fog is removed from the monitoring picture of the drilling platform through the model.

Description

Method for removing water mist from drilling platform monitoring picture by machine learning
Technical Field
The invention belongs to the field of combination of image processing and machine learning in specific industries, and particularly relates to a method for removing water mist from a drilling platform monitoring picture by utilizing machine learning.
Background
In the oil and gas exploration process, when video monitoring equipment is deployed on a drilling platform, the monitoring is seriously influenced due to the problems that fog generated by vapor transpiration at the bottom of the platform has great interference on images collected by a camera, even the situation cannot be identified and the like.
The above information disclosed in this background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method for removing water mist from a drilling platform monitoring picture by using machine learning, which solves the existing defects, and recovers a high-quality clear image by removing the influence of platform mist in the image through continuous frames in a video.
According to the method, modeling of atmospheric parameters is not needed, the restored image is directly output by analyzing context information in the video in an end-to-end mode, the method is more suitable for the actual drilling platform scene, and the robustness is higher.
The purpose of the invention is realized by the following technical scheme:
a method for removing water mist from a drilling platform monitoring picture by machine learning comprises the following steps:
s100: acquiring a video sample of a drilling platform through a camera according to a preset frame rate, and performing frame extraction on the video sample, wherein the video sample comprises a fog-containing video and a fog-free video;
s200: taking continuous 8 frames of pictures from the fog-containing video, and combining the pictures into a video frame sequence X, wherein X = { X = { (X) }i,xi+1,…,xi+7Selecting a frame of actual picture X 'similar to any image in the video frame sequence X from the fog-free video as a target picture, and forming a training set X by the X' and the video frame sequence Xtrain={X,x’}；
S300: the prepared training set XtrainThe training model is divided into A, B parts, wherein, part A accounts for 30% of the training set, the rest part is part B, and the model is trained according to the part A = { G = {model,DmodelIn which G ismodelTo generate a model, DmodelIs a discriminant model; wherein, the training is divided into 2 stages, and A, B parts of the training set are respectively used as G in the 2 stagesmodelThe input of (1);
s400: after the model is trained, a generative model G in the model is utilizedmodelDefogging a video-based continuous frame image of a drilling rig.
Preferably, the predetermined frame rate is a frame rate of 25 frames.
Preferably, step S300 includes:
stage 1 of training:
1) inputting part A of training set into part GmodelObtaining GmodelThe output restored picture y';
2) calculating the mean square error loss of y 'and x':
Figure 623817DEST_PATH_IMAGE001
wherein x isj’、yj' are pixel values of x ' and y ', respectively, n is the number of pixels, and j is a value from 0 to n;
3) if loss is not less than the set first threshold, then l is comparedThe oss derivative calculates the gradient and propagates the gradient back to GmodelUpdating the parameters in the model until loss is less than the set first threshold;
if loss is less than the first threshold, entering the 2 nd stage of training;
stage 2 of training:
1) inputting part B of training set into part GmodelObtaining GmodelThe output restored picture y';
2) respectively inputting y 'and x' into a discriminant model DmodelObtaining two discrimination probabilities dfAnd drWhile the maximum value is found by:
max(log(df)+log(1-dr))，
and updating D with the maximized valuemodelWhen two discrimination probabilities df、drWhen the value of (A) approaches 0.5, the updating is stopped;
3) obtaining DmodelIntermediate features in the calculation of the discrimination probability to calculate the perception error Ploss:
Figure 427694DEST_PATH_IMAGE002
wherein v isf、vrThe intermediate features of y 'and x' respectively, m is the number of the intermediate features, and k is from 0 to m;
if Ploss is not less than the second threshold, then the Ploss is propagated backwards to update GmodelUntil Ploss is less than said set second threshold;
and when Ploss is smaller than a set second threshold value, finishing training.
Preferably, when GmodelWhen the neural network is a deep neural network, the method also comprises the following steps:
s301: manually marking each image for training the deep neural network, and marking the positions of key points to obtain each marked image;
s302: and taking each marked image as the input of a corresponding stage of training to construct an auxiliary neural network, and assisting the deep neural network in learning and training the images.
Preferably, step S302 includes the steps of:
s3021: taking each marked image as input, selecting a proper middle layer from the deep neural network, and acquiring the output of the middle layer;
s3022: establishing an auxiliary neural network formed by convolution functions;
s3023: inputting the output of the middle layer and the corresponding attitude estimation matrix of each image before labeling into an auxiliary neural network;
s3024: and combining the outputs of the auxiliary neural network and the deep neural network, and jointly inputting the outputs into a loss function of the deep neural network to optimize the learning of the deep neural network.
Preferably, the attitude estimation matrix in step S3023 is obtained by:
s30231: calibrating a camera of the camera, and solving internal parameters of the camera, wherein the internal parameters comprise: the image optical axis principal point, the focal lengths in the X direction and the Y direction, the tangential distortion coefficient and the radial distortion coefficient;
s30232: the attitude estimation matrix is further solved as follows:
solving an attitude estimation matrix [ R | t ] (X = M [ R | t ])/X),
wherein M is an internal parameter of the camera, X is a world coordinate system, and X is an image pixel coordinate of a known shot object; r, t are the rotation vector and the translation vector of the attitude estimation matrix, respectively.
Preferably, step S3021 includes:
the internal parameters of the camera are solved by a Zhang Zhengyou calibration method and a checkerboard with known size.
Preferably, the grid is 10cm by 10 cm.
Preferably, the deep neural network selects ResNet 50.
Preferably, the auxiliary neural network selects ResNet 18.
In addition, the invention also discloses a method for removing water mist from the monitoring picture of the drilling platform by using machine learning, which comprises the following steps:
in the first step, a video sample is collected through a camera according to a preset frame rate, and frame extraction is carried out on the video sample:
Figure 717861DEST_PATH_IMAGE003
wherein X is a sequence of video frames comprising fog, X' is a sequence of video frames having no or negligible fog effect, Xn、xm'are images extracted after corresponding video frames are extracted from X and X', respectively;
in a second step, 8 consecutive images are taken from the video sequence X containing fog, combined into a video frame sequence,
Figure 457147DEST_PATH_IMAGE004
selecting and from fog-free video
Figure 320847DEST_PATH_IMAGE005
A video frame x' with similar images is taken as a target picture to form a training set
Figure 413568DEST_PATH_IMAGE006
；
In the third step, the prepared training set is used
Figure 807509DEST_PATH_IMAGE007
Dividing the training set into two parts, namely A and B, wherein the part A accounts for 30% of the training set, and the rest part is B, so as to train the model;
and in the fourth step, the trained model is used for removing water mist from the images of the drilling platform based on the video continuous frames.
Compared with the traditional marking method, the method has the advantages that the influence of platform fog in the image is removed through continuous frames in the video, and the computer vision algorithm of the high-quality clear image is restored. Different from other traditional methods of image enhancement based on a single picture, such as estimation of an atmospheric degradation model or histogram equalization, the method does not need to model atmospheric parameters, directly outputs the restored image by analyzing context information in a video in an end-to-end mode, better accords with the actual application of a drilling platform scene, and has higher robustness.
The above description is only an overview of the technical solutions of the present invention, and in order to make the technical means of the present invention more clearly apparent, and to make the implementation of the content of the description possible for those skilled in the art, and to make the above and other objects, features and advantages of the present invention more obvious, the following description is given by way of example of the specific embodiments of the present invention.
Drawings
Various other advantages and benefits of the present invention will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. Also, like parts are designated by like reference numerals throughout the drawings.
In the drawings:
FIG. 1 is a schematic diagram of a model training process according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a model training process according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a model training process according to an embodiment of the present invention;
FIG. 4 is a flow chart illustrating an embodiment of the present invention.
The invention is further explained below with reference to the figures and examples.
Detailed Description
Specific embodiments of the present invention will be described in more detail below with reference to fig. 1 to 4. While specific embodiments of the invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
It should be noted that certain terms are used throughout the description and claims to refer to particular components. As one skilled in the art will appreciate, various names may be used to refer to a component. This specification and claims do not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. The description which follows is a preferred embodiment of the invention, but is made for the purpose of illustrating the general principles of the invention and not for the purpose of limiting the scope of the invention. The scope of the present invention is defined by the appended claims.
For the purpose of facilitating understanding of the embodiments of the present invention, the following description will be made by taking specific embodiments as examples with reference to the accompanying drawings, and the drawings are not to be construed as limiting the embodiments of the present invention.
A method for defogging a rig monitor screen using machine learning, the method comprising the steps of:
in the first step, a video sample is collected through a camera according to a preset frame rate, and frame extraction is carried out on the video sample:
Figure 858642DEST_PATH_IMAGE003
wherein X is a sequence of video frames comprising fog, X' is a sequence of video frames having no or negligible fog effect, Xn、xm'are images extracted after corresponding video frames are extracted from X and X', respectively;
in a second step, 8 consecutive images are taken from the video sequence X containing fog, combined into a video frame sequence,
Figure 315031DEST_PATH_IMAGE004
selecting and from fog-free video
Figure 962175DEST_PATH_IMAGE005
A video frame x' with similar images is taken as a target picture to form a training set
Figure 961355DEST_PATH_IMAGE006
；
In the third step, the prepared training set is divided into two parts A and B, wherein the part A accounts for 30% of the training set, and the rest part is B, so as to train the model;
and in the fourth step, the trained model is used for removing water mist from the images of the drilling platform based on the video continuous frames.
In the third step, if the training can not meet the requirements, returning to the second step again, re-selecting another continuous 8 images, re-selecting and re-selecting
Figure 167078DEST_PATH_IMAGE005
A new video frame x' with similar image is used as the target picture, and the second step and the third step are executed again.
In a preferred embodiment of the method, the predetermined frame is a frame rate of 25 frames.
To further understand the present invention, in one embodiment, the method trains a model capable of reconstructing a clear image from an original image disturbed by fog by using a video acquired from an actual scene based on a deep learning computer vision model. The model is divided into two parts: model = { G =model,Dmodel}；
Wherein G ismodelTo generate a model, DmodelIs a discriminant model. In the training process, the method inputs G the video sequence interfered by water mistmodelAnd with the expectation that it will get the defogged picture, while passing through DmodelFor GmodelG, judging and supervising produced recovered picturesmodelIn the training process of (1), wherein GmodelIs a deep neural network. Preferably, in one embodiment, DmodelIs a deep neural network. More preferablyIn one embodiment, only the generative model G needs to be deployed in the actual algorithm deploymentmodelTo obtain a restored picture.
For the present invention, the preparation process of the training sample is as follows:
1) acquiring a video sample according to a frame rate of 25 frames by a camera, and performing frame extraction on the video sample:
Figure 376342DEST_PATH_IMAGE008
wherein X and X' are respectively a video frame sequence containing fog and a video frame sequence without fog influence or with small fog influence, and Xn、xm'are images extracted after corresponding video frames are decimated from X and X', respectively. In the training process of other deep learning models for image restoration or image reconstruction, a restored image which is completely consistent with the content in the original image to be reconstructed needs to be provided as a training set. In this method, the inventors found that: monitoring is carried out aiming at the drilling platform, and better constraint is carried out on a reconstructed image. Therefore, only the background of the shot contents of the two is needed, the foreground is similar, and the illumination conditions are not obviously different.
2) Preprocessing the acquired image:
in general, an image has foreground and background portions, where the foreground portion is often transformed and the background portion is less transformed. In the field of water mist removal and restoration of images, a deep learning model usually needs two nearly identical images without mist influence and affected by mist to perform pair training.
The inventor notices that: if the foreground and background parts are input to the network indiscriminately, the algorithm model fails to train because the input picture contains the irrespective part when the foreground part changes obviously. To solve this problem, see fig. 3, where the left ellipse in fig. 3 represents the foreground part that changes frequently, and the foreground part actually applied to the training picture by means of the mask will be masked as shown in the right diagram. Therefore, the foreground part is shielded in a mask mode, and the deep learning model is only focused on sensing the image part affected by the water mist. Therefore, for the above-mentioned video frame sequence containing fog and the video frame sequence without fog influence or with less fog influence, only the background of the contents shot by the two video frames is needed, the foreground is similar, and the illumination condition has no significant difference. During preprocessing, the foreground part 1 is shielded in a mask mode.
3) Taking 8 continuous images from the video sequence containing fog, preprocessing the images, and combining the images into a video frame sequence
Figure 76445DEST_PATH_IMAGE009
，
Figure 943513DEST_PATH_IMAGE010
4) Selecting a video frame x' similar to the image in the video sequence extracted in the previous step from the fog-free video sequence as a target picture to form a training set
Figure 805290DEST_PATH_IMAGE011
。
The model training method disclosed in the method is shown in fig. 1 and fig. 2, and the method needs to train two models so as to generate a model GmodelA restored picture y 'closer to the actual picture x' can be generated from the input fog picture (note: the degree of proximity is determined by the first threshold value hereinafter, i.e., the threshold value in fig. 1). Then inputting x 'and y' into discriminant model DmodelSo that the system can distinguish which is the actual picture and which is the real picture, and feed back the information to the generation model GmodelSo that it produces a more realistic reconstructed picture (note: the degree of realism is determined by a second threshold hereinafter, i.e. the threshold in fig. 2, the first and second thresholds being determinable from empirical values).
In another embodiment, the invention discloses a staged training method: the training is divided into two phases:
first, training G with a small portion of datamodelMake it reach a more stable state to avoid training G at the same timemodelAnd DmodelThe phenomenon of model collapse easily occurs;
then, most of the remaining data is reused to train G at the same timemodelAnd Dmodel。
The following are exemplary: the prepared training set is divided into two parts, A and B, wherein the part A accounts for 30% of the training set (namely the small part of data), and the rest part is B.
Stage 1 of training:
1) inputting part A of training set into part GmodelObtaining GmodelThe output restored picture y';
2) calculating the mean square error loss of y 'and x':
Figure 892064DEST_PATH_IMAGE001
wherein x isj’、yj' are pixel values of x ' and y ', respectively, n is the number of pixels, and j is a value from 0 to n;
3) if the loss is not less than the set first threshold, calculating the gradient by differentiating the loss, and reversely propagating the gradient to GmodelUpdating the parameters in the model until loss is less than the set first threshold;
if loss is less than the first threshold, entering the 2 nd stage of training;
stage 2 of training:
1) inputting part B of training set into part GmodelObtaining GmodelThe output restored picture y';
2) respectively inputting y 'and x' into a discriminant model DmodelObtaining two discrimination probabilities dfAnd drWhile the maximum value is found by:
max(log(df)+log(1-dr))，
and updating D with the maximized valuemodelWhen two discrimination probabilities df、drWhen the value of (A) approaches 0.5, the updating is stopped;
3) obtainGet DmodelIntermediate features in the calculation of the discrimination probability to calculate the perception error Ploss:
Figure 130278DEST_PATH_IMAGE002
wherein v isf、vrThe intermediate features of y 'and x' respectively, m is the number of the intermediate features, and k is from 0 to m;
if Ploss is not less than the second threshold, then the Ploss is propagated backwards to update GmodelUntil Ploss is less than said set second threshold; when Ploss is smaller than a set second threshold value, finishing training;
considering that today's neural networks involving images tend to feed the features of the last layer into the last classifier Softmax, which means that the features are filtered layer by layer in order to express the last "useful" feature as an image, the problem is that this results in the loss of some features and the omission of information, therefore, the present embodiment is to use the intermediate features of some or all intermediate feature layers as well to calculate the perception error Ploss.
Referring to FIGS. 1 and 2, it can be seen that, similar to loss, the perceived error Ploss also controls whether G is updated continuouslymodelThe parameter (c) of (c).
7) If Ploss is not less than the second threshold, then the Ploss is propagated backwards to update GmodelUntil Ploss is less than said set second threshold; as shown in fig. 2, when Ploss is smaller than the set second threshold, the training is ended.
Thus, through two training phases, the final goal is to achieve GmodelAnd (4) training. For the present invention, in the case of GmodelAfter the training is completed, G can be directly connectedmodelDeployed to an on-site server. The inputted picture passes through GmodelThe water mist is removed after the treatment is recovered, and the picture with the water mist removed can be sent to other applications for subsequent use.
It can be appreciated that the back propagation Ploss can be similar to loss, with the gradient calculated by taking the derivative of Ploss and back propagatingGradient to Gmodel。
In another embodiment of the present invention, the substrate is,
when G ismodelWhen the neural network is a deep neural network, the method also comprises the following steps:
s301: manually marking each image for training the deep neural network, and marking the positions of key points to obtain each marked image;
s302: and taking each marked image as the input of a corresponding stage of training to construct an auxiliary neural network, and assisting the deep neural network in learning and training the images.
With this embodiment, it is considered that the final result of the image recognition is to output the coordinates of the respective key points on the image. However, if done according to the prior art: the deep neural network learns the image and directly outputs two-dimensional coordinates to perform optimization learning, which is an extremely nonlinear process, and the constraint of a loss function for optimization on the weight in the neural network is weak during the optimization learning. Therefore, in this embodiment, we finally construct an intermediate state by constructing an auxiliary neural network according to the trained images and the positions of the manually labeled key points, so as to assist the training and learning of the deep neural network. It can be understood that when D ismodelWhen the image is also a deep neural network, the above-mentioned auxiliary neural network mode can be adopted to assist the corresponding neural network to learn and train the image.
In another embodiment, step S302 includes:
s3021: taking each marked image as input, selecting a proper middle layer from the deep neural network, and acquiring the output of the middle layer;
s3022: establishing an auxiliary neural network formed by convolution functions;
s3023: inputting the output of the middle layer and the corresponding attitude estimation matrix of each image before labeling into an auxiliary neural network;
s3024: and combining the outputs of the auxiliary neural network and the deep neural network, and jointly inputting the outputs into a loss function of the deep neural network to optimize the learning of the deep neural network.
In another embodiment of the present invention, the substrate is,
the attitude estimation matrix in step S3023 is obtained by the following steps:
s30231: calibrating the camera, and solving the intrinsic parameters of the camera, wherein the intrinsic parameters comprise: the image optical axis principal point, the focal lengths in the X direction and the Y direction, the tangential distortion coefficient and the radial distortion coefficient;
s30232: the attitude estimation matrix is further solved as follows:
solving an attitude estimation matrix [ R | t ] (X = M [ R | t ])/X),
wherein M is an internal parameter of the camera, X is a world coordinate system, and X is an image pixel coordinate of a known shot object; r, t are the rotation vector and the translation vector of the attitude estimation matrix, respectively.
In another embodiment of the present invention, the substrate is,
step S3021 includes:
the internal parameters of the camera are solved by a Zhang Zhengyou calibration method and a checkerboard with known size.
For example, the intrinsic parameters of the camera are solved by shooting the pixel coordinates of a checkerboard with known dimensions in different directions and different positions in an image coordinate system and by a Zhang friend calibration method.
In another embodiment of the present invention, the substrate is,
the grid is 10cm by 10 cm.
In another embodiment of the present invention, the substrate is,
the deep neural network selects ResNet 50. Typically, the process is carried out by Python.
In another embodiment of the present invention, the substrate is,
the auxiliary neural network selects ResNet 18.
In another embodiment of the present invention, the substrate is,
the convolution function is Conv (input, w), where input represents the input and w represents the weight.
In another embodiment, the loss function is selected as a mean square error function.
Referring to fig. 4, in another embodiment, the intermediate layer outputs described previously select the outputs of intermediate layer C5:7 × 2048.
For the related embodiments described above, the present disclosure effectively reduces the fitting difficulty during the training of the relevant model, and simultaneously improves the robustness of the model. In the same test set, after the images are used as a training set and the training and optimizing methods are adopted, the map @0.5 precision of the model is 2.76% higher than that of the model which is not adopted. It is further noted that, when the attitude estimation matrix is solved by the PNP based on the rannsac algorithm, even if the average error of the PNP solution is found to be about 5% in the actual scene verification, the difficulty of fitting during the training of the relevant model is effectively reduced and the robustness of the model is improved, so that the subsequent actual scene verification is not affected by the error of the attitude estimation matrix.
Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-described embodiments and application fields, and the above-described embodiments are illustrative, instructive, and not restrictive. Those skilled in the art, having the benefit of this disclosure, may effect numerous modifications thereto without departing from the scope of the invention as defined by the appended claims.

Claims (10)

1. The method for removing the water mist from the monitoring picture of the drilling platform by using machine learning is characterized by comprising the following steps of:
s100: acquiring a video sample of a drilling platform through a camera according to a preset frame rate, and performing frame extraction on the video sample, wherein the video sample comprises a fog-containing video and a fog-free video;
s200: taking continuous 8 frames of pictures from the fog-containing video, and combining the pictures into a video frame sequence X, wherein X = { X = { (X) }i,xi+1,…,xi+7Selecting a frame of actual picture X 'similar to any image in the video frame sequence X from the fog-free video as a target picture, and forming a training set X by the X' and the video frame sequence Xtrain={X,x’}；
S300: the prepared training set XtrainThe training model is divided into A, B parts, wherein, part A accounts for 30% of the training set, the rest part is part B, and the model is trained according to the part A = { G = {model,DmodelIn which G ismodelTo generate a model, DmodelIs a discriminant model; wherein, the training is divided into 2 stages, and A, B parts of the training set are respectively used as G in the 2 stagesmodelThe input of (1);
s400: after the model is trained, a generative model G in the model is utilizedmodelDefogging a video-based continuous frame image of a drilling rig.
2. The method of claim 1, wherein the predetermined frame is a frame rate of 25 frames.
3. The method of claim 1, wherein step S300 comprises:
stage 1 of training:
1) inputting part A of training set into part GmodelObtaining GmodelThe output restored picture y';
2) calculating the mean square error loss of y 'and x':
Figure 231960DEST_PATH_IMAGE001
wherein x isj’、yj' are pixel values of x ' and y ', respectively, n is the number of pixels, and j is a value from 0 to n;
3) if the loss is not less than the set first threshold, calculating the gradient by differentiating the loss, and reversely propagating the gradient to GmodelUpdating the parameters in the model until loss is less than the set first threshold;
if loss is less than the first threshold, entering the 2 nd stage of training;
stage 2 of training:
1) inputting part B of training set into part GmodelObtaining GmodelOutput restoration imageA sheet y';
2) respectively inputting y 'and x' into a discriminant model DmodelObtaining two discrimination probabilities dfAnd drWhile the maximum value is found by:
max(log(df)+log(1-dr))，
and updating D with the maximized valuemodelWhen two discrimination probabilities df、drWhen the value of (A) approaches 0.5, the updating is stopped;
3) obtaining DmodelIntermediate features in the calculation of the discrimination probability to calculate the perception error Ploss:
Figure 290046DEST_PATH_IMAGE002
wherein v isf、vrThe intermediate features of y 'and x' respectively, m is the number of the intermediate features, and k is from 0 to m;
if Ploss is not less than the second threshold, then the Ploss is propagated backwards to update GmodelUntil Ploss is less than said set second threshold;
and when Ploss is smaller than a set second threshold value, finishing training.
4. The method of claim 1, wherein when G is presentmodelWhen the neural network is a deep neural network, the method also comprises the following steps:
s301: manually marking each image for training the deep neural network, and marking the positions of key points to obtain each marked image;
s302: and taking each marked image as the input of a corresponding stage of training to construct an auxiliary neural network, and assisting the deep neural network in learning and training the images.
5. The method according to claim 4, wherein step S302 comprises the steps of:
s3021: taking each marked image as input, selecting a proper middle layer from the deep neural network, and acquiring the output of the middle layer;
s3022: establishing an auxiliary neural network formed by convolution functions;
s3023: inputting the output of the middle layer and the corresponding attitude estimation matrix of each image before labeling into an auxiliary neural network;
s3024: and combining the outputs of the auxiliary neural network and the deep neural network, and jointly inputting the outputs into a loss function of the deep neural network to optimize the learning of the deep neural network.
6. The method according to claim 5, wherein the attitude estimation matrix in step S3023 is obtained by:
s30231: calibrating a camera of the camera, and solving internal parameters of the camera, wherein the internal parameters comprise: the image optical axis principal point, the focal lengths in the X direction and the Y direction, the tangential distortion coefficient and the radial distortion coefficient;
s30232: the attitude estimation matrix is further solved as follows:
solving an attitude estimation matrix [ R | t ] (X = M [ R | t ])/X),
wherein M is an internal parameter of the camera, X is a world coordinate system, and X is an image pixel coordinate of a known shot object; r, t are the rotation vector and the translation vector of the attitude estimation matrix, respectively.
7. The method of claim 5, wherein step S3021 comprises:
the internal parameters of the camera are solved by a Zhang Zhengyou calibration method and a checkerboard with known size.
8. The method of claim 7, wherein the checkerboard is 10cm by 10 cm.
9. The method of claim 5, wherein the deep neural network selects ResNet 50.
10. The method of claim 5, wherein the accessory neural network selects ResNet 18.
CN202110803908.6A 2021-07-16 2021-07-16 Method for removing water mist from drilling platform monitoring picture by machine learning Active CN113256541B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110803908.6A CN113256541B (en) 2021-07-16 2021-07-16 Method for removing water mist from drilling platform monitoring picture by machine learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110803908.6A CN113256541B (en) 2021-07-16 2021-07-16 Method for removing water mist from drilling platform monitoring picture by machine learning

Publications (2)

Publication Number Publication Date
CN113256541A true CN113256541A (en) 2021-08-13
CN113256541B CN113256541B (en) 2021-09-17

Family

ID=77180506

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110803908.6A Active CN113256541B (en) 2021-07-16 2021-07-16 Method for removing water mist from drilling platform monitoring picture by machine learning

Country Status (1)

Country Link
CN (1) CN113256541B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117218329A (en) * 2023-11-09 2023-12-12 四川泓宝润业工程技术有限公司 Wellhead valve detection method and device, storage medium and electronic equipment

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106127702A (en) * 2016-06-17 2016-11-16 兰州理工大学 A kind of image mist elimination algorithm based on degree of depth study
CN107798669A (en) * 2017-12-08 2018-03-13 北京小米移动软件有限公司 Image defogging method, device and computer-readable recording medium
CN108230264A (en) * 2017-12-11 2018-06-29 华南农业大学 A kind of single image to the fog method based on ResNet neural networks
CN109360155A (en) * 2018-08-17 2019-02-19 上海交通大学 Single-frame images rain removing method based on multi-scale feature fusion
CN109447918A (en) * 2018-11-02 2019-03-08 北京交通大学 Removing rain based on single image method based on attention mechanism
CN109493303A (en) * 2018-05-30 2019-03-19 湘潭大学 A kind of image defogging method based on generation confrontation network
CN110136175A (en) * 2019-05-21 2019-08-16 杭州电子科技大学 A kind of indoor typical scene matching locating method neural network based
US20200193927A1 (en) * 2017-01-26 2020-06-18 Semiconductor Energy Laboratory Co., Ltd. Semiconductor device and electronic device including the semiconductor device
WO2020190561A1 (en) * 2019-03-15 2020-09-24 Nvidia Corporation Techniques to train a neural network using transformations
CN111738940A (en) * 2020-06-02 2020-10-02 大连理工大学 Human face image eye completing method for generating confrontation network based on self-attention mechanism model
CN111738942A (en) * 2020-06-10 2020-10-02 南京邮电大学 Generation countermeasure network image defogging method fusing feature pyramid
CN111815523A (en) * 2020-06-08 2020-10-23 天津中科智能识别产业技术研究院有限公司 Image restoration method based on generation countermeasure network
US20200387750A1 (en) * 2019-06-06 2020-12-10 Samsung Electronics Co., Ltd. Method and apparatus for training neural network model for enhancing image detail
CN112541864A (en) * 2020-09-25 2021-03-23 中国石油大学(华东) Image restoration method based on multi-scale generation type confrontation network model
CN112614070A (en) * 2020-12-28 2021-04-06 南京信息工程大学 DefogNet-based single image defogging method
CN112669242A (en) * 2021-03-16 2021-04-16 四川大学 Night scene restoration method based on improved image enhancement algorithm and generation countermeasure network

Patent Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106127702A (en) * 2016-06-17 2016-11-16 兰州理工大学 A kind of image mist elimination algorithm based on degree of depth study
US20200193927A1 (en) * 2017-01-26 2020-06-18 Semiconductor Energy Laboratory Co., Ltd. Semiconductor device and electronic device including the semiconductor device
CN107798669A (en) * 2017-12-08 2018-03-13 北京小米移动软件有限公司 Image defogging method, device and computer-readable recording medium
CN108230264A (en) * 2017-12-11 2018-06-29 华南农业大学 A kind of single image to the fog method based on ResNet neural networks
CN109493303A (en) * 2018-05-30 2019-03-19 湘潭大学 A kind of image defogging method based on generation confrontation network
CN109360155A (en) * 2018-08-17 2019-02-19 上海交通大学 Single-frame images rain removing method based on multi-scale feature fusion
CN109447918A (en) * 2018-11-02 2019-03-08 北京交通大学 Removing rain based on single image method based on attention mechanism
WO2020190561A1 (en) * 2019-03-15 2020-09-24 Nvidia Corporation Techniques to train a neural network using transformations
CN110136175A (en) * 2019-05-21 2019-08-16 杭州电子科技大学 A kind of indoor typical scene matching locating method neural network based
US20200387750A1 (en) * 2019-06-06 2020-12-10 Samsung Electronics Co., Ltd. Method and apparatus for training neural network model for enhancing image detail
CN111738940A (en) * 2020-06-02 2020-10-02 大连理工大学 Human face image eye completing method for generating confrontation network based on self-attention mechanism model
CN111815523A (en) * 2020-06-08 2020-10-23 天津中科智能识别产业技术研究院有限公司 Image restoration method based on generation countermeasure network
CN111738942A (en) * 2020-06-10 2020-10-02 南京邮电大学 Generation countermeasure network image defogging method fusing feature pyramid
CN112541864A (en) * 2020-09-25 2021-03-23 中国石油大学(华东) Image restoration method based on multi-scale generation type confrontation network model
CN112614070A (en) * 2020-12-28 2021-04-06 南京信息工程大学 DefogNet-based single image defogging method
CN112669242A (en) * 2021-03-16 2021-04-16 四川大学 Night scene restoration method based on improved image enhancement algorithm and generation countermeasure network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
REN W Q等: "Single image dehazing via multi-scale convolutional neural networks", 《PROCEEDINGS OF THE 2016 EUROPEAN CONFERENCE ON COMPUTER VISION》 *
孙潇等: "基于级联生成对抗网络的遥感图像去雾方法", 《计算机应用》 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117218329A (en) * 2023-11-09 2023-12-12 四川泓宝润业工程技术有限公司 Wellhead valve detection method and device, storage medium and electronic equipment
CN117218329B (en) * 2023-11-09 2024-01-26 四川泓宝润业工程技术有限公司 Wellhead valve detection method and device, storage medium and electronic equipment

Also Published As

Publication number Publication date
CN113256541B (en) 2021-09-17

Similar Documents

Publication Publication Date Title
CN110008817B (en) Model training method, image processing method, device, electronic equipment and computer readable storage medium
CN108520503B (en) Face defect image restoration method based on self-encoder and generation countermeasure network
CN111047522B (en) Image restoration method based on edge generation
CN111696196B (en) Three-dimensional face model reconstruction method and device
CN110009573B (en) Model training method, image processing method, device, electronic equipment and storage medium
CN110705353A (en) Method and device for identifying face to be shielded based on attention mechanism
CN112528782B (en) Underwater fish target detection method and device
WO2023066173A1 (en) Image processing method and apparatus, and storage medium and electronic device
CN112991193A (en) Depth image restoration method, device and computer-readable storage medium
CN113284061B (en) Underwater image enhancement method based on gradient network
CN113256541B (en) Method for removing water mist from drilling platform monitoring picture by machine learning
US11783454B2 (en) Saliency map generation method and image processing system using the same
CN113516604B (en) Image restoration method
CN112508814B (en) Image tone restoration type defogging enhancement method based on unmanned aerial vehicle at low altitude visual angle
CN117036189A (en) Underwater image restoration model based on lumen consistency and complementary color correction joint learning
Wei et al. Facial image inpainting with deep generative model and patch search using region weight
CN116342649A (en) Method, device and equipment for tracking specific target object in low-resolution image in complex scene
CN112232221A (en) Method, system and program carrier for processing human image
WO2020044630A1 (en) Detector generation device, monitoring device, detector generation method, and detector generation program
CN116051407A (en) Image restoration method
CN113378980B (en) Mask face shielding recovery method based on self-adaptive context attention mechanism
CN112215766B (en) Image defogging method combining image restoration and image enhancement and convolution network thereof
CN114821580A (en) Noise-containing image segmentation method by stage-by-stage merging with denoising module
CN111461959B (en) Face emotion synthesis method and device
CN113792629A (en) Helmet wearing detection method and system based on deep neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder

Address after: 610056 No.05, 11th floor, building 1, 99 mengzhuiwan Hengjie, Chenghua District, Chengdu City, Sichuan Province

Patentee after: Sichuan hongbaorunye Engineering Technology Co.,Ltd.

Patentee after: Chongqing Hongbao Technology Co.,Ltd.

Patentee after: Beijing Baolong Hongrui Technology Co.,Ltd.

Address before: 610056 No.05, 11th floor, building 1, 99 mengzhuiwan Hengjie, Chenghua District, Chengdu City, Sichuan Province

Patentee before: Sichuan hongbaorunye Engineering Technology Co.,Ltd.

Patentee before: Chongqing Qingyun Petroleum Engineering Technology Co.,Ltd.

Patentee before: Beijing Baolong Hongrui Technology Co.,Ltd.

CP01 Change in the name or title of a patent holder