CN115019107A - Sonar simulation image generation method, system and medium based on style migration - Google Patents

Sonar simulation image generation method, system and medium based on style migration Download PDF

Info

Publication number
CN115019107A
CN115019107A CN202210752897.8A CN202210752897A CN115019107A CN 115019107 A CN115019107 A CN 115019107A CN 202210752897 A CN202210752897 A CN 202210752897A CN 115019107 A CN115019107 A CN 115019107A
Authority
CN
China
Prior art keywords
image
sonar
style
satellite
style migration
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210752897.8A
Other languages
Chinese (zh)
Other versions
CN115019107B (en
Inventor
陈德山
李卓翼
王之森
吴兵
汪洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University of Technology WUT
Original Assignee
Wuhan University of Technology WUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University of Technology WUT filed Critical Wuhan University of Technology WUT
Priority to CN202210752897.8A priority Critical patent/CN115019107B/en
Publication of CN115019107A publication Critical patent/CN115019107A/en
Application granted granted Critical
Publication of CN115019107B publication Critical patent/CN115019107B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Remote Sensing (AREA)
  • Astronomy & Astrophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)

Abstract

The invention discloses a sonar simulation image generation method, a system and a medium based on style migration, wherein the method comprises the following steps: according to the type of a target object contained in a real sonar image, a satellite remote sensing image is taken as input, the satellite remote sensing image is identified, an identification object with the same type as the target object is identified, and the range where the identification object is located is marked; carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite sub-image data set by taking the category of the target object as classification; and constructing a style migration network, taking the real sonar image as a style image, carrying out style migration on the segmented satellite sub-image, and generating a sonar simulation image to be used as a training sample image. The invention obtains the simulated sonar image by using the style migration network based on the remote sensing satellite image, can effectively expand the number of sonar data sets, and provides a basis for subsequent related work based on the sonar image. The invention can be widely applied to the technical field of image processing.

Description

Sonar simulation image generation method, system and medium based on style migration
Technical Field
The invention relates to the technical field of image processing, in particular to a sonar simulation image generation method, system and medium based on style migration.
Background
Sonar is used as underwater imaging equipment with high resolution, multiple purposes and low cost, and is widely applied to oceans, rivers, lakes and other water areas. The method can obtain high-resolution continuous submarine images, and is widely applied to the fields of marine surveying and mapping, offshore exploration, underwater search and rescue and the like. During underwater searching and rescuing activities, the sonar can effectively detect underwater targets such as airplane wreckages, sunken ships and the like. In a long-time underwater searching and rescuing task, rescuers need to continuously and carefully check sonar images to determine whether target objects exist. After working for a period of time, the staff can generate visual fatigue and easily miss the rescue target. In order to effectively reduce staff's work load, reduce the wrong judgement that visual fatigue arouses, improve work efficiency, carry out automatic classification to sonar image and have realistic meaning. However, the prior art has the following disadvantages:
in recent years, the underwater image automatic classification algorithm with higher accuracy uses a deep neural network algorithm, and the algorithm needs a large amount of sonar image data for training. However, because sonar images are often related to emergency rescue or national defense, the difficulty in acquiring data sets is high, and no unified large-batch sonar images are disclosed yet. This makes convolutional neural networks always unavailable for a wide range of applications in the sonar area.
Disclosure of Invention
In order to solve at least one of the technical problems in the prior art to a certain extent, the invention aims to provide a sonar simulation image generation method, system and medium based on style migration.
The technical scheme adopted by the invention is as follows:
a sonar simulation image generation method based on style migration comprises the following steps:
according to the type of a target object contained in a real sonar image, a satellite remote sensing image is taken as input, the satellite remote sensing image is identified, an identified object with the same type as the target object is identified, and the range where the identified object is located is marked;
carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite sub-image data set by taking the category of the target object as classification;
and constructing a style migration network, taking the real sonar image as a style image, carrying out style migration on the segmented satellite sub-image, and generating a sonar simulation image to be used as a training sample image.
Further, the identifying the satellite remote sensing image, identifying an identification object with the same type as the target object, and marking the range of the identification object includes:
adopting a target recognition network trained by a preset data set to perform target detection on the satellite remote sensing image, and dividing a detection result into n +1 types, wherein n is the number of target object types;
and marking the area where the target object is located, wherein the resolution of the marked area is adjusted according to different input images.
Further, the image segmentation of the identified and marked satellite remote sensing image and the construction of a satellite sub-image data set by taking the target object class as a classification comprise:
after the identification object is marked, obtaining a target area, and reading parameters x, y, w and h of the target area, wherein x and y are horizontal and vertical coordinates of the upper left corner of the target area, and w and h are horizontal and vertical dimensions of the target area;
according to the read parameters, carrying out self-adaptive resolution segmentation on the target area, wherein the segmented areas of the satellite subimages are as follows:
Figure BDA0003721594240000021
wherein,
Figure BDA0003721594240000022
the adaptive resolution scaling factor is determined by the horizontal and vertical dimensions of the original picture;
Figure BDA0003721594240000023
for the range of segmentation of the original mark,
Figure BDA0003721594240000024
the subsequent segmentation range is scaled for adaptive resolution.
Further, the adaptive resolution scaling factor λ is obtained by:
Figure BDA0003721594240000025
wherein f (w, h) is w, and the smaller side of h.
Further, the style migration network extracts style feature maps by using a void space pyramid structure.
Further, the constructing a style migration network, using a real sonar image as a style image, performing style migration on the segmented satellite sub-image, generating a sonar simulation image as a training sample image, and including:
and constructing a convolutional neural network, extracting the characteristics of a real sonar image and a satellite sub-image by using the convolutional neural network, and optimizing by taking the Euclidean distance between the characteristics of the two images and the Euclidean distance of a gram matrix as a loss function to obtain a sonar simulation image.
Further, the constructing a convolutional neural network, extracting the features of a real sonar image and a satellite sub-image by using the convolutional neural network, and optimizing by using the euclidean distance between two image features and the euclidean distance of a gram matrix as a loss function to obtain a sonar simulation image includes:
based on the front 17 layer of the VGG16 network, short connection is added in the front 17 layer of the VGG16 to construct a convolutional neural network;
the convolutional neural network calculation after the shortcut is added is represented as:
x l+1 ＝x l +F(x l )
in the formula, x l And x l+1 Respectively representing the characteristics of the ith layer and the (l + 1) th layer of the network; f (x) l ) Represents the convolution and activation operations of the network; the network back propagation process after the shortcut is added is as follows:
Figure BDA0003721594240000031
using the front 17 layers of the trained VGG16 network as the backbone of the convolutional neural network, using a real sonar image as a style image, using a satellite sub-image as a content image, inputting the content image into a VGG16 network with short, and respectively extracting a style feature S and a content feature C;
and taking the Euclidean distance between the style characteristic S and the content characteristic C as content loss, wherein the content loss is expressed as:
Figure BDA0003721594240000032
wherein i represents the ith element in the feature map;
the style loss function is:
Figure BDA0003721594240000033
Figure BDA0003721594240000034
Figure BDA0003721594240000035
in the formula
Figure BDA0003721594240000036
The method is characterized in that style characteristics S (also called style picture S) are output at the (i, j, k) th position of the first layer of the CNN, wherein the (i, j, k) corresponds to three dimensions of height, width and channel;
Figure BDA0003721594240000037
the method is characterized in that the style picture is output at the (i, j, k ') th position of the first layer of the CNN, wherein the (i, j, k') corresponds to three dimensions of height, width and channel;
Figure BDA0003721594240000038
outputting the network generated image G at the (i, j, k) th position of the ith layer of the CNN, wherein the (i, j, k) corresponds to three dimensions of height, width and channel;
Figure BDA0003721594240000039
is the total number in the height dimension of the l layer of CNN;
Figure BDA00037215942400000310
is the total number in the width dimension of the l-th layer of the CNN;
Figure BDA00037215942400000311
the total number of the CNN layer I channel dimensions;
Figure BDA0003721594240000041
for the style feature S of the first layer of CNN in channel k ′ And a gram matrix for channel k;
Figure BDA0003721594240000042
for the content characteristic C of the I th layer of CNN in channel k ′ And a gram matrix for channel k;
the loss function for style migration is:
L＝αL c +βL s
wherein alpha and beta are weight coefficients.
The invention adopts another technical scheme that:
a sonar simulation image generation system based on style migration comprises:
the satellite image processing module is used for identifying the satellite remote sensing image by taking the satellite remote sensing image as input according to the target object category contained in the real sonar image, identifying an identification object with the same category as the target object, and marking the range of the identification object;
the satellite image segmentation module is used for carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite subimage data set by taking the category of the target object as classification;
and the image style migration module is used for constructing a style migration network, taking the real sonar image as a style image, performing style migration on the segmented satellite sub-image, and generating a sonar simulation image to be used as a training sample image.
The other technical scheme adopted by the invention is as follows:
a sonar simulation image generation system based on style migration comprises:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method described above.
The other technical scheme adopted by the invention is as follows:
a computer readable storage medium in which a processor executable program is stored, which when executed by a processor is for performing the method as described above.
The invention has the beneficial effects that: the invention obtains the simulated sonar image by taking the remote sensing satellite image as input, taking the real sonar image as a target and using the style migration network, can effectively expand the number of sonar data sets, and provides a basis for subsequent related work based on the sonar image.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description is made on the drawings of the embodiments of the present invention or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flowchart of the steps of a sonar simulation image generation method based on style migration in the embodiment of the present invention;
FIG. 2 is a schematic flow chart of a sonar simulation image generation method based on style migration in an embodiment of the present invention;
FIG. 3 is a block diagram of a convolutional neural network in an embodiment of the present invention;
FIG. 4 is a flowchart illustrating operation of a VGG16 network-based style migration network in an embodiment of the present invention;
FIG. 5 is a schematic illustration of a satellite remote sensing image used in an embodiment of the present invention;
FIG. 6 is a schematic diagram of a satellite remote sensing image after object recognition according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a satellite remote sensing subimage after image cropping according to an embodiment of the invention;
fig. 8 is a schematic diagram of a simulated sonar image obtained through style migration in the embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
In the description of the present invention, it should be understood that the orientation or positional relationship referred to in the description of the orientation, such as the upper, lower, front, rear, left, right, etc., is based on the orientation or positional relationship shown in the drawings, and is only for convenience of description and simplification of description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present invention.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number. If the first and second are described for the purpose of distinguishing technical features, they are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the description of the present invention, unless otherwise specifically limited, terms such as set, installation, connection and the like should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above terms in the present invention by combining the specific contents of the technical solutions.
As shown in fig. 1, the present embodiment provides a sonar simulation image generation method based on style migration, including the following steps:
and S1, according to the type of the target object contained in the real sonar image, using the satellite remote sensing image as input, identifying the satellite remote sensing image, identifying the identified object with the same type as the target object, and marking the range where the identified object is located.
According to the existing target object type and the target object type needing to be added contained in the real sonar image, the satellite remote sensing image is used as input, and the same satellite remote sensing image area as the target object type is identified.
(1) There are two types in the real sonar image dataset: the types of the ship remains and the airplane remains are few, and the number distribution is uneven, so that subsequent work based on the real sonar image data set cannot be expanded. In order to make the sonar image data sets diversified in type and sufficient in quantity, the invention takes ship debris, airplane debris, personnel falling into water and stones as target object types.
(2) The method comprises the steps of taking a satellite remote sensing image as an original image as an input, identifying the target object in the image to obtain an identification object, marking an image area where the identification object is located, and marking data as follows: the category of the identification object, the coordinates x and y of the upper left corner of the area of the identification object, and the dimensions h and w of the area of the identification object in the height direction and the width direction of the image.
And S2, carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite sub-image data set by taking the target object class as a classification.
And according to the identification object marking data in the step, carrying out image cutting on the marking area, judging the length, the width and the proportion of the cut sub-image, and carrying out telescopic transformation on the image according to different length, width and length-width ratios.
The method comprises the following specific steps:
(1) and according to the x, y, h and w obtained in the step, cutting the original image, wherein the cutting range is as follows:
{(x，y)，(x+w，y)，(x，y+h)，(x+w，y+h)}
the four coordinates in the above formula are the four-corner coordinates of the cut image, and the image is cut by taking the four-corner coordinates as a cutting range to obtain a sub-image.
(2) And performing image scaling transformation by using 416 × 416 pixels as a standard according to the sub-images obtained in the above steps. The image scaling transformation processing formula is as follows:
Figure BDA0003721594240000061
and S3, constructing a style migration network, taking the real sonar image as a style image, performing style migration on the segmented satellite subimage, and generating a sonar simulation image serving as a training sample image.
And constructing a convolutional neural network, extracting the characteristics of the real sonar image and the subimage by using the network, and optimizing by taking the Euclidean distance between the characteristics of the two images and the Euclidean distance of a gram matrix as a loss function to obtain a simulated sonar image.
The method comprises the following specific steps:
(1) based on the front 17 layer of the VGG16 network, short connections are added to the front 17 layer of the VGG16 to construct a convolutional neural network. As shown in fig. 3.
The Shortcut connection is the addition of one input to the output of the function. The output after the short is added can be explicitly represented as a linear superposition of F (x) and x. The output is expressed as a linear superposition of the input and a non-linear transformation of the input. And the training effect of the network is ensured under the condition that the network structure is deepened. The network computation after adding the shortcut can be expressed as:
x l+1 ＝x l +F(x l )
in the above formula x l And x l+1 Respectively representing the characteristics of the l < th > and l +1 < th > layers of the network. F (x) l ) Representing the convolution and activation operations of the network. The network back propagation process after the shortcut is added is as follows:
Figure BDA0003721594240000071
(2) and loading the parameters of the convolutional neural network, and inputting the sub-image obtained in the step S2 and the real sonar image into the network for style migration. As shown in fig. 4.
Using the front 17 layers of the trained VGG16 network as the backbone of the convolutional neural network, using the real sonar image as the style image, and inputting the sub-image obtained in step S2 as the content image into the VGG16 network with short to extract the style feature S and the content feature C respectively.
Taking the euclidean distance between the style feature S and the content feature C as the content loss, the content loss can be expressed as:
Figure BDA0003721594240000072
where i represents the ith element in the feature map, each element in the feature map being a numerical value between [0, 1).
The style loss function is:
Figure BDA0003721594240000073
Figure BDA0003721594240000074
Figure BDA0003721594240000075
in the formula
Figure BDA0003721594240000076
Is the output of the style picture at the (i, j, k) th position of the I th layer of the CNN, wherein (i, j, k) corresponds to the height, width and channel
The loss function for style migration is:
L＝αL c +βL s
wherein alpha and beta are weight coefficients
(3) And setting a contrast experiment to verify the similarity between the simulated sonar image and the real sonar image.
Experimental group data set up: combining the real sonar image and the simulated sonar image into a mixed data set, and dividing the data set into a training set and a verification set according to the proportion of 7: 3. Setting comparison group data: only the real sonar image is set as a data set, and the training set and the validation set are divided according to a 7:3 ratio. After the image classification network inspection, the accuracy rates of the two groups of experimental data sets are similar.
The above method is explained in detail with reference to the figures and the specific embodiments.
Referring to fig. 2, fig. 2 shows a flowchart of a sonar simulation image generation method based on style migration according to an embodiment of the present invention, where the method includes the following steps:
step S101, according to the existing target object type and the target object type needing to be added contained in the real sonar image, the satellite remote sensing image is taken as input, and the same satellite remote sensing image area as the target object type is identified.
(1) There are two types of real sonar image data sets involved in this embodiment: the method comprises the steps of collecting ship debris, airplane debris, simulation sonar image target object types, rescue salvage and underwater detection, and adding falling personnel and stones as the target object types.
(2) Fig. 5 is one of the satellite remote sensing images used in this example, and the satellite remote sensing image is used as an input, an object in the remote sensing image is identified according to a classification standard, and the identified region is labeled. Fig. 5 includes three airplanes, so after identification, three airplanes and corresponding areas should be marked. The identified label graph is shown in fig. 6. The labeled data are respectively:
{airplane,(642,187),120,120}
{airplane,(151,360),210,180}
{airplane,(329,552),174,147}
and S102, cutting the image of the marked area according to the marked data of the identification object in the step, judging the length, the width and the proportion of the cut sub-image, and performing telescopic transformation on the image according to different length, width and length-width ratios.
(1) According to { classes, (x, y), w, h } obtained in the above steps, the range of clipping of the three images is:
{(642,187),(762,187),(642,370),(762,370)}
{(151,360),(361,360),(151,542),(361,542)}
{(329,552),(503,552),(329,699),(503,699)}
image clipping is performed by using the four-corner coordinates as a clipping range, and a sub-image is obtained as shown in fig. 7.
(2) And performing image scaling transformation by using 416 × 416 pixels as a standard according to the sub-images obtained in the above steps. The image scaling transformation processing formula is as follows:
Figure BDA0003721594240000081
the sizes of the three scaled sub-images are respectively as follows: 416 × 416, 416 × 357, 416 × 351.
S103: and constructing a convolutional neural network, extracting the characteristics of the real sonar image and the subimage by using the network, and optimizing by taking the Euclidean distance between the characteristics of the two images and the Euclidean distance of a gram matrix as a loss function to obtain a simulated sonar image.
(1) Based on the front 17 layer of the VGG16 network, short connections are added to the front 17 layer of the VGG16 to construct a convolutional neural network.
Short structure: except for necessary down-sampling operation, the data input into the network is not processed, and is directly fused with the feature map after convolution, so that the capability of extracting features by the neural network is improved.
(2) The parameters of the convolutional neural network described above were loaded, which were from the ImageNet dataset training. And inputting the sub-image obtained in the step S102 and the real sonar image into a network for style migration.
The simulated sonar image after the style migration is shown in fig. 8.
(3) And setting a contrast experiment to verify the similarity between the simulated sonar image and the real sonar image.
Experimental group data set up: combining the real sonar image and the simulated sonar image into a mixed data set, and dividing the data set into a training set and a verification set according to the proportion of 7: 3. The number of training sets and validation sets was 84, 36, respectively. Setting comparison group data: only the real sonar image is set as a data set, and the training set and the validation set are divided according to a 7:3 ratio. The number of training and validation sets was 44, 18 respectively. After the image classification network reasoning, the accuracy rates of the two groups of data are respectively as follows: 86.11% and 83.33%. The difference between the two groups of data is stronger, which shows that for the same image classification network, the real data set and the mixed data set have stronger similarity.
In the embodiment, a remote sensing satellite aircraft image is used as input, a real aircraft debris sonar image is used as a target, a style migration network is used, a simulated sonar image is obtained, and the number of aircraft sonar image data sets is effectively expanded.
In summary, compared with the prior art, the method of the embodiment has the following advantages and beneficial effects:
the method takes a remote sensing satellite image as input, takes a real sonar image as a target, and uses a style migration network to obtain a simulated sonar image. In addition, in the process from the input of the image to the final acquisition of the simulated sonar image, the method does not need to manually add image features, can quickly acquire the simulated sonar image according to the remote sensing images with different resolutions, and the acquired simulated sonar image has the style of a real sonar image, can effectively expand the number of sonar data sets, and provides a basis for subsequent related work based on the sonar image.
The embodiment also provides a sonar simulation image generation system based on style migration, which comprises:
the satellite image processing module is used for identifying the satellite remote sensing image by taking the satellite remote sensing image as input according to the target object type contained in the real sonar image, identifying an identification object with the same type as the target object, and marking the range where the identification object is located;
the satellite image segmentation module is used for carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite subimage data set by taking the category of the target object as classification;
and the image style migration module is used for constructing a style migration network, taking the real sonar image as a style image, performing style migration on the segmented satellite sub-image, and generating a sonar simulation image to be used as a training sample image.
The sonar simulation image generation system based on style migration can execute the sonar simulation image generation method based on style migration provided by the method embodiment of the invention, can execute any combination implementation steps of the method embodiment, and has corresponding functions and beneficial effects of the method.
The embodiment also provides a sonar simulation image generation system based on style migration, which comprises:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method of fig. 1.
According to the sonar simulation image generation system based on style migration, the sonar simulation image generation method based on style migration provided by the method embodiment of the invention can be executed, the implementation steps of any combination of the method embodiments can be executed, and the sonar simulation image generation system based on style migration has corresponding functions and beneficial effects of the method.
The embodiment of the application also discloses a computer program product or a computer program, which comprises computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and executed by the processor to cause the computer device to perform the method illustrated in fig. 1.
The embodiment also provides a storage medium, which stores instructions or programs capable of executing the sonar simulation image generation method based on style migration provided by the method embodiment of the invention, and when the instructions or the programs are run, the method embodiment can be executed in any combination of implementation steps, and the method has corresponding functions and beneficial effects.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those of ordinary skill in the art will be able to practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the foregoing description of the specification, reference to the description of "one embodiment/example," "another embodiment/example," or "certain embodiments/examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (10)

1. A sonar simulation image generation method based on style migration is characterized by comprising the following steps:
according to the type of a target object contained in a real sonar image, a satellite remote sensing image is taken as input, the satellite remote sensing image is identified, an identified object with the same type as the target object is identified, and the range where the identified object is located is marked;
carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite sub-image data set by taking the category of the target object as classification;
and constructing a style migration network, taking the real sonar image as a style image, carrying out style migration on the segmented satellite sub-image, and generating a sonar simulation image to be used as a training sample image.
2. The method for generating the sonar simulation image based on the style migration according to claim 1, wherein the identifying the satellite remote sensing image, identifying the recognized object with the same type as the target object, and marking the range where the recognized object is located comprises:
adopting a target recognition network trained by a preset data set to perform target detection on the satellite remote sensing image, and dividing a detection result into n +1 types, wherein n is the number of target object types;
and marking the area where the target object is located.
3. The sonar simulation image generation method based on style migration according to claim 1, wherein the image segmentation is performed on the identified and marked satellite remote sensing image, and a satellite sub-image data set is constructed by taking the category of a target object as a classification, and the method comprises the following steps:
after the identification object is marked, obtaining a target area, and reading parameters x, y, w and h of the target area, wherein x and y are horizontal and vertical coordinates of the upper left corner of the target area, and w and h are horizontal and vertical dimensions of the target area;
according to the read parameters, carrying out self-adaptive resolution segmentation on the target area, wherein the segmented areas of the satellite subimages are as follows:
Figure FDA0003721594230000011
wherein,
Figure FDA0003721594230000012
the adaptive resolution scaling factor is determined by the horizontal and vertical dimensions of the original picture;
Figure FDA0003721594230000013
for the range of segmentation of the original mark,
Figure FDA0003721594230000014
the subsequent segmentation range is scaled for adaptive resolution.
4. The sonar simulation image generation method based on style migration according to claim 3, wherein the adaptive resolution scaling factor λ is obtained by:
Figure FDA0003721594230000015
wherein f (w, h) is w, and the smaller side of h.
5. The sonar simulation image generation method based on style migration according to claim 1, wherein the style migration network extracts style feature maps by using a hollow space pyramid structure.
6. The method for generating sonar simulation images based on style migration according to claim 1, wherein the constructing a style migration network, using a real sonar image as a style image, performing style migration on segmented satellite sub-images to generate sonar simulation images as training sample images, comprises:
and constructing a convolutional neural network, extracting the characteristics of a real sonar image and a satellite sub-image by using the convolutional neural network, and optimizing by taking the Euclidean distance between the characteristics of the two images and the Euclidean distance of a gram matrix as a loss function to obtain a sonar simulation image.
7. The method for generating a sonar simulation image based on style migration according to claim 6, wherein the constructing a convolutional neural network, extracting features of a real sonar image and a satellite sub-image by using the convolutional neural network, and optimizing by using a Euclidean distance between two image features and a Euclidean distance of a Graham matrix as a loss function to obtain the sonar simulation image comprises:
based on the front 17 layer of the VGG16 network, short connections are added to the front 17 layer of the VGG16 to construct a convolutional neural network;
the convolutional neural network calculation after the shortcut is added is represented as:
x l+1 ＝x l +F(x l )
in the formula, x l And x l+1 Respectively representing the characteristics of the ith layer and the (l + 1) th layer of the network; f (x) l ) Represents the convolution and activation operations of the network; the network back propagation process after the shortcut is added is as follows:
Figure FDA0003721594230000021
using the front 17 layers of the trained VGG16 network as the backbone of the convolutional neural network, using a real sonar image as a style image, using a satellite sub-image as a content image, inputting the content image into a VGG16 network with short, and respectively extracting a style feature S and a content feature C;
and taking the Euclidean distance between the style characteristic S and the content characteristic C as content loss, wherein the content loss is expressed as:
Figure FDA0003721594230000022
wherein i represents the ith element in the feature map;
the style loss function is:
Figure FDA0003721594230000031
Figure FDA0003721594230000032
Figure FDA0003721594230000033
in the formula
Figure FDA0003721594230000034
The method is characterized in that the style characteristic S is output at the (i, j, k) th position of the first layer of the CNN, wherein the (i, j, k) corresponds to three dimensions of height, width and channel;
Figure FDA0003721594230000035
the method is characterized in that the style picture is output at the (i, j, k ') th position of the first layer of the CNN, wherein the (i, j, k') corresponds to three dimensions of height, width and channel;
Figure FDA0003721594230000036
outputting the network generated image G at the (i, j, k) th position of the ith layer of the CNN, wherein the (i, j, k) corresponds to three dimensions of height, width and channel;
Figure FDA0003721594230000037
is the total number in the height dimension of the l layer of CNN;
Figure FDA0003721594230000038
is the total number in the width dimension of the l-th layer of the CNN;
Figure FDA0003721594230000039
the total number of the CNN layer I channel dimensions;
Figure FDA00037215942300000310
graham matrix for the style feature S at layer l of CNN at channel k' and channel k；
Figure FDA00037215942300000311
A gram matrix of content features C of the I < th > layer of the CNN in a channel k' and a channel k;
the loss function for style migration is:
L＝αL c +βL s
wherein alpha and beta are weight coefficients.
8. A sonar simulation image generation system based on style migration is characterized by comprising:
the satellite image processing module is used for identifying the satellite remote sensing image by taking the satellite remote sensing image as input according to the target object type contained in the real sonar image, identifying an identification object with the same type as the target object, and marking the range where the identification object is located;
the satellite image segmentation module is used for carrying out image segmentation on the identified and marked satellite remote sensing image, and constructing a satellite subimage data set by taking the category of the target object as classification;
and the image style migration module is used for constructing a style migration network, taking the real sonar image as a style image, performing style migration on the segmented satellite sub-image, and generating a sonar simulation image to be used as a training sample image.
9. A sonar simulation image generation system based on style migration is characterized by comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, in which a program executable by a processor is stored, wherein the program executable by the processor is adapted to perform the method according to any one of claims 1 to 7 when executed by the processor.
CN202210752897.8A 2022-06-29 2022-06-29 Sonar simulation image generation method, system and medium based on style migration Active CN115019107B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210752897.8A CN115019107B (en) 2022-06-29 2022-06-29 Sonar simulation image generation method, system and medium based on style migration

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210752897.8A CN115019107B (en) 2022-06-29 2022-06-29 Sonar simulation image generation method, system and medium based on style migration

Publications (2)

Publication Number Publication Date
CN115019107A true CN115019107A (en) 2022-09-06
CN115019107B CN115019107B (en) 2024-09-06

Family

ID=83079038

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210752897.8A Active CN115019107B (en) 2022-06-29 2022-06-29 Sonar simulation image generation method, system and medium based on style migration

Country Status (1)

Country Link
CN (1) CN115019107B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116596753A (en) * 2023-07-20 2023-08-15 哈尔滨工程大学三亚南海创新发展基地 Acoustic image dataset expansion method and system based on style migration network

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110991516A (en) * 2019-11-28 2020-04-10 哈尔滨工程大学 Side-scan sonar image target classification method based on style migration
CN111582403A (en) * 2020-05-18 2020-08-25 哈尔滨工程大学 Zero-sample side-scan sonar image target classification method
CN113052215A (en) * 2021-03-15 2021-06-29 浙江大学 Sonar image automatic target identification method based on neural network visualization
US20210256304A1 (en) * 2018-10-10 2021-08-19 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Method and apparatus for training machine learning model, apparatus for video style transfer

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210256304A1 (en) * 2018-10-10 2021-08-19 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Method and apparatus for training machine learning model, apparatus for video style transfer
CN110991516A (en) * 2019-11-28 2020-04-10 哈尔滨工程大学 Side-scan sonar image target classification method based on style migration
CN111582403A (en) * 2020-05-18 2020-08-25 哈尔滨工程大学 Zero-sample side-scan sonar image target classification method
CN113052215A (en) * 2021-03-15 2021-06-29 浙江大学 Sonar image automatic target identification method based on neural network visualization

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
朱兆彤;付学志;胡友峰;: "一种利用迁移学习训练卷积神经网络的声呐图像识别方法", 水下无人系统学报, no. 01, 15 February 2020 (2020-02-15) *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116596753A (en) * 2023-07-20 2023-08-15 哈尔滨工程大学三亚南海创新发展基地 Acoustic image dataset expansion method and system based on style migration network
CN116596753B (en) * 2023-07-20 2024-02-02 哈尔滨工程大学三亚南海创新发展基地 Acoustic image dataset expansion method and system based on style migration network

Also Published As

Publication number Publication date
CN115019107B (en) 2024-09-06

Similar Documents

Publication Publication Date Title
KR102414452B1 (en) Target detection and training of target detection networks
CN112884760A (en) Near-water bridge multi-type disease intelligent detection method and unmanned ship equipment
US8319793B2 (en) Analyzing pixel data by imprinting objects of a computer-implemented network structure into other objects
CN110796048B (en) Ship target real-time detection method based on deep neural network
CN110310264A (en) A kind of large scale object detection method, device based on DCNN
CN111079739B (en) Multi-scale attention feature detection method
CN111369581A (en) Image processing method, device, equipment and storage medium
CN110647802A (en) Remote sensing image ship target detection method based on deep learning
CN117690128B (en) Embryo cell multi-core target detection system, method and computer readable storage medium
CN113221731B (en) Multi-scale remote sensing image target detection method and system
CN110503651A (en) A kind of significant object segmentation methods of image and device
CN113111740A (en) Characteristic weaving method for remote sensing image target detection
CN114677596A (en) Remote sensing image ship detection method and device based on attention model
CN115601371A (en) Multi-organ segmentation method, training method, medium, and electronic device
CN115841629A (en) SAR image ship detection method based on convolutional neural network
CN116681894A (en) Adjacent layer feature fusion Unet multi-organ segmentation method, system, equipment and medium combining large-kernel convolution
CN115019107A (en) Sonar simulation image generation method, system and medium based on style migration
CN113805178A (en) Method for detecting static obstructive objects on water surface
CN112966558A (en) Port automatic identification method and system based on optimized SSD target detection model
Tsai et al. Multi-dimensional underwater point cloud detection based on deep learning
CN111914766B (en) Method for detecting business trip behavior of city management service
CN113610178A (en) Inland ship target detection method and device based on video monitoring image
CN113807354A (en) Image semantic segmentation method, device, equipment and storage medium
CN117893744A (en) Remote sensing image segmentation method based on improved boundary guide context aggregation network
CN114663743B (en) Ship target re-identification method, terminal equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant