CN112381745B - Side-scan sonar image data enhancement method and system based on multi-source data synthesis - Google Patents

Side-scan sonar image data enhancement method and system based on multi-source data synthesis Download PDF

Info

Publication number
CN112381745B
CN112381745B CN202011231617.6A CN202011231617A CN112381745B CN 112381745 B CN112381745 B CN 112381745B CN 202011231617 A CN202011231617 A CN 202011231617A CN 112381745 B CN112381745 B CN 112381745B
Authority
CN
China
Prior art keywords
image
sonar image
target
information
source
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011231617.6A
Other languages
Chinese (zh)
Other versions
CN112381745A (en
Inventor
宋艳
李沂滨
崔明
王代超
郭庆稳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong University
Original Assignee
Shandong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong University filed Critical Shandong University
Priority to CN202011231617.6A priority Critical patent/CN112381745B/en
Publication of CN112381745A publication Critical patent/CN112381745A/en
Application granted granted Critical
Publication of CN112381745B publication Critical patent/CN112381745B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/13Edge detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10032Satellite or aerial image; Remote sensing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)

Abstract

The invention provides a side scan sonar image data enhancement method and a side scan sonar image data enhancement system based on multi-source data synthesis, which acquire target contour information and pixel information of a multi-source image; based on the contour information and the pixel information, performing pixel filling on the contour, and primarily synthesizing a sonar image; and placing the primarily synthesized image in a submarine reverberation area to obtain a synthesized side-scan sonar image. According to the invention, contour information of different visual angles and different models of the same type of targets in the sonar image is obtained by combining the multi-source image, and the contour is subjected to pixel filling through the pixel information of the sonar image and the contour information of the multi-source image, so that the number of a few target samples is increased.

Description

Side-scan sonar image data enhancement method and system based on multi-source data synthesis
Technical Field
The invention belongs to the field of image data processing, and particularly relates to a side scan sonar image data enhancement method and system based on multi-source data synthesis.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
As fewer submarine targets exist in the natural environment, most areas in the side scan sonar image have no targets. This results in a larger number of samples for non-target areas and smaller samples for target areas in the existing side scan sonar image dataset, with a significant difference in the number of samples for each class. The class imbalance problem greatly affects the fitting and generalization capabilities of the segmentation model based on deep learning, which results in over-fitting of the model and neglects the learning of small sample classes. Typical problems are as in practical environments, the number of seabed sunken vessels is far less than the number of coral reefs. If the segmentation model adopts a standard algorithm to target the highest accuracy, the classification of the coral reefs is preferred, and the submarine sunken ship is predicted to be the coral reef, so that the ocean detection efficiency is seriously affected.
Data enhancement is the main method to increase small sample data and solve the over-fitting problem. The existing data enhancement method increases the number of data samples by performing operations such as rotation, mirroring, translation, scaling and the like on the image. However, since the submarine image acquired by the side scan sonar only has images of a plurality of view angles of a certain target, the data volume required for training the deep learning model cannot be achieved even though the submarine image is processed by data enhancement methods such as rotation, mirror image and translation.
Disclosure of Invention
In order to solve the problems, the invention provides a side-scan sonar image data enhancement method and a side-scan sonar image data enhancement system based on multi-source data synthesis.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
the first aspect of the invention provides a side scan sonar image data enhancement method based on multi-source data synthesis.
A side-scan sonar image data enhancement method based on multi-source data synthesis comprises the following steps:
acquiring multi-source contour information and pixel information of a target sonar image;
based on the contour information and the pixel information, performing pixel filling on the contour, and primarily synthesizing a sonar image;
and placing the primarily synthesized image in a submarine reverberation area to obtain a synthesized side-scan sonar image.
As an alternative embodiment, the multi-source image includes several of an underwater camera image, a remote sensing image, and a network image.
In an alternative embodiment, the specific process of acquiring the target contour information of the multi-source image includes: and extracting the target contour from the multi-source image, and acquiring contour information of different visual angles and different models of the same type of target in the sonar image.
As a further definition, the specific process of extracting the target contour includes: target contour extraction is accomplished using threshold segmentation or k-means clustering.
As an alternative embodiment, based on the contour information and the pixel information, the specific process of filling the contour with pixels includes: based on a non-parametric sampling algorithm, taking an image block selected from the side-scan sonar image as a seed image, and gradually generating a new image based on the seed image to finish contour filling.
As an alternative embodiment, the specific process of placing the preliminarily synthesized image in the sub-sea reverberation area includes: randomly fusing the filled target area in a submarine reverberation area of the real sonar image.
As an alternative embodiment, any operation of rotation, mirroring, translation, and scaling is performed to increase the amount of image data.
The second aspect of the invention provides a side-scan sonar image data enhancement system based on multi-source data synthesis.
A multi-source data synthesis based side scan sonar image data enhancement system, comprising:
the data acquisition module is configured to acquire target contour information and pixel information of the multi-source image;
the image filling module is configured to perform pixel filling on the outline based on the outline information and the pixel information, and preliminarily synthesize a sonar image;
and the data enhancement module is configured to place the primarily synthesized image in the submarine reverberation area to obtain a synthesized side scan sonar image.
A third aspect of the present invention provides a computer-readable storage medium.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of a multi-source data synthesis based side scan sonar image data enhancement method as described above.
A fourth aspect of the invention provides a computer device.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in a multi-source data synthesis based side scan sonar image data enhancement method as described above when the program is executed.
Compared with the prior art, the invention has the beneficial effects that:
(1) Based on the data enhancement idea, the invention synthesizes the side scan sonar images by combining the multi-source images, and increases the number of the sonar image data set samples;
(2) The contour information of the invention has more sources, and compared with the existing data enhancement method, the method can greatly increase the number of synthesis targets;
(3) According to the method, according to the missing part in the adjacent pixel or edge pixel patch image, the image texture information and the gray information are comprehensively considered, the synthesized image is similar to the real image, and the size of the synthesized image is not limited;
(4) The gray value of the synthesized target sample is obtained based on echo data of the existing sonar target, and the synthesized target is close to the real sonar target; since the target sample contour and the location of the seafloor reverberation region are known in the semi-synthetic image, the segmented labels of the semi-synthetic image can be obtained simultaneously.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.
FIG. 1 is a flow chart of a method for enhancing side-scan sonar image data according to an embodiment of the present invention;
FIG. 2 is a schematic illustration of an aircraft in a Klein 5000 side scan sonar image according to an embodiment of the present invention;
FIG. 3 is a schematic view of a multi-view, multi-model aircraft binary image in accordance with an embodiment of the present invention;
fig. 4 (a) and 4 (b) are examples of sand wave sonar image synthesis based on NPS.
Detailed Description
The invention will be further described with reference to the drawings and examples.
It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present invention. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
Example 1
Just as in the background art, since the submarine image acquired by the side-scan sonar only has images of a certain target with a plurality of view angles, even if the submarine image is processed by data enhancement methods such as rotation, mirror image, translation and the like, the data volume required for training the deep learning model cannot be achieved. Aiming at the problems, the invention combines multi-source images (such as an underwater camera image, a remote sensing image, a network image and the like) to obtain contour information of different visual angles and different models of the same type of targets in a sonar image, and fills pixels in the contour through the pixel information of the sonar image and the contour information of the multi-source image, so that the number of a few target samples is increased. And in specific application, the contribution of various target samples to the training of the deep learning segmentation model is balanced, and the phenomenon of over-fitting and preference of the deep learning model is prevented.
The invention is based on the data enhancement idea, and the number of the sonar image data set samples is increased by combining the multi-source images to synthesize the side scan sonar image. The specific process is as shown in fig. 1:
(1) Acquiring multi-source profile information of a target
The submarine image acquired by the side scan sonar is a gray scale image, and often only images of several views of a certain target. As shown in fig. 2, the plane remains in Klein 5000 side scan sonar images, even though the data enhancement processing such as rotation, mirror image, translation, etc. is performed, the data volume required for training the deep learning model cannot be achieved. The invention combines images of various sources (such as an underwater camera image, a remote sensing image, a network image and the like) to obtain contour information of different visual angles and different types of similar targets in a sonar image. FIG. 3 is a binary image of an aircraft of different model and view angles, using thresholding or k-means clustering to accomplish target profile extraction in accordance with the present invention.
(2) Filling contours based on target pixel information
After the target contour is acquired, how to fill the contour area and further synthesize a new target sonar image is a key problem to be solved. The present invention addresses this problem based on the Non-parametric sampling (Non-Parametric Sampling, NPS) algorithm. The algorithm utilizes a given seed image to generate a similar image of greater resolution or patches missing portions in the image from adjacent pixels or edge pixels. The NPS algorithm takes an image block selected from the side scan sonar image as a seed image, the selected image block contains target pixel information, then a new image is gradually generated based on the image block, and contour filling is completed. Fig. 4 (a) and 4 (b) are examples of sand wave sonar image synthesis based on NPS. Fig. 4 (a) is a 64×64 pixel seed sonar image, fig. 4 (b) is a 200×200 pixel composite image obtained based on NPS algorithm, and the seed image is in a rectangular frame. The NPS algorithm comprehensively considers image texture information and gray information, and the synthesized image is similar to the real image and has no limitation on the size of the synthesized image.
(3) Side scan sonar image synthesis
In order to obtain a near-real synthesized image, the filled target area is randomly fused in a submarine reverberation area of the real sonar image, and the synthesized side-scan sonar image is obtained.
In this embodiment, the synthesized target image and the submarine reverberation area may be simply fused, that is, the synthesized image is rotated, mirrored, and scaled in size, and then stacked in the submarine reverberation area image to replace the original area of the submarine reverberation area image.
Of course, in other embodiments, fusion may be performed in other manners, which are not described herein.
Embodiment two:
a multi-source data synthesis based side scan sonar image data enhancement system, comprising:
the data acquisition module is configured to acquire target contour information and pixel information of the multi-source image;
the image filling module is configured to perform pixel filling on the outline based on the outline information and the pixel information, and preliminarily synthesize a sonar image;
and the data enhancement module is configured to place the primarily synthesized image in the submarine reverberation area to obtain a synthesized side scan sonar image.
Embodiment III:
a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of a multi-source data synthesis based side scan sonar image data enhancement method as described in embodiment one.
Embodiment four:
a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in a multi-source data synthesis based side scan sonar image data enhancement method as in embodiment one.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random access Memory (Random AccessMemory, RAM), or the like.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (6)

1. A side scan sonar image data enhancement method based on multi-source data synthesis is characterized by comprising the following steps: comprising the following steps:
acquiring multi-source image target contour information and sonar image pixel information of a target sonar image;
the multi-source image comprises a plurality of underwater camera images, remote sensing images and network images;
based on the contour information and the pixel information, performing pixel filling on the contour, and primarily synthesizing a sonar image;
placing the primarily synthesized image in a submarine reverberation area to obtain a synthesized side-scan sonar image;
the specific process for acquiring the target contour information of the multi-source image comprises the following steps: extracting the outline of a target from a multi-source image, and acquiring outline information of the same type of target with different visual angles and different types;
the specific process for extracting the target contour comprises the following steps: finishing target contour extraction by using threshold segmentation or k-means clustering;
based on the target contour information and the pixel information, the specific process of filling the contour with pixels comprises the following steps: based on a non-parametric sampling algorithm, taking an image block selected from the side-scan sonar image as a seed image, and gradually generating a new image based on the seed image to finish contour filling.
2. The method for enhancing the side scan sonar image data based on multi-source data synthesis as set forth in claim 1, wherein the method comprises the following steps: the specific process of placing the primarily synthesized image in the sub-sea reverberation area includes: randomly fusing the filled target area in a submarine reverberation area of the real sonar image.
3. The method for enhancing the side scan sonar image data based on multi-source data synthesis as set forth in claim 1, wherein the method comprises the following steps: and any operation of rotation, mirroring, translation and scaling is performed to increase the amount of image data.
4. A multi-source data synthesis-based side scan sonar image data enhancement system for implementing the multi-source data synthesis-based side scan sonar image data enhancement method as set forth in any one of claims 1-3, characterized in that: comprising the following steps:
the data acquisition module is configured to acquire target contour information of the multi-source image and pixel information of the sonar image; the specific process for acquiring the target contour information of the multi-source image comprises the following steps: extracting the outline of a target from a multi-source image, and acquiring outline information of the same type of target with different visual angles and different types;
the image filling module is configured to perform pixel filling on the outline based on the outline information and the pixel information, and preliminarily synthesize a sonar image;
and the data enhancement module is configured to place the primarily synthesized image in the submarine reverberation area to obtain a synthesized side scan sonar image.
5. A computer-readable storage medium, characterized by: a computer program stored thereon, which when executed by a processor, implements the steps of a multi-source data synthesis based side scan sonar image data enhancement method as claimed in any of claims 1-3.
6. A computer device, characterized by: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, said processor implementing the steps in a multi-source data synthesis based side scan sonar image data enhancement method as claimed in any one of claims 1-3 when said program is executed.
CN202011231617.6A 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis Active CN112381745B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011231617.6A CN112381745B (en) 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011231617.6A CN112381745B (en) 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis

Publications (2)

Publication Number Publication Date
CN112381745A CN112381745A (en) 2021-02-19
CN112381745B true CN112381745B (en) 2023-09-22

Family

ID=74579126

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011231617.6A Active CN112381745B (en) 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis

Country Status (1)

Country Link
CN (1) CN112381745B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101404059A (en) * 2008-09-24 2009-04-08 中国科学院自动化研究所 Iris image database synthesis method based on block texture sampling
CN105701835A (en) * 2016-02-26 2016-06-22 华北电力大学 Image edge detection method and system facing electric power facilities
CN108510458A (en) * 2018-03-29 2018-09-07 中国海洋大学 Sidescan-sonar image synthetic method based on deep learning method and non-parametric sampling
CN109934834A (en) * 2017-12-19 2019-06-25 北京京东尚科信息技术有限公司 Image outline extracting method and system
CN110766736A (en) * 2019-10-29 2020-02-07 京东方科技集团股份有限公司 Defect detection method, defect detection device, electronic equipment and storage medium
CN111582403A (en) * 2020-05-18 2020-08-25 哈尔滨工程大学 Zero-sample side-scan sonar image target classification method
CN111652149A (en) * 2020-06-04 2020-09-11 青岛理工大学 Deep convolutional neural network-based benthonic oil sonar detection image identification method

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101404059A (en) * 2008-09-24 2009-04-08 中国科学院自动化研究所 Iris image database synthesis method based on block texture sampling
CN105701835A (en) * 2016-02-26 2016-06-22 华北电力大学 Image edge detection method and system facing electric power facilities
CN109934834A (en) * 2017-12-19 2019-06-25 北京京东尚科信息技术有限公司 Image outline extracting method and system
CN108510458A (en) * 2018-03-29 2018-09-07 中国海洋大学 Sidescan-sonar image synthetic method based on deep learning method and non-parametric sampling
CN110766736A (en) * 2019-10-29 2020-02-07 京东方科技集团股份有限公司 Defect detection method, defect detection device, electronic equipment and storage medium
CN111582403A (en) * 2020-05-18 2020-08-25 哈尔滨工程大学 Zero-sample side-scan sonar image target classification method
CN111652149A (en) * 2020-06-04 2020-09-11 青岛理工大学 Deep convolutional neural network-based benthonic oil sonar detection image identification method

Also Published As

Publication number Publication date
CN112381745A (en) 2021-02-19

Similar Documents

Publication Publication Date Title
CN113706714A (en) New visual angle synthesis method based on depth image and nerve radiation field
Abdollahi et al. Improving road semantic segmentation using generative adversarial network
CN106845440B (en) Augmented reality image processing method and system
Li et al. Zero shot objects classification method of side scan sonar image based on synthesis of pseudo samples
WO2022195285A1 (en) Image processing using machine learning
US20230051960A1 (en) Coding scheme for video data using down-sampling/up-sampling and non-linear filter for depth map
EP3973507B1 (en) Segmentation for holographic images
Toriya et al. SAR2OPT: Image alignment between multi-modal images using generative adversarial networks
CN114821263A (en) Weak texture target pose estimation method based on feature fusion
CN115049556A (en) StyleGAN-based face image restoration method
Agrafiotis et al. Underwater Image Enhancement before Three-Dimensional (3D) Reconstruction and Orthoimage Production Steps: Is It Worth?
CN112381745B (en) Side-scan sonar image data enhancement method and system based on multi-source data synthesis
Dong et al. ShipGAN: Generative Adversarial Network based simulation-to-real image translation for ships
Dietrich-Sussner et al. Synthetic glacier SAR image generation from arbitrary masks using pix2pix algorithm
CN114742950B (en) Ship shape 3D digital reconstruction method and device, storage medium and electronic equipment
CN113505650B (en) Topographic feature line extraction method, device and equipment
Kallasi et al. Object detection and pose estimation algorithms for underwater manipulation
CN113034675A (en) Scene model construction method, intelligent terminal and computer readable storage medium
Johnston et al. Single View 3D Point Cloud Reconstruction using Novel View Synthesis and Self-Supervised Depth Estimation
Huang et al. Seg2Sonar: a full-class sample synthesis method applied to underwater sonar image target detection, recognition, and segmentation tasks
Fuchs et al. GAN-enhanced simulated sonar images for deep learning based detection and classification
Dahle et al. Semantic segmentation of historical photographs of the Antarctica Peninsula
CN117333627B (en) Reconstruction and complement method, system and storage medium for automatic driving scene
Yin et al. General model-agnostic transfer learning for natural degradation image enhancement
Fu Application and Analysis of RGB-D Salient Object Detection in Photographic Camera Vision Processing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant