CN112381745A - Side-scan sonar image data enhancement method and system based on multi-source data synthesis - Google Patents

Side-scan sonar image data enhancement method and system based on multi-source data synthesis Download PDF

Info

Publication number
CN112381745A
CN112381745A CN202011231617.6A CN202011231617A CN112381745A CN 112381745 A CN112381745 A CN 112381745A CN 202011231617 A CN202011231617 A CN 202011231617A CN 112381745 A CN112381745 A CN 112381745A
Authority
CN
China
Prior art keywords
image
sonar image
scan sonar
contour
source
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011231617.6A
Other languages
Chinese (zh)
Other versions
CN112381745B (en
Inventor
宋艳
李沂滨
崔明
王代超
郭庆稳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong University
Original Assignee
Shandong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong University filed Critical Shandong University
Priority to CN202011231617.6A priority Critical patent/CN112381745B/en
Publication of CN112381745A publication Critical patent/CN112381745A/en
Application granted granted Critical
Publication of CN112381745B publication Critical patent/CN112381745B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/13Edge detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10032Satellite or aerial image; Remote sensing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)

Abstract

The invention provides a side-scan sonar image data enhancement method and system based on multi-source data synthesis, which are used for acquiring target contour information and pixel information of a multi-source image; based on the contour information and the pixel information, carrying out pixel filling on the contour, and preliminarily synthesizing a sonar image; and placing the primarily synthesized image in a submarine reverberation area to obtain a synthesized side-scan sonar image. The method combines the multisource images to obtain the contour information of the same type of targets in the sonar images in different viewing angles and different types, and pixel filling is carried out on the contours through the sonar image pixel information and the multisource image contour information, so that the number of a small number of target samples is increased.

Description

Side-scan sonar image data enhancement method and system based on multi-source data synthesis
Technical Field
The invention belongs to the field of image data processing, and particularly relates to a side-scan sonar image data enhancement method and system based on multi-source data synthesis.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
Because the number of submarine targets in the natural environment is small, most of the side-scan sonar images have no targets. This results in the existing side scan sonar image data set, there are many samples in the non-target area and few samples in the target area, and there are obvious differences in the number of samples in each category. The problem of class imbalance greatly influences the fitting and generalization capability of a segmentation model based on deep learning, so that the model generates an overfitting condition, and the learning of small sample classes is neglected. Typical problems are that in practical environments, the number of sunken ships at the sea bottom is far less than that of coral reefs. If the segmentation model adopts a standard algorithm and aims at the highest accuracy, the coral reef category is preferred, and the seabed sunken ship is predicted to be the coral reef, so that the ocean exploration efficiency is seriously influenced.
Data enhancement is the main method to add small sample data and solve the overfitting problem. The existing data enhancement method increases the number of data samples by rotating, mirroring, translating, scaling and other operations on an image. However, the submarine image acquired by the side scan sonar only has images from a plurality of viewing angles of a certain target, and even if the submarine image is processed by data enhancement methods such as rotation, mirror image and translation, the data size required by the deep learning model cannot be achieved.
Disclosure of Invention
In order to solve the problems, the invention provides a side-scan sonar image data enhancement method and system based on multi-source data synthesis.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a side-scan sonar image data enhancement method based on multi-source data synthesis.
A side scan sonar image data enhancement method based on multi-source data synthesis comprises the following steps:
acquiring multi-source contour information and pixel information of a target sonar image;
based on the contour information and the pixel information, carrying out pixel filling on the contour, and preliminarily synthesizing a sonar image;
and placing the primarily synthesized image in a submarine reverberation area to obtain a synthesized side-scan sonar image.
As an alternative embodiment, the multi-source image includes several of an underwater camera image, a remote sensing image, and a network image.
As an alternative embodiment, the specific process of acquiring the target contour information of the multi-source image includes: and extracting the target contour of the multi-source image, and acquiring contour information of the same type of target in the sonar image at different viewing angles and different models.
As a further limitation, the specific process of extracting the target contour includes: and (4) completing target contour extraction by using threshold segmentation or k-means clustering.
As an alternative embodiment, based on the contour information and the pixel information, the specific process of pixel filling the contour includes: based on a non-parametric sampling algorithm, image blocks selected from the side-scan sonar image are used as seed images, new images are generated step by step based on the seed images, and contour filling is completed.
As an alternative embodiment, the specific process of placing the preliminary synthesized image in the submarine reverberation region includes: and randomly fusing the filled target area in the submarine reverberation area of the real sonar image.
As an alternative embodiment, any operation of rotation, mirror image, translation and scaling is further included, and the amount of image data is increased.
The invention provides a side-scan sonar image data enhancement system based on multi-source data synthesis.
A side-scan sonar image data enhancement system based on multi-source data synthesis, comprising:
the data acquisition module is configured to acquire target contour information and pixel information of the multi-source image;
the image filling module is configured to perform pixel filling on the outline based on the outline information and the pixel information, and preliminarily synthesize a sonar image;
and the data enhancement module is configured to place the primarily synthesized image in the submarine reverberation region to obtain a synthesized side-scan sonar image.
A third aspect of the invention provides a computer-readable storage medium.
A computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in a side-scan sonar image data enhancement method based on multi-source data synthesis as described above.
A fourth aspect of the invention provides a computer apparatus.
A computer device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the method for enhancing side-scan sonar image data based on multi-source data synthesis.
Compared with the prior art, the invention has the beneficial effects that:
(1) the method is based on a data enhancement idea, and combines a multi-source image to synthesize a side-scan sonar image, so that the number of samples of a sonar image data set is increased;
(2) the invention has more contour information sources, and compared with the existing data enhancement method, the method can greatly increase the number of synthetic targets;
(3) by using a non-parametric sampling method, repairing missing parts in the image according to adjacent pixels or edge pixels, comprehensively considering image texture information and gray scale information, enabling the synthesized image to be similar to a real image, and not limiting the size of the synthesized image;
(4) the gray value of the synthesized target sample is obtained based on the existing sonar target echo data, and the synthesized target is close to a real sonar target; since the target sample contour and the position of the seafloor reverberation region are known in the semi-synthetic image, the segmentation labels of the semi-synthetic image can be obtained simultaneously.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.
FIG. 1 is a flowchart of a side-scan sonar image data enhancement method according to an embodiment of the present invention;
FIG. 2 is a schematic view of an aircraft in a Klein 5000 side scan sonar image of an embodiment of the present invention;
FIG. 3 is a schematic diagram of a multi-view, multi-model aircraft binary image in accordance with an embodiment of the present invention;
fig. 4(a) and 4(b) are examples of synthesizing a sand-wave sonar image based on NPS.
Detailed Description
The invention is further described with reference to the following figures and examples.
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example one
As described in the background art, the submarine image acquired by the side scan sonar only has images from a plurality of viewing angles of a certain target, and even if the submarine image is processed by data enhancement methods such as rotation, mirroring, translation and the like, the amount of data required for training the deep learning model cannot be achieved. Aiming at the problems, the invention combines multi-source images (such as underwater camera images, remote sensing images, network images and the like) to obtain contour information of the same kind of targets in sonar images in different visual angles and different models, and pixel filling is carried out on the contours through sonar image pixel information and multi-source image contour information, so that the number of a small number of target samples is increased. In specific application, the contribution of various target samples to deep learning segmentation model training is balanced, and the phenomena of overfitting and preference of the deep learning model are prevented.
The invention is based on the data enhancement idea, and combines the multi-source image to synthesize the side-scan sonar image, thereby increasing the sample number of the sonar image data set. The specific process is shown in figure 1:
(1) obtaining multi-source contour information of an object
The image of the sea floor acquired by the side scan sonar is a grayscale image and often has only images of a certain target from several perspectives. As shown in fig. 2, the airplane debris in the Klein 5000 side scan sonar image cannot reach the amount of data required for training the deep learning model even if the airplane debris is subjected to data enhancement processing such as rotation, mirroring, translation and the like. The method combines images from various sources (such as underwater camera images, remote sensing images, network images and the like) to obtain the contour information of the same type of target in the sonar images at different viewing angles and different models. FIG. 3 is a binary image of an aircraft including different models and different perspectives, where the invention uses threshold segmentation or k-means clustering to complete target contour extraction.
(2) Filling contours based on target pixel information
After the target contour is obtained, how to fill the contour region and further synthesize a new target sonar image is a key problem to be solved. The present invention solves this problem based on a Non-Parametric Sampling (NPS) algorithm. The algorithm generates a similar image of greater resolution with a given seed image, or patches missing portions in the image from neighboring pixels or edge pixels. The NPS algorithm takes image blocks selected from the side-scan sonar images as seed images, the selected image blocks contain target pixel information, then new images are generated step by step based on the image blocks, and contour filling is completed. Fig. 4(a) and 4(b) are examples of synthesizing a sand-wave sonar image based on NPS. Fig. 4(a) is a 64 × 64 pixel seed sonar image, fig. 4(b) is a composite image of 200 × 200 pixels obtained by the NPS algorithm, and the inside of the rectangular frame is a seed image. The NPS algorithm comprehensively considers image texture information and gray information, the synthesized image is similar to a real image, and the size of the synthesized image is not limited.
(3) Side scan sonar image synthesis
In order to obtain an approximate real synthetic image, the filled target area is randomly fused in the submarine reverberation area of the real sonar image, and a synthetic side-scan sonar image is obtained.
In this embodiment, the synthesized target image and the reverberation region of the ocean floor can be simply fused, that is, the synthesized image is rotated, mirrored, and scaled in size, and then is superimposed on the image of the reverberation region of the ocean floor to replace the original region of the image of the reverberation region of the ocean floor.
Of course, in other embodiments, other methods may be used for fusion, which are not described herein again.
Example two:
a side-scan sonar image data enhancement system based on multi-source data synthesis, comprising:
the data acquisition module is configured to acquire target contour information and pixel information of the multi-source image;
the image filling module is configured to perform pixel filling on the outline based on the outline information and the pixel information, and preliminarily synthesize a sonar image;
and the data enhancement module is configured to place the primarily synthesized image in the submarine reverberation region to obtain a synthesized side-scan sonar image.
Example three:
a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in a side scan sonar image data enhancement method based on multi-source data synthesis according to an embodiment.
Example four:
a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a method for enhancing side scan sonar image data based on multi-source data synthesis according to an embodiment.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A side scan sonar image data enhancement method based on multi-source data synthesis is characterized by comprising the following steps: the method comprises the following steps:
acquiring multi-source contour information and pixel information of a target sonar image;
based on the contour information and the pixel information, carrying out pixel filling on the contour, and preliminarily synthesizing a sonar image;
and placing the primarily synthesized image in a submarine reverberation area to obtain a synthesized side-scan sonar image.
2. The side-scan sonar image data enhancement method based on multi-source data synthesis according to claim 1, comprising: the multi-source image comprises a plurality of underwater camera images, remote sensing images and network images.
3. The side-scan sonar image data enhancement method based on multi-source data synthesis according to claim 1, comprising: the specific process for acquiring the target contour information of the multi-source image comprises the following steps: and extracting the target contour of the multi-source image, and acquiring contour information of the same type of target in the sonar image at different viewing angles and different models.
4. The side-scan sonar image data enhancement method based on multi-source data synthesis according to claim 3, comprising: the specific process of extracting the target contour comprises the following steps: and (4) completing target contour extraction by using threshold segmentation or k-means clustering.
5. The side-scan sonar image data enhancement method based on multi-source data synthesis according to claim 1, comprising: based on the contour information and the pixel information, the specific process of filling pixels in the contour comprises the following steps: based on a non-parametric sampling algorithm, image blocks selected from the side-scan sonar image are used as seed images, new images are generated step by step based on the seed images, and contour filling is completed.
6. The side-scan sonar image data enhancement method based on multi-source data synthesis according to claim 1, comprising: the specific process of placing the preliminarily synthesized image in the submarine reverberation region comprises the following steps: and randomly fusing the filled target area in the submarine reverberation area of the real sonar image.
7. The side-scan sonar image data enhancement method based on multi-source data synthesis according to claim 1, comprising: and the method also comprises the step of carrying out any operation of rotation, mirror image, translation and scaling to increase the quantity of image data.
8. A side scan sonar image data enhancement system based on multi-source data synthesis is characterized in that: the method comprises the following steps:
the data acquisition module is configured to acquire target contour information and pixel information of the multi-source image;
the image filling module is configured to perform pixel filling on the outline based on the outline information and the pixel information, and preliminarily synthesize a sonar image;
and the data enhancement module is configured to place the primarily synthesized image in the submarine reverberation region to obtain a synthesized side-scan sonar image.
9. A computer-readable storage medium characterized by: a computer program stored thereon, which when executed by a processor implements the steps in a side scan sonar image data enhancement method based on multi-source data synthesis according to any one of claims 1-7.
10. A computer device, characterized by: the method comprises a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor executes the program to realize the steps in the multi-source data synthesis-based side scan sonar image data enhancement method according to any one of claims 1-7.
CN202011231617.6A 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis Active CN112381745B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011231617.6A CN112381745B (en) 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011231617.6A CN112381745B (en) 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis

Publications (2)

Publication Number Publication Date
CN112381745A true CN112381745A (en) 2021-02-19
CN112381745B CN112381745B (en) 2023-09-22

Family

ID=74579126

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011231617.6A Active CN112381745B (en) 2020-11-06 2020-11-06 Side-scan sonar image data enhancement method and system based on multi-source data synthesis

Country Status (1)

Country Link
CN (1) CN112381745B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101404059A (en) * 2008-09-24 2009-04-08 中国科学院自动化研究所 Iris image database synthesis method based on block texture sampling
CN105701835A (en) * 2016-02-26 2016-06-22 华北电力大学 Image edge detection method and system facing electric power facilities
CN108510458A (en) * 2018-03-29 2018-09-07 中国海洋大学 Sidescan-sonar image synthetic method based on deep learning method and non-parametric sampling
CN109934834A (en) * 2017-12-19 2019-06-25 北京京东尚科信息技术有限公司 Image outline extracting method and system
CN110766736A (en) * 2019-10-29 2020-02-07 京东方科技集团股份有限公司 Defect detection method, defect detection device, electronic equipment and storage medium
CN111582403A (en) * 2020-05-18 2020-08-25 哈尔滨工程大学 Zero-sample side-scan sonar image target classification method
CN111652149A (en) * 2020-06-04 2020-09-11 青岛理工大学 Deep convolutional neural network-based benthonic oil sonar detection image identification method

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101404059A (en) * 2008-09-24 2009-04-08 中国科学院自动化研究所 Iris image database synthesis method based on block texture sampling
CN105701835A (en) * 2016-02-26 2016-06-22 华北电力大学 Image edge detection method and system facing electric power facilities
CN109934834A (en) * 2017-12-19 2019-06-25 北京京东尚科信息技术有限公司 Image outline extracting method and system
CN108510458A (en) * 2018-03-29 2018-09-07 中国海洋大学 Sidescan-sonar image synthetic method based on deep learning method and non-parametric sampling
CN110766736A (en) * 2019-10-29 2020-02-07 京东方科技集团股份有限公司 Defect detection method, defect detection device, electronic equipment and storage medium
CN111582403A (en) * 2020-05-18 2020-08-25 哈尔滨工程大学 Zero-sample side-scan sonar image target classification method
CN111652149A (en) * 2020-06-04 2020-09-11 青岛理工大学 Deep convolutional neural network-based benthonic oil sonar detection image identification method

Also Published As

Publication number Publication date
CN112381745B (en) 2023-09-22

Similar Documents

Publication Publication Date Title
Bovcon et al. The mastr1325 dataset for training deep usv obstacle detection models
CN113706714A (en) New visual angle synthesis method based on depth image and nerve radiation field
Alonso et al. CoralSeg: Learning coral segmentation from sparse annotations
CN110889844B (en) Coral distribution and health condition assessment method based on deep clustering analysis
CN108416751A (en) A kind of new viewpoint image combining method assisting full resolution network based on depth
Wang et al. A feature-supervised generative adversarial network for environmental monitoring during hazy days
CN114581571A (en) Monocular human body reconstruction method and device based on IMU and forward deformation field
CN115428027A (en) Neural opaque point cloud
CN115049556A (en) StyleGAN-based face image restoration method
Periyasamy et al. How to get the most out of u-net for glacier calving front segmentation
Agrafiotis et al. Underwater Image Enhancement before Three-Dimensional (3D) Reconstruction and Orthoimage Production Steps: Is It Worth?
Sun et al. IRDCLNet: Instance segmentation of ship images based on interference reduction and dynamic contour learning in foggy scenes
Dong et al. ShipGAN: Generative Adversarial Network based simulation-to-real image translation for ships
Dietrich-Sussner et al. Synthetic glacier SAR image generation from arbitrary masks using pix2pix algorithm
CN112381745B (en) Side-scan sonar image data enhancement method and system based on multi-source data synthesis
CN112132164A (en) Target detection method, system, computer device and storage medium
CN114742950B (en) Ship shape 3D digital reconstruction method and device, storage medium and electronic equipment
Marques et al. Self-supervised subsea slam for autonomous operations
Gracias et al. Application challenges of underwater vision
CN112950481B (en) Water bloom shielding image data collection method based on image mosaic network
CN115311544A (en) Underwater fish target detection method and device
CN113505650A (en) Method, device and equipment for extracting topographic feature line
Zhang et al. Novel approaches to enhance coral reefs monitoring with underwater image segmentation
CN113034675A (en) Scene model construction method, intelligent terminal and computer readable storage medium
Schoening et al. Seabed classification using a bag-of-prototypes feature representation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant