KR20230110861A - Method for Detecting Fish in Sonar Video using Deep Learning - Google Patents

Abstract

The present invention is a fish detection method using a fish detection device including a control module, a deep learning module, a sonar sensor, and a display module, wherein the control module captures a sonar image generated by receiving a sonar sensor and generates an original image to generate an original image including a fish, a labeled image generation step in which the deep learning module labels a fish region in the original image where the fish exists to generate a labeled image, and the deep learning module generates a labeled image of the fish region and the fish in the original image. A mask generation step of generating a mask displaying remaining regions excluding regions in different colors; a deep learning model learning step of learning a deep learning model using the labeled image and the mask by the deep learning module;
A fish detection step in which the control module detects fish by applying the deep learning model to a sonar image captured by the sonar sensor, and a fish display step in which a fish detection image including only the detected fish is displayed on the display module is disclosed.

Description

Method for detecting fish in sonar video using deep learning {Method for Detecting Fish in Sonar Video using Deep Learning}

The present invention relates to a method for detecting fish in a sonar image using deep learning.

A sonar device transmits sound wave pulses through water and determines the position of an object by measuring the time it takes for the sound wave pulse to be reflected while receiving the reflected sound wave pulse after contacting an object such as a fish, plant, or seabed. In addition, the sonar device can distinguish the type of object by measuring the intensity of the reflected sound wave pulse. The sonar device may form an element image by a received sound wave pulse.

Most of the fish may appear in an arch shape on sonar images. The reason is that the reflectance values at both ends of the fish in the sonar's conical range are farther away than at the midpoint. In addition, the size of the shape of the fish changes within the sonar image according to the location (or depth). Therefore, care must be taken in finding the size (or type) of fish within the sonar image.

The image processing technique is a method of recognizing an object (fish) through image preprocessing technology and feature point extraction, but it can be difficult to stably detect fish due to severe image noise caused by reflected waves reflected from the surface of the water, floor, or wall. Therefore, attempts are being made to accurately detect fish by applying various techniques. As a technique used in the fish detection, a filter noise removal technique may be applied to extract an object area by removing noise from an image using a median filter or the like. The filter noise removal technique is effective in removing small noise, but when the noise is large, such as in a sonar image, image distortion becomes severe, which may cause errors in fish detection.

An object of the present invention is to provide a method for detecting fish in a sonar image using deep learning, which can more stably detect fish in a sonar image.

A fish detection method in a sonar image using deep learning of the present invention is a fish detection method using a fish detection device including a control module, a deep learning module, a sonar sensor, and a display module, wherein the control module captures a sonar image generated by receiving a sonar sensor and generates an original image to generate an original image including a fish, and a labeled image generation step in which the deep learning module labels a fish area in the original image to generate a labeled image; A mask generation step in which a module generates a mask displaying the fish region and remaining regions excluding the fish region in different colors in the original image, a deep learning model learning step in which the deep learning module learns a deep learning model using the labeled image and the mask, a fish detection step in which the control module detects fish by applying the deep learning model to a sonar image captured by the sonar sensor, and a fish display step displaying a fish detection image including only the detected fish on the display module. characterized by

In addition, in the fish detection method in the sonar image using the deep learning, the sonar image may include a terrain feature including a floor and a hill and an object including the fish.

Also, in the generating of the labeled image, all pixels of the original image may be labeled using semantic segmentation.

In the mask generating step, a mask for the fish region may be generated by using a label for the fish region in the original image.

In the mask generating step, the mask may display the fish area in white and the remaining area in black.

The method for detecting fish in an original image using deep learning of the present invention detects fish in a sonar image based on deep learning using a labeled image generated from a sonar image and a mask image by masked semantic segmentation, so that the fish can be stably detected without being sensitive to changes in the surrounding environment.

1 is a block diagram of a fish detection device for a method of detecting fish in a sonar image using deep learning according to an embodiment of the present invention.
2 is a process flow chart for a method of detecting fish in a sonar image using deep learning according to an embodiment of the present invention.
3 is an original image generated by the method of detecting fish in a sonar image using deep learning according to the present invention.
4 is a labeled image generated in the method of detecting fish in a sonar image using deep learning of the present invention.
5 is a mask image generated in the fish detection method in a sonar image using deep learning of the present invention.

Hereinafter, the fish detection method in a sonar image using deep learning of the present invention will be described in more detail through examples and accompanying drawings.

First, a fish detection device for a fish detection method according to an embodiment of the present invention will be schematically described.

1 is a block diagram of a fish detection device for a fish detection device according to an embodiment of the present invention.

Referring to FIG. 1 , the fish detection device 100 may include a control module 110, a deep learning module 120, a sonar sensor 130, and a display module 140. The control module 110 and the deep learning module 120 may be integrally formed.

The fish detection device 100 may be applied to a fish detection method according to an embodiment of the present invention.

The control module 110 may control each step of the fish detection method. The control module 110 may control deep learning model learning. The control module 110 may control the operation of the sonar sensor 130 . The control module 110 may receive a signal for a sonar image from the sonar sensor 130 . In addition, the control module 110 may extract an original image from a captured sonar image. The control module 110 may generate a fish detection image for the detected fish.

The control module 110 may detect fish in real time from sonar images captured using the learned deep learning model. The control module 110 may perform fish detection by analyzing sonar images in real time. Also, the control module 110 may classify fish in real time.

The control module 110 may have a separate storage device. The storage device may store sonar images transmitted from the sonar sensor 130 . In addition, the storage device may store various information required for deep learning model learning. Also, the storage device may store the learned deep learning model.

The deep learning module 120 may proceed with deep learning model learning. The deep learning module 120 is interlocked with the control module 110 and can receive sonar images. The deep learning module 120 may extract an original image from a sonar image and generate a labeling image and a mask. In addition, the deep learning module 120 may learn a deep learning model using the generated labeling image and mask. The deep learning module 120 may transmit the learned deep learning model to the control module 110 .

The sonar sensor 130 may be located in the sea and generate signals for sonar images. The sonar sensor 130 may transmit sound wave pulses through water and receive reflected sound pulses after contacting an object such as fish, plants, or the bottom of the sea. The sonar sensor 130 may transmit a received signal to the control module 110 . The sonar sensor 130 may be formed as a general sensor used for fish detection. The sonar sensor 130 may be mounted on the lower end of a separate floating device (not shown) and located underwater.

The display module 140 can be created with a general device capable of generating a video or image. The display module 140 may be formed in two pieces. The display module 140 may be formed of a display module 140 displaying a sonar image and a display module 140 displaying a fish detection image.

Next, a method for detecting fish in a sonar image using deep learning according to an embodiment of the present invention will be described.

2 is a process flow chart for a method of detecting fish in a sonar image using deep learning according to the present invention. 3 is an original image generated by the method of detecting fish in a sonar image using deep learning according to the present invention. 4 is a labeled image generated in the method of detecting fish in a sonar image using deep learning of the present invention. 5 is a mask image generated in the fish detection method in a sonar image using deep learning of the present invention.

Referring to FIGS. 2 to 5 , the method for detecting fish in a sonar image using deep learning according to an embodiment of the present invention may include generating an original image (S10), generating a labeled image (S20), generating a mask (S30), learning a deep learning model (S40), detecting fish (S50), and displaying fish (S60).

The fish detection method may use the fish detection device 100 of FIG. 1 . In the fish detection method, a deep learning model may be formed by generating a labeled image and a mask from a sonar image of a device image acquired by the sonar sensor 130 in advance. In addition, the fish detection method may learn a deep learning model and detect a fish from a sonar image generated by an element device based on the deep learning model. Here, the sonar image is an image generated by the sonar sensor 130, and the sonar image may be an image selected at a specific time from the sonar image.

The fish detection method may use masked semantic segmentation unlike commonly used segmentation. Since the fish detection method uses masked semantic segmentation, it is possible to accurately and stably detect fish by suppressing other features in a sonar image and reinforcing the fish feature to be detected. In addition, since the fish detection method in the sonar image can extract only fish features, it can be corrected according to the size of the fish, and the fish can be classified. Therefore, the fish detection method in the sonar image can reduce detection errors that may appear due to the fact that a part of the sea bottom and a part of fish have similar characteristics regardless of the reflected wave intensity of the sound wave pulse.

The fish detection method may be applied to various types of fish. The fish detection method may be applied to detect various fish included in a sonar image by learning a deep learning model for each fish.

The original image generating step (S10) is a step of generating an original image by capturing a sonar image generated by receiving a sound wave pulse by the sonar sensor 130. The original image generating step (S10) may be performed in the control module 110. Alternatively, in the original image generating step (S10), the deep learning module 120 may receive the sonar image from the control module 110 and generate the original image. The sonar sensor 130 may generate a sonar image by emitting a sound wave pulse in the sea and receiving a sound wave pulse reflected from a feature such as a bottom or a hill, or an object such as seaweed or fish. The fish may be various types of fish, and the sonar image may include various types of fish. can be Accordingly, the original image may include various types of fish. Also, the original image may be generated for each fish.

The sonar image may be displayed on a display device such as a monitor. The original image may be generated by capturing an image including a fish in a sonar image. Accordingly, the original image may include a feature such as a floor or a hill along with a fish area. The original image may be displayed in different colors according to objects or features. Also, the original image may be displayed in different colors for each fish.

The labeled image generation step (S20) is a step in which the deep learning module 120 labels the fish region in the original image to generate a labeled image. As mentioned above, the original image may include an object such as seaweed or a feature such as a hill along with fish. In the labeled image generating step (S20), all pixels in the original image may be labeled using semantic segmentation. The semantic segmentation may be performed using a convolutional neural network.

Accordingly, the semantic segmentation may use classification in units of pixels. The semantic segmentation may label a fish region included in the original image. The fish area is an area where fish are located, and is an area where fish exist in the device image. In addition, the fish area may be set and labeled for each fish.

As shown in FIG. 3 , the labeled image may be displayed as a boundary line for fish and a fish area having a color within the boundary line. The labeled image may be divided into a fish area and a residual area excluding the fish area. In the labeled image, all other features in the image may disappear and only fish features may remain. In addition, the labeled image may have different characteristics for each fish. Therefore, when the deep learning model is trained using the labeled image, it is possible to remove the surrounding environment and noise at once and detect the characteristics of the fish.

The mask generation step (S30) is a step in which the deep learning module 120 generates a mask displaying the fish area and the remaining remaining area in different colors in the original image. One or a plurality of fish areas may exist according to the sonar image. The mask may be generated using a labeled image by semantic segmentation. In the mask generation step ( S30 ), a mask for the fish region may be generated by using a label for the fish region with respect to the original image.

In the mask generation step (S30), the confidence level of the segmentation prior information of each pixel or each pixel group including a plurality of pixels is determined in consideration of the distribution and numerical value of the segmentation dictionary information among the pixels in the original image, and the segmentation dictionary information and confidence level for each pixel or each pixel group can be used together when setting the final segmentation dictionary information. Also, in the mask generating step ( S30 ), pixels having a value of 1 may be generated as a mask only for the fish region to be considered. For example, in the mask generating step (S30), when there is a fish in the image, the mask may generate an image having a value of 1 for a mask area corresponding to the fish and a value of 0 for the remaining areas. The mask may display the fish in white and display the rest in black.

The deep learning model learning step (S40) is a step in which the deep learning module 120 learns a deep learning model using a labeled image and a mask. In the deep learning model learning step (S40), a deep learning model for fish detection may be learned from a sonar image using a labeled image and a mask generated from the sonar image. Here, the deep learning model can be learned using a general artificial neural network used in deep learning. Accordingly, the deep learning model may form a trained deep learning model for fish. In the deep learning model learning step (S40), deep learning model learning may be performed for various fish.

The fish detection step (S50) is a step in which the control module 110 detects fish in the sonar image by applying a deep learning model to the sonar image captured by the sonar sensor 130. In the fish detection step (S50), a sonar image may be captured from a sonar image captured in real time. In the fish detection step (S50), fish may be detected in the sonar image by applying a deep learning model to the captured sonar image. The fish detection step (S50) may be performed by capturing a plurality of sonar images. The fish detection step (S50) may be performed by capturing a plurality of sonar images at predetermined time intervals. In the fish detection step (S50), fish detection may be performed from the sonar image in real time using the learned deep learning model. In addition, the fish detection step (S50) may be performed for each fish.

The fish displaying step (S60) is a step of displaying a fish detection image including only the detected fish. The fish detection image may include only fish detected in the sonar image. In the detected fish image, the fish is displayed in white or white, and the background may be displayed in a different color. In the fish displaying step ( S60 ), a fish detection image may be displayed on the display module 140 .

What has been described above is only one embodiment for implementing a method of detecting fish in a cow or image using deep learning according to an embodiment of the present invention. The present invention is not limited to the above-described embodiments, and as claimed in the following claims, it will be said that the technical spirit of the present invention exists to the extent that various changes can be made by anyone skilled in the art without departing from the gist of the present invention.

100: fish detection device
110: control module 120: deep learning module
130: sonar sensor 140: display module

Claims (5)

A fish detection method using a fish detection device including a control module, a deep learning module, a sonar sensor, and a display module,
An original image generating step in which the control module generates an original image including a fish by capturing a sonar image generated by receiving a sound wave pulse from the sonar sensor;
a labeled image generation step of generating a labeled image by labeling, by the deep learning module, a fish area where the fish exists in the original image;
a mask generation step of, by the deep learning module, generating a mask displaying the fish region and a remaining region excluding the fish region in the original image in different colors;
A deep learning model learning step in which the deep learning module learns a deep learning model using the labeled image and the mask;
A fish detection step in which the control module detects fish by applying the deep learning model to the sonar image captured by the sonar sensor; and
A fish detection method in a sonar image using deep learning, characterized in that it comprises a fish display step of displaying a fish detection image including only the detected fish on the display module. According to claim 1,
The method of detecting fish in a sonar image using deep learning, characterized in that the sonar image includes a terrain feature including a floor and a hill and an object including the fish. According to claim 1,
The method of detecting fish in a sonar image using deep learning, characterized in that the step of generating the labeled image labels all pixels in the original image using semantic segmentation. According to claim 1,
The method of detecting fish in a sonar image using deep learning, characterized in that in the mask generating step, a mask for the fish area is generated by using a label for the fish area with respect to the original image. According to claim 1,
The fish detection method in a sonar image using deep learning, characterized in that the mask displays the fish area in white and the remaining area in black.