JP2024027023A - Fish species learning device, fish species learning method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a fish species learning device, a fish species learning method, and a program that can more accurately perform machine learning for fish species discrimination.

SOLUTION: A server 20 (fish species learning device) comprises: a storage unit 202 that stores annotation data for every unit echo image; and a control unit 201. The control unit 201 extracts, from annotation data of a first unit echo image and a second unit echo image, which are temporally continuous, first annotation data and second annotation data of a fish school present between the first unit echo image and second unit echo image, and integrates the extracted first annotation data and second annotation data to generate annotation data of the entire fish school present between the first unit echo image and second unit echo image.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a fish species learning device, a fish species learning system, a fish species learning method, and a program that perform machine learning for fish species discrimination using teacher data (annotation data).

2. Description of the Related Art Fish detection devices that detect schools of fish in water are conventionally known. In this type of fish finding device, ultrasonic waves are transmitted underwater and the reflected waves are received. Echo data is generated according to the intensity of the received reflected wave, and an echo image is displayed based on the generated echo data. The user can confirm the school of fish from the echo image and can smoothly capture the school of fish.

In this case, it is preferable that the fish species of the school of fish on the echo image be further determined and displayed. Thereby, the user can efficiently catch the fish species he or she desires.

Such fish species discrimination may be performed using, for example, a machine learning model (machine learning algorithm). Learning is performed on a machine learning model using a large amount of training data (annotation data). Each annotation data includes echo data of the fish school, the range (water depth, time) of the fish school, and the fish species of the fish school. Patent Document 1 below describes a configuration in which a user generates such annotation data.

JP 2019-200175 Publication

Generally, annotation data is generated for each unit echo image. That is, a series of echo images for a predetermined time period is divided into a plurality of echo images with a time width corresponding to one frame, and a plurality of unit echo images from which annotation data is generated are generated. An operator such as an expert specifies the range of a school of fish for each unit echo image, and further assigns a fish species label to the specified school of fish. As a result, annotation data is generated in which the range of the school of fish, the species of fish within the range of the school of fish, and the image data (echo data) included in the range of the school of fish are associated.

With such a generation method, the entire school of fish is not necessarily included in the unit echo image. That is, one school of fish may straddle two temporally consecutive unit echo images. In such a case, two fish school portions obtained by dividing one fish school will be included in each of the two echo images. In the above generation method, since processing is performed in units of echoes, annotation data is generated individually for each divided fish school portion.

However, in the machine learning described above, characteristics arising from the entire school of fish, such as the tailing and distribution of echoes unique to the fish species, can affect the accuracy of the machine learning. Therefore, as described above, when annotation data generated for a part of a school of fish is used for machine learning, the accuracy of machine learning may decrease.

In view of this problem, an object of the present invention is to provide a fish species learning device, a fish species learning method, and a program that can perform machine learning for fish species discrimination with higher accuracy.

A first aspect of the present invention relates to a fish species learning device. The fish species learning device according to this aspect includes a storage unit that stores annotation data for each unit echo image, and a control unit. The control unit generates a first annotation of a school of fish spanning between the first unit echo image and the second unit echo image from the annotation data of the temporally continuous first unit echo image and second unit echo image. The first annotation data and the second annotation data are respectively extracted, and the extracted first annotation data and the second annotation data are integrated to generate annotation data for the entire school of fish.

According to the fish species learning device according to this aspect, the first annotation data and the second annotation data of the fish school spanning between the first unit echo image and the second unit echo image are integrated, and the annotation data of the entire fish school is is generated. Therefore, since the annotation data for the entire school of fish can be used for machine learning, machine learning for identifying fish species can be performed with higher accuracy.

In the fish species learning device according to the present aspect, the control unit may control a first school of fish that extends to a vicinity of a first boundary on the second unit echo image side on the first unit echo image, and a second unit echo image that A second school of fish extending to near the second boundary on the side of the first unit echo image is identified in the above, and when the depth range of the first school of fish and the depth range of the second school of fish substantially match, the The annotation data of the first school of fish and the second school of fish may be configured to be extracted as the first annotation data and the second annotation data, respectively.

According to this configuration, the first school of fish and the second school of fish, which have a high probability of forming one school of fish, can be smoothly identified in the first unit echo image and the second unit echo image. Therefore, by integrating the annotation data of these fish schools, annotation data of one entire fish school can be generated with high accuracy.

In this configuration, the control unit determines whether the depth range of the first school of fish and the depth range of the second school of fish substantially match or not. The determination may be made based on a matching rate with the depth range of the second school of fish near the second boundary.

According to this configuration, the depth range of the first fish school and the depth range of the second fish school are different from each other in the vicinity of the first boundary and the second boundary, which are the boundaries between the first unit echo image and the second unit echo image. Since it is determined whether they substantially match, it is possible to accurately determine whether the first fish school and the second fish school are continuous with each other. Therefore, annotation data for one entire school of fish can be generated with high accuracy from annotation data for the first school of fish and the second school of fish.

Further, in this configuration, the control unit extracts a school of fish that is at least partially included in a predetermined time range from the first boundary as the first fish school, and extracts a school of fish that is at least partially included in a predetermined time range from the second boundary. The second fish school may be configured to extract a fish school in which the second fish school is included.

According to this configuration, even when the distribution of fish is sparse near the boundary, the first fish school and the second fish school constituting the fish school spanning the first unit echo image and the second unit echo image are transferred to the first unit echo image. It can be specified without missing anything in the echo image and the second unit echo image. Therefore, annotation data for one entire school of fish can be appropriately generated.

In the fish species learning device according to the present aspect, when the fish species included in the first annotation data and the fish species included in the second annotation data are different, the control unit controls the It may be configured to set the fish species of the annotation data of the fish school.

In this case, the setting condition may include, for example, setting the fish species of the newer annotation data of the first annotation data and the second annotation data.

According to these configurations, when the fish species included in the first annotation data and the second annotation data to be integrated are different from each other, a highly reliable fish species can be set in the integrated annotation data. Therefore, the accuracy of machine learning can be improved.

A second aspect of the present invention relates to a method for learning fish species. In the fish species learning method according to this aspect, from the annotation data of the first unit echo image and the second unit echo image that are temporally continuous, the method straddles between the first unit echo image and the second unit echo image. First annotation data and second annotation data of the school of fish are respectively extracted, and the extracted first annotation data and second annotation data are integrated to generate annotation data of the entire school of fish.

A third aspect of the present invention relates to a program that causes a computer to execute a predetermined function. The program according to this aspect is configured to calculate information on a school of fish that straddles between the first unit echo image and the second unit echo image from the annotation data of the temporally continuous first unit echo image and second unit echo image. The method includes a function of extracting first annotation data and second annotation data, respectively, and a function of integrating the extracted first annotation data and second annotation data to generate annotation data of the entire school of fish.

According to the second and third aspects described above, effects similar to those of the first aspect described above are achieved.

As described above, according to the present invention, it is possible to provide a fish species learning device, a fish species learning method, and a program that can perform machine learning for fish species discrimination with higher accuracy.

The effects and significance of the present invention will become clearer from the following description of the embodiments. However, the embodiment shown below is merely one example of implementing the present invention, and the present invention is not limited to what is described in the embodiment below.

FIG. 1 is a diagram showing the configuration of a fish species discrimination system according to an embodiment. FIG. 2 is a block diagram showing the configuration of the fish species discrimination system according to the embodiment. FIG. 3 is a diagram schematically showing fish species discrimination processing using a neural network according to the embodiment. FIG. 4 is a diagram schematically showing a display example of an echo image including a fish species discrimination result according to the embodiment. FIG. 5 is a diagram schematically showing a method of generating annotation data performed in a terminal device according to the embodiment. FIG. 6 is a diagram illustrating the storage format of annotation data in the storage unit of the server according to the embodiment. FIG. 7 is a flowchart illustrating annotation data integration processing executed by the control unit of the server according to the embodiment. FIG. 8 is a flowchart illustrating a process for extracting annotation data of a school of fish spanning two unit echo images, according to the embodiment. FIGS. 9A and 9B are diagrams each schematically showing the state of a school of fish near the boundary between the first unit echo image and the second unit echo image, according to the embodiment. FIGS. 10(a) and 10(b) are diagrams showing a method for integrating annotation data of a first school of fish and a second school of fish, respectively, according to the embodiment. FIG. 11A is a diagram schematically showing the state of a school of fish near the boundary between the first unit echo image and the second unit echo image, according to modification example 1. FIG. 11(b) is a diagram illustrating a method of integrating annotation data of the first fish school and the second fish school according to modification example 1. FIG. 12 is a flowchart illustrating a fish species setting process executed in an annotation data integration process according to modification example 2.

Embodiments of the present invention will be described below with reference to the drawings.

In the following embodiments, the server 20 corresponds to a "fish species learning device" described in the claims. However, the "fish species learning device" according to the present invention is not necessarily limited to the server 20; for example, other devices such as the terminal device 50 can perform the functions of the "fish species learning device" according to the present invention. May be executed.

FIG. 1 is a diagram showing the configuration of a fish species discrimination system 1. As shown in FIG.

The fish species discrimination system 1 includes an underwater detection device 10 and a server 20. The underwater detection device 10 is a fish finder installed on the boat 2. Underwater detection device 10 is capable of communicating with server 20 via external communication network 30 (for example, the Internet) and base station 40 . The underwater detection device 10 and the server 20 each hold address information for mutual communication. The respective address information is set in the underwater detection device 10 and the server 20 at the time of initial setting.

The underwater detection device 10 includes a transducer 11 and a control unit 12. The transducer 11 is installed on the bottom of the ship 2, and the control unit 12 is installed in the ship's wheelhouse or the like. The transducer 11 and the control unit 12 are connected by a signal cable (not shown). The transducer 11 includes an ultrasonic transducer for transmitting and receiving waves. The transducer 11 transmits an ultrasonic wave 3 (transmitted wave) toward the seabed 4 using an ultrasonic transducer under control from the control unit 12, and receives the reflected wave. The transducer 11 transmits a received signal based on the received reflected wave to the control unit 12 .

Control unit 12 processes the received signals to generate echo data indicating the echo intensity at each depth. The control unit 12 arranges the echo intensities at each depth based on the echo data in time series, and generates one screen worth of echo images. The control unit 12 displays the generated echo image on the display section. The control unit 12 updates the echo image every time ultrasonic waves are transmitted and received. The user can grasp the existence and position of the fish school 5 by referring to the echo image.

Furthermore, the control unit 12 transmits the generated echo data to the server 20 at any time. The server 20 stores the received echo data and generates echo images similar to the control unit 12. The server 20 uses a machine learning model (machine learning algorithm) to calculate a predicted probability (probability of being that fish species) for each fish species for a school of fish included in the echo image.

The server 20 obtains the fish species discrimination result for the fish school based on the predicted probability for each fish species calculated by the machine learning model. The server 20 transmits the thus obtained discrimination result of the fish species, together with the range (depth, time) of the school of fish to be discriminated, to the underwater detection device 10 that receives the echo data.

Based on the received discrimination results and the range (depth, time) of the fish school, the underwater detection device 10 displays the fish species discrimination results superimposed on the corresponding range on the echo image. Thereby, the user can confirm the fish species of each fish school on the echo image, and can smoothly proceed with catching the desired fish.

Further, the server 20 acquires annotation data (teacher data) for learning the machine learning model from the plurality of terminal devices 50 via the external communication network 30. That is, the server 20 distributes the echo data received from the underwater detection device 10 to any one of the plurality of terminal devices 50. The terminal device 50 is used to generate annotation data from received echo data.

That is, the terminal device 50 is owned by an operator, such as an expert, who generates annotation data. The operator causes the terminal device 50 to display echo images based on the received echo data, and sets the fish species for each school of fish included in these echo images. The set fish species is associated with the range (depth, time) of the school of fish and the echo data of the range. The fish species, range of the fish school, and echo data thus associated constitute annotation data for the fish school. The annotation data is transmitted to the server 20 together with the identification information of the echo image in which the annotation data was generated.

The server 20 uses the received annotation data to perform learning on the machine learning model. This increases the accuracy of fish species discrimination of the machine learning model.

Note that here, annotation data is generated for the echo data provided by the server 20 to the terminal device 50, and the generated annotation data is returned to the server 20. However, the method of providing the annotation data to the server 20 is as follows. It is not limited to this. For example, echo data may be provided to the terminal device 50 from a device other than the server 20 to generate annotation data, and the generated annotation data may be provided to the server 20.

Further, although only one underwater detection device 10 is illustrated in FIG. 1, in reality, a large number of underwater detection devices 10 can communicate with the server 20 via the external communication network 30 and the nearest base station. . Further, the underwater detection device 10 that communicates with the server 20 includes, in addition to the one installed on the ship 2 as shown in FIG. 1, several types of underwater detection devices for different fishing methods, such as an underwater detection device installed in a fixed net. It can be done.

FIG. 2 is a block diagram showing the configuration of the fish species discrimination system 1. As shown in FIG.

The underwater detection device 10 includes a control section 101, a display section 102, an input section 103, a wave transmitting/receiving section 104, a signal processing section 105, a communication section 106, and a position detection section 107.

The control unit 101 is composed of a microcomputer, a memory, and the like. The control section 101 controls each section of the underwater detection device 10 according to a program stored in the memory.

The display unit 102 includes a monitor and displays a predetermined image under control from the control unit 101. The input unit 103 includes a trackball for moving a cursor on the image displayed on the display unit 102, operation keys, and the like, and outputs a signal to the control unit 101 in response to a user's operation. The display unit 102 and the input unit 103 may be integrally configured by a liquid crystal touch panel or the like.

The wave transmitter/receiver 104 includes the wave transmitter/receiver 11 shown in FIG. 105. The transmitting circuit and receiving circuit are included in the control unit 12 of FIG.

The wave transmitting/receiving unit 104 transmits a transmission wave (ultrasonic wave) of a predetermined frequency according to control from the control unit 101. The wave transmitting/receiving unit 104 receives the reflected wave of the transmitted wave and outputs a received signal. The receiving circuit extracts a received signal of the frequency of the transmitted wave and outputs it to the signal processing section 105.

The signal processing unit 105 generates echo data indicating the intensity of reflected waves according to the depth from the received signal input from the wave transmitting/receiving unit 104, and outputs the generated echo data to the control unit 101. The elapsed time from the timing of transmitting the transmission wave corresponds to the depth. Here, the intensity of the reflected wave attenuates as the depth increases. Therefore, in order to be able to quantitatively handle the echo data regardless of the difference in depth, the signal processing unit 105 corrects the intensity of the reflected waves that attenuate according to the elapsed time, and controls the echo data with the intensity corrected. It outputs to section 101.

The control unit 101 generates an echo image based on the received echo data and displays it on the display unit 102. The control unit 101 generates a row of images in the depth direction in which the echo intensity at each depth is expressed in gradations using a color scale, from the echo data. The control unit 101 integrates images in each column from the current time to a predetermined time ago in the time direction to generate an echo image for one screen.

The communication unit 106 is a communication module that can communicate wirelessly with the base station 40. The position detection unit 107 is equipped with a GPS and detects the position of the underwater detection device 10. The position detection unit 107 outputs the detected position information to the control unit 101.

As described with reference to FIG. 1, the control unit 101 transmits echo data to the server 20 via the communication unit 106 at any time. Further, the control unit 101 receives the fish species determination result from the server 20 via the communication unit 106. The control unit 101 further transmits the position information detected by the position detection unit 107 to the server 20.

As shown in FIG. 2, in addition to the underwater detection device 10, a large number of underwater detection devices 10a, 10b, ... can communicate with the server 20 via the external communication network 30 and the nearest base stations 40a, 40b, ... It is. As mentioned above, the underwater detection device 10 that communicates with the server 20 can be used for several types of underwater detection devices for different fishing methods, such as the one installed on the ship 2 as shown in FIG. Includes equipment. The basic configuration of other underwater detection devices is similar to the underwater detection device 10 of FIG. 2.

The server 20 includes a control section 201, a storage section 202, and a communication section 203. The control unit 201 is composed of a CPU and the like. The storage unit 202 is composed of ROM, RAM, hard disk, and the like. The storage unit 202 stores a program for fish species discrimination and a program for machine learning. The control unit 201 controls each unit using a program stored in the storage unit 202. The communication unit 203 communicates with the underwater detection device 10 via the external communication network 30 and the base station 40 under the control of the control unit 201 . Further, the server 20 communicates with a plurality of terminal devices 50 via the external communication network 30.

The terminal device 50 is a device capable of inputting and outputting information, such as a personal computer or a tablet computer. The terminal device 50 includes a control section 501, a storage section 502, a display section 503, an input section 504, and a communication section 505.

The control unit 501 is composed of a CPU and the like. The storage unit 502 is composed of ROM, RAM, hard disk, and the like. The storage unit 502 stores a program for generating annotation data. The control unit 501 controls each unit using a program stored in the storage unit 502. The display unit 503 is configured with a liquid crystal monitor or the like, and displays a predetermined image under control from the control unit 501. The input unit 504 includes input means such as a mouse and a keyboard. The communication unit 505 communicates with the server 20 under control from the control unit 501.

FIG. 3 is a diagram schematically showing fish species discrimination processing using a neural network.

In this embodiment, machine learning using a neural network is applied as machine learning. For example, a deep learning neural network that combines neurons in multiple stages is applied. However, the applied machine learning is not limited to this, and other machine learning such as support vector machines and decision trees may be applied.

The control unit 201 of the server 20 extracts the range (depth, time) of the school of fish from one screen worth of echo data to be processed. On the echo image, areas where the echo intensity is greater than a predetermined threshold and where the echo intensities are connected are extracted as a school of fish, and a rectangular range formed by the maximum time width and maximum depth width of this school of fish is extracted as a school of fish. Extracted as a range. Regarding the method for extracting fish schools, the description of International Publication No. 2019/003759, which the applicant previously filed, can be incorporated by reference.

The control unit 201 applies the echo data in the range of the extracted fish school to the input 301a of the machine learning model (machine learning algorithm using a neural network) 301 in FIG.

The output 301b of the machine learning model 301 is assigned items of fish species such as sardines, horse mackerel, and mackerel. When the echo data of the range of the fish school is applied to the input 301a of the machine learning model 301, from each item of the output 301b of the machine learning model 301, the probability that the fish species of the fish school is the fish species of each item (predicted probability) is output. In the example of FIG. 3, a predicted probability of 85% is output from the sardine item, a 70% predicted probability is output from the mackerel item, and a 10% predicted probability is output from the horse mackerel item.

The predicted probability of each item is checked against the output condition 302. As the output condition 302, for example, a condition is applied to output the fish species of the item whose predicted probability is equal to or higher than a predetermined lower limit value and whose rank is the highest (highest) as the discrimination result 303. The lower limit value is set to prevent fish species with low accuracy from being output as a discrimination result. In the example of FIG. 3, sardines with a predicted probability of 85% are output as the fish species discrimination result 303.

Machine learning for the machine learning model 301 is performed by sequentially applying a series of annotation data (teacher data) to the input 301a and output 301b of the machine learning model 301. That is, the echo data of a school of fish included in one annotation data is input to the input 301a of the machine learning model 301, and the output 301b of the machine learning model 301 includes 100% in the item corresponding to the fish species included in this annotation data. % is set, other items are set to 0%, and machine learning is performed.

In addition to echo data of fish schools, the input 301a of the machine learning model 301 may include other information that can be used for fish species identification, such as the location where the echo data was obtained and sea state data at that location. good. The annotation data may further include these other information.

FIG. 4 is a diagram schematically showing a display example of the echo image P1 including the fish species discrimination results. For convenience, in FIG. 4, lines in the depth direction are attached only to portions where the echo intensity is high.

Upon receiving the determination result and the range of the fish school (depth, time) from the server 20, the control unit 201 of the underwater detection device 10 selects an area on the echo image P1 corresponding to the depth width and time width corresponding to the received range of the fish school. A frame-shaped marker M0 indicating the range of the school of fish is displayed. Further, the control unit 201 further displays a label L0 indicating the received fish species determination result around this marker M0.

In the example of FIG. 4, markers M0 are displayed for fish schools F1 to F8 based on the discrimination results received from the server 20 and the range of fish schools (depth, time), and furthermore, fish species are displayed around these markers M0. A label L0 indicating the determination result is displayed. The current date and time is displayed near the upper left corner of the echo image P1.

In the example of FIG. 4, no marker or label is displayed for fish school F9 because the fish species discrimination result was not output according to the machine learning model 301 and output condition 302 of FIG. 3. This may occur, for example, when the predicted probability of the fish school F9 by the machine learning model 301 does not satisfy the output condition 302. In such a case, the determination result for this school of fish is not transmitted from the server 20 to the underwater detection device 10, so the result of the determination of the fish species will not be displayed, as in the case of the school of fish F9 in FIG. 4.

FIG. 5 is a diagram schematically showing a method of generating annotation data performed in the terminal device 50.

Generation of annotation data is performed by displaying an echo image based on the echo data to be processed on the display unit 503 in the terminal device 50 .

An operator such as an expert specifies the range of the school of fish on the displayed echo image. For example, the range of a school of fish is specified by a rectangular range. The operator specifies the range of the school of fish by operating the input unit 504 and specifying the diagonal vertices of the rectangle. Further, the operator operates the input unit 504 to set the fish species for each school of fish. For example, in response to the designation of the fish school range, a list of fish species selection candidates is displayed on the display section 503, and the operator selects the fish species to be set from this list. As a result, the fish species is set within the range of the school of fish.

Here, the annotation data is generated for each unit echo image, as shown in FIG. That is, a series of echo images for a predetermined time period is divided into a plurality of echo images with a time width corresponding to one frame, and a plurality of unit echo images from which annotation data is generated are generated. Among the boundaries B1 and B2 of the unit echo images in the time direction, the boundary B1 on the newer side matches the boundary B2 on the older side of the next unit echo image.

An operator such as an expert specifies the range of the fish school for each unit echo image, and further sets the fish species for the specified fish school. As a result, annotation data is generated in which the range of the school of fish, the species of fish within the range of the school of fish, and the image data (echo data) included in the range of the school of fish are associated.

In the example of FIG. 5, the ranges of six fish schools (ranges indicated by rectangular broken lines) are specified by the annotation process An for the unit echo image Pn, and the fish species is set in the range of each fish school. Further, by annotation processing An+1 for unit echo image Pn+1, ranges of five fish schools (ranges indicated by rectangular broken lines) are specified, and fish species are set in the range of each fish school. Therefore, from the unit echo image Pn, for each of the six schools of fish, the range of the school of fish, the fish species within the range of the school of fish, and the image data (echo data) included in the range of the school of fish are associated. Annotation data is generated. Furthermore, from the unit echo image Pn+1, for each of the five fish schools, the range of the fish school, the fish species within the range of the fish school, and the image data (echo data) included in the range of the fish school are associated. Annotation data is generated.

The annotation data generated in this way is transmitted from the terminal device 50 to the server 20 and stored in the storage unit 202 of the server 20.

FIG. 6 is a diagram showing a storage format of annotation data in the storage unit 202 of the server 20.

The storage unit 202 stores unit echo data and annotation data in association with echo IDs and image IDs.

The echo ID is identification information for identifying a series of echo images (echo data) for the above-mentioned predetermined time period. The echo ID is uniquely set by the server 20. The image ID is identification information for identifying a unit echo image. The image ID is, for example, a number assigned to each unit echo image in order from oldest to oldest. The unit echo data is one frame worth of echo data corresponding to each unit echo image. The annotation data is annotation data generated from each unit echo image.

When the control unit 201 of the server 20 newly receives echo data for a predetermined period of time to be subjected to annotation processing from the underwater detection device 10 or the like, it attaches an echo ID to this echo data. Furthermore, the control unit 201 divides this echo data into time widths of one frame from the beginning on the older side, and generates unit echo data corresponding to each unit echo image. Then, the control unit 201 attaches an image ID to each unit echo data, and stores the unit echo data in the storage unit 202 in association with the image ID. As a result, the echo ID, image ID, and unit echo data portions of the data structure shown in FIG. 6 are configured.

Thereafter, the control unit 201 transmits each unit echo data to the terminal device 50 along with the echo ID and image ID. The control unit 501 of the terminal device 50 stores the received data in the storage unit 502, and provides the data for annotation processing by the operator. When the operator performs an annotation processing operation, the control unit 501 causes the display unit 503 to display a unit echo image based on the unit echo data to be processed. The operator performs annotation processing on the unit echo image to be processed by the processing shown in FIG. As a result, annotation data for the unit echo image is generated.

When the annotation processing for all unit echo data received from the server 20 is thus completed, the control unit 501 associates the annotation data generated for each unit echo image with the image ID of each unit echo image, and It is sent to the server 20 along with the ID. The control unit 201 of the server 20 stores the received annotation data in the storage unit 202 in association with the image ID of FIG. As shown in FIG. 5, each image ID is associated with annotation data corresponding to the number of fish schools included in the unit echo image. In this way, by storing the annotation data in the storage unit 202, the entire data structure shown in FIG. 6 is constructed.

By the way, the unit echo image shown in FIG. 5 does not necessarily include the entire school of fish, and one school of fish may straddle two temporally consecutive unit echo images.

For example, in the example of FIG. 5, the school of sardines at the top right of the unit echo image Pn is considered to be connected to the school of sardines at the top left of the next unit echo image Pn+1. However, in the above generation method, a school of fish is specified for each unit echo image, so a school of sardines that straddles the unit echo image Pn and the unit echo image Pn+1 is divided into a portion of the unit echo image Pn and a portion of the unit echo image Pn+1. It is divided into two parts and specified. Then, annotation data will be generated individually for each divided fish school portion.

However, in the machine learning described above, characteristics arising from the entire school of fish, such as echo tailing and distribution unique to fish species, can affect the accuracy of machine learning. For this reason, as described above, when annotation data generated for a part of a school of fish is used for machine learning, the accuracy of machine learning will decrease.

In order to solve such a problem, in this embodiment, annotation data of a school of fish that spans between these unit echo images is extracted from annotation data of two temporally consecutive unit echo images, and Annotation data for the school of fish is generated by integrating the annotation data. That is, in the example of FIG. 5, the annotation data of the school of sardines at the top right of the unit echo image Pn and the annotation data of the school of sardines at the top left of the unit echo image Pn+1 are integrated to create annotation data for the entire school of sardines. is generated.

FIG. 7 is a flowchart showing the annotation data integration process executed by the control unit 201 of the server 20.

The control unit 201 refers to the annotation data of two temporally consecutive unit echo images among the unit echo images (unit echo data) having the same echo ID (S11). Next, the control unit 201 extracts annotation data of a school of fish spanning these two unit echo images from the referenced annotation data (S12). Then, the control unit 201 integrates the extracted annotation data to generate annotation data for the entire school of fish spanning the two unit echo images, and stores the generated annotation data in the storage unit 202 (S13). The control unit 201 executes the process shown in FIG. 7 on all unit echo images (unit echo data) having the same echo ID.

The annotation data of FIG. 6 generated by the terminal device 50 and the annotation data generated by the process of FIG. 7 are used for learning the machine learning model 301 of FIG. 3. At this time, the two annotation data integrated by the process of FIG. 7 may be excluded from the annotation data used for machine learning. For example, the two annotation data integrated by the process in FIG. 7 may be deleted from the data structure in FIG. 6 constructed in the storage unit 202.

FIG. 8 is a flowchart showing an example of the process in step S12 of FIG.

For convenience, steps S11 and S13 in FIG. 7 are included in FIG. Steps S101 to S105 in FIG. 8 correspond to step S12 in FIG.

In step S11, the control unit 201 refers to the annotation data of two temporally consecutive unit echo images and grasps the range of the fish school in each annotation data. Next, the control unit 201 determines whether or not a school of fish exists near the right boundary, that is, near the boundary B1 in FIG. 5, in the first unit echo image that is older among these two unit echo images. is determined (S101). If a school of fish does not exist near the boundary B1 on the right side of the first unit echo (S101: NO), the control unit 201 ends the process of FIG. 8. If a school of fish exists near the boundary B1 on the right side of the first unit echo (S101: YES), the control unit 201 specifies this school of fish as the first school of fish to be integrated (S102).

Next, the control unit 201 determines whether or not there is a school of fish near the left boundary, that is, near the boundary B2 in FIG. is determined (S103). If there is no school of fish near the left boundary B2 of the second unit echo (S103: NO), the control unit 201 ends the process of FIG. 8. If a school of fish exists near the left boundary B2 of the second unit echo (S103: YES), the control unit 201 specifies this school of fish as a second school of fish to be integrated (S104).

Next, the control unit 201 determines whether the depth range of the first school of fish and the depth range of the second school of fish substantially match (S105). More specifically, the control unit 201 determines whether the matching rate between the depth range of the first school of fish and the depth range of the second school of fish is greater than or equal to a predetermined threshold Th1. If the determination in step S105 is NO, the control unit 201 ends the process in FIG. 8 without performing the process in step S13.

If the determination in step S105 is YES, the control unit 201 determines that the first fish school and the second fish school constitute a fish school spanning the first unit echo image and the second unit echo image. The annotation data of the entire fish school are integrated to generate annotation data of the entire fish school (S13). Thereby, the control unit 201 ends the processing on the annotation data of these two unit echo images.

The control unit 201 executes the process shown in FIG. 8 on the annotation data of the next two temporally consecutive unit echo images. In this way, the control unit 201 performs the process shown in FIG. 8 on the annotation data of all unit echo images assigned the same echo ID.

FIGS. 9A and 9B are diagrams schematically showing the state of a school of fish near the boundary between the first unit echo image and the second unit echo image.

FIG. 9A shows the upper right part of the unit echo image Pn (first unit echo image) and the upper left part of the unit echo image Pn+1 (second unit echo image) in FIG. 5. Fa and Fb are sardine fish schools, and Ra and Rb are ranges (time range, depth range) of the fish schools included in the annotation data. As described above, the ranges Ra and Rb of the school of fish are set to be rectangular.

In step S101 of FIG. 8, it is determined whether a school of fish exists within a predetermined time range Ta from the boundary B1 of the first unit echo image Pn, and in step S103, it is determined whether a school of fish exists within a predetermined time range Ta from the boundary B1 of the second unit echo image Pn+1. It is determined whether a school of fish exists in the time range Tb. The time ranges Ta and Tb are set to approximately the maximum value of the gap in the time axis direction on the echo image that can occur in one school of fish. The time ranges Ta and Tb may be set to the same time width.

In the example of FIG. 9A, since the fish school Fa (fish school range Ra) exists within the time range Ta, the determination in step S101 of FIG. 8 is YES, and the fish school Fa is specified as the first fish school. Furthermore, in the example of FIG. 9(a), since the fish school Fb (fish school range Rb) exists within the time range Tb, the determination in step S103 of FIG. 8 is YES, and the fish school Fb is identified as the second fish school. .

On the other hand, in the example of FIG. 9(b), since the fish school Fa (fish school range Ra) does not exist within the time range Ta, the determination in step S101 of FIG. 8 is NO, and the fish school Fa is the first fish school. Not specified. Further, in the example of FIG. 9(b), since the fish school Fb (fish school range Rb) does not exist within the time range Tb, the determination in step S103 of FIG. 8 is NO, and the fish school Fb is not specified as the second fish school.

Further, in step S105 in FIG. 8, a determination is made using the following equation (1).

As shown in FIG. 9(a), Amax and Amin are the maximum depth and minimum depth, respectively, of the first fish school Fa (fish school range Ra) in the first unit echo image Pn, and Bmax and Bmin are, respectively, These are the maximum depth and minimum depth of the second fish school Fb (fish school range Rb) in the second unit echo image Pn+1.

In other words, the denominator on the left side of equation (1) is the depth range from the smaller of Amin and Bmin (shallower) to the larger of Amax and Bmax (deeper), and in the example of FIG. 9(a), from Amin to The depth range is up to Bmax. In addition, the numerator on the left side of equation (1) is the depth range from the larger one of Amin and Bmin (deeper one) to the smaller one of Amax and Bmax (shallower one), and in the example of FIG. 9(a), from Bmin to The depth range is up to Amax. The matching rate between the depth range of the first fish school Fa and the depth range of the second fish school Fb is calculated from the left side of equation (1). In step S105 of FIG. 8, it is determined whether the depth range of the first fish school Fa and the depth range of the second fish school Fb substantially match, depending on whether the matching rate is equal to or greater than the threshold Th1. The threshold Th1 may be a statistically set default value, or may be arbitrarily set by the administrator of the server 20.

FIGS. 10A and 10B are diagrams showing a method for integrating the annotation data of the first fish school and the second fish school in step S13 of FIG. 8.

In step S13 of FIG. 8, the control unit 201 matches the depth ranges of the fish school range Ra of the first fish school Fa and the fish school range Rb of the second fish school Fb. That is, as shown in FIG. 10A, the control unit 201 matches the smaller one (Amax) of the maximum depths Amax and Bmax with the larger one (Amax), and matches the larger one (Bmin) of the minimum depths Amin and Bmin. The depth ranges of the fish school ranges Ra and Rb are corrected to match the smaller one (Amin). Then, as shown in FIG. 10(b), the control unit 201 integrates the annotation data of the corrected ranges Ra and Rb of the fish school to generate annotation data of one fish school consisting of the fish schools Fa and Fb. do.

The generated annotation data is composed of a fish school range Rab that integrates the corrected fish school ranges Ra and Rb, echo data included in this fish school range Rab, and the fish species of the fish school (in this case, sardines). be done. The fish school range Rab is also rectangular like the fish school ranges Ra and Rb, and the fish schools Fa and Fb are included in this rectangular range. As described above, the control unit 201 of the server 20 uses the integrated annotation data for learning the machine learning model 301.

In addition, in FIGS. 9(a) to 10(b), fish schools in the upper right part of the first unit echo image Pn and the upper left part of the second unit echo image Pn+1 are illustrated, but the first unit echo image If there are a plurality of schools of fish extending over the second unit echo image and the second unit echo image, the same processing as described above is performed for each school of fish, and a plurality of pieces of integrated annotation data are generated.

Furthermore, when one school of fish straddles three or more unit echo images, the above processing is sequentially performed on adjacent unit echo images among these unit echo images. As a result, the annotation data of parts of the fish school divided into three or more unit echo images are integrated, and annotation data for one fish school is generated.

<Effects of embodiment>
According to the embodiment, the following effects can be achieved.

As shown in FIGS. 7 to 10(b), the annotation data (first annotation data and second annotation data) of the fish school spanning between the first unit echo image Pn and the second unit echo image Pn+1 are integrated. Then, annotation data for the entire school of fish is generated. Therefore, the annotation data for the entire school of fish can be used for machine learning, so the accuracy of machine learning for fish species discrimination can be improved.

As shown in FIG. 9A, the control unit 201 converts the fish school Fa extending to the vicinity of the boundary B1 (first boundary) on the second unit echo image Pn+1 side on the first unit echo image Pn into the first fish group. , the fish school Fb extending to the vicinity of the boundary B2 (second boundary) on the first unit echo image Pn side on the second unit echo image Pn+1 is specified as the second fish school, and the fish school Fa (first fish school) is identified as the second fish school. When the range Ra and the range Rb of the fish school Fb (second fish school) approximately match, the annotation data of these fish schools Fa and Fb (first fish school, second fish school) are combined with the annotation data to be integrated (first annotation data). and second annotation data). Thereby, fish schools Fa and Fb (first fish school, second fish school) that have a high probability of forming one fish school can be smoothly identified in the first unit echo image Pn and the second unit echo image Pn+1. Therefore, by integrating the annotation data of these fish schools, annotation data of one entire fish school can be generated with high accuracy.

As shown in FIG. 9(a), the control unit 201 extracts a school of fish Fa that is at least partially included in a predetermined time range Ta from the boundary B1 (first boundary) as a school of fish to be integrated (first school of fish). Then, a school of fish Fb that is at least partially included in a predetermined time range Tb from the boundary B2 (second boundary) is extracted as a school of fish to be integrated (second school of fish). As a result, even if the distribution of fish is sparse near the boundaries B1 and B2, the first and second schools of fish constituting the school of fish spanning the first unit echo image Pn and the second unit echo image Pn+1 can be separated from each other. It can be specified without missing anything in the 1-unit echo image Pn and the 2nd unit echo image Pn+1. Therefore, annotation data for one entire school of fish can be appropriately generated.

<Change example 1>
The present invention is not limited to the above embodiments, and the embodiments of the present invention can be modified in various ways in addition to the above configurations.

For example, in the above embodiment, the range (time range, depth range) of the fish school that makes up the annotation data is rectangular, but the range of the fish school that makes up the annotation data may have a shape other than a rectangle. .

For example, as shown in FIG. 11(a), the ranges Ra and Rb of the fish schools may be shaped along the outer edges of the fish schools Fa and Fb. In this case, as described above, areas on the echo image where the echo intensity is equal to or higher than a predetermined threshold and where the echo intensities are connected are extracted as fish schools Fa and Fb, and the outer edges of the extracted fish schools Fa and Fb are extracted as fish schools Fa and Fb. The range along is extracted as the ranges Ra and Rb of the school of fish. When the operator specifies the position of a school of fish on the echo image, the control unit 501 of the terminal device 50 displays the range of the school of fish extracted by the above processing for the school of fish including the specified position, superimposed on the echo image. .

In this case, as in the first embodiment, in step S101 of FIG. 8, it is determined whether a school of fish exists in the time range Ta from the boundary B1, and in step S103, it is determined whether a school of fish exists in the time range Tb from the boundary B2. is determined. Thereby, as shown in FIG. 11(a), the fish schools Fa and Fb are specified as the first fish school and the second fish school, respectively.

In this case, in step S105 of FIG. 11(a), it is determined whether the depth range of the fish school Fa (first fish school) and the depth range of the fish school Fb (second fish school) almost match. The determination is made based on the matching rate between the depth range of the fish school Fa (first fish school) near the first boundary) and the depth range of the fish school Fb (second fish school) near the boundary B2 (second boundary).

That is, for the fish school Fa, the maximum depth Amax and the minimum depth Amin are obtained for the part included in the time range Ta of the range Ra of the fish school, and for the fish school Fb, the maximum depth Amax and the minimum depth Amin are obtained for the part included in the time range Tb of the range Rb of the fish school. A maximum depth Bmax and a minimum depth Bmin are obtained for the portion that is to be stored.

The control unit 201 applies the maximum depths Amax, Bmax and minimum depths Amin, Bmin thus obtained to the above equation (1) to perform the determination in step S105. In this way, in the vicinity of boundary B1 (first boundary) and boundary B2 (second boundary), the depth range of fish school Fa (first fish school) and the depth range of fish school Fb (second fish school) approximately match. By determining, it is possible to accurately determine whether these fish schools Fa and Fb (first fish school, second fish school) are continuous with each other. Therefore, the annotation data for one entire fish school can be generated with high accuracy from the annotation data for the fish schools Fa and Fb (first fish school, second fish school).

In this case, in step S13 of FIG. 11(a), as shown in FIG. 11(b), the positions of the ranges Ra and Rb of the fish schools from which the maximum depths Amax and Bmax have been obtained are connected to each other, and the minimum depths Amin and Bmin are determined. By connecting the positions of the respective acquired fish school ranges Ra and Rb, the fish school range Rab of the entire fish school can be set. Thereby, the range Rab of the fish school that constitutes the annotation data of the entire fish school can be set smoothly and appropriately.

However, the method of setting the range Rab of the fish school is not limited to this. For example, by connecting the positions of ranges Ra and Rb of fish schools slightly closer to the boundaries B1 and B2 than the positions of ranges Ra and Rb of fish schools for which maximum depths Amax and Bmax have been obtained, respectively, a school of fish for which minimum depths Amin and Bmin have been obtained, respectively. The range Rab of the fish school may be set by connecting the positions of the ranges Ra and Rb of the fish school that are slightly closer to the boundaries B1 and B2 than the positions of the ranges Ra and Rb. In this case as well, the range Rab of the fish school that constitutes the annotation data of the entire fish school can be set smoothly and appropriately.

<Change example 2>
In the embodiment described above, it was assumed that the fish species that constitute the annotation data of the first fish school and the fish species that constitute the annotation data of the second fish school are the same. However, it may be difficult for an operator to determine the species of fish based on images of schools of fish. For this reason, it is possible that the fish species of the annotation data respectively set for the first fish school and the second fish school that are specified as forming one fish school are different from each other.

In modification example 2, in such a case, the fish species is set for the integrated annotation data based on predetermined conditions.

FIG. 12 is a flowchart showing the fish species setting process executed in the annotation data integration process in step S13 of FIG.

The control unit 201 determines whether the fish species in the annotation data of the first fish school and the second fish school to be integrated match (S201). If the two match (S201: YES), the control unit 201 sets the matched fish species as the fish species of the fish group after integration (S202). On the other hand, if the two do not match, the control unit 201 sets the fish species of the integrated fish school according to predetermined setting conditions (S203).

The setting conditions in step S203 may include, for example, setting the annotation data to the fish species of the newer one of the annotation data (the first annotation data and the second annotation data) of the first school of fish and the second school of fish.

For example, in the example of FIG. 11(a), if the fish species constituting the annotation data of fish school Fa (first fish school) and fish school Fb (second fish school) are different, the control unit 201 selects the newer unit echo. The fish species of the annotation data of the fish school Fb acquired from the image Pn+1 is set as the fish species for the range Rab of the fish school after integration.

Note that the setting conditions in step S203 are not limited to the condition that the new annotation data is set to the fish species as described above (the first condition), but the setting condition is that the fish species with high reliability is set to the annotation data after the integration. Other conditions may be used as long as they can be set in the data.

For example, among the annotation data of the first and second fish schools to be integrated (first annotation data, second annotation data), the fish species of the annotation data that has a larger range of fish schools can be added to the annotation data of the integrated fish school. The condition of setting the fish species (second condition) may be included in the setting conditions of step S203.

Alternatively, the condition (third condition) that the fish species obtained by processing the integrated annotation data with the fish species discrimination machine learning model 301 is set as the fish species of the annotation data of the fish school after integration is set in step S203. may be included in the setting conditions.

In this case, for example, when the fish species cannot be set due to the second condition (for example, when the range of the first fish school and the range of the second fish school are approximately the same), the first condition or the third condition is applied. , fish species may be set. Alternatively, among the fish species obtained according to the first condition to the third condition, the fish species with the largest number of mutually matching fish species may be set as the fish species of the annotation data of the fish school after integration. The setting conditions are not limited to the first to third conditions, and may further include other conditions.

In this way, when the fish species composing the annotation data of fish school Fa (first fish school) and fish school Fb (second fish school) are different, by performing the process of step S203, highly reliable fish species can be selected. , can be set in the annotation data after integration. Therefore, the accuracy of machine learning can be improved.

Note that if the determination in step S201 is NO, the process of integrating the annotation data of the first school of fish and the second school of fish may be canceled. In this case, the control unit 201 may exclude both the annotation data of the first fish school and the second fish school from the annotation data used for machine learning of the machine learning model 301, or may exclude one of these annotation data (e.g. , the older one or the one with a smaller fish school range) may be excluded from the annotation data used for machine learning in the machine learning model 301.

<Other change examples>
In the above embodiment and modified examples 1 and 2, the processes shown in FIGS. 7 and 8 were performed in the control unit 201 of the server 20, but these processes were performed in a device other than the server 20, and were integrated. Later annotation data may be provided to server 20. For example, the processes in FIGS. 7 and 8 are performed by the control unit 501 of the terminal device 50 after the operator generates annotation data, and the annotation data generated by the operator and the annotation data generated by the processes in FIGS. 7 and 8 are may be transmitted by the control unit 501 to the server 20. In this case, the terminal device 50 corresponds to a fish species learning device described in the claims.

In addition, in the above embodiment and modifications 1 and 2, it is determined by the above formula (1) whether the depth ranges of the first fish school and the second fish school substantially match. It is not limited to. For example, in FIG. 9A, the difference between the first depth width between the maximum depth Amax and the minimum depth Amin and the second depth width between the maximum depth Bmax and the minimum depth Bmin is smaller than a predetermined threshold value. , and when the condition that the difference between the intermediate depth of the first depth width and the intermediate depth of the second depth width is smaller than a predetermined threshold is satisfied, the depth ranges of the first fish school and the second fish school substantially match. , if this condition is not met, it may be determined that the depth ranges of these schools of fish do not match. In the case of FIG. 11(a) as well, it may be determined whether the depth ranges of the first fish school and the second fish school substantially match using a similar determination method.

Further, the method for extracting annotation data of a school of fish across the first unit echo image and the second unit echo image is not limited to the method shown in steps S101 to S105 in FIG. 8. For example, it may be determined whether the first school of fish and the second school of fish constitute one school of fish, based on the continuity of the distribution of the echo data of the first school of fish and the echo data of the second school of fish.

Furthermore, in the above embodiment and modifications 1 and 2, it was assumed that the frequency of the transmission wave transmitted by the wave transmitting/receiving section 104 is one, but the transmission waves of two different frequencies are transmitted and received. The wave may be transmitted from the section 104. In this case, echo data based on received signals of each frequency may be applied to the machine learning model 301 for each school of fish to determine the fish species of each school of fish. By transmitting and receiving waves at two types of frequencies in this way, the machine learning model 301 can discriminate fish species with higher accuracy. For example, the echo intensity of each frequency varies depending on the presence or absence of fins. Therefore, by referring to the difference in the intensity of echoes from a school of fish, the species of fish in the school of fish can be determined with high accuracy.

In this case, the annotation data generated from the unit echo image may be generated for each frequency for each school of fish. Further, the annotation data integration process shown in FIGS. 7 and 8 may be performed individually for each frequency of echo data (unit echo image).

Further, in the above embodiment, the fish species discrimination process using the machine learning model 301 was performed on the server 20 side, but this discrimination process may be performed on the underwater detection device 10 side. In this case, the machine learning model 301 updated with the annotation data is transmitted to the underwater detection device 10 at any time and stored in the underwater detection device 10. The control unit 101 of the underwater detection device 10 executes fish species discrimination using the machine learning model 301, similar to the control unit 201 of the server 20, based on the echo data acquired by the wave transmitting/receiving unit 104 and the signal processing unit 105. Then, the discrimination results are displayed on the echo image.

Further, in the above embodiment, the annotation data is generated in the terminal device 50, but the device that generates the annotation data is not limited to this. For example, a user of the underwater detection device 10 inputs a school of fish on an echo image and its fish species from his own fishing results, and the echo data of this school of fish and the fish species of the school of fish are sent to the server 20 as annotation data. Good too.

Further, in the above embodiment, the underwater detection device 10 is a fish finder, but the underwater detection device 10 may be a device other than a fish finder, such as a sonar.

In addition, the embodiments of the present invention can be modified in various ways within the scope of the claims.

20 Server (machine learning device)
201 Control unit 202 Storage unit 301 Machine learning model Pn, Pn+1 Unit echo image (first unit echo image, second unit echo image)
Fa, Fb Fish school (1st fish school, 2nd fish school)
B1, B2 boundary (first boundary, second boundary)
Ta, Tb time range

Claims (8)

a storage unit that stores annotation data for each unit echo image;
comprising a control unit;
The control unit includes:
From the annotation data of the temporally continuous first unit echo image and second unit echo image, first annotation data and second annotation data of a school of fish spanning between the first unit echo image and the second unit echo image are obtained. Extract each data,
integrating the extracted first annotation data and the second annotation data to generate annotation data for the entire school of fish;
A fish species learning device characterized by: The fish species learning device according to claim 1,
The control unit includes:
A first school of fish extending to near the first boundary on the second unit echo image side on the first unit echo image, and extending to near the second boundary on the first unit echo image side on the second unit echo image. Identify the second school of fish that is extending,
When the depth range of the first fish school and the depth range of the second fish school substantially match, the annotation data of the first fish school and the second fish school are used as the first annotation data and the second annotation data. Extract each
A fish species learning device characterized by: The fish species learning device according to claim 2,
The control unit determines whether the depth range of the first school of fish and the depth range of the second school of fish substantially match. Determining based on the matching rate with the depth range of the second school of fish,
A fish species learning device characterized by: The fish species learning device according to claim 2,
The control unit extracts a school of fish that is at least partially included in a predetermined time range from the first boundary as the first school of fish, and extracts a school of fish that is at least partially included in a predetermined time range from the second boundary as the first school of fish. Extract as the second school of fish,
A fish species learning device characterized by: The fish species learning device according to claim 1,
When the fish species included in the first annotation data and the fish species included in the second annotation data are different, the control unit sets the fish species of the annotation data of the fish school based on predetermined setting conditions. ,
A fish species learning device characterized by: The fish species learning device according to claim 5,
The setting conditions are:
setting the newer one of the first annotation data and the second annotation data to the fish species;
A fish species learning device characterized by: From the annotation data of the temporally continuous first unit echo image and second unit echo image, first annotation data and second annotation data of a school of fish spanning between the first unit echo image and the second unit echo image are obtained. Extract each data,
integrating the extracted first annotation data and the second annotation data to generate annotation data for the entire school of fish;
A method of learning about fish species. to the computer,
From the annotation data of the temporally continuous first unit echo image and second unit echo image, first annotation data and second annotation data of a school of fish spanning between the first unit echo image and the second unit echo image are obtained. A function to extract each data,
A program that executes a function of integrating the extracted first annotation data and the second annotation data to generate annotation data for the entire school of fish.

Images