WO2024018989A1

WO2024018989A1 - Fish species discrimination system, server, fish species discrimination method and program

Info

Publication number: WO2024018989A1
Application number: PCT/JP2023/025896
Authority: WO
Inventors: Akinori KASAI; Yuta HIRABAYASHI; Masashi Muragaki
Original assignee: Furuno Electric Co., Ltd.
Priority date: 2022-07-20
Filing date: 2023-07-13
Publication date: 2024-01-25
Also published as: JP2024013658A

Abstract

To provide a fish species discrimination system, a server, a fish species discrimination method, and a program that can improve the accuracy of fish species discrimination results using a machine learning model. To solve this problem, a fish species discrimination system 1 comprises an echo data acquisition unit 110 that acquires echo data in water, a storage unit 202, and a control unit 201. The storage unit 202 stores a machine learning model that outputs prediction probabilities for each fish species based on the echo data, know-how information for each fish species related to the fish school set by the user, and a know-how model that modifies prediction probabilities for each fish species based on the know-how information. The control unit 201 determines the fish species of the fish school based on the modification result obtained by modifying the prediction probabilities for each fish species acquired by the machine learning model by the know-how model.

Description

FISH SPECIES DISCRIMINATION SYSTEM, SERVER, FISH SPECIES DISCRIMINATION METHOD AND PROGRAM

The present disclosure relates to a fish species discrimination system that uses a machine learning model (machine learning algorithm) for fish species discrimination, a server and a fish species discrimination method, and a program that makes a computer execute a function for fish species discrimination using a machine learning model.

Background

Fish finders have been known to detect fish schools in the water. In this type of fish-finder, ultrasonic waves are sent underwater and the reflected waves are received. Echo data is generated according to the intensity of the reflected waves received, and an echo image is displayed based on the echo data generated. The user can confirm the fish school from the echo image, and the capture of the fish school can proceed smoothly.

In this case, it is preferable that the fish species can be further discriminated for the fish school on the echo image. Thus, the user can efficiently catch the fish of the fish species the user wants.

To determine the fish species, for example, a machine learning model can be used. In this case, learning is performed on the machine learning model using the echo data output from the fish finder as input data and the fish species of the fish school on the echo data as teacher data, and a learned model is generated. The fish species of the fish school on the echo data (teacher data) is input by the user based on the actual catch, for example. Patent Document 1 below describes a fish species estimation system for this species.

The Patent publication is JP2019-200175.

Summary

In the method described above, a unique machine learning model can be generated for each user. However, in the method described above, the number of teacher data is limited because the teacher data is input by the user based on the actual fish catch. Therefore, it is difficult to accurately determine the fish species by machine learning models.

One possible solution to this problem is to aggregate standard data generated by experts for machine learning and use the aggregated standard data as teacher data for learning machine learning models.

However, the characteristics of fish in the water, such as swimming depth, swimming speed, and school style, may vary depending on the region. In addition, the sea conditions suitable for fish of each fish species, such as water temperature, salinity, and current speed, may also vary depending on the region, and furthermore, the time and place at which fish of each fish species can be caught may also vary depending on the region. For this reason, if the species of fish in a school is determined by a machine learning model based on standard data as described above, it is possible that highly accurate discrimination results may not be obtained in each region.

In view of such issues, the present disclosure aims to provide a fish species determination system, a server, a fish species determination method and a program that can improve the accuracy of the fish species discrimination results using a machine learning model.

The first embodiment of the disclosure relates to a fish species discrimination system. The fish species discrimination system according to this embodiment is provided with an echo data acquisition unit for acquiring echo data in water, a storage unit, and a control unit. The storage unit stores a machine learning model for outputting prediction probabilities for each fish species based on the echo data, the know-how information for each fish species about the fish school set by the user, and a know-how model for modifying the prediction probabilities for each fish species based on the know-how information. The control unit determines the fish species of the fish school based on the modification result of modifying the prediction probabilities for each fish species acquired by the machine learning model by the know-how model.

According to the fish species determination system according to this embodiment, the prediction probabilities of each fish species by the machine learning model are modified based on the know-how model using the know-how information from the user, so that the prediction probabilities of each fish species after the modification are more easily adapted to the region to which the user belongs or the fishing area. Therefore, the accuracy of the fish species discrimination results can be improved.

In the fish species determination system according to this embodiment, the know-how information may include information about the characteristics of the fish of the species in the water.

The characteristics of each fish species in the water, such as swimming depth, swimming speed and school style, can vary depending on the region. Therefore, when the know-how information includes the characteristics of each fish species in the water, the modified results of the prediction probabilities of each fish species by the machine learning model modified by the know-how model can be approximated to the prediction probabilities according to the regionality of each fish species. Therefore, the accuracy of the fish species discrimination results can be enhanced.

In the fish species discrimination system according to this embodiment, the know-how information may include information on the sea conditions suitable for the fish of the said fish species.

The sea conditions suitable for the fish of each fish species, for example, water temperature, salinity concentration, current speed, etc. may vary depending on the region. Thus, by including the sea conditions suitable for the fish of each fish species in the know-how information, the modified results obtained by modifying the prediction probabilities of each fish species by the machine learning model by the know-how model can be approximated to the prediction probabilities according to the regionality of each fish species. Therefore, the accuracy of the fish species discrimination results can be enhanced.

In the fish species discrimination system according to this embodiment, the know-how information may include information about at least one of the time and place of catching the fish of the species.

The time and place of catching the fish of each species may vary depending on the region. Thus, by including at least one of the time and place of catching the fish of each species in the know-how information, the modified results of the prediction probabilities of each species by the machine learning model modified by the know-how model can be approximated to the prediction probabilities according to the regionality of the fish of each species. Therefore, the accuracy of the fish species discrimination results can be enhanced.

In the fish species discrimination system according to this embodiment, the control unit can be configured to change the degree of modification of the prediction probability in the know-how model based on feedback information indicating the content of the user’s modification to the discrimination results.

According to this configuration, since the degree of modification of the prediction probability in the know-how model can be changed at any time according to the content of the user’s modification to the discrimination results, even if, for example, the user accidentally sets inappropriate know-how information, the prediction probability after the modification can be made close to the prediction probability corresponding to the actual fish species.

In the fish species discrimination system, the control unit may be configured to change the degree of modification of the prediction probability in the know-how model based on the learning progress of the machine learning.

According to this configuration, for example, when the learning progress of the machine learning model is low and the accuracy of the prediction probability of each fish species is low, the degree of modification of the prediction probability in the know-how model is enhanced, and then, as the learning progress of the machine learning model increases and the accuracy of the prediction probability of each fish species increases, the degree of modification of the prediction probability in the know-how model is reduced. Thus, when the machine learning model and the know-how model work in a complementary manner, the accuracy of the prediction probability after the modification can be efficiently enhanced and the accuracy of the fish species discrimination result can be enhanced.

The fish species discrimination system according to this embodiment may be provided with an underwater detection device for detecting a school of fish in the water and a server capable of communicating with the underwater detection device. Here, the echo data acquisition unit may be arranged in the underwater detection device and the storage unit and the control unit may be arranged in the server.

According to this configuration, the construction of machine learning models and know-how models necessary for fish species discrimination and the processing of fish species discrimination using these models are mainly executed in the server. Therefore, the fish species discrimination process can be efficiently performed while reducing the burden on the underwater detection device installed on the ship or vessel and so on.

The second embodiment of the present disclosure relates to a server capable of communicating with an underwater detection device for detecting a school of fish in the water. The server according to this embodiment includes a storage unit and a control unit. The storage unit stores a machine learning model for outputting prediction probabilities for each fish species based on echo data received from the underwater detection device, the know-how information for each fish species related to the fish school set by the user, and the know-how model for modifying the prediction probabilities for each fish species by the machine learning model based on the know-how information. The control unit determines the fish species of the fish school based on the modification result of modifying the prediction probabilities for each fish species acquired by the machine learning model by the know-how model.

The third embodiment of the present disclosure relates to a fish species determination method. The fish species determination method according to this embodiment involves acquiring echo data in water, calculating prediction probabilities for each fish species based on the echo data by a machine learning model, storing know-how information from the user about the fish school for each fish species, modifying the prediction probabilities for each fish species obtained by the machine learning model based on the know-how information, and discriminating the fish species of the fish school based on the prediction probabilities for each fish species modified by the know-how model.

The fourth embodiment of the present disclosure relates to a program that causes a computer to perform a predetermined function. The program in this embodiment includes a function for calculating the prediction probability for each fish species based on the echo data acquired from the water by a machine learning model, a function for storing the know-how information from the user about the fish school for each fish species, a function for modifying the prediction probability for each fish species acquired by the machine learning model by a know-how model for modifying the prediction probability for each fish species based on the know-how information, and a function for discriminating the fish species of the fish school based on the prediction probability for each fish species modified by the know-how model.

According to the second to third embodiments, the same effect as the first embodiment is achieved.
Effect of the disclosure

As described above, the present disclosure can provide a fish species determination system, a server, a fish species determination method, and a program that can enhance the accuracy of the fish species discrimination result using a machine learning model.

The effect or significance of the present disclosure will be further clarified by the following description of the embodiment. However, the following embodiment is only one example when implementing the present disclosure, and the present disclosure is not in any way limited to those described in the following embodiment.

Fig. 1 is a diagram showing the configuration of a fish species discrimination system according to an embodiment. Fig. 2 is a block diagram showing the configuration of a fish species discrimination system according to an embodiment. Fig. 3 is a diagram showing the management status of various information in the storage unit of a server according to an embodiment. Fig. 4 (a) to (c) are diagrams showing the structure of individual data according to an embodiment, respectively. Fig. 5 is a diagram schematically showing the fish species discrimination processing by a neural network according to an embodiment. Fig. 6 is a diagram showing the input screen of know-how information according to an embodiment. Fig. 7 is a diagram illustrating an example of processing in which the prediction probability of each fish species by the machine learning model is modified by the know-how model according to an embodiment. Fig. 8 is a flowchart illustrating the fish species discrimination processing according to an embodiment. Fig. 9 is a diagram schematically showing a display example of an echo image including the result of identifying the fish species according to the embodiment. Fig. 10 (a) is a flowchart showing the transmission processing of feedback information executed by the control unit of the underwater detection device according to the embodiment. Fig. 10 (b) is a flowchart showing the reception processing of feedback information executed by the control unit of the server according to the embodiment. Fig. 11 is a diagram schematically showing a screen for receiving modification of fish species from the user according to the embodiment. Fig. 12 (a) is a flowchart showing modification processing of the know-how model performed by the control unit of the server according to the embodiment. Fig. 12 (b) is a flowchart showing an example of processing in Step S312 of Fig. 12 (a) according to the embodiment. Fig. 13 (a) and (b) are diagrams showing an example of modification of the know-how model according to the embodiment. Fig. 14 (a) is a flowchart showing the process of modifying the degree of modification of the prediction probability in the know-how model according to the learning progress of machine learning according to the modification example 1. Fig. 14 (b) is a diagram showing the structure of the table used for the process of Fig. 14 (a) according to the modification example 1. Fig. 15 is a diagram showing an example of modification of the know-how information according to the modification example 2.

DETAILED DESCRIPTION

Fig. 1 is a diagram showing the configuration of the fish species discrimination system 1.

The fish species discrimination system 1 comprises an underwater detection device 10 and a server 20. The underwater detection device 10 is a fish finder installed in the ship ( vessel) 2. The underwater detection device 10 can communicate with the server 20 via an external communication network 30 (For example, the Internet) and a base station 40. The underwater detection device 10 and the server 20 each hold address information for communicating with each other. The respective address information is set in the underwater detection device 10 and the server 20 at the initial setting.

The underwater detector 10 comprises a transmitter and receiver 11 and a control unit 12. The transmitter and receiver 11 is installed on the bottom of the ship 2, and the control unit 12 is installed in the wheelhouse or the like of the ship. The transmitter and receiver 11 and the control unit 12 are connected by a signal cable (not shown). The transmitter and receiver 11 comprises an ultrasonic vibrator for transmitting and receiving waves. The transmitter and receiver 11 transmits an ultrasonic wave 3 (transmitted wave) toward the seabed 4 and receives its reflected wave by the ultrasonic vibrator in response to control from the control unit 12. The transmitter and receiver 11 transmits a received signal based on the received reflected wave to the control unit 12.

The control unit 12 processes the received signal to generate echo data indicating the echo intensity at each depth. The control unit 12 arranges the echo intensity at each depth based on the echo data in time series to generate an echo image for one screen. The control unit 12 displays the generated echo screen on the display unit. The control unit 12 updates the echo screen for each ultrasonic wave transmitted and received. The user can grasp the presence and location of the fish school 5 by referring to the echo screen.

Furthermore, the control unit 12 transmits the generated echo data to the server 20 at any time. The server 20 stores the received echo data and generates an echo image similar to that of the control unit 12. The server 20 calculates a prediction probability (probability of being the fish species) for each fish species included in the echo image by a machine learning model. Furthermore, the server 20 modifies the calculated prediction probability for each fish species by a know-how model based on the know-how information set by the user.

Here, the know-how information is information for each fish species with respect to a school of fish and may include the characteristics of each fish species in the water (Swimming depth, swimming speed, school style, etc.), information on the sea conditions suitable for each fish species (water temperature, salinity, current speed, etc.), or the time and place of catching the fish of each fish species.

The server 20 modifies the prediction probabilities for each fish species calculated by the machine learning model by the know-how model based on the know-how information set by the user, calculates the modified prediction probabilities for each fish species, and acquires the discrimination result of the fish species for the fish school based on the modified prediction probabilities of each fish species. The server 20 transmits the obtained discrimination result of the fish species, together with the range (Depth, Time) of the fish school to be judged, to the underwater detection device 10 at the receiver of the echo data.

Based on the received discrimination result and the range (Depth, Time) of the fish school, the underwater detection device 10 superimposes the discrimination result of the fish species on the corresponding range on the echo image. Thus, the user can confirm the fish species of each fish school on the echo image and smoothly advance the fishing of the desired fish.

When the discrimination result of the fish species provided by the server 20 is different from the fish species of the fish actually caught, the user sends feedback information for modifying the fish species to the server 20. For example, after the user finishes fishing for a day, the user performs an operation to acquire echo data for a predetermined time period of the day from the server 20 via the input unit of the underwater detection device 10. Accordingly, the server 20 transmits the echo data for the designated day and time period to the underwater detection device 10 along with the result of the discrimination of the fish species (including the range of the fish school to be determined). Based on the received echo data and the result of the discrimination, the underwater detection device 10 causes the display unit to display the echo image including the result of the discrimination.

Through the input unit, the user performs an operation to modify the discrimination result of the fish species displayed on the echo image to the fish species of the fish he has captured. With this, the underwater detection device 10 transmits feedback information including the user’s modification of the discrimination result and the range (Depth, Time) of the fish school corresponding to the discrimination result to the server 20. The server 20 changes the degree of modification of the prediction probability in the know-how model described above based on the received feedback information. This makes the know-how model appropriate to reflect the actual fishing results of the user.

It should be noted that only one underwater detection device 10 is shown in Fig. 1, but in fact, many underwater detection devices 10 can communicate with the server 20 via the external communication network 30 and the nearest base station. In addition, the underwater detection devices 10 that communicate with the server 20 include those installed in the ship 2 as shown in Fig. 1, as well as several types of underwater detection devices with different fishing methods, such as underwater detection devices installed in fixed nets.

Fig. 2 is a block diagram showing the configuration of the fish species discrimination system 1.

An underwater detection device 10 is provided with a control unit 101, a display unit 102, an input unit 103, a transmitting and receiving wave unit 104, a signal processing unit 105, a communication unit 106, and a position detection unit 107. The transmitting and receiving wave unit 104 and the signal processing unit 105 constitute an echo data acquisition unit 110 for acquiring underwater echo data.

The control unit 101 consists of a microcomputer, a memory, etc. The control unit 101 controls each part of the underwater detector 10 according to a program stored in the memory. This program includes functions related to the reception and display of fish species discrimination results described below and functions related to the reception and transmission of feedback and know-how information.

A display unit 102 comprises a monitor and displays a prescribed image by control from a control unit 101. The input unit 103 comprises a trackball for moving a cursor on the image displayed on the display unit 102, an operation key, etc., and outputs a signal corresponding to the operation from the user to the control unit 101. The display unit 102 and the input unit 103 may be integrally constituted by a liquid crystal touch panel or the like.

The transmitting and receiving wave unit 104 is provided with the transmitter and receiver 11 shown in Fig. 1, a transmitting circuit for supplying a transmission signal to the transmitter and receiver 11, and a receiving circuit for processing the received signal output from the transmitter and receiver 11 and outputting it to the signal processing unit 105. The transmitting circuit and receiving circuit are included in the control unit 12 of Fig. 1.

The transmitting and receiving wave unit 104 transmits the transmitted wave (ultrasonic wave) according to the control from the control unit 101. Here, in 1 sequence, two kinds of transmitted waves with different frequencies are transmitted. The transmitting and receiving wave unit 104 receives the reflected waves of the transmitted waves of each transmitted frequency and outputs the received signal. The receiving circuit extracts the received signal of the frequency of each transmitted wave and outputs it to the signal processing unit 105.

The reason why the transmitted and received waves are carried out at two different frequencies is to perform the fish species discrimination described later with higher accuracy. For example, the presence or absence of an airbladder causes a difference in the echo intensity at each frequency. Therefore, by referring to the difference in the echo intensity from a school of fish, the species of fish in that school can be accurately identified.

The signal processing unit 105 generates echo data indicating the intensity of the reflected wave according to the depth from the received signal of each frequency input from the transmitting and receiving wave unit 104, and outputs the generated 2 kinds of echo data to the control unit 101. The elapsed time from the timing of transmitting the transmitted wave of each frequency corresponds to the depth. Here, the intensity of the reflected wave attenuates as the depth increases. Therefore, the signal processing unit 105 amends the intensity of the reflected wave that attenuates according to the elapsed time and outputs the amended intensity echo data to the control unit 101 so that echo data can be quantitatively handled regardless of the difference in depth.

The control unit 101 generates an echo image based on the received echo data and causes the display unit 102 to display it. The control unit 101 generates echo data using echo data corresponding to one of the frequencies. The user may be able to switch as appropriate which frequency of echo data is used to generate the echo image. The control unit 101 generates one row of images in the depth direction, in which the echo intensity at each depth is expressed in gradation by a color scale, from the echo data. The control unit 101 integrates the images in each row from the present time to a predetermined time ago in the time direction to generate an echo image for one screen.

In the following explanation, when we refer to echo data (including echo data included in feedback information), we mean echo data of two different frequencies unless otherwise specified.

The communication unit 106 is a communication module capable of wireless communication with the base station 40. The position detection unit 107 comprises GPS and detects the position of the underwater detection device 10. The position detection unit 107 outputs the detected position information to the control unit 101.

As described with reference to Fig. 1, the control unit 101 transmits echo data, feedback information and know-how information to the server 20 via the communication unit 106 at any time. The control unit 101 also receives the fish species discrimination result from the server 20 via the communication unit 106. The control unit 101 further transmits the position information detected by the position detection unit 107 to the server 20.

As shown in Fig. 2, in addition to the underwater detection device 10, a number of underwater detection devices 10a, 10b and ... can communicate with the server 20 via the external communication network 30 and the

nearest base stations

40a, 40b and .... As described above, the underwater detection device 10 communicating with the server 20 is shown in Fig. 1. In addition to the one installed in the ship 2 as shown, several types of underwater detection devices with different fishing methods are included, such as underwater detection devices installed in fixed nets. The basic configuration of the other underwater detection devices is similar to that of the underwater detection device 10 in Fig. 2.

However, the underwater detection device installed in the fixed net may consist of an offshore unit installed in the fixed net and a terminal capable of communicating with the offshore unit via an external communication network for the user to remotely monitor the status of fish in the fixed net. The echo data acquired by the offshore unit is transmitted to the terminal via an external communication network. This causes the terminal to display the echo image. The terminal may be a personal computer, or a mobile terminal owned by the user such as a mobile phone or tablet.

In this case, the terminal may transmit the echo data to the server 20, or the offshore unit may also transmit the echo data to the server 20 in parallel with the transmission of the echo data to the terminal. Feedback information and know-how information may also be input via the terminal and transmitted from the terminal to the server 20. The fish species discrimination result may be transmitted directly from the server 20 to the terminal without going through the offshore unit. Also, the echo data may be transmitted to the terminal from the server 20. That is, the server 20 may receive the echo data from the offshore unit and transmit the received echo data to the terminal.

The server 20 includes a control unit 201, a storage unit 202, and a communication unit 203. The control unit 201 is composed of a CPU or the like. The storage unit 202 is composed of a ROM, a RAM, a hard disk or the like. The storage unit 202 stores a program for fish species discrimination. The control unit 201 controls each part according to the program stored in the storage unit 202. The communication unit 203 communicates with the underwater detector 10 through the external communication network 30 and the base station 40 under the control of the control unit 201.

The control unit 201 generates a know-how model applied to each underwater detector 10 by the above program. The control unit 201 also stores the echo data, feedback information and know-how information received from each underwater detector 10 in the storage unit 202 in association with each underwater detector 10. The control unit 201 uses the know-how information 1received from each underwater detector 10 to generate a know-how model applicable to the underwater detector 10, and further uses the feedback information received from each underwater detector 10 to update the know-how model applicable to the underwater detector 10.

It should be noted that the echo data transmitted from each underwater detector 10 to the server 20 may be decimated to a predetermined particle size to reduce communication traffic and the capacity load of the server 20. In this case, the server 20 performs fish species determination and machine learning using the echo data with the decimation amended by interpolation processing. Alternatively, fish species determination and machine learning may be performed using the echo data in the decimated state. However, for more accurate fish species determination and machine learning, it is preferable to use the echo data amended for decimation by interpolation processing for fish species determination and machine learning.

In addition, in order to quantitatively handle the echo data received from each underwater detection device 10, the server 20 may perform fish species determination and machine learning by amending the echo data received from each underwater detection device 10 based on underwater acoustic theory and taking into account the characteristics (For example, sensitivity, amplification, etc.) of the underwater detection device 10 and the transmitter and receiver 11. Thus, fish species determination by the machine learning model and machine learning for the machine learning model can be performed with higher accuracy.

Fig. 3 is a diagram showing the management status of various information in the storage unit 202 of the server 20.

The storage unit 202 stores standard data 301, a machine learning model 302, oceanographic data 303,

individual data

311 and 321, and know-

how models

312 and 322.

The standard data 301 is standard teacher data for machine learning. The standard data 301 is data that combines echo data for the range (Depth, Time) of a school of fish with the species of fish in that school. The standard data 301 is sequentially generated by experts and registered by managers. This gradually increases the number of data in the standard data.

The machine learning model 302 is a machine learning model generated by machine learning using the standard data 301. In response to the update of the standard data 301, machine learning is performed on the machine learning model 302 and the machine learning model 302 is updated. Thus, the learning progress of the machine learning model 302 is enhanced.

In this embodiment, machine learning using a neural network is applied as machine learning. For example, a neural network with deep learning combining neurons in multiple stages is applied. However, the machine learning applied is not limited to this, and other machine learning such as support vector machines and decision trees may be applied.

The oceanographic data 303 is data on oceanographic conditions such as water temperature, salinity concentration and tidal current speed. The oceanographic data 303 is detected by detectors installed on buoys at sea and transmitted periodically to the server 20 by radio communication from each detector. The detectors comprises GPS and transmit the position information detected by GPS along with the oceanographic data to the server 20. The server 20 stores oceanographic data in a storage unit 202 for each position of the buoys (detectors). If the underwater detector 10 comprises a detector for acquiring oceanographic data, oceanographic data may be transmitted from the underwater detector 10 to the server 20 along with position information.

Individual data

311 and 321 are data acquired from each user's underwater detector 10. As described above, the know-

how models

312 and 322 are models for modifying the prediction probabilities for each fish species calculated by the machine learning model 302 based on the know-how information for each user.

In Fig. 3, the individual data 311 and the know-how model 312 are for the user U1, and the individual data 321 and the know-how model 322 are for the user U2. The individual data, the know-how information and the know-how model are similarly managed for each user other than the users U1 and U2.

Figs. 4 (a) to 4 (c) show the structure of the individual data.

As shown in Fig. 4 (a) - (c), various individual data are managed in association with user IDs. The user ID is information for identifying a user (underwater detector 10). For example, the product code of underwater detector 10 may be used as the user ID, or a randomly assigned code may be used as the user ID. The user ID is transmitted and received from time to time when information is transmitted and received between underwater detector 10 and server 20.

Fig. 4 (a) shows individual data related to echo data. The underwater detector 10 sequentially transmits echo data obtained by 1 sequence of transmitted and received waves to the server 20 along with the date and time of acquisition. In the storage unit 202 of the server 20, the acquisition start date and time and the acquisition end date and time of the echo data, and a group of echo data acquired during that time and the acquisition date and time are stored corresponding to the user ID.

Furthermore, the result of fish species discrimination obtained by the machine learning model from a group of echo data from the start date and time to the end date and time is further matched to the echo data of each group. When multiple results of fish species discrimination are obtained, the multiple results of fish species discrimination are matched to the group of echo data. Each result of fish species discrimination consists of a range (Depth, Time) of fish schools and a discrimination result (fish species).

Fig. 4 (b) shows individual data on feedback information. Here, the feedback information acquired from the underwater detection device 10 corresponding to the user ID is stored in time series. Fig. 4 (c) shows the individual data related to the know-how information. Here, the know-how information acquired from the underwater detection device 10 corresponding to the relevant user ID is stored in time series.

Fig. 5 is a diagram schematically showing the fish species discrimination processing by the neural network.

The control unit 201 of the server 20 extracts the range (Depth, Time) of the fish school from the echo data of 1 screen to be processed. The range of the fish school is extracted as the range where the echo intensity is above a predetermined threshold and is connected to the echo intensity on the echo image. For the extraction method of the range of the fish school, the description in the applicant's earlier application, International Publication No. 2019/003759, may be incorporated by reference.

The control unit 201 applies the echo data of the extracted range of the fish school to the input 302a of the machine learning model (machine learning algorithm by neural network) 302 in Fig. 5.

Items of fish species such as sardine, horse mackerel and mackerel are assigned to the output 302b of the machine learning model 302. When echo data of a range of fish schools are applied to the input 302a of the machine learning model 302, the probability (prediction probability) that the fish species of the fish school is the fish species of each item is output from each item of the output 302b of the machine learning model 302. In the example in Fig. 5, the prediction probability of 85% is output from the mackerel item, the prediction probability of 70% is output from the sardine item, and the prediction probability of 10% is output from the tuna item.

The predicted probabilities for each item are modified by the know-how model 312. As described above, the know-how model 312 is a model (algorithm) that modifies the predicted probabilities for each fish species (item) based on the know-how information of the relevant user. In the example in Fig. 5, the predicted probabilities for the mackerel item were modified from 85% to 25%, the predicted probabilities for the sardine item were modified from 70% to 94%, and the predicted probabilities for the tuna item were modified from 10% to 1%.

The modified predicted probabilities for each item are checked against the output condition 401. For the output condition 401, for example, a condition is applied to output as the discrimination result 402 the fish species of the item whose modified predicted probabilities are equal to or higher than a predetermined lower limit and have the highest ranking (highest). The lower limit is set to prevent the fish species with low accuracy from being output as the discrimination result. In the example of Fig. 5, a sardine with a modified prediction probability of 94% is output as the fish species discrimination result 402.

Machine learning for the machine learning model 302 is performed by sequentially applying a series of teacher data to inputs 302a and outputs 302b of the machine learning model 302. That is, the echo data of the fish school included in one teacher data is input to inputs 302a of the machine learning model 302, and the items corresponding to the fish species included in this teacher data are set to 100% in outputs 302b of the machine learning model 302, and the other items are set to 0%, and the machine learning is performed.

The machine learning model 302 of Fig. 3 is generated by setting standard data 301 (Echo data for the range of fish schools, fish species) sequentially to inputs 302a and outputs 302b of the machine learning model 302 to perform machine learning.

Besides the echo data of the fish school, other information that can be used for fish species discrimination, such as the position where the echo data is obtained and oceanographic data of the position, may be input to inputs 302a of the machine learning model 302. In this case, the standard data may also further include these other information.

Fig. 6 shows an example of the configuration of the input screen 500 for know-how information.

The input screen 500 includes a fish species setting item 501, a know-how input area 502, a reliability input area 503, and a confirmation key 504. The input screen 500 is displayed on a display unit 102 in Fig. 2 and accepts input from a user via an input unit 103.

The fish species setting item 501 is an item for the user to set the fish species for which the know-how information is to be set. When the fish species setting item 501 is selected, a selection candidate fish species is displayed by a pull-down list directly under the fish species setting item 501. The user selects a fish species from the displayed fish species candidates for which he or she intends to input know-how information. Thus, the selected fish species is displayed in the fish species setting item 501. In the example of Fig. 6, a mackerel is shown selected.

The know-how input area 502 is an item for the user to input his/her own know-how information. The know-how input area 502 includes the title 502a indicating the types of know-how information and an input item 502b for the user to input these types of know-how information. Here, as the types of know-how, swimming depth, swimming speed, school style, proper water temperature, fishing time and fishing location are exemplified as the know-how information that can be input. However, the types of know-how information are not limited to these.

The swimming depth is the range of depth at which fish of the relevant species (Here, mackerel) swim, and the swimming speed is the average speed at which fish of the relevant species swim. The school style is the school style of fish of the relevant species (how they form a school of fish). The proper water temperature is the water temperature appropriate for the fish of the relevant species, and the time and place of fishing is the time and place where the fish of the relevant species are caught. The user appropriately inputs each kind of know-how information in the case of catching the fish of the relevant species in his/her own fishing ground into the input item 502b. When the input item 502b of the school method is selected, selection candidates are displayed by a pre-down list directly under the input item 502b. Fishing locations are set by longitude and latitude ranges.

The reliability level input area 503 is an area for inputting the reliability level (confidence level) of each know-how information. The reliability level input area 503 includes items for selecting whether the confidence level of the know-how information is high or average for each know-how information included in the know-how input area 502.

After setting the fish species in the fish species setting item 501, the user enters the items to be set in the input item 502b for each type of know-how information item displayed in the know-how input area 502. Furthermore, the user enters a reliability level (confidence level) for each type of know-how information entered by selecting either a high or normal selection item in the reliability level input area 503.

Thus, when the input to the fish species setting item 501, the know-how input area 502 and the reliability level input area 503 is completed, the user operates a confirmation key 504. Accordingly, the control unit 101 of the underwater detection device 10 transmits the know-how information and the confidence level input to the know-how input area 502 and the reliability level input area 503, respectively, to the server 20 together with the fish species set in the fish species setting item 501. The server 20 stores these received information in the storage unit 202 as individual data of the user in Fig. 3, as shown in Fig. 4 (c).

Fig. 7 is a diagram illustrating an example of processing in which the prediction probability of each fish species by the machine learning model is modified by the know-how model.

In this example, in the input screen 500 of Fig. 6, the know-how model is generated based on the know-how information whose reliability is set to high by the user. Since the user has set high reliability for the swimming depth of mackerel and sardine and the time of catching tuna, the know-how model is generated using these know-how information. Specifically, the know-how information used to generate the know-how model is as follows.

・Swimming depth of mackerel ... 40 ~ 120 m
・Swimming depth of sardines ... 0 ~ 30 m (shallower than 30 m)
・Catching season of tuna ... March to July

The reliability of these know-how information was 100% in the initial setup by the user, but the following changes were made by subsequent adjustments based on feedback information.

・Mackerel swimming depth ... 70%
・Sardine swimming depth ... 80%
・Tuna catching time ... 90%

The know-how model modifies the predicted probabilities of each fish species by the machine learning model by, for example, the following equation:

Adjusted prediction probability = 100% x {1- (1-Rp) x (1-Rn)} ... (1)

Prediction probability after modification = Prediction probability x (1 - Rn) ... (2)

In Equations (1) and (2) above, Rp is the prediction probability for the target fish species expressed in decimal, and Rn is the confidence level of the know-how information for the target fish species expressed in decimal. For example, in the example in Fig. 7, if the target fish species is mackerel, Rp is 0.85 and Rn is 0.7. If the target fish species is sardine, Rp is 0.7 and Rn is 0.8.

In the example in Fig. 7, the depth range of the fish school for species determination was 10 ~ 20 m. In contrast, the swimming depth of the mackerel in the know-how information is 40 ~ 120 m, so the swimming depth of the fish school does not satisfy the condition of the swimming depth of the mackerel in the know-how information. Therefore, the prediction probability (85%) of the mackerel by the machine learning model is modified by Equation (2) above. Thus, the modified prediction probability of the mackerel is calculated as 25.5% and rounded to the first decimal place of this value to 26%.

On the other hand, since the swimming depth of the sardine in the know-how information is 0 ~ 30 m, the swimming depth of the fish school for fish species determination satisfies the condition of the swimming depth of the sardine in the know-how information. Therefore, the prediction probability (70%) of the sardine by the machine learning model is modified according to the above equation (1). This gives a modified prediction probability of 94% for the sardine.

Also, because the catch time of tuna in the know-how information is from March to July, the catch time (October 2) of the group of fish subject to species determination does not satisfy the condition of the catch time of tuna in the know-how information. Therefore, the prediction probability (10%) of tuna by the machine learning model is modified according to the above equation (1). This gives a modified prediction probability of 1% for tuna.

Thus, in the modified prediction probability, sardines have the highest prediction probability. Therefore, sardines are obtained as the discrimination result 402 in Fig. 5.

Note that in the example in Fig. 7, there is only one kind of know-how information for each fish species, but when there is more than one kind of know-how information for each fish species, for example, the prediction probability after the modification is calculated by the following equation.

Prediction probability after the modification = sigma (result of calculation by Equation 1 or Equation 2 above)/Number of types of know-how information ... (3)

That is, for the target fish species, the prediction probability after the modification is calculated by Equation (1) or Equation (2) above for each type of know-how information, and the average value of all the calculated predicted probabilities is obtained as the prediction probability after the modification for the target fish species.

Note that the method for modifying the prediction probability in the know-how model is not limited to the methods shown in Equations (1) - (3) above, and other modification methods may be used as long as the user's know-how information is reflected in the modification result.

In the above method, only the know-how information set to high reliability in the input screen 500 of Fig. 6 is used for generating the know-how model, but the know-how information set to ordinary reliability in the input screen 500 may be used for generating the know-how model further. In this case, for example, the ordinary reliability at the time of setting is set to 50% and is changed by subsequent feedback information. Then, from all the know-how information of the high and ordinary reliability, for example, the modified predicted probabilities are calculated for each fish species according to the above equations (1), (2) and (3).

In addition, the reliability input area 503 may be omitted in the input screen 500 of Fig. 6. In this case, the know-how information input in the know-how input area 502 is, for example, all set to a reliability of 100% and changed by subsequent feedback information. Then, from all the know-how information set by the user, for example, the prediction probability after the modification is calculated for each fish species according to the above equation (3).

Fig. 8 is a flow chart showing the fish species discrimination process.

When the control unit 201 of the server 20 starts receiving the echo data from the underwater detector 10 (S101: YES), the control unit stores the received echo data in the storage unit 202 as individual data in Fig. 3 (S102). Further, the control unit 201 sequentially constructs an echo image from the received echo data and specifies the range (Depth, Time) of the fish school on the echo image. Then, the control unit 201 applies the echo data of the specified range of the fish school to the machine learning model to calculate the prediction probability for each fish species (S103). Furthermore, the control unit 201 modifies the calculated prediction probability by the know-how model of the underwater detection device 10 (user ID) as described above to acquire the modified prediction probability (S104). Then, the control unit 201 applies the modified prediction probability to the output condition 401 of Fig. 5 to determine the result of discriminating the species of the fish school (S105).

The control unit 201 transmits the result of discriminating the acquired species of fish, together with the range (Depth, Time) of the fish school for which the result of the discrimination was acquired, to the underwater detection device 10, and further stores this information in the storage unit 202 (S106).

Thereafter, the control unit 201 repeatedly executes the processing of steps S102 to S106 until the reception of echo data from the underwater detector 10 is terminated (S107: NO). As a result, each time the range (Depth, Time) of the fish school is newly identified from the echo image, the fish species of the fish school is determined by the machine learning model and the know-how model of the user. The result of the discrimination of the newly obtained fish school and the range (Depth, Time) of the fish school are transmitted to the underwater detection device 10 as needed and stored in the storage unit 202 of the server 20.

Thus, when the reception of the echo data from the underwater detection device 10 is finished (S107: YES), the control unit 201 terminates the processing of Fig. 8. Thus, the individual data of one row of Fig. 4 (a) is stored in the storage unit 202. As described above, the echo data column of Fig. 4 (a) holds all the echo data received from the underwater detector 10 in the processing of Fig. 8. In addition, the fish species discrimination results column of Fig. 4 (a) holds all the fish species discrimination results obtained by the processing of Fig. 8 along with the range (Depth, Time) of the fish school.

Fig. 9 is a diagram schematically showing a display example of echo image P1 including the result of fish species discrimination. For convenience, in Fig. 9, a line in the depth direction is attached only to the part where the echo intensity is high.

When the control unit 201 of the underwater detection device 10 receives the discrimination result and the range (Depth, Time) of the fish school from the server 20, a frame-shaped marker M0 indicating the range of the fish school is displayed in the area on the echo image P1 corresponding to the depth width and time width corresponding to the range of the received fish school. Furthermore, the control unit 201 further displays a label L0 indicating the discrimination result of the received fish school around this marker M0. In the example of Fig. 9, based on the discrimination result received from the server 20 and the range (Depth, Time) of the fish school, a marker M0 is displayed for the fish schools F1 to F8, and further, a label L0 indicating the discrimination result of the fish species is displayed around these markers M0. The current date and time is displayed near the upper left corner of the echo image P1.

Here, in the example of Fig. 9, the marker and label are not displayed in the fish school F9 because the result of fish species discrimination was not output according to the machine learning model 302, the know-how model 312 and the output condition 401 in Fig. 5. This could happen, for example, if the modified prediction probability of the fish school F9 by the machine learning model 302 and the know-how model 312 did not satisfy the output condition 401. For example, if the output condition is to output a fish species with a prediction probability of the first rank (highest) that is greater than or equal to a specified lower limit value, the discrimination result for this fish school will not be output if the prediction probability of the first rank among the modified predicted probabilities of each fish species by the machine learning model 302 and the know-how model 312 is less than this lower limit value. In such a case, the discrimination result is not transmitted from the server 20 to the underwater detection device 10 for this fish school, so that the discrimination result of the fish species is not displayed as in the case of the fish school F9 in Fig. 9.

Fig. 10 (a) is a flow chart showing the feedback information transmission processing performed by the control unit 101 of the underwater detection device 10. Fig. 10 (b) is a flow chart showing the feedback information reception processing performed by the control unit 201 of the server 20.

The user performs an operation (feedback operation) to transmit feedback information to the server 20 for modifying the fish species of the given fish school to the underwater detection device 10 via the input unit 103 of Fig. 2, for example, when the fish species discrimination result of the given fish school displayed in the echo image P1 differs from the fish species of the fish school he actually captured, or when he actually captured the fish school for which the fish species discrimination result was not displayed in the echo image P1 and grasped the fish species of the fish school. In this case, the user inputs the date and time range of the echo image including the fish school to be modified via the input unit 103, and further performs an operation to acquire the fish species discrimination result and echo data (Hereafter referred to as "historical information") of the range from the server 20.

Referring to Fig. 10 (a), when a feedback operation is input from the user via the input unit 103 (S201: YES), the control unit 101 of the underwater detector 10 sends a request to transmit history information including the date and time range input by the user to the server 20, and acquires the history information from the server 20 (S202). Referring to Fig. 10 (b), when the control unit 201 of the server 20 receives a request to transmit the history information transmitted in step S202 of Fig. 10 (a) (S301: YES), it extracts the history information (echo data and fish species discrimination result) of the date and time range included in the transmission request from the individual data of the underwater detector 10 of the transmission source of the transmission request, and transmits the extracted history information to the underwater detector 10 of the transmission source (S302).

Referring to Fig. 10 (a), when the control unit 101 of the underwater detector 10 receives the history information from the server 20 (S202), it causes the display unit 102 to display an echo screen based on the received history information, and accepts the modification of the fish species from the user (S203).

Fig. 11 is a diagram schematically showing a screen for accepting the modification of the fish species from the user in step S203 of Fig. 10 (a).

The control unit 101 of the underwater detection device 10 causes the display unit 102 to display the echo image and the discrimination result of the leading time zone in the time range specified by the user when requesting the transmission of history information. The user operates the scroll bar B0 via the input unit 103 to transition the echo image in the time direction and display a screen containing the echo image and the discrimination result of the desired time zone. Thus, the screen of the time zone in Fig. 11 is displayed.

In this screen, the user specifies the marker M0 of the fish school that he/she intends to modify via an input unit 103. In the screen of Fig. 11, the marker M0 of the fish school F5 with the discrimination result of mackerel is specified by the user. With this, the marker M0 of the fish school F5 is highlighted, and the selection candidate C0 of the fish species is displayed around this marker M0 with a scroll bar. The user manipulates the scroll bar of the selection candidate C0 to display the desired fish species, and then selects the fish species he intends to change. In the example of Fig. 9, Sea bream is selected as the fish species of the fish school F5.

In addition, when the user inputs the fish species for the fish school to which the discrimination result is not attached on this screen, the user designates the range of the fish school via the input unit 103. As shown in Fig. 7, the fish school F9 does not have the discrimination result of the fish species. When the user inputs the fish species of this fish school F9, the user specifies the range of the fish school F9 via the input unit 103. As a result, as shown in Fig. 11, a new marker M1 is displayed in the range of the specified fish school F9, and the selection candidate C0 of the fish species is displayed around this marker M1 with a scroll bar. The user manipulates the scroll bar of the selection candidate C0 to display the desired fish species, and then selects the fish species he or she intends to input. In the example of Fig. 9, tuna is selected as the fish species of the fish school F9.

Thus, after performing an operation to change or set the fish species, the user inputs an operation to confirm these operations via an input unit 103.

With reference to Fig. 10 (a), when a confirmation operation is input from the user (S204: YES), the control unit 101 transmits feedback information including the range of the fish school designated by the user in step S203 and the fish species input by the user for the fish school to the server 20 (S205). With this, the control unit 101 terminates the processing of Fig. 10 (a).

Referring to Fig. 10 (b), when the control unit 201 of the server 20 receives feedback information from the control unit 101 of the underwater detector 10 (S303: YES), it causes the storage unit 202 to store the received feedback information as individual data of the underwater detector 10 (S304). As a result, the feedback information of 1 line in Fig. 4 (b) is stored in the storage unit 202. With this, the control unit 201 terminates the processing shown in Fig. 10 (b).

Fig. 12 (a) is a flowchart showing the modification processing of the know-how model executed by the control unit 201 of the server 20.

When the control unit 201 receives feedback information from the underwater detector 10 (S401: YES), it modifies the know-how model corresponding to the underwater detector 10 based on the received know-how information (S402).

Fig. 12 (b) is a flow chart showing an example of the processing in step S402 of Fig. 12 (a).

The control unit 201 compares the information about the fish school included in the feedback information with the conditions of the know-how information of each fish species of the user, and extracts the know-how information including the information about the fish school in the conditions from the individual data of the user (S411). The information about the fish school includes the depth range, day and time range, position (Longitude, latitude) of the fish school, and oceanographic data of the position. In addition, the know-how information to be compared is the know-how information used to generate the know-how model, for example, in the example of Fig. 6, the know-how information for which high reliability is set.

Next, the control unit 201 compares the fish species of each extracted know-how information with the fish species after the modification made by the user to the fish school (S412). Then, in each extracted know-how information, the control unit 201 increases the reliability of the know-how information in which the fish species after the modification and the fish species match (S413), and decreases the reliability of the know-how information in which the fish species after the modification and the fish species do not match (S414).

The increase and decrease in the reliability in steps S413 and S414 are performed, for example, by changing the reliability by a predetermined value (e.g., 5%). However, the method of increasing and decreasing the reliability is not limited to this and may be, for example, by changing the reliability by a predetermined percentage. Also, the increase and decrease in the reliability need not be the same and, for example, the increase may be larger than the decrease. The upper limit of the reliability is 100%. The lower limit of the reliability may be set by default to a predetermined value (e.g., 50%) or it may be user configurable.

Figs. 13 (a) and (b) show examples of modification of the know-how model by the processing shown in Fig. 12 (b).

In this example, in step S411 of Fig. 12 (b), among the know-how information of mackerel and sea bream, swimming depth is extracted. That is, since the depth of the fish school for which the user modified the fish species was included in the swimming depth conditions of 40 ~ 120 m and 60 ~ 150 m for mackerel and sea bream , which is one of the know-how information of the user, these know-how information is extracted in step S411 of Fig. 12 (b).

In this example, the fish species is modified from mackerel to sea bream, for example, by the modification to the fish school F5 of Fig. 11. Therefore, as shown in Fig. 13 (a), the confidence level of the know-how information on the swimming depth of the extracted mackerel is reduced by 5% in Step S414 of Fig. 12 (b). Also, as shown in Fig. 13 (b), the confidence level of the know-how information on the swimming depth of the extracted sea mackerel is increased by 5% in Step S413 of Fig. 12 (b).

By modifying the reliability of each modified know-how information, the know-how model of the user is modified. In the next fish species determination process, the prediction probability of each fish species by the machine learning model is modified using the modified know-how model, and the result of fish species discrimination is acquired by the modified prediction probability.

According to the embodiment, the following effects can be achieved.

As explained with reference to Figs. 5 and 7, the prediction probabilities of each fish species by the machine learning model are modified based on the know-how model using the know-how information from the user, so that the modified prediction probabilities of each fish species are more easily adapted to the region to which the user belongs and the fishing area. Therefore, the accuracy of the fish species discrimination results can be enhanced.

As shown in Fig. 6, the know-how information includes information about the characteristics of each fish species in the water (Here, swimming depth, swimming speed and school style). The characteristics of each fish species in the water, such as swimming depth, swimming speed and school style, can vary from region to region. Therefore, by including the characteristics of each fish species in the water in this way, the modified results of the prediction probabilities of each fish species by the machine learning model modified by the know-how model can be approximated to the prediction probabilities according to the regionality of each fish species. Therefore, the accuracy of the fish species discrimination results can be enhanced.

As shown in Fig. 6, the know-how information includes information about the sea conditions (proper water temperature) suitable for each fish species. The sea conditions suitable for each fish species, such as water temperature, salinity concentration and current speed, can vary depending on the region. Thus, by including the sea conditions suitable for each fish species in the know-how information, the modified results of the prediction probabilities of each fish species by the machine learning model modified by the know-how model can be approximated to the prediction probabilities according to the regionality of each fish species. Therefore, the accuracy of the fish species discrimination results can be enhanced.

As shown in Fig. 6, the know-how information includes information about when and where fish of each species are caught. The time and location at which fish of each species can be caught may vary from region to region. Therefore, by including at least one of the time and location at which fish of each species are caught in the know-how information in this way, it is possible to approximate the modification result of the prediction probability of each species by the machine learning model modified by the know-how model to the prediction probability corresponding to the regionality of fish of each species. Therefore, the accuracy of the fish species discrimination result can be enhanced.

As shown in Fig. 12 (a) and (b), the control unit 201 changes the degree of modification of the prediction probability in the know-how model (the reliability of the know-how information) based on feedback information indicating the content of the user’s modification to the discrimination result. As a result, the degree of modification of the prediction probability in the know-how model (the reliability of the know-how information) is changed at any time according to the content of the user’s modification to the discrimination result, so that even if, for example, the user accidentally sets inappropriate know-how information, the prediction probability after the modification can be made close to the prediction probability corresponding to the actual fish species.

In the processing of Fig. 12 (b), the degree of modification of the prediction probability in the know-how model is changed by changing the reliability of each know-how information, but the method of changing the degree of modification of the prediction probability in the know-how model is not limited to this. For example, if the prediction probability is modified by the following equations (4) and (5) instead of the above equations (1) and (2), the adjustment rate Ra may be modified by the feedback information.

Prediction probability after modification = Prediction probability x (1 + Ra) ... (4)

Prediction probability after modification = Prediction probability x (1 - Ra) ... (5)

Here, the initial value of the adjustment rate Ra is set to, for example, 0.5, and subsequent feedback information changes the adjustment rate Ra in the range of 0.1 ~ 1. In this case, for example, in step S413 of Fig. 12 (b), the adjustment rate Ra applied to the know-how information to be processed is increased by 0.05, and in step S414, the adjustment rate Ra applied to the know-how information to be processed is decreased by 0.05.

As shown in Fig. 2, the fish species determination system 1 includes an underwater detection device 10 for detecting a school of fish in the water and a server 20 capable of communicating with the underwater detection device 10. Here, the echo data acquisition unit 110 is arranged in the underwater detector 10, and the storage unit 202 for storing the machine learning model, know-how information and know-how model, and the control unit 201 for discriminating the fish species of the fish school using these are arranged in the server 20. According to this configuration, the construction of the machine learning model and know-how model necessary for fish species discrimination and the processing of fish species discrimination using these are mainly executed in the server 20. Therefore, the fish species discrimination processing can be efficiently executed while reducing the burden on the underwater detection device 10 installed in the ship.

The present disclosure is not limited to the above embodiment, and the embodiment of the present disclosure can be modified in various ways other than the above configuration.

For example, the degree of modification of the prediction probability in the know-how model may be modified based on the learning progress of machine learning.

Fig. 14 (a) is a flow chart showing the process of modifying the degree of modification of the prediction probability in the know-how model based on the learning progress of machine learning.

Every time the machine learning model is updated with new teacher data (S501: YES), the control unit 201 sets a modification rate according to the learning depth of the machine learning (S502). In this case, the above expressions (1) and (2) are modified as follows, for example.

Prediction probability after modification = 100% x {1- (1-Rp) x (1-Rn x Rm)} ... (6)

Prediction probability after modification = Prediction probability x (1 - Rn x Rm) ... (7)

Rm in equations (6) and (7) above is the modification rate in step S502 in Fig. 14 (a). The initial value of the modification rate Rm is 1, which is reduced from 1 as the learning progress of the machine learning model increases. Therefore, as the learning progress of the machine learning model increases, the influence of the confidence Rn (minority representation) in equations (6) and (7) is weakened, and the degree of modification of the prediction probability in the know-how model is reduced.

The control unit 201 sets the modification rate Rm, for example, according to the table in Fig. 14 (b). The learning progress is defined, for example, by the total number of teacher data used for learning the machine learning model. When the learning progress reaches P1, the modification rate Rm is set from 1 to R1 (R1 is less than 1), and then, when the learning progress reaches P2, the modification rate Rm is set from R1 to R2 (R2 is less than R1). Similarly, each time the next learning progress is reached, the modification rate Rm is reduced. The difference in the modification rate Rm between the learning progress need not be constant.

In addition, when equatione (4) and (5) above are used, they may be modified as follows:

Prediction probability after modification = Prediction probability x (1 + Ra x Rm) ... (8)

Prediction probability after modification = Prediction probability x (1 - Ra x Rm) ... (9)

If Equations (6) and (7) or (8) and (9) are used, Equation (3) is also used as appropriate.

According to the processing in Fig. 14 (a), for example, when the learning progress of the machine learning model is low and the accuracy of the prediction probability of each fish species is low, the degree of modification of the prediction probability in the know-how model is enhanced, and then, as the learning progress of the machine learning model increases and the accuracy of the prediction probability of each fish species increases, the degree of modification of the prediction probability in the know-how model is reduced. Thus, when the machine learning model and the know-how model work in a complementary manner, the accuracy of the prediction probability after the modification can be efficiently enhanced and the accuracy of the fish species discrimination result can be enhanced.

The modification rate Rm set in Fig. 14 (a) need not necessarily be applied to all fish species, for example, even if the learning progress of the machine learning model is enhanced, the modification rate Rm may be set to 1 for fish species with low discrimination accuracy, that is, fish species whose discrimination results are frequently modified by the user, and the prediction probability by the machine learning model may be modified by the know-how model in the same manner as in Equations (1) and (2) above.

In the above embodiment, the know-how information of each fish species set by the user is used to generate the know-how model as it is, but if, for example, the know-how information set by the user overlaps among fish species, the control unit 201 may modify the know-how information so that there is no overlap among fish species.

Fig. 15 is a diagram showing an example of modification of the know-how information.

In this example, the know-how information (here, swimming depth) of each fish species on the left side overlaps with each other. In this case, the control unit 201 modifies the know-how information of each fish species so that there is no overlap among fish species. The right side of Fig. 15 shows the modified know-how information. In this case, the control unit 201 uses the modified know-how information to perform the same fish species discrimination processing as in the above embodiment.

The method of modifying the know-how information is not limited to the method shown in Fig. 15. For example, the swimming depth of mackerel may be modified to 55 m to 120 m and the swimming depth of sardine may be modified to 20 ~ 55 m. Also, in the example in Fig. 15, the know-how information of each fish species is modified so that the overlap is completely eliminated, but the know-how information of each fish species may be modified so that the range of overlap is reduced. For example, the swimming depth of mackerel may be modified to 60 m to 120 m, and the swimming depth of sardine may be modified to 20 ~ 80 m.

In addition, instead of or in conjunction with the processing for modifying the range of know-how information that overlaps among fish species as described above, the initial value of the reliability of the know-how information that overlaps among fish species may be modified. For example, in the example of Fig. 15, the swimming depths of mackerel and sardine can be assumed to overlap not only among these fish species but also with those of tuna and many other fish species because of their wide range. In this way, the confidence level of the know-how information that overlaps with more than a predetermined number of fish species may be initially set to a value less than 100% (for example, 60%).

As described above, by modifying the range of know-how information that overlaps with each other or by modifying the confidence level of these know-how information, it can be assumed that the know-how model is more appropriate and more appropriate fish species discrimination results can be obtained.

In the above embodiment, the process of automatically modifying the know-how model based on the feedback information is performed, but this process may be omitted. For example, with respect to the know-how information that caused the fish species to be modified by the user, the process of urging the user to reset the know-how information may be performed when the frequency or number of such modifications exceeds a predetermined threshold. In this case, the control unit 201 of the server 20 transmits a notification for resetting the know-how information to the underwater detector 10, and the control unit 201 of the underwater detector 10 causes the display unit 102 to display a screen for resetting the know-how information based on the reception of the notification. In this case, the control unit 201 may cause the display unit 102 to further display the reason why resetting is necessary, for example, that the fish school discrimination result based on this know-how information is frequently modified by the user.

In the above embodiment, an example of the kind of know-how information set by the user is shown in the know-how input area 502 of the input screen 500 in Fig. 6, but the kind of know-how information set by the user is not limited to this. For example, other kinds of know-how information such as salinity concentration and tidal current speed may be included in the know-how information that can be set by the user.

In the input screen 500 of Fig. 6, the user can set the reliability of each know-how information, but the reliability of each know-how information need not be settable. In this case, for example, all the know-how information input by the user to the know-how input area 502 of the input screen 500 is set to high reliability. For this reason, a message for allowing only the confident information to be input may be further displayed on the input screen 500.

In the above embodiment, the machine learning model 302 and the know-how model 312 are separately configured as shown in Fig. 5, but the know-how model 312 may be incorporated into the machine learning model 302.

In the above embodiment, the storage of the machine learning model, the know-how information and the know-how model and the determination of the fish species using these were performed on the server 20 side, but these storage and determination processing may be performed on the underwater detector 10 side.

In this case, the server 20 transmits the latest machine learning model to the underwater detector 10 from time to time, and the control unit 101 of the underwater detector 10 stores the latest machine learning model received from the server 20 in the memory. The control unit 101 of the underwater detector 10 also stores the know-how information of each fish species set by the user in the memory, and further stores the know-how model generated using this know-how information in the memory. Based on the echo data acquired by the echo data acquisition unit 110, the control unit 101 performs fish species discrimination using the machine learning model and the know-how model, as in the control unit 201 of the server 20. Further, the control unit 101 stores the same information as in Fig. 4 (a) in the memory, and performs the modification of the know-how model, as in the above, based on the feedback information input via the echo image P1 in Fig. 11.

The storage of the machine learning model, the know-how information and the know-how model, and the determination of the fish species using these may be shared by the underwater detection device 10 and the server 20.

In the above embodiment, feedback information from the user is input by the screen shown in Fig. 11, but the method of input of feedback information is not limited to this.

In the above embodiment, the machine learning model is generated by machine learning using teacher data created by experts, but the teacher data used for machine learning of the machine learning model is not limited to this. For example, each user may input a school of fish and its species on an echo image from the user’s own catch results, and the echo data of this school of fish and the species of fish in the school of fish may be used as the teacher data of the machine learning model. In this case, the server 20 stores the school of fish and its echo data and the species of fish in the school of fish input by each user as the teacher data. The server 20 may, for example, aggregate this teacher data by region and generate a machine learning model by region. In this case, the control unit 201 of the server 20 may perform machine learning on the machine learning model by region by the teacher data aggregated by region.

In the above embodiment, the underwater detector 10 is a fish finder, but the underwater detector 10 may be a device other than a fish finder such as sonar.

In addition, the embodiment of the present disclosure may be modified as appropriate to the extent stated in the claims.

List of Reference Numerals
1 Fish species discrimination system
10 Underwater detection system
20 Server
201 Control unit
202 Storage unit
302 Machine learning model
312, 322 Know-how model

Claims

A fish species discrimination system, comprising:
an echo data acquisition unit for acquiring underwater echo data;
a storage unit; and
a control unit;
wherein the storage unit stores
a machine learning model for outputting prediction probabilities for each fish species based on the echo data;
the know-how information for each fish species for a fish school set by the user; and
the know-how model for modifying the prediction probability for each fish species based on the know-how information; and
memorizing, and
the control unit discriminate the fish species of the said fish school based on the modification result obtained by modifying the said predicted probabilities for each of the said fish species obtained by the said machine learning model by the said know-how model.
The fish species discrimination system of claim 1, wherein:
the know-how information includes information about the characteristics of the fish of the said fish species in water,
a fish species discrimination system characterized by:
The fish species discrimination system of claim 1, wherein:
said know-how information includes information on the sea conditions suitable for the fish of said fish species,
a fish species discrimination system characterized by:
The fish species discrimination system of claim 1, wherein:
the know-how information includes information about at least one of the time and place of catching the fish of the said species,
a fish species discrimination system.
The fish species discrimination system of claim 1, wherein:
the control unit changes the degree of modification of the prediction probability in the know-how model based on feedback information indicating the user’s modification to the discrimination result.
This is a fish species discrimination system.
The fish species discrimination system of claim 1, wherein:
the control unit changes the degree of modification of the prediction probability in the know-how model based on the learning progress of the machine learning.
This is a fish species discrimination system.
The fish species discrimination system of claim 1, further comprising:
an underwater detection device for detecting a school of fish in the water, and
a server capable of communicating with the underwater detection device, and wherein:
the echo data acquisition unit is arranged in the underwater detection device, and
the storage unit and the control unit are arranged in the server,
a fish species discrimination system.
The fish species discrimination system of claim 7, wherein:
the underwater detection device:
a display unit, and
an input unit, and
a control unit that displays the discrimination result in the display unit, accepts the modification of the discrimination result through the input unit, and transmits the modification to the server as feedback information.
A server capable of communicating with an underwater detection device for detecting a school of fish in the water, comprising:
a storage unit; and
with a control unit; and
wherein the storage unit stores:
a machine learning model for outputting prediction probabilities for each fish species based on echo data received from the underwater detector; and
the know-how information for each fish species with respect to a school of fish set by the user; and
the know-how model for modifying the prediction probability for each fish species by the machine learning model based on the know-how information, and
the control unit discriminates the fish species of the fish school based on the modification result obtained by modifying the prediction probability for each of the fish species acquired by the machine learning model by the know-how model;
A fish species discrimination method , comprising:
obtaining underwater echo data,
calculating the predicted probabilities for each fish species by a machine learning model based on the echo data, and
storing know-how information from the user about a fish school for each fish species, and
modifying the predicted probabilities for each fish species acquired by the machine learning model by a know-how model for modifying the predicted probabilities for each fish species based on the know-how information, and
discriminating the fish species of the fish school based on the predicted probabilities for each fish species modified by the know-how model.
A program comprising functions which, when executed by a computer, cause the computer:
to calculate the prediction probability for each fish species based on echo data acquired from underwater using a machine learning model, and
to store know-how information from users about fish schools for each fish species, and
to modify the predicted probabilities for each fish species acquired by the machine learning model by a know-how model for modifying the predicted probabilities for each fish species based on the know-how information, and
to discriminate the fish species of the fish school based on the predicted probabilities for each fish species modified by the know-how model.
a method of identifying fish species characterized by: