DE102018215096A1 - Autonomous underwater vehicle to support fishing - Google Patents

Abstract

The invention provides an autonomous underwater vehicle that is set up to accommodate a classification module. The autonomous underwater vehicle comprises (100) a camera system (126, 200) which is set up to take digital images (310). The classification module is trained to capture fish in the digital images (310) and to assign a fish species to the captured fish (208) in the digital images. A processor (302) is arranged to control the autonomous underwater vehicle to: control (352) the autonomous underwater vehicle by maneuvering through a school of fish (208); control (354) the camera system for capturing the digital images while the autonomous underwater vehicle is maneuvering through the school of fish; calculate (356) statistical data (318) of the school of fish using the digital images and using a classification module; and store the statistical school of fish data in memory (358).

Description

Field of the Invention

The invention relates to autonomous underwater vehicles, in particular the use of autonomous underwater vehicles to support fishing.

Background and state of the art

When fishing in the sea, fishing vessels are usually equipped with active sonar systems for locating schools of fish. While the fishing vessel is moving, it can repeatedly ping the active sonar system and generate an echogram. Echograms are effective in locating schools of fish. However, the fishermen do not necessarily know which type of fish or which types of fish the shoal consists of.

One problem for the fishermen is that they only recognize which fish the shoal consists of after they have caught the fish. This can lead to unwanted fish being caught that cannot be used economically and must be discarded. These unwanted fish are called bycatch. Discarding bycatch can adversely affect the health of some fish stocks. In some regions there are restrictions on the total number of fish that a fishing vessel can catch. The fishing vessel can therefore become more economical by reducing by-catches.

The magazine article Handoko et. al., "Designing embedded fish sensor for underwater robot", Indian Journal of Marine Science, vol. 38 (3), September 2009, pp. 308-315 (hereinafter "Handoko et. al.") discloses an underwater robot that uses an artificial neural network that is implemented on an integrated microcontroller that uses sonar to identify fish.

U.S. patent application US 20170105397A1 discloses a device that enables monitoring of the underwater environment of a trawl or other seafood collection device. The device consists of a cylindrical shell. The cylindrical shell includes a sensor that generates data regarding the underwater environment of the device. The cylindrical shell also includes a light source. The light source is a light emitting device or a reflected light beam. The sensor and light source are rotatable about an axis that extends to the ends of the cylindrical shell. The device further comprises one or more electrical components, which are coupled to the sensor and the light source and additionally communicatively coupled to a data processing device. At least part of the cylindrical shell is transparent.

Brief description

The invention provides in the independent claims an autonomous underwater vehicle, a school of fish location, a method for reducing by-catch and a method for providing an updated training machine learning module and an updated classification module. Embodiments are specified in the dependent claims.

Embodiments may provide a system that can reduce by-catch during a catch. This can be achieved by using an autonomous underwater vehicle with a camera system. The camera system can include at least one camera. A processor within the autonomous underwater vehicle controls the autonomous underwater vehicle to swim in a school of fish. Digital images taken by the camera system are entered into a classification module within the autonomous underwater vehicle, which can detect fish and also assign an identification number. Since the detection and identification takes place within the autonomous underwater vehicle, this information can be quickly transferred to a fishing vessel.

Normally, the water conditions and / or the depth of the fish in the sea may not allow photographic methods for the correct detection of fish. Handoko et. al. disclosed an embedded fish sensor using sonar and mentioned in the first paragraph of "Materials and Methods" that visual sensors such as cameras cannot normally be used in water because light effects interfere with the detection. Embodiments of the present invention overcome this by maneuvering the autonomous underwater vehicle through the school of fish. Even in adverse water and light conditions, the autonomous underwater vehicle is able to take pictures of fish in the immediate vicinity of the autonomous underwater vehicle. Maneuvering the autonomous underwater vehicle through the school of fish and taking digital images is comparable to taking samples from the fish population of a school of fish. Light sources (visible or even infrared) can be attached to the autonomous underwater vehicle to enable measurements in poor and missing lighting conditions.

In one aspect, the invention provides an autonomous underwater vehicle that is set up to accommodate a classification module. The classification module is set up to recognize fish in digital images and to assign a fish species to the recognized fish. The classification module could be implemented in hardware, software or a combination of hardware and software. The classification module can be set up to assign a fish species to a limited number of fish species. For example, the assigned fish species could come from a set of predefined fish species for which the classification module is set up. In some embodiments, an unknown species of fish could be one of the classifications available. In this case, the fish species is assigned a label as an unknown fish species if the classification module does not recognize the fish species of a specific detected fish.

The classification module is an implementation of the classification functionality provided by a trained machine learning module, or it is the trained machine learning module.

The autonomous underwater vehicle also includes a camera system that is set up to take digital images. The camera system is set up to take pictures outside or around the autonomous underwater vehicle. The autonomous underwater vehicle further includes a memory for storing statistical schools of fish.

The autonomous underwater vehicle further includes a processor for controlling the autonomous underwater vehicle. The processor is set up to control the autonomous underwater vehicle for maneuvering through a school of fish. The control of the autonomous underwater vehicle for maneuvering by the school of fish can be different in different examples. In some cases, the location of the school of fish can be preprogrammed or transferred to the autonomous underwater vehicle. In other cases, the autonomous underwater vehicle can go through a search routine and first look for the school of fish before passing through it. The processor is also configured to control the camera system for capturing the digital images while the autonomous underwater vehicle is maneuvering through the school of fish.

The processor is also configured to calculate shoal statistical data using the digital images and the classification module. If the classification module is implemented as software, this could include access to the memory by the processor to provide the shoal statistical data using the trained machine learning module by capturing fish in the digital images and assigning the species to the captured fish.

The trained machine learning module could be configured to use the detection of the fish and the assignment of the species to the detected fish to calculate the statistical shoal of fish data, or the trained machine learning module could provide its output to the processor, which is also configured to calculate the statistical fish shoal data .

In other examples, the processor is configured to calculate shoal statistical data using the digital images, and in some cases the trained machine learning module could alternatively be formulated as follows: The processor is also configured to calculate shoal statistical data by including fish in the digital images and the fish species of the fish in the digital images determined with the trained machine learning module.

The processor is also set up to store the statistical shoal of fish data in the memory. This embodiment can be advantageous because it can provide an effective means of determining the composition of a school of fish. This can be useful, for example, for fishing certain economically valuable species and also for avoiding fishing for fish that cannot be used commercially. This can reduce the burden on fisheries.

Moving the autonomous underwater vehicle through the school of fish can provide a sample of the statistical school of fish data while the autonomous underwater vehicle is moving and capturing the digital image data. The digital images can be viewed as a sample of a distribution that is recorded periodically. The accuracy of this data can be increased, for example, by repeatedly maneuvering the autonomous underwater vehicle through the school of fish.

In another embodiment, the autonomous underwater vehicle comprises the classification module. For example, software-implemented components of the classification module can be stored in the memory of the autonomous underwater vehicle. All hardware components of the classification module, such as special integrated circuits, can be installed in the autonomous underwater vehicle.

If the classification module has not yet been installed, the autonomous underwater vehicle is set up for taking digital images, which can be used for training a trained machine learning module. In one example, this would include the processor performing the steps of controlling the autonomous underwater vehicle to maneuver through a school of fish and controlling the camera system to capture the digital images while the autonomous underwater vehicle is maneuvering through the school of fish when the classification module is not installed. This can be advantageous since the autonomous underwater vehicle can be delivered before the classification module is provided and can be used to record digital training images in the real environment in which the autonomous underwater vehicle is used.

In another embodiment, the autonomous underwater vehicle is a light autonomous underwater vehicle or LAUV. A LAUV is defined herein as a portable, autonomous, one-man underwater vehicle. This is an autonomous underwater vehicle that can be lifted by a single adult for deployment and recovery.

In another embodiment, the calculation of the statistical schooling fish data is carried out at least partially during the acquisition of the digital images. This can have the advantage that the time for calculating the statistical fish school data is reduced. In some embodiments, the calculation of the schooling statistical fish data is performed after groups of digital images are captured. With an increasing number of groups of digital images, the statistical shoal data are continuously improved with larger samples of fish depicted.

In another embodiment, the fish species is one of a set of predetermined fish species. The processor is configured to receive a selection of one or more selected types. The set of predetermined fish species includes the one or more selected species. Statistical shoal data are calculated for the one or more selected species. This embodiment can be advantageous because the autonomous underwater vehicle can be programmed to only compute statistical shoal data for the one or more selected species. This can reduce the computing power required to calculate the statistical schooling fish data. It can also have the advantage of reducing the amount of data that has to be transmitted in order to transmit the statistical shoal of fish data to a fishing vessel. The reduction in the amount of data to be transmitted would be particularly advantageous if an acoustic modem is used.

The set of predetermined fish species can be determined by the training process used to manufacture the classification module or the trained machine learning module. Further training of the trained machine learning module can be used, for example, to expand the number of identified types. In some cases there may be a shortlist of fish such as mackerel and herring. Other species that are not mackerel or herring in this example can be identified as other species or bycatch species.

The processor is configured to receive a selection of one or more selected types. The set of predetermined fish species includes the one or more selected species. The selection can be made in different ways; in some cases, the autonomous underwater vehicle is provided through a user interface, network, or other data system. In other cases, the autonomous underwater vehicle can be pre-programmed and the selection can be called up from the memory.

In another embodiment, the autonomous underwater vehicle comprises the classification module.

In another embodiment, the classification module is implemented as hardware, as software, executable instructions and / or a data structure.

In another embodiment, the classification module is a trained machine learning module.

In another embodiment, the memory stores the trained machine learning module.

In another embodiment, the trained machine learning module comprises at least one convolutional neural network. In one example, there may be a convolutional neural network used to first capture the position of each fish and provide bounding frames, and then a second convolutional neural network that looks at the image within each bounding frame and assigns a species or species of fish . In another example, a single convolutional neural network performs both actions.

In another embodiment, the trained machine learning module comprises a neural network.

In another embodiment, the trained machine learning module comprises a deformable template comparison algorithm. Various image templates that can be used to represent individual fish species are used and can be deformed according to the fish captured in the digital images.

In another embodiment, the trained machine learning module comprises a local orientation histogram with a support vector machine algorithm that is trained to identify the different fish species.

In another embodiment, the autonomous underwater vehicle further comprises at least one light source, which is set up to illuminate a field of view of at least one camera of the camera system during the recording of the digital images. This embodiment can be advantageous because the autonomous underwater vehicle can penetrate to depths in which there is insufficient light for taking clear images or clear digital images. The use of a light source can therefore enable the use of the autonomous underwater vehicle to provide the statistical school of fish data in poor visibility or poor water conditions. On the open sea it is possible that a lack of light, restless or rough weather conditions above the sea or pollutants or other substances in the water make imaging over long distances difficult. If the autonomous underwater vehicle swims or maneuvers through a shoal of fish, the effect of sampling to detect fish does not depend on the image capture of fish that are far from the autonomous underwater vehicle. Even under conditions where cameras of the camera system have a limited range, they can still take a significant statistical sample while the autonomous underwater vehicle maneuvers through the school of fish.

In some embodiments, there is at least one light source per camera of the camera system. These can be used to illuminate part or all of the camera's field of view.

In another embodiment, the light source is a visible light source.

In another embodiment, the light source is an infrared light source. When using an infrared light source, the camera with which the infrared light source is coupled can also be a camera that is capable of imaging infrared light.

Using an infrared camera can be beneficial because it may be less of a nuisance to fish in the school of fish. This can help the autonomous underwater vehicle to move through the school of fish without frightening the fish or changing their behavior. This can allow a more accurate statistical sample of the fish species.

In another embodiment, the autonomous underwater vehicle comprises a hull. The hull can take various forms, for example. In one embodiment, the fuselage is a cylindrical fuselage.

In another embodiment, the hull is less than 1.5 meters in length.

In another embodiment, the hull is less than 1 meter in length.

In another embodiment, the hull is less than 0.5 meters in length.

A shorter hull can be advantageous since the autonomous underwater vehicle interferes less with the school of fish when passing through the school of fish and this can lead to a better measurement of the statistical school of fish data.

In another embodiment, the hull has a circumference that is less than 80 cm.

In another embodiment, the hull has a circumference that is less than 50 cm.

In another embodiment, the hull is less than 20 cm in circumference.

In another embodiment, the hull has a circumference of less than 10 cm.

A hull with a smaller circumference can be advantageous because the overall size of the autonomous underwater vehicle is smaller. The smaller the autonomous underwater vehicle, the less likely it is that the autonomous underwater vehicle will interfere with the school of fish when taking measurements.

In another embodiment, the processor is further configured to control the autonomous underwater vehicle so that it glides through the school of fish at least part of the time when driving. Sliding as used herein means sliding in an idle state. For example, the autonomous Have underwater vehicle thrusters or propellers that are used to propel the autonomous underwater vehicle. However, using the propellers or thrusters can interfere with the fish by making noise or warning them. By sliding for at least part of the time, the accuracy of the measurement of the statistical fish school data can be improved.

In another embodiment, the processor is further configured to control the autonomous underwater vehicle to use an intermittent drive when maneuvering through the school of fish. This embodiment may be advantageous because it can minimize the use of the autonomous underwater vehicle propulsion system. This can be effective in less frightening or scaring the fish in the school of fish. This can improve the quality of the schooling fish statistical data.

In another embodiment, the memory comprises a removable memory component. The digital images are recorded on the removable memory component. For example, there may be a memory card or other removable memory that can be removed from the autonomous underwater vehicle. This can be an efficient means of reading out the digital images. This can be useful, for example, for further training of the trained machine learning module.

In some other embodiments, there may be more than just the digital images stored on the removable storage component. The shoal statistical data as well as the data indicating the detection of fish such as bounding frames and the identification of fish species for each of the bounding frames can also be recorded on the removable memory component. This can be helpful when assessing the accuracy of the trained machine learning module.

The processor is also configured to receive shoal position data describing the size and position of the shoal of fish. The swarm position data includes any of the following: a depth range of the school of fish, a size of the school of fish, an echogram of the school of fish, and combinations thereof. The processor is further configured to at least partially control the maneuvering of the autonomous underwater vehicle by the school of fish using the swarm position data. This embodiment can be advantageous since it provides some initial data for the autonomous underwater vehicle in order to find the school of fish more efficiently and quickly.

The shoal position data can be entered, for example, into a user interface in which the depth range and the width or size of the shoal of fish are entered. In other examples, the echogram itself could be transferred to the autonomous underwater vehicle, and this could serve as a guide for the search for the school of fish. For example, the sonar system on a fishing vessel could connect or control the autonomous underwater vehicle.

In another embodiment, the autonomous underwater vehicle further comprises a sonar system that is set up to acquire sonar data. The processor is also set up to control the drive system in order to at least partially maneuver the autonomous underwater vehicle through the school of fish using the sonar data. For example, sonar data can be useful in determining the number of fish in a particular area and the size of the fish located there. The position of the fish can be used to choose a direction in which the autonomous underwater vehicle is to be moved.

In another embodiment, the sonar data are used to supplement the calculation of the statistical fish school data on the basis of the sonar data. As mentioned earlier, the relative number and size of fish can be used to provide further better statistics. For example, the use of sonar data can enable an accurate determination of the relative size distribution of the fish without the use of a camera.

In another embodiment, the processor is further configured to determine a position of the school of fish by capturing the individual fish in the digital images. The processor is also configured to change the maneuvering of the autonomous underwater vehicle to swim in the direction of the school of fish by using the position of the school of fish determined on the basis of the digital images. For example, the image data can indicate that a greater concentration of fish can occur in a certain direction. In further embodiments, both the position of the fish in the images and sonar data can be used to precisely steer the autonomous underwater vehicle and to maximize the number of encounters with fish.

In another embodiment, the camera system includes a forward-facing one Camera, the field of vision of which is aligned with a drive direction of the autonomous underwater vehicle. The autonomous underwater vehicle can be driven, for example, in a direction that has an axis of symmetry with a cylindrical hull. The use of the forward camera can be advantageous because it can take pictures of fish that are present in an amount that has not been potentially disturbed by the autonomous underwater vehicle.

In another embodiment, the camera system comprises at least one laterally mounted camera, the field of vision of which is oriented essentially perpendicular to the drive direction of the autonomous underwater vehicle. This can be advantageous since a further additional or more cameras can be added and can take up a larger spatial volume in the vicinity of the autonomous underwater vehicle. Imaging this larger volume can improve the quality of the statistical schooling fish data.

In another embodiment, the shoal statistical data includes fish quantity statistics calculated using the recognition of the individual fish in the digital images. For example, the number of bounding frames that fish display in the digital images can be used as statistics, either as a mean or as a total, to estimate the total number of fish in a school of fish.

In another embodiment, the shoal statistical data includes fish size statistics calculated using the capture of the individual fish in the digital images. For example, the extent to which the fish are based on visibility in the water can be estimated, or a stereo camera can be used, and an accurate estimate of the distribution of the fish sizes can be calculated. In some cases, this data can be supplemented or replaced with sonar data, which can also be used to measure the average size or distribution of fish sizes.

In another embodiment, the shoal statistical data further includes fish species statistics describing the identification of the one or more selected species in the digital images. This can be broken down, for example, according to the recognized types of fish and also identify fish that do not match those contained in the set of predefined fish species.

In another embodiment, the autonomous underwater vehicle comprises a communication module that is set up to communicate with a remote computing device. The remote computing device can be, for example, an AUV control system. The communication module may include one of the following: an acoustic modem, a WLAN system, a LoRa system, and combinations thereof. The WLAN system and the LoRa system work with radio communication and can require that the autonomous underwater vehicle shows up before use. The acoustic modem, on the other hand, can be used to transmit information with a smaller bandwidth while it is completely or completely submerged.

In another embodiment, the processor is further configured to transmit the statistical schooling fish data using the communication module. This embodiment may be advantageous because the school of shoal statistical data is then provided to the operator or the fishing vessel to make a decision as to whether the school of fish should be caught before the school of fish has dispersed or moved on.

In another embodiment, the processor is further configured to calculate a catch decision by comparing the statistical fish school data with a predetermined criterion. The fishing decision consists of an instruction to catch the school of fish or to avoid the school of fish. The processor is also set up to append the catch decision to the statistical schooling fish data. This embodiment can be advantageous because it can provide guidance to a fisherman without the use of an expensive computer system.

In another embodiment, the processor is further configured to control the autonomous underwater vehicle to hold a predetermined position within the school of fish if the catch decision includes the command to catch the school of fish after providing the catch decision. This embodiment can be advantageous since the autonomous underwater vehicle can be brought into a position in which it is caught or retrieved by the fishing nets. This can be more efficient for operators of a fishing vessel as they do not have to track or catch up with the autonomous underwater vehicle. You can just catch the school of fish and it will be caught up in the fishing nets.

In another embodiment, the processor is further configured to control the autonomous underwater vehicle to hold a predetermined position within the school of fish. This embodiment may be advantageous since it is desirable for the autonomous underwater vehicle may be to get a larger sample of data or to stay in the predetermined position to avoid frightening the fish.

In a further aspect, the invention provides a school of fish location. A fish school location system as used herein includes a device that can be used to locate or identify fish. The fish school tracking system includes an autonomous underwater vehicle according to one embodiment. The school of fish location system further includes a remote computing device. The remote computing device may also be referred to herein as an AUV control system. The remote computer device comprises a communication system that is set up to establish a communication connection with the communication module of the autonomous underwater vehicle. The remote computer device is set up to receive the statistical shoal of fish data from the autonomous underwater vehicle via the communication link. This embodiment is advantageous because it can provide a means to quickly provide the shoal statistical data to the operator of a fishing vessel.

In another embodiment, the remote computing device is one of the following: a tablet computer, a laptop computer, a computer, a smartphone, and an echolocator. The integration of the remote computer device into an echolocator can be advantageous since the functions of the sonar system with which the fish are located can also be used as a control and analysis system for the autonomous underwater vehicle. For example, the echolocator of a fish trawler could have a WLAN system or be connected to an acoustic modem. After the school of fish has been located with the sonar, the echolocator can download the echogram or data about the echogram to the autonomous underwater vehicle either before or after a start, depending on the type of communication established between the two.

In another embodiment, the computer system includes a display. The computer system is set up to calculate estimated shoal of fish data from the statistical shoal of fish data. The school of fish data includes any of the following: an estimated number of the one or more selected species of the school of fish, an estimated mass of the one or more selected species of the school of fish, an estimated size of the fish within the school of fish, an estimated number of by-catch fish in the school of fish , an estimate of the mass of by-catch fish in the school of fish, the types of fish in percent and combinations thereof. In this embodiment, the processor is further configured to display the estimated shoal of fish data on the display. This embodiment can be advantageous because it can be used to convert the school of fish statistical data into a form that can be useful in deciding whether or not a school of fish is caught.

In another embodiment, the school of fish location system further comprises a central training server. The central training server is set up to collect the recorded digital images from the autonomous underwater vehicle via the remote computer device. In this embodiment, for example, there may be a network connection that is established continuously or intermittently between the remote computing device and the central training server. The central training server is also set up to receive markings which indicate the position of individual fish in the recorded digital images and / or an assignment of the fish species to the individual fish in the recorded digital images. For example, these can be downloaded from another source or entered by an operator on a user interface. The central training server is also set up to generate an updated, trained machine learning module from the markings and the recorded digital images. If the updated trained machine learning module (s) is a neural network, such as a convolutional neural network, this can be done, for example, by deep learning. This embodiment can be advantageous since the data obtained during the actual fishing activities can be used to provide a trained machine learning module.

In another embodiment, the central training server is also set up to transmit the updated trained machine learning module to the autonomous underwater vehicle via the remote computer device. The autonomous underwater vehicle is set up to install the updated trained machine learning module as a trained machine learning module. In some embodiments, the central training server may also be configured to generate the updated trained machine learning module by training an untrained machine learning module from the autonomous underwater vehicle using the markers and the captured digital images. The untrained machine learning module may be in an initial state before being trained. For example, if the untrained machine learning module is a neural Network is, the structure of the untrained machine learning module can be defined in advance, but the different weights for the neural connections cannot be trained yet.

In some embodiments, the central training server may further be configured to generate the updated trained machine learning module by training a copy of the trained machine learning module from the autonomous underwater vehicle using the markings and the captured digital images. This embodiment can be advantageous since the performance of the trained machine learning module is continuously improved when using the schooling fish location system. This can be useful to improve the accuracy of the schooling fish statistical data.

In another aspect, the invention provides a method of reducing bycatch in accordance with one embodiment. The method involves locating a school of fish. The method also includes deploying the autonomous underwater vehicle. The method further includes control of the autonomous underwater vehicle by the processor according to one embodiment. The method further includes receiving the shoal statistical data via the remote computing device. The method further includes outputting a signal as to whether the school of fish should be caught. The signal can be, for example, a message on a display or another signal such as an optical or acoustic display. This embodiment can be advantageous because it can provide an effective means of alerting a fishing vessel whether or not to catch a particular school of fish.

In another embodiment, the method further comprises controlling the autonomous underwater vehicle to maintain a predetermined position within the school of fish when the school of fish is caught. The method further includes retrieving the autonomous underwater vehicle by catching the school of fish with a net when the underwater vehicle is to maintain the predetermined position within the school of fish. This can be advantageous as it can provide a means of easily retrieving the autonomous underwater vehicle.

In another aspect, the invention provides a method for providing an updated trained machine learning module. The updated trained machine learning module is trained to capture fish in digital images. The trained machine learning module is trained to assign a fish species to the fish in the digital images. The method further includes collecting captured digital images from a fleet of autonomous underwater vehicles, all of which correspond to one embodiment. The captured digital images are taken by the autonomous underwater vehicles in the sea during the catch processes. This can be advantageous since a certain type of fish and also the optical conditions under which the images are taken are then the same ones that would be used in further capture processes.

The method further includes receiving markings that indicate the position of individual fish in the recorded digital images and / or an association of the fish species with the individual fish in the recorded digital images. The method then enables the updated trained machine learning module to be generated using the markers and the captured digital images. If the trained machine learning module is a neural network, this training can be accomplished through deep learning. The method further includes transferring the updated trained machine learning module to at least one of the fleet of autonomous underwater vehicles in the fleet. The method further includes installing the updated trained machine learning module of the at least one from the fleet of autonomous underwater vehicles as a trained machine learning module. This embodiment can be advantageous because it provides an effective means of continuously updating the trained machine learning module, which leads to a better estimate of the statistical schooling fish data.

In another embodiment, the method further comprises providing an untrained machine learning module. The updated trained machine learning module is generated by training the untrained machine learning module and using the markers and the captured digital images. The untrained machine learning module may be in an initial state before being trained. For example, if the untrained machine learning module is a neural network, the structure of the untrained machine learning module can be defined in advance, but the various weights for the neural connections cannot yet be trained.

In another embodiment, the method further comprises providing a trained machine learning module. The updated trained machine learning module is created by training a copy of the trained machine learning module and using the markers and the captured digital images.

It goes without saying that one or more of the aforementioned embodiments of the invention can be combined as long as the combined embodiments are not mutually exclusive.

As understood by those skilled in the art, aspects of the present invention can be embodied as an apparatus, method, or computer program product. Accordingly, aspects of the present invention may take the form of a full hardware embodiment, a full software embodiment (including firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware, all herein as "circuitry," " Module ”or“ System ”can be called. In addition, aspects of the present invention may take the form of a computer program product contained in one or more computer-readable media with executable program code contained thereon.

Any combination of one or more computer readable media can be used. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A "computer readable storage medium" as used herein includes any physical storage medium that can store instructions that are executable by a processor of a computing device. The computer-readable storage medium can be referred to as a computer-readable, non-volatile storage medium. The computer-readable storage medium can also be referred to as a physical, computer-readable medium. In some embodiments, a computer readable storage medium may also store data that is accessible by the processor of the computing device. Examples of computer-readable storage media include: a floppy disk, a magnetic hard disk drive, a solid-state hard disk, a flash memory, a USB stick, random access memory (RAM), read-only memory (ROM), an optical disk, a magneto-optical disc and the register file of the processor. Examples of optical disks are compact disks (CD) and digital versatile disks (DVD), e.g. B. CD-ROM, CD-RW, CD-R, CD-R, DVD-ROM, DVD-RW or DVD-R discs. The term computer readable storage medium also refers to various types of recording media that the computing device can access via a network or communication link. For example, data can be accessed via a modem, the Internet or a local network. Computer executable code on a computer readable medium can be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic cable, RF, etc., or a suitable combination thereof.

A computer readable signal medium may comprise a spread data signal with computer executable code contained therein, for example in the baseband or as part of a carrier wave. Such a propagated signal can take a variety of forms including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, transmit, or transport a program for use by or in connection with an instruction execution system, device, or device.

"Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible to a processor. "Computer memory" or "memory" is another example of a computer-readable storage medium. Computer memory is any non-volatile, computer-readable storage medium. In some embodiments, the computer memory can also be a computer memory or vice versa.

A "processor" as used herein includes an electronic component capable of executing a program or a machine-executable instruction or a computer-executable code. References to the computing device with "one processor" are to be interpreted as possibly containing more than one processor or processing core. The processor can be a multicore processor, for example. A processor can also refer to a collection of processors within a single computer system or distributed across multiple computer systems. The term "computing device" should also be interpreted to refer to a collection or network of computing devices, each comprising a processor or processors. The computer-executable code can be executed by several processors, which can be located within the same computer device or which can even be distributed over several computer devices.

Computer executable code may include machine executable instructions or a program that causes a processor to execute an aspect of the present invention. Computer-executable code for performing operations for aspects of the present invention can be in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C ++ or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages, and compiled into machine-executable instructions. In some cases, the computer-executable code can be in the form of a high-level language or in a precompiled form and can be used in conjunction with an interpreter that generates the machine-executable instructions during execution.

The computer-executable code can also include executable code from a script language and / or precompiled code from the script language. For example, the computer-executable code can be implemented at least partially in a scripting language such as Python, Ruby, Pearl, Lua or as a shell script.

The computer-executable code can be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (e.g., over the Internet using an Internet) Service provider).

Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the invention. It is understood that each block or a part of the blocks of the flowchart, the illustrations and / or the block diagrams can optionally be implemented by computer program instructions in the form of computer-executable code. It is further understood that combinations of blocks in different flowcharts, illustrations and / or block diagrams can be combined if these are not mutually exclusive. These computer program instructions may be provided to a general purpose computer processor, dedicated computer, or other programmable computing devices to create a machine so that the instructions executed by the computer processor or other programmable computing devices, means for executing those in the flowchart and / or generate the functions / actions specified for the block or blocks of the block diagram.

These computer program instructions can also be stored in a computer readable medium that can instruct a computer, other programmable data processing devices or other devices to function in a certain way so that the instructions stored in the computer readable medium produce a product together with the instructions that are contained in the Execute the flow diagram and / or the function / action specified for the block or blocks of the block diagram.

The computer program instructions can also be loaded onto a computer, other computing devices, or other devices to cause a series of operations to be performed on the computer, other programmable devices, or other devices to create a computer-executed process so that the instructions that run on the computer or other programmable devices, provide processes for performing the functions / actions indicated in the flowchart and / or the block or blocks of the block diagram.

A "user interface" as used herein is an interface that enables a user or operator to interact with a computer or computer system. A “user interface” can also be referred to as a “human-machine interface”. A user interface can provide the operator with information or data and / or receive information or data from the operator. A user interface can allow operator input to be received by the computer and can provide user output from the computer. In other words, the user interface may allow an operator to control or direct a computer, and the interface may allow the computer to display the effects of the control or instruction by the operator. The display of data or information on a display or graphical user interface is an example of providing information to an operator. Receiving data via keyboard, mouse, trackball, touchpad, trackpoint, graphics tablet, joystick, gamepad, webcam, headset, pedals, cable glove, remote control and acceleration sensor are examples of components of the user interface that enable information or data to be received by an operator .

A "hardware interface" as used herein includes an interface that it can Processor of a computer system enables to interact with and / or to control an external computer device and / or an external computer device. A hardware interface may allow a processor to send control signals or instructions to an external computing device and / or an external computing device. A hardware interface can also enable a processor to exchange data with an external computing device and / or an external computing device. Examples of a hardware interface include: a universal serial bus, IEEE 1394 port, parallel port, IEEE 1284 port, serial port, RS-232 port, IEEE 488 port, Bluetooth connection, wireless local area network connection, TCP / IP connection, Ethernet connection, control voltage interface, MIDI interface, analog input interface and digital input interface.

A "display" or "display device" as used herein includes an output device or user interface suitable for displaying images or data. A display can output visual, acoustic and / or tactile data. Examples of a display include: a computer monitor, a television screen, a touch screen, a tactile electronic display, a braille display, a cathode ray tube (CRT), a storage tube, a bistable display, electronic paper, a vector display, a flat screen display, one Vacuum fluorescent display (VF), a light-emitting diode display (LED), an electroluminescent display (ELD), plasma display panels (PDP), a liquid crystal display (LCD), an organic light-emitting diode display (OLED), a projector and head-mounted display.

An autonomous underwater vehicle (AUV) as used herein includes an underwater vehicle that is capable of independently maneuvering underwater without constant external control. The autonomous underwater vehicle can have a drive and control system.

Figure list

• 1 ein Beispiel für ein autonomes Unterwasserfahrzeug veranschau licht;
• 2 die Aufnahme digitaler Bilder durch das autonome Unterwasserfahrzeug von 1 veranschaulicht;
• 3 ein Beispiel für ein Steuerungssystem für das autonome Unterwasserfahrzeug von 1 veranschaulicht;
• 4 ein Ablaufschema zeigt, das ein Verfahren zum Betreiben des autonomen Unterwasserfahrzeugs von 1 veranschaulicht;
• 5 einen Schritt zum Auffinden eines Fischschwarms mit einem Echolocator veranschaulicht
• 6 ein Sonogramm zeigt, das vom Fischereifahrzeug erzeugt werden könnte, während es über den Fischschwarm fährt und periodisch Sonarmessungen durchführt
• 7 den Schritt des Ausbringens 600 des autonomen Unterwasserfahrzeugs veranschaulicht
• 8 den Schritt des Manövrierens durch einen Fischschwarm und des Aufnehmens der digitalen Bilder durch das autonome Unterwasserfahrzeug veranschaulicht
• 9 den Schritt des Empfangens der statistischen Fischschwarmdaten durch ein entferntes Computergerät zeigt
• 10 ein Beispiel für ein Fischschwarmortungssystem veranschaulicht
• 11 ein weiteres Beispiel für ein Fischschwarmortungssystem veranschau licht
• 12 ein Ablaufschema zeigt, das ein Verfahren zum Bereitstellen eines aktualisierten, trainierten Maschinenlernmoduls veranschaulicht
• 13 ein Funktionsschema des autonomen Unterwasserfahrzeugs zeigt
• 14 ein Beispiel für eine Implementierung des trainierten Maschinenlernmoduls veranschaulicht
• 15 den Aufbau und Einsatz des trainierten Maschinenlernmoduls veranschaulicht, wenn es sich um ein durch tiefes Lernen trainiertes Convolutional Neural Network handelt

In the following, embodiments of the invention are described purely by way of example with reference to the accompanying drawings, in which:

• 1 illustrates an example of an autonomous underwater vehicle;
• 2 the taking of digital pictures by the autonomous underwater vehicle from 1 illustrated;
• 3 an example of a control system for the autonomous underwater vehicle of 1 illustrated;
• 4 FIG. 12 shows a flow diagram illustrating a method for operating the autonomous underwater vehicle of FIG 1 illustrated;
• 5 illustrates a step to locate a school of fish with an echolocator
• 6 shows a sonogram that could be generated by the fishing vessel as it passes over the school of fish and periodically takes sonar measurements
• 7 the step of deploying 600 of the autonomous underwater vehicle
• 8th illustrates the step of maneuvering through a school of fish and capturing the digital images through the autonomous underwater vehicle
• 9 shows the step of receiving the shoal statistical data from a remote computing device
• 10 illustrates an example of a school of fish location
• 11 Another example of a school of fish location is illustrated
• 12 FIG. 12 shows a flow diagram illustrating a method for providing an updated, trained machine learning module
• 13 shows a functional diagram of the autonomous underwater vehicle
• 14 illustrates an example of an implementation of the trained machine learning module
• 15 illustrates the structure and use of the trained machine learning module if it is a convolutional neural network trained through deep learning

Detailed description

Elements with the same numbers in these figures are either equivalent elements or have the same function. Elements previously discussed are not necessarily discussed in later figures if the function is equivalent.

1 illustrates an example of an autonomous underwater vehicle 100 , The autonomous underwater vehicle 100 is as a hull 102 presented comprehensively. The hull has a length 104 and a diameter 106 , It can be seen that in this example the hull 102 is a cylindrical hull.

1 shows a side or sectional view of an idealized autonomous Submersible 100 , The autonomous underwater vehicle 100 also includes a drive system 108 , In this example it is a series of four electric motors with propellers.

The autonomous underwater vehicle 100 is like a battery 110 presented comprehensively. The battery 110 can be used to power the drive system 108 be used. The autonomous underwater vehicle 100 is further as a control system 112 comprehensively illustrated, the processor and a memory for controlling the autonomous underwater vehicle 100 may include. The battery 110 powers the control system 112 and all other electronic components within the autonomous underwater vehicle 100 with electricity.

The autonomous underwater vehicle 100 is an optional control system 114 shown containing. The control system 114 can be an inertial control system, for example. The autonomous underwater vehicle 100 is an antenna for a WLAN or LoRa system 116 presented comprehensively. This allows the autonomous underwater vehicle 100 establish communication after surfacing. The autonomous underwater vehicle 100 can optionally also use an acoustic modem 120 include. The acoustic modem 120 enables one-way or two-way communication with the autonomous underwater vehicle when diving 100 , The autonomous underwater vehicle 100 is also an optional sensor package 118 presented comprehensively. The sensor package 118 is designed to display any number of sensors in the autonomous underwater vehicle 100 included or can optionally be included. For example, the sensor package 118 Sensors such as a temperature sensor, a salinity sensor, a depth or pressure sensor and a plankton sensor include

The drive system 108 is set up to be the autonomous underwater vehicle 100 primarily in the drive direction 122 to move. In this example there are four individual propellers for the drive system 108 available. By controlling the drive of each of the individual propellers, this can steer the autonomous underwater vehicle 100 be used. In other examples, slats may be provided, which are also used to control or steer the movement of the autonomous underwater vehicle 100 be used.

The autonomous underwater vehicle 100 is an optional variable buoyancy system 124 or VBS comprehensively presented. The variable buoyancy system 124 comprises a chamber that can be filled with sea water, and the sea water can be expelled either with an oil-filled membrane or mechanically with a piston. This allows the autonomous underwater vehicle 100 stop its lift. In some embodiments, the propulsion system 108 can be used to control the depth at which the autonomous underwater vehicle operates. In other examples, the thrust can be combined with the variable buoyancy system 124 be used. In this example is the variable buoyancy system 124 in front of the battery 110 stored. This can have the advantage that the submarine's dive angle tilts forward when the variable buoyancy system 124 is filled with sea water; the drive system 108 can then be more easily used for the autonomous underwater vehicle 100 to assist in reaching a shallower depth. Likewise, when water is ejected from the variable buoyancy system 124 the diving angle is positive, which makes it the autonomous underwater vehicle 100 relieved to reach the surface. The use of a variable buoyancy system 124 can also help the autonomous underwater vehicle reach greater depths, such as hundreds of meters. The variable buoyancy system 124 can be useful in the autonomous underwater vehicle for reaching greater depths very quickly. The variable buoyancy system 124 can also be the autonomous underwater vehicle 100 support in the production of kinetic energy. The autonomous underwater vehicle can do this 100 enable a speed with minimal use of the drive system 108 to reach.

In this example, the camera system includes a forward camera 126 , which is mounted so that it has a field of vision primarily in the drive direction 122 having. In this example the camera is 126 on the outer surface of the fuselage 102 stored. In other examples, the camera can 126 in the fuselage 102 be integrated. There may also be additional cameras either on the outside of the fuselage 102 attached or integrated into this.

It is an optional lamp 128 with the field of view of the forward-facing camera 126 is at least partially illuminated.

The autonomous underwater vehicle 100 is also available as an optional sonar converter 130 presented comprehensively. The sonar converter 130 is also connected to the optional sonar electronics 132 connected. In this example, the sonar converter 130 in a bow area of the autonomous underwater vehicle 100 be stored so that it can be used to catch fish in the front area of the autonomous underwater vehicle 100 can be used. This can be useful, for example, in a direction for steering the autonomous underwater vehicle 100 with the drive system 108 to determine.

2 shows a top view of the autonomous underwater vehicle 100 , In this example the camera is forward 126 in the fuselage 102 integrated. There is also a camera on the side 200 present, which is also in the fuselage 102 is integrated. Both the side-mounted camera 200 as well as the forward-facing camera 126 both cameras could be the outside on an outer surface of the fuselage 102 are attached. Both the forward-facing camera 126 as well as the side-mounted camera 200 each have a lamp 128 and 202 on. The dashed lines 204 show the field of view of the forward-facing camera 126 on. The lamp 128 illuminates the field of vision 204 at least partially. The dashed lines also show 206 the field of view of the camera 200 on. It is an optional lamp 202 present that the field of view 206 the side-mounted camera 200 as at least partially illuminating.

In some examples, the lights 128 and 202 Be luminaires that generate visible light. In other examples, it could be infrared lights and the cameras 126 . 200 can be infrared cameras. The autonomous underwater vehicle 100 is considered to be through a school of fish 208 shown moving. Some of the fish 208 are in the field of vision 204 and some of the other fish 208 are in the field of vision 206 , While the autonomous underwater vehicle 100 through the school of fish 208 moves, samples of the size and / or type of fish are effectively taken with the images it captures. By moving through the school of fish 208 can the autonomous underwater vehicle 100 make a reasonable estimate of the type of fish within a shoal.

3 illustrates an example of a control system 112 for the autonomous underwater vehicle. The control system 112 includes a hardware interface 300 that enables the other components of the autonomous underwater vehicle 100 to control. The hardware interface 300 is with a processor 302 connected, which is also connected to a memory 304 connected is. The memory 304 can include any combination of memory required for the processor 302 is accessible, and may have elements such as a hard drive and / or a removable memory card. The memory 304 is called the machine-executable instructions 306 containing shown. The machine-executable instructions 306 contain instructions that it the processor 302 enable the autonomous underwater vehicle 100 to operate and control. The memory 304 is further shown as a trained machine learning module containing 308. Alternatively, the trained machine learning module can be replaced by a classification module that is implemented in hardware and / or software. The trained machine learning module is trained to capture individual fish in digital images. The trained machine learning module can be trained further in the digital images to assign a fish species to the individual fish. The fish species is a set of predetermined fish species.

The memory 304 is also as digital pictures 310 containing shown either with the camera attached to the side 200 or the forward-facing camera 126 were recorded. The memory 304 is further as a selection of one or more types 312 shown. The memory 304 is also used as positions of bounding boxes 314 in the digital images 310 for the selected one 312 Depicted containing species. The memory 304 is also an identification of the fish species 316 for the bounding box 314 shown containing. The memory 304 is also available as statistical schools of fish 318 depicted using the bounding box 314 or the position 316 the bounding box 314 were calculated. The memory 304 is also at least one optional predetermined criterion 320 shown containing. The predetermined criterion 320 can, for example, be a criterion that matches the statistical schooling fish data 318 can be compared to determine whether a recommended schooling of the school of fish is recommended or not. The memory 304 is as an optional catch decision 322 containing, by comparing the predetermined criterion 320 with the statistical school of fish data 318 was hit. For example, the catch decision 322 be transmitted to the fish trawler via the acoustic modem. The catch decision 322 could also be used to decide that the autonomous underwater vehicle remains with the school of fish so that it is caught by the nets and does not have to be caught separately.

The memory 304 is as optional sonar data 324 included shown with the optional sonar converter 130 and sonar electronics 132 were recorded. In the presence of sonar data 324 can the sonar data 324 to supplement the statistical school of fish data 318 be used. The sonar data 324 can also be used to control the direction of travel of the autonomous underwater vehicle 100 be used.

The memory 304 is also as swarm position data 326 shown containing. This can be data describing a position of the school of fish or a previous position, such as the depth at which the school is located, the size and possibly even the swimming direction of the school of fish. This can be useful to the autonomous underwater vehicle 100 at the Support tracking and swimming in the school of fish.

4 shows a flow chart that shows a method for operating the in the 1-3 shown autonomous underwater vehicle 100 illustrated. The process begins with the optional step 350 , In step 350 will be the selection of one or more types 312 receive. The next step 352 is running. In step 352 becomes the autonomous underwater vehicle 100 controlled by a school of fish for maneuvering. Next in step 354 the camera system 126 . 200 to take the digital pictures 310 controlled. This happens during the movement of the autonomous underwater vehicle through the school of fish. Next in step 356 the statistical school of fish data 318 calculated cumulatively by using the individual fish in the digital images using the bounding box 314 recorded and the fish species of the fish can be determined in the digital images; this can identify the species of fish 316 for each of the bounding boxes 314 his. If the optional step 350 statistical shoal of fish data may be limited to the selection of the one or more species. And finally in step 358 the statistical school of fish data 318 In the storage room 304 saved.

5 illustrates a step to locate a school of fish with an echolocator. In this step 400 a fishing vessel moves 402 on the surface of the sea 404 , In this figure, the relative size of the ship and other objects is not to scale. The seabed is marked with 406. Inside the sea 404 there is a school of fish 208 , The fishing vessel 402 has a sonar system or an echolocator for locating the fish 208 , The dashed lines 408 indicate the path of the sonar waves generated. When the sonar waves the fish 208 hit, they become a fishing vessel 408 thrown back. The fishing vessel 402 moves in the direction 412 and takes periodic measurements with the sonar system. These measurements can be used to create a sonogram of the school of fish.

6 shows a sonogram 500 from the fishing vessel 402 could be built while it was over the school of fish 208 drives and periodically performs sonar measurements. The sonogram 500 shows a sea depth 502 as well as a minimum depth 504 of the school of fish 208 on. The width 506 and height 508 of the school of fish 208 is also shown. The information in the sonogram 500 would be useful information for the autonomous underwater vehicle 100 when planning a route through the school of fish 208 , This data could take the form of swarm position data 326 to be provided. The swarm position data 326 for example, the actual sonogram 500 be that in the autonomous underwater vehicle 100 downloaded, or it may contain metadata or a summary of relevant information such as the minimum depth of the school of fish 504 as well as their size 506 . 508 his.

7 illustrates the spreading step 600 of the autonomous underwater vehicle 100 , After the fishing vessel 402 the school of fish 208 has happened, the autonomous underwater vehicle 100 by placing or depositing it in the sea 404 applied. The autonomous underwater vehicle 100 then dives to the school of fish 208 and follow the path 602 , The way 602 could, for example, with the sonogram 500 be calculated. When following the path 602 could the optional variable buoyancy system 124 be filled with water for quick diving. If the autonomous underwater vehicle 100 the school of fish 208 approaching, water can be expelled, and the autonomous underwater vehicle 100 stays at this height. With the help of slats, the movement downwards into a horizontal movement of the autonomous underwater vehicle 100 be transmitted. This can reduce the need to use battery power.

8th illustrates the step of executing 700 the machine-executable instructions 306 through the processor 302 , During the in 7 The section of the method shown can include the steps of 4 be performed. The autonomous underwater vehicle 100 moves through the school of fish and captures them in the captured digital images using the trained machine learning module. The autonomous underwater vehicle 100 can be so deep that there is not enough light or other conditions prevail that prevent the cameras from working effectively over long distances. The optional lights can be used to illuminate the fish and take pictures of the fish in close proximity. While the autonomous underwater vehicle is within the school of fish 208 it can move around and move through the school of fish 208 to sample the different species and possibly the size of the fish. After the statistical school of fish data 318 have been determined sufficiently, the method then goes to that in 8th illustrated about.

9 shows the step of receiving 800 statistical school of fish data by a remote computing device. In this step, the autonomous underwater vehicle 100 the raising of Data from the school of fish 208 ended and is by following the path 602 surfaced. The WLAN or LoRa communication is then with the remote computer device on the fishing vessel 402 over a communication link 802 manufactured. The corresponding computer device is in 9 illustrated.

10 illustrates an example of a school of fish location 900 , The school of fish location system includes the autonomous underwater vehicle 100 and a remote computing device 901 , The remote computing device 901 includes a processor 902 using a communication system 904 of the computer device and a memory 906 the computer device is connected. The communication system 904 the computing device is set up to communicate 802 with the autonomous underwater vehicle 100 build up. The memory 906 is than the statistical school of fish data 318 depicting that from the autonomous underwater vehicle 100 were retrieved. The processor 902 can then from the statistical school of fish data 318 a signal 912 produce. This shows the step of spending 900 of a signal 912 whether or not to catch the school of fish. In this case, the remote computing device includes 901 an ad 908 used to display a user interface 910 is used. In this case, the user interface shows 910 the sonogram 500 of the school of fish and a signal 912 on. In this case the signal picks up 912 the form of an estimate of the amount of commercially available fish and by-catch.

11 illustrates another example of a school of fish location 1000 , In this example is a fleet 1002 available for fishing. Each ship has an autonomous underwater vehicle 100 as well as a remote computing device 901 on. Any of the remote computing devices 901 also has a communication link 802 with the communication system 1006 of a central training server 1004 built up. The communication links 802 are then used to transfer the cumulatively captured digital images 310 used. The central training server 1004 assigns one to the communication system 1006 connected processor 1008 and a memory 1010 on. The digital pictures 310 of the autonomous underwater vehicles 100 are as in store 1010 shown saved. The memory 1010 also includes the trained machine learning module 308 by the autonomous underwater vehicles 100 is used. The memory 1010 is as with the markings and markings 1016 for the digital pictures 310 shown containing. This is then used to train the trained machine learning module to an updated trained machine learning module 1018 to create. After creating the updated trained machine learning module 1018 can then do this via the communication connections 802 to the individual autonomous underwater vehicles 100 be transferred back. The updated trained machine learning module 1018 then replaces the current trained machine learning module.

12 shows a flow diagram illustrating a method for providing an updated trained machine learning module 1018 illustrated. First in step 1200 optionally a trained machine learning module 308 provided. Next in step 1202 the digital images taken 310 from a fleet 1002 of autonomous underwater vehicles 100 added. The captured digital images 310 autonomous underwater vehicles 100 in the sea 404 added. Next in step 1204 Markings and markings 1016 receive. The markings and markings show the position and type of fish in the digital images 310 on. Next in step 1206 the updated trained machine learning module 1018 using the markers 1016 and the digital images 310 generated. If the optional step 1100 the updated trained machine learning module 1018 by training the copy of the trained machine learning module 308 generated. Next in step 1208 the updated trained machine learning module 1018 to at least one of the autonomous underwater vehicles 100 the fleet 1002 transfer. Finally in step 1210 the trained machine learning module from at least one of the autonomous underwater vehicles 100 the fleet 1002 through the updated trained machine learning module 1018 replaced.

Examples can provide a means for real-time underwater detection and location of fish. Data and images can be captured by cameras as well as using sonar data. The data can then be fed into an embedded device. The data can be processed by a modern neural network, such as a CNN-based object detector. The position of the object in a bounding box and the classification are retrieved. The data can then be processed to count fish species, size and other characteristics. The data can be completed by adding relevant characteristics such as bycatch, density, biomass or other statistics. The data can then be saved and later transferred to a large data system on land, which can be used for further training.

13 shows a functional diagram of the autonomous underwater vehicle 100 , In this diagram, box represents 126 one of the cameras. Optionally, a second camera 200 and possibly also a forward-looking sonar system 130-132 to be available. All three of these sensor systems 126 . 200 . 130-132 deliver data to the processor 302 , The processor can be an integrated low energy processor that performs real time processing. The processor then stores the data on an SD card or other storage 304 , This can be used, for example, to save the images or the digital images 310 be used. This can be done through the computer system and the communication link 802 with the database on land, the central training server 1004 , be streamed.

14 shows an example of an implementation of the trained machine learning module 308 , In this example, the trained machine learning module is an object recognition model through deep learning. The digital pictures 310 or images are placed in a feature extractor of a convolutional neural network 1400 entered. This is used to create bounding boxes. The bounding boxes and the image data 310 are then integrated into a neural area and classification network 1402 entered. The output of the modules 1400 and 1402 are the bounding boxes 1404 and the object classifications 1406 , This information then becomes the statistical school of fish data 318 generated. This data can, for example, be a measure of the number of fish counted or their detection 1408 , the estimated catch amount 1410 , a measure of the school of fish and the tracking and tracing of the school 1412 , a size measurement 1414 or include the distribution of sizes and can also include additional statistical measures 1416 include.

15 illustrates the structure and use of the trained machine learning module if it is a convolutional neural network trained through deep learning. The steps are as recording and labeling 1500 shown. The next group of steps is training 1502 , and the last set of steps is deployment 1504 of the trained machine learning module. In the recording and marking phase 1500 are the digital images first 310 collected. This can also include sonar data. Next in step 1016 Bounding frames and markings for the digital images 310 provided. This then becomes a labeled image database 1510 built up. The featured image database 1510 is then used as input for the training phase 1502 used.

In the training phase, the marked image database is then created 1510 preprocessed images 1520 generated. From these, a pre-processed marked database is created 1522 generated. This is then when training object recognition through deep learning 1524 used, which generates the trained machine learning module, which in this case is an object recognition model through deep learning. This object recognition module through deep learning 308 is then in the deployment phase 1504 used. In the deployment phase, the digital images or the unmarked fish image data are first made available. This then has the preprocessing 1530 cycle through to generate preprocessed images. These preprocessed images 1530 are then entered into the object recognition module through deep learning 308 entered. The object recognition module through deep learning 308 then gives the bounding box 1404 and the object classifications 1406 which in this case are the fish species.

Reference list

100

Autonomous underwater vehicle

102

hull

104

Hull length

106

Hull diameter

108

Drive system

110

battery

112

Control system

114

Control system

116

WLAN or LoRa system

118

Sensor package

120

Acoustic modem

122

Drive direction

124

Variable buoyancy system

126

Forward-facing camera

128

lamp

130

Sonar converter

132

Sonar electronics

200

Camera attached to the side

202

Luminaire attached to the side

204

Field of view of the forward-facing camera

206

Field of vision of the side-mounted camera

208

fish

300

Hardware interface

302

processor

304

Storage

306

Machine-executable instructions

308

Trained machine learning module

310

Digital images

312

Selection of one or more types

314

Display the bounding box

316

Identification of fish species

318

Statistical schools of fish

320

Predefined criteria

322

Catch decision

324

Sonar data

326

Swarm position data

350

Recipient of a selection of one or more selected species

352

Control the autonomous underwater vehicle to maneuver through a school of fish

354

Control the camera system to capture the digital images as the autonomous underwater vehicle maneuvers through the school of fish

356

Cumulative calculation of statistical school of fish data by capturing the fish in the digital images and determining the fish species of the individual fish in the digital images

358

Store the statistical school of fish data in memory

400

Locating a school of fish using an echolocator

402

Fishing vessel

404

Sea surface

406

Seabed

408

Path of the sonar waves

412

Direction of travel of the fishing vessel

500

Sonogram

502

Sea depth

504

Minimum depth of school of fish

506

Width of the school of fish

508

Shoal of fish

600

Deployment of the autonomous underwater vehicle

602

Autonomous submarine route

700

Execute machine-executable instructions

800

Transmission of the statistical shoal of fish data to the fishing vessel

802

Communication link

900

School of fish location

901

Remote computing device

902

Computer device processor

904

Communication system of the computer device

906

Computer device memory

908

display

910

user interface

912

signal

1000

School of fish location

1002

Fishing fleet

1004

Central training server

1006

Communication system

1008

processor

1010

Storage

1016

Markings and markings

1018

Updated trained machine learning module

1200

Provision of a trained machine learning module

1202

Collect captured digital images from a fleet of autonomous underwater vehicles that were captured in the sea during fishing operations

1204

Receive markings that indicate a position of individual fish in the captured digital images and / or an association of the species of fish with the individual fish in the captured digital images

1206

Generate the updated trained machine learning module by training a copy of the trained machine learning module using the markers and the captured digital images

1208

Transfer the updated trained machine learning module to at least one of the fleet of autonomous underwater vehicles in the fleet

1210

Replace the trained machine learning module of at least one of the autonomous underwater vehicles of the fleet through the updated trained machine learning module

1400

Feature extractor of the Convolutional Neural Net

1402

Neural area and classification network

1404

Bounding box

1406

Object classifications

1408

Fish counting / capturing

1410

By-catch

1412

School of fish tracking / tracing

1414

Size measurements

1416

Additional statistical measures

1500

Record and mark

1502

Work out

1504

Output

1510

Featured image database

1520

Preprocess images

1522

Pre-processed labeled database

1524

Training object recognition through deep learning

1530

Pre-processed images

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 20170105397 A1 [0005]

Non-patent literature cited

• Handoko et. al., "Designing embedded fish sensor for underwater robot", Indian Journal of Marine Science, Vol. 38 (3), September 2009, pp. 308-315 [0004]

Claims (37)

Autonomous underwater vehicle (100) configured to receive a classification module (308), the autonomous underwater vehicle (100) comprising: a camera system (126, 200) which is set up for taking digital images (310), - a memory (304); and a processor (302) for controlling the autonomous underwater vehicle, wherein the classification module (308) is arranged to capture in the captured images (310) each fish that belongs to a set of predetermined fish species and to assign the relevant fish species to the captured fish (208); wherein the processor (302) is configured to: - Controlling (352) the autonomous underwater vehicle to maneuver through a school of fish (208); - controlling (354) the camera system to capture the digital images (310) as the autonomous underwater vehicle maneuvers through the school of fish; - computing (356) statistical schools of fish data (318) using the digital images (310) and the classification module (308); and - storing (358) the schooling statistical fish data in the memory (304). Autonomous underwater vehicle after Claim 1 , wherein the autonomous underwater vehicle (100) further comprises at least one light source (128, 202), which is set up to illuminate a field of view of at least one camera of the camera system during the recording of the digital images. Autonomous underwater vehicle after Claim 2 wherein the light source comprises one of the following: a visible light source, an infrared light source, and combinations thereof. Autonomous underwater vehicle according to one of the preceding claims; the autonomous underwater vehicle (100) comprising a hull and the hull having a length (104) that is less than one of the following: 1.5 meters long, 1 meter long and 0.5 meters long; wherein the fuselage has a perimeter (106) that is less than one of the following: 80 cm, 50 cm, 20 cm and 10 cm. An autonomous underwater vehicle according to any one of the preceding claims, wherein the processor (302) is arranged to control the autonomous underwater vehicle so that it glides through the school of fish at least part of the time when maneuvering. An autonomous underwater vehicle according to any one of the preceding claims, wherein the processor (302) is arranged to control the autonomous underwater vehicle (100) to use an intermittent drive when maneuvering through the school of fish. An autonomous underwater vehicle according to any one of the preceding claims, wherein the memory (304) comprises a removable memory component and wherein the digital images are recorded on the removable memory component. Autonomous underwater vehicle according to one of the preceding claims; wherein the processor (302) is arranged to: - receiving shoal position data (326) describing a size (506, 508) and a position (504) of the shoal of fish (208); wherein the shoal position data includes one of a depth of the school of fish (208), a size of the school of fish (208), an echogram (500) of the school of fish, and combinations thereof; and Control the maneuvering of the autonomous underwater vehicle by the school of fish, at least in part, using the swarm position data. Autonomous underwater vehicle according to one of the preceding claims, wherein the autonomous underwater vehicle (100) further comprises a sonar system (130, 132), which is set up for recording sonar data (324), wherein the processor (302) is set up to perform one of the following: - controlling the propulsion system to at least partially maneuver the autonomous underwater vehicle (100) through the school of fish (208) using the sonar data; - Completing the calculation of the schooling fish statistical data using the sonar data; and - combinations of these. Autonomous underwater vehicle according to one of the preceding claims, wherein the processor (302) is set up to: - determining a position of the school of fish by capturing the fish (208) in the digital images (310) by the classification module; and - Change the maneuvering of the autonomous underwater vehicle to swim towards the school of fish based on the position of the school of fish determined using the digital images. Autonomous underwater vehicle according to one of the preceding claims, wherein the camera system comprises one of the following: a forward-looking camera (126) with its field of view aligned with a drive direction (122) of the autonomous underwater vehicle; - At least one camera (200) mounted on the side, the field of vision of which is oriented essentially perpendicular to the direction of drive of the autonomous underwater vehicle; and - combinations thereof. An autonomous underwater vehicle according to any one of the preceding claims, wherein the schooling statistical fish data comprises one of the following: - Fish quantity statistics, which is calculated by capturing the fish (208) in the digital images (310); - a fish size statistic, which is calculated by capturing the fish (208) in the digital images (310); - fish species statistics describing the identification of the one or more selected species in the digital images (310); and - combinations of these. Autonomous underwater vehicle after Claim 12 , wherein the autonomous underwater vehicle (100) comprises a communication module configured to communicate with a remote computing device (901), the communication module comprising one of the following: an acoustic modem (120), a WLAN system (116), a LoRa System (116) and combinations thereof. Autonomous underwater vehicle after Claim 13 wherein the processor (302) is configured to transmit (800) the statistical schooling fish data using the communication module. Autonomous underwater vehicle after Claim 14 wherein the processor (302) is arranged to: - calculate a catch decision (322) by comparing the statistical shoal of fish data with a predetermined criterion (320), the catch decision comprising an instruction to catch the shoal of fish (208) or to avoid the shoal of fish; and - appending the fishing decision to the schooling statistical fish data. Autonomous underwater vehicle after Claim 15 wherein the processor (302) is configured to control the autonomous underwater vehicle (100) to hold a predetermined position within the school of fish (208) when the catch decision includes instructing to catch the school of fish after providing the catch decision. Autonomous underwater vehicle according to one of the Claims 1 to 15 wherein the processor (302) is configured to control the autonomous underwater vehicle (100) to maintain a predetermined position within the school of fish (208). Autonomous underwater vehicle according to one of the preceding claims, wherein the calculation of the statistical school of fish data is carried out at least partially during the recording of the digital images (310). An autonomous underwater vehicle according to any preceding claim, wherein the fish species is one of a set of predetermined fish species; wherein the processor (302) is arranged to receive (350) a selection (312) of one or more selected species, the set of predetermined fish species comprising the one or more selected species, and wherein the calculation of the statistical schooling fish data for the one or the multiple selected species is calculated. Autonomous underwater vehicle according to one of the preceding claims, wherein the autonomous underwater vehicle (100) comprises the classification module (308). Autonomous underwater vehicle according to one of the preceding claims, wherein the classification module (308) is implemented as one of the following: a hardware module, an emulation of a trained machine learning module and combinations thereof. Autonomous underwater vehicle according to one of the Claims 1 to 20th , wherein the memory (304) is arranged to store the classification module (308), the classification module (308) being a trained machine learning module. Autonomous underwater vehicle after Claim 22 , wherein the trained machine learning module comprises one of the following: at least one convolutional neural network (1400), a neural network, a deformable template comparison algorithm and a local orientation histogram with a support vector machine algorithm. School of fish location (900, 1000), comprising: - an autonomous underwater vehicle (100) according to one of the Claims 1 to 23 ; and - a remote computing device (901), the remote computing device comprising a communication system (904) configured to establish a communication link (802) with the autonomous underwater vehicle (100), the remote computing device (901) for receiving the statistical school of fish data (318) from the autonomous underwater vehicle (100) via the communication link (802). School of fish location according to Claim 24 wherein the remote computing device (901) is one of the following: a tablet computer, a laptop computer, a computer, a smartphone, and an echolocator. School of fish location according to Claim 24 or 25th , wherein the remote computer system (901) includes a display (908), the remote computer system (901) configured to: - calculate estimated shoal of fish data (912) using the shoal statistical data, the shoal fish data comprising one of the following: an estimated one Number of the one or more selected species in the school of fish (208), an estimated mass of the one or more selected species in the school of fish (208), an estimated size of the fish in the school of fish, an estimated number of by-catch fish in the school of fish (208), one Estimating the mass of by-catch fish in the school of fish (208), an estimated percentage of the one or more selected species in the school of fish (208), and combinations thereof; - Viewing the estimated shoal of fish data on the display (908). School of fish location according to one of the Claims 24 to 26 , wherein the school of fish location system further comprises a central training server (1004), the central training server being set up for: - collecting a sample of the captured digital images (310) from the autonomous underwater vehicle (100), preferably via the remote computer device (901); - receiving markings (1016) indicating a position of individual fish in the sample of the captured digital images and / or an association of the species of fish with the individual fish in the captured digital images; and - generating an updated trained machine learning module (1018) using the markers and the captured digital images. School of fish location according to Claim 27 , the autonomous underwater vehicle according to Claim 22 or 23 wherein the central training server (1004) is further configured to transmit the updated trained machine learning module to the autonomous underwater vehicle (100) via the remote computing device (901), the underwater vehicle configured to install the updated trained machine learning module as the trained machine learning module. School of fish location according to Claim 28 wherein the updated trained machine learning module is generated by training a copy of the trained machine learning module using the markers and the captured digital images (310). School of fish location according to Claim 28 or 29 wherein the updated trained machine learning module is generated by training an untrained machine learning module using the markers and the captured digital images. Method of reducing by-catch using a school of fish location system (900, 1000) according to one of the Claims 24 to 30th The method comprising: - locating (400) a school of fish (208); - deploying (600) the autonomous underwater vehicle; - executing (700) the machine-executable instructions by the processor of the autonomous underwater vehicle; - receiving (800) the schooling statistical fish data via the remote computing device; and - outputting (900) a signal (912) whether the school of fish should be caught. A method of reducing by-catch using a school of fish location system (900, 1000), the school of fish location system (900, 1000) comprising: an autonomous underwater vehicle (100) with a camera system (126, 200) and a classification module (308) and - a remote one Computer device (901) and the method comprising the steps of: - locating (400) a school of fish (208); - The autonomous underwater vehicle (100) is deployed (600); - The autonomous underwater vehicle (100) maneuvers through the school of fish (208); - The camera system (126, 200) takes digital images (310) while the autonomous underwater vehicle (100) maneuvers through the school of fish (208); - the classification module (308) in the captured images (310) captures each fish belonging to a set of predetermined fish species and assigns the fish species in question to the captured fish (208); - the classification module (308) calculates statistical schooling fish data (318) using the assignments of fish species to the recorded fish (208); the calculated statistical school of fish data (318) are transmitted from the autonomous underwater vehicle (100) to the remote computer device (901) and - the remote computing device (901), in a human-perceivable form, outputs a signal indicative of the shoal statistical fish data (318). Method of reducing bycatch after Claim 31 or 32 the method further comprising: controlling the autonomous underwater vehicle to hold a predetermined position within the school of fish when the school of fish is caught; and - retrieving the autonomous underwater vehicle by catching the school of fish with a net if the underwater vehicle is to hold the predetermined position within the school of fish. A method for providing an updated trained machine learning module, the method comprising: collecting (1202) a sample of recorded digital image data (308) from a fleet of autonomous underwater vehicles (1002) Claim 22 or 23 , the captured digital images being captured by the autonomous marine underwater vehicles during the capture processes; - receiving (1204) markers (1016) indicating a position of fish in the sample of the captured digital images and / or an association of the species of fish with the fish; - generating (1206) the updated trained machine learning module (1018) using the markers and the captured digital images; - transferring (1208) the updated trained machine learning module to at least one of the fleet of autonomous underwater vehicles, the updated trained machine learning module being trained to capture fish in the digital images and to associate a fish species with the captured fish in the digital image; - Install (1210) the updated, trained machine learning module in at least one of the fleet of autonomous underwater vehicles. Method for providing an updated trained machine learning module according to Claim 34 The method further comprising providing (1200) an untrained machine learning module (308), the updated trained machine learning module being generated by training an untrained machine learning module using the markers and the captured digital images. Method for providing an updated trained machine learning module according to Claim 34 the method further comprising providing (1200) a trained machine learning module (308), the updated trained machine learning module being generated by training a copy of the trained machine learning module using the markers and the captured digital images. Method for creating a classification module (308) using a fleet of autonomous underwater vehicles (1002), each comprising a camera system (126, 200), the method comprising the steps of - Maneuver the autonomous underwater vehicles (1002) through schools of fish (208); - The camera systems (126, 200) record digital images (310); - a sample of captured digital image data (308) is collected (1202); - markers (1016) each indicating a position of a fish belonging to a set of predetermined fish species and the fish species concerned of that fish are received (1204); - the classification module (308) is generated (1206) using the received markings (1016) and the recorded digital images (310), so that the generated classification module (308) is arranged to recognize each fish belonging to the set of fish in a digital image (310) and to assign the relevant fish species to the detected fish (208); and - The generated classification module (308) is installed on board at least one autonomous underwater vehicle (1002) of the fleet.

Images