NO348007B1

NO348007B1 - A system for monitoring of dead fish

Info

Publication number: NO348007B1
Application number: NO20201382A
Authority: NO
Inventors: Even Bringsdal; Patcharee Thongtra
Original assignee: Createview As
Priority date: 2020-12-16
Filing date: 2020-12-16
Publication date: 2024-06-17
Also published as: WO2022129362A1; NO20201382A1

Description

TECHNICAL FIELD

The present invention relates to a system and a method for making observations of aquatic animals. More particularly, the present invention relates to a system and a method which provides real-time observations of dead fish, the observations including counting, tracking, weight estimation and detection of welfare indicators like deformity, wound(s), scale loss, lice or the like by using optical setup, image and/or geometrical processing methods.

BACKGROUND

Intensive production of fishes, i.e. breeding of fishes, is a rapidly growing industry. It is anticipated that its importance will grow strongly in extent. In connection with the intensive production of fishes, the recording of number of fishes and the size classification of fishes are frequently required.

Sale of fish is usually done based on number and, as a sale may comprise several hundreds of thousands, the execution of a rapid and accurate recording of fish number (i.e. live fish) becomes of great importance. Furthermore, in order to achieve the largest possible growth, it is important to regularly carry out size classification of the live fishes, such that the variation of size of fish in different storage arrangements, for instance a net pen, fish tank or the like, is kept at a limited amount, as the individual fishes grow with varying speed and larger fishes will restrict the growth of smaller fishes kept in the same storage arrangement.

Today there are several methods for the counting and size classification of live fish. Usually, the counting of fishes is performed manually on-site observation.

However, several systems for the automatic recording of number have emerged recently. These automatic counting systems require a distance between adjacent fishes, in order to achieve an accurate recording of number. This requirement restricts the capacity of the systems substantially. There are also known to use different methods for the size classification of fish, where these methods are based on thickness of the fish, the relation between the thickness and size of the fish or the like.

Both when recording the live fish number and when classifying fishes by size, the live fishes have to be treated as leniently as possible.

However, none of the known systems and/or methods have dealt with observations and/or analysis of dead fish.

However, none of the known systems and/or methods have engaged in observations and/or analyzes of dead fish. The lack of good and/or adequate observations of dead fish has resulted in, due to lack of supervision, in a number of large-scale deaths of fish worldwide, which brings a serious burden to the national economy and greatly restricts the development of aquaculture.

From the above it appears that the cause of fish death could also play an important role in the management of aquaculture.

WO 2014/098614 A1 relates to a method and system for calculating physical dimensions for freely movable objects in water, by illuminating the object or projecting a known or selected light pattern on the object by means of at least one light source at selected wavelength or wavelengths, recording illuminated object or objects with the projected light pattern by means of recording means in the form of at least one 2D camera provided with filters arranged to only accept light having selected wavelengths or provided with image producing sensors arranged to only collect light with selected wavelengths, including generation of a 3D model based on recorded images and/or video from the recording means as basis for calculating the physical dimensions.

GB 2.571.003 A relates to a method or apparatus for collecting marine life, where a submersible device for collecting fish or other water creatures comprising a suction system and a porous collection means to contain the fish and permit water and undesirable materials to pass through There is a method of using the device to collect dead or dying salmon, such as those affected by sea-lice, in aquaculture enclosures or fish farms. The device may have propulsion means such that it can be maneuvered in three dimensions. The device may be attached or retrofitted to an underwater vehicle, which may be autonomous, or user controlled via an umbilical or wirelessly. There may be a camera to identify fish or other structures within an aquaculture enclosure. The suction system may comprise an educator powered water jets fed by a subsea or surface pump.

WO 2018/061926 A1 relates to a counter system and counting method, where a counter system is provided with a plurality of photograph devices and an information processing device. The plurality of photograph devices are arranged in a predetermined photograph space with a space in the height direction therebetween. The information processing device measures the number of objects to be measured in the photograph space on the basis of captured images by the photograph devices.

M. A. R. PINHEIRO, Master Thesis University of Stavanger, 2018, «Identification of Innovative Improvements for Aquaculture Sector Using Arising Technologies» discloses new technologies such as internet of things, machine learning, 3D printing, virtual reality and among others, that are starting to transform industries in different sectors, already calling this phenomenon as the 4 Industrial Revolution or Industry 4.0, and as any industrial revolution the companies must be aware of the importance to use these technologies to maintain or improve their position in the market.

The Norwegian Aquaculture sector was chosen for identify possible innovations, due to its growth and its importance for future generation due to the fast growth of the world population, and also given that the Norwegian aquaculture can be considered as a conservative sector there is a lot of space for improvements.

Innovative ideas were generated by utilizing a method, which identified improvements opportunities in the services, products and segments of salmon farming sector applying the arising technologies. After this process the ideas generated were evaluated in problem solving potential, economic potential and patent protection to classify the ideas with most potential of developing a business. A few ideas presented very good potential making possible the idea of developing a business, showing that the method utilized can be a model to identify opportunities for new undertakings.

WO2019002880 A1 discloses a method and apparatus for collecting and/or preprocessing data related to feeding animals in water. More particularly, the present invention relates to a method and apparatus for minimising wasted feed used in a fish farm. According to an aspect, there is a provided a computer-implemented method for detecting motion in relation to one or more aquatic animals, the method comprising the steps of: receiving sensor data; determining from the sensor data one or more moving objects using one or more learned functions; and generating output data in relation to the determined one or more moving objects.

CN 111738139 A discloses a cultured fish monitoring method based on image recognition. The cultured fish monitoring method comprises the following steps: monitoring cultured fish; constructing and training a YOLO network model; obtaining dead fish identification model weight; collecting a plurality of cultured fish images; splicing the plurality of cultured fish images into a complete cultured fishimage; calling the dead fish recognition model weight from the trained YOLO network model and inputting the dead fish recognition model weight into the complete cultured fish image for real-time detection; storing the real-time detection result in a storage device; displaying the real-time detection result on an image interface, calculating the survival rate of the cultured fishes, and/or uploading the survival rate of the cultured fishes to a cloud database. The method and the system can monitor the survival conditions of the cultured fishes on the water surface and the water bottom of the culture pond in real time, and can monitor the survival states of the cultured fishes in real time more quickly, more accurately and more intelligently.

A. BOCHKOVSKIY et al., 2020, «YOLOv4: Optimal Speed and Accuracy of Object Detection» relates to features which are said to improve Convolutional Neural Network (CNN) accuracy. Practical testing of combinations of such features on large datasets, and theoretical justiﬁcation of the result, is required. Some features operate on certain models exclusively and for certain problems exclusively, or only for small-scale datasets; while some features, such as batch-normalization and residual-connections, are applicable to the majority of models, tasks, and datasets. We assume that such universal features include Weighted-Residual-Connections (WRC), Cross-Stage-Partial-connections (CSP), Cross mini-Batch Normalization (CmBN), Self-adversarial-training (SAT) and Mish-activation. We use new features: WRC, CSP, CmBN, SAT, Mish activation, Mosaic data augmentation, CmBN, DropBlock regularization, and CIoU loss, and combine some of them to achieve state-of-the-art results: 43.5% AP (65.7% AP50) for the MS COCO dataset at a realtime speed of 65 FPS on Tesla V100. Source code is at https://github.com/AlexeyAB/darknet.

There is thus a need for alternatives to today's measuring and counting systems and methods, or at least supplementary solutions for such measuring and counting systems and methods.

An object according to the present invention is to remedy at least one of the abovementioned disadvantages or problems.

Another object according to the present invention is to provide a system and method for making remote measurements of dead fish which indirectly will ensure better fish welfare.

SUMMARY OF THE INVENTION

These objects are achieved according to the present invention with a system and a method for making remote observations and measurements as defined in the independent claims 1 and 6. Further embodiments of the invention are defined in the dependent claims.

When dead fish in a fish pen is to be counted, this is obtained by a manual operation. The dead fish from the net pen is pumped out from a bottom of the net pen and is transferred to a counting place where a number of people will count the dead fish. This operation is time-consuming, tedious and tends to be error prone.

An automatic system for monitoring of dead fish will gain more accuracy and save worker-hours. However, such an automatic system for monitoring of dead fish faces some challenges, such as the fast movement of multiple dead fish at a pumping counter, the difference in shape and size of the dead fish, and the overlapping issues caused by the movement.

The software for such an automatic system for monitoring of dead fish in real-time may be written in Python. However, it is to be understood that the software could be written in other languages, for instance R, C/C++, Java or the like. These programming languages are well suited for Machine Learning (ML).

The present invention relates to a system for monitoring of dead fish in real-time, where the system comprises :

- a pumping system equipment for pumping the dead fish from the bottom of a net pen;

- at least one image acquisition device;

- a detection and tracking unit,

- a communication unit adapted for sending and receiving data from the pumping system equipment,

- a data uploading unit,

wherein the data uploading unit is adapted to upload detected dead fish data and images into a storage unit;

characterized in that:

- the system comprises a straining box for receiving the dead fish from the pumping system equipment;

- the system comprises a weight estimation unit;

- the at least one image acquisition device is mounted above the straining box and is capturing images of the dead fish passing the image acquisition device; wherein the data uploading unit is adapted to upload cage identification and weight and length of the dead fish onto a storage unit.

According to one aspect of the present invention, the image and acquisition device may comprise a stereo camera. However, it could also be envisaged that the system for monitoring of dead fish according to the present invention could comprise more than one camera, for instance two or more cameras, where the cameras could be arranged with a distance between them and where each camera could be arranged to collect different data, to take images with different speed, rate, resolution etc. The cameras could be of same or different type.

According to one exemplary embodiment the detection and tracking unit may comprise a detector and a tracker. The detector may, for instance, be a YOLOv4 detector. The tracker may, for instance, be a Kalman filter.

The YOLOv4 detector may, for instance, be divided into three parts: backbone, neck and head. The function of each part is different. The backbone part will mainly extract the features, the neck part is used to combine the features extracted from the main part and the function of the head part is to predict, including predicting the bounding boxes and the object classification.

The YOLOv4’s model may also be improved, for instance, in relation to loss function. The method uses IOU (Intersection over Union) to determine the degree of overlap between the predicted and the ground-truth bounding box. IOU is represented as follow

where M is the prediction bounding box, represented by (xcenter, ycenter, w, h). N is the ground-truth bounding box (x, y, w, h). However, the optimization method is not always able to optimize non-overlapping parts. Therefore, a generalized IOU (GIOU) is implemented, represented as follow:

where A represents the minimum bounding box between the predicted bounding box and the ground-truth bounding box. U is the union of the predicted and the groundtruth bounding boxes, i.e. M∪N. The bounding box regression loss function used will be:

LOSS = 1 - GIOU

This generalized loss function not only pays attention to the overlapping area, but also focuses on the non-overlapping area of the two kinds of boxes which better reflects the overlap of the two boxes.

The present invention also relates to a method for monitoring of dead fish, where the method comprises the following steps:

- pumping dead fish from a bottom of a net pen and thereafter transferring it to a straining box for draining off water;

- capturing images of the dead fish passing an image area localized over water by using at least one image acquisition device mounted above the straining box;

-analysing images from the at least one image acquisition device, which characterize death condition;

-by means of the identified features, form a registry of each dead fish having passed the at least one image acquisition device, and store in a storage unit; and -record the features characterizing the death condition for each individual.

DETAILED DESCRIPTION

Other advantages and features of the invention will become apparent from the following detailed description, the accompanying drawings and the following claims, where

Figure 1 is a schematic figure of the different components of a system for monitoring of dead fish in real-time according to the present invention,

Figure 2 shows in a schematic way how the system for monitoring of dead fish in real-time can be arranged and used and

Figure 3 shows how the system for monitoring of dead fish in real-time may be used for counting of lice on fish.

Figure 1 shows the different components of the system for monitoring of dead fish in real-time according to the present invention, where the system comprises an image acquisition device 1, a detection and tracking unit 2, a communication unit 3, a weight estimation unit 4 and a data uploading unit 5.

The image acquisition device 1 may, for instance, be a stereo camera.

The image acquisition unit 1 is used to capture images at a pumping counter in realtime. The image acquisition device 1 used an SDK (Software Development Kit) to control the parameters of the camera, for instance the resolution, frame rate, exposure, gain, brightness, contrast or the like, to calibrate the camera as well as to capture images. Each image will thereafter be processed to generate depth maps or raw data, where distance values (Z) for each pixel (X, Y) in the image are stored. These depth maps or raw data are used to calculate a distance (in metric units) of each dead fish in the image to the camera.

The stored depth maps will be processed by the detection and tracking unit 2, where the detection and tracking unit 2 use deep learning to detect the dead fish in the image. Such deep learning or also Deep neural network (DNN), is a multi-layered neural network that is modelled to work like human brain, i.e. able to learn from large amounts of data. Deep learning algorithms in each layer of the neural network will perform calculations and make predictions repeatedly, thereby progressively learning and improving the accuracy of the outcome over time.

Depending on the application areas for the system for monitoring of dead fish in real-time according to the present invention, it is to be understood that many different types of DNNs may be used. According to the present invention, a conventional neural network may be used, in order to obtain a real-time image classification and object detection. The neural network will check a captured image by blocks, starting from the left upper corner and moving further pixel by pixel up to a successful completion.

Then the result of every verification is passed through a convolutional layer, where some or specific data elements have connections while others don’t. Based on this data, the system can produce the result of the verifications and can conclude what object is in the picture (image classification) and also identify the location of object (object detection).

In one embodiment according to the present invention, YOLOv4 could be used to detect the dead fish in the real-time. YOLOv4 is designed to be a super-fast object detector for production systems. YOLOv4 may be trained offline with all dead fish images that are collected.

Such training of YOLOv4 will also result in that other parameters from the processed image may be analyzed, such as wounds on the fish, deformity of the fish, state or condition of the fish, in order to be able to determine a possible death cause of the fish.

The size distribution of the dead fish may, for instance, may indicate this is a small or large fish. If the fish is small, it can, as one possibility, be considered that the fish has not eaten, i.e. starved to death.

The detection and tracking unit 2 comprises further Kalman filtering techniques to track individual dead fish motion. It can be large differences in velocity and movement of the dead fish between different locations where the dead fish are counted, and according to the present invention one or more specific image datasets from one location may be used to obtain better accuracy. This can be done by tuning parameters in the Kalman filter in an offline manner.

A communication unit 3 is thereafter used to send and receive data from a pumping system equipment located at each fish pen, such that an identification of which fish pen the data is received from and what the number of dead fish is. The communication unit 3 will also send the total of dead fish from all fish pens to a pumping system screen.

A weight estimation unit 4, where the weight estimation unit 4 uses another CNN-based architecture, so-called Mask RCNN, is thereafter used to solve dead fish instance segmentation. The segmentation model masks each dead fish in the image. With the distance value received during the image acquisition and masked area, the system for monitoring of dead fish in real-time is able to estimate an actual size and weight of the dead fish.

A data uploading unit 5 is thereafter used to upload processed information of detected dead fish, i.e. captured and/or pumped timestamp, cage identification, weight etc., as well as their images into Google BigQuery and Google Cloud respectively.

The images are also used for training the dead fish detector and segmentation model.

When the system for monitoring of dead fish in real-time has performed the above described processing of the data, the results, i.e. the number of dead fish and other clean fish, the average weight, images of each individual fish etc., may be visualized through Web interface. According to one embodiment the web interface has been implemented by React and Firebase. However, it is to be understood that also other Web Application Frameworks may be used, for instance Angular, Vue, Ember etc.

Figure 2 shows how the system for monitoring of dead fish 8 in real-time can be arranged and used, where it can be seen that the dead fish is/are pumped up from a bottom of a fish pen 6 and thereafter transferred to a straining box 7 (a box used to drain off water) through a hose or pipe. An image acquisition unit 1 mounted above the straining box 7 will thereafter take an image of the dead fish 8 or dead fishes passing the image acquisition unit 1.

Before an image of the dead fish is taken, the SDK will be used to adjust or set different parameters of the camera 1.

Each of the images taken by the image acquisition unit 1 will be processed to generate depth maps or raw data, in order to store distance values for each pixel in the image, such that a distance can be calculated.

The stored depth maps or raw date are thereafter sent to the detection and tracking unit 2, where the detection and tracking unit 2, based on deep learning, is used to detect the dead fish in each image.

Meanwhile, the communication unit 3 will receive data from a pumping system equipment connected to each fish pen, such that it can be confirmed from which fish pen the dead fish is collected and counted.

If, for instance, the fish farm comprises eight fish pens, the communication unit 3 will receive data from the first fish pen and the number of dead fish in this first net pen, data from the second net pen and the number of dead fish in the second fish pen and correspondingly for the remaining fish pens in the fish farm. The total of dead fish from the fish farm is thereafter visualized on a pumping system HMI screen.

Data for the different fish pens/net pens and the corresponding number of dead fish may also be set manually from the pumping system screen.

Furthermore, the weight estimation unit 4 is used to calculate dead fish instance segmentation in order to estimate the actual size and weight of each dead fish.

Information of detected and counted dead fish, fish pen identification, size, weight, deformity and images are sent to a data uploading unit 5.

The system for monitoring of dead fish in real-time will therefore comprise the steps of passing dead fish over an image area where the image acquisition unit 2 is arranged in order to capture images of the dead fish, analyzing the images in order to confirm the possible death condition, and by means of the identified death condition to form a registry of each dead fish and thereafter store the data in a storage unit.

The system for monitoring of dead fish in real-time according to the present invention may also be used for counting of lice on fish. In such an application of the system for monitoring of dead fish in real-time, the lice is counted before the fish are treated for lice. When the fish have been treated for lice, the lice are again counted in order to be able to adjust the treatment of fish.

Figure 3 shows a renewed or repeated counting of lice after the lice have been counted a first time and the fish thereafter have been treated for lice, where the system for monitoring of dead fish in real-time is used to count lice on each fish passing the image acquisition unit 1.

The fish may be treated for lice through methods using flushing or rinsing, thermal methods (i.e. use of heated water) or the like.

If, for instance, the number of lice after the renewed or repeated counting of lice is considered to be high, then these data may be used to adjust the pressure, temperature etc., during the treatment of the fish, in order to improve the treatment.

The system for monitoring of dead fish in real-time according to the present invention will therefore provide data such as weight, size distribution, operational welfare indicators, etc., where these data on dead fish are stored in a database for future analysis and learning of production.

As such, the system and method for monitoring of dead fish in real-time may be used to estimate cause of death, welfare and/or health status of fish in a fish pen.

Furthermore, such monitoring or scan of dead fish may be used with monitoring or scan of live fish in order to get an overall picture of the fish pens in the fish farm. This may be important as monitoring or scan of live fish most often see fresh fish and not the fish that is about to die, as this fish usually is swimming in the upper layer of the fish pen.

The invention has now been explained with several non-limiting examples. One skilled in the art will appreciate that a variety of variations and modifications can be made to the system and the method for monitoring of dead fish in real-time as described within the scope of the invention as defined in the appended claims.

Claims

1. A system (S) for monitoring of dead fish in real-time,

the system (S) comprising:

- a pumping system equipment for pumping the dead fish from the bottom of a net pen (6);

- at least one image acquisition device (1);

- a detection and tracking unit (2),

- a communication unit (3) adapted for sending and receiving data from the pumping system equipment,

- a data uploading unit (5),

wherein the data uploading unit (5) is adapted to upload detected dead fish data and images into a storage unit;

characterized in that:

- the system (S) comprises a straining box (7) for receiving the dead fish from the pumping system equipment;

- the system (S) comprises a weight estimation unit (4);

- the at least one image acquisition device (1) is mounted above the straining box (7) and is capturing images of the dead fish passing the image acquisition device (1);

wherein the data uploading unit (5) is adapted to upload cage identification and weight and length of the dead fish onto a storage unit.

2. The system (S) according to claim 1, characterized in that the image acquisition device (1) comprises a stereo camera.

3. The system (S) according to claim 1, characterized in that the detection and tracking unit (2) comprises a detector and a filter.

4. The system (S) according to claim 3, characterized in that the detector is a YOLOv4 detector.

5. The system (S) according to claim 3, characterized in that the filter is a Kalman filter.

6. A method for monitoring of dead fish in real-time, the method comprising the following steps:

- pumping dead fish from a bottom of a net pen (6) and thereafter transferring it to a straining box (7) for draining off water;

- capturing images of the dead fish passing an image area localized over water by using at least one image acquisition device (1) mounted above the straining box (7); -analysing images from the at least one image acquisition device (1), which characterize death condition;

-by means of the identified features, form a registry of each dead fish having passed the at least one image acquisition device (1), and store in a storage unit; and -record the features characterizing the death condition for each individual.

7. The method according to claim 6, where the step of analysing images comprises detection and determination of welfare indicators such as scale loss, wound(s), short gills, deformity or the like.

8. Use of the system (S) and method for monitoring of dead fish in real-time according to claims 1-7 for counting of lice on fish.

9. Use of the system (S) and method for monitoring of dead fish in real-time according to claims 1-7 for adjusting parameter(s) for lice treatment,

wherein the number of lice is counted before the fish was treated for lice and after the fish was treated for lice, the number of lice after treatment indicating whether the treatment method needs to be improved by adjusting treatment parameters such as pressure or temperature.

10. Use of the system (S) and method for monitoring of dead fish in real-time according to claims 1-7 to estimate cause of death, welfare and/or health status of the fish in the fish pen (6).