WO2024108318A1

WO2024108318A1 - System and method for identifying pelagic species outside the water

Info

Publication number: WO2024108318A1
Application number: PCT/CL2023/050100
Authority: WO
Inventors: Mauricio URBINA FONERON; Sergio Neftalí TORRES INOSTROZA; Ariel Patricio TORRES SÁNCHEZ; Vincenzo Alexo CARO FUENTES; Danny Alberto LUARTE CANTO; Sebastián Eugenio GODOY MEDEL; Jorge Edgardo PEZOA NUÑEZ
Original assignee: Universidad De Concepcion
Priority date: 2022-11-24
Filing date: 2023-11-03
Publication date: 2024-05-30
Also published as: CL2022003307A1

Abstract

The present invention relates to a system for identifying pelagic species outside the water that comprises (a) an illumination system that illuminates one or more pelagic species to be analysed with controlled light; (b) an optical filter set for selecting spectral bands; (c) an arrangement of cameras with one or more wavelength ranges that record broadband or narrowband images of the illuminated pelagic species by means of optical filters; (d) an analysis unit connected to a database of pelagic species, wherein said unit comprises at least one processor configured to receive the images, compare spectral and morphological particularities that distinguish the illuminated pelagic species with records available in the database, and identify the illuminated pelagic species according to similarity criteria with the records available in the database of pelagic species. The invention further relates to a method for identifying species.

Description

A SYSTEM AND METHOD FOR THE IDENTIFICATION OF PELAGIC SPECIES OUT OF WATER.

Technical Sector

The present invention is related to the fishing industry, more particularly, it is related to a system composed of an apparatus and a method that allows digitally acquiring, measuring, counting, identifying and classifying the different species that are unloaded from fishing vessels.

Previous Technique

Currently in the fishing industry, many countries establish fishing quotas to better manage their marine resources, however, the supervision and control so that these quotas are met is difficult to carry out.

Organizations in charge of quota compliance must supervise, control and register fisheries subject to catch quotas. In some cases, fishing landings must be inspected on the ground by inspection personnel. The inspection process is not simple because the volumes of fishing extracted by boat can be hundreds or thousands of kilograms of pelagic species, the number of daily landings is high and the number of inspectors on the ground is limited.

The analysis carried out on landings usually consists of the in-situ analysis of less than a dozen fishing samples taken in buckets of approx. 3 liters. The samples are analyzed manually to count the individuals of each species within the buckets and then, once all the resources have been weighed, the percentage contribution of each one is calculated with respect to the total weight of the sample and this value is assumed as the distribution of species to be landed. This procedure is imprecise, it is not automated, it does not have traceability, and it does not allow the associated error to be estimated, since the fishing in the volume analyzed from the buckets is not necessarily representative of the total fishing of each landing.

Below, the most relevant scientific and technical alternatives are reviewed that at first glance present a certain degree of similarity to the system for the classification of fish species described in this document, but that do not solve the same problem or do not do so with the same technique.

Regarding classification out of water, the publication “Automated measurement of species and length of fish by computer vision" by White et al. proposes computer vision methods to distinguish certain species of fish and determine their size. For this, it is determined the orientation of the fish using an invariant moment method, and identified with high precision the typology of the fish: flat or round, from relationships between the intensities of certain pixels in the image and its length. The study uses only fish with a high aspect ratio and, therefore, its performance is not guaranteed in species with different morphological characteristics. This technology solves this problem by considering global features in the training of the neural network. The above method is not applicable in situations where leash conditions are constantly changing (lighting, dirt, cuts, etc.), and it is not at all useful when the fish come in bunches, which hinders many more traditional segmentation methods such as edge detection plus thresholding or other morphological operations, (DJ White, C. Svellingen, NJC Strachan, “Automated measurement of species and length of fish by computer vision” Fisheries Research, Volume 80, Issues 2-3, September 2006, Pages 203 -210.)

On the other hand, the publication “Fish species recognition using computer vision and a neural network” by Storbeck et al. describes that fish species recognition systems have been developed based on a computer vision system and neural networks. In this case, the vision system measures a number of characteristics, including width and height at various locations along the body of the fish as it is transported by a belt. However, the model used for species classification is based on a traditional neural network architecture with backpropagation, which integrates length and width information from various areas of the fish as its input features. The measurement of length and width is carried out based on the distortion measurement provided by a helium-neon laser integrated into the vision system, (Frank Storbeck, Berent Daan, “Fish species recognition using computer vision and a neural network” Fisheries Research, Volume 51, Issue 1, April 2001, Pages 11 -15).

The publication "Classification of Fish Species with Augmented Data using Deep Convolutional Neural Network" by Montalbo etal., describes the use of convolutional neural networks (CNN) for fish classification. The VGG16 architecture is used and transfer learning is performed to adapt the classifiers to the existing fish species on the Green Island of Thailand. However, currently, classifiers of this style are no longer used since they impact the processing capacity of the system (in FPS). The more detections there are and without applying some method to parallelize the classification of each bounding box that is generated, the FPS drops too low to perform real-time processing. Additionally, it is not being applied in any industrial environment that uses conveyor belts or any particular vision system, they are only concerned with the classification method (F. J. P. Montalbo and A. A. Hernandez, "Classification of Fish Species with Augmented Data using Deep Convolutional Neural Network, " 2019 IEEE 9th International Conference on System Engineering and Technology (ICSET), Shah Alam, Malaysia, 2019, pp. 396-401, doi: 10.1 109/ICSEngT.2O19.8906433).

Patent application W02008/056988(A1) describes a method and a system for automatic classification of fish underwater, comprising a camera system that takes images of fish and an analysis unit ready to receive and analyze each image to find the outline of each fish. Furthermore, this method involves dividing the fish into a certain number of segments and analyzing the color in each segment, where the characteristic features are compared with the data of the fish in a knowledge database for the classification of fish with the help of analysis. discriminatory. However, this patent uses basic analytical methods, such as discriminant analysis, which are lagging in performance compared to current complex methods, which use deep learning. Nor does it make use of additional information such as that provided by specific spectral bands. It is focused on underwater measurements.

Patent application US2012/306644(A1) describes a fish identification device consisting of a housing, a touch screen that allows the user to view and operate functions, an internal processor operated by software, an internal GPS, a camera with a lens and flash that photographs a fish, an internal memory and an operation software. The device allows the user to determine local fishing regulations, photograph a fish, identify it, and then save the information to internal memory for later use. However, this technology requires the manual use of a device to identify individual fish, therefore, it is not applicable to the problem of monitoring landings.

It should also be noted that phone applications such as Google Lens, which use a computer's camera to recognize various objects when searching on Google Search Engine. This type of application can be used to recognize pelagic species, but the results it provides are not consistent because they are strongly affected by factors such as luminosity, camera focus, simultaneity of objects, among others.

Therefore, a technology is required that allows analyzing the fish taken from fishing vessels, taking a greater number of samples in a shorter time, in addition to maintaining more objective and consistent comparison standards and that allows generating traceability and registration of the species analyzed with better quantification of the results.

Brief description of the invention

The present technology refers to an apparatus, a method and a system for identification and classification of pelagic species at the fishing landing point, which more particularly consists of an automated multi-camera vision apparatus, a method for counting, recording, classification and biomethco analysis of fishing and a protocol for integrating both into a single system.

This system is composed of a device that includes a gantry with embedded computer vision technology for multimodal digital video acquisition and a method based on machine learning and image processing and analysis programs to discriminate different species of fish such as, for example, but not limited to, small pelagic species and their accompanying fauna in artisanal fishing and other forms of industrial fishing.

Description of the figures

Figure 1 shows an internal diagram of the device and the system, whose main components are: a gantry where the acquisition system (100) is installed that contains an array of cameras (101) with optical spectral filters, a remote communication router ( 102) that packages the information acquired by the module and transmits it securely over the Internet to the user. The acquisition requires a controlled lighting system (103), and must be ventilated (104) for correct operation. The acquired and transmitted information is received in an image processing system (200) where the method of identification, counting and classification of the samples is implemented.

Figure 2 shows a diagram of the implementation of the method within the system (200) whose main components are a module for receiving images transmitted from a gantry (201), a deep convolutional neural network that identifies and classifies the species in the image. captured (202). A false identification discard algorithm (203) that automatically eliminates false positives from the sample and those that remain are recorded by a system (204) and stored in parallel (205). Finally, the video feed with the overlay of the identified relevant data is displayed to the user by a user interface (206).

Figure 3 shows an illustration of the diagram of the operating process of the present technology applied to the example of fish landing. The gantry captures broadband and narrowband images with the camera array. These images are processed and generate a report that is shown to the end user, where (a) corresponds to user interface software, reports and evidence; (b) classification algorithm; (c) cameras and optical filters; (d) preprocessing and control software; (e) controlled lighting

Figure 4 shows a graph with spectral fingerprints obtained from complete anatomy images of the example of fish whose species are of interest for national fishing, where (a) corresponds to sardine, (b) to anchovy, (c) to mote , (d) horse mackerel, (e) hake and (f) pippin. Specifically, Figure 4A shows a graph with spectral fingerprints from full anatomy images of species of interest and Figure 4B shows a graph with spectral fingerprints from images of the lateral zone of the species.

Figure 5 shows a schematic of the deep convolutional neural network architecture of the present technology to discriminate the species of interest, where (I) corresponds to the input with (a) narrow band segmented images for N bands; (II) feature extraction stage with (b) 2D convolutional layer and (c) repeating 2D Maxpooling sequentially; (III) classification stage with fully connected networks; and (IV) output stage that delivers classification results

Figure 6 shows a representation of the camera system and lighting system of an exemplary embodiment of the present technology. Specifically, in (a) the system of multiple cameras and optical filters is shown; (b) shows the controlled lighting system and (c) shows the ventilation system required for the operation of the embedded opto-electronics of the gantry.

Figure 7 shows a confusion matrix of the validation results carried out by certifiers.

Detailed description of the invention

The present technology refers to an integrated system that incorporates a specialized apparatus and a method that facilitates the identification and classification of objects that pass through the field of vision of the apparatus such as, for example, but not limited to the classification of pelagic species passing through a conveyor belt after disembarkation. Under this scenario, the integrated system comprises an automatic vision system that illuminates one or more pelagic species to be analyzed with controlled or structured light for the counting, registration, classification and biometric analysis of pelagic species, but which can be easily extended to new situations or configurations.

Specifically, the system comprises computer vision technology, machine learning methodologies integrated with image processing techniques designed to discriminate and analyze different objects passing through their field of view, particularly but not exclusively, to analyze different pelagic species, especially, small pelagic species and their accompanying fauna in artisanal sardine and anchovy fishing in different regions of Chile and the world.

Advantageously, this technology automatically discriminates the objects or species of interest, generating a representative sampling of the composition of the analyzed species, for example, but not limited to fishing landings of pelagic species, in addition to providing statistics of relevant biometric estimates of the fishery such as , for example, but not limited to the sizes and weight of the classified specimens.

The specialized computer vision apparatus has a controlled lighting system of the preferred type, controlled arrangement of LED panels or arrangement of one or more cameras at one or more wavelengths, with fixed or mobile optical filters for the acquisition of multiple spectral bands of interest and embedded computer vision systems.

The infrastructure of the device generates lighting conditions specifically selected for the application in which the device is being used, such that it enables the acquisition of broadband or high-band images. narrow according to the application, through the use of optical filters that highlight the discriminatory spectral and morphological differences of the species to be classified, independent of the external lighting conditions. Depending on the conditions of the acquisition environment, the device can have uniform or structured lighting in order to optimize discrimination between the different classes to be identified.

The optical filters are selected in tune with the spectral bands that promote the contrast difference of the species in the images taken by the infrared spectrum or visible spectrum camera array, and may or may not be essential depending on the morphology of the species to be classified. . More particularly, spectral filters in the bands 300 nm to 500 nm promote the difference in contrast between, but not limited to, sardine and anchovy.

The broadband or narrow band images are entered into an analysis unit connected to a database of pelagic species, where they are computationally analyzed to carry out the identification, counting, registration and biomethco analysis of size and weight of the species classified in the images. This analysis unit comprises at least one processor configured to receive the images, compare discriminatory spectral and morphological particularities of the illuminated pelagic species with existing records in the pelagic species database, and identify the illuminated pelagic species according to criteria of similarity with the existing records in the pelagic species database.

The unit of analysis is the one that implements the method of this technology. To do this, it uses previously trained machine-learning algorithms to identify objects (for example, fish) in real time, from the multimodal and multispectral video feed of a conveyor belt; determines relevant information (such as, for example, the species and size of pelagic species) and reports the results to a user. These reports are the complete record with the level of detail that the user requires and contain the digital evidence of the objects and species analyzed in images and the statistics associated with an analysis time window that includes the number, relative proportion and estimation of relevant parameters (e.g. size and weight of classified species). The technology also comprises a user interface connected to the on-site device and to the back-end of an analysis unit to generate the aforementioned reports. An illustration of the operation of the system in the example of classifying pelagic species is shown in Figure 3, where (a) corresponds to software, user interface, reports and evidence; (b) to classification software; (c) to camera and narrow band filters; (d) preprocessing and control software; and (e) LED lighting.

Figures 4 A and B show graphs with records of spectral fingerprints of species of interest obtained with the present technology. The spectral fingerprint is defined as the ratio between the amount of light reflected by a pelagic species, as a function of wavelength, with respect to the total light reflected by a known reference, and where the specific measurements of the spectrum are spatially averaged using all those pixels in an image. Specifically, Figure 4A shows a graph referring to the complete anatomy of the pelagic species under analysis, while Figure 4B shows a graph referring to the lateral zone of the pelagic species under analysis.

Figure 5 shows a diagram of the species classification system based on a deep convolutional neural network architecture used by this technology to discriminate the species of interest, where (I) corresponds to the input, (II) to the property extraction, (III) to the output, (a) to the narrow band segmented images, N bands, (b) to 2D convolutional layer and (c) to 2D max pooling. The computational analysis system consists mainly of a set of pelagic species classifiers based on machine learning technology and image processing programs that operate in embedded computer vision systems.

Figure 6 shows a render of the designed, manufactured and installed gantry, where the gantry comprises an anchoring system that is installed on the conveyor belt, a controlled lighting system and an array of cameras with optical filters to improve discrimination. of species.

Exemplary embodiments are described in detail below to illustrate the principles of the invention. Embodiments are provided to illustrate aspects of the invention, but are not limited to any particular embodiment. The scope of the invention encompasses numerous alternatives, modifications and equivalents, limited only by the embodiments of the claims.

Application examples

Example 1 .

In an exemplary embodiment of the present technology, part of a fish landing was analyzed by passing through a conveyor belt, where specifically 70 tons were analyzed. For the comparison of this example, three 3-liter buckets were extracted with pelagic reference species, where it was estimated that the landing comprised 79% of anchovy and 21% of sardine.

Subsequently, a lighting system, a camera system and a set of optical filters of the present technology were installed on the conveyor belt through which the fish were unloaded and broadband images of the pelagic species were recorded as they passed under the camera system. The images were transmitted to an analysis unit connected to a database of pelagic species from the Biobío and Ñuble regions in Chile. The analysis unit compared discriminatory morphological particularities of the illuminated pelagic species with the existing records in the database, identifying the pelagic species illuminated according to similarity criteria using a deep convolutional neural network algorithm, thus obtaining that the landing included 82.5% anchovy and 17.5% sardine. The results indicate a similar distribution of species both by what was recorded by the inspector and by the identification and classification method. The main difference of this system is that, advantageously, it can achieve representative sampling by being able to analyze a greater number of species and in a distributed manner throughout the duration of the download. On the other hand, another advantage is that the inspector will not need to be in person at the unloading point, since he can carry out the work remotely by analyzing the results delivered by the recognition system.

Example 2.

In another example, a validation process of the results was carried out with certifying personnel. In this exercise, a total of 240 different images were selected, and the identification system was used to identify all possible species. Then, these already identified images were distributed to 5 expert certifiers, in order for them to indicate the wrong identifications of the system.

This validation showed that the average precision of the system for the correct identification of each of the 4 pelagic species available corresponded to: 98.2% for horse mackerel, 86.6% for mackerel, 75.4% for anchovy, and 93.4 % for sardine (see Figure 7 where a confusion matrix of the validation results carried out by certifiers is shown). Therefore, through the analysis system, high precision was achieved in the identification of species unloaded from fishing vessels.

Claims

1. A system for the identification of pelagic species out of water CHARACTERIZED because it comprises: a lighting system that illuminates one or more pelagic species to be analyzed with controlled light; a set of fixed or mobile optical filters for the selection of spectral bands; a camera array comprising one or more cameras in one or more wavelength ranges that record broadband or narrow band images of illuminated pelagic species using optical filters; an analysis unit connected to a database of pelagic species, wherein the analysis unit comprises at least one processor configured to receive the images, compare discriminatory spectral and morphological particularities of the illuminated pelagic species with existing records in the database of pelagic species, and identify the illuminated pelagic species according to similarity criteria with the existing records in the pelagic species database.

2. The system according to claim 1, CHARACTERIZED in that the lighting system comprises at least one uniform lighting source.

3. The system according to claim 1, CHARACTERIZED in that the lighting system comprises at least one controlled or structured lighting source.

4. The system according to claim 1, CHARACTERIZED in that the camera system comprises at least one infrared spectrum camera or a visible spectrum camera.

5. The system according to claim 1, CHARACTERIZED in that the set of optical filters filters in the bands 300 nm to 500 nm.

6. The system according to claim 1, CHARACTERIZED in that the identification of the illuminated pelagic species according to similarity criteria is determined by convolutional neural networks.

7. The system according to claim 6, CHARACTERIZED because the convolutional neural networks are deep neural networks.

8. A method for the identification of pelagic species out of water, CHARACTERIZED because it includes the steps of: providing a unit of analysis connected to a database of pelagic species; provide a controlled lighting system, a camera array system and a set of optical filters on one or more pelagic species to be analyzed;

illuminate one or more pelagic species with controlled lighting; record broadband and narrow band images of pelagic species using the multispectral camera system through the set of optical filters; receiving the images of the illuminated fish in the analysis unit; compare discriminatory spectral and morphological particularities of illuminated fish with existing records in the pelagic species database; identify the illuminated pelagic species according to similarity criteria with existing records in the pelagic species database.

9. The method according to claim 8, CHARACTERIZED because the similarity criteria are determined by convolutional neural networks.

10. The method according to claim 8, CHARACTERIZED because the convolutional neural networks are deep neural networks.