CN114049477B

CN114049477B - Fish passing fishway system and dynamic identification and tracking method for fish quantity and fish type

Info

Publication number: CN114049477B
Application number: CN202111357204.7A
Authority: CN
Inventors: 柳春娜; 李健源; 吴必朗
Original assignee: China Institute of Water Resources and Hydropower Research
Current assignee: China Institute of Water Resources and Hydropower Research
Priority date: 2021-11-16
Filing date: 2021-11-16
Publication date: 2023-04-07
Anticipated expiration: 2041-11-16
Also published as: CN114049477A

Abstract

The invention relates to a fish passing fishway system and a dynamic identification tracking method for the number and the types of fishes, which belong to the technical field of fish passing fishways.A fish blocking iron net is used on two sides of a fish passing monitoring section in a fishway to reduce the fish passing channel, the mutual shielding proportion of the fishes is reduced, two underwater cameras are used for shooting images of the fishes passing through between the fish blocking iron net and a light supplementing light wall, the shot images are input into a trained YOLOv5 network classification model, an identification image labeled with the position and the type of a prediction frame of each fish is output, and then a Deepsort multi-target tracking algorithm based on a counting function is adopted to obtain the uplink number and the downlink number of each kind of fishes in the fish passing fishway corresponding to each underwater camera in a monitoring period, so that the dynamic intelligent identification tracking of the number and the types of the fishes is realized; and the defects of inaccurate fish identification and high counting difficulty of the traditional manual work are overcome, and the efficiency and the accuracy of fish identification are improved.

Description

Fish passing fishway system and dynamic identification and tracking method for fish quantity and fish type

Technical Field

The invention relates to the technical field of fish passing fishways, in particular to a fish passing fishway system and a dynamic identification and tracking method for the number and the types of fishes.

Background

The water conservancy and hydropower engineering plays a great role in promoting the sustainable development of social economy and guaranteeing the energy safety as an important means for developing and utilizing hydroenergy and water resources in human society. However, the construction of the engineering can cause the longitudinal connectivity of the river to disappear, so that a dam blocking effect is generated, the migration of the fishes and the gene communication of upstream and downstream populations are influenced, and the biological diversity of the fishes is reduced. In order to reduce the influence of the hydroelectric engineering on the environment and fishes, the fish passing facilities are built to assist the migratory fishes to go up and down so as to maintain the ecological balance of a river system.

The fish passing effect is the basis for measuring the fishway construction value and optimizing and transforming. The development of the effect monitoring of the fish passing facilities is a necessary link for improving the fish passing efficiency and verifying the effectiveness of the fishway protection measures. In the process of monitoring the effect of the fish passing facilities, the biological index monitoring technology is the key point of monitoring the fish passing effect, such as the type and the quantity of fish, the body length, the weight, the age and the like.

The fish passing effect of fishways at home and abroad can be divided into two categories of direct methods and indirect methods according to monitoring modes, wherein manual observation, a net-opening method, a blocking method and the like belong to the direct methods, the manual observation is time-consuming and labor-consuming, and the net-opening method needs to be off-line at different time intervals every day and is not suitable for long-term monitoring; the blocking method needs to put the net at the outlet of the fishway to drain water, has large disturbance to the fish, can be implemented at most once a month, and the like. The monitoring mode can only acquire monitoring data of a short-time sequence, cannot acquire real-time monitoring data, and also has the problem that the type and the quantity of the entering fishway have great difference due to the fact that natural influence factors such as water level difference, water temperature and hydrological state of different months change, so that the short-time monitoring data cannot replace long-sequence monitoring data. Although parameters such as the number, the type, the specification and the like of fish passing in a fishway can be obtained by the direct method, the direct method excessively depends on manual operation, cannot automatically count the number of the fish passing, and has low efficiency and accuracy and high cost. While monitoring by acoustics, telemetry and the like belongs to indirect methods, fish passing parameters are obtained by deep analysis of data acquired by equipment, wherein the accuracy of the fish passing parameters depends on a later data analysis method, and the method is mainly used for fish quantity identification, and although the sonar acoustics, PIT telemetry and the like can be used for monitoring fishes for a long time, the method can only generally acquire the fish quantity and swimming speed, but is difficult to realize fish species identification.

Disclosure of Invention

The invention aims to provide a fish passing fishway system and a dynamic identification and tracking method for the number and the types of fishes, so as to realize dynamic intelligent identification and tracking of the number and the types of the fishes and improve the efficiency and the accuracy of fish identification.

In order to achieve the purpose, the invention provides the following scheme:

a fish pass fishway system, comprising: the system comprises a server, a light supplementing light wall, two fish blocking iron nets and two underwater cameras;

the two fish blocking iron nets are oppositely arranged on two sides of the fish passing fishway, and the light supplementing light wall is arranged in the fish passing fishway between the two fish blocking iron nets, so that fish pass through the space between the fish blocking iron nets and the light supplementing light wall;

an underwater camera is arranged between each fish blocking iron net and the inner wall of the fish passing channel; the lenses of the two underwater cameras are aligned to the light supplementing wall;

the signal output ends of the two underwater cameras are connected with the server, and the two underwater cameras are used for respectively shooting images of fish passing through the space between the fish blocking iron net and the light supplementing light wall and transmitting the images to the server;

the server is used for obtaining the number and the types of the fishes passing through the fish channel according to the images shot by the two underwater cameras.

Optionally, the server includes: the system comprises an image acquisition module, an identification module, a tracking module and a fish number counting module;

the image acquisition module is used for acquiring a plurality of images shot by each underwater camera in a monitoring period;

the recognition module is used for inputting each image shot by each underwater camera into a trained YOLOv5 network classification model and outputting a recognition image labeled with a prediction frame and a type of each fish;

the tracking module is used for dividing each identification image corresponding to each underwater camera into 4 regions, tracking the fish in the identification images by utilizing a Deepsort multi-target tracking algorithm based on a counting function according to the identification images after the regions are divided corresponding to each underwater camera and counting the number of the fish in the identification images, and acquiring the uplink number and the downlink number of each kind of fish in the fish passage channel corresponding to each underwater camera in the monitoring time period;

and the fish number counting module is used for taking the sum of the uplink number and the downlink number of the same kind of fishes in the fish passing fishways corresponding to all the underwater cameras in the monitoring period as the number of each kind of fishes in the fish passing fishways in the monitoring period.

A dynamic identification and tracking method for the number and the types of fishes based on the fish passing fishway system comprises the following steps:

acquiring a plurality of images shot by each underwater camera in a monitoring period;

inputting each image shot by each underwater camera into a trained YOLOv5 network classification model, and outputting an identification image labeled with a prediction frame and a type of each fish;

dividing each identification image corresponding to each underwater camera into 4 areas, tracking the fish in the identification images by using a Deepsort multi-target tracking algorithm based on a counting function according to the identification images after the areas are divided corresponding to each underwater camera, and counting the number of the fishes in the identification images to obtain the uplink number and the downlink number of each kind of fish in the fish passage corresponding to each underwater camera in a monitoring period;

and taking the sum of the uplink quantity and the downlink quantity of the same type of fishes in the fish-passing fishways corresponding to all the underwater cameras in the monitoring period as the quantity of each type of fishes in the fish-passing fishways in the monitoring period.

Optionally, the acquiring a plurality of images captured by each underwater camera in the monitoring period further includes:

collecting a picture sample of each target fish; the picture samples comprise pictures of the wriggling state of the target fish, pictures containing all characteristics of the target fish and pictures of local unique characteristics of the target fish;

labeling the collected pictures of each target fish by using LabelImg to obtain a picture label of each target fish;

forming a fish data sample set by the pictures of all the target fishes and the picture labels of all the target fishes;

and dividing the fish data sample set into a training set and a verification set, training a Yolov5 network model by using the training set, and detecting the trained Yolov5 network model by using the verification set to obtain a trained Yolov5 network classification model.

Optionally, the dividing each identification image corresponding to each underwater camera into 4 regions, performing trajectory tracking and quantity statistics on the fish in the identification images according to the identification image after the division corresponding to each underwater camera by using a deep sort multi-target tracking algorithm based on a counting function, and obtaining the uplink quantity and the downlink quantity of each fish in the fish passage corresponding to each underwater camera within the monitoring period specifically includes:

dividing each identification image corresponding to the underwater camera into 4 areas by using a cross line to obtain area division images;

the number k of the preset area division image frames is equal to 1;

according to the area division images of the kth frame and the (k + 1) th frame, matching the fish in the area division images of the two frames by using a deep sort multi-target tracking algorithm to obtain the position of a prediction frame of the fish successfully matched in the area division images of the (k + 1) th frame;

dividing the regions of the images according to the regions of the successfully matched fish in the (k + 1) th frame and the k-th frame, and determining the direction of the successfully matched fish; the traveling direction is ascending or descending;

according to the swimming direction and the belonged fishes of the successfully matched fishes in the (k + 1) th frame region division image, counting the ascending quantity and the descending quantity of the belonged fishes of the successfully matched fishes in the (k + 1) th frame region division image;

increasing the value of k by 1, returning to the step of matching the fish in the two frame region division images by a Deepsort multi-target tracking algorithm according to the k frame and the k +1 frame region division images to obtain the position of a prediction frame of the successfully matched fish in the two frame region division images in the k +1 frame region division images until the value of k +1 is equal to the total number of all images shot by the underwater camera in the monitoring period, and outputting the uplink quantity and the downlink quantity of the fish to which the successfully matched fish belongs in the k +1 frame region division images as the uplink quantity and the downlink quantity of each fish in the fish passage channel corresponding to each underwater camera in the monitoring period.

Optionally, the determining the walking direction of the successfully matched fish according to the areas where the successfully matched fish is located in the k +1 th frame and the k frame and by dividing the areas where the images are located includes:

defining an upper right area of the area division image as a first area, an upper left area as a second area, a lower left area as a third area and a lower right area as a fourth area;

if the center of the position of the prediction frame of the fish successfully matched is in the first area of the k frame area divided image, and the center of the position of the prediction frame of the fish successfully matched is in the second area of the k +1 frame area divided image, determining the direction of the wandering line of the fish successfully matched as an ascending line;

and if the center of the position of the prediction frame of the fish successfully matched is in the third area of the k frame area divided image, and the center of the position of the prediction frame of the fish successfully matched is in the fourth area of the k +1 frame area divided image, determining the direction of the wandering line of the fish successfully matched as a downlink.

A system for dynamic identification and tracking of fish numbers and species, the system comprising:

Optionally, the system further includes:

the image sample acquisition module is used for acquiring an image sample of each target fish; the picture samples comprise pictures of the wriggling state of the target fish, pictures containing all characteristics of the target fish and pictures of local unique characteristics of the target fish;

the picture tag obtaining module is used for labeling the collected pictures of each target fish by using LabelImg to obtain a picture tag of each target fish;

the fish data sample set forming module is used for forming a fish data sample set by the pictures of all the target fishes and the picture labels of all the target fishes;

and the training module is used for dividing the fish data sample set into a training set and a verification set, training the Yolov5 network model by using the training set, detecting the trained Yolov5 network model by using the verification set, and obtaining the trained Yolov5 network classification model.

Optionally, the tracking module specifically includes:

the area division submodule is used for dividing each identification image corresponding to the underwater camera into 4 areas by using a cross line to obtain area division images;

a preset sub-module for presetting the number k of the region division image frames to be equal to 1;

the matching sub-module is used for matching the fish in the two frames of the area division images by using a deep sort multi-target tracking algorithm according to the k frame and the k +1 frame of the area division images to obtain the position of a prediction frame of the fish successfully matched in the two frames of the area division images in the k +1 frame of the area division images;

the swim direction determining submodule is used for dividing the regions of the images according to the regions of the successfully matched fish in the (k + 1) th frame and the kth frame and determining the swim direction of the successfully matched fish; the traveling direction is ascending or descending;

the number counting submodule is used for counting the uplink number and the downlink number of the fishes to which the successfully matched fishes belong in the (k + 1) th frame region division image according to the swimming direction and the fishes to which the successfully matched fishes belong in the (k + 1) th frame region division image;

and the circulating submodule is used for increasing the value of k by 1, returning to the step of matching the fish in the two frame region division images by using a Deepsort multi-target tracking algorithm according to the k frame and the k +1 frame region division images to obtain the position of a prediction frame of the fish successfully matched in the two frame region division images in the k +1 frame region division images until the value of k +1 is equal to the total number of all images shot by the underwater camera in the monitoring period, and outputting the uplink quantity and the downlink quantity of the fish to which the fish successfully matched in the k +1 frame region division images belongs as the uplink quantity and the downlink quantity of each fish in the fish passage fishway corresponding to each underwater camera in the monitoring period.

Optionally, the tour direction determining sub-module specifically includes:

the image processing device comprises an area defining unit, a first image processing unit and a second image processing unit, wherein the area defining unit is used for defining an upper right area of an area division image as a first area, an upper left area as a second area, a lower left area as a third area and a lower right area as a fourth area;

an ascending determination unit, configured to determine that the direction of the swimming of the successfully matched fish is ascending if the center of the position of the prediction frame of the successfully matched fish is in the first region of the k-th frame region partition image and the center of the position of the prediction frame of the successfully matched fish is in the second region of the k + 1-th frame region partition image;

and a downlink determining unit, configured to determine that the traveling direction of the successfully matched fish is downlink if the center of the position of the prediction frame of the successfully matched fish is in the third region of the k-th frame region divided image and the center of the position of the prediction frame of the successfully matched fish is in the fourth region of the k + 1-th frame region divided image.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects:

the invention discloses a fish passing fishway system and a dynamic identification and tracking method for the number and the types of fishes.A fish passing channel is reduced by using fish blocking iron nets at two sides of a fish passing monitoring section in a fishway, the mutual shielding proportion of the fishes is reduced, two underwater cameras are used for shooting images of the fishes passing through between the fish blocking iron nets and a light supplementing light wall, the shot images are input into a trained YOLOv5 network classification model, an identification image labeled with the position and the type of a prediction frame of each fish is output, and then the uplink number and the downlink number of each kind of fishes in the fish passing fishway corresponding to each underwater camera in a monitoring period are obtained by adopting a Deepsort multi-target tracking algorithm based on a counting function, so that the dynamic intelligent identification and tracking of the number and the type of the fishes are realized; and the defects of inaccurate fish identification and high counting difficulty of the traditional manual work are overcome, and the efficiency and the accuracy of fish identification are improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

FIG. 1 is a block diagram of a fish pass fishway system according to the present invention;

FIG. 2 is a flow chart of a method for dynamically identifying and tracking the number and species of fish according to the present invention;

FIG. 3 is a diagram of the intersection region of the prediction box and the real box during the training of the YOLOv5 network model provided by the present invention;

FIG. 4 is a diagram of the YOLOv5 detection process provided by the present invention;

FIG. 5 is a schematic diagram of a YOLOv5 network structure provided by the present invention;

FIG. 6 is a schematic diagram of the Deepsort algorithm provided by the present invention;

FIG. 7 is a schematic diagram of the interface partitioning provided by the present invention;

FIG. 8 is a schematic diagram of an example interface partitioning provided by the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

In order to make the aforementioned objects, features and advantages of the present invention more comprehensible, the present invention is described in detail with reference to the accompanying drawings and the detailed description thereof.

The present invention provides a fish passing fishway system, as shown in fig. 1, the fish passing fishway system includes: the system comprises a server, a light supplementing light wall, two fish blocking iron nets and two underwater cameras.

Two block the relative both sides that set up at crossing the fish fishway of fish iron-net, during light filling light wall set up the fish fishway of crossing between two block the fish iron-nets for the fish passes through between fish iron-net and the light filling light wall from blocking.

An underwater camera is arranged between each fish blocking iron net and the inner wall of the fish passing channel; the lens of two underwater cameras are both aligned to the light filling wall.

The signal output ends of the two underwater cameras are connected with the server, and the two underwater cameras are used for respectively shooting the images of the fish passing through the fish blocking iron net and the light supplementing light wall and transmitting the images to the server.

The server includes: the device comprises an image acquisition module, an identification module, a tracking module and a fish number counting module.

the identification module is used for inputting each image shot by each underwater camera into a trained YOLOv5 network classification model and outputting an identification image labeled with a prediction frame and a type of each fish;

The fish passing fishway designed by the invention can be used for allowing one fish to pass through, and also can be used for allowing fish shoals to pass through simultaneously, in order to detect the mutually-covered fishes as far as possible, the two sides of the fish passing monitoring section in the fishway are provided with the iron frame nets to reduce the fish passing channel, so that the mutual shielding ratio is reduced, and the fish can be detected as long as more than 1/5 of the fish can be shot. Because of meeting bad hydrology season occasionally, particulate matters such as silt, impurity are mixed in the water, in order to increase the visibility of fish when the camera is shot, add a colour is variable, luminance adjustable light filling light wall in the middle of the fishway after reducing, the fish can pass through in worn-out fur both sides, arrange high definition underwater camera in both sides and carry out long-time monitoring to this fishway, it is the multi-target dynamic tracking identification technique that the video passes through RTSP video stream real-time transmission and uses the code to read in real time and input to the server, add up the data that the camera obtained of both sides finally, can obtain fishway fish quantity and kind.

Aiming at the difficulties that the existing fish-passing effect monitoring technology needs to rely on high level of service for identifying fishes by managers and the difficulty of counting fish-passing targets is high, the data of the types, the quantity, the specifications, the fish-passing efficiency and the like of the fish-passing targets are obtained in real time, the ecological environment is damaged, the aquatic ecosystem is influenced, and the efficiency is low, the deep learning neural network is utilized to simulate human neurons to learn and classify the fish images, the YOLOv5 network model is used to detect and identify the fishes, and the improved deep Sort target tracking algorithm is combined to track and count the fishes, so that the dynamic intelligent identification tracking and the multi-target automatic counting of the fish types are realized.

The invention provides a dynamic identification and tracking method for the number and the type of fishes based on the fish passing fishway system, as shown in fig. 2, the method comprises the following steps:

101, acquiring a plurality of images shot by each underwater camera in a monitoring period;

step 102, inputting each image shot by each underwater camera into a trained YOLOv5 network classification model, and outputting an identification image labeled with a prediction frame and a type of each fish;

103, dividing each identification image corresponding to each underwater camera into 4 areas, tracking the fish in the identification images by using a Deepsort multi-target tracking algorithm based on a counting function according to the identification images after the areas are divided corresponding to each underwater camera, and counting the number of the fishes in the identification images to obtain the uplink number and the downlink number of each kind of fish in the fish passage and fishway corresponding to each underwater camera in the monitoring time period.

The method specifically comprises the following steps:

the number k of the preset area division image frames is equal to 1;

dividing the areas of the images according to the areas of the successfully matched fish in the (k + 1) th frame and the kth frame, and determining the swimming direction of the successfully matched fish; the traveling direction is ascending or descending;

according to the swimming direction and the belonged fishes of the fishes successfully matched in the k +1 frame region division image, counting the uplink quantity and the downlink quantity of the belonged fishes of the fishes successfully matched in the k +1 frame region division image;

increasing the value of k by 1, returning to the step of matching the fish in the two frame region division images by a Deepsort multi-target tracking algorithm according to the k frame and the k +1 frame region division image to obtain the position of a prediction frame of the successfully matched fish in the k +1 frame region division image in the two frame region division images until the value of k +1 is equal to the total number of all images shot by the underwater cameras in the monitoring period, and outputting the uplink quantity and the downlink quantity of the fishes to which the successfully matched fish belongs in the k +1 frame region division image as the uplink quantity and the downlink quantity of each kind of fish in the fish passage corresponding to each underwater camera in the monitoring period.

The method for determining the swimming direction of the successfully matched fish comprises the following steps:

if the center of the position of the prediction frame of the successfully matched fish is in the first area of the k frame area divided image, and the center of the position of the prediction frame of the successfully matched fish is in the second area of the k +1 frame area divided image, determining the direction of the wandering line of the successfully matched fish as an ascending line;

and if the center of the position of the prediction frame of the fish successfully matched is in the third area of the k frame area divided image, and the center of the position of the prediction frame of the fish successfully matched is in the fourth area of the k +1 frame area divided image, determining the direction of the wandering line of the fish successfully matched as a descending line.

And step 104, taking the sum of the uplink quantity and the downlink quantity of the same type of fishes in the fish-passing fishways corresponding to all the underwater cameras in the monitoring period as the quantity of each type of fishes in the fish-passing fishways in the monitoring period.

The YOLOv5 network model also needs to be trained before step 101 is performed:

collecting a picture sample of each target fish; the picture samples comprise pictures of the wriggling state of the target fishes, pictures containing all characteristics of the target fishes and pictures of local unique characteristics of the target fishes;

dividing the fish data sample set into a training set and a verification set, training a YOLOv5 network model by using the training set, and detecting the trained YOLOv5 network model by using the verification set to obtain a trained YOLOv5 network classification model.

According to the invention, under the condition of not interfering normal swimming of fishes, a fish effect monitoring technology is researched, a deep learning neural network is utilized to simulate human neurons to learn and classify fish images, a novel fish detection counting method and a novel fish tracking counting method are developed in an effort to realize dynamic intelligent identification tracking of the quantity and the type of the fishes, the problem of low accuracy and efficiency of a traditional fish static identification technology is solved, the high efficiency and the comprehensiveness of the fish monitoring effect are improved, long-sequence monitoring data of the fish effect are obtained, support is provided for fish facility effect evaluation and structure optimization improvement, unnecessary interference on fish habitats is reduced, and the health of an aquatic ecosystem is maintained.

The dynamic identification and tracking method for the number and the type of the fishes is further explained aiming at the fishway construction of the Tibet region.

The deep learning has strong feature extraction and expression capability, can extract deep visual features, and completes target classification and detection. The main fish passing through the fishway are tip naked carp, giant beard schizothorax fish, lassa schizothorax fish and heterodentate schizothorax fish, and the target fish needing training is shot by using a shooting tool respectively, and the acquisition requirements of pictures comprise: (1) Photographing the target fish from left to right (from head to tail to head) and from top to bottom (from dorsal fin to belly to dorsal fin) as much as possible for 360 degrees around the target fish to extract all the characteristics of the target fish; (2) Pictures of the wriggling (swimming, head twisting, turning, etc.) state of the target fish so as to improve the accuracy of dynamic identification; (3) Clear pictures with local unique characteristics (such as body surfaces with obvious head and front half body characteristics) for target fishes; (4) Not more than two pictures (avoiding excessive repeated pictures) are continuously shot at the same position and angle, and the pictures are required to be clear. In order to improve the detection accuracy, the collection of the four target fish pictures reaches ten thousand, but the four target fish pictures are not completely limited to the four fishes, and the types of the target fish can be expanded if the target fish pictures are needed, so that the method is not limited.

And marking the processed pictures by using a tool LabelImg to prepare a Tibet specific fish data set, wherein 90% of the data set is divided into a training set for training, and 10% of the data set is divided into a testing set for testing the detection effect after training. And (3) performing fish training on the self-made Tibet specific fish data set by using a YOLOv5 network model to obtain the network identification weight of the data set. The training loss function of the network adopts a GIoU loss function, and when the fluctuation of the loss value is not reduced within a certain small range, the network can be judged to be convergent, so that the training effect is good.

Ac is the area of the minimum closure region, which is colloquially understood as the area of the minimum box simultaneously containing a prediction box during training and a real box manually marked by using LabelImg; the area of intersection of the prediction box and the real box is shown in fig. 3.

U: the size of the intersection of the prediction box during training and the real box manually labeled by LabelImg

The obtained network identification weight of the Tibet specific fish is connected with a detection module of yolov5, so that the fish species can be directly detected, and then an independent ID is distributed to each identified fish species by combining with the modified tracker Deepsort and the tracker records the identification weight so as to perform real-time tracking. The DeepsSort algorithm uses recursive Kalman filtering to process data relevance frame by frame and uses Hungarian algorithm to perform target screening and cross-frame matching on the output of the detector. The identification technique of the system is as follows:

a detector module: the YOLOv5 algorithm framework is mainly divided into four parts, the first part is an input end, and three-channel pictures with the size of 640 x 640 are input. The second part is a BackBone BackBone network, and the BackBone network takes Focus and CSPNet network frames as models to carry out feature extraction on imagesAnd (4) taking. The third part is a hack module, which is located between the backphone and the final output layer. The module comprises a space pooling Structure (SPP) adopting a maximum pooling method and a path aggregation network structure (PANet) under an instance segmentation framework, and repeatedly performs feature fusion and extraction on shallow information and deep information in three feature layers. And the fourth part predicts and predictively decodes the generated three feature maps (yoloead) to directly obtain the position and the type of each fish prediction frame in the image, namely, a 20 × 20, 40 × 40 and 80 × 80 feature map grid is output after an image passes through a Backbone BackBone network, SPP and PANet, and if the center of a certain target falls in the grid, the grid is responsible for predicting the target. B boundary boxes with different sizes and aspect ratios are preset in each grid, and each boundary box comprises 5 predicted values: t is t _x 、t _y 、t _w 、t _h And confidence, calculating the center coordinates (b) of the prediction frame according to the formula (1) _x ，b _y ) And width w, height h, the prediction box is position shifted and scaled based on the bounding box.

In the formula:

σ (x) - -Logistic function:

c _x 、c _y -the coordinates of the upper left corner of each grid in the feature map;

p _w 、p _h -the width and height of the bounding box relative to the feature map;

t _x 、t _y 、t _w 、t _h -the central coordinates and width and height of the model predictions;

(2) After the step (1), each grid generates B prediction frames, then non-maximum value suppression is carried out, grid boundary frames without targets are directly removed, grids with targets highlight boundary frames with the highest confidence level in all categories, boundary frames of other categories in the grids are removed, and the confidence level is predicted by the continuous iterative training of the networks.

(3) And performing intersection comparison (IOU) on the boundary box with the highest confidence coefficient of the grid and the boundary boxes of other grids after non-maximum value inhibition and a real box, if the IOU is more than 0.5, determining the boundary boxes are the same target, removing the boundary boxes with lower confidence coefficient than the middle confidence coefficient, and continuing the operation until only one boundary box with the highest confidence coefficient is left in the target.

The YOLOv5 detection process is shown in fig. 4, and the YOLOv5 network structure is shown in fig. 5.

A tracker: based on a YOLOv5 target detector, a detection result of the YOLOv5 is used as input of subsequent tracking, a Deepsort algorithm framework is combined, data association is carried out by utilizing a motion model and apparent information, end-to-end multi-target visual tracking is achieved, and the principle of the Deepsort algorithm is shown in FIG. 6. Deepsort uses X = [ u, v, r, h, u ', v', r ', h']The state vector is used as a direct observation model of the target, wherein (u, v) is the central coordinate of the bounding box, r and h are the aspect ratio and the height of the bounding box respectively, and the other 4 vectors represent the corresponding speed information. Image vision and motion information are extracted through a Kalman algorithm, the position of the next moment is predicted based on the position of the previous moment of the target, namely when the target moves, through two target frames of an upper frame and a lower frame, parameters such as the speed of the two frames are obtained by utilizing the pixel distance of the movement of the central points of the two target frames of the two frames, and similar data can be obtained all the time by analogy, and the trajectory Tracks are predicted and updated. The Hungarian algorithm is used for data association, the predicted trajectory Tracks are matched with the detection in the current frame (cascade matching and IOU matching), and the fact that whether a certain target of the current frame is the same as a certain target of the previous frame or not is informed to people, so that the ID distribution problem is solved. The cost matrix is obtained by integrating a motion model and an appearance model, wherein the motion model is obtained by square Mahalanobis distance, and the appearance model is obtained by minimum cosine distance. Mahalanobis distance meterIndicating the deviation degree of the detected target from the target track average position, measuring the matching degree of the target state predicted by the Kalman filter and the detected value by using the Mahalanobis distance, and using y _i Indicating the target prediction frame position of the i-th tracker, d _j For the jth detection frame position, S _i Is a covariance matrix between the detected position and the tracked position. The mahalanobis distance calculation formula is:

screening left and right detected targets through the Mahalanobis distance, and setting a threshold value t ⁽¹⁾ =9.4877. If the associated mahalanobis distance d ⁽¹⁾ And if the value is less than the threshold value, setting the motion state association to be successful. I.e., the moving distance between the upper and lower frames is not too far apart, the closer the coordinates are, the more likely it is that the same object is.

Mahalanobis distance can measure well the relationship between the detected target and the trajectory when the motion uncertainty is low, but the correlation method fails when the camera is shaken vigorously. Therefore, convolutional neural networks were introduced for correlation with surface feature matching. Finding each detection target d _j Characteristic vector r of _j And r _j I | =1. For tracker i, the nearest 100 frame feature vectors corresponding to its kth track are saved in the set { r (i) k } ^Lk _k＝1 In, L _k =100. And using the minimum cosine distance between the feature set and the feature vector of the jth detection result of the current frame to represent the relation between the detection target and the track:

setting a threshold value and determining whether the two are related or not. Finally, two measures are used for linear weighting as a final measure:

c _i,j ＝λd ⁽¹⁾ (i,j)+(1-λ)d ⁽²⁾ (i,j)

after prediction and correlation are carried out by a Deepsort algorithm, a fish multi-target counting module is added into a tracking module, namely, after targets to be detected and identified are classified and an ID is independently allocated to each target, 4 areas are divided on an interface to be detected so as to carry out target uplink and downlink and total quantity counting, if a red line 1 and a blue line 2 (the two channels are separated by the line 2 and cannot alternately pass through) are used for dividing the interface into 4 areas, the areas are named as I, II, III and IV in sequence, each position where the target passes is recorded, and if the condition is met, counting is carried out, as shown in figure 7.

Referring to fig. 7-8, when the center point of the target in the area i reaches the area ii, that is, the center coordinate of the target is smaller than the X-axis coordinate of the line 1 on the X-axis and the Y-axis coordinate of the line 2 on the Y-axis, 1 is added to the uplink number according to the target type; similarly, if the central point of the target in the area iii reaches the area iv, that is, the central coordinate of the target is greater than the X-axis coordinate of the line 1 on the X-axis and greater than the Y-axis coordinate of the line 2 on the Y-axis, the downlink number is increased by 1 according to the type of the target.

And finally, accumulating the data obtained by the cameras on the two sides to obtain the number and the types of the fish in the fishway.

The traditional fishway fish passing effect evaluation mainly adopts manual observation, a net-opening method and the like, the transition depends on manual work, the action is extremely inconvenient, time, material and labor are consumed, the efficiency of the method is low, and the interference on fishes is large. On the basis of a YOLOv5 target detection algorithm combined with an improved DeepSort multi-target tracking algorithm, dynamic matching of fish characteristic values, intelligent fish identification tracking and multi-target automatic counting are achieved, the problems that the technical level of field personnel is excessively depended on, and the difficulty in counting the fish passing targets is high are solved, the defects that the accuracy rate and the efficiency of a traditional fish static identification technology are low are overcome, the data of the types, the quantity, the specifications, the fish passing efficiency and the like of the fish passing targets can be obtained in real time, the dynamic intelligent identification tracking of the quantity and the types of the fish is achieved, the operation management cost is reduced, and the long-term fish passing effect monitoring is achieved.

The invention also provides a system for dynamically identifying and tracking the number and the types of the fishes, which comprises:

The system further comprises:

the fish data sample set forming module is used for forming the pictures of all the target fishes and the picture labels of all the target fishes into a fish data sample set;

The tracking module specifically comprises:

the presetting submodule is used for presetting that the number k of the image frames divided by the area is equal to 1;

The tour direction determination submodule specifically includes:

an ascending determination unit, configured to determine that the direction of the row of the successfully matched fish is an ascending direction if the center of the position of the prediction frame of the successfully matched fish is in the first region of the k-th frame region partition image and the center of the position of the prediction frame of the successfully matched fish is in the second region of the k + 1-th frame region partition image;

and the descending determining unit is used for determining the traveling direction of the successfully matched fish as descending if the center of the position of the prediction frame of the successfully matched fish is in the third area of the k frame area and the center of the position of the prediction frame of the successfully matched fish is in the fourth area of the k +1 frame area.

In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.

Claims

1. A fish passing fishway system, comprising: the system comprises a server, a light supplementing light wall, two fish blocking iron nets and two underwater cameras;

the server is used for obtaining the number and the types of the fishes passing through the fish channel according to the images shot by the two underwater cameras;

the server includes: the system comprises an image acquisition module, an identification module, a tracking module and a fish number counting module;

the fish number counting module is used for taking the sum of the uplink number and the downlink number of the same type of fishes in the fish-passing fishways corresponding to all the underwater cameras in the monitoring time period as the number of each type of fishes in the fish-passing fishways in the monitoring time period;

the YOLOv5 network classification model specifically includes:

an input terminal for inputting three-channel images having a size of 640 x 640;

the BackBone BackBone network takes Focus and CSPNet network frames as models and is used for carrying out feature extraction on images;

the Neck module adopts a space pooling structure of a maximum pooling method and a path aggregation network structure under an instance segmentation frame, and is used for repeatedly performing feature fusion on shallow information and deep information in the three feature layers and then extracting and generating three feature graphs; the three feature maps are 20 × 20, 40 × 40 and 80 × 80 feature map grids;

the output layer is used for predicting and predicting decoding the three characteristic graphs to obtain the position and the category of each fish prediction frame in the image; b bounding boxes with different sizes and aspect ratios are preset in each feature map grid; each bounding box contains 5 predictors: t is t _x 、t _y 、t _w 、t _h And a confidence level; the prediction frame carries out position movement and size scaling on the basis of the bounding box; the specific formula is as follows:

in the formula: σ (x) is a Logistic function,

(c _x ，c _y ) Coordinates of the upper left corner of each grid in the feature map; p is a radical of _w 、p _h Width and height of the bounding box relative to the feature map; t is t _x 、t _y 、t _w 、t _h The center coordinates and width and height of the bounding box; (b) _x ，b _y ) The center coordinate of the prediction frame, w the width of the prediction frame and h the height of the prediction frame; b bounding boxes are generated by each feature map grid, the bounding boxes are subjected to non-maximum value suppression, the bounding boxes without targets are directly removed, the feature map grids with the targets highlight the bounding boxes with the highest confidence level in all categories, and the bounding boxes of other categories in the feature map grids are removed; the confidence coefficient is predicted by continuous iterative training of a YOLOv5 network classification model; the boundary box with the highest confidence coefficient and the boundary box after the other characteristic graph grids are subjected to non-maximum value inhibition are subjected to intersection and comparison with a real box; if the intersection ratio is larger than 0.5, the target is regarded as the same target, the boundary box with the intersection ratio lower than the middle confidence level is removed, and finally a boundary box with the highest confidence level is obtained;

the tracking module specifically includes:

and the circulating submodule is used for increasing the value of k by 1, returning to the step of matching the fish in the two frame region division images according to the k frame and the k +1 frame region division images by using a DeepSort multi-target tracking algorithm to obtain the position of a prediction frame of the successfully matched fish in the two frame region division images in the k +1 frame region division images until the value of k +1 is equal to the total number of all images shot by the underwater cameras in the monitoring period, and outputting the uplink quantity and the downlink quantity of the fishes to which the successfully matched fish belongs in the k +1 frame region division images as the uplink quantity and the downlink quantity of each kind of fish in the fish passage corresponding to each underwater camera in the monitoring period.

2. A method for dynamically identifying and tracking the number and the type of fishes based on the fish pass-through channel system of claim 1, wherein the method comprises:

dividing each identification image corresponding to each underwater camera into 4 areas, tracking the fish in the identification images by using a Deepsort multi-target tracking algorithm based on a counting function according to the identification images after the areas are divided corresponding to each underwater camera, and counting the number of the fishes in the identification images to obtain the uplink number and the downlink number of each kind of fish in the fish passage corresponding to each underwater camera in a monitoring period; the method specifically comprises the following steps: dividing each identification image corresponding to the underwater camera into 4 areas by using a cross line to obtain area division images; the number k of the preset area division image frames is equal to 1; according to the area division images of the kth frame and the (k + 1) th frame, matching the fish in the area division images of the two frames by using a deep sort multi-target tracking algorithm to obtain the position of a prediction frame of the fish successfully matched in the area division images of the (k + 1) th frame; dividing the regions of the images according to the regions of the successfully matched fish in the (k + 1) th frame and the k-th frame, and determining the direction of the successfully matched fish; the traveling direction is ascending or descending; according to the swimming direction and the belonged fishes of the fishes successfully matched in the k +1 frame region division image, counting the uplink quantity and the downlink quantity of the belonged fishes of the fishes successfully matched in the k +1 frame region division image; increasing the value of k by 1, returning to the step of matching the fish in the two frame region division images by a Deepsort multi-target tracking algorithm according to the k frame and the k +1 frame region division image to obtain the position of a prediction frame of the successfully matched fish in the k +1 frame region division image in the two frame region division images until the value of k +1 is equal to the total number of all images shot by the underwater cameras in the monitoring period, and outputting the uplink quantity and the downlink quantity of the fishes to which the successfully matched fish belongs in the k +1 frame region division image as the uplink quantity and the downlink quantity of each kind of fish in the fish passage corresponding to each underwater camera in the monitoring period;

3. The method for dynamically identifying and tracking the number and the type of the fishes according to claim 2, wherein the acquiring a plurality of images shot by each underwater camera in the monitoring period further comprises:

4. The method for dynamically identifying and tracking the number and the type of the fishes according to claim 2, wherein the step of determining the direction of the wandering of the successfully matched fishes by dividing the regions of the images according to the regions of the successfully matched fishes in the (k + 1) th frame and the kth frame comprises the following steps:

5. A system for dynamically identifying and tracking the number and type of fish, the system comprising:

the fish number counting module is used for taking the sum of the uplink number and the downlink number of the same type of fishes in the fish passing fishways corresponding to all the underwater cameras in the monitoring period as the number of each type of fishes in the fish passing fishways in the monitoring period;

wherein, the YOLOv5 network classification model specifically includes:

the BackBone trunk network takes Focus and CSPNet network frames as models and is used for carrying out feature extraction on the images;

in the formula: σ (x) is a Logistic function,

(c _x ，c _y ) Coordinates of the upper left corner of each grid in the characteristic diagram; p is a radical of _w 、p _h Width and height of the bounding box relative to the feature map; t is t _x 、t _y 、t _w 、t _h The center coordinates and width and height of the bounding box; (b) _x ，b _y ) The center coordinate of the prediction frame, w the width of the prediction frame and h the height of the prediction frame; b boundary frames are generated by each feature map grid, the boundary frames are subjected to non-maximum value suppression, the boundary frames without targets are directly removed, the feature map grids with targets highlight the boundary frames with the highest confidence level in all categories, and the boundary frames of other categories in the feature map grids are removed; the confidence coefficient is predicted by continuous iterative training of a YOLOv5 network classification model; the boundary box with the highest confidence coefficient and the boundary box after the other characteristic graph grids are subjected to non-maximum value inhibition are subjected to intersection and comparison with a real box; if the intersection ratio is larger than 0.5, the target is regarded as the same target, the boundary box with the intersection ratio lower than the middle confidence level is removed, and finally a boundary box with the highest confidence level is obtained;

the tracking module specifically includes:

the matching sub-module is used for matching the fish in the two frames of the regional division images by using a Deepsort multi-target tracking algorithm according to the regional division images of the kth frame and the (k + 1) th frame, and obtaining the position of a prediction frame of the successfully matched fish in the two frames of the regional division images in the (k + 1) th frame;

the parade direction determining submodule is used for dividing the areas of the images in the k +1 th frame and the k frame according to the successfully matched fish areas and determining the parade direction of the successfully matched fish; the traveling direction is ascending or descending;

6. The system for dynamically identifying and tracking the number and types of fish as claimed in claim 5, further comprising:

the image sample acquisition module is used for acquiring an image sample of each target fish; the picture samples comprise pictures of the wriggling state of the target fishes, pictures containing all characteristics of the target fishes and pictures of local unique characteristics of the target fishes;

7. The system for dynamically identifying and tracking the number and the type of fish according to claim 5, wherein the parade direction determination submodule specifically comprises:

and a downlink determining unit, configured to determine that the traveling direction of the successfully matched fish is downlink if the center of the position of the prediction frame of the successfully matched fish is in the third region of the k frame region, and the center of the position of the prediction frame of the successfully matched fish is in the fourth region of the k +1 frame region.