CN116758480B - Method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination - Google Patents
Method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination Download PDFInfo
- Publication number
- CN116758480B CN116758480B CN202310769965.6A CN202310769965A CN116758480B CN 116758480 B CN116758480 B CN 116758480B CN 202310769965 A CN202310769965 A CN 202310769965A CN 116758480 B CN116758480 B CN 116758480B
- Authority
- CN
- China
- Prior art keywords
- fishing boat
- fishing
- target
- preset
- boat
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000004927 fusion Effects 0.000 title claims abstract description 14
- 238000001514 detection method Methods 0.000 claims abstract description 84
- 238000001914 filtration Methods 0.000 claims abstract description 33
- 230000011218 segmentation Effects 0.000 claims abstract description 33
- 238000012549 training Methods 0.000 claims abstract description 23
- 238000012544 monitoring process Methods 0.000 claims description 56
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 125000004122 cyclic group Chemical group 0.000 claims description 2
- 239000000463 material Substances 0.000 description 15
- 238000012545 processing Methods 0.000 description 4
- 230000006978 adaptation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination, which comprises the following steps of S1, training a fishing boat detection model, a personnel detection model and a river surface segmentation model; s2, sending a fishing boat detection model and a personnel detection model, and sending a river surface segmentation model again if the model meets the requirements, so as to obtain a group of splitting events; step S3, filtering and splitting the event according to a preset parameter set, determining a fishing boat target, and otherwise, returning to the step S2; s4, analyzing a fishing boat target according to preset conditions of the fishing state of the fishing boat to obtain an effective event, otherwise, returning to the step S2; s5, calculating the speed of the fishing boat according to the front and rear effective events, judging whether the speed of the fishing boat is greater than a preset speed, and if so, eliminating illegal fishing operation, and returning to the step S2; and S6, calculating the comprehensive confidence coefficient of each group of targets according to the fishing boat grouping targets by taking the fishing boat as a unit, and obtaining an early warning event if the comprehensive confidence coefficient is higher than a preset threshold value. The invention improves the identification precision of fishing boat fishing and reduces the false alarm rate.
Description
Technical Field
The invention relates to the technical field of fishing vessel fishing target tracking, in particular to a method for identifying illegal fishing vessel fishing based on multi-model fusion and multi-parameter combination.
Background
The camera video monitoring and AI analysis has wide application in the fields of security protection, fishery supervision, vehicle license plate recognition and the like, and is more urgent in recognition of illegal fishing boat fishing events along with the ten-year forbidding of fishing in Yangtze river basin. The non-capturing identification multi-use AI algorithm model analyzes the monitoring video in real time, but the traditional model is easily influenced by a plurality of factors such as light, distance, angle, gesture and the like in the practical application environment. The current difficulty in identifying non-catching events is that ships are different in shape, random in azimuth and sailing speed, and are easily influenced by cargo ships, passenger ships, navigation mark ships, shorelines and the like, the number of people on the ships is different, the colors of clothes are different, the shapes of the people are different, the position of the people are frequently changed, fishing tools comprise fishing rods, dip nets and the like, and the training materials are difficult to collect, so that single model training is difficult, and the error rate of detected events is high. Therefore, improving the recognition accuracy of the non-capture event is a problem to be solved at present.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination, which improves the identification precision of illegal fishing boat fishing events, reduces false alarm rate and leads training materials to be difficult to collect.
The technical scheme of the invention is as follows:
a method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination comprises the following steps:
Step S1, training a fishing boat detection model, a personnel detection model and a river surface segmentation model;
S2, sending the monitoring video image collected by the camera into the fishing boat detection model and the personnel detection model for reasoning, if the monitoring video image meets the requirements of the fishing boat and the personnel at the same time, sending the monitoring video image into the river surface segmentation model for reasoning, obtaining a group of splitting events, otherwise, re-collecting the monitoring video image;
Step S3, filtering the splitting event according to a preset parameter set, determining a fishing boat target if the preset parameter is met, and returning to the step S2 if the preset parameter is not met;
s4, analyzing the fishing boat target according to the preset condition of the fishing state of the fishing boat, if the condition is met, obtaining an effective event, otherwise, returning to the step S2;
step S5, calculating the speed of the fishing boat according to the effective events, judging whether the speed of the fishing boat is greater than a preset speed, and if so, eliminating illegal fishing operation, and returning to the step S2;
And S6, grouping according to the fishing state of the fishing boat in the step S4, calculating the comprehensive confidence coefficient of each group, and if the comprehensive confidence coefficient is higher than a preset threshold value, obtaining an early warning event.
Further: further comprising step S7: uploading the early warning event to a monitoring platform and sending early warning information.
Further: step S1, training a fishing boat detection model, a personnel detection model and a river surface segmentation model, wherein the step S1 comprises the following steps:
the target detection models are a fishing boat detection model and a personnel detection model, and the fishing boat detection model and the personnel detection model are trained based on YOLOv;
The example segmentation model is a river surface segmentation model, and the river surface segmentation model is trained based on YOLOv-seg.
Further: step S2, firstly sending a monitoring video image acquired by a camera into the plurality of target detection models for reasoning, and sending the monitoring video image into the instance segmentation model for reasoning if a plurality of targets are met at the same time, so as to obtain a group of splitting events; otherwise, the method re-collects the monitoring video image, which comprises the following steps:
Extracting camera real-time monitoring video images according to a certain frame rate, sending the monitoring video images into the plurality of target detection models at certain intervals, sending the monitoring video images into the instance segmentation model for reasoning if a plurality of targets are met at the same time, obtaining a group of splitting events if a segmentation result exists, and simultaneously storing an original image; otherwise, the camera real-time monitoring video image is extracted again.
Further: step S3, filtering the splitting event according to a preset parameter set, determining a fishing boat target if the preset parameter is met, otherwise returning to step S2, and comprising the steps of:
step S3.1: removing irrelevant areas and blurred images according to the acquired preset point positions and motion states of the camera:
The camera preset point position is obtained through onvif protocol, and irrelevant areas are eliminated;
The camera motion state is obtained through a manufacturer SDK or onvif protocol;
Step S3.2: filtering according to different preset confidence degrees of the detection time period, discarding the target lower than the confidence degrees, and discarding the event if the filtered target meeting the confidence degrees is not met by the fishing boat, the personnel and the river surface;
Step S3.3: filtering according to a preset target size of the fishing boat, and determining the target of the fishing boat if the target size of the fishing boat is in accordance with the preset target size of the fishing boat; if the target of the fishing boat is smaller, the magnification camera is obtained according to the comparison of the fishing boat size (w a,ha) and the standard fishing boat size (w s,hs) And (2) adjusting the camera according to the magnification factor of the camera by adopting a linkage camera, carrying out 3D magnification on the fishing boat target, and returning to the step (S2) for repeated circulation after any amplified fishing boat target is not in line with the preset fishing boat target size requirement.
Further: the step S3.2 is as follows: the filtering is performed according to different confidence coefficient preset in the detection time period, and the filtering is performed according to different targets and different confidence coefficient preset in the detection time period.
Further: step S4, analyzing the fishing boat target according to the preset condition of the fishing state of the fishing boat, if the condition is met, obtaining an effective event, otherwise, returning to step S2, wherein the step S comprises the following steps:
Step S4.1: analyzing the integrity of the fishing boat target, taking out the fishing boat target position information (x a,ya,wa,ha) and the target image size (w o,ho), calculating the integrity B＝(xa＞10&(xa+wa-wo)＞10)&(ya＞5&(ya+ha-ho)＞5), of the fishing boat, if B is false, the fishing boat is incomplete, and removing the fishing boat target;
S4.2, respectively calculating the intersection ratio of the fishing boat target and the personnel target, the fishing rod target, the dip net target and the trawl target, excluding the onshore target, grouping the targets according to the fishing boat, and discarding the event if the intersection ratio is smaller than 0.15;
s4.3, sequentially calculating the distance D from the fishing boat target to the river surface shoreline by using opencv, and obtaining an effective event if the distance is positive and the fishing boat is positioned on the river surface; otherwise, the fishing boat is in the berthing on the shore or outside the shoreline, and the step S2 is returned to repeat the cycle.
Further: and S5, calculating the speed of the fishing boat according to the effective events, judging whether the speed of the fishing boat is greater than a preset speed, and if so, eliminating illegal fishing operation, returning to the step S2, wherein the step S comprises the following steps:
The fishing boat speed is calculated by the effective event identified by the front and back frames, under normal conditions, the fishing boat is slowly moved or is static during fishing operation, the fishing boat targets in the effective event of the front and back frames are circularly compared, the fishing boat targets of the front and back frames are respectively B, N, and the similarity of the two is calculated
S=1-2×iou+c+w+h, wherein IoU is the front-back target overlap ratio, C is the confidence difference absolute value c= |c B-CN |, W, H is the wide-high ratio difference And satisfies IoU >0.1, C <0.1, W, H <0.15, and the minimum S value of the front and rear target calculation speeds;
When (when) The fishing boat runs slowly and is far away from the shoreline, so that illegal fishing operation is met;
When (when) The running speed of the fishing boat is moderate, continuous observation is needed, and calculation errors caused by camera shake are prevented;
when V is more than max (w, h), the fishing boat runs fast, illegal fishing operation is eliminated, and the step S2 is returned to carry out cyclic operation.
Further: in the step S5, the speed of the fishing boat is calculated according to the effective events before and after, and before judging whether the speed of the fishing boat is greater than a preset speed, the method further comprises the following steps:
and caching the effective event after the filtering, taking out the target of the last effective event for comparison, and if the targets of the same type of labels exist, eliminating the abnormal influence of the image and detecting the output error of the model.
Further: the step S6: taking a fishing boat as a unit, grouping targets according to the fishing boat, calculating the comprehensive confidence coefficient of each group of targets, and if the comprehensive confidence coefficient is higher than a preset threshold value, obtaining an early warning event, further comprising:
Filtering according to the preset early warning event time interval of the preset point position, calculating the similarity S F of the fishing boat in the last early warning event of the same preset point position, and if S F is smaller than 0.5 and the time interval is smaller than the preset early warning event time interval value, considering the fishing boat as the same fishing boat target, and removing the fishing boat and the boat target.
The technical scheme has the following beneficial effects:
The invention solves the problems of low identification precision, easy false report and difficult material training of the existing illegal fishing boat fishing event, establishes a multi-target detection model of the fishing boat and personnel and a river surface segmentation model, realizes the combined application of the target detection model and the example segmentation model, does not use a classification model, greatly reduces the collection difficulty of the training material, reduces the deep width requirement of a model network and the hardware cost, and is convenient for quick training, conversion and on-line deployment and replacement of the model. While converting a single non-captured event into a combination of multiple events for multi-feature recognition. Filtering the event according to the identification time, the target confidence coefficient and the size parameter to obtain a fishing boat target; and obtaining an effective event according to preset conditions of fishing states of the fishing vessel such as the intersection ratio of the fishing vessel, personnel, a fishing rod, a dip net and a trawl in the fishing vessel target, the integrity and the like, and finally obtaining an early warning event according to whether the fishing vessel target movement speed and the shoreline distance are illegal fishing vessel fishing events or not. The identification method has the advantages that the detection rate and the precision of a single target are improved greatly, and the false alarm condition of the illegal fishing boat fishing event is reduced.
Before the speed of the fishing boat is calculated according to the effective events, the effective events after the filtering are cached, the last effective event targets are taken out for comparison, if targets with the same type of labels exist, the abnormal influence of images and the output errors of a detection model are eliminated, and multi-frame detection is adopted, so that the continuity of the effective event detection is ensured.
Filtering according to the preset early warning event time interval of the preset point position, calculating the similarity S F of the fishing boat in the last early warning event of the same preset point position, and if S F is smaller than 0.5 and the time interval is smaller than the preset early warning event time interval value, considering the fishing boat as the same fishing boat target, and removing the fishing boat and the boat target. Because the fishing boat works slowly or stays for a long time, the same fishing boat can be repeatedly identified, and therefore, the reporting time interval needs to be set, and the system resource consumption and the manual processing cost are reduced.
Further description is provided below with reference to the drawings and detailed description.
Drawings
FIG. 1 is a block flow diagram of an embodiment 1;
FIG. 2 is a flow chart of embodiment 1;
fig. 3 is a flow chart of embodiment 2.
Detailed Description
Specific example 1:
As shown in fig. 1 and 2, the method for identifying the illegal fishing boat fishing based on the combination of multiple models and multiple parameters is based on an illegal fishing boat fishing system constructed by a camera, a network and a server, wherein the camera is connected with an AI operation box, the AI operation box is connected with the server through a network, the network is a wired network or a wireless network, and the camera adopts a linkage camera.
Step S1, training a fishing boat detection model, a personnel detection model and a river surface segmentation model;
the fishing boat detection model and the personnel detection model form a target detection model, and the river surface segmentation model is an example segmentation model and comprises the following components:
And training a fishing boat detection model and a personnel detection model based on YOLOv. The method specifically comprises the following steps:
The training fishing boat detection model adopts the method that the image materials of a marked cargo boat, a navigation mark boat, a passenger boat and a fishing boat are collected in the same proportion based on camera monitoring, the materials are collected in the daytime and at night, and the existing open source algorithm YOLOv is used for training the fishing boat detection model; calculating a standard size (w s,hs) of the fishing boat by using a K-Mean clustering algorithm, wherein w is wide and h is high; and counting the relationship M 0 between the scaling multiple of the camera and the size of the fishing boat in the image.
The training personnel detection model adopts the steps of collecting and marking personnel standing, squatting, sitting, fishing rod, dip net, trawl and umbrella materials, and training personnel and fishing tool detection models, namely training personnel detection models, by using the existing open source algorithm YOLOv. Under normal conditions, whether the fishing boat works or not is judged, and personnel and fishing tools on the boat need to be detected, so that the identification precision of the non-fishing event can be greatly improved. Because the acquisition difficulty of the fishing tool materials from the camera monitoring is high, the fishing tool materials are acquired from the network first and then gradually replaced according to the actual detection event.
Training a river surface segmentation model based on YOLOv-seg, specifically comprising:
The river surface segmentation model adopts a camera-based monitoring and collecting marked river surface materials, the materials at day and night are respectively half of the materials at day and night, and the conventional YOLOv-seg training river surface segmentation model is used. Usually, the ship is operated in the river and is located at a certain distance from the coastline, and the adoption of the division of the river surface is beneficial to filtering out the fishing ship targets berthed on the shore and the coast.
In the specific embodiment, the fishing boat and the river surface are manually intercepted from monitoring, the initial stage of personnel, fishing rods, dip nets and the like are collected from a network, and the later stage of the fishing boat and the river surface are real pictures which are selected from early warning events in a gradual iteration mode. According to the invention, a single non-capturing event is converted into a combination of a plurality of events identified by fishing boats, personnel and river surfaces, then the detection model is used for detecting the boats and the personnel, and if the boats and the personnel exist simultaneously, the river surface shoreline is segmented by using the segmentation model, so that a group of split events are obtained, and the system performance is improved.
S2, sending the monitoring video image collected by the camera into the fishing boat detection model and the personnel detection model for reasoning, and if the monitoring video image meets the requirements of the fishing boat and the personnel at the same time, sending the monitoring video image into the river surface segmentation model for reasoning to obtain a group of splitting events; otherwise, the monitoring video image is collected again. Comprising the following steps:
Extracting camera real-time monitoring video images according to a certain frame rate, sending the monitoring video images into the fishing boat detection model and the personnel detection model at certain intervals, sending the monitoring video images into the river surface segmentation model for reasoning if the fishing boat and the personnel are simultaneously met, obtaining a group of splitting events if a segmentation result exists, and simultaneously storing an original image; otherwise, the camera real-time monitoring video image is extracted again.
In this embodiment, the cameras are fixedly installed at a plurality of location points to monitor, and the cameras are linked cameras, that is, the cameras can be controlled to rotate, and operation and maintenance personnel can set some fixed monitoring places according to the river surface. The camera can only monitor one place at the same time and switch to the next place at regular intervals.
And extracting N frames of real-time monitoring video images every second by using an OpenCV (open control system) according to the number of cameras by a first thread, storing the N frames of real-time monitoring video images into a queue Q, taking out the monitoring video images from the Q by a second thread, and respectively sending the monitoring video images into the fishing boat detection model and the personnel detection model at certain intervals for reasoning. The interval time is set manually by event target continuity and saving system resources, and the embodiment sets one second for each camera. Judging whether fishing boat and personnel targets exist at the same time, if so, storing the detection result and continuing to send the detection result into the river surface segmentation model for reasoning; otherwise, re-extracting the real-time monitoring video image of the camera, taking out the monitoring video image, respectively sending the monitoring video image into the fishing boat detection model and the personnel detection model for reasoning and judgment, and repeating the cycle. If a river surface segmentation result exists, a group of splitting event sets is obtained, meanwhile, an original image is saved, the time for saving the original image is not limited, the obtained original image is saved in an AI box so as to draw pictures and synthesize videos at a later stage, and the processed pictures and videos are uploaded to a monitoring platform, and in the specific embodiment, the original image is saved for 10 seconds.
Step S3, filtering the splitting event according to a preset parameter set, determining a fishing boat target if the preset parameter is met, otherwise, returning to the step S2, and specifically, the step S:
step S3.1: removing irrelevant areas and blurred images according to the acquired preset point positions and motion states of the camera; comprising the following steps:
the camera preset point position is obtained through onvif protocol, and for monitoring and identifying fishing operation, a plurality of monitoring points are arranged along the river surface and the shoreline to exclude a plurality of irrelevant areas, so that the identification precision is improved. If present on the river shores, grasslands, etc., are ineffective. Setting the camera to cruise only on the river surface can find valid events most probably.
The camera motion state is obtained through a manufacturer SDK or onvif protocol; if the camera moves at a high speed, the monitored image is blurred, the target position changes greatly, and the speed calculation is inaccurate, so that events generated in the camera movement need to be eliminated. According to the real-time position judgment of the camera, the embodiment adopts the steps of taking two camera positions at intervals of 0.2 seconds for comparison, if no change exists, the camera positions are not moved, and the camera positions are obtained through onvif protocols.
Step S3.2: filtering according to different confidence degrees preset in the detection time period, wherein the light is sufficient in daytime, more training materials are used, the model detection rate is higher, more false detection is carried out, and the higher confidence degree is set; if the camera generates a gray level image through laser imaging at night, the target features are fewer, detection is fewer, the confidence of the model is reduced, and the output of detection events is increased. The higher the confidence of the model output is, the better, so that only the lowest confidence is required to be set, and the confidence is a decimal between 0 and 1. And according to the comparison of the confidence coefficient output by the fishing boat detection model and the personnel detection model with different preset confidence coefficients, discarding the target lower than the confidence coefficient, and if the filtered target which does not meet the confidence coefficient is not available for the fishing boat, personnel and river, discarding the event.
Specific: firstly, filtering the confidence coefficient of the fishing boat, and if the confidence coefficient requirement of the fishing boat is met; filtering the confidence coefficient of the personnel, and if the confidence coefficient requirement of the fishing boat is met; finally, filtering the river surface confidence coefficient; if any link does not meet the requirement, the subsequent link processing is not needed, and the event is directly discarded.
Possibly, the filtering is performed by presetting different confidence degrees according to the detection time period, and further includes filtering by presetting different confidence degrees according to different targets and the detection time period. Setting confidence degrees for different targets at daytime and nighttime, and discarding targets lower than the confidence degrees. For the targets with obvious characteristics such as fishing boats, personnel, umbrellas and the like, setting 0.85 in daytime and 0.8 at night; for targets with unobvious characteristics such as fishing rods, dip nets, trawl nets and the like and low detection rate, setting 0.75 in daytime and 0.6 at night; for river surface division, day time was set to 0.8 and night time was set to 0.65.
Step S3.3: and filtering according to the preset target size of the fishing boat, if the target size of the fishing boat is met, comparing the length and the width of the fishing boat with the length and the width of the preset fishing boat, determining the target of the fishing boat if any one of the target size and the width of the fishing boat is met, and entering the next step. If the target of the fishing boat is smaller, when the fishing boat detection model is trained according to the step 1, calculating a standard size (w s,hs) of the fishing boat by using a K-Mean clustering algorithm, counting the relationship M 0 between the scaling factor of the camera and the size of the fishing boat in the image, and comparing the target size (w a,ha) of the fishing boat with the standard size (w s,hs) of the fishing boat to obtain the scaling factor of the cameraAdjusting a linkage camera according to the magnification factor M of the magnification camera, and performing 3D magnification on the fishing boat target; and (3) repeating the steps 1 to 3, and discarding any amplified event which does not meet the target size requirement of the preset fishing boat. Because the small-size targets have fewer characteristics, the recognition accuracy of the target detection model on the small targets is low, and therefore the small targets need to be filtered out by setting reasonable sizes. In this embodiment: according to practical experience, taking a fishing boat standard size (1920,1080), a fishing boat target size is preset as (150,75), namely, the fishing boat target size is preset, personnel target and umbrella target sizes are preset as (75, 75), a fishing rod target, a landing net target and a trawl target size are preset as (140,60), and if the fishing boat target after filtering does not meet the size, an event is eliminated; if the filtered fishing boat target meets the size; then judging whether the filtered personnel target and the filtered umbrella target meet the sizes, and if so, judging whether the personnel target and the umbrella target meet the sizes; finally judging whether the filtered fishing rod target, the trawl target and the trawl target meet the size; if any link does not meet the requirement, the subsequent link processing is not needed, the event is directly discarded, and the step S2 is returned to repeat the cycle.
And S4, analyzing the fishing boat target according to the preset condition of the fishing state of the fishing boat, if the condition is met, obtaining an effective event, otherwise, returning to the step S2. Comprising the following steps:
step S4.1: and analyzing the integrity of the fishing boat target, and taking out the fishing boat target position information (x a,ya,wa,ha) and the target image size (w o,ho) to calculate the fishing boat integrity B＝(xa＞10&(xa+wa-wo)＞10)&(ya＞5&(ya+ha-ho)＞5),, wherein the fishing boat integrity B＝(xa＞10&(xa+wa-wo)＞10)&(ya＞5&(ya+ha-ho)＞5), indicates that the target is within 10 pixels on the left or right of the image, and the target can be considered to be incomplete within 5 pixels on the upper or lower side of the image, because the model can also identify the incomplete target, but the error rate is high, namely B is false.
If B is false, the fishing boat is incomplete, and the incomplete fishing boat target in the target image is removed. Incomplete fishing boat cargo vessel, navigation mark ship, stone extremely easily report into the fishing boat by mistake, consequently need get rid of the fishing boat at image edge, because the mobility of fishing boat and camera monitoring analysis's continuity, the fishing boat can appear in image central authorities at last, and the condition of missing to examine is very few, increases substantially and detects the rate of accuracy.
Step S4.2: and respectively calculating the intersection ratio of the fishing boat target and the personnel target, the fishing rod target, the dip net target and the trawl target, and eliminating the onshore target and the grouping target according to the fishing boat. If the intersection ratio of the fishing boat target and the personnel is less than 0.15, the fishing boat target and the personnel are all the discarding events. If the intersection ratio of the fishing boat target and the personnel is more than or equal to 0.15, then judging that any one of the intersection ratio of the fishing boat target and the fishing rod target, the dip net target and the trawl target meets the intersection ratio of more than or equal to 0.15, and entering the next step. The method adopts the method for judging the cross-over ratio of the related targets on the ship, mainly aims at eliminating targets which are not on the fishing ship, and is more accurate than the method for judging the use distance when people are generally on the fishing ship during fishing. Taking personnel as an example, the personnel target position information (x b,yb,wb,hb) is sequentially compared with the fishing vessel target position information (x a,ya,wa,ha) and the area intersection I is calculated.
If y b-hb＞ya-ha|wb＞wa is true, discarding the fishing boat, and eliminating false alarm of the fishing boat;
If xa＞＝xb+wb|xa+wa＜＝xb|y＞＝yb+hb|ya+ha＜＝yb is true, i=0; otherwise I＝(min(xa+wa,xb+wb)-max(xa,xb))*(min(ya+ha,yb+hb)-max(ya,yb), when the cross-over ratio When the number of the workers is less than 0.15, the workers are removed, and the shoreside workers and other operators are excluded. And (5) calculating the cross ratio of the fishing rod target and the fishing boat target in sequence according to the personnel target cross ratio method, and discarding the fishing rod if the cross ratio is smaller than 0.15. And if the intersection ratio of the dip net target and the fishing boat target is smaller than 0.15, the dip net is removed. And if the intersection ratio of the trawl target and the fishing boat target is smaller than 0.15, the trawl is removed.
Step S4.3: sequentially calculating the distance D from a fishing boat target to a river surface shoreline by using a cross-platform computer vision and machine learning software library OpenCV, and obtaining an effective event if the distance is a positive number and the fishing boat is positioned on the river surface; otherwise, the fishing boat is in the berthing on the shore or outside the shoreline, and the step S2 is returned to repeat the cycle. According to practical application experience, when the fishing boat is out of the shoreline or berthed alongside the fishing boat at the distance Dv0, the fishing boat is an invalid event, and the step S2 is returned. When the distance D is more than 0, the fishing boat is positioned in the river surface, and an effective event is obtained.
And S5, calculating the speed of the fishing boat according to the effective events, judging whether the speed of the fishing boat is greater than a preset speed, and if so, eliminating illegal fishing operation, and returning to the step S2. Comprising the following steps:
The method comprises the steps of calculating the speed of a fishing boat through effective events identified by front and rear frames, moving slowly or stillly during fishing boat fishing operation under normal conditions, circularly comparing fishing boat targets in the effective events of the front and rear frames, setting the fishing boat targets of the front and rear frames as B, N respectively, and calculating the similarity S=1-2 x IoU+C+W+H of the fishing boat targets, wherein IoU is the cross-over ratio of the front and rear targets C is the absolute value of the confidence difference, C= |C B-CN |, W, H is the wide-high ratio difference/>The front and rear target calculation speeds with minimum S values are met, ioU >0.1, C <0.1, W and H <0.15, and the front and rear events are adopted to be effective in 3 seconds;
In the formula, t B、tN represents the time of the front and rear frame fishing vessel target B, N event respectively.
When (when)The fishing boat runs slowly and is far away from the shoreline, so that illegal fishing operation is met;
When (when) The running speed of the fishing boat is moderate, continuous observation is needed, and calculation errors caused by camera shake are prevented;
When V is larger than max (w, h), representing the moving speed of the target fishing boat on the monitoring picture, taking pixels as a unit, if the moving speed is larger than the maximum value max (w, h) of the target length and width, considering that the fishing boat is fast to run, excluding illegal fishing operation, and returning to the step S2 for repeated circulation.
Possibly, in the step S5, the method further includes calculating the speed of the fishing boat according to the effective events, and determining whether the speed of the fishing boat is greater than a preset speed:
buffering the effective event after the filtering, in this embodiment: and caching the effective event after the filtering for 3 seconds, taking out the target of the last effective event for comparison, and if the targets of the same type of labels exist, eliminating the abnormal influence of the image and detecting the output error of the model.
And S6, calculating the comprehensive confidence coefficient of each group of targets according to the fishing boat grouping targets by taking the fishing boat as a unit, and obtaining an early warning event if the comprehensive confidence coefficient is higher than a preset threshold value. Comprising the following steps: taking a fishing boat as a unit, grouping according to the intersection ratio of the fishing boat target and the personnel target, the fishing rod target, the dip net target and the trawl target calculated in the step S4.2, and calculating the comprehensive confidence coefficient of each group(N is the number of targets in each group, C i is the confidence corresponding to each target), removing event groups smaller than the threshold value, and caching event groups higher than the threshold value to obtain early warning events. The specific embodiment is as follows: a fishing boat is taken as a main body, and people crossing the fishing boat and fishing tools are taken as a group of events. If there are m fishing vessels there are m sets of events. And counting the threshold according to the early warning event, wherein the threshold is 0.8, and if the threshold is higher than the event group of 0.8, the event group belongs to the early warning event.
Possibly, the step S6 is to calculate the comprehensive confidence coefficient of each group of targets according to the grouping targets of the fishing boat by taking the fishing boat as a unit, and if the comprehensive confidence coefficient is higher than a preset threshold value, the step S6 further comprises the following steps:
Filtering according to preset early warning event time intervals of preset points, calculating the similarity S F of the fishing boat in the last early warning event of the same preset point according to the similarity S method of the preset points of the camera obtained in the step S3 and the fishing boat targets of the front frame and the rear frame calculated in the step S5, and setting the early warning event time interval according to system storage and related requirements for generally 5 minutes if S F is smaller than 0.5 and the time interval is smaller than the preset early warning event time interval value.
Then the same fishing vessel target is considered and the fishing vessel and the on-board target are removed. Because the fishing boat works slowly or stays for a long time, the same fishing boat can be repeatedly identified, and therefore, the reporting time interval needs to be set, and the system resource consumption and the manual processing cost are reduced.
According to the invention, the identification of the single illegal fishing boat fishing event is converted into the combined detection of targets such as river surface, fishing boat, personnel, fishing rod, dip net and the like, and the combined application of the YOLOv target detection model and the example segmentation model greatly reduces the collection difficulty of training materials, and simultaneously improves the detection rate and the precision of the single target. Meanwhile, the deep width requirement and hardware cost of the model network are reduced, and the model is convenient to train, convert and deploy and replace online.
The filtering event is optimized through the combination of the parameters such as the confidence coefficient, the detection time period, the size, the integrity, the intersection ratio, the river bank line distance, the movement speed and the like, so that the overall error probability of the system is reduced, and the overall detection rate of the system is increased. The problems of low identification precision, easy false alarm and difficult collection of training materials of the existing illegal fishing boat fishing event are solved, and the precision requirement of the identification method is improved.
The technical scheme provided by the invention is described in detail. The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the method of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
Example 2
Referring to fig. 3, step S7: and drawing a detection frame and synthesizing a video according to preset configuration, uploading the early warning event to a monitoring platform, sending early warning information and sending a push event. The preset configuration in this embodiment includes: the color, width, event type, confidence font size, color, etc. of the wire are detected.
And (2) drawing a detection frame and synthesizing a video by utilizing the original image stored by the AI box in the step (S2) and uploading the early warning event to a monitoring platform, so that the monitoring platform can conveniently conduct remote control and timely inform and remind fishery monitoring personnel.
Other features are the same as those of embodiment 1, so this embodiment is omitted here.
Claims (8)
1. A method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination is characterized in that:
Step S1, training a fishing boat detection model, a personnel detection model and a river surface segmentation model;
S2, sending the monitoring video image collected by the camera into the fishing boat detection model and the personnel detection model for reasoning, if the monitoring video image meets the requirements of the fishing boat and the personnel at the same time, sending the monitoring video image into the river surface segmentation model for reasoning, obtaining a group of splitting events, otherwise, re-collecting the monitoring video image;
Step S3, filtering the splitting event according to a preset parameter set, determining a fishing boat target if the preset parameter is met, and returning to the step S2 if the preset parameter is not met; comprising the following steps:
step S3.1: removing irrelevant areas and blurred images according to the acquired preset point positions and motion states of the camera:
The camera preset point position is obtained through onvif protocol, and irrelevant areas are eliminated;
The camera motion state is obtained through a manufacturer SDK or onvif protocol;
Step S3.2: filtering according to different preset confidence degrees of the detection time period, discarding the target lower than the confidence degrees, and discarding the event if the filtered target meeting the confidence degrees is not met by the fishing boat, the personnel and the river surface;
Step S3.3: filtering according to a preset target size of the fishing boat, and determining the target of the fishing boat if the target size of the fishing boat is in accordance with the preset target size of the fishing boat; if the target of the fishing boat is smaller, the magnification camera is obtained according to the comparison of the fishing boat size (w a,ha) and the standard fishing boat size (w s,hs) The camera adopts a linkage camera, the camera is adjusted according to the magnification of the camera, 3D magnification is carried out on the fishing boat target, any amplified fishing boat target which does not meet the size requirement of the preset fishing boat target is removed, and the step S2 is returned to repeat the cycle;
S4, analyzing the fishing boat target according to the preset condition of the fishing state of the fishing boat, if the condition is met, obtaining an effective event, otherwise, returning to the step S2; comprising the following steps:
Step S4.1: analyzing the integrity of the fishing boat target, taking out the fishing boat target position information (x a,ya,wa,ha) and the target image size (w o,ho), calculating the integrity B＝(xa＞10&(xa+wa-wo)＞10)&(ya＞5&(ya+ha-ho)＞5), of the fishing boat, if B is false, the fishing boat is incomplete, and removing the fishing boat target;
S4.2, respectively calculating the intersection ratio of the fishing boat target and the personnel target, the fishing rod target, the dip net target and the trawl target, excluding the onshore target, grouping the targets according to the fishing boat, and discarding the event if the intersection ratio is smaller than 0.15;
s4.3, sequentially calculating the distance D from the fishing boat target to the river surface shoreline by using opencv, and obtaining an effective event if the distance is positive and the fishing boat is positioned on the river surface; otherwise, the fishing boat is in the berthing of the shore or outside the shore line, and returns to the step S2 to repeat the cycle;
step S5, calculating the speed of the fishing boat according to the effective events, judging whether the speed of the fishing boat is greater than a preset speed, and if so, eliminating illegal fishing operation, and returning to the step S2;
And S6, calculating the comprehensive confidence coefficient of each group of targets according to the fishing boat grouping targets by taking the fishing boat as a unit, and obtaining an early warning event if the comprehensive confidence coefficient is higher than a preset threshold value.
2. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1, wherein the method comprises the following steps: further comprising step S7: uploading the early warning event to a monitoring platform and sending early warning information.
3. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1 or 2, wherein the method comprises the following steps: the step S1 is to train a fishing boat detection model, a personnel detection model and a river surface segmentation model, and comprises the following steps:
training a fishing boat detection model and a personnel detection model based on YOLOv;
a river surface segmentation model is trained based on YOLOv-seg.
4. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1 or 2, wherein the method comprises the following steps: step S2, sending the monitoring video image collected by the camera into the fishing boat detection model and the personnel detection model to infer, if the monitoring video image meets the requirements of the fishing boat and the personnel at the same time, sending the monitoring video image into the river surface segmentation model to infer, obtaining a group of splitting events, otherwise, re-collecting the monitoring video image, wherein the step S2 comprises the following steps:
Extracting camera real-time monitoring video images according to a certain frame rate, sending the monitoring video images into the fishing boat detection model and the personnel detection model at certain intervals, sending the monitoring video images into the river surface segmentation model for reasoning if the fishing boat and the personnel are simultaneously met, obtaining a group of splitting events if a segmentation result exists, and simultaneously storing an original image; otherwise, the camera real-time monitoring video image is extracted again.
5. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1, wherein the method comprises the following steps: the step S3.2 is as follows: the filtering is performed according to different confidence coefficient preset in the detection time period, and the filtering is performed according to different targets and different confidence coefficient preset in the detection time period.
6. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1 or 2, wherein the method comprises the following steps: and S5, calculating the speed of the fishing boat according to the effective events, judging whether the speed of the fishing boat is greater than a preset speed, and if so, eliminating illegal fishing operation, returning to the step S2, wherein the step S comprises the following steps:
The method comprises the steps of calculating the speed of a fishing boat through an effective event identified by a front frame and a rear frame, moving slowly or stillly during fishing boat fishing operation under normal conditions, circularly comparing fishing boat targets in the effective event of the front frame and the rear frame, setting the fishing boat targets in the front frame and the rear frame as B, N, calculating similarity S=1-2 times IoU+C+W+H of the fishing boat targets, wherein IoU is the front-rear target intersection ratio, C is the absolute value C= |C B-CN |, W and H are the wide-high proportion difference value And satisfies IoU >0.1, C <0.1, W, H <0.15, and the minimum S value of the front and rear target calculation speeds;
When (when) The fishing boat runs slowly and is far away from the shoreline, so that illegal fishing operation is met;
When (when) The running speed of the fishing boat is moderate, continuous observation is needed, and calculation errors caused by camera shake are prevented;
when V is more than max (w, h), the fishing boat runs fast, illegal fishing operation is eliminated, and the step S2 is returned to carry out cyclic operation.
7. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1 or 2, wherein the method comprises the following steps: in the step S5, the speed of the fishing boat is calculated according to the effective events before and after, and before judging whether the speed of the fishing boat is greater than a preset speed, the method further comprises the following steps:
and caching the effective event after the filtering, taking out the target of the last effective event for comparison, and if the targets of the same type of labels exist, eliminating the abnormal influence of the image and detecting the output error of the model.
8. The method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination according to claim 1 or 2, wherein the method comprises the following steps: and S6, calculating the comprehensive confidence coefficient of each group of targets according to the group targets of the fishing boat by taking the fishing boat as a unit, and if the comprehensive confidence coefficient is higher than a preset threshold value, obtaining an early warning event, further comprising:
Filtering according to the preset early warning event time interval of the preset point position, calculating the similarity S F of the fishing boat in the last early warning event of the same preset point position, and if S F is smaller than 0.5 and the time interval is smaller than the preset early warning event time interval value, considering the fishing boat as the same fishing boat target, and removing the fishing boat and the boat target.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310769965.6A CN116758480B (en) | 2023-06-27 | 2023-06-27 | Method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310769965.6A CN116758480B (en) | 2023-06-27 | 2023-06-27 | Method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116758480A CN116758480A (en) | 2023-09-15 |
| CN116758480B true CN116758480B (en) | 2024-04-19 |
Family
ID=87960664
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310769965.6A Active CN116758480B (en) | 2023-06-27 | 2023-06-27 | Method for identifying illegal fishing boat fishing based on multi-model fusion and multi-parameter combination |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116758480B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111222526A (en) * | 2018-11-27 | 2020-06-02 | 中电科海洋信息技术研究院有限公司 | Fishing boat real-time fishing behavior recognition method, device, equipment and storage medium |
| CN113269121A (en) * | 2021-06-08 | 2021-08-17 | 兰州大学 | Fishing boat fishing state identification method based on fusion CNN model |
| CN113591540A (en) * | 2021-06-02 | 2021-11-02 | 中国水产科学研究院南海水产研究所 | Method and system for monitoring and early warning illegal fishing in fishery |
| KR20220025537A (en) * | 2020-08-24 | 2022-03-03 | 제주대학교 산학협력단 | Method for Identification of Fishing Gear Type using Trajectory Data |
-
2023
- 2023-06-27 CN CN202310769965.6A patent/CN116758480B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111222526A (en) * | 2018-11-27 | 2020-06-02 | 中电科海洋信息技术研究院有限公司 | Fishing boat real-time fishing behavior recognition method, device, equipment and storage medium |
| KR20220025537A (en) * | 2020-08-24 | 2022-03-03 | 제주대학교 산학협력단 | Method for Identification of Fishing Gear Type using Trajectory Data |
| CN113591540A (en) * | 2021-06-02 | 2021-11-02 | 中国水产科学研究院南海水产研究所 | Method and system for monitoring and early warning illegal fishing in fishery |
| CN113269121A (en) * | 2021-06-08 | 2021-08-17 | 兰州大学 | Fishing boat fishing state identification method based on fusion CNN model |
Non-Patent Citations (1)
| Title |
|---|
| 基于可变形部件模型的渔船安全监控系统;洪志恒;陈明;秦玉芳;李净;;计算机应用与软件;20180228(第02期);第188-193页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116758480A (en) | 2023-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA3100569C (en) | Ship identity recognition method base on fusion of ais data and video data | |
| CN109190636B (en) | Remote sensing image ship target information extraction method | |
| Alpatov et al. | Vehicle detection and counting system for real-time traffic surveillance | |
| KR20200007084A (en) | Ship detection method and system based on multi-dimensional features of scene | |
| CN112232349A (en) | Model training method, image segmentation method and device | |
| CN102867196B (en) | Based on the complicated sea remote sensing image Ship Detection of Gist feature learning | |
| CN112800860B (en) | High-speed object scattering detection method and system with coordination of event camera and visual camera | |
| CN113324864B (en) | Pantograph carbon slide plate abrasion detection method based on deep learning target detection | |
| CN106331636A (en) | Intelligent video monitoring system and method of oil pipelines based on behavioral event triggering | |
| Bedruz et al. | Real-time vehicle detection and tracking using a mean-shift based blob analysis and tracking approach | |
| CN112149550A (en) | Automatic driving vehicle 3D target detection method based on multi-sensor fusion | |
| CN111178182A (en) | Real-time detection method for garbage loss behavior | |
| CN107948465A (en) | A kind of method and apparatus for detecting camera and being disturbed | |
| CN109817013A (en) | Parking stall state identification method and device based on video flowing | |
| CN104966304A (en) | Kalman filtering and nonparametric background model-based multi-target detection tracking method | |
| CN108596169A (en) | Piecemeal signal conversion based on video streaming image and object detection method and device | |
| CN112069985B (en) | High-resolution field image rice spike detection and counting method based on deep learning | |
| Tourani et al. | Motion-based vehicle speed measurement for intelligent transportation systems | |
| CN110636281B (en) | Real-time monitoring camera shielding detection method based on background model | |
| CN111461078A (en) | Anti-fishing monitoring method based on computer vision technology | |
| CN109919026A (en) | A kind of unmanned surface vehicle local paths planning method | |
| CN110321855A (en) | A kind of greasy weather detection prior-warning device | |
| Su et al. | A new local-main-gradient-orientation HOG and contour differences based algorithm for object classification | |
| CN113034378A (en) | Method for distinguishing electric automobile from fuel automobile | |
| CN115240086A (en) | Unmanned aerial vehicle-based river channel ship detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |