CN115100732A - Phishing detection method, device, computer equipment and storage medium - Google Patents

Abstract

The invention discloses a fishing detection method, a fishing detection device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a target detection frame comprising a pedestrian and a fishing rod; performing phishing behavior recognition on the target detection frame through the phishing behavior detection model, and outputting a first result of which the recognition result is 'phishing behavior'; carrying out fishing behavior recognition on the target detection frame through the human fishing rod key point detection model, and outputting a second result of which the recognition result is 'fishing behavior'; and voting the first result and the second result to output a final recognition result. The fishing detection method provided by the invention can be used for detecting and managing the sneaking fishing behaviors in 24 hours all day under the scene of day and night, so that the labor cost can be greatly saved, the illegal fishing behaviors in a large water area can be identified in 24 hours all day at low cost, the management level of the water area in which fishing is forbidden is improved, the final identification result is output by voting according to the average value of the first result and the second result, and the detection accuracy is improved.

Description

Phishing detection method, device, computer equipment and storage medium

technical field

The present invention relates to the technical field of computer visual recognition, and in particular, to a fishing detection method, device, computer equipment and storage medium.

Background technique

Fishing is an outdoor leisure activity with a large audience. You can often see fishing enthusiasts by rivers, lakes or reservoirs. However, for management needs and commercial interests, fishing is not allowed in many places, such as scenic lakes. , Privately contracted reservoirs. However, there are still some poaching behaviors from time to time, which not only bring hidden dangers to the safety management of the relevant waters, but also may cause personal injury to the poachers themselves, such as accidental falling into the water, drowning, or touching high-voltage electricity, etc. , and the waste generated in the process of poaching is also very likely to cause water pollution and so on.

Existing methods for prohibiting anglers from illegal fishing in controlled waters mainly include setting up warning signs prohibiting fishing in corresponding areas, or arranging special management personnel to conduct patrols. However, in practice, it is found that setting up warning signs cannot effectively drive away illegal fishermen, and arranging special management personnel to patrol is a huge waste of human resources, and it cannot achieve 24-hour patrol monitoring.

In view of this, some existing technologies also provide means based on computer vision recognition to perform target detection, recognition and management on illegal phishers. Object detection is one of the basic tasks in the field of computer vision, and it has been studied in academia for nearly two decades. In recent years, with the rapid development of deep learning technology, the target detection algorithm has also shifted from the traditional algorithm based on manual features to the detection technology based on deep neural network.

However, the main principle of the existing computer illegal phishing identification technology is based on the traditional pixel comparison method between the front and back frames. The principle of the pixel comparison method is to compare the frame data before and after the collected image by collecting real-time images of the area to be detected. Specifically, the image pixel difference values at time k and time k+1 are calculated to detect the image difference between the previous and the previous time. Then, the probability distribution of the difference pixel values is calculated and compared with the existing fishing image database to determine whether there is fishing behavior. Due to the limited fishing image data that can be collected, that is, the distribution of standard data that can be compared is limited, and the numerical distribution of pixel differences collected and calculated is random and diverse, resulting in two problems with this method: (1) Often the Pedestrians mistaken for anglers; (2) no fishing behavior can be detected. At the same time, the image data collected by the existing computer illegal fishing identification technology are mainly visible light images, which are more sensitive to light, and many fishing enthusiasts like to fish at night. Therefore, under the cover of night, these illegal fishermen can still avoid monitoring. There is no real-time monitoring of phishing behavior 24 hours a day.

SUMMARY OF THE INVENTION

To solve the above technical problems, embodiments of the present invention provide a method, device, computer equipment and storage medium for phishing detection, which can perform 24-hour detection and management of phishing behavior in day and night scenarios with high detection accuracy.

A phishing detection method comprising:

obtaining a target detection frame, wherein the target detection frame includes pedestrians and fishing rods;

Perform phishing behavior recognition on the target detection frame through the phishing behavior detection model, and output the first result of the recognition result as "phishing behavior";

Perform fishing behavior recognition on the target detection frame through the human fishing rod key point detection model, and output the recognition result as the second result of "fishing behavior";

The first result and the second result are voted to output the final recognition result.

Preferably, in the above fishing detection method, the acquiring the target detection frame includes:

Extract the pedestrian frame and fishing rod frame in the image to be detected;

matching the fishing rod frame with the pedestrian frame in the same image to be detected;

If the intersection ratio obtained by matching is greater than the preset intersection ratio threshold, it is determined that the pedestrian in the pedestrian frame at this time is associated with the fishing rod in the fishing rod frame;

A target detection frame is generated based on the pedestrian and fishing rod.

Preferably, in the above fishing detection method, a pre-trained target detection model is used to extract the pedestrian frame and the fishing rod frame in the image to be detected. Before extracting the pedestrian frame and the fishing rod frame in the image to be detected, the Methods also include:

Obtain preprocessed sample data;

Inputting the preprocessed sample data into a preset initial target detection model to obtain an output result;

According to the preset focus loss function and the output result, adjust the sample type weight in the initial target detection model to obtain a parameter-adjusted target detection model;

The parameter-adjusted target detection model is trained by a batch stochastic gradient descent algorithm to obtain a pre-trained target detection model.

Preferably, in the above fishing detection method, the obtaining of the preprocessed sample data includes:

obtaining the first original image and the second original image in the sample data;

The first original image and the second original image are mixed and enhanced according to the mixed weight to obtain enhanced sample data.

Preferably, in the above-mentioned phishing detection method, before the phishing behavior recognition is performed on the target detection frame through the phishing behavior detection model, and before outputting the first result of the recognition result as "fishing behavior", the method further includes:

extracting a human-rod frame in the preprocessed sample data, wherein the human-rod frame includes a sample pedestrian and a sample fishing rod;

Train a preset classification model according to the human pole frame and batch stochastic gradient descent algorithm;

If the training result obtained through training reaches the preset training threshold, the classification model trained at this time is used as the fishing behavior detection model.

Preferably, in the above fishing detection method, before the target detection frame is identified as fishing behavior by using a human fishing rod key point detection model, and before outputting the second result of the identification result as "fishing behavior", the method further include:

Carrying out key point feature marking on the human-rod frame to obtain human key points corresponding to the sample pedestrian and fishing rod key points corresponding to the sample fishing rod;

The preset segmentation model is trained according to the human body key points, the fishing rod key points and the batch stochastic gradient descent algorithm, and the trained segmentation model is used as the human fishing rod key point detection model.

Preferably, in the above phishing detection method, the voting on the first result and the second result to output the final identification result includes:

respectively obtaining the mean value of the first result and the confidence level of the second result;

Voting is calculated according to the mean and the confidence to output the final recognition result.

A fishing detection device, comprising:

The RGB-IR image acquisition module is used to acquire the image to be detected in the monitoring water scene;

The target detection module extracts the pedestrian frame and the fishing rod frame in the image to be detected, and generates the target detection frame including the pedestrian and the fishing rod according to the threshold of the intersection ratio;

The phishing behavior detection module performs phishing behavior recognition on the target detection frame, and outputs the first result of the recognition result as "phishing behavior";

A human fishing rod key point detection module, which performs fishing behavior identification on the target detection frame, and outputs the second result of the identification result as "fishing behavior";

The voting module performs voting calculation according to the mean value of the first result and the confidence level of the second result, and outputs the final recognition result.

A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the phishing detection method when the processor executes the computer program.

A computer-readable storage medium stores a computer program, and the computer program implements the phishing detection method when executed by a processor.

The beneficial effects of the present invention are as follows: the fishing detection method proposed by the present invention can detect and manage the poaching behavior 24 hours a day in the night scene, and output the final identification result by voting on the mean value of the first result and the second result. the detection accuracy. Through automatic detection 24 hours a day, labor costs are greatly saved, so that illegal fishing in large waters can be identified 24 hours a day at low cost and with high accuracy, improving the management level of waters where fishing is prohibited.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the drawings that are used in the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

Fig. 1 is the flow chart of the fishing detection method described in the present invention;

Fig. 2 is the flow chart of obtaining described target detection frame;

Fig. 3 is the pre-training flow chart of the target detection model described in the present invention;

Fig. 4 is the training flow chart of the fishing behavior detection model described in the present invention;

Fig. 5 is the training flow chart of the key point detection model of the human fishing rod described in the present invention;

6 is a schematic structural diagram of the fishing detection device described in the present invention;

FIG. 7 is a schematic diagram of the internal structure of an embodiment of the computer device according to the present invention;

FIG. 8 is a schematic diagram of the internal structure of another embodiment of the computer device according to the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1. As shown in the figure, a phishing detection method proposed by an embodiment of the present invention includes:

Step S100, acquiring a target detection frame, wherein the target detection frame includes pedestrians and fishing rods.

Specifically, the to-be-detected image can be collected in real time by an image sensor arranged in the monitoring water scene, and it can be a time-series frame image or a time-series frame image in a streaming media video. The time-series frame images extracted from the streaming media video may be continuous frame images, or may be interval frame images obtained by adopting the interval frame-taking rule in a time-series frame image set. For example, if the streaming media video includes M frames of images to be detected, at least one frame of images to be detected is acquired from the M frames of images to be detected every N frames. It should be noted that the frame rate of the image to be detected in the streaming video is generally more than 25 frames per second. If each frame of the image to be detected is detected, it will increase the amount of computation and reduce the response timeliness of phishing behavior identification and detection. In this embodiment, multiple frames of images to be detected are obtained at intervals from the streaming media video, which can reduce the computational complexity of image processing, improve the speed of identification and detection of phishing behaviors, achieve rapid real-time feedback of detection results, and effectively combat illegal phishing. of poachers.

The images to be detected obtained on site in real time can be transmitted to the monitoring center server by wired or wireless means, and then the server will detect and identify the pedestrians and fishing rods appearing in the images to be detected, and mark the pedestrians on the images to be detected with pedestrian frames. Frame, mark and frame the fishing rod on the to-be-detected image with a fishing rod frame. When marking frames for pedestrians and fishing rods, the image to be detected is scanned and detected by a sliding window in the form of an automatic sliding window.

Specifically, by traversing the pedestrian frame on each frame of the image to be detected, and determining the pedestrian frame associated with the fishing rod frame on the image to be detected according to a preset intersection ratio threshold, a target containing both pedestrians and fishing rods is generated. Check box.

Specifically, the pedestrian frame and the fishing rod frame on the image to be detected detected in step S100 may be unrelated to each other, have no matching relationship, or may be associated and matched. The association matching referred to here means that the pedestrian frame and the fishing rod frame are a feature set of a poaching fisher, that is, the pedestrian frame is the human body feature reflected by the poaching fisher's body, and the fishing rod frame is the poaching fisher's human body. The characteristics of the fishing rod reflected by the person's fishing rod, that is, there is a poaching fisher at this time. By associating and matching the pedestrian frame and the fishing rod frame, a target detection frame containing both the pedestrian and the fishing rod can be generated, which can avoid identifying the passing unrelated passers-by as poachers, and improve the accuracy of identification and detection.

In step S200, the target detection frame is identified as a phishing behavior by using a phishing behavior detection model, and the first result of the identification result of "fishing behavior" is output.

Specifically, the target detection frame is a marker frame that includes both a pedestrian and a fishing rod, that is, two associated matching features of a poacher with an associated matching relationship detected and identified in step S200, that is, the pedestrian feature and the fishing rod. One or more poachers in the to-be-detected image can be detected by identifying the target detection frame containing pedestrians and fishing rods, and a target detection frame represents a poacher. In an image to be detected, there may be no target detection frame, or there may be one or more target detection frames. In the case where there is no target detection frame, the to-be-detected image does not enter the fishing behavior identification process of step S200. In the case where there is one target detection frame or multiple target detection frames, the image to be detected enters step S200 to identify the fishing behavior, and the output result is the first result of "fishing behavior".

Step S300 , the target detection frame is identified by the fishing behavior of the key point detection model of the human fishing rod, and the identification result is output as the second result of "fishing behavior".

Specifically, when detecting the key points of the human fishing rod, it is to load and output the target detection frame of the multi-frame image of the frame image of the first result in a time sequence, that is, to output the frame image of the first result in a time sequence. The target detection frames in the multi-frame images below are loaded at the same time, and then the key point features of the human fishing rod in the target detection frames in the multi-frame images are recognized for fishing behavior. It should be noted that the fishing behavior identification here is different from the fishing behavior identification in step S200. Here, the key point features of the human body and the key point features of the fishing rod of the poacher are identified, not the target. Detection frame recognition. By obtaining multi-frame human fishing rod key point feature data, and then using STGCN (space-time graph convolutional network model) to extract key point feature information for fishing behavior identification, the identification and detection results of fishing behavior can be obtained more accurately.

Step S400, voting on the first result and the second result to output the final identification result.

Specifically, the first result of the multi-frame images under one time sequence is averaged, and then the second result of the identification and output of the multi-frame images under the one time sequence based on the key point feature information of the human fishing rod is voted, and the final result is output. As a result, the detection and recognition accuracy of phishing behaviors is further improved.

In some embodiments of the present invention, the image to be detected is acquired by an RGB-IR image sensor set in the monitoring water scene. The RGB-IR image sensor is an image sensor that can simultaneously perform visible light and infrared light signal sensing, and the infrared light The signal is used to provide scene brightness information in a low-light environment, and the object of interest can be presented at night with poor lighting through the infrared feature map. Here, the subjects of interest include human targets and fishing rod targets. When fishing at night, lights are usually projected, and the fishing rod can reflect infrared characteristics under the illumination of lights, while poachers or pedestrians can reflect their infrared characteristics due to the spontaneous infrared signals of the human body. In the case of normal illumination, the visible light components of R, G, and B can achieve better imaging effects in the image sensor, which are mainly used in the case of good illumination during the day.

Further, in some embodiments of the present invention, the image to be detected is a multi-frame real-time static image captured in a pulsed manner by an RGB-IR image sensor set in the monitoring water scene. Specifically, the RGB-IR image sensor captures the current scene image in the form of pulses. Specifically, the pulse signal can be set according to the actual situation. At the peak of the pulse signal, the RGB-IR image sensor collects the image of the current scene as the image to be detected. At the trough of the pulse signal, the RGB-IR image sensor does not collect the image of the current scene, that is, it sleeps intermittently. working status. That is, another form of interval frame, the difference is that it does not belong to the interval frame in streaming media video. In time series, it is a continuous still frame image. Through the above-mentioned pulsed capturing of the image to be detected, the processing amount of the image data can be reduced, the calculation amount can be increased while the energy consumption is saved, and unnecessary calculation power can be avoided.

Further, in some embodiments of the present invention, the to-be-detected image can also be obtained under a unit time sequence based on the interval frame rule from the streaming media video recorded by the RGB-IR image sensor set in the monitoring water scene. of multi-frame real-time still images. Specifically, as another example, the interval frame selection rule may be 1+(n-1), where n is the frame selection period, that is, within the frame selection period n, only one frame of the image to be detected is acquired. In the 1+(n-1) mode, the detection time is 1/n of the frame period detection. For example, if n is 5, the number of image frames to be detected accounts for 20% of the number of video frames to be detected, which can be further improved. Detect the video processing speed and the number of video access channels to be detected. Obtaining multiple frames of images to be detected at intervals from the streaming video can improve the processing speed of the images to be detected, improve the response timeliness of phishing behavior identification and detection, and reduce the delay of phishing behavior identification and detection.

Further, in some embodiments of the present invention, as shown in FIG. 2 , the step of acquiring the target detection frame includes:

Step S110, extracting the pedestrian frame and the fishing rod frame in the image to be detected;

Step S120, respectively matching the fishing rod frame with the pedestrian frame in the same to-be-detected image;

Step S130, if the intersection ratio obtained by matching is greater than the preset intersection ratio threshold, determine that the pedestrian in the pedestrian frame at this time is associated with the fishing rod in the fishing rod frame;

Step S140, generating a target detection frame according to the pedestrian and the fishing rod.

Specifically, in the embodiment of the present invention, the calculation rule of the intersection ratio is as follows:

The coordinates of the upper left corner and the lower right corner of the pedestrian box a are marked as bbox _a = [(x _a1 , y _a1 ), (x _a2 , y _a2 )], where the coordinates of the upper left corner of the pedestrian box a are marked as (x _a1 , y _a1 ), the coordinates of the lower right corner of the pedestrian frame a are marked as (x _a2 ,y _a2 );

The coordinates of the upper left corner and the lower right corner of the rod frame b are marked as bbox _b = [(x _1b , y _1b ), (x ₂ , y _2b )], where the coordinates of the upper left corner of the rod frame b are marked as (x _1b , y _1b ), the coordinates of the lower right corner of the rod frame b are marked as (x ₂ , y _2b );

The intersection ratio IoU is the intersection ratio between pedestrian frame a and fishing rod frame b, which can be expressed as:

Further, the preset intersection ratio threshold in the embodiment of the present invention is set to be 0.1˜0.3. Wherein, as a preferred embodiment of the present invention, the preset intersection ratio threshold is set to 0.2. In the actual test process, it is found that when the intersection ratio of a fishing rod frame and a pedestrian frame is greater than 0.2, the areas contained in these two frames can be considered as a person and the fishing rod related to him; when the intersection ratio threshold When it is less than 0.2, it is considered that the fishing rods and pedestrians in these two boxes are irrelevant, so it is more appropriate to set the intersection ratio threshold to 0.2. Under the premise of not increasing the amount of computation, a high detection and recognition accuracy rate for phishing behaviors is guaranteed.

In a preferred embodiment of the present invention, the pedestrian frame and the fishing rod frame on the image to be detected are extracted through a pre-trained target detection model. Specifically, as shown in Figure 3, the training process of the target detection model includes:

Step S1101, obtaining preprocessed sample data;

Step S1102, inputting the preprocessed sample data into a preset initial target detection model to obtain an output result;

Step S1103, according to the preset focus loss function and the output result, adjust the sample type weight in the initial target detection model to obtain a parameter-adjusted target detection model;

Step S1104: Train the parameter-adjusted target detection model through a batch stochastic gradient descent algorithm to obtain a pre-trained target detection model.

Specifically, in a preferred embodiment of the present invention, the initial target detection model is a YOLOv3 model, and the sample types include positive samples and negative samples.

Wherein, the obtaining of the preprocessed sample data is specifically:

Step S11011, acquiring the first original image and the second original image in the sample data;

Step S11012: Perform mixed enhancement processing on the first original image and the second original image according to the mixed weight to obtain enhanced sample data.

Specifically, the data enhancement processing method can be expressed as:

image _mix = lambda*image _a +(1-lambda)*image _b ;

bbox _mix ∈ bboxes _mix ;

bbox _a ∈ bboxes _a ;

bbox _b ∈ bboxes _b ;

bboxes _mix = bboxes _a ∪bboxes _b ;

Among them, image _mix represents the mixed enhanced image, image _a represents the first original image, image _b represents the second original image, bboxes _mix represents the feature annotation frame of the mixed enhanced image, bboxes _a represents the feature annotation frame of the first original image , bboxes _b represents the feature label box of the second original image, and lambda is the mixing weight. In a preferred embodiment of the present invention, the value of the lambda mixing weight is 0.5, so that the first original image and the second original image have the same weight in the mixed enhanced image. This data enhancement processing method can greatly improve the The rich training samples of the pedestrian/fishing rod detection model improve the detection efficiency and accuracy of pedestrian and fishing rod features, so as to quickly generate the corresponding pedestrian and fishing rod frames. Specifically, the pedestrian frame includes the pedestrian feature, the fishing rod frame includes the fishing rod feature, and the target detection frame includes both the pedestrian feature and the fishing rod feature, that is, a feature marker of a fishing behavior identified in the present invention.

In the embodiment of the present invention, the sample data is derived from several visible light images and infrared images corresponding to the existing poaching behavior in the monitoring water scene, specifically the visible light images or infrared images marked with pedestrian frames and fishing rod frames, so as to This is used as sample data for model training.

Among them, in the embodiment of the present invention, in order to improve the accuracy of model training, 1000 frames of visible light images and infrared images corresponding to the phishing behavior can be used for repeated training and iterative training for many times. In order to improve the reading speed of the image and the accuracy of recognition, it is also necessary to align the visible light image and the infrared image before marking the pedestrian or fishing rod on the infrared image. Specifically, the image alignment includes feature-based image alignment and data-based alignment. The feature-based image alignment is to map the floating image to the reference image by finding a spatial transformation, so that the points corresponding to the same spatial position in the two images correspond one-to-one, so as to achieve the purpose of information fusion and facilitate the localization of features, Extraction and detection recognition. In the embodiment of the present invention, preferably, the visible light image is used as the reference image, and the infrared image is used as the floating image, and the infrared image is mapped to the visible light image through spatial transformation, so that at a time node, pedestrians on the infrared image The features and fishing rod features are aligned with the pedestrian features and fishing rod features, respectively, on the visible light image. Specifically, the feature-based image alignment may be processed by using a Homography (homography) algorithm, a Mesh Warps (variant) algorithm or an Optical flow (optical flow) algorithm in the prior art. Among them, the data-based alignment is to spatially arrange the infrared image data and the visible light image data according to certain rules, rather than sequentially arranging them one by one. Since each hardware platform handles storage space very differently, some platforms can only access certain specific types of data from certain specific addresses. If the data storage is not aligned according to the requirements of its platform, it will bring loss in access efficiency. For example, on some platforms, each read starts from an even address. If an int type (assuming a 32-bit system) is stored at the beginning of an even address, then one read cycle can be read, and if it is stored at an odd address In some places, it may take 2 read cycles, and the high and low bytes of the results of the two reads can be pieced together to get the int data. Obviously there is a lot of drop in read efficiency. Therefore, the alignment of the infrared image data and the visible light image data based on the data realizes the game in space and time, improves the data reading efficiency, and improves the calculation amount and response timeliness. Specifically, data alignment is a conventional technique commonly used in data access processing, and four-byte alignment is usually adopted, which will not be repeated here.

Specifically, in the embodiment of the present invention, in the step S11012, performing hybrid enhancement processing on the sample data is to perform weighting processing on the difficult samples in the positive and negative samples of the focal loss function, and the sample quality can be improved through the weighting processing. , reduce the loss, and improve the accuracy of judging difficult samples during the model training process, so as to reduce the problem of imbalanced sample categories and sample classification difficulty. Specifically, in the embodiment of the present invention, data enhancement processing is performed on the input data based on the YOLOv3 model.

Further, in some embodiments of the present invention, as shown in FIG. 4 , the training process of the fishing behavior detection model includes:

Step S210, extracting the human-rod frame in the preprocessed sample data, wherein the human-rod frame includes a sample pedestrian and a sample fishing rod;

Step S220, training a preset classification model according to the human pole frame and batch stochastic gradient descent algorithm;

Step S230: If the training result obtained through training reaches the preset training threshold, the classification model trained at this time is used as the fishing behavior detection model.

Specifically, in a preferred embodiment of the present invention, the classification model is a mobilenetv1 (a lightweight neural network) model.

Further, in some embodiments of the present invention, as shown in FIG. 5 , the training process of the human fishing rod key point detection model includes:

Step S310, marking the human rod frame with key point features to obtain human key points corresponding to the sample pedestrian and fishing rod key points corresponding to the sample fishing rod;

Step S320: Train a preset segmentation model according to the human body key points, the fishing rod key points and the batch stochastic gradient descent algorithm, and use the trained segmentation model as the human fishing rod key point detection model.

Specifically, in a preferred embodiment of the present invention, the segmentation model is an improved Unet model.

Specifically, in a preferred embodiment of the present invention, the human body fishing rod key point feature information includes 17 human body key point features and one fishing rod key point feature corresponding to human joints. Among them, 17 human key point features include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee , left ankle, right ankle. 1 key point feature of the rod includes: the end of the rod that is attached to the line.

In step 320, because the fishing rod is relatively thin, it is difficult to identify in the image. The improved Unet (segmentation network) model of the present invention is based on the existing Unet model, and the ordinary convolution layer is replaced by a hollow volume. product, improve the receptive field during convolution calculation, and characterize the existence of the fishing rod by strengthening the acquisition of the context (background) information around the fishing rod, thereby improving the accuracy of the key point detection of the fishing rod. Specifically, the Unet model is a well-known network used in the field of medical image segmentation. Since its fusion idea of low-level features and high-level features can well preserve the detailed feature information of the image, the Unet model is used as the key in the embodiment of the present invention. At the same time, in order to further improve the detection rate of the key points of small details, that is, the key points of the fishing rod, the present invention replaces the convolutional layer with a hollow convolutional layer to improve the time of convolution calculation. receptive field. In step S320, multiple frames of human fishing rod key point data can be obtained, and then STGCN (spatiotemporal graph convolutional network model) is used to extract key point feature information for fishing behavior identification, so as to obtain the final result more accurately.

Further, in some embodiments of the present invention, the voting on the first result and the second result to output the final identification result specifically includes:

Step S410, obtaining the mean value of the first result and the confidence level of the second result respectively;

Step S420: Perform voting calculation according to the mean value and the confidence level to output the final recognition result.

Specifically, the mean value of the first result refers to the mean value of the first results of multiple frames of images in a time series, and the confidence level of the second result refers to using the multi-frame images in a time series to extract the key points of the human fishing rod feature information to obtain a final phishing behavior recognition confidence result representing the second result;

Among them, the voting rules can be expressed as:

Among them, score _STGCN is a final phishing behavior recognition confidence result obtained by STGCN (Spatial Temporal Graph Convolutional Networks, spatiotemporal graph convolutional network model) using multiple frames of time series data; score _imagemodeli is the result of the ith frame obtained in step S300 ; lambda is the weight parameter.

In a preferred embodiment of the present invention, the voting weight of the mean value of the first result is 0.4<n<0.6, and the voting weight of the fishing behavior recognition confidence result is 0.4<m<0.6, where m+n=1.

Specifically, as a preferred embodiment of the present invention, the weight parameter lambda is set to 0.5 here, which represents the average detection and recognition results of the two models, the fishing behavior detection model and the human fishing rod key point detection model. Among them, n is the number of frames of the multi-frame used, which is set to 5 here. 5 frames can not only achieve a better recognition rate, but also have a good performance in speed.

Further, in the embodiment of the present invention, the to-be-detected image is acquired by an RGB-IR image sensor set in the monitoring water scene, including:

Receive the human body induction signal, wake up the dormant RGB-IR image sensor after the induction is triggered, and collect the real-time picture in the current field of view;

The collected real-time images are sent back to the monitoring center server.

By triggering the human body induction signal to wake up the dormant RGB-IR image sensor to collect the real-time picture in the current field of view, it can greatly save the data storage space and reduce the power consumption of the terminal RGB-IR image sensor.

On the other hand, another embodiment of the present invention also provides a fishing detection device, which is in one-to-one correspondence with the fishing detection method in the above embodiment. Specifically, as shown in FIG. 6 , the fishing detection device includes: an RGB-IR image acquisition module 10 , a target detection module 20 , a fishing behavior detection module 30 , a human fishing rod key point detection module 40 and a voting module 50 .

Wherein, the RGB-IR image acquisition module 10 is used to acquire the image to be detected in the monitoring water scene. The target detection module 20 is used to extract the pedestrian frame and the fishing rod frame in the image to be detected, and then generate a target detection frame including both the pedestrian and the fishing rod according to the intersection ratio threshold. The phishing behavior detection module 30 is configured to perform phishing behavior recognition on the target detection frame, and output the first result of the recognition result as "fishing behavior". The human fishing rod key point detection module 40 is used to identify the fishing behavior on the target detection frame, and output the second result of the identification result as "fishing behavior". The voting module 50 is configured to perform voting calculation according to the mean value of the first result and the confidence level of the second result, and output the final recognition result.

Specifically, the work completed by the target detection module 20 is as follows: first, perform detection and identification of pedestrians and fishing rods on the loaded images to be detected, and generate corresponding pedestrian frames and fishing rod frames; then, traverse each frame of the images to be detected The pedestrian frame on the frame is matched with the fishing rod frame on the image to be detected in the frame, and the pedestrian frame associated with the fishing rod frame is determined according to the preset intersection ratio threshold; Object detection boxes for pedestrians and fishing rods.

For the specific working principle of the fishing detection device based on RGB-IR image data, reference may be made to the workflow of the above fishing detection method, which will not be repeated here. Each module in the above-mentioned fishing detection device can be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

On the other hand, in an embodiment, as shown in FIG. 7 , the present invention further provides a computer device, the computer device may be a server, and a schematic diagram of its internal structure is shown in FIG. 7 . The computer equipment includes a data processor, memory, a network interface and a database connected by a device bus. Wherein, the computer equipment is provided with a plurality of data processors, and the data processors are used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating device, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The device's database is used to store data involved in image processing. The network interface of the device is used to communicate with external terminals through a network connection.

The memory stores a computer program that can run on the processor, and the processor implements the above-mentioned phishing detection method when the computer program is executed.

In an embodiment, the present invention also provides a computer device, the computer device may be a terminal, and a schematic diagram of its internal structure is shown in FIG. 8 . The computer equipment includes a data processor, memory, a network interface, a display screen, and an input device connected by a system bus. Wherein, the computer equipment is provided with a plurality of data processors, and the data processors are used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection.

Wherein, the memory stores a computer program that can be executed on the processor, and when the computer program is executed by the processor, realizes the above-mentioned phishing detection method.

Specifically, the display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or buttons, trackballs or buttons provided on the shell of the computer equipment. The touchpad can also be an external keyboard, touchpad or mouse.

Those skilled in the art can understand that the structures shown in FIG. 7 and FIG. 8 are only block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer equipment to which the solution of the present application is applied. A device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In another aspect, the present invention also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned phishing detection method is implemented.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

Exemplarily, the computer program may be divided into one or more modules/units, and the one or more modules/units are stored in a memory and executed by a processor to accomplish the present invention. One or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in a computer device.

The computer device may be a desktop computer, a notebook, a palmtop computer, a cloud server, and other computing devices. Computer equipment may include, but is not limited to, processors, memory. Those skilled in the art can understand that FIG. 7 and FIG. 8 are only examples of computer equipment, and do not constitute a limitation to the computer equipment, and may include more or less components than those shown, or combine certain components, or different Components, such as computer devices, may also include input-output devices, network access devices, buses, and the like.

The processor may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf processors Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. The memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. . Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Excluded are electrical carrier signals and telecommunication signals.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it is still possible to implement the foregoing implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the within the protection scope of the present invention.

Claims (10)

1. A fishing detection method, comprising:

obtaining a target detection frame, wherein the target detection frame comprises a pedestrian and a fishing rod;

performing phishing behavior recognition on the target detection frame through a phishing behavior detection model, and outputting a first result of which the recognition result is 'phishing behavior';

carrying out fishing behavior recognition on the target detection frame through a human fishing rod key point detection model, and outputting a second result of which the recognition result is 'fishing behavior';

and voting the first result and the second result to output a final recognition result.

2. A fishing detection method according to claim 1, wherein said acquiring a target detection frame includes:

extracting a pedestrian frame and a fishing rod frame in an image to be detected;

respectively matching the fishing rod frames with pedestrian frames in the same image to be detected;

if the intersection ratio obtained by matching is larger than a preset intersection ratio threshold value, determining that the pedestrian in the pedestrian frame is associated with the fishing rod in the fishing rod frame;

and generating a target detection frame according to the pedestrian and the fishing rod.

3. A fishing detection method according to claim 2, wherein a pedestrian frame and a fishing rod frame in the image to be detected are extracted using a pre-trained target detection model, and before the extraction of the pedestrian frame and the fishing rod frame in the image to be detected, the method further comprises:

acquiring preprocessed sample data;

inputting the preprocessed sample data into a preset initial target detection model to obtain an output result;

adjusting sample type weight in the initial target detection model according to a preset focus loss function and the output result to obtain a parameter-adjusted target detection model;

and training the parameter-adjusted target detection model through a batch stochastic gradient descent algorithm to obtain a pre-trained target detection model.

4. A phishing detection method according to claim 3 where said obtaining pre-processed sample data comprises:

acquiring a first original image and a second original image in sample data;

and performing mixed enhancement processing on the first original image and the second original image according to the mixed weight to obtain enhanced sample data.

5. A phishing detection method according to claim 3 or 4, wherein before performing phishing behavior recognition on the target detection frame by the phishing behavior detection model and outputting a first result that a recognition result is "phishing behavior", the method further comprises:

extracting a rod frame in the preprocessed sample data, wherein the rod frame comprises a sample pedestrian and a sample fishing rod;

training a preset classification model according to the rod frame and the batch random gradient descent algorithm;

and if the training result obtained through training reaches a preset training threshold value, taking the trained classification model as a fishing behavior detection model.

6. A fishing detection method according to claim 5, wherein before the fishing behavior recognition is performed on the target detection frame by the human fishing rod key point detection model and the second result that the recognition result is "fishing behavior" is output, the method further comprises:

carrying out key point feature marking on the fishing rod frame to obtain a human body key point corresponding to a sample pedestrian and a fishing rod key point corresponding to a sample fishing rod;

and training a preset segmentation model according to the human body key points, the fishing rod key points and the batch stochastic gradient descent algorithm, and taking the trained segmentation model as the human body fishing rod key point detection model.

7. A phishing detection method as claimed in claim 1 wherein said voting for said first and second results to output a final recognition result comprises:

respectively acquiring the mean value of the first result and the confidence coefficient of the second result;

and voting calculation is carried out according to the mean value and the confidence coefficient so as to output a final recognition result.

8. A fishing detection device, comprising:

the RGB-IR image acquisition module is used for acquiring an image to be detected in a scene of a monitoring water area;

the target detection module is used for extracting a pedestrian frame and a fishing rod frame in the image to be detected and generating a target detection frame simultaneously comprising the pedestrian and the fishing rod according to the intersection ratio threshold;

the fishing behavior detection module is used for carrying out fishing behavior identification on the target detection frame and outputting a first result of which the identification result is 'fishing behavior';

the human fishing rod key point detection module is used for carrying out fishing behavior recognition on the target detection frame and outputting a second result of which the recognition result is 'fishing behavior';

and the voting module is used for voting according to the mean value of the first result and the confidence coefficient of the second result and outputting a final recognition result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the phishing detection method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a phishing detection method according to any one of claims 1 to 7.

Images