CN114511499A

CN114511499A - Method and system for detecting abnormal behavior of object charging and electronic device

Info

Publication number: CN114511499A
Application number: CN202111628178.7A
Authority: CN
Inventors: 赵蕾; 孙海涛; 李宁钏; 杨剑波; 熊剑平
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2021-12-28
Filing date: 2021-12-28
Publication date: 2022-05-17

Abstract

The application relates to an abnormal behavior detection method, system and electronic device for object charging, wherein the abnormal behavior detection method comprises the following steps: acquiring an image to be detected acquired in the process of feeding the article; inputting the image to be detected into a pre-trained article detection model, and determining an article area of the image to be detected; and determining whether an abnormality exists in the article charging process according to the association information of the article area and a target detection area, wherein the target detection area is determined based on the area where the article is added in the article charging process. Through the method and the device, the problem of low efficiency of abnormal article charging detection is solved, and a high-efficiency and accurate detection method for whether the article charging is abnormal is realized.

Description

Method and system for detecting abnormal behavior of object charging and electronic device

Technical Field

The present application relates to the field of detecting abnormal material charging, and in particular, to a method, a system, and an electronic device for detecting abnormal material charging behavior.

Background

In an actual production scene, the feeding amount of articles such as the quantity of tobacco leaves on a similar tobacco leaf conveyor belt is a problem that a production line needs to pay attention to all the time, and particularly when the feeding amount of the articles is less than a certain threshold value, the normal work of subsequent equipment can be influenced, so that the production efficiency is influenced, and the feeding condition of the articles such as the tobacco leaves needs to be detected. In the related art, the control of the feeding amount is usually completed by manual monitoring, the mode has low efficiency, and if the supervision is implemented, huge labor is needed, and the labor cost is high; or the sensor is used for feeding monitoring, but the application range of the sensor is limited by equipment, so that the monitoring flexibility is low, and the efficiency of detecting abnormal feeding of the articles is low.

At present, no effective solution is provided aiming at the problem of low efficiency of abnormal article charging detection in the related technology.

Disclosure of Invention

The embodiment of the application provides a method and a system for detecting abnormal behavior of object charging and an electronic device, and aims to at least solve the problem of low efficiency of abnormal detection of tobacco charging in the related technology.

In a first aspect, an embodiment of the present application provides a method for detecting abnormal behavior of an article charging, where the method includes:

acquiring an image to be detected acquired in the process of feeding the article;

inputting the image to be detected into a pre-trained article detection model, and determining an article area of the image to be detected;

and determining whether abnormal behaviors exist in the article charging process based on the associated information of the article area and a target detection area, wherein the target detection area is determined based on the area where the articles are added in the article charging process.

In some embodiments, before acquiring the image to be detected in the article charging process, the method further includes:

acquiring a preset article image and article marking information;

labeling the preset article image according to the article labeling information to obtain first training data, and preprocessing the first training data by using a mean subtraction method to obtain second training data;

and inputting the second training data into a preset high-resolution network for training to obtain the article detection model.

In some embodiments, the inputting the second training data into a preset high-resolution network for training to obtain the article detection model includes:

inputting the training data into at least two stage sub-networks of the high-resolution network, so that each stage sub-network respectively outputs an article characteristic and a resolution down-sampling result, and fusing the article characteristic and the resolution down-sampling result to obtain corresponding characteristic extraction information;

performing upsampling processing on each feature extraction information to obtain an upsampling result, and fusing all the upsampling results to obtain a fusion result;

and setting a learning rate and an attenuation mode, and training the high-resolution network according to the learning rate, the attenuation mode and the fusion result so as to obtain the article detection model after the second training data iteration is finished.

In some embodiments, the determining whether there is an abnormal behavior in the process of charging the article based on the correlation information between the article region and the target detection region includes:

determining a degree of overlap of the target detection zone and the item zone;

determining whether abnormal behaviors exist in the process of charging the articles according to the overlapping degree

In some embodiments, the determining the degree of overlap between the target detection area and the item area includes:

and calculating to obtain an article area intersection result between the target detection area and the article area, and calculating to obtain the article charging result according to the article area intersection result and the target detection area.

In some embodiments, said determining whether there is abnormal behavior during said filling of said article based on said degree of overlap comprises:

acquiring a preset overlapping degree range, and determining an actual overlapping degree range which is matched with the overlapping degree in the preset overlapping degree range;

and determining that a target abnormal behavior exists in the material charging process according to the actual overlapping degree range based on the mapping relation between the preset overlapping degree range and the sub-abnormal behaviors, wherein the target abnormal behavior is the abnormal behavior mapped in the actual overlapping degree range determined in the sub-abnormal behaviors.

In some embodiments, after determining that the target abnormal behavior exists in the article charging process, the method further includes:

and sending alarm grade information corresponding to the target abnormal behavior to an alarm application program or terminal equipment.

In some embodiments, before determining whether there is an abnormal behavior in the process of charging the article based on the correlation information between the article region and the target detection region, the method further includes:

responding to a region selection instruction triggered by the image to be detected, and determining an image region indicated by the region selection instruction as the target detection region; or

Determining an image area sent by a terminal device as the target detection area, wherein the image area is determined by the terminal device based on an area selection instruction triggered by the image to be detected, and the area selection instruction is used for indicating the image area.

In a second aspect, an embodiment of the present application provides an anomaly detection behavior system for charging an article, the system including: an image acquisition device and a control device;

the image acquisition equipment is used for acquiring an image to be detected in the article charging process and sending the image to be detected to the control device;

the control device is configured to execute the method for detecting abnormal behavior of the charging of the article according to the first aspect.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the processor implements the method for detecting abnormal behavior of charging an article according to the first aspect.

In a fourth aspect, the present application provides a storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for detecting abnormal behavior of charging an article according to the first aspect.

Compared with the related art, the method, the system and the electronic device for detecting the abnormal behavior of the object charging provided by the embodiment of the application acquire the image to be detected acquired in the object charging process; inputting the image to be detected into a pre-trained article detection model, and determining an article area of the image to be detected; and determining whether the object charging process is abnormal or not according to the associated information of the object area and the target detection area, wherein the target detection area is determined based on the area where the object is added in the object charging process, so that the problem of low efficiency of detecting the abnormal object charging is solved, and the efficient and accurate detection method for detecting whether the object charging is abnormal or not is realized.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is an application environment diagram of an abnormal behavior detection method for charging an article according to an embodiment of the present application;

FIG. 2 is a flow chart of a method of detecting abnormal behavior of an article charge according to an embodiment of the present application;

FIG. 3A is a schematic diagram of a high resolution network architecture according to an embodiment of the present application;

FIG. 3B is a schematic diagram of a decoding network architecture according to an embodiment of the present application;

FIG. 4 is a flow chart of a method for detecting abnormal behavior in the loading of an article according to a preferred embodiment of the present application;

FIG. 5A is a schematic diagram of an image to be detected according to an embodiment of the present application;

FIG. 5B is a schematic diagram of a target detection area according to an embodiment of the present application;

FIG. 5C is a diagram illustrating an image segmentation result according to an embodiment of the present application;

FIG. 5D is a graphical illustration of the result of loading an article according to an embodiment of the present application;

FIG. 6 is a block diagram of an apparatus for anomaly detection behavior of charging an article according to an embodiment of the present application;

FIG. 7 is a block diagram of an anomaly detection behavior system for article charging in accordance with an embodiment of the present application;

fig. 8 is a block diagram of the inside of a computer device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference herein to "a plurality" means greater than or equal to two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

The method for detecting the abnormal behavior of the object charging provided by the embodiment of the application can be applied to the application environment shown in fig. 1. Wherein the image acquisition apparatus 12 communicates with the control device 14 via a network. The control device 14 obtains an image to be detected in the article feeding process through the image acquisition device 12, inputs the image to be detected into a well-trained article detection model to output an image segmentation result for the article in the image to be detected, calculates an article feeding result in the image to be detected according to the image segmentation result and a preset target detection area, and finally determines whether an abnormality exists in the article feeding process according to the article feeding result. The image capturing device 12 may be, but is not limited to, various binocular cameras, a dome camera, or other devices for capturing images, and the control device 14 may be, but is not limited to, various server devices, terminal devices such as a personal computer or a notebook computer, a processing chip, or other devices for controlling; the server device may be implemented as a stand-alone server or as a server cluster consisting of a plurality of servers.

The embodiment provides a method for detecting abnormal behavior of an article charging, fig. 2 is a flowchart of a method for detecting abnormal behavior of an article charging according to an embodiment of the present application, and as shown in fig. 2, the flowchart includes the following steps:

step S202, acquiring an image to be detected acquired in the article charging process.

Wherein, the image to be detected is an image comprising an article feeding channel; the image to be detected can be obtained by extracting continuous frame images from the video stream acquired by the image acquisition equipment by the control devices such as the server equipment, the personal computer, the notebook computer, the processing chip and the like.

Step S204, inputting the image to be detected into a pre-trained article detection model, and determining an article area of the image to be detected.

The article detection model is a model obtained by deep neural network training and used for segmenting an article image. The article detection model can be a neural network model such as a Deep residual network (ResNet) model, a VGG-19 model, a DenseNet model or a High-Resolution network (HRNet) model; preferably, the HRNet model is adopted as the article detection model in the embodiment, so that the efficiency and accuracy of the model can be higher. The object detection model is used for training the image to be detected, the image segmentation result for the object can be automatically generated, namely partial images of the object in the image to be detected are extracted according to the outline of the object, so that the object area in the image to be detected is determined, and subsequent calculation is facilitated.

Step S206, determining whether abnormal behaviors exist in the material filling process of the article based on the related information of the article area and a target detection area, wherein the target detection area is determined based on the area where the article is added in the material filling process.

The target detection area refers to the area range of the object feeding channel which is defined in the image to be detected and needs to count the object ratio. Specifically, the article loading channel region is generally a quadrangular region, the extreme end of the article loading channel is taken as the upper side of the quadrangular region, and the left and right sides of the article loading channel are taken as the left and right sides of the quadrangular region. The target detection area can be generated by a user on the terminal device based on the pre-labeling of the image to be detected, or the control device can perform algorithm processing such as edge detection on the image to be detected, and then the target detection area is automatically extracted. After the control device acquires the target detection area, parameters are configured based on characteristic coordinates such as corner points of the target detection area, so that the division of the object charging channel detection area is realized. The related information between the article area and the target detection area refers to the area overlapping degree between the article area and the target detection area, and the area overlapping degree can be represented by data such as an area ratio or an area difference; for example, after the image segmentation result and the target detection region are acquired, the control device may calculate the area ratio between the image segmentation result and the target detection region to determine the article charging condition, or may determine the difference between the image segmentation result and the target detection region as the article charging condition, or the like.

Specifically, if the area proportion of the region where the article is located in the image segmentation result calculated by the control device based on the correlation information to the target detection region is large, it may be determined that there is no abnormality in the article charging process, and normal operation may be continued; otherwise, determining that the object charging process is abnormal indeed, and reminding workers, namely the user charges in time.

Through the steps S202 to S206, the image to be detected is input into the article detection model to determine the article area of the image to be detected, and whether the article charging process is abnormal or not is finally determined according to the article area and the acquired target detection area, so that the problem of low detection efficiency caused by only manually detecting or detecting the article charging condition by a sensor is avoided, the problem of low detection efficiency of article charging abnormity is solved, and the efficient and accurate detection method for detecting whether the article charging is abnormal or not is realized. In addition, the image segmentation precision and the calculation precision of the article charging result in the embodiment can be continuously improved according to the accumulation of materials, and meanwhile, the method has higher flexibility and can be generally suitable for similar scenes such as tobacco charging and the like.

In some embodiments, before the image to be detected in the process of charging the article is acquired, the abnormal behavior detection method further includes the following steps:

step S210, acquiring a preset article image and article marking information; labeling the preset article image according to the article labeling information to obtain first training data, and preprocessing the first training data by using a mean subtraction method to obtain second training data; and inputting the training data into a preset high-resolution network for training to obtain the article detection model.

Wherein, the preset article image is a prepared image including the article; the preset article image may be multiple. The article marking information can be generated by marking the article in each preset article image by a modeling worker so as to facilitate the subsequent high-resolution network to identify the article; for example, the modeler labels the region of the article in the channel using the polygonal features of the article loading channel, only the region where the article is present is labeled, and the background portion is not labeled. After the control device performs labeling and segmentation processing on the preset article image by using article labeling information generated by labeling of a modeling worker to obtain the first training data, the first training data can be divided into a training set and a test set, and the training set and the test set are used as input images for training the high-resolution network. The control device may also perform preprocessing on the first training data, for example, in order to improve the sensitivity of the high-resolution network to image features, three channels r, g, and b of the image in the first training data may be subjected to mean subtraction respectively by means of mean subtraction, so that feature information of each channel is obvious, and the discrimination between different features is improved. It can be understood that the control device may also perform preprocessing such as edge removal, clipping, and noise reduction on the first training data, and perform data enhancement on the second training data obtained after the preprocessing, so as to improve the processing efficiency of the article detection model. And finally, inputting the second training data into the high-resolution network by the control device for segmentation processing, and further training to generate the article detection model.

Through the step S210, the preset article image is labeled in advance through the article labeling information to obtain the first training data, and the first training data is preprocessed through the mean subtraction method to improve the feature discrimination, so that the accuracy and the efficiency of article charging abnormity detection are improved.

In some embodiments, the inputting the training data into a preset high-resolution network for training to obtain the article detection model further includes the following steps:

step S211, inputting the training data into at least two stage subnetworks of the high resolution network, so that each stage subnetwork respectively outputs the article feature and the resolution down-sampling result, and fusing the article feature and the resolution down-sampling result to obtain corresponding feature extraction information.

The above step S211 provides a feature extraction process of the high resolution network. Specifically, in consideration of the real-time requirement of the application scenario, the model calculation amount should be as small as possible, so in this embodiment, a relatively light HRNet-W32 is selected as the basic network, and the HRNet-W32 network structure is as shown in fig. 3A. Wherein the HRNet-W32 network comprises 4-phase subnetworks as shown in the first row of tiles in FIG. 3A; the sub-network in the first stage comprises 4 residual units, and the main function of the residual units is to enable the output result of the high-resolution network to keep a better effect under the condition that the number of layers is increased. The second, third and fourth stages respectively comprise 1, 4 and 3 switching blocks, and one switching block consists of 4 residual error units. Feature extraction is performed on the input image by the phase sub-network of the four phases. The process from each stage to the next stage is also performed with down-sampling computation in parallel, as shown in fig. 3A, by the second row of blocks, the third row of blocks, and the fourth row of blocks, i.e., the resolution of the feature map of each stage is reduced by half, and the features of the image object are better abstracted through down-sampling performed in each stage. Meanwhile, because the output size of each stage is different, the resolution of the information obtained by down-sampling in each stage is different, and in order to utilize a plurality of features with different resolutions, the resolution down-sampling result and the feature of the article are fused from each stage to the next stage to obtain the feature extraction information so as to obtain more sufficient feature information.

Step S212, performing upsampling processing on each piece of feature extraction information to obtain an upsampling result, and fusing all the upsampling results to obtain a fusion result.

The above step S212 provides a decoding process of the high resolution network. Specifically, as shown in fig. 3B, in step S211, the image is down-sampled at each stage to obtain an abstract feature, and at the same time, some spatial position information is lost, so that the decoding process is used to supplement the image spatial information to obtain the same size as the input image, so that each pixel corresponding to the input image can be classified, and the result of each pixel is summarized to obtain the image segmentation result. The decoding process mainly comprises the steps of respectively carrying out up-sampling on the result images obtained in the 4 stages, directly carrying out up-sampling with opposite multiples by using the down-sampling result in the encoder, restoring the images to the original size, then fusing the four results through the convolution layer, and classifying each pixel to obtain a fused result.

And S213, setting a learning rate and an attenuation mode, and training the high-resolution network according to the learning rate, the attenuation mode and the fusion result so as to obtain the article detection model after the second training data iteration is finished.

The attenuation modes comprise exponential attenuation, natural exponential attenuation, piecewise constant attenuation or other attenuation modes, and a final article detection model is obtained after a plurality of times of training second data iteration. The learning rate attenuation is realized by setting the initial value and the attenuation mode of the learning rate, and after the training reaches a certain degree, the precision is improved by using a small learning rate, so that the training model can be converged more quickly, and the algorithm for training the article detection model is further optimized.

Through the steps S211 to S213, the article detection model obtained through high-resolution network training is used for segmenting the article area image in the image to be detected, so that the problem that certain difficulty exists in boundary definition due to the fact that the target at the tail end of the article charging channel is small is solved, and the accuracy of article charging abnormity detection is further improved.

In some embodiments, the determining whether an abnormal behavior exists in the charging process of the article based on the correlation information between the article area and the target detection area further includes the following steps: determining the degree of overlap of the target detection area and the item area; and determining whether abnormal behaviors exist in the article charging process according to the overlapping degree.

In some embodiments, the determining the degree of overlap between the target detection area and the article area further includes: and calculating to obtain an article area intersection result between the target detection area and the image segmentation result, and calculating to obtain an article charging result according to the article area intersection result and the target detection area.

The control device may calculate the ratio of the area of the article to the area of the article by using the detection area of the article feeding channel, which is configured in advance, i.e., the target detection area, and the image segmentation result. In particular, R may be used_{Article with a cover}The area of the region actually occupied by the article in the image segmentation result is represented by R_TargetThe area occupied by the target detection area is represented, and the areas of the target detection area and the target detection area are both represented by the number of pixels, and the intersection result of the areas of the objects is R_{Article with a cover}∩R_TargetThe calculation formula of the article charging result E is shown as formula 1:

E＝(R_{article with a cover}∩R_Target)/R_TargetEquation 1

Through the embodiment, the article area intersection result between the target detection area and the image segmentation result is calculated, and the article charging result is calculated based on the article area intersection result, so that the condition that the article outside the article charging channel is wrongly counted into the image segmentation result to cause a large article charging result calculation error is avoided, and the accuracy of the abnormal detection of article charging is improved.

In some embodiments, the determining whether there is abnormal behavior during the charging process of the article according to the overlapping degree further includes the following steps:

step S241, acquiring a preset overlap degree range, and determining an actual overlap degree range matching the overlap degree in the preset overlap degree range.

Wherein, the preset overlapping degree range can be preset by a user; for example, the detected number of items may be divided into a plurality of sections according to the range of the item ratio, as shown in table 1 below:

TABLE 1 comparison table of quantity information and ratio range of articles

Item quantity information reporting	Area of goods ratio
		Sufficient articles	G₁
Slight lack of articles	G₂
		Serious lack of articles	G₃
The articles being almost empty	G₄

Taking Table 1 as an example, the above-mentioned article proportion range G indicating sufficient articles₁Can refer to the range of 100% -90% of the ratio of the article, and the range of the ratio of the article G for indicating the slight lack of the article₂Can refer to the range of 90% -70% of the ratio of the item, indicating the range of the ratio of the item G in which the item is seriously absent₃Can refer to a range of 70% to 20% of the item ratio, indicating an item ratio G where the item is almost empty₄May be in the range of 20% to 0% by weight of the article. The preset overlap degree range of the four interval ranges can be determined based on the table 1; after determining the overlap degree, the control device may select an actual overlap degree range matching the overlap degree from the preset overlap degree ranges; for example, if the data value of the detected overlap degree is in the range of 90% to 70%, it may be determined that the actual overlap degree range is G2 in the preset overlap degree range.

Step S242, determining that a target abnormal behavior exists in the article charging process according to the actual overlap degree range based on the mapping relationship between the preset overlap degree range and the sub-abnormal behavior, where the target abnormal behavior is an abnormal behavior mapped to the actual overlap degree range determined in the sub-abnormal behaviors.

Specifically, the control device may detect a section in which the overlap degree falls within the article proportion range information based on the mapping relationship, which may be, for example, table 1: the predetermined overlap range is G₂In the interval, the corresponding sub-abnormal condition indicating that the article is slightly lacked can be determined in the feeding process; the predetermined overlap range is G₃In the interval, the corresponding sub-abnormal condition indicating that the article is seriously lacked can be determined in the feeding process; the predetermined overlap range is G₄Interval, then at this point it can be determined that there is a sub-anomaly corresponding to it that indicates that the article is nearly empty during the loading process. Therefore, after the control device determines the actual overlap degree range in the preset overlap degree range through the step S241, it may determine a sub abnormal behavior corresponding to the actual overlap degree in the mapping relationship according to the actual overlap degree range, and determine this sub abnormal behavior as the target abnormal behavior.

Through the steps S201 to S282, whether the target abnormal behavior still exists in the article charging process is determined by determining the actual overlap degree range matched with the overlap degree in the preset overlap degree range, and the article quantity is divided into sections by using the article charging result, so that the fault tolerance rate of the article charging abnormal behavior detection method can be effectively improved, and the accuracy of the article charging abnormal behavior detection is further improved.

In some embodiments, after determining that the target abnormal behavior exists in the charging process of the article, the abnormal behavior detection method further includes the following steps: and sending alarm grade information corresponding to the target abnormal behavior to an alarm application program or terminal equipment.

The control device may adopt the terminal device, and at this time, the control device may send the alarm level information to an alarm application program associated therewith, the alarm application program being based onThe alarm level information runs an alarm program. Alternatively, the control device may be a server device, a processing chip, or the like, other than the terminal device, and at this time, the control device may send alarm level information corresponding to the target abnormal behavior to the terminal device; taking Table 1 as an example, if the result of charging the article is in G₄Within the interval, the control means may correspond to G at this time₄The first-level alarm level of the interval is sent to the terminal equipment, and the terminal equipment indicates an acousto-optic device connected with the terminal equipment to alarm in the modes of red light flashing or voice alarm prompting and the like based on the first-level alarm level; if the article charging result is in G₃Within the interval, the control means may correspond to G at this time₃Sending the second-level alarm grade of the interval to the terminal equipment, and executing alarm in a mode of 'please manually feed' text prompt and the like by the terminal equipment based on the second-level alarm grade; if the article is in the feeding position G₂Within the interval, the control means may correspond to G at this time₂And the secondary alarm grade of the interval is sent to the terminal equipment, and the terminal equipment carries out alarm in the modes of yellow light flashing or character prompt of 'slight lack of article quantity and attention' and the like based on the secondary alarm grade. That is, when the level of the corresponding alarm level generated by the control device is higher, the alarm operation executed by the control terminal device is milder, so that the abnormal result is fed back to the production line to remind the user of the abnormal alarm method for handling the abnormality in time, and the abnormal alarm management in multiple sections is realized.

It can be understood that the control device may also generate a corresponding article quantity information report result based on the charging detection interval result similar to table 1, and send the article quantity information report result to the alarm application program or the terminal device for displaying, so that the user can grasp and inquire the article charging situation at any time.

In some embodiments, the acquiring the target detection area further includes: and in response to the area selection instruction triggered by the image to be detected, determining the image area indicated by the area selection instruction as the target detection area. Specifically, in a case where the control device is a terminal device, the control device may detect a region selection instruction generated by a user drawing or intercepting a certain image region on the control device, and determine a target detection region corresponding to the region selection instruction in response to the region selection instruction. Or determining an image area sent by the terminal device as the target detection area, where the image area is determined by the terminal device based on an area selection instruction triggered for the image to be detected, and the area selection instruction is used to indicate the image area. Specifically, when the control device employs a server device or other device except a terminal device, the control device may send the image to be detected to the terminal device for display, so that the user may draw a certain image area on the image to be detected by interacting with the terminal device, and further trigger an area selection instruction corresponding to the certain image area, the terminal device detects an image area input by the user in response to the area selection instruction, and sends the image area to the control device, and the control device takes the image area as a target detection area, so as to perform subsequent abnormal behavior detection calculation processing.

Through the embodiment, the image area indicated by the area selection instruction is determined as the target detection area, or the image area acquired by the terminal device based on the area selection instruction is determined as the target detection area, so that the user can conveniently and visually operate, the man-machine interaction is improved, and the user experience is improved.

An embodiment of the present application is described in detail below with reference to an actual application scenario, taking tobacco leaf charging anomaly detection as an example, and fig. 4 is a flowchart of an anomaly behavior detection method for article charging according to a preferred embodiment of the present application, and as shown in fig. 4, the flowchart includes the following steps:

step S401, an image to be detected is input, as shown in fig. 5A.

Step S402, parameter configuration, namely, defining a target detection area corresponding to a tobacco channel, namely defining an area range in which the ratio of tobacco leaves needs to be counted; the quadrilateral frame marked by the white line in fig. 5B is the target detection area.

Step S403, labeling tobacco data, and making a training set; the data, namely the images of the preset articles, can be acquired by a camera fixed at the upper end of a tobacco leaf channel area, the tobacco leaf area in the channel is marked by utilizing a polygon, only the area where the tobacco leaves exist is marked, the background part is not marked, and a training set and a test set are divided at the same time to be used as input images of a training network.

S404, training an article detection model, and extracting a tobacco leaf area; after an article detection model is generated based on the training of the training set, an image to be detected is input into the article detection model for image segmentation processing, and a tobacco leaf area, namely the image segmentation result, is obtained; the image segmentation result is shown in fig. 5C.

Step S405, calculating the ratio of the articles; as shown in fig. 5D, if the edge of the region where the article is located in the image segmentation result obtained in step S404 is marked in the target detection region by a black line, the tobacco leaf ratio is calculated by using the image segmentation result and the target detection region configured in step S402, and the method is to perform intersection operation on the two regions to obtain the area ratio of the tobacco leaf region, that is, the article charging result.

Step S406, reporting the proportion degree information and giving an alarm aiming at the abnormal condition; and comparing the tobacco leaf area occupation ratios obtained in the step S405 according to preset article occupation ratio range information, returning alarm levels such as 'sufficient' and 'material shortage' according to the article occupation ratio range interval, and triggering an alarm function to remind a worker to feed materials in time when the occupation ratios are smaller than a certain threshold value.

It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.

The embodiment also provides an anomaly detection behavior device for article charging, which is used for implementing the above embodiments and preferred embodiments, and the description of the device is omitted. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 6 is a block diagram of a device for detecting an abnormal behavior in charging an article according to an embodiment of the present application, where, as shown in fig. 6, the device includes: an acquisition module 62, a segmentation module 64, and an exception module 66; the acquisition module 62 is configured to acquire an image to be detected acquired in an article charging process; the segmentation module 64 is configured to input the image to be detected into a pre-trained article detection model, and determine an article region of the image to be detected; the anomaly module 66 is configured to determine whether there is anomalous behavior in the loading process of the article based on the correlation information between the region of the article and a target detection region, wherein the target detection region is determined based on the region of the article added in the loading process of the article.

Through the embodiment, the image to be detected is input into the article detection model through the segmentation module 64 so as to determine the article area of the image to be detected, the proportion module 66 calculates the article charging result according to the image segmentation result and the preset target detection area, and the abnormity module 66 finally determines whether the article charging process is abnormal or not according to the article area and the obtained target detection area, so that the low detection efficiency caused by only manually detecting or detecting the article charging condition by a sensor is avoided, the problem of low efficiency of abnormal article charging detection is solved, and the efficient and accurate detection device for detecting whether the article charging is abnormal or not is realized.

In some embodiments, the above apparatus for detecting an abnormal behavior in the loading of an article further comprises a training module; the training module is used for acquiring a preset article image and article marking information; the training module performs labeling processing on the preset article image according to the article labeling information to obtain first training data, and performs preprocessing on the first training data by using a mean subtraction method to obtain second training data; the training module inputs the second training data into a preset high-resolution network for training so as to obtain the article detection model.

In some embodiments, the training module is further configured to input the training data into at least two stage subnetworks of the high resolution network, so that each stage subnetwork outputs an article feature and a resolution down-sampling result, and performs fusion processing on the article feature and the resolution down-sampling result to obtain corresponding feature extraction information; the training module performs upsampling processing on each feature extraction information to obtain an upsampling result, and fuses all the upsampling results to obtain a fusion result; the training module sets a learning rate and an attenuation mode, and trains the high-resolution network according to the learning rate, the attenuation mode and the fusion result, so that the article detection model is obtained after the second training data iteration is finished.

In some embodiments, the anomaly module 66 is further configured to determine a degree of overlap between the target detection area and the item area; the anomaly module 66 determines whether there is anomalous behavior in the loading of the item based on the degree of overlap.

In some embodiments, the anomaly module 66 is further configured to calculate an article area intersection result between the target detection area and the article area, and calculate the overlapping degree according to the article area intersection result and the target detection area.

In some embodiments, the exception module 66 is further configured to obtain a preset overlap degree range, and determine an actual overlap degree range, which matches the overlap degree, in the preset overlap degree range; the anomaly module 66 determines that a target anomalous behavior exists in the charging process of the article according to the actual overlap degree range based on the mapping relationship between the preset overlap degree range and the sub-anomalous behavior, where the target anomalous behavior is an anomalous behavior mapped to the actual overlap degree range determined in the sub-anomalous behavior.

In some embodiments, the above device for detecting an abnormality in charging an article further comprises an alarm module; the alarm module is used for sending alarm grade information corresponding to the target abnormal behavior to an alarm application program or terminal equipment.

In some embodiments, the obtaining module 62 is further configured to, in response to a region selection instruction triggered for the image to be detected, determine an image region indicated by the region selection instruction as the target detection region; or, the obtaining module 62 determines an image area sent by the terminal device as the target detection area, where the image area is determined by the terminal device based on an area selection instruction triggered for the image to be detected, and the area selection instruction is used to indicate the image area.

The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.

This embodiment also provides an anomaly detection behavior system for article charging, and fig. 7 is a block diagram of a structure of an anomaly detection behavior system for article charging according to an embodiment of the present application, as shown in fig. 7, the system includes: an image acquisition device 12 and a control device 14; wherein the image capturing device 12 may be fixedly mounted to the upper end of the article loading passage area. The image acquisition equipment 12 is used for acquiring an image to be detected in the article charging process and sending the image to be detected to the control device 14; the control device 14 is configured to input the image to be detected into a pre-trained article detection model, and determine an article region of the image to be detected; the control device 14 determines whether there is abnormal behavior in the process of charging the article based on the correlation information of the article region and the target detection region determined based on the region where the article is added in the process of charging the article.

Through the embodiment, the control device 14 inputs the image to be detected into the article detection model to determine the article area of the image to be detected, calculates the article feeding result according to the article area and the acquired target detection area, and finally determines whether the article feeding process is abnormal or not based on the article feeding result, so that the problem that the detection efficiency is low due to the fact that only manual detection or a sensor detects the article feeding condition is avoided, the problem that the efficiency of detecting the article feeding abnormality is low is solved, and a high-efficiency and accurate detection system for detecting whether the article feeding is abnormal or not is realized.

In some embodiments, the control device 14 is further configured to obtain a preset article image and article labeling information; the control device 14 performs labeling processing on the preset article image according to the article labeling information to obtain first training data, and performs preprocessing on the first training data by using a mean subtraction method to obtain second training data; the control device 14 inputs the second training data into a preset high-resolution network for training, so as to obtain the article detection model.

In some embodiments, the control device 14 is further configured to input the training data into at least two stage subnetworks of the high resolution network, so that each stage subnetwork outputs an article feature and a resolution down-sampling result, and performs fusion processing on the article feature and the resolution down-sampling result to obtain corresponding feature extraction information; the control device 14 performs upsampling processing on each feature extraction information to obtain an upsampling result, and fuses all the upsampling results to obtain a fusion result; the control device 14 sets a learning rate and an attenuation mode, and trains the high-resolution network according to the learning rate, the attenuation mode and the fusion result, so that the article detection model is obtained after the second training data iteration is finished.

In some embodiments, the control device 14 is further configured to determine a degree of overlap between the target detection area and the item area; the control device 14 determines whether there is abnormal behavior in the process of charging the article based on the degree of overlap.

In some embodiments, the control device 14 is further configured to calculate an article area intersection result between the target detection area and the article area, and calculate the article charging result according to the article area intersection result and the target detection area.

In some embodiments, the control device 14 is further configured to obtain a preset overlap degree range, and determine an actual overlap degree range, which matches the overlap degree, in the preset overlap degree range; the control device 14 determines that there is a target abnormal behavior in the charging process of the article according to the actual overlap degree range based on the mapping relationship between the preset overlap degree range and the sub-abnormal behaviors, where the target abnormal behavior is an abnormal behavior mapped to the actual overlap degree range determined in the sub-abnormal behaviors

In some embodiments, the system for detecting an abnormality in the charging of an article further includes a terminal device; the terminal device can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers, internet of things devices and portable wearable devices; the terminal device may be connected to the control device 14 through a network; it should be noted that the terminal device and the control device may also be the same hardware device. The control device 14 is further configured to send alarm level information corresponding to the target abnormal behavior to an alarm application or the terminal device.

In some embodiments, the control device 14 is further configured to, in response to a region selection instruction triggered for the image to be detected, determine an image region indicated by the region selection instruction as the target detection region; or, the control device 14 determines an image area sent by the terminal device as the target detection area, where the image area is determined by the terminal device based on an area selection instruction triggered for the image to be detected, and the area selection instruction is used to indicate the image area.

In some embodiments, a computer device is provided, and the computer device may be a server, and fig. 8 is a structural diagram of the inside of a computer device according to the embodiment of the present application, as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The database of the computer device is used to store the item detection model. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to realize the abnormal behavior detection method for charging the article.

Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

The present embodiment also provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

and S1, acquiring an image to be detected acquired in the article charging process.

S2, inputting the image to be detected into a pre-trained article detection model, and determining the article area of the image to be detected;

and S3, determining whether abnormal behaviors exist in the product filling process based on the related information of the product area and the target detection area, wherein the target detection area is determined based on the area of the product added in the product filling process.

It should be noted that, for specific examples in this embodiment, reference may be made to examples described in the foregoing embodiments and optional implementations, and details of this embodiment are not described herein again.

In addition, in combination with the method for detecting abnormal behavior of article charging in the above embodiments, the embodiments of the present application may be implemented by providing a storage medium. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the above-described embodiments of the method for detecting abnormal behavior in the filling of an article.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

It should be understood by those skilled in the art that various features of the above-described embodiments can be combined in any combination, and for the sake of brevity, all possible combinations of features in the above-described embodiments are not described in detail, but rather, all combinations of features which are not inconsistent with each other should be construed as being within the scope of the present disclosure.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method for detecting abnormal behavior of an article charging, the method comprising:

2. The abnormal behavior detection method according to claim 1, wherein before the obtaining of the image to be detected in the process of charging the article, the method further comprises:

acquiring a preset article image and article marking information;

3. The abnormal behavior detection method according to claim 2, wherein the inputting the second training data into a preset high-resolution network for training to obtain the item detection model comprises:

4. The abnormal behavior detection method according to claim 1, wherein the determining whether the abnormal behavior exists in the process of charging the article based on the associated information of the article area and the target detection area comprises:

determining a degree of overlap of the target detection zone and the item zone;

and determining whether abnormal behaviors exist in the article charging process according to the overlapping degree.

5. The abnormal behavior detection method according to claim 4, wherein the determining the degree of overlap of the target detection area and the item area comprises:

and calculating to obtain an article area intersection result between the target detection area and the article area, and calculating to obtain the overlapping degree according to the article intersection result and the target detection area.

6. The abnormal behavior detection method according to claim 4 or 5, wherein the determining whether the abnormal behavior exists in the process of charging the article according to the degree of overlap comprises:

7. The abnormal behavior detection method according to claim 6, further comprising, after determining that the target abnormal behavior exists during the charging of the article:

8. The abnormal behavior detection method according to any one of claims 1 to 5, wherein before determining whether the abnormal behavior exists in the process of charging the article based on the correlation information between the article region and the target detection region, the method further comprises:

9. An abnormal behavior detection system for charging an article, the system comprising: an image acquisition device and a control device;

the image acquisition equipment is used for acquiring an image to be detected in the article feeding process and sending the image to be detected to the control device;

the control device is used for executing the abnormal behavior detection method for the charging of the articles according to any one of claims 1 to 8.

10. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is configured to execute the computer program to perform the method of detecting abnormal behavior in the charging of an item of any one of claims 1 to 8.

11. A storage medium having stored thereon a computer program, wherein the computer program is arranged to execute the method of detecting abnormal behaviour in the filling of an item of any one of claims 1 to 8 when running.