CN111553355A - Method for detecting out-of-store operation and notifying management shop owner based on monitoring video - Google Patents
Method for detecting out-of-store operation and notifying management shop owner based on monitoring video Download PDFInfo
- Publication number
- CN111553355A CN111553355A CN202010421513.5A CN202010421513A CN111553355A CN 111553355 A CN111553355 A CN 111553355A CN 202010421513 A CN202010421513 A CN 202010421513A CN 111553355 A CN111553355 A CN 111553355A
- Authority
- CN
- China
- Prior art keywords
- shop
- store
- name
- operation detection
- video image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012544 monitoring process Methods 0.000 title claims abstract description 19
- 238000001514 detection method Methods 0.000 claims abstract description 48
- 238000000605 extraction Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 12
- 238000011176 pooling Methods 0.000 claims description 8
- 238000012790 confirmation Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000005070 sampling Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 235000011888 snacks Nutrition 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/40—Document-oriented image-based pattern recognition
- G06V30/41—Analysis of document content
- G06V30/413—Classification of content, e.g. text, photographs or tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Tourism & Hospitality (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Educational Administration (AREA)
- Evolutionary Biology (AREA)
- Development Economics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Closed-Circuit Television Systems (AREA)
- Alarm Systems (AREA)
Abstract
The invention provides a method for detecting and informing a management shop owner of the out-shop operation based on a monitoring video, which comprises the steps of obtaining a video image picture of the out-shop operation detection through a camera, uploading the video image picture to a city management system, inputting the image obtained by the video into a shop name text extraction and shop name type identification model through the city management system, inputting an original video image into the out-shop operation detection model through the city management system after obtaining a shop name identification result, and respectively carrying out corresponding event processing by the city management system according to different conditions by combining different retrieval results in an illegal out-shop operation to-be-processed database; and when the out-of-store operation detection model detects that the out-of-store operation of the store is changed from out-of-store operation to out-of-store operation, cutting the event folder from the to-be-processed database into the end-of-record database for storage.
Description
Technical Field
The invention relates to the field of deep learning image processing, relates to technologies such as deformable convolution, deformable pooling, a convolutional neural network, deep learning and image classification, and particularly relates to a method for detecting out-of-store operation and notifying a manager of the out-of-store operation based on a monitoring video.
Background
China is fast in economic development, and the standardized operation of stores in daily life is very important for city construction and resident life. The quick and effective operation detection and correction of the shop leaving and the subsequent check are more and more emphasized by the urban management department. The needs of going out of store operation detection, informing store owners of rectification and post-check are carried out by using images obtained by urban management monitoring videos, and the needs are more and more urgent in the field of store standard management.
The achievements of the current recognition technology and the image matching technology are dramatically advanced in the academic research field, but most of the current recognition systems and image matching systems are applied to face recognition, satellite images, medical treatment and the like, but are not widely and deeply applied to store standard management in real life.
Meanwhile, the detection of the out-of-store operation by utilizing the image recognition matching technology is challenging, because the store name in the monitoring video is seriously deformed, and the commodities operated outside the store are various, the false alarm rate is increased, the current after-check is time-consuming, labor-consuming and labor-consuming, the key for solving the problems lies in how to determine that the articles in the store operation range belong to the store operation commodities, and the current image recognition technology cannot well solve the technical problem.
Disclosure of Invention
The invention aims to overcome the problems and provides a method for detecting and informing a management shop owner based on monitoring video, and aims to provide a plurality of urban management assisting tools and methods for assisting urban management personnel to timely and accurately detect the operation of the shop owner or the shop at the street according to the monitoring video image, conveniently complete the tasks of informing the shop owner of rectification and video check, and improve the urban management efficiency. The technical scheme provides a monitoring video-based method for detecting out-of-store operation and informing management of store owners, which comprises the following steps:
step (1): acquiring a video image picture;
step (2): inputting a video image picture into a shop name text extraction and shop name type identification model to obtain a parameter group of a shop name prediction box, a shop name in a text form and a shop name category, wherein the parameter group comprises a coordinate value of the upper left corner of the prediction box and a width and height value of the coordinate value;
and (3): inputting the original video image picture into a store-out operation detection model, and judging whether the store is operated after being taken out of the store;
and (4): step (4.1) when the out-of-store operation detection model does not detect that the out-of-store operation exists in the video image picture, inputting a next picture to repeatedly perform the step (2) and the step (3) to identify the store name and detect the out-of-store operation; and (4.2) when the out-of-store operation detection model detects that the out-of-store operation condition exists in the video image picture, searching and obtaining the shop owner information through the shop information determined by the camera position corresponding to the video image picture, and informing the shop owner.
In addition, a corresponding electronic device and computer program medium are provided.
Compared with the prior art, the technical scheme has the following characteristics and beneficial effects:
the method comprises the steps that a video image picture needing to be subjected to store-out operation detection is obtained through a monitoring camera and uploaded to a city management system, the city management system inputs an image obtained through a video into a store name text extraction and store name type identification model, the identification precision and efficiency of store names are improved through a deep learning model, the problem that store names in a monitoring video are seriously deformed is solved, after a store name identification result is obtained, an original video image is input into the store-out operation detection model through the city management system, and corresponding event processing is respectively carried out on the city management system according to different conditions by combining different retrieval results in an illegal store-out operation to-be-processed database; when the out-store operation detection model detects that the store is changed from out-store operation to out-store operation, the event folder is cut from the database to be processed to the end-record database for storage, and the follow-up shop owner adjustment and video check tasks are intelligently and efficiently completed.
Drawings
FIG. 1 is a flow chart of a method for a surveillance video based store operation detection and notification management circuit.
Fig. 2 is a diagram of a store name text extraction and store name type identification network.
Fig. 3 is a network structure diagram of a text feature extraction module.
Fig. 4 is a diagram of a store operation detection model network structure.
FIG. 5 is a diagram of a network architecture of the HG module.
Fig. 6 is a flow chart of the out-of-store operation secondary confirmation module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
It will be understood by those skilled in the art that in the present disclosure, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are used in an orientation or positional relationship indicated in the drawings for ease of description and simplicity of description, and do not indicate or imply that the referenced devices or components must be constructed and operated in a particular orientation and thus are not to be considered limiting.
It is understood that the terms "a" and "an" should be interpreted as meaning that a number of one element or element is one in one embodiment, while a number of other elements is one in another embodiment, and the terms "a" and "an" should not be interpreted as limiting the number.
The scheme provides a monitoring video-based method for detecting the out-of-store operation and informing the management of the shop owner, wherein the monitoring video-based method for detecting the out-of-store operation is used for detecting the condition of the shop operation in the monitoring video, and the monitoring video-based method for informing the management of the shop owner is combined with the out-of-store operation detection method to complete the tasks of informing the shop owner of correction and video check.
In this embodiment, as shown in fig. 1, a method for detecting and notifying a management owner of an out-store operation based on a surveillance video includes the following steps:
step (1): acquiring a video image picture needing to be subjected to out-of-store operation detection through a camera, and uploading the video image picture to a city management system;
step (2): inputting a video image picture into a shop name text extraction and shop name type identification model by a city management system to obtain a parameter group of a shop name prediction box, a shop name in a text form and a shop name category, wherein the parameter group comprises coordinate values (x, y) of the upper left corner of the prediction box and width and height values thereof, and the parameter groups are all formed by the following parameters;
and (3): inputting the original video image picture into a store-out operation detection model by the urban management system, and judging whether the store is operated after being taken out of the store;
and (4):
step (4.1) when the out-of-store operation detection model does not detect that the out-of-store operation exists in the video image picture, the urban management system inputs the next picture and repeatedly performs the step (2) and the step (3) to recognize the store name and detect the out-of-store operation;
step (4.2) when the out-of-store operation detection model detects that the out-of-store operation condition exists in the video image picture, determining the store information (including the location and the name of the store) through the position of the camera, searching and obtaining the shop owner information (including the relevant information and the contact way of the shop owner) of the shop owner in the shop management database, and informing the shop owner in the modes of telephone, WeChat or short message and the like; and simultaneously creating a folder for storing pictures of the video image pictures corresponding to different task numbers n, wherein the pictures of the video image pictures are detected to be out of store for business, specifically, creating a folder with the task number n as a folder name, wherein n is 1, 2, 3 …, and storing the pictures detected to be out of store into the folder, and the picture name format is as follows: time, place and shop name, wherein the time and the place are acquired in real time through a camera, and then the folder is stored into a violation 'ex-shop operation' to-be-processed database as an event;
and (5): inputting a video image picture of the camera after a set time T into a shop name text extraction and shop name type identification model again for shop name detection, inputting the shop name text extraction and the shop name type identification model for shop-out operation detection, intelligently searching in an illegal 'shop-out operation' to-be-processed database according to the recognized shop name and a shop location determined by the position of the camera, and inputting a next picture by a city management system for shop name identification and shop-out operation detection if the same shop name and place are not searched in the illegal 'shop-out operation' to-be-processed database and the shop-out operation condition of the shop is not detected; if the same store name and place are not searched in the illegal 'out-of-store operation' to-be-processed database, but the out-of-store operation condition of the store is detected, returning to the step (4.2); if the same store name and place are searched in the illegal 'store-out operation' to-be-processed database, and the condition that the store still runs out is detected, relevant workers are assigned to carry out on-site processing on the task n, and meanwhile, pictures of the detected store-out operation are stored in an event folder with the task number n in a naming format of time + place + store name + relevant worker number.
And (6): after a set time period, acquiring a video image picture of a store in a violation 'store-out operation' to-be-processed database, inputting the video image picture into a store-out operation detection model to perform store-out operation detection, and if store-out operation is not detected, performing end-to-end processing, namely, saving a picture of a video image picture corresponding to the no store-out operation into a folder corresponding to an event, and transferring the folder of the event folder into an end-to-end database by taking time + place + store name as a naming format, namely, cutting the folder from the to-be-processed database into the end-to-end database to be saved.
Further, the shop name text extraction and shop name type recognition model and the training process in the step (2) are as follows:
step (2.1) constructing a training and testing data set: the method comprises the steps of obtaining and labeling image data containing stores to obtain training and testing labels, wherein the labels used for a store name text extraction and a store name type identification model are triples and comprise (1) a store name frame marked by a rectangular frame in real image data obtained by a camera, (2) the types of the store name frame, namely the types of the store name, such as catering, furniture, automobile stores and the like, and (3) parameter group data of the store name frame, wherein the parameter group data comprise coordinate values (x, y) of the upper left corner of the store name frame and width and height values of the coordinate values.
Step (2.2) model structure design: a network structure of a shop name text extraction and shop name type identification model is characterized in that a characteristic pyramid network structure is adopted, the network structure comprises i 16 layers of classical convolution layers and i 16 layers of upper sampling layers, a characteristic diagram output by a k-th layer of classical convolution and a characteristic diagram output by an (i-k) th layer of upper sampling layers are input into the (i +1-k) th layer of upper sampling layers through channel fusion and used for extracting basic characteristics, then the characteristic diagrams extracted by each layer of upper sampling layers are respectively input into an initial text processing module, and the initial text processing module comprises L6 layers of classical convolution layers and j 3 layers of deformable convolution layers. And secondly, performing feature fusion on the obtained i-16 feature maps through a splicing layer (namely, splicing the feature maps with the same size). Finally, inputting the fused feature map into an improved region generation network (RPN) to obtain text characters;
wherein, the regional generation network is improved in that an ROI (region of interest) pooling layer in the network is changed into a deformable PS (Position-Sensitive) ROI pooling layer, and mask instance segmentation is carried out after a prediction frame is obtained; the mask example segmentation is to multiply the real frame in the label with the image to be processed to obtain the image of the region of interest, namely, the image value in the region of interest is kept unchanged, and the image value outside the region is 0.
Wherein, the formula of the deformable convolution layer is as follows:
in the formula P0Is represented by PnIs the offset (in integer) of each point of the convolution output relative to each point on the receptive field, the offset Δ PnIs obtained by convolution of the previous layer;
the deformable PS (Position-Sensitive) ROI pooling layer formula is as follows:
the boxes (ROI) based on the feature map are divided into K3 boxes (bin), where p0For the coordinates of the upper left corner of each box (bin), p is the coordinates of each point in the box relative to p0Amount of coordinate shift of, nijIs the number of dots in the (i, j) th box, Δ pijIs the offset per box, i.e. the offset Δ pijIs for the whole box, Δ p for each point in a boxijThe values are all the same;
step (2.3) model training: assigning an initialization value to the network parameter, and setting the maximum iteration number m of the network to be 1000000; and inputting the prepared data set into a network for training. If the loss value is decreased all the time, continuing training until a final model is obtained after iteration for m times; if the loss value tends to be stable in the midway, stopping iteration to obtain a final model;
wherein the loss function formula is as follows:
L＝LRcls+LRreg+λLmask
in the formula LRclsAnd LRregIs a loss function of classification and regression of the frames by the region-generated network (RPN), LmaskThe loss function is a loss function of a segmentation task, for example, a cross entropy loss function, λ ═ 0.6 is a fixed weight parameter, and can be set according to actual needs;
wherein L isRclsAnd LRregThe formula is as follows:
in the formula NclsIs the total number of anchor points (anchors), PiIs the predicted probability that the ith anchor point is the target,
for a true tag value, the value is 1 when the anchor is positive, otherwise it is 0. t is tiIs a vector composed of 4 parameters representing the prediction boundary, which are the center point coordinates x, y and width and height, respectively
Is a vector composed of 4 parameters of a real bounding box related to a positive anchor point, and R is smoothL1A function;
the anchor point judges whether the real frame is positive or negative through the intersection ratio of the real frame and the prediction frame, when the intersection ratio is greater than a threshold value a which is 0.5, the anchor point is positive, and the intersection ratio formula is as follows:
in the formula, IOU represents the cross-over ratio, Pred and Truth represent prediction and reality respectively, area represents the area of rectangular Box, BoxpAnd BoxtRespectively representing a prediction box and a real box;
step (2.4) model use: inputting a picture to be recognized, if a shop name text extraction and a shop name type recognition model detect that a shop name exists in the picture, outputting a parameter group P of a shop name prediction box, wherein the parameter group P comprises an upper left coordinate value (x, y) and a width and height value of the box, and simultaneously outputting a shop name and a shop category on the picture in a text form, such as catering-xx snacks or home-xx homes and the like; if the store name recognition model does not detect the store name in the picture, a signal that the store name is not in the picture is output, for example, 0 is output.
Further, the out-of-store operation detection model and the training process in the step (3) are as follows:
step (3.1) constructing a training and testing data set: all shops operated by the shop are framed out by a rectangular frame in the video image, the frame comprises a shop name, a shop front and a shop front belonging to the shop operation range, and the label name is the shop operation;
step (3.2) model structure: the out-of-store operation detection model mainly comprises an HG module, a residual convolution network and an out-of-store operation secondary confirmation module. The HG module is a pyramid network consisting of 8 layers of convolution layers and 8 layers of upper sampling layers; the residual convolution network consists of 12 convolution layers, 6 residual modules, a full-connection layer and a classification layer, wherein each residual module consists of 2 convolution layers, 3 convolution layers and 1 convolution layer, and f is 1 convolution layer;
go out of store and manage secondary and confirm the module: firstly, calculating the coincidence rate of a shop name prediction frame obtained in a shop name text extraction and shop name type recognition model and a shop-going operation prediction frame obtained in a classification layer, selecting the class to which the shop name with the highest coincidence rate belongs, screening all target objects obtained in the classification layer, calculating the coincidence rate of a prediction frame parameter group of the selected target object belonging to the shop name class and a shop-crossing region frame parameter group, and outputting the parameter group of the shop-going operation prediction frame if the target object with the coincidence rate larger than a threshold value d of 0.8 exists;
wherein, the calculation formula of the coincidence ratio CR is as follows:
in the formula, x1、y1、w1And h1Is a set of parameter sets, x2、y2、w2And h2A set of parameter sets, and the upper left corner of the image is taken as an origin;
step (3.3) model training: the out-of-store operation detection network firstly carries out primary training by using a target identification data set, such as a COCO data set, and then carries out secondary training by using an out-of-store operation training data set marked in a real video;
and assigning initialization values to the network parameters and the weights of the target identification network, and setting the maximum iteration number m of the network to be 1000000. Inputting the prepared data set into a network for training; if the loss value is decreased all the time, continuing training until a final model is obtained after iteration for m times; if the loss value tends to be stable in the midway, stopping iteration to obtain a final model;
step (3.4) model use: respectively drawing an off-store business district frame for each store needing out-store business operation detection on a camera video, wherein the off-store business district frame belongs to an off-store business range in front of a corresponding store door, and takes a store name as the name of the off-store business district frame of the store, and simultaneously records the parameter group of the frame;
then, the picture to be detected, the shop name type obtained by the shop name detection model, the parameter set of the shop name prediction box and the parameter set of the out-of-store operation area box are input into the out-of-store operation detection model, and when the out-of-store operation detection model detects that the out-of-store operation situation exists in the picture, a signal for out-of-store operation is output, for example, 1 is output.
In addition, according to another aspect of the present invention, the present invention provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the above-mentioned steps of the method for detecting the out-of-store operation based on the urban management monitoring video and notifying the management owner.
According to another aspect of the present invention, the present solution provides a computer-readable storage medium having stored thereon a computer program, which when executed by a processor, performs the above-mentioned steps of the method for detecting and notifying an out-of-store operation based on an urban management monitoring video to a managing store owner.
The present invention is not limited to the above-mentioned preferred embodiments, and any other products in various forms can be obtained by anyone in the light of the present invention, but any changes in the shape or structure thereof, which have the same or similar technical solutions as those of the present application, fall within the protection scope of the present invention.
Claims (10)
1. A monitoring video-based method for detecting out-of-store operation and informing management of store owners is characterized by comprising the following steps:
step (1): acquiring a video image picture;
step (2): inputting a video image picture into a shop name text extraction and shop name type identification model to obtain a parameter group of a shop name prediction box, a shop name in a text form and a shop name category, wherein the parameter group comprises a coordinate value of the upper left corner of the prediction box and a width and height value of the coordinate value;
and (3): inputting the original video image picture into a store-out operation detection model, and judging whether the store is operated after being taken out of the store;
and (4): step (4.1) when the out-of-store operation detection model does not detect that the out-of-store operation exists in the video image picture, inputting a next picture to repeatedly perform the step (2) and the step (3) to identify the store name and detect the out-of-store operation; and (4.2) when the out-of-store operation detection model detects that the out-of-store operation condition exists in the video image picture, searching and obtaining the shop owner information through the shop information determined by the camera position corresponding to the video image picture, and informing the shop owner.
2. The monitored video-based out-of-store operation detection and notification management shopkeeper method according to claim 1,
in the step (4.2), a folder for storing pictures of video image pictures corresponding to different task numbers n and detected out-of-store operation is created, wherein the picture format is time + place + store name, and the folder is stored into a violation "out-of-store operation" to-be-processed database as an event;
and (5): inputting a video image picture of the camera after a set time T into a shop name text extraction and shop name type identification model again for shop name detection, inputting the shop name text extraction and the shop name type identification model for shop-out operation detection, searching the illegal 'shop-out operation' to-be-processed database according to the recognized shop name and a shop location determined by the position of the camera, and inputting a next picture for shop name identification and shop-out operation detection if the same shop name and place are not searched in the illegal 'shop-out operation' to-be-processed database and the shop-out operation condition of the shop is not detected; if the same store name and place are not searched in the illegal 'out-of-store operation' to-be-processed database, but the out-of-store operation condition of the store is detected, returning to the step (4.2); if the same store name and place are searched in the illegal 'store-out operation' to-be-processed database, and the condition that the store still runs out is detected, relevant workers are assigned to process the task n, and meanwhile, pictures of the detected store-out operation are stored in a folder with the task number n in a naming format of time + place + store name + relevant + worker number.
3. The monitored video-based out-of-store operation detection and notification management store owner method according to claim 2, wherein the step (6): after a set time period, acquiring a video image picture of a store in a violation 'store-out operation' to-be-processed database, inputting the video image picture into a store-out operation detection model to perform store-out operation detection, and performing settlement processing if store-out operation is not detected, namely, storing a picture of a video image picture corresponding to no store-out operation into a folder corresponding to an event, and transferring the folder into a settlement database by taking time + place + store name as a naming format.
4. The shop exit operation detection and notification management shop owner method based on surveillance video according to claim 1, wherein the shop name text extraction and shop name type identification model adopts a feature pyramid network structure and comprises a classical convolutional layer and an upsampling layer, feature maps extracted by each upsampling layer are respectively input into an initial text processing module, the initial text processing module comprises a classical convolutional layer and a deformable convolutional layer, the obtained feature maps are fused and input into an improved regional generation network to obtain web characters, the regional generation network changes an ROI pooling layer in the network into a deformable PSROI pooling layer, and mask instance segmentation is performed after a prediction frame is obtained.
5. The method for store owner management based on monitored video outlet operation detection and notification according to claim 4, wherein the label for the store name text extraction and the store name type identification model is a triple including at least (1) a store name box marked with a rectangular box, (2) a category of the store name box, and (3) parameter group data of the store name box.
6. The shop operation detection and notification management shop owner method based on surveillance video according to claim 1, wherein the shop operation detection model is composed of an HG module, a residual convolutional network, and a shop operation secondary confirmation module, wherein the shop operation secondary confirmation module calculates a coincidence rate of the shop operation prediction frame obtained by the shop name prediction frame obtained in the shop name text extraction and shop name type identification model and the shop operation prediction frame obtained by the residual convolutional network, selects a category to which the shop name with the highest coincidence rate belongs, performs category screening on all target objects obtained by the residual convolutional network, performs coincidence rate calculation on the prediction frame parameter group of the selected target object belonging to the shop name category and the shop-outside camp area frame parameter group, and outputs the parameter group of the shop operation prediction frame if there is a target object with a coincidence rate greater than a threshold value.
7. The shop exit operation detection and notification management shop owner method based on surveillance video according to claim 1, wherein the shop name text extraction and shop name type identification model adopts a feature pyramid network structure and comprises a classical convolutional layer and an upsampling layer, feature maps extracted by each upsampling layer are respectively input into an initial text processing module, the initial text processing module comprises a classical convolutional layer and a deformable convolutional layer, the obtained feature maps are fused and input into an improved regional generation network to obtain web characters, the regional generation network changes an ROI pooling layer in the network into a deformable PSROI pooling layer, and mask instance segmentation is performed after a prediction frame is obtained.
8. The shop operation detection and notification management shop owner method based on surveillance video according to claim 1, wherein the shop operation detection model is composed of an HG module, a residual convolutional network, and a shop operation secondary confirmation module, wherein the shop operation secondary confirmation module calculates a coincidence ratio of a shop name prediction box obtained in the shop name text extraction and shop name type recognition model and a shop operation prediction box obtained by the residual convolutional network, selects a category to which the shop name with the highest coincidence ratio belongs, performs category screening on all target objects obtained by the residual convolutional network, performs coincidence ratio calculation on a prediction box parameter group of the selected target object belonging to the shop name category and an off-shop camp area frame parameter group, and outputs the parameter group of the shop operation prediction box if there is a target object with a coincidence ratio greater than a threshold.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor when executing the program performs the steps of the method for detecting and notifying a manager of an out-of-store operation based on a city management video according to any one of claims 1 to 6.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method for detecting and notifying a managing store owner of an out-of-store operation based on an urban management monitoring video according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010421513.5A CN111553355B (en) | 2020-05-18 | 2020-05-18 | Monitoring video-based method for detecting and notifying store outgoing business and managing store owner |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010421513.5A CN111553355B (en) | 2020-05-18 | 2020-05-18 | Monitoring video-based method for detecting and notifying store outgoing business and managing store owner |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111553355A true CN111553355A (en) | 2020-08-18 |
| CN111553355B CN111553355B (en) | 2023-07-28 |
Family
ID=72000431
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010421513.5A Active CN111553355B (en) | 2020-05-18 | 2020-05-18 | Monitoring video-based method for detecting and notifying store outgoing business and managing store owner |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111553355B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112200631A (en) * | 2020-10-12 | 2021-01-08 | 支付宝(杭州)信息技术有限公司 | Industry classification model training method and device |
| CN112541772A (en) * | 2020-12-04 | 2021-03-23 | 浪潮云信息技术股份公司 | Merchant-oriented qualification authentication method |
| CN112613569A (en) * | 2020-12-29 | 2021-04-06 | 北京百度网讯科技有限公司 | Image recognition method, and training method and device of image classification model |
| CN113344064A (en) * | 2021-05-31 | 2021-09-03 | 北京百度网讯科技有限公司 | Event processing method and device |
| CN114140734A (en) * | 2021-12-10 | 2022-03-04 | 北京商海文天科技发展有限公司 | Video data-based off-store business analysis method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180293684A1 (en) * | 2016-09-07 | 2018-10-11 | Southeast University | Supervision and penalty method and system for expressway emergency lane occupancy |
| CN108985169A (en) * | 2018-06-15 | 2018-12-11 | 浙江工业大学 | Across the door operation detection method in shop based on deep learning target detection and dynamic background modeling |
| CN109063612A (en) * | 2018-07-19 | 2018-12-21 | 中智城信息技术有限公司 | City intelligent red line management method and machine readable storage medium |
| CN109345435A (en) * | 2018-12-07 | 2019-02-15 | 山东晴天环保科技有限公司 | Occupy-street-exploit managing device and method |
| CN109977782A (en) * | 2019-02-27 | 2019-07-05 | 浙江工业大学 | Across shop operation detection method based on target position information reasoning |
| WO2019192397A1 (en) * | 2018-04-04 | 2019-10-10 | 华中科技大学 | End-to-end recognition method for scene text in any shape |
-
2020
- 2020-05-18 CN CN202010421513.5A patent/CN111553355B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180293684A1 (en) * | 2016-09-07 | 2018-10-11 | Southeast University | Supervision and penalty method and system for expressway emergency lane occupancy |
| WO2019192397A1 (en) * | 2018-04-04 | 2019-10-10 | 华中科技大学 | End-to-end recognition method for scene text in any shape |
| CN108985169A (en) * | 2018-06-15 | 2018-12-11 | 浙江工业大学 | Across the door operation detection method in shop based on deep learning target detection and dynamic background modeling |
| CN109063612A (en) * | 2018-07-19 | 2018-12-21 | 中智城信息技术有限公司 | City intelligent red line management method and machine readable storage medium |
| CN109345435A (en) * | 2018-12-07 | 2019-02-15 | 山东晴天环保科技有限公司 | Occupy-street-exploit managing device and method |
| CN109977782A (en) * | 2019-02-27 | 2019-07-05 | 浙江工业大学 | Across shop operation detection method based on target position information reasoning |
Non-Patent Citations (4)
| Title |
|---|
| YUANPING ZHU等: "Curved Scene Text Detection Based on Mask R-CNN" * |
| 徐向波;: "基于智慧化的城市管理发现问题模式" * |
| 王建新;王子亚;田萱;: "基于深度学习的自然场景文本检测与识别综述" * |
| 郝力;: "数字城管迈向智慧城管(三)" * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112200631A (en) * | 2020-10-12 | 2021-01-08 | 支付宝(杭州)信息技术有限公司 | Industry classification model training method and device |
| CN112541772A (en) * | 2020-12-04 | 2021-03-23 | 浪潮云信息技术股份公司 | Merchant-oriented qualification authentication method |
| CN112613569A (en) * | 2020-12-29 | 2021-04-06 | 北京百度网讯科技有限公司 | Image recognition method, and training method and device of image classification model |
| CN112613569B (en) * | 2020-12-29 | 2024-04-09 | 北京百度网讯科技有限公司 | Image recognition method, training method and device for image classification model |
| CN113344064A (en) * | 2021-05-31 | 2021-09-03 | 北京百度网讯科技有限公司 | Event processing method and device |
| CN114140734A (en) * | 2021-12-10 | 2022-03-04 | 北京商海文天科技发展有限公司 | Video data-based off-store business analysis method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111553355B (en) | 2023-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111553355B (en) | Monitoring video-based method for detecting and notifying store outgoing business and managing store owner | |
| CN110059554B (en) | Multi-branch target detection method based on traffic scene | |
| US11455805B2 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| CN109977782B (en) | Cross-store operation behavior detection method based on target position information reasoning | |
| Serna et al. | Detection, segmentation and classification of 3D urban objects using mathematical morphology and supervised learning | |
| CN106951830B (en) | Image scene multi-object marking method based on prior condition constraint | |
| CN110796141A (en) | Target detection method and related equipment | |
| CN112801050B (en) | Intelligent luggage tracking and monitoring method and system | |
| CN110188807A (en) | Tunnel pedestrian target detection method based on cascade super-resolution network and improvement Faster R-CNN | |
| CN105005760B (en) | A kind of recognition methods again of the pedestrian based on Finite mixture model | |
| CN110555420B (en) | Fusion model network and method based on pedestrian regional feature extraction and re-identification | |
| WO2012139228A1 (en) | Video-based detection of multiple object types under varying poses | |
| CN111931582A (en) | Image processing-based highway traffic incident detection method | |
| WO2023206904A1 (en) | Pedestrian trajectory tracking method and system, and related apparatus | |
| CN113723361A (en) | Video monitoring method and device based on deep learning | |
| CN112560675A (en) | Bird visual target detection method combining YOLO and rotation-fusion strategy | |
| CN112766069A (en) | Vehicle illegal parking detection method and device based on deep learning and electronic equipment | |
| CN112836657A (en) | Pedestrian detection method and system based on lightweight YOLOv3 | |
| CN114003672A (en) | Method, device, equipment and medium for processing road dynamic event | |
| CN115731477A (en) | Image recognition method, illicit detection method, terminal device, and storage medium | |
| Hiremath et al. | A Smart App for Pothole Detection Using Yolo Model | |
| You et al. | PANDA: predicting road risks after natural disasters leveraging heterogeneous urban data | |
| CN115131826B (en) | Article detection and identification method, and network model training method and device | |
| CN114882020B (en) | Product defect detection method, device, equipment and computer readable medium | |
| CN115223003A (en) | Yolov 5-based intrusion detection model training method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |