CN115620186A - Medicine sale monitoring method and system - Google Patents

Medicine sale monitoring method and system Download PDF

Info

Publication number
CN115620186A
CN115620186A CN202211051299.4A CN202211051299A CN115620186A CN 115620186 A CN115620186 A CN 115620186A CN 202211051299 A CN202211051299 A CN 202211051299A CN 115620186 A CN115620186 A CN 115620186A
Authority
CN
China
Prior art keywords
behavior
target area
medicine
monitoring
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211051299.4A
Other languages
Chinese (zh)
Inventor
程冰
田梦超
蔡杭洲
王�义
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Yuntian Lifei Technology Co ltd
Shenzhen Intellifusion Technologies Co Ltd
Original Assignee
Qingdao Yuntian Lifei Technology Co ltd
Shenzhen Intellifusion Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Yuntian Lifei Technology Co ltd, Shenzhen Intellifusion Technologies Co Ltd filed Critical Qingdao Yuntian Lifei Technology Co ltd
Priority to CN202211051299.4A priority Critical patent/CN115620186A/en
Publication of CN115620186A publication Critical patent/CN115620186A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the invention provides a medicine selling monitoring method and system, and belongs to the field of data processing. The method comprises the following steps: when a first object enters a first target area according to a first monitoring video corresponding to the first target area, determining whether the behavior of the first object in the first target area is a medicine taking behavior or not according to the first monitoring video; when the behavior of the first object in the first target area is determined to be the medicine taking behavior, generating medicine taking behavior information; when a second object enters a second target area according to a second monitoring video corresponding to the second target area, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be payment behavior, generating payment behavior information; and when the generated payment behavior information is inquired within the preset time length of the generated medicine taking behavior information, generating a medicine selling event. The method improves the authenticity of the medicine selling condition.

Description

Medicine sale monitoring method and system
Technical Field
The invention relates to the technical field of image processing, in particular to a medicine sale monitoring method and system.
Background
Medicines are indispensable parts in daily life of people, and medicine safety concerns people's health and life safety, and pharmacy personnel need sell the medicine according to national or regional regulation system, mainly scans the sign indicating number to the medicine through cashier's system at present and records pharmacy selling condition of pharmacy personnel, nevertheless has the situation that pharmacy personnel falsify, can't guarantee authenticity and the accuracy of pharmacy selling condition.
Disclosure of Invention
The embodiment of the invention provides a medicine selling monitoring method and system, aiming at improving the authenticity and accuracy of medicine selling conditions of drugstores.
In a first aspect, an embodiment of the present invention provides a medicine vending monitoring method, including: when a first object enters a first target area according to a first monitoring video corresponding to the first target area, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video, wherein the first target area comprises a medicine storage area; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; when a second object enters a second target area according to a second monitoring video corresponding to the second target area, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video, wherein the second target area comprises a medicine checkout area; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information, wherein the payment behavior information is used for indicating that the behavior of the second object in the second target area is a payment behavior; and when the generated payment behavior information is inquired within the preset time length of the medicine taking behavior information, generating a medicine selling event.
In an embodiment, the determining whether the behavior of the first object in the first target area is a drug taking behavior according to the first monitoring video includes: inputting the multi-frame images in the first monitoring video into a preset target detection model for processing to obtain target detection results of each frame of images in the multi-frame images; tracking the first object according to the target detection result of each frame of the image; when the first object is tracked to leave the first target area, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the target detection result of each frame of the image.
In an embodiment, the tracking the first object according to the target detection result of each frame of the image includes: calculating the intersection ratio between the head and shoulder detection frame of the first object in the first image and the head and shoulder detection frame of each object to be detected in the second image; determining the maximum intersection ratio in the intersection ratios as a target intersection ratio, and determining whether the target intersection ratio is greater than or equal to a preset intersection ratio threshold value; and when the target cross-over ratio is greater than or equal to a preset cross-over ratio threshold value, marking the target cross-over ratio corresponding to the object to be detected in the second image as the first object.
In an embodiment, the determining, according to the target detection result of each frame of the image, whether the behavior of the first object in the first target area is a drug taking behavior includes: counting the number of images corresponding to the hand state label as a medicine placing state label to obtain a first image number, wherein the medicine placing state label describes that the hand of the first object is in a medicine placing state; determining a percentage of the first number of images to the total number of the plurality of frame images; when the percentage is greater than or equal to a preset percentage threshold value, determining that the behavior of the first object in the first target area is a medicine taking behavior.
In an embodiment, the determining, according to the target detection result of each frame of the image, whether the behavior of the first object in the first target area is a drug taking behavior includes: determining a first target image, wherein the first target image is an image corresponding to a medicine placing state label which is the hand state label detected for the first time in the multi-frame image, and the medicine placing state label describes that the hand of the first object is in a medicine placing state; counting the number of second target images corresponding to the hand state label and the medicine placing state label, wherein the second target images are images behind the first target images in the multi-frame images; adding 1 to the number of the second target images to obtain the number of second images; when the second image number is larger than or equal to a preset image number threshold value, determining that the behavior of the first object in the first target area is a medicine taking behavior.
In an embodiment, the medicine taking behavior information includes a medicine taking behavior identifier and body feature information of the first object, the body feature information includes at least one of face feature information and head-shoulder feature information, and the medicine sale monitoring method further includes: calculating a first similarity between the body characteristic information of the first object and preset body characteristic information; and when the first similarity is greater than or equal to a preset similarity threshold value, generating a medicine selling event.
In an embodiment, the payment behavior information includes a payment behavior identifier and physical characteristic information of the second object, and after calculating a first similarity between the physical characteristic information of the first object and preset physical characteristic information, the method further includes: when the first similarity is smaller than a preset similarity threshold value, calculating a second similarity between the body characteristic information of the first object and the body characteristic information of the second object; and when the second similarity is greater than or equal to the similarity threshold, generating a medicine selling event.
In a second aspect, an embodiment of the present invention further provides a medicine vending monitoring system, where the medicine vending monitoring system includes a first monitoring device, a second monitoring device, and a server, and the first monitoring device and the second monitoring device are respectively in communication connection with the server; the first monitoring device is used for monitoring a first target area to obtain a first monitoring video, and the first target area comprises a medicine storage area; when the first target area is determined to have a first object entering according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior or not according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information and sending the medicine taking behavior information to the server, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; the second monitoring device is used for monitoring a second target area to obtain a second monitoring video, wherein the second target area comprises a medicine checkout area; when a second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information and sending the payment behavior information to the server, wherein the payment behavior information is used for indicating that the behavior of the second object in the drug checkout area is the payment behavior; the server is used for determining whether the payment behavior information sent by the second monitoring equipment is received within a preset time length or not when the medicine taking behavior information is received; and generating a medicine selling event when the payment behavior information sent by the second monitoring device is received within a preset time.
In a third aspect, an embodiment of the present invention further provides a medicine vending monitoring system, where the medicine vending monitoring system includes a first monitoring device, a second monitoring device, and a server, and the first monitoring device and the second monitoring device are respectively in communication connection with the server; the first monitoring device is used for monitoring a first target area to obtain a first monitoring video and sending the first monitoring video to the server, wherein the first target area comprises a medicine storage area; the second monitoring device is used for monitoring a second target area to obtain a second monitoring video and sending the second monitoring video to the server, wherein the second target area comprises a medicine checkout area; the server is used for acquiring a first monitoring video sent by the first monitoring equipment and a second monitoring video sent by the second monitoring equipment; when it is determined that a first object enters the first target area according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; when a second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information, wherein the payment behavior information is used for indicating that the behavior of the second object in the second target area is a payment behavior; and when the generated payment behavior information is inquired within the preset time length of the medicine taking behavior information, generating a medicine selling event.
The embodiment of the invention provides a medicine sale monitoring method and a medicine sale monitoring system, wherein when a first object enters a medicine storage area, the medicine sale monitoring method monitors whether the behavior of the first object in the medicine storage area is a medicine taking behavior, so that when the behavior of the first object in the medicine storage area is monitored to be the medicine taking behavior, medicine taking behavior information is generated, and when a second object enters a medicine checkout area, the behavior of the second object in the medicine checkout area is monitored to be a payment behavior, so that when the behavior of the second object in the medicine checkout area is monitored to be the payment behavior, payment behavior information is generated, and when the generated payment behavior information is inquired within the preset time length of the generated medicine taking behavior information, a medicine event is generated, so that medicine shop staff cannot falsely sell medicines, and the authenticity and the accuracy of the medicine sale condition are greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a scenario for implementing a method for monitoring drug sales provided by an embodiment of the present invention;
FIG. 2 is a schematic flow chart diagram illustrating a method for monitoring the sale of a pharmaceutical product according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a frame of an image in a first surveillance video in an embodiment of the invention;
FIG. 4 is a schematic flow chart illustrating sub-steps of the pharmaceutical vending monitoring method of FIG. 2;
FIG. 5 is a network level diagram of a default target detection model in an embodiment of the invention;
FIG. 6 is a diagram illustrating a target detection result of multiple frame images according to an embodiment of the present invention;
fig. 7 is another schematic diagram of a target detection result of a plurality of frame images in the embodiment of the present invention;
fig. 8 is a block diagram illustrating a structure of a medicine vending monitoring system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution order may be changed according to the actual situation.
It is to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a schematic view of a scenario for implementing a medicine vending monitoring method according to an embodiment of the present invention.
As shown in fig. 1, the scenario includes a first monitoring device 100, a second monitoring device 200, and a server 300, the first monitoring device 100 and the second monitoring device 200 are respectively in communication connection with the server 300, the first monitoring device 100 may be installed on a top or side wall of a first target area of a pharmacy, such that the first target area is located within a monitoring range of the first monitoring device 100, and the first target area includes a medicine storage area; the second monitoring device 200 may be mounted on a top or side wall of a second target area, including a drug checkout area, which is an active area of a customer when checkout at a checkout counter accessory, such that the drug checkout area is within the monitoring range of the second monitoring device 200.
In an embodiment, the first monitoring device 100 monitors a first target area to obtain a first monitoring video, where the first target area includes a medicine storage area; when the first object enters the first target area according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior or not according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information and sending the medicine taking behavior information to the server 300, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; the second monitoring device 200 monitors a second target area to obtain a second monitoring video, wherein the second target area comprises a medicine checkout area; when the second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information, and sending the payment behavior information to the server 300, wherein the payment behavior information is used for indicating that the behavior of the second object in the drug checkout area is the payment behavior; when receiving the medicine taking behavior information sent by the first monitoring device 100, the server 300 determines whether to receive the payment behavior information sent by the second monitoring device 200 within a preset time period, and generates a medicine selling event when receiving the payment behavior information sent by the second monitoring device 200 within the preset time period.
In an embodiment, the first monitoring device 100 monitors a first target area to obtain a first monitoring video, and sends the first monitoring video to the server 300, where the first target area includes a medicine storage area; the second monitoring device 200 monitors a second target area to obtain a second monitoring video, and sends the second monitoring video to the server 300, wherein the second target area comprises a medicine checkout area; the server 300 acquires a first monitoring video sent by the first monitoring device 100 and a second monitoring video sent by the second monitoring device 200; when the first object enters the first target area according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior or not according to the first monitoring video; when the behavior of the first object in the first target area is determined to be the medicine taking behavior, generating medicine taking behavior information, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; when the second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be the payment behavior, generating payment behavior information, wherein the payment behavior information is used for indicating that the behavior of the second object in the second target area is the payment behavior; and when the generated payment behavior information is inquired within the preset time for generating the medicine taking behavior information, generating a medicine selling event.
The first monitoring device 100 or the second monitoring device 200 may include one or more shooting devices and chips, the server 300 may be an independent server, a server cluster including a plurality of servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, a cloud computing, a cloud function, a cloud storage, a Network service, a cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and an artificial intelligence platform.
Hereinafter, the medicine sale monitoring method provided by the embodiment of the present invention will be described in detail with reference to the scenario in fig. 1. It should be noted that the scenario in fig. 1 is only used for explaining the medicine vending monitoring method provided in the embodiment of the present invention, and does not constitute a limitation to an application scenario of the medicine vending monitoring method provided in the embodiment of the present invention.
Referring to fig. 2, fig. 2 is a schematic flow chart of a medicine vending monitoring method according to an embodiment of the present invention.
As shown in fig. 2, the medicine sales monitoring method includes steps S101 to S105.
Step S101, when it is determined that a first object enters the first target area according to a first monitoring video corresponding to the first target area, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video.
In an embodiment of the present invention, the first target area includes a medicine storage area, the first object may include a pharmacist or a customer in a pharmacy, the medicine storage area may include a first medicine storage area and/or a second medicine storage area, the first medicine storage area is used for storing medicines that need to be registered to be sold, the medicines that need to be registered to be sold may include prescription medicines, cough stopping medicines, fever reducing medicines, antiviral medicines, antibiotic medicines, and the like, and the second medicine storage area is used for storing medicines that can be sold without being registered.
In one embodiment, a multi-frame image is obtained from a first monitoring video, and a first background image corresponding to a first target area is obtained; calculating pixel difference between each frame image in the multi-frame images and the first background image, determining that the first object enters the first target area when the pixel difference in each frame image is larger than or equal to the image corresponding to the preset pixel difference threshold value, and determining that no first object enters the first target area when the pixel difference in each frame image is smaller than the pixel difference threshold value. The first background image is an image acquired by the first monitoring device when the first object is not in the first target area. For example, as shown in fig. 3, a first object 42 is included in a region of interest 41 in one frame image in a first surveillance video, and it can be determined that the first object enters a first target region.
In one embodiment, a plurality of frames of images are obtained from a first monitoring video, and each frame of image in the plurality of frames of images is input into a preset target detection model to be processed, so that a target detection result of each frame of image is obtained; when the target detection result in each frame of image comprises the image corresponding to the head and shoulder detection frame or the face detection frame, determining that a first object enters a first target area, and when the target detection result corresponding to each frame of image does not comprise the head and shoulder detection frame or the face detection frame, determining that no first object enters the first target area. Whether the first object enters the first target area or not can be accurately determined through the target detection result of the image, so that the authenticity and the accuracy of the medicine selling event can be improved.
In one embodiment, a multi-frame image is obtained from a first monitoring video, and a first background image corresponding to a first target area is obtained; calculating pixel difference between each frame image in the multi-frame images and the first background image, acquiring an image corresponding to the pixel difference which is greater than or equal to a preset pixel difference threshold value from the multi-frame images to obtain a target image, and inputting the target image into a preset target detection model for processing to obtain a target detection result of the target image; when the target detection result of the target image includes the head-shoulder detection frame or the face detection frame, it is determined that the first object enters the first target region, and when the target detection result of the target image does not include the head-shoulder detection frame or the face detection frame, it is determined that no first object enters the first target region. By comprehensively considering the target detection result of the target image and the pixel difference between the target image and the background image, the detection accuracy of whether the first object enters the first target area can be further improved, and the authenticity and the accuracy of the medicine selling condition can be improved.
In one embodiment, as shown in fig. 4, step S101 includes sub-step S1011 to sub-step S1013.
And a substep S1011, inputting the multi-frame images in the first monitoring video into a preset target detection model for processing, and obtaining target detection results of each frame image in the multi-frame images.
In the embodiment of the invention, the target detection result comprises a first object or a head and shoulder detection frame of the object to be detected and a hand state label of the first object, wherein the hand state label comprises a medicine setting state label or an idle state label, the medicine setting state label is used for describing that the hand of the first object is in a medicine setting state, and the idle state label is used for describing that the hand of the first object is in an idle state.
In one embodiment, the preset target detection model includes a lightweight backbone Network, a bottleneck Network and a detection head Network, and the bottleneck Network includes a transposed convolutional layer and a Path Aggregation Network (PAN). By using the lightweight backbone network, the data volume of the target detection model can be reduced and the running speed of the target detection model can be improved, and the path aggregation network in the bottleneck network uses the transposed convolution layer, so that the precision loss in the conversion and quantification processes of the target detection model can be reduced, and the accuracy of the target detection model deployed on the monitoring equipment or the server can be ensured.
In one embodiment, the image is normalized, and the normalized image is subjected to feature extraction through a lightweight backbone network to obtain a plurality of first feature maps, wherein the first feature maps are different in size; processing the plurality of first feature maps through the transposed convolutional layer and the path aggregation network to obtain a plurality of second feature maps; processing the plurality of second characteristic graphs through the detection head network to obtain a plurality of target characteristic graphs; and determining a target detection result of the first object according to the plurality of target feature maps.
In one embodiment, the lightweight backbone Network includes a convolutional Network and a Cross Stage Partial Network (CSP). Compared with the existing lightweight network yolov5s, the target detection model in the implementation of the invention uses a convolution network and a cross-stage local network, does not need to frequently perform downsampling slicing operation on the image, can reduce the cache occupation of the monitoring equipment or the server, and also improves the deployment flexibility of the target detection model.
In one embodiment, the lightweight backbone network includes ShuffleNet V2. Compared with the existing lightweight network yolov5s, the trunk network of the target detection model in the implementation of the invention uses ShuffleNet V2, and does not need frequent downsampling slicing operation on the image, so that the cache occupation of monitoring equipment or a server can be reduced, the deployment flexibility of the target detection model is improved, and the data volume of the ShuffleNet V2 is smaller, so that the data volume of the target detection model can be reduced.
In one embodiment, the images after the normalization processing are convoluted through a convolution network to obtain convolution images, and the convolution images are subjected to feature extraction through a cross-stage local network to obtain a plurality of first feature maps; or, performing feature extraction on the normalized image through ShuffleNet V2 to obtain a plurality of first feature maps. The preset target detection model is explained below by taking an example in which the lightweight trunk network includes a convolutional network and a cross-stage local network, and the bottleneck network includes a transposed convolutional layer and a path aggregation network.
As shown in fig. 5, the preset target detection network includes a lightweight backbone network 10, a bottleneck network 20, and a detection head network 30, where the lightweight backbone network 10 includes a convolutional network 11, a first CSP layer 12, a second CSP layer 13, and a third CSP layer 14, and the convolutional network 11 is connected to the first CSP layer 12, the first CSP layer 12 is connected to the second CSP layer 13, and the second CSP layer 13 is connected to the third CSP layer 14. The bottleneck network 20 includes a first transposed convolutional layer 211, a second transposed convolutional layer 212, a first splice layer 213, a second splice layer 214, a fourth CSP layer 215, a first convolutional layer 216, a third splice layer 217, a fifth CSP layer 218, a second convolutional layer 219, a fourth splice layer 220, and a sixth CSP layer 221. The detector head network 30 includes a third convolutional layer 31, a fourth convolutional layer 32, and a fifth convolutional layer 33.
The first transposed buildup layer 211 is connected to the third CSP layer 14 and the first splice layer 213, the second transposed buildup layer 212 is connected to the first splice layer 213 and the second splice layer 214, the first splice layer 213 is connected to the second CSP layer 13, the second splice layer 214 is connected to the first CSP layer 12 and the fourth CSP layer 215, the fourth CSP layer 215 is connected to the first buildup layer 216, the first buildup layer 216 is connected to the third splice layer 217, the third splice layer 217 is connected to the first splice layer 213, the fifth CSP layer 218 is connected to the second buildup layer 219, the second buildup layer 219 is connected to the fourth splice layer 220, and the fourth splice layer 220 is connected to the third CSP layer 14 and the sixth CSP layer 221. The third buildup layer 31 is connected to the fourth CSP layer 215, the fourth buildup layer 32 is connected to the fifth CSP layer 218, and the fifth buildup layer 33 is connected to the sixth CSP layer 221.
In an embodiment, the image is normalized, and the normalized image is convolved by the convolution network 11 to obtain a convolved image; performing feature extraction on the convolution image through the first CSP layer 12 to obtain a feature map A 1 Feature map A is mapped by second CSP layer 13 1 Performing feature extraction to obtain a feature map A 2 Feature map A is aligned by the third CSP layer 14 2 Extracting the characteristics to obtain a characteristic diagram A 3 (ii) a The feature map A is mapped by the first transposed convolution layer 211 3 Performing transposition convolution to obtain a characteristic diagram B 1 (ii) a Feature map A is aligned by the first stitching layer 213 2 And feature map B 1 Splicing to obtain a characteristic diagram B 2 (ii) a The feature map B is aligned with the second transposed convolution layer 212 2 Performing transposition convolution to obtain a characteristic diagram B 3 (ii) a Feature map A is aligned by the second stitching layer 214 1 And a characteristic diagram B 3 Splicing to obtain a characteristic diagram B 4 (ii) a Pair of feature maps B by fourth CSP layer 215 4 Processing to obtain a characteristic diagram C 1 (ii) a By the third convolution layer 31 to the feature map C 1 Performing convolution processing to obtain a target characteristic diagram D 1 。
Pairing feature maps C with the first convolutional layer 216 1 Performing convolution processing to obtain a characteristic diagram B 5 (ii) a Feature map B is aligned by the third stitching layer 217 2 And a characteristic diagram B 5 Splicing to obtain a characteristic diagram B 6 (ii) a Feature map B by the fifth CSP layer 6 Processing to obtain a characteristic diagram C 2 By the fourth convolution layer 32, the characteristic diagram C 2 Performing convolution processing to obtain a target characteristic diagram D 2 . The fifth feature map C is mapped by the second convolution layer 219 2 Performing convolution processingObtaining a characteristic diagram B 7 (ii) a Feature map A by the fourth stitching layer 220 3 And a characteristic diagram B 7 Splicing to obtain a characteristic diagram B 8 (ii) a Feature map B by sixth CSP layer 8 Processing to obtain a characteristic diagram C 3 (ii) a Feature map C by the fifth convolution layer 33 3 Performing convolution processing to obtain a target characteristic diagram D 3 。
In sub-step S1012, the first object is tracked based on the target detection result of each frame image.
In this embodiment of the present invention, the multi-frame image includes a first image and a second image adjacent to the first image, and the manner of tracking the first object according to the target detection result of each frame image may be: calculating the intersection ratio between the head and shoulder detection frame of the first object in the first image and the head and shoulder detection frame of each object to be detected in the second image; determining the maximum intersection ratio in the intersection ratios as a target intersection ratio, and determining whether the target intersection ratio is greater than or equal to a preset intersection ratio threshold value; and when the target cross-over ratio is greater than or equal to a preset cross-over ratio threshold value, marking the target cross-over ratio corresponding to the object to be detected in the second image as a first object. The intersection ratio threshold may be set based on actual conditions, which is not specifically limited in the embodiment of the present invention. The sizes of the head and the shoulders of the person are relatively fixed, and the head and the shoulders of the person are usually not shielded, so that the head and the shoulders can be accurately identified, the cross-over ratio between the head and shoulder detection frames can be accurately calculated, the accuracy of tracking the first object can be improved, the calculation amount of the cross-over ratio between the head and shoulder detection frames is small, and the efficiency of tracking the first object can be improved.
In an embodiment, when the target cross-over ratio is smaller than the cross-over ratio threshold, the target cross-over ratio in the second image is marked as a new first object corresponding to the object to be detected. For example, let S be the area of the head and shoulder detection frame of the first object a in the first image A Object B to be detected in the second image 1 Object B to be detected 2 And an object B to be detected 3 The areas of the head and shoulder detection frames are respectively S B1 、S B2 And S B3 Then the head and shoulder detection frame of the first object A and the object B to be detected 1 And an object B to be detected 2 And an object B to be detected 3 The intersection-parallel ratio between the head and shoulder detecting frames is (S) A ∩S B1 )/(S A ∪S B1 )、(S A ∩S B2 )/(S A ∪S B2 ) And (S) A ∩S B3 )/(S A ∪S B3 ) Due to (S) A ∩S B2 )/(S A ∪S B2 ) Maximum, if (S) A ∩S B2 )/(S A ∪S B2 ) If the intersection ratio is larger than or equal to the preset intersection ratio threshold value, marking the object B2 to be detected as a first object A, and if (S) A ∩S B2 )/(S A ∪S B2 ) And if the intersection ratio is smaller than or equal to a preset intersection ratio threshold value, marking the object B2 to be detected as a new first object C.
And a substep S1013 of determining whether the behavior of the first object in the first target region is a drug-taking behavior according to the target detection result of each frame of image when the first object is tracked to leave the first target region.
Through substeps 1011 to substep S1013, when it is monitored that a first object enters the medicine storage area, the target detection model may be used to perform target detection on the acquired image, and the first object is tracked according to the target detection result, so that when the first object is tracked to leave the medicine storage area, whether the behavior of the first object in the medicine storage area is a medicine taking behavior can be determined accurately in real time based on the target detection result.
In an embodiment of the present invention, the target detection result includes a hand state tag of the first object, where the hand state tag includes a medicine placing state tag or an idle state tag, the medicine placing state tag is used to describe that the hand of the first object is in a medicine placing state, the idle state tag is used to describe that the hand of the first object is in an idle state, the hand of the first object in the medicine placing state means that the hand of the first object holds a medicine, and the hand of the first object in the idle state means that the hand of the first object does not hold the medicine.
In one embodiment, the number of images corresponding to the hand state label is counted as a medicine placing state label, and a first image number is obtained; determining a percentage of the first number of images to the total number of the plurality of frame images; when the percentage is larger than or equal to a preset percentage threshold value, determining that the behavior of the first object in the first target area is a medicine taking behavior; and when the percentage is smaller than a preset percentage threshold value, determining that the behavior of the first object in the first target area is not the medicine taking behavior. The total number of the multiple frames of images and the preset percentage threshold may be set based on an actual situation, which is not specifically limited in the embodiment of the present invention. For example, the total number of the images of the plurality of frames is 10, and the preset percentage threshold is 70% or 50%. By comprehensively considering the hand label state in the target detection result of the multi-frame image, the interference caused by the error identification of the target detection model can be reduced, and the identification accuracy of the behavior of the first object in the first target area is improved.
For example, the total number of the images of the multiple frames is 10 frames, and the hand state label of the first object in the target detection result of the 10 frames of images is as shown in fig. 6, "1" in fig. 6 is a medicine setting state label indicating that the hand of the first object is in a medicine setting state, "0" is an idle state label indicating that the hand of the first object is in an idle state or is erroneously identified, and it can be obtained through statistics that the number of images corresponding to the hand state label of "1" is 7 frames, the percentage of the images corresponding to the hand state label of "1" in the total number of the images of the multiple frames is calculated to be 70%, and the percentage threshold value is 50%, so that it can be determined that the behavior of the first object in the first target area is a medicine taking behavior.
In one embodiment, a first target image is determined, wherein the first target image is an image corresponding to a medicine placing state label of a hand detected for the first time in a multi-frame image, and the medicine placing state label describes that the hand of a first object is in a medicine placing state; counting the number of second target images corresponding to the medicine placing state labels, wherein the second target images are images behind the first target images in the multi-frame images; adding 1 to the number of the second target images to obtain the number of second images; and when the second image number is larger than or equal to a preset image number threshold value, determining that the behavior of the first object in the first target area is a medicine taking behavior. The image quantity threshold may be set based on actual conditions, which is not specifically limited in the embodiment of the present invention, and for example, the image quantity threshold is 3 or 5. By comprehensively considering the hand label state in the target detection result of the multi-frame image, the interference caused by the error identification of the target detection model can be reduced, and the identification accuracy of the behavior of the first object in the first target area is improved.
For example, the total number of the multi-frame images is 10 frames, and the hand state label of the first object in the target detection results of the 10 frames of images is as shown in fig. 7, "1" in fig. 7 is a medicine setting state label indicating that the hand of the first object is in a medicine setting state, "0" is an idle state label indicating that the hand of the first object is in an idle state or is erroneously recognized, and it can be obtained from fig. 7 that the image with the hand state label first being the medicine setting state label is the image 51, and the hand state label in the target detection results of 5 frames of images in the images located after the image 51 is "1", and therefore, the number of second target images corresponding to the hand state label being the medicine setting state label is 5, the number of second images is 6, the number of images threshold is 5, and since 6 is greater than 5, it can be determined that the behavior of the first object in the first target area is the medicine taking behavior.
In one embodiment, target detection results corresponding to respective inverse multi-frame images in the multi-frame images are obtained, and the number of images corresponding to the head and shoulder detection frames without the first object in the target detection results is counted to obtain a third image number; and calculating the percentage of the number of the third images to the total number of the last multi-frame images, and determining that the tracked first object leaves the first target area when the percentage is greater than or equal to a preset percentage. By considering the target detection results of the multi-frame images, it is possible to accurately determine whether the tracked first object leaves the first target region.
The total number and the preset percentage of the frames of images that are counted backwards may be set based on an actual situation, which is not specifically limited in the embodiment of the present invention. For example, the total number of reciprocal multiframe images is 4, and the preset percentage is 50%. And acquiring target detection results corresponding to 4 frames of images from the last 10 frames of continuous images, and determining that the tracked first object leaves the first target area when the head and shoulder detection frame of the first object does not exist in the target detection results of more than 2 frames of images in the 4 frames of images.
Step S102, when the behavior of the first object in the first target area is determined to be the medicine taking behavior, medicine taking behavior information is generated.
In this embodiment of the present invention, the medicine taking behavior information is used to indicate that a behavior of the first object in the first target area is a medicine taking behavior, and the medicine taking behavior information includes a medicine taking behavior identifier, or the medicine taking behavior information includes a medicine taking behavior identifier and body feature information of the first object, the body feature information of the first object may be obtained by the first monitoring device by recognizing the first object, and the body feature information may include face feature information or head-shoulder feature information.
Step S103, when it is determined that a second object enters the second target area according to a second monitoring video corresponding to the second target area, whether the behavior of the second object in the second target area is a payment behavior is determined according to the second monitoring video.
In an embodiment of the present invention, the second target area includes a medicine checkout area, and the medicine checkout area is an active area when a customer checks out at the checkout counter accessory.
In one embodiment, a plurality of frame images are obtained from the second monitoring video, and a second background image corresponding to the second target area is obtained; and calculating the pixel difference between each frame of image in the multi-frame images and the second background image, determining that the second object enters the second target area when the pixel difference in each frame of image is greater than or equal to the image corresponding to the preset pixel difference threshold value, and determining that no second object enters the second target area when the pixel difference in each frame of image is less than the pixel difference threshold value. And the second background image is an image acquired by the second monitoring device when the second object is not in the second target area.
In one embodiment, multiple frames of images are obtained from the second monitoring video, and each frame of image in the multiple frames of images is input into a preset target detection model to be processed, so that a target detection result of each frame of image is obtained; and when the target detection result corresponding to each frame of image does not comprise the head-shoulder detection frame or the face detection frame, determining that no second object enters the second target area. Whether a second object enters a second target area can be accurately determined through the target detection result of the image, so that the authenticity and the accuracy of the medicine selling event can be improved.
In one embodiment, a plurality of frame images are obtained from the second monitoring video, and a second background image corresponding to the second target area is obtained; calculating the pixel difference between each frame of image in the multi-frame images and the second background image, acquiring an image corresponding to the pixel difference which is greater than or equal to a preset pixel difference threshold value from the multi-frame images to obtain a target image, and inputting the target image into a preset target detection model for processing to obtain a target detection result of the target image; and when the target detection result of the target image does not comprise the head and shoulder detection frame or the face detection frame, determining that no second object enters the second target area. By comprehensively considering the target detection result of the target image and the pixel difference between the target image and the background image, the detection accuracy of whether a second object enters the second target area can be further improved, and the authenticity and the accuracy of the medicine selling condition can be improved.
In an embodiment, the manner of determining whether the behavior of the second object in the second target area is a payment behavior according to the second surveillance video may be: determining whether the duration of the second object in the second target area is greater than a preset duration threshold or not according to the second monitoring video; and when the duration that the second object is located in the second target area is determined to be greater than the preset duration threshold, determining that the behavior of the second object in the second target area is payment behavior. The preset time threshold may be set based on an actual situation, which is not specifically limited in the embodiment of the present invention. For example, the preset time threshold is 3 minutes.
In an embodiment, the manner of determining whether the behavior of the second object in the second target area is the payment behavior according to the second surveillance video may be: inputting a plurality of frames of images in the second monitoring video into the payment behavior recognition model for processing to obtain action behavior labels of the frames of images; counting the number of images corresponding to the payment behavior labels of the action behavior labels to obtain the number of target images; determining the percentage of the number of the target images in the total number of the multi-frame images in the second monitoring video; and when the percentage is greater than or equal to a preset percentage threshold value, determining that the behavior of the second object in the second target area is a payment behavior. By comprehensively considering the action behavior tags of the multi-frame images, the interference caused by the error identification of the payment behavior tags can be reduced, and the identification accuracy of the behavior of the second object in the second target area is improved.
In the embodiment of the invention, the payment behavior recognition model is obtained by performing iterative training on the neural network model according to a plurality of positive example training samples and a plurality of negative example training samples, the positive example training samples comprise positive example sample images and labeled payment behavior labels, the action behaviors of objects in the positive example sample images are code scanning payment by using electronic equipment, the negative example training samples comprise negative example sample images and labeled non-payment behavior labels, and the action behaviors of the objects in the negative example sample images are not code scanning payment action by using the electronic equipment, such as simple action of playing the electronic equipment or action of self-shooting by using the electronic equipment.
And step S104, generating payment behavior information when the behavior of the second object in the second target area is determined to be the payment behavior.
In the embodiment of the present invention, the payment behavior information is used to indicate that the behavior of the second object in the second target area is a payment behavior, the payment behavior information includes a payment behavior identifier, or the payment behavior information includes the payment behavior identifier and identity feature information of the second object, the identity feature information of the second object may be obtained by the second monitoring device by identifying the second object, and the identity feature information may include face feature information and head and shoulder feature information.
And S105, when the generated payment behavior information is inquired within the preset time length for generating the medicine taking behavior information, generating a medicine selling event.
In the embodiment of the present invention, the preset duration may be set based on an actual situation, which is not specifically limited in the embodiment of the present invention. For example, the preset time duration is 5 minutes, so that when the generated payment behavior information is queried within 5 minutes of the generation of the medicine taking behavior information, a medicine selling event is generated, that is, within 5 minutes after the generation of the medicine taking behavior information, if the payment behavior information is generated, the medicine selling event is generated.
In an embodiment, when a new drug registration record is not inquired to be generated within a preset time length of a generated drug selling event, a third monitoring video acquired in a preset time period for the generated drug selling event is acquired, wherein the third monitoring video comprises a monitoring video acquired by monitoring a first target area and a monitoring video acquired by monitoring a second target area; and acquiring an illegal verification result aiming at the third monitoring video, and outputting prompt information of whether the shop assistant has illegal behaviors or not based on the illegal verification result. When a new medicine registration record is generated within the preset time length of the medicine selling event, a clerk in the pharmacy is suspected of illegally selling the medicines, and whether the clerk has an illegal behavior can be further judged by acquiring an illegal checking result aiming at the third monitoring video, so that the detection efficiency and the accuracy of whether the clerk illegally sells the medicines are greatly improved.
When a customer purchases a medicine which needs to be registered and can be sold, such as a prescription medicine, an antitussive medicine, an antipyretic, an antiviral medicine, an antibiotic medicine and the like, the customer needs to input medicine registration information through a mobile phone code scanning mode, the mobile phone uploads the medicine registration information to a server through a network, and the server generates a medicine registration record based on the received medicine registration information and stores the medicine registration record when receiving the medicine registration information. The drug registration information may include the customer's identification number, contact phone number, drug category of the drug purchased, etc.
It can be understood that the violation checking result may be input by the checker by viewing the third monitoring video, or may be obtained by analyzing the third monitoring video, which is not specifically limited in this embodiment of the present invention. And when the violation checking result indicates that the clerk does not have the violation, outputting a second prompt message to the associated electronic equipment, wherein the second prompt message is used for prompting that the clerk does not have the violation.
In an embodiment, the manner of analyzing the third monitoring video to obtain the violation checking result may be: under the condition that the behavior of the first object in the first target area is determined to be a medicine taking behavior and the behavior of the second object in the second target area is determined to be a payment behavior according to the third monitoring video, a video clip containing the first target area is obtained from the third monitoring video, and character recognition and character information are carried out on each frame of image in the video clip; under the condition that the text information contains at least one preset keyword, determining that the violation checking result is that a store employee has a violation; and under the condition that the text information does not contain the preset keywords, determining that the violation checking result is that the shop assistant does not violate the rule. The preset keywords may include prescription drugs, antitussives, antipyretics, antiviral drugs, antibiotic drugs, and the like.
In one embodiment, under the condition that the generated payment behavior information is inquired within a preset time length of the generation of the medicine taking behavior information, calculating a first similarity between the body characteristic information of the first object and the preset body characteristic information; and when the first similarity is greater than or equal to a preset similarity threshold, generating a medicine selling event. The body characteristic information can include face characteristic information and/or head and shoulder characteristic information, and the preset body characteristic information is body characteristic information of a pharmacist which is input in advance. Through generating getting it filled within a long time to inquire under the circumstances of the payment action information that generates in predetermineeing of action information, further based on the health characteristic information of first object and predetermine the similarity between the health characteristic information, judge whether pharmacist of pharmacy has the medicine of selling, can improve the judgement accuracy of medicine selling event to further improve the authenticity and the accuracy of the medicine selling condition of pharmacy.
In one embodiment, under the condition that the generated payment behavior information is inquired within a preset time length for generating the medicine taking behavior information, calculating a first similarity between the body characteristic information of the first object and the preset body characteristic information; when the first similarity is smaller than a preset similarity threshold value, calculating a second similarity between the body characteristic information of the first object and the body characteristic information of the second object; and when the second similarity is greater than or equal to the similarity threshold, generating a medicine selling event. When the first similarity is smaller than the preset similarity threshold value, the object for getting the medicine can be determined to be a customer, and whether the customer for getting the medicine and paying the medicine can be determined to be the same through the second similarity between the body characteristic information of the first object and the body characteristic information of the second object, so that the customer can get the medicine and pay the medicine by oneself, and the authenticity and the accuracy of the medicine selling condition of a pharmacy can be improved.
It can be understood that the medicine storage area can comprise a first medicine storage area and/or a second medicine storage area, the first medicine storage area is used for storing medicines which need to be registered and can be sold, the second area is used for storing medicines which can be sold without being registered, a customer cannot go to the first medicine storage area to take medicines, a clerk of a pharmacy can go to the first medicine storage area to take or put medicines, and both the customer and the clerk of the pharmacy can go to the second medicine storage area to take or put medicines.
For example, under the scenario that the first medicine storage area is within the monitoring range of the first monitoring device and the medicine checkout area is within the monitoring range of the second monitoring device, the first monitoring device can monitor the head-shoulder and hand states of a clerk of the pharmacy in the first medicine storage area, so that the first monitoring device can track the clerk through the head-shoulder of the clerk and identify whether the behavior of the clerk of the pharmacy in the first medicine storage area is a medicine taking behavior through the hand state of the clerk, and when the behavior of the clerk of the pharmacy in the first medicine storage area is a medicine taking behavior, the first monitoring device sends a medicine taking behavior identifier to the server. The second monitoring device can monitor whether the action of the customer in the medicine checkout area is a payment action, and when the second monitoring device identifies that the action of the customer in the medicine checkout area is a payment action, the second monitoring device sends a payment action identifier to the server, so that the server can determine whether a medicine selling event is generated through the medicine taking action identifier sent by the first monitoring device and the payment action identifier sent by the second monitoring device, and therefore when the medicine selling event is determined to be generated, a medicine selling monitoring record can be generated and stored.
For another example, in a scenario where the second medicine storage area is within the monitoring range of the first monitoring device and the medicine checkout area is within the monitoring range of the second monitoring device, the first monitoring device can monitor the head, face and hand states of the clerk or customer of the pharmacy in the second medicine storage area, so that the first monitoring device can track the clerk through the head or face of the clerk or customer, and identify whether the behavior of the clerk or customer in the second medicine storage area is a medicine taking behavior through the hand state of the clerk or customer, and when the behavior of the clerk or customer in the second medicine storage area is identified as a medicine taking behavior, the first monitoring device sends a medicine taking behavior identifier and face feature information of the clerk or customer to the server.
The second monitoring device can monitor whether the action of a customer in a drug checkout area is a payment action, and when the second monitoring device recognizes that the action of the customer in the drug checkout area is a payment action, the second monitoring device sends a payment action identifier and face feature information of the customer to the server, so that the server can determine whether a drug selling event is generated through the drug taking action identifier sent by the first monitoring device, the face feature information, the payment action identifier sent by the second monitoring device and the face feature information, namely when the face feature information sent by the first monitoring device is matched with face feature information of a pre-recorded salesman, the server can determine that the drug is taken by the salesman, when the face feature information sent by the first monitoring device is matched with the face feature information sent by the second monitoring device, the server can determine that the drug is taken by the customer, when the drug is taken by the customer or the salesman, the server can determine that one drug selling event is generated, and generate and store a drug monitoring record.
It is understood that steps S101 and S102 may be performed by a first monitoring device, steps S103 and S104 may be performed by a second monitoring device, and step S105 may be performed by a server. Or, the first monitoring device sends the first monitoring video to the server, the second monitoring device sends the second monitoring device to the server, and steps S101 to S105 are all executed by the server. The execution subject of steps S101 to S105 is not particularly limited in the embodiment of the present invention.
According to the medicine selling monitoring method provided by the embodiment, when it is monitored that the first object enters the medicine storage area, whether the behavior of the first object in the medicine storage area is the medicine taking behavior is monitored, so that the medicine taking behavior information is generated when the behavior of the first object in the medicine storage area is monitored to be the medicine taking behavior, and meanwhile, when it is monitored that the second object enters the medicine checkout area, whether the behavior of the second object in the medicine checkout area is the payment behavior is monitored, so that the payment behavior information is generated when the behavior of the second object in the medicine checkout area is monitored to be the payment behavior, so that when the generated payment behavior information is inquired within the preset time length of the generated medicine taking behavior information, a medicine event is generated, and staff in a pharmacy cannot falsely sell medicines, and the authenticity and the accuracy of the medicine selling condition are greatly improved.
Referring to fig. 8, fig. 8 is a block diagram schematically illustrating a structure of a medicine vending monitoring system according to an embodiment of the present invention.
As shown in FIG. 8, drug vending monitoring system 400 includes a first monitoring device 410, a second monitoring device 420, and a server 430, each of first monitoring device 410 and second monitoring device 420 communicatively coupled to server 430.
In an embodiment, the first monitoring device 410 is configured to monitor a first target area, so as to obtain a first monitoring video, where the first target area includes a medicine storage area; when the first target area is determined to have a first object entering according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior or not according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information and sending the medicine taking behavior information to the server, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior;
the second monitoring device 420 is configured to monitor a second target area to obtain a second monitoring video, where the second target area includes a drug checkout area; when a second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information and sending the payment behavior information to the server, wherein the payment behavior information is used for indicating that the behavior of the second object in the drug checkout area is the payment behavior;
the server 430 is configured to determine whether the payment behavior information sent by the second monitoring device is received within a preset time period when the medicine taking behavior information is received; and generating a medicine selling event when the payment behavior information sent by the second monitoring equipment is received within a preset time length.
In an embodiment, the first monitoring device 410 is configured to monitor a first target area, obtain a first monitoring video, and send the first monitoring video to a server, where the first target area includes a medicine storage area;
the second monitoring device 420 is configured to monitor a second target area, obtain a second monitoring video, and send the second monitoring video to a server, where the second target area includes a drug checkout area;
the server 430 is configured to obtain a first monitoring video sent by the first monitoring device and a second monitoring video sent by the second monitoring device; when it is determined that a first object enters the first target area according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; when a second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be payment behavior, generating payment behavior information, wherein the payment behavior information is used for indicating that the behavior of the second object in the second target area is payment behavior; and when the generated payment behavior information is inquired within the preset time length for generating the medicine taking behavior information, generating a medicine selling event.
In an embodiment, the second monitoring device 420 or the server 430 is further configured to determine, according to the second monitoring video, whether a duration that the second object is located in the second target area is greater than a preset duration threshold; determining that the behavior of the second object in the second target area is a payment behavior when the duration is determined to be greater than a preset duration threshold.
In an embodiment, the server 430 is further configured to, when generation of a new drug registration record is not queried within a preset time period for generating the drug selling event, acquire a third monitoring video acquired within a preset time period for the generated drug selling event, where the third monitoring video includes a monitoring video obtained by monitoring the first target area and a monitoring video obtained by monitoring the second target area; and acquiring an illegal checking result aiming at the third monitoring video, and outputting prompt information whether a clerk has an illegal action or not based on the illegal checking result.
In an embodiment, the first monitoring device 410 or the server 430 is further configured to input multiple frames of images in the first monitoring video into a preset target detection model for processing, so as to obtain a target detection result of each frame of the images in the multiple frames of images; tracking the first object according to the target detection result of each frame of the image; when the first object is tracked to leave the first target area, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the target detection result of each frame of the image.
In an embodiment, the multiple frames of images at least include a first image and a second image adjacent to the first image, and the first monitoring device 410 or the server 430 is further configured to calculate an intersection-parallel ratio between a head-shoulder detection frame of the first object in the first image and a head-shoulder detection frame of each object to be detected in the second image; determining the maximum intersection ratio in the intersection ratios as a target intersection ratio, and determining whether the target intersection ratio is greater than or equal to a preset intersection ratio threshold value; and when the target cross-over ratio is greater than or equal to a preset cross-over ratio threshold value, marking the target cross-over ratio corresponding to the object to be detected in the second image as the first object.
In an embodiment, the target detection result includes a hand state label of the first object, and the first monitoring device 410 or the server 430 is further configured to count the number of images corresponding to the hand state label as a medicine placing state label to obtain a first image number, where the medicine placing state label describes that the hand of the first object is in a medicine placing state; determining a percentage of the first number of images to the total number of the plurality of frame images; when the percentage is greater than or equal to a preset percentage threshold, determining that the behavior of the first object in the first target area is a drug taking behavior.
In an embodiment, the target detection result includes a hand state tag of the first object, and the first monitoring device 410 or the server 430 is further configured to determine a first target image, where the first target image is an image corresponding to a medicine placing state tag, where the hand state tag detected for the first time in the multiple frames of images is an image corresponding to the medicine placing state tag, and the medicine placing state tag describes that a hand of the first object is in a medicine placing state; counting the number of second target images corresponding to the hand state label and the medicine placing state label, wherein the second target images are images behind the first target images in the multi-frame images; adding 1 to the number of the second target images to obtain the number of second images; when the second number of images is greater than or equal to a preset number of images threshold, determining that the behavior of the first object in the first target area is a drug taking behavior.
In an embodiment, the medicine taking behavior information includes a medicine taking behavior identifier and body feature information of the first object, the body feature information includes at least one of face feature information and head-shoulder feature information, and the server 430 is further configured to calculate a first similarity between the body feature information of the first object and preset body feature information; and when the first similarity is greater than or equal to a preset similarity threshold value, generating a medicine selling event.
In an embodiment, the payment behavior information includes a payment behavior identifier and body feature information of the second object, and the server 430 is further configured to calculate a second similarity between the body feature information of the first object and the body feature information of the second object when the first similarity is smaller than a preset similarity threshold; and when the second similarity is greater than or equal to the similarity threshold, generating a medicine selling event.
It should be noted that, as will be clearly understood by those skilled in the art, for convenience and simplicity of description, in the specific working process of the medicine selling monitoring system described above, reference may be made to the corresponding process in the foregoing medicine selling monitoring method embodiment, and details are not described herein again.
Embodiments of the present invention also provide a storage medium for a computer-readable storage, where the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement any one of the medicine sales monitoring methods provided in the description of the embodiments of the present invention.
The storage medium may be an internal storage unit of the first monitoring device, the second monitoring device, or the server described in the foregoing embodiment, for example, a hard disk or a memory of the first monitoring device, the second monitoring device, or the server. The storage medium may also be an external storage device of the first monitoring device, the second monitoring device, or the server, for example, a plug-in hard disk provided on the first monitoring device, the second monitoring device, or the server, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, or suitable combinations thereof. In a hardware embodiment, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
It should be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or system comprising the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (11)

1. A medicine sale monitoring method is characterized by comprising the following steps:
when a first object enters a first target area according to a first monitoring video corresponding to the first target area, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video, wherein the first target area comprises a medicine storage area;
when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior;
when a second object enters a second target area according to a second monitoring video corresponding to the second target area, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video, wherein the second target area comprises a medicine checkout area;
when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information, wherein the payment behavior information is used for indicating that the behavior of the second object in the second target area is a payment behavior;
and when the generated payment behavior information is inquired within the preset time length for generating the medicine taking behavior information, generating a medicine selling event.
2. The method for monitoring the sale of drugs according to claim 1, wherein the determining whether the second object's behavior in the second target area is payment behavior according to the second monitoring video comprises:
determining whether the time length of the second object in the second target area is greater than a preset time length threshold value or not according to the second monitoring video;
and when the duration is determined to be greater than a preset duration threshold, determining that the behavior of the second object in the second target area is a payment behavior.
3. The medication sale monitoring method of claim 1, wherein after generating the medication sale event, further comprising:
when new drug registration records are not inquired and generated within a preset time length for generating the drug selling event, acquiring a third monitoring video acquired in a preset time period aiming at the generated drug selling event, wherein the third monitoring video comprises a monitoring video obtained by monitoring the first target area and a monitoring video obtained by monitoring the second target area;
and acquiring an illegal checking result aiming at the third monitoring video, and outputting prompt information whether the shop assistant has illegal behaviors or not based on the illegal checking result.
4. A method for monitoring the sale of medicines according to any one of claims 1-3, wherein the determining whether the first object's behavior in the first target area is a medicine taking behavior according to the first monitoring video comprises:
inputting the multi-frame images in the first monitoring video into a preset target detection model for processing to obtain target detection results of all frames of images in the multi-frame images;
tracking the first object according to the target detection result of each frame of the image;
when the first object is tracked to leave the first target area, whether the behavior of the first object in the first target area is a medicine taking behavior is determined according to the target detection result of each frame of image.
5. The method of claim 4, wherein the plurality of frames of images include at least a first image and a second image adjacent to the first image, and wherein tracking the first object according to the target detection result of each frame of image comprises:
calculating the intersection ratio between the head and shoulder detection frame of the first object in the first image and the head and shoulder detection frame of each object to be detected in the second image;
determining the maximum intersection ratio in the intersection ratios as a target intersection ratio, and determining whether the target intersection ratio is greater than or equal to a preset intersection ratio threshold value;
when the target cross-over ratio is greater than or equal to a preset cross-over ratio threshold, marking the target cross-over ratio corresponding to the object to be detected in the second image as the first object.
6. The medicine vending monitoring method of claim 4, wherein the target detection result comprises a hand state label of the first object, and the determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the target detection result of each frame of the image comprises:
counting the number of images corresponding to the hand state label as a medicine placing state label to obtain a first image number, wherein the medicine placing state label describes that the hand of the first object is in a medicine placing state;
determining a percentage of the first number of images to the total number of images of the plurality of frames;
when the percentage is greater than or equal to a preset percentage threshold, determining that the behavior of the first object in the first target area is a drug-taking behavior.
7. The method for monitoring medicine vending of claim 4, wherein the target detection result comprises a hand state label of the first object, and the determining whether the behavior of the first object in the first target area is the medicine taking behavior according to the target detection result of each frame of the image comprises:
determining a first target image, wherein the first target image is an image corresponding to a medicine placing state label, which is detected in the multi-frame image for the first time, and the medicine placing state label describes that a hand of the first object is in a medicine placing state;
counting the number of second target images corresponding to the hand state label and the medicine placing state label, wherein the second target images are images behind the first target images in the multi-frame images;
adding 1 to the number of the second target images to obtain the number of second images;
when the second number of images is greater than or equal to a preset number of images threshold, determining that the behavior of the first object in the first target area is a drug taking behavior.
8. A medication vending monitoring method according to any one of claims 1-3, wherein the medication dispensing behavior information includes a medication dispensing behavior identifier and body characteristic information of the first object, the body characteristic information including at least one of face characteristic information and head-shoulder characteristic information, the medication vending monitoring method further comprising:
calculating a first similarity between the body characteristic information of the first object and preset body characteristic information;
and when the first similarity is greater than or equal to a preset similarity threshold value, generating a medicine selling event.
9. The medication vending monitoring method of claim 7, wherein the payment behavior information includes a payment behavior identifier and physical characteristic information of the second object, and after calculating the first similarity between the physical characteristic information of the first object and preset physical characteristic information, the method further comprises:
when the first similarity is smaller than a preset similarity threshold value, calculating a second similarity between the body characteristic information of the first object and the body characteristic information of the second object;
and when the second similarity is greater than or equal to the similarity threshold, generating a medicine selling event.
10. The medicine selling monitoring system is characterized by comprising a first monitoring device, a second monitoring device and a server, wherein the first monitoring device and the second monitoring device are in communication connection with the server respectively;
the first monitoring device is used for monitoring a first target area to obtain a first monitoring video, and the first target area comprises a medicine storage area; when it is determined that a first object enters the first target area according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information and sending the medicine taking behavior information to the server, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior;
the second monitoring device is used for monitoring a second target area to obtain a second monitoring video, and the second target area comprises a medicine checkout area; when a second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information and sending the payment behavior information to the server, wherein the payment behavior information is used for indicating that the behavior of the second object in the drug checkout area is the payment behavior;
the server is used for determining whether the payment behavior information sent by the second monitoring equipment is received within a preset time length or not when the medicine taking behavior information is received; and generating a medicine selling event when the payment behavior information sent by the second monitoring equipment is received within a preset time length.
11. The medicine selling monitoring system is characterized by comprising a first monitoring device, a second monitoring device and a server, wherein the first monitoring device and the second monitoring device are in communication connection with the server respectively;
the first monitoring device is used for monitoring a first target area to obtain a first monitoring video and sending the first monitoring video to the server, wherein the first target area comprises a medicine storage area;
the second monitoring device is used for monitoring a second target area to obtain a second monitoring video and sending the second monitoring video to the server, wherein the second target area comprises a medicine checkout area;
the server is used for acquiring a first monitoring video sent by the first monitoring equipment and a second monitoring video sent by the second monitoring equipment; when it is determined that a first object enters the first target area according to the first monitoring video, determining whether the behavior of the first object in the first target area is a medicine taking behavior according to the first monitoring video; when the behavior of the first object in the first target area is determined to be a medicine taking behavior, generating medicine taking behavior information, wherein the medicine taking behavior information is used for indicating that the behavior of the first object in the first target area is the medicine taking behavior; when a second object enters the second target area according to the second monitoring video, determining whether the behavior of the second object in the second target area is a payment behavior according to the second monitoring video; when the behavior of the second object in the second target area is determined to be a payment behavior, generating payment behavior information, wherein the payment behavior information is used for indicating that the behavior of the second object in the second target area is a payment behavior; and when the generated payment behavior information is inquired within the preset time length of the medicine taking behavior information, generating a medicine selling event.
CN202211051299.4A 2022-08-30 2022-08-30 Medicine sale monitoring method and system Pending CN115620186A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211051299.4A CN115620186A (en) 2022-08-30 2022-08-30 Medicine sale monitoring method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211051299.4A CN115620186A (en) 2022-08-30 2022-08-30 Medicine sale monitoring method and system

Publications (1)

Publication Number Publication Date
CN115620186A true CN115620186A (en) 2023-01-17

Family

ID=84856902

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211051299.4A Pending CN115620186A (en) 2022-08-30 2022-08-30 Medicine sale monitoring method and system

Country Status (1)

Country Link
CN (1) CN115620186A (en)

Similar Documents

Publication Publication Date Title
US10410278B2 (en) Method and apparatus for integrated image capture for vehicles to track damage
CN109920174B (en) Book borrowing method and device, electronic equipment and storage medium
US8117071B1 (en) Method and system for matching via an image search query at a point of sale
US20100023400A1 (en) Image Recognition Authentication and Advertising System
US20080140432A1 (en) Verification and data-tracking systems, methods, and devices
WO2016070138A1 (en) Identification scan in compliance with jurisdictional or other rules
US9569938B1 (en) Video content analysis using point of sale data
JP2012208854A (en) Action history management system and action history management method
CN111476559A (en) Merchant authentication method and device, computer equipment and storage medium
US20130328765A1 (en) Signage system and display method by the same
CN112788269B (en) Video processing method, device, server and storage medium
CN109685580A (en) A kind of member management method and system based on recognition of face
JP2015090579A (en) Behavior analysis system
CN115600249A (en) Meta-universe e-commerce shopping big data security protection method and system based on block chain
CN111612657A (en) Client type identification method and device, electronic equipment and storage medium
FR2877122A1 (en) SYSTEM AND METHOD FOR AUTHENTICATING AN OBJECT
CN115222427A (en) Artificial intelligence-based fraud risk identification method and related equipment
WO2015003287A1 (en) Behavior recognition and tracking system and operation method therefor
US20220180463A1 (en) Systems, apparatus, and methods for integrating and streamlining the process of issuing citations while simultaneously enhancing security of law enforcement officers (leos)
WO2017032056A1 (en) Point-of-sale-based cash-out determining method and apparatus
CN108171286B (en) Unmanned selling method and system
CN111191995A (en) Method for realizing store intelligent monitoring and member-to-store intelligent reminding based on Internet platform
CN115620186A (en) Medicine sale monitoring method and system
CN111540106A (en) Unmanned supermarket system
FR3067846B1 (en) ANTI-THEFT MANAGEMENT SYSTEM AND ASSOCIATED METHOD

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination