CN112164439A

CN112164439A - Medicine taking monitoring method and device

Info

Publication number: CN112164439A
Application number: CN202010895777.4A
Authority: CN
Inventors: 张志艺
Original assignee: Xiamen Chenli Technology Co ltd
Current assignee: Xiamen Chenli Technology Co ltd
Priority date: 2020-08-31
Filing date: 2020-08-31
Publication date: 2021-01-01
Anticipated expiration: 2040-08-31
Also published as: CN112164439B

Abstract

A medicine taking monitoring method and a device thereof comprise the following steps: 1) acquiring image information of a medicine taking person in real time, and performing limb detection on the image information to obtain key point positions; 2) judging whether one hand of a medicine taking person performs a specified action or not according to the position of the key point, if so, controlling the medicine discharging machine to discharge medicine, entering the step 3), and if not, judging that the medicine discharging machine is abnormal and entering the step 5); 3) judging whether the other hand of the medicine taking person performs the medicine taking action or not according to the position of the key point, judging whether one hand of the medicine taking person keeps the specified action or not, if so, entering the step 4), otherwise, judging that the other hand is abnormal and entering the step 5); 4) judging whether the medicine taking is finished or not, if so, finishing; if not, judging the abnormality and entering the step 5); 5) and (6) exception handling. The invention monitors, identifies and tracks the whole medicine taking process through intelligent monitoring and self-service medicine taking, realizes the standardization of the medicine taking process and effectively reduces the labor cost.

Description

Medicine taking monitoring method and device

Technical Field

The invention relates to the field of machine vision, in particular to a medicine taking monitoring method and a medicine taking monitoring device.

Background

Methadone, also known as methadone hydrochloride, methadone, and amirone, is an opioid receptor agonist, is an narcotic analgesic, and is a morphine-substituting narcotic analgesic synthesized in germany during world war ii. The drug has the drug effect of treating heroin dependence detoxification and replacing maintenance therapy in the early 60 s of the 20 th century, is a drug adopted by 'maintenance therapy of drug in opioid addicts community' in China at present, and is used for detoxification therapy of drug addicts.

At present, each link in the methadone clinic is very strict in supervision flow, and mainly comprises taking medicine from a storehouse, applying for a patient, taking medicine by the patient and the like. The storehouse is got the medicine and is included: morning 8 per day: 30 hours, the warehouse door is opened by the outpatient medicine distribution department staff, the security guard and the office warehouse management staff together, and the outpatient medicine distribution department staff and the warehouse management staff unlock respectively; taking methadone by storehouse management personnel according to the needs, and signing and confirming the staff at the outpatient department and the storehouse management personnel on a methadone warehouse-out registration book respectively; the security and outpatient department staff carry the medicines out of the warehouse, and the warehouse manager confirms that no one exists in the warehouse, and then locks the door; security and outpatient department staff carry the drugs to the outpatient department, and the security and outpatient department staff verify the drugs and sign on the drug use register.

The medicine taking process of a patient comprises the following steps: doctor's prescription → data entry, get therapeutic dose card → medicine room get medicine → see clothes, and require drinking more than one cup of drinking water → waiting room observe, receive health education → confirm no adverse reaction after leaving by oneself.

The above processes need manual treatment and are very complicated, and particularly in the drug taking process, errors are easy to occur, so that self-service drug administration equipment is available in the market, and the self-service drug administration equipment is provided with a verification system, an IC card reader, a drug supply device, a control host and the like, wherein the verification system is used for verifying the identity of a drug-dropping person, the IC card reader is used for reading an IC card held by the drug-dropping person, the drug supply device is used for supplying drugs to the drug-dropping person, and the drug supply device comprises a drug cup dispensing assembly for dispensing a drug cup, a conveying assembly capable of conveying the drug cup, a drug solution storage container for storing drug solution, and a drug solution peristaltic pump for pumping the drug solution in the drug solution storage container into the drug cup for the drug-dropping person to obtain the drug solution from a drug solution supply port. However, the self-service administration equipment needs to carry an IC card, and cannot monitor whether the administration personnel take the liquid medicine on the spot, which has a monitoring leak.

Disclosure of Invention

The invention mainly aims to overcome the defects in the prior art, and provides a medicine taking monitoring method and a medicine taking monitoring device, which can intelligently monitor the whole medicine taking process, effectively reduce the labor cost and improve the supervision level.

The invention adopts the following technical scheme:

the medicine taking monitoring method is characterized by comprising the following steps:

1) acquiring image information of a medicine taking person in real time, and performing limb detection on the image information to obtain key point positions;

2) judging whether one hand of a medicine taking person performs a specified action or not according to the position of the key point, if so, controlling the medicine discharging machine to discharge medicine, entering the step 3), and if not, judging that the medicine discharging machine is abnormal and entering the step 5);

3) judging whether the other hand of the medicine taking person performs the medicine taking action or not according to the position of the key point, judging whether one hand of the medicine taking person keeps the specified action or not, if so, entering the step 4), otherwise, judging that the other hand is abnormal and entering the step 5);

4) judging whether the medicine taking is finished or not, if so, finishing; if not, judging the abnormality and entering the step 5);

5) and (6) exception handling.

Preferably, in the step 1), the acquired image information of the medicine taking person includes image information of at least two directions.

Preferably, the key point positions in step 1) include one or more combinations of wrist positions, elbow positions, shoulder positions, neck positions and facial five-sense organ positions.

Preferably, the step 1) further comprises judging whether the number of people taking medicine is greater than 0 according to limb detection, and if not, repeating the step; if yes, judging whether the number of medicine taking personnel is equal to 1, if not, judging that multiple persons are abnormal and entering the step 5), and if yes, acquiring the positions of key points and entering the step 2).

Preferably, in the step 2) or the step 3), the designating operation is to keep one of the hands at the designated position, and the judging whether one of the hands of the administrator performs the designating operation according to the key point position is to judge whether the wrist of the administrator is within a preset area of the designated position according to the key point position.

Preferably, in the step 2), after the medicine is discharged from the medicine discharging machine, whether the medicine liquid is detected in the medicine cup is also judged, if not, the medicine discharging is judged to be abnormal, the process enters the step 5), and if yes, the process enters the step 4).

Preferably, in the step 3), the medicine taking action comprises holding a cup and delivering the cup to the mouth; whether the cup is held or not is judged by judging whether the liquid medicine is detected at the wrist position or not, and whether the medicine cup is delivered to the mouth or not is judged by judging whether the positions of key points meet collinear conditions and angle conditions or not.

Preferably, the collinear condition means that the positions of a plurality of key points are on the same straight line; the angle condition is that the angle change of the included angle between the wrist position and one end of the edge is changed from positive to negative by taking the connecting line of the positions of two key points as the edge.

Preferably, in the step 4), the mouth region is first defined by the key point position, and then it is determined whether the liquid medicine in the medicine cup in the mouth region is present or absent, and if so, the medicine taking is completed.

Preferably, whether the medicine cup has the medicine liquid is judged by detecting whether the specific color target exists in the medicine cup.

Preferably, in the step 4), after the medicine is taken, whether the medicine cup has the medicine liquid left is detected, if so, the step 5) is performed, and if not, the step is finished.

Preferably, in the step 4), whether the medicine taking personnel lifts the medicine cup to a specified position is judged, and then whether the medicine cup has residual medicine liquid is detected.

A medicine taking monitoring device is characterized by comprising

The image acquisition module is used for acquiring image information of a medicine taking person in real time;

the limb detection module is used for carrying out limb detection on the image information to obtain the position of a key point;

the medicine taking judgment module judges whether one hand of a medicine taking person performs a specified action and the other hand performs a medicine taking action according to the key point position, judges whether the medicine taking is finished and sends abnormal information;

and the exception module is used for executing exception handling according to the exception information.

As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following advantages:

1. according to the method, whether one hand of a medicine taking person executes an appointed action or not is sequentially judged through the key point position, whether the other hand of the medicine taking person executes the medicine taking action or not is judged, whether the medicine taking is finished or not is judged, the whole medicine taking process is monitored, identified and tracked through intelligent monitoring and non-contact self-service medicine taking, the medicine taking process is standardized, and the labor cost is effectively reduced.

2. According to the method, the positions of the key points are detected and identified through limbs, the positions comprise one or more combinations of wrist positions, elbow positions, shoulder positions, neck positions and facial five-sense-organ positions, and the like, and different functional identification requirements can be realized through different position combinations.

3. According to the method, the acquired image information of the medicine taking personnel comprises the image information in at least two directions, so that the user, the liquid medicine and the like can be conveniently identified through different angles, and the identification result is more accurate.

4. According to the method, the number of the medicine taking personnel is judged through limb detection, so that the problem that the supervision is loophole due to interference of a medicine taking process caused by multi-person operation is avoided.

5. According to the method, a medicine taking person keeps one hand in the specified position or makes a stroke with the specified gesture, so that the hand cannot interfere with the medicine taking process, the hand is fixed in a non-contact manner, and the method is safer and more convenient to operate.

6. The method judges whether to hold the cup or not by action decomposition and combining the positions of key points, judges whether to send the medicine cup to the mouth or not by meeting collinear conditions and angle conditions, further judges whether to execute medicine taking action or not, and has simple algorithm and high accuracy.

8. The method of the invention firstly demarcates the mouth area through the key point position, then judges whether the liquid medicine in the medicine cup in the mouth area exists or not, and can accurately distinguish the abnormal conditions existing in the medicine taking process, such as residual liquid medicine in the mouth, liquid medicine pouring in the midway and the like.

Drawings

FIG. 1 is a block diagram of an apparatus according to the present invention;

FIG. 2 is a main flow chart of the method of the present invention;

FIG. 3 is a flowchart of the image acquisition module determination;

FIG. 4 is a limb detection flow chart;

FIG. 5 is a schematic view of limb detection (multi-person);

FIG. 6 is a schematic view of limb detection (single person);

FIG. 7 is a flowchart illustrating a left-hand location determination process;

FIG. 8 is an exemplary diagram of a mirror;

FIG. 9 is a left hand position schematic;

FIG. 10 is a flow chart of the drug discharge machine;

FIG. 11 is a flowchart illustrating a cup holding determination process;

FIG. 12 is a flowchart for detecting the drug solution;

FIG. 13 is a flow chart illustrating the process of identifying the delivery of a cup to the mouth;

FIG. 14 is a schematic view of a cup lifting process;

FIG. 15 is a schematic of a collinear condition and an angular condition;

FIG. 16 is a flowchart illustrating the process of detecting completion of taking medicine;

fig. 17 is a flowchart for detecting whether a drug remains.

The invention is described in further detail below with reference to the figures and specific examples.

Detailed Description

The invention is further described below by means of specific embodiments.

The invention provides a medicine taking monitoring device, which is shown in figure 1 and comprises the following main modules: the device comprises an image acquisition module, a limb detection module, a medicine taking judgment module and an abnormality module. The image acquisition module is used for acquiring the image information of the medicine taking personnel in real time. In order to acquire image information in different directions, at least two cameras can be erected and can be installed at different positions. For example, one of them is a front camera mounted about 60cm in front of the person taking the medicine, with a height of about 1.8m and a viewing angle towards the upper body of the person taking the medicine. The other camera is arranged at a position about 2.4m above the ground and below the medicine taking person, and can acquire pictures in front of and above the medicine taking person in real time. The front camera has the function of a mirror at the same time, and a person taking the medicine can see the action of the person through the display screen. The mirror surface is shown in fig. 8, which is convenient for the medicine taking person to view and execute the designated action in the medicine taking process.

When the module works, whether the camera in front of the module is opened or not is judged, and if not, a prompt for inserting the front camera is sent. On the premise of successfully opening the front camera, whether the upper camera is opened or not is judged, and if not, a prompt for inserting the upper camera is sent. And reading real-time image frames from the two cameras on the premise that the two cameras are opened, and performing image preprocessing, wherein the preprocessing is down-sampling.

The limb detection module is used for carrying out limb detection on the image information to obtain the position of a key point. The medicine taking judgment module is used for judging whether one hand of a medicine taking person performs the specified action or not and whether the other hand performs the medicine taking action or not according to the key point position, judging whether the medicine taking is finished or not and sending abnormal information. And the exception module is used for executing exception handling according to the exception information.

Referring to fig. 2, the present invention further provides a medication monitoring method, which includes the following steps:

1) the image information of the medicine taking personnel is collected in real time, and limb detection is carried out on the image information to obtain the position of a key point.

In this step, the acquired image information of the medicine taking person includes image information of at least two directions, such as two common views, i.e. a front view, a top view, a side view, and the like, preferably, the front view and the top view. In this step, the collected image information is down-sampled, and the obtained processed image includes a front image fim (front image) and an upper image is denoted as uim (upper image).

In the step, the adopted limb action recognition technology is a technology for recognizing and tracking limbs by acquiring human body images through videos, and each limb part of each person in a camera picture can be detected in real time. The key point positions detected by the limbs include various combinations of wrist positions, elbow positions, shoulder positions, neck positions, facial five sense organ positions, wherein the facial five sense organ positions may be at least one of a nose, a mouth, eyes, ears, and eyebrows, see fig. 4.

Specifically, after the front image Fim is used for detecting the limbs, and the number of detected people is 1, the obtained key point positions include that the left wrist (left wrost) position is represented as lw, the left elbow (left elbow) position is represented as le, the left shoulder (left shoulder) position is represented as ls, the right wrist (right wrost) position is represented as rw, the right elbow (right elbow) position is represented as re, the right shoulder (right summer) position is represented as rs, the neck (neck) position is represented as nc, and the nose (nose) position is represented as ns.

Referring to fig. 6, the origin of the coordinate system in the image is set as the upper left corner of the image, and the positive direction of the x axis is towards the right and the positive direction of the y axis is towards the bottom. In the following, the position variable is added with the suffixes of upper and lower coordinates to represent coordinate values in respective directions, such as lw _ x which represents the x coordinate value of the position of the key point of the left wrist.

The method only allows one person to take the medicine alone, and other people cannot appear around the medicine. Therefore, in the step, whether the number of people taking medicine is greater than 0 is judged according to limb detection, and if not, the step is repeated; if yes, judging whether the number of medicine taking personnel is equal to 1, if not, judging that multiple persons are abnormal and entering the step 5), and if yes, acquiring the positions of key points and entering the step 2).

2) And judging whether one hand of the medicine taking personnel performs a specified action or not according to the position of the key point, if so, controlling the medicine discharging machine to discharge medicine, entering the step 3), otherwise, judging the abnormality and entering the step 5). The medicine outlet machine can adopt the existing medicine outlet equipment, and can output medicine cups with quantitative liquid medicine according to commands.

In the invention, one hand is designed to carry out the specified action and is required to hold the cup by the other hand, so that the hand cannot interfere with the medicine taking process and the hand is fixed in a non-contact manner. The designated action is to keep the hand at the designated position, and the judgment of whether one hand of the medicine taking person performs the designated action according to the key point position is to judge whether the wrist of the person is in a preset area of the designated position according to the key point position. According to the habit of most people, the left hand is used to perform a specific action, and the cup is held by the right hand. The specified action is not limited to this, but may be in other ways, such as holding a particular part with the left hand, or swiping a particular gesture with the left hand, etc.

Specifically, referring to fig. 8, a medicine taking person can see a real-time picture acquired by the front camera on a display screen of the medicine dispensing machine.

The key point position required in this step is left wrist position information lw. As shown in fig. 9, the left rectangular area in the figure is a designated position, and the side length is hw, (h _ x, h _ y) represents the position of the center of the designated position in Fim. The setting conditions are as follows: the left wrist key point positions lw _ x and lw _ y are located right below the rectangular area, the center of the preset area is located at the middle point of the lower boundary of the rectangular area, the width is T _ x, the length is T _ y, and then the required meeting conditions are described as follows:

|lw_x-h_x|<T_x，|lw_y-(h_y+hw/2)|<T_y (1)

in this step, a state variable LPC may be introduced indicating whether the left hand is put in place, the LPC being initialized to FALSE, indicating that it is not in the specified position. If the left wrist position lw meets the condition shown in the formula (1), which indicates that the left wrist is located in the preset area, setting the LPC to TRUE, which indicates that the left wrist is located at the designated position.

Referring to fig. 7, if the condition shown in formula (1) is not satisfied, it is further determined whether the left hand is fixed and then removed or not fixed. On the premise that the condition shown in the formula (1) is not satisfied, if the LPC is FALSE, it is indicated that the LPC is not fixed all the time, and at the moment, the next frame of image is directly read for continuous judgment; and if the LPC is TRUE, the left hand is fixed and then moves away, and the left hand is judged to be abnormal in moving away. In the case of no abnormality, after detecting that the left hand is placed at the designated position, a cup discharge request is sent.

Referring to fig. 10, in this step, the medicine discharging machine requests medicine discharging according to a cup discharging request, and after the cup discharging is successful, it is further determined whether medicine liquid is detected in the medicine cup, if not, it is determined that the medicine discharging is abnormal, and the process proceeds to step 5), and if so, the process proceeds to step 3). Specifically, the liquid medicine is detected from the image Uim acquired by the upper camera.

3) And judging whether the other hand of the medicine taking person performs the medicine taking action or not according to the position of the key point, judging whether the left hand keeps the specified action or not, if so, entering a step 4), otherwise, judging that the left hand is abnormal and entering a step 5).

In the step, the medicine taking action comprises holding the cup first and then conveying the medicine cup to the mouth; whether the cup is held or not is judged by judging whether the liquid medicine is detected at the wrist position or not, and whether the medicine cup is delivered to the mouth or not is judged by judging whether the positions of key points meet collinear conditions and angle conditions or not.

Referring to fig. 11, for detecting whether to hold a cup, the image Fim acquired by the front camera is used for identification, the key point position is the right wrist position information rw, the rw is used as a base point to define the position range of the cup held in the right hand, the image corresponding to the position range is captured and recorded as RWim, and then whether a liquid medicine exists in the RWim is detected.

The parameter RWNum is set for recording the times of continuously detecting the liquid medicine by following the wrist. This parameter is reset to 0 after completion of a dosing sequence (including re-entering the dosing sequence after reporting an abnormality). And setting a state parameter RWS, wherein when the RWS is TRUE, the value is initialized to FALSE corresponding to the cup holding the liquid medicine in the right hand. And setting a state parameter RWNS, wherein when the RWNS is TRUE, the value is initialized to FALSE, and the value corresponds to that no liquid medicine exists in the cup which is held by the right hand after the medicine is taken.

If the liquid medicine is detected in the right wrist area, indicating that a cup with the liquid medicine is held in the right hand, the counting parameter RWNum is increased by 1, if the counting parameter RWNum is increased to exceed a threshold value RWDRESH, the RWS is set to TRUE, and the next stage is carried out. The main reason for setting the threshold here is that there is a certain amount of time from the right hand lifting of the cup to the mouth side, which corresponds to the number of frames of the front camera processed in this time.

On the premise that no medical liquid is detected in the right wrist area, there are two possible situations: firstly, the cup is not grabbed by the right hand; secondly, the liquid medicine is taken or poured out, and no liquid medicine exists in the cup. And in the first case, the RWS is FALSE, and in the second case, the RWS is TRUE, so that when the RWS is FALSE, the next frame of image is continuously read for identification, and when the RWS is TRUE, the RWS is set to TRUE, the medicine taking is finished or the medicine is poured out, and the subsequent steps are further used for judging.

For detecting whether the medicine cup is delivered to the mouth, the relative position characteristics between the positions of the key points of the limbs are adopted for identification. The collinear condition means that the positions of a plurality of key points are on the same straight line; the angle condition is that the angle change of the included angle between the wrist position and one end of the edge is changed from positive to negative by taking the connecting line of the positions of two key points as the edge. The image Fim that accessible leading camera acquireed is discerned, and a plurality of key point positions that need use can include right wrist position information rw, right elbow re, right shoulder rs and nose ns etc..

When the right hand lifts the cup to the periphery of the mouth, the wrist, the elbow and the nose are basically in a straight line, and the included angle between the wrist, the elbow and the shoulder is positive or negative. Only if the collinear condition and the angular condition are satisfied is the right-hand cup held near the mouth considered. In order to increase robustness and reduce false detection, a variable LCNum is set, the times of continuously meeting the two conditions are recorded, and the limb condition is considered to be met when a certain number of times is reached. This variable is reset to 0 after completion of a dosing sequence (including re-entering the dosing sequence after reporting an abnormality).

The collinear condition and the angular condition are given below, respectively. The right-hand cup lifting is disassembled as shown in fig. 14, in which (a) is a posture of just lifting the cup, (b) is a posture of lifting the cup to a position adjacent to the mouth, and (c) is a posture of lifting the cup to the mouth.

Collinear rule: note the relative positions of the three key points of the right wrist, elbow and nose, which are not in a straight line when the cup is lifted, and gradually adjust positions as the cup is lifted to the mouth, which are substantially in the same straight line when the cup is lifted to the mouth. This process is shown in fig. 15(a), rw is the right wrist keypoint location, ns is the nose location, re is the right shoulder location, and the arrow direction describes the right wrist keypoint movement direction. The change in the three-point positional relationship can be described by the angle β with the elbow as the apex and the line connecting the wrist and neck as the sides, and when they are collinear, β becomes 0.

The angle change rule is as follows: observing the angle change with the elbow as the vertex and the line connecting the wrist and the shoulder as the side, as shown in fig. 15(b), and specifying the elbow to shoulder side as the reference line and the counterclockwise direction as the positive direction, as the cup is lifted to the mouth side, the angle gradually decreases until it becomes 0 and then becomes negative, i.e., the angle α as shown in fig. 15(b) gradually decreases from positive to 0 and then again becomes negative. The absolute value of the turning negative angle is smaller and does not exceed a certain threshold value.

Based on the above rules, the conditions for lifting the cup to the mouth side by the right hand are provided as follows:

collinear conditions: rw, re and ns are collinear, i.e. β is 0 °, i.e. cos β is 1.

The angle condition is as follows: the elbow is the apex and the angle to the wrist and shoulder connecting the edges changes from positive to negative and is smaller in absolute value, i.e. cos α gradually increases to 1 and then decreases to a value close to 1.

All the above conditions use the angle calculation problem, and the vertex p is given₀Point p on the reference edge₁And another point p₂Then, the cosine of the included angle θ is calculated as follows:

in the invention, whether the medicine cup has the liquid medicine is judged by detecting whether the specific color target exists in the medicine cup, namely whether the specific color target exists is detected in the RGB color image. Referring to the flow shown in FIG. 12: firstly, converting image information into HSV color space from an RGB color model to obtain images H, S and V of three channels, wherein H corresponds to a chrominance channel, S corresponds to a saturation channel, and V corresponds to a luminance channel.

Setting the HSV value range of the target color, wherein the maximum value and the small value of the three components are respectively as follows: h _ M, H _ M, S _ M, S _ M, V _ M, V _ M. For each pixel (x, y), the condition is set:

H_m<H(x,y)<H_M，S_m<S(x,y)<S_M，V_m<V(x,y)<V_M (2)

constructing a binary image B in the following way:

if the condition (2) is satisfied at (x, y), B (x, y) is 1; otherwise, B (x, y) is 0.

Then, an opening operation is performed on the binary image B, wherein the structural element is a circular disc with the radius of 14, and the purpose of the step is to remove small targets and prevent noise interference. And if the binary image after the on operation is a pure black image, indicating that no target exists, otherwise, indicating that the target exists. There are many methods for determining whether a pure black image is present, and a simple method is to add the gray values of all pixels of a binary image, and if the sum is 0, the pure black image is obtained.

the mouth area is firstly demarcated according to the key point position, and then whether the liquid medicine of the medicine cup in the mouth area is from the existence to the nonexistence is judged, if so, the medicine taking is finished.

Specifically, a parameter MNum is set for recording the number of times that the liquid medicine is continuously detected following the mouth. This parameter is reset to 0 after completion of a dosing sequence (including re-entering the dosing sequence after reporting an abnormality). And setting a parameter MNNum for recording the frequency of continuously detecting no liquid medicine along with the mouth. This parameter is reset to 0 after completion of a dosing sequence (including re-entering the dosing sequence after reporting an abnormality). And setting a state parameter MNS, wherein the MNS is that the liquid medicine cannot be detected around the corresponding mouth of the TRUE, and the value is initialized to FALSE. Setting a state parameter MS, initializing to FALSE, setting MS to TRUE when a target is detected around the mouth, and indicating that the liquid medicine is available when the right hand lifts the cup to the mouth.

The step is completed by an image Fim acquired by a front camera, a left wrist position information ns and a neck position information nc are needed to be used, a position range of the cup lifted to the mouth side is defined by taking ns and nc as base points, an image corresponding to the position range is captured and recorded as Mim, then a liquid medicine is detected in Mim, and the flow shown in fig. 12 is adopted for detecting the liquid medicine.

If the liquid medicine is detected, adding 1 to Mnum, if the liquid medicine is continuously detected, namely Mnum is larger than MTHRESH, and the MTHRESH is a preset threshold value, setting MS as TRUE, otherwise, acquiring the next frame of image for continuous detection.

In the case where no liquid medicine is detected, there are two possibilities: firstly, the cup is not lifted to the mouth edge; secondly, the drinking is finished. The first case corresponds to a value of MS FALSE, at which point the next frame of image is acquired for further detection. The second case corresponds to a MS value of TRUE, i.e. indicating that the cup has been lifted around the mouth, MNNum is increased by 1. After the drinking state is detected for many times, namely MNNum is greater than MTHRESH, the MNS is set as TRUE.

In order to prevent the medicine taking personnel from intentionally shielding the medicine liquid when putting the medicine liquid to the mouth, the misjudgment of the completion of the medicine taking is caused, after the medicine taking is completed, the method also comprises the step of detecting whether the medicine liquid is remained in the medicine cup, if so, the step 5 is carried out, and if not, the method is ended. Or whether the medicine taking personnel lift the medicine cup to the designated position is judged firstly, and then whether the medicine cup has residual medicine liquid is detected.

Specifically, referring to fig. 17, state variables RWS, RWNS, MS, and MNS are obtained, and if RWS is false, it indicates that a cup has not been held yet, and a reminder is sent to use the right hand to hold the cup. If the RWS is true, this indicates that the cup has been picked up. When RWNS, MS and MNS are all true, this indicates that the cup is lifted to the mouth and is consumed.

In order to further check whether residues exist or not and avoid malicious shielding, a medicine taking person is required to lift the medicine cup to a specified position, wherein the specific position is a right rectangular area shown in fig. 8. Right wrist position information rw is acquired. The determination method is the same as the determination method in which the left hand is located at the designated position. And if the right hand is not placed at the specified position, sending a prompt that the right hand lifts the cup to the specified position, otherwise, detecting the liquid medicine in the image Uim acquired by the upper camera. If no medicine liquid is detected, indicating no medicine residue, sending a medicine taking completion message. If the liquid medicine is detected, the liquid medicine is judged to be residual, and the liquid medicine is judged to be residual.

And if the RWNS, the MS and the MNS are not true at the same time, the exception occurs, and the step of exception processing is directly carried out to carry out exception processing. The states corresponding to the values of the state variables are shown in table 1.

TABLE 1

5) And (6) exception handling. In this step, exception handling includes terminating a flow, recording, or alerting. Specifically, the exception information and corresponding processing from the foregoing steps are shown in table 2.

Wherein the medicine discharging abnormity comes from the medicine discharging process of the medicine discharging machine in the step 2), and the details are shown in a figure 10. Left-hand positional exceptions result from the left-hand specified action determination process of step 2), see FIG. 7 for details. The multiple person anomaly comes from the limb detection process of step 1), see fig. 5 for details. The drug residue comes from the judgment process of whether the drug taking in the step 4) is finished, and the details are shown in figure 17. The other three exceptions are judged by state variables.

TABLE 2

The above description is only an embodiment of the present invention, but the design concept of the present invention is not limited thereto, and any insubstantial modifications made by using the design concept should fall within the scope of infringing the present invention.

Claims

1. The medicine taking monitoring method is characterized by comprising the following steps:

5) and (6) exception handling.

2. The medication monitoring method according to claim 1, wherein the image information of the person taking the medicine collected in step 1) includes image information of at least two directions.

3. The medication monitoring method of claim 1, wherein the key point positions in step 1) comprise one or more combinations of wrist positions, elbow positions, shoulder positions, neck positions, and facial five-sense-organ positions.

4. The medication monitoring method according to claim 1, wherein the step 1) further comprises determining whether the number of people taking medicine is greater than 0 according to the detection of limbs, and if not, repeating the step; if yes, judging whether the number of medicine taking personnel is equal to 1, if not, judging that multiple persons are abnormal and entering the step 5), and if yes, acquiring the positions of key points and entering the step 2).

5. The medication intake monitoring method according to claim 1, wherein in step 2) or step 3), the action of designating is to keep one of the hands at the designated position, and the judgment of whether one of the hands performs the designated action according to the position of the key point is to judge whether the wrist thereof is within the preset area of the designated position according to the position of the key point.

6. The medication monitoring method according to claim 1, wherein in step 2), after the medicine is discharged from the medicine discharge machine, it is further determined whether or not the medicine liquid is detected in the medicine cup, and if not, it is determined that the medicine discharge is abnormal and the process proceeds to step 5), and if so, the process proceeds to step 4).

7. The medication monitoring method according to claim 1, wherein in step 3), the medication action comprises holding a cup and delivering the cup to the mouth; whether the cup is held or not is judged by judging whether the liquid medicine is detected at the wrist position or not, and whether the medicine cup is delivered to the mouth or not is judged by judging whether the positions of key points meet collinear conditions and angle conditions or not.

8. The medication monitoring method of claim 7, wherein the collinear condition is that the positions of a plurality of key points are on the same straight line; the angle condition is that the angle change of the included angle between the wrist position and one end of the edge is changed from positive to negative by taking the connecting line of the positions of two key points as the edge.

9. The medication monitoring method according to claim 1, wherein in step 4), the mouth area is defined by the key points, and it is determined whether the liquid medicine in the medicine cup in the mouth area is present or absent, and if so, the medication is completed.

10. The medication monitoring method according to claim 6 or 8, wherein the presence of the liquid medicine in the cup is determined by detecting the presence of a specific color target in the cup.

11. The medication monitoring method according to claim 1, wherein the step 4) further comprises detecting whether the medicine cup has residual medicine after the medication is completed, if yes, the step 5 is performed, and if not, the process is ended.

12. The medication monitoring method according to claim 10, wherein in step 4), it is determined whether the person taking the medicine lifts the cup to a predetermined position, and then whether the medicine remains in the cup is detected.

13. A medicine taking monitoring device is characterized by comprising