CN116824175A - Medicine sorting and checking method and device, storage medium and electronic equipment - Google Patents
Medicine sorting and checking method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN116824175A CN116824175A CN202310130148.6A CN202310130148A CN116824175A CN 116824175 A CN116824175 A CN 116824175A CN 202310130148 A CN202310130148 A CN 202310130148A CN 116824175 A CN116824175 A CN 116824175A
- Authority
- CN
- China
- Prior art keywords
- image
- medicine
- target
- information
- identified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000003814 drug Substances 0.000 title claims abstract description 397
- 229940079593 drug Drugs 0.000 title claims abstract description 180
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000001514 detection method Methods 0.000 claims description 27
- 238000012549 training Methods 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 15
- 238000007689 inspection Methods 0.000 claims description 8
- 238000012795 verification Methods 0.000 claims description 5
- 230000002194 synthesizing effect Effects 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims description 2
- 239000000825 pharmaceutical preparation Substances 0.000 claims 1
- 229940127557 pharmaceutical product Drugs 0.000 claims 1
- 230000005540 biological transmission Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 9
- 230000006872 improvement Effects 0.000 description 8
- 238000006073 displacement reaction Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000007599 discharging Methods 0.000 description 3
- 238000012377 drug delivery Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004806 packaging method and process Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010998 test method Methods 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
According to the drug sorting and checking method provided by the specification, corresponding drugs are sorted according to the received orders, images of the drugs in the transmission process are obtained, image features and text information of the drugs in the images are extracted, the extracted image features and text information are matched with those of standard drugs, the types of the sorted drugs are determined, whether the sorted drugs are consistent with the drugs on the orders or not is judged, and drug sorting accuracy is improved.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method and apparatus for sorting and inspecting medicines, a storage medium, and an electronic device.
Background
With the development of internet technology, intelligent pharmacy is gradually rising, users can utilize electronic equipment to independently place an order according to requirements, the order is submitted to a platform system of the intelligent pharmacy, medicines on the order are automatically sorted onto a conveying belt after the order is received, the sorted medicines are conveyed to a container and packaged by the conveying belt, and finally, the packaged medicines are conveyed to the hands of the users by a rider, so that great convenience is brought to life.
However, the sorting work of the medicines is completed by the machine, whether the sorted medicines are completely consistent with the medicines on the order cannot be determined, and in the face of the situation, it is necessary to check the sorted medicines before packaging to verify whether the sorted medicines are completely consistent with the medicines on the order. Therefore, the application provides a medicine sorting and checking method so as to achieve the aim of checking the accuracy of sorted medicines.
Disclosure of Invention
Embodiments of the present disclosure provide a drug sorting inspection method, apparatus, storage medium, and electronic device to at least partially solve the above-mentioned problems.
The embodiment of the specification adopts the following technical scheme:
the present specification provides a method of drug sorting inspection, the method comprising:
receiving an order, and sorting corresponding target medicines from an automatic medicine taking device onto a conveyor belt according to the order so as to convey the target medicines to a container through the conveyor belt;
in the process of conveying the target medicine by the conveyor belt, acquiring each image of the target medicine conveyed by the conveyor belt to form an image sequence;
extracting information to be identified of a target medicine in each image in the image sequence, wherein the information to be identified comprises at least one of image characteristics and text information;
Matching the extracted information to be identified with standard information of standard medicines to determine standard medicines matched with the target medicines in the image;
and verifying whether the sorted target medicine is consistent with the medicine required by the order according to the standard medicine matched with the target medicine in the image.
Optionally, before extracting the information to be identified of the target drug in the image, the method further includes:
tracking a target medicine contained in each frame of image in the image sequence;
for any two adjacent images in the image sequence, judging whether the target medicine contained in the next image in the two images appears in the previous image according to the tracking result of the target medicine contained in the previous image in the two images;
if not, updating the number of the detected target medicines, and verifying whether the number of the updated target medicines is consistent with the number of the medicines on the order.
Optionally, tracking the target drug in each image in the image sequence specifically includes:
aiming at any two adjacent images in the image sequence, taking each target medicine contained in the next image in the two images as a medicine to be determined;
For each pending medicine, determining the displacement of the position of the pending medicine contained in the next frame of image in the two frames of images relative to the position of the pending medicine in the previous frame of image;
translating a bounding box of the undetermined medicine in the next frame of image according to the displacement;
determining the superposition area of the translated bounding box and the bounding box of the undetermined medicine in the previous frame image;
if the overlapping area exceeds a first preset threshold value, determining that the undetermined medicines contained in the two frames of images are the same medicine, otherwise, determining that the undetermined medicines contained in the two frames of images are not the same medicine.
Optionally, matching the extracted information to be identified with standard information of standard medicines specifically includes:
when the extracted information to be identified is an image feature, matching the image feature of the specified pixel point in the information to be identified with the image feature of the standard medicine to determine each specified pixel point matched with the image feature of the standard medicine in the information to be identified as each matched pixel point;
determining the corresponding area of each matched pixel point in the image;
judging whether the sum of areas corresponding to the matched pixel points is larger than a second preset threshold value or not;
If yes, the image features of the information to be identified and the standard medicine are successfully matched.
Optionally, matching the extracted information to be identified with standard information of standard medicines specifically includes:
when the extracted information to be identified is text information, judging whether the text information of the standard medicine is contained in the information to be identified, and judging whether the text information of the standard medicine is contained in the information to be identified;
if the information to be identified contains text information of the standard medicine or the text information of the standard medicine contains the information to be identified, the information to be identified and the text information of the standard medicine are determined to be successfully matched.
Optionally, before determining whether the sum of the areas corresponding to the matched pixel points is greater than the second preset threshold, the method further includes:
obtaining historically stored similarities;
determining a second preset threshold according to the acquired similarity;
after the information to be identified is successfully matched with the image features of the standard medicine, the method further comprises the following steps:
and determining the similarity of the information to be identified and the image characteristics of the standard medicine, and storing the similarity.
Optionally, determining a second preset threshold according to the acquired similarity specifically includes:
Screening the acquired similarity meeting the specified conditions;
and determining a second preset threshold according to the average value of the screened similarity.
Optionally, extracting information to be identified of the target drug in the image specifically includes:
inputting the image into a pre-trained rotating target detection network model to obtain a bounding box of a target medicine in the image output by the rotating target detection network model;
extracting information to be identified of the target medicine in the image according to the bounding box;
pre-training a rotating target detection network model, specifically comprising:
acquiring an image of a sample medicine and a sorting background image of the sample medicine in a sorting process;
synthesizing the image of the sample medicine with the sorting background image to obtain a training sample, and determining a label corresponding to the training sample according to the position of the image of the sample medicine in the training sample synthesized in the sorting background image;
inputting the training sample into the rotating target detection model to obtain an output result;
and adjusting model parameters of the rotating target detection model according to the output result and the labeling.
The present specification also provides a drug sorting inspection device, the device comprising:
The receiving module is used for receiving the order and sorting corresponding target medicines from the automatic medicine taking device onto the conveyor belt according to the order so as to convey the target medicines to the container through the conveyor belt;
the acquisition module is used for acquiring each image of the target medicine transmitted on the conveyor belt in the process of transmitting the target medicine by the conveyor belt to form an image sequence;
the extraction module is used for extracting information to be identified of a target medicine in each image in the image sequence, wherein the information to be identified comprises at least one of image characteristics and text information;
the matching module is used for matching the extracted information to be identified with the standard information of the standard medicine so as to determine the standard medicine matched with the target medicine in the image;
and the verification module is used for verifying whether the sorted target medicines are consistent with the medicines required by the order according to the standard medicines matched with the target medicines in the image.
A computer readable storage medium is provided in the present specification, the storage medium storing a computer program which, when executed by a processor, implements the above-described drug sort test method.
The electronic equipment provided by the specification comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor realizes the medicine sorting and checking method when executing the program.
The above-mentioned at least one technical scheme that this description embodiment adopted can reach following beneficial effect:
according to the drug sorting and checking method provided by the specification, corresponding drugs are sorted according to the received orders, images of the drugs in the transmission process are obtained, image features and text information of the drugs in the images are extracted, the extracted image features and text information are matched with those of standard drugs, the types of the sorted drugs are determined, whether the sorted drugs are consistent with the drugs on the orders or not is judged, and drug sorting accuracy is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a schematic flow chart of a drug sorting inspection method according to an embodiment of the present disclosure;
FIG. 2 is a schematic illustration of a drug sorting process provided herein;
fig. 3 is a schematic view of an image acquired by the image acquisition device according to the embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a rotation target detection network model output image according to an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a training rotating object detection network model according to an embodiment of the present disclosure;
fig. 6 is a schematic diagram of a drug sorting inspection device according to an embodiment of the present disclosure;
fig. 7 is a schematic view of an electronic device corresponding to fig. 1 provided in an embodiment of the present disclosure;
fig. 8 is a schematic view of a dispensing machine according to an embodiment of the present disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art without the exercise of inventive faculty, are intended to be within the scope of the application, based on the embodiments in the specification.
The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of a drug sorting and checking method provided in the present specification, specifically including the following steps:
s100: and receiving an order, and sorting corresponding target medicines from the automatic medicine taking device onto a conveyor belt according to the order so as to convey the target medicines to the container medicines through the conveyor belt.
The execution subject of the drug sorting test provided in the present specification may be a server, or may be an electronic device such as a personal computer (Personal Computer, PC) or a mobile phone, and for convenience of description, the method of model training provided in the present specification will be described below with only the server as the execution subject.
In this embodiment of the present disclosure, when the server receives the order, the server may determine the identifier of the target drug carried in the order first, generate a sorting instruction carrying the identifier of the target drug, and send the sorting instruction to the drug delivery machine, where after receiving the sorting instruction, the drug delivery machine sorts the corresponding drug according to the identifier of the target drug carried in the sorting instruction, where the type of the target drug may be one type or may be multiple types, and according to the specific situation of the received order, it is determined that the number of types of the target drug is not limited in this disclosure.
In the embodiment of the present specification, as shown in fig. 8, the dispensing machine (800) includes a medicine storage area (801) and a conveying area (802), the medicine storage area (801) is used for storing various medicines, the conveying area (802) is used for conveying target medicines, and the output area (803) is used for outputting the sorted target medicines from the interior of the dispensing machine to a container. When the medicine dispensing machine (800) receives a sorting instruction sent by the server, the medicine dispensing machine sorts corresponding target medicines from the medicine storage area (801) according to the sorting instruction, the target medicines are placed on a conveying belt of the conveying area (802) through the medicine taking device (804), and when the conveyed target medicines are determined to be sorted without errors, the target medicines are conveyed to the output area (803) and are output to be packaged in a container.
S101: and in the process of conveying the target medicine by the conveyor belt, acquiring each image of the target medicine conveyed by the conveyor belt to form an image sequence.
In this embodiment of the present disclosure, as shown in fig. 2, when the drug dispensing machine sorts the target drugs, the sorted target drugs may be sequentially placed on the conveyor belt, and the drug dispensing machine includes image capturing devices (such as cameras, etc.) for capturing the drugs transferred on the conveyor belt, which are located on both sides of the conveyor belt (the image capturing devices are not shown in fig. 8). The medicine dispensing machine can instruct the image acquisition device to acquire images on the conveyor belt when receiving sorting instructions sent by the server, the image acquisition device starts to acquire the images, when the medicine dispensing machine sorts corresponding medicines on the conveyor belt according to the sorting instructions, the medicine dispensing machine can wait according to preset time and then send instructions for stopping photographing to the image acquisition device, and when the medicine dispensing machine sends instructions for stopping photographing to the image acquisition device, the medicine on the conveyor belt is discharged; the method can also send unloading completion information to the server after the conveyor belt is unloaded, and send a photographing stopping instruction to the image acquisition device after the server receives the unloading completion information, wherein the judgment of the unloading completion of the conveyor belt can be determined according to a sensor or preset rotation time, and the preset time can be predetermined according to the speed and the length of the conveyor belt in the medicine dispensing machine.
For example, the preset time is 8 seconds, after the medicine dispensing machine sorts all corresponding medicines onto the conveyor belt according to the sorting instruction, the medicine dispensing machine waits for 8 seconds, after 8 seconds, the medicine dispensing machine instructs the image acquisition device to stop photographing, the image acquisition device stops photographing after receiving the instruction, or the medicine dispensing machine sorts all corresponding medicines onto the conveyor belt according to the sorting instruction, the medicine is discharged after 8 seconds of conveying, after the discharging is completed, the conveyor belt sends information of the discharging completion to the server, and after the server receives the information of the discharging completion, the server sends an instruction of stopping photographing to the image acquisition device. As shown in fig. 3, fig. 3 is one of a sequence of images including a conveyor belt for conveying sorted medicines, and target medicines a, b, and c.
In the embodiment of the present disclosure, the collected images are formed into an image sequence instead of collecting a single image containing all the target medicines, because when the number of the target medicines is large, the single image may cause that each target medicine is unclear and thus it is inconvenient to extract the characteristics of each target medicine, and after the image sequence is formed, all the target medicines are not necessarily constrained on the same image, so that the target medicines are clearer and the characteristics of each target medicine are convenient to extract.
S102: and extracting information to be identified of the target medicine in each image in the image sequence, wherein the information to be identified comprises at least one medicine in image characteristics and text information.
The image shown in fig. 3 is input into the rotating target detection network model, so that bounding boxes of each target medicine can be obtained, and the output result is shown in fig. 4, and a dashed box in fig. 4, which respectively encloses the target medicine a, the target medicine b and the target medicine c, is the bounding box output by the rotating target detection network model.
S103: and extracting information to be identified of the target medicine in each image in the image sequence, wherein the information to be identified comprises at least one of image characteristics and text information.
In the embodiment of the present disclosure, the information to be identified of the target drug may be directly extracted according to the acquired image, in order to improve the effect of extracting the information to be identified, the acquired image may be input into the rotating target detection network model, the bounding box of the target drug in the image may be output by the model, and then the information to be identified may be extracted for the target drug in the bounding box, so that the information irrelevant to the information to be identified, such as the image background and the conveyor belt in the image, may be eliminated, and the accuracy of extracting the information to be identified is improved.
In this embodiment of the present disclosure, the extracted information to be identified may be an image feature or a text message, when the extracted information to be identified is an image feature, the image feature may be matched with an image feature of a standard drug, and when the extracted information to be identified is a text message, the text message is matched with a text message of the standard drug.
Specifically, for each standard medicine stored in advance, the image similarity between the target medicine and the standard medicine can be determined according to the image feature to be identified extracted from the target medicine and the image feature of the standard medicine, the text similarity between the target medicine and the standard medicine is determined according to the text information to be identified extracted from the target medicine and the text information of the standard medicine, the image similarity is weighted with the text similarity, the weighted result is compared with a preset similarity threshold, if the weighted result is higher than the preset similarity threshold, the target medicine and the standard medicine are determined to be successfully matched, and if the weighted result is lower than the preset similarity threshold, the target medicine and the standard medicine are determined to be failed to be matched.
The standard medicine matched with the target medicine can be determined according to the fact that the image feature to be identified extracted from the target medicine is matched with the image feature of each standard medicine, meanwhile, the standard medicine matched with the target medicine is determined according to the fact that the text information to be identified extracted from the target medicine is matched with the text information of each standard medicine, if the matching result determined through the image feature matching is inconsistent with the matching result determined through the text information, the standard medicine determined according to the image feature can be used as the standard medicine matched with the target medicine, the standard medicine determined according to the text information can be used as the standard medicine matched with the target medicine, the image similarity determined through the image feature matching can be compared with the text similarity determined through the text information, and the standard medicine matched with the target medicine can be determined according to the matching result with higher similarity.
S104: and verifying whether the sorted target medicine is consistent with the medicine required by the order according to the standard medicine matched with the target medicine in the image.
When the determined standard medicine matched with the target medicine in the image is consistent with the medicine required in the order, the error-free sorting can be determined, and the sorted target medicine is output from the medicine dispensing machine to the container for packaging.
When the determined standard medicine matched with the target medicine in the image is inconsistent with the medicine required by the order, the sorting is likely to be wrong, at the moment, whether the sorted target medicine is consistent with the medicine required by the order or not can be checked again by checking the image sequence manually, and if yes, the sorted medicine is output from the medicine dispensing machine to a container for packaging; if not, the server sends a sorting instruction to the medicine dispensing machine again, and the medicine dispensing machine re-sorts the medicine with wrong sorting in the order after receiving the sorting instruction.
In the embodiment of the specification, the verification of the target medicine, whether the sorted target medicine is consistent with the medicine required in the order, is completed in the process of conveying the target medicine by the driving belt.
According to the drug sorting and checking method, corresponding target drugs are sorted on the conveyor belt of the drug dispensing machine according to the received orders, the target drugs on the conveyor belt are photographed, an image sequence is obtained, image features and text information of the target drugs contained in images in the image sequence are extracted, the extracted image features and text information are respectively matched with the image features and the text information of standard drugs, standard drugs matched with the sorted target drugs are determined, whether the sorted target drugs are completely consistent with the drugs on the orders or not is further judged, and accuracy in the drug sorting process is improved.
In step S102 shown in fig. 1, it is necessary to determine bounding boxes of each target drug in the image by rotating the target detection network model, so that training of the model needs to be completed in advance, and a specific training method may be: the method comprises the steps of obtaining a sorting background image of a sample medicine in a sorting process, specifically, as shown in fig. 5, the sorting background image does not contain any sample medicine, synthesizing the determined sorting background image of the sample medicine and an image of the sample medicine, taking the synthesized image as a training sample, inputting the training sample into the rotating target detection network model to obtain an output result, determining a label corresponding to the training sample according to the position of the image of the sample medicine in the training sample, comparing the output result with the label, and iteratively adjusting parameters of the rotating target detection network model, so that the training purpose is achieved, and the trained rotating target detection network model can accurately output a bounding box of the target medicine in the input image. The label may be a bounding box of the sample medicine, or may be an image of the sample medicine itself. When the label is the bounding box of the sample medicine, the position of the bounding box of the sample medicine in the image output by the rotating target detection network model can be compared with the position of the label, the position difference of the bounding box of the sample medicine and the position difference of the bounding box of the sample medicine can be determined, the larger the position difference is, the larger the loss is, and the parameters of the rotating target detection network model are adjusted according to the loss. When the label is the image of the sample medicine, comparing the image defined by the enclosed frame in the image output by the rotating target detection network model with the label, determining the similarity of the image defined by the enclosed frame and the label, and adjusting the parameters of the rotating target detection network model according to the loss, wherein the smaller the similarity is, the larger the loss is.
In step S103 shown in fig. 1, the image features may be matched with the image features of the standard medicine, specifically, the pixels in the image features are screened, the screened pixels are used as designated pixels, the image features of the designated pixels are matched with the image features of the standard medicine, the designated pixels successfully matched with the image features of the standard medicine are used as matched pixels, the areas of the areas corresponding to all the matched pixels are added, wherein the area corresponding to the matched pixels is an area with a designated size centered on the matched pixels, and if the addition result is greater than a second preset threshold, it is determined that the to-be-identified information is successfully matched with the standard information.
The second preset threshold may be a preset fixed value or a dynamic value, and when the second preset threshold is a dynamic value, the similarity stored in history may be obtained before determining whether the sum of the areas corresponding to the matched pixel points is greater than the second preset threshold, and the second preset threshold may be determined according to the obtained similarity.
Specifically, the historically stored similarity may be used as the similarity to be determined, the similarity satisfying the specified condition may be selected, where the similarity satisfying the specified condition refers to the similarity falling into the specified range, the specified range may be a range preset according to a specific situation, the mean value and standard deviation of each similarity to be determined according to the mean value and the standard deviation, for example, the mean value is 17, the standard deviation is 0.5, 17± (2×0.5) is used as the specified range, the similarity to be determined may be further sorted in the order from small to large or from large to small, the first n and/or the last m similarities of the sorted similarity to be removed, and the remaining similarities are used as the similarities satisfying the specified condition, that is, the range corresponding to the remaining similarities is determined as the specified range. And finally, determining the average value of the similarity meeting the specified condition, and determining a second preset threshold value according to the average value, wherein the values of n and m can be preset according to specific conditions. If the proportion of the similarity meeting the specified condition to all the undetermined similarities does not exceed the preset proportion, the similarity meeting the specified condition is regarded as the undetermined similarity again until the proportion of the similarity meeting the specified condition to the undetermined similarity exceeds the preset proportion; if the proportion of the similarity meeting the specified condition to the undetermined similarity exceeds the preset proportion, determining the average value of the similarity meeting the specified condition, and determining a second preset threshold value according to the average value.
When the second preset threshold is determined according to the average value, the product of the average value and the preset value can be used as the second preset threshold, wherein the preset value is preset according to specific conditions.
Judging whether the information to be identified is successfully matched with the standard information according to the determined second preset threshold value, if so, determining the similarity between the information to be identified and the image characteristics of the standard medicine after the information to be identified is successfully matched with the image characteristics of the standard medicine, and storing the determined similarity for determining the dynamic second preset threshold value next time.
When the extracted information to be identified is text information, the text information and the text information of the standard medicine can be converted into vectors, text similarity can be determined according to Euclidean distance between the vectors, if the text similarity is higher than a preset text similarity threshold, the text information of the standard medicine and the information to be identified are successfully matched, and if the text similarity is lower than the preset text similarity threshold, the text information of the standard medicine and the information to be identified are unsuccessfully matched.
And whether the text information of the standard medicine is contained in the information to be identified or not or whether the text information of the standard medicine is contained in the information to be identified or not can be judged, if the text information of the standard medicine is contained in the information to be identified or the text information of the standard medicine is contained in the information to be identified, the matching of the information to be identified and the text information of the standard medicine is determined to be successful, otherwise, the matching of the information to be identified and the text information of the standard medicine is failed.
In a practical application scenario, since the photographing height is reduced and the photographing field is reduced to ensure that the photographed pictures of the target medicines are sufficiently clear, the number of target medicines included in each image in the image sequence may be different, for example, medicines included in image 1 are a and b, and medicines included in image 2 are a, b and c. At this time, in order to improve the matching efficiency of the information to be identified and the standard information of the standard medicine, before step S103 shown in fig. 1, the target medicine included in each image in the image sequence needs to be tracked, the number of the target medicines is updated by tracking the target medicines, when the tracking of all the target medicines is completed, whether the number of the target medicines is consistent with the number of medicines required by the order is judged, if yes, the information to be identified of the extracted target medicines can be matched with the standard information of the standard medicine continuously through step S103; if not, determining that the sorting is wrong, and not needing to be matched.
The method for tracking the target medicine contained in each image in the image sequence specifically includes taking each target medicine contained in a later frame of image as a to-be-determined medicine according to the shooting sequence of the images in the image sequence, calculating the displacement of the position of the to-be-determined medicine contained in the later frame of image relative to the position of the to-be-determined medicine in the earlier frame of image according to each to-be-determined medicine, moving the bounding box of the to-be-determined medicine in the later frame of image according to the calculated displacement result, determining the overlapping area of the translated bounding box and the bounding box of the to-be-determined medicine in the earlier frame of image, and if the overlapping area exceeds a first preset threshold, knowing that the number of the target medicines is not changed and updating is not needed; if the overlapping area does not exceed the first preset threshold, it can be known that the number of the target medicines is changed, and at this time, the number of the sorted target medicines needs to be updated, wherein the first preset threshold can be preset according to specific conditions of the medicines.
In this embodiment of the present disclosure, after the image sequence is acquired, each drug image in the image sequence may be further screened to screen out an image containing an incomplete drug, and the remaining complete drug images are stored in a standard drug library, where the standard drug library is used to store standard information of standard drugs, so as to implement updating of the standard drug library.
The drug sorting inspection method provided above for one or more embodiments of the present specification also provides a corresponding drug sorting inspection device based on the same thought, as shown in fig. 6.
Fig. 6 is a schematic diagram of a drug sorting and inspecting apparatus provided in the present specification, specifically including:
a receiving module 601, configured to receive an order, and sort a corresponding target drug from an automatic drug delivery device onto a conveyor belt according to the order, so as to convey the target drug to a container via the conveyor belt;
an acquisition module 602, configured to acquire images of the target drug conveyed on the conveyor belt during the process of conveying the target drug by the conveyor belt, and form an image sequence;
an extracting module 603, configured to extract, for each image in the image sequence, information to be identified of a target drug in the image, where the information to be identified includes at least one of image features and text information;
a matching module 604, configured to match the extracted information to be identified with standard information of a standard drug, so as to determine a standard drug that matches with the target drug in the image;
a verification module 605 is configured to verify whether the sorted target drug is consistent with the drug required by the order according to the standard drug matched with the target drug in the image.
Optionally, the apparatus further comprises:
the tracking module 606 is configured to track a target drug contained in each frame of image in the image sequence; for any two adjacent images in the image sequence, judging whether the target medicine contained in the next image in the two images appears in the previous image according to the tracking result of the target medicine contained in the previous image in the two images; if not, the number of the detected target medicines is updated, and whether the number of the updated target medicines is consistent with the number of the medicines on the order is verified.
Optionally, the tracking module 606 is specifically configured to take, as the undetermined drug, each target drug included in a next frame image of the two frame images for any two adjacent frame images in the image sequence; for each pending medicine, determining the displacement of the position of the pending medicine contained in the next frame of image in the two frames of images relative to the position of the pending medicine in the previous frame of image; translating a bounding box of the undetermined medicine in the next frame of image according to the displacement; determining the superposition area of the translated bounding box and the bounding box of the undetermined medicine in the previous frame image; if the overlapping area exceeds a first preset threshold value, determining that the undetermined medicines contained in the two frames of images are the same medicine, otherwise, determining that the undetermined medicines contained in the two frames of images are not the same medicine.
Optionally, the matching module 604 is specifically configured to, when the extracted information to be identified is an image feature, match an image feature of a specified pixel point in the information to be identified with an image feature of a standard drug, so as to determine each specified pixel point in the information to be identified, which is matched with the image feature of the standard drug, as each matched pixel point; determining the corresponding area of each matched pixel point in the image; judging whether the sum of areas corresponding to the matched pixel points is larger than a second preset threshold value or not; if yes, the image features of the information to be identified and the standard medicine are successfully matched.
Optionally, the matching module 604 is specifically configured to determine whether the text information of the standard drug is included in the extracted information to be identified when the extracted information to be identified is text information, and whether the text information of the standard drug is included in the text information of the standard drug; if the information to be identified contains text information of the standard medicine or the text information of the standard medicine contains the information to be identified, the information to be identified and the text information of the standard medicine are determined to be successfully matched.
Optionally, the matching module 604 is further configured to: obtaining historically stored similarities; determining a second preset threshold according to the acquired similarity;
The verification module 605 is further configured to determine a similarity between the information to be identified and the image feature of the standard drug, and store the similarity.
Optionally, the matching module 604 is specifically configured to screen, from the acquired similarities, the similarities that meet the specified condition, where the similarities that meet the specified condition are similarities that fall within a specified range; and determining a second preset threshold according to the average value of the similarity meeting the specified condition.
Optionally, the extracting module 603 is specifically configured to input the image into a pre-trained rotating target detection network model, and obtain a bounding box of the target drug in the image output by the rotating target detection network model; and extracting information to be identified of the target medicine in the image according to the bounding box.
Optionally, the extracting module 603 is specifically configured to obtain an image of the sample medicine and a sorting background image of the sample medicine in the sorting process; synthesizing the image of the sample medicine with the sorting background image to obtain a training sample, and determining a label corresponding to the training sample according to the position of the image of the sample medicine in the training sample synthesized in the sorting background image; inputting the training sample into the rotating target detection model to obtain an output result; and adjusting model parameters of the rotating target detection model according to the output result and the labeling.
The present specification also provides a computer readable storage medium having stored thereon a computer program operable to perform the method of drug sort test provided in fig. 1, described above.
The embodiment of the present specification also proposes a schematic structural diagram of the electronic device shown in fig. 7. At the hardware level, as in fig. 7, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile storage, although it may include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the drug sort test method described above with respect to fig. 1. Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present description, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, medicament or device comprising the element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (10)
1. A method of sorting and testing pharmaceutical products, the method comprising:
receiving an order, and sorting corresponding target medicines from an automatic medicine taking device onto a conveyor belt according to the order so as to convey the target medicines to a container through the conveyor belt;
in the process of conveying the target medicine by the conveyor belt, acquiring each image of the target medicine conveyed by the conveyor belt to form an image sequence;
Extracting information to be identified of a target medicine in each image in the image sequence, wherein the information to be identified comprises at least one of image characteristics and text information;
matching the extracted information to be identified with standard information of standard medicines to determine standard medicines matched with the target medicines in the image;
and verifying whether the sorted target medicine is consistent with the medicine required by the order according to the standard medicine matched with the target medicine in the image.
2. The method of claim 1, wherein prior to extracting information to be identified of the target drug in the image, the method further comprises:
tracking a target commodity contained in each frame of image in the image sequence;
for any two adjacent images in the image sequence, judging whether the target medicine contained in the next image in the two images appears in the previous image according to the tracking result of the target medicine contained in the previous image in the two images;
if not, updating the number of the detected target medicines, and verifying whether the number of the updated target medicines is consistent with the number of the medicines on the order.
3. The method according to claim 1, wherein matching the extracted information to be identified with standard information of standard medicines, specifically comprises:
when the extracted information to be identified is an image feature, matching the image feature of the specified pixel point in the information to be identified with the image feature of the standard medicine to determine each specified pixel point matched with the image feature of the standard medicine in the information to be identified as each matched pixel point;
determining the corresponding area of each matched pixel point in the image;
judging whether the sum of areas corresponding to the matched pixel points is larger than a second preset threshold value or not;
if yes, the image features of the information to be identified and the standard medicine are successfully matched.
4. The method according to claim 1, wherein matching the extracted information to be identified with standard information of standard medicines, specifically comprises:
when the extracted information to be identified is text information, judging whether the text information of the standard medicine is contained in the information to be identified, and judging whether the text information of the standard medicine is contained in the information to be identified;
if the information to be identified contains text information of the standard medicine or the text information of the standard medicine contains the information to be identified, the information to be identified and the text information of the standard medicine are determined to be successfully matched.
5. The method of claim 3, wherein before determining whether the sum of the areas corresponding to the matched pixels is greater than the second preset threshold, the method further comprises:
obtaining historically stored similarities;
determining a second preset threshold according to the acquired similarity;
after the information to be identified is successfully matched with the image features of the standard medicine, the method further comprises the following steps:
and determining the similarity of the information to be identified and the image characteristics of the standard medicine, and storing the similarity.
6. The method of claim 5, wherein determining the second preset threshold according to the obtained similarity comprises:
screening the acquired similarities which meet the specified conditions, wherein the similarities which meet the specified conditions are the similarities which fall into the specified range;
and determining a second preset threshold according to the average value of the similarity meeting the specified condition.
7. The method of claim 1, wherein extracting information to be identified of the target drug in the image specifically comprises:
inputting the image into a pre-trained rotating target detection network model to obtain a bounding box of a target medicine in the image output by the rotating target detection network model;
Extracting information to be identified of the target medicine in the image according to the bounding box;
pre-training a rotating target detection network model, specifically comprising:
acquiring an image of a sample medicine and a sorting background image of the sample medicine in a sorting process;
synthesizing the image of the sample medicine with the sorting background image to obtain a training sample, and determining a label corresponding to the training sample according to the position of the image of the sample medicine in the training sample synthesized in the sorting background image;
inputting the training sample into the rotating target detection model to obtain an output result;
and adjusting model parameters of the rotating target detection model according to the output result and the labeling.
8. A medication sorting inspection device, the device comprising:
the receiving module is used for receiving the order and sorting corresponding target medicines from the automatic medicine taking device onto the conveyor belt according to the order so as to convey the target medicines to the container through the conveyor belt;
the acquisition module is used for acquiring each image of the target medicine transmitted on the conveyor belt in the process of transmitting the target medicine by the conveyor belt to form an image sequence;
The extraction module is used for extracting information to be identified of a target medicine in each image in the image sequence, wherein the information to be identified comprises at least one of image characteristics and text information;
the matching module is used for matching the extracted information to be identified with the standard information of the standard medicine so as to determine the standard medicine matched with the target medicine in the image;
and the verification module is used for verifying whether the sorted target medicines are consistent with the medicines required by the order according to the standard medicines matched with the target medicines in the image.
9. A computer readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of the preceding claims 1-7 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310130148.6A CN116824175A (en) | 2023-02-08 | 2023-02-08 | Medicine sorting and checking method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310130148.6A CN116824175A (en) | 2023-02-08 | 2023-02-08 | Medicine sorting and checking method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116824175A true CN116824175A (en) | 2023-09-29 |
Family
ID=88128040
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310130148.6A Pending CN116824175A (en) | 2023-02-08 | 2023-02-08 | Medicine sorting and checking method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116824175A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117208455A (en) * | 2023-11-06 | 2023-12-12 | 苏州艾隆科技股份有限公司 | Medicine supplementing method and device for medicine supplementing equipment, storage medium and electronic terminal |
| CN117422708A (en) * | 2023-12-04 | 2024-01-19 | 广州方舟信息科技有限公司 | Medicine boxing detection method, device, electronic equipment and storage medium |
| CN117954045B (en) * | 2024-03-27 | 2024-06-04 | 吉林大学 | Automatic drug sorting management system and method based on prescription data analysis |
-
2023
- 2023-02-08 CN CN202310130148.6A patent/CN116824175A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117208455A (en) * | 2023-11-06 | 2023-12-12 | 苏州艾隆科技股份有限公司 | Medicine supplementing method and device for medicine supplementing equipment, storage medium and electronic terminal |
| CN117208455B (en) * | 2023-11-06 | 2024-01-23 | 苏州艾隆科技股份有限公司 | Medicine supplementing method and device for medicine supplementing equipment, storage medium and electronic terminal |
| CN117422708A (en) * | 2023-12-04 | 2024-01-19 | 广州方舟信息科技有限公司 | Medicine boxing detection method, device, electronic equipment and storage medium |
| CN117954045B (en) * | 2024-03-27 | 2024-06-04 | 吉林大学 | Automatic drug sorting management system and method based on prescription data analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116824175A (en) | Medicine sorting and checking method and device, storage medium and electronic equipment | |
| US10824936B2 (en) | Recycling system and method based on deep-learning and computer vision technology | |
| US11049278B2 (en) | System and method for visual identification, and system and method for classifying and sorting | |
| JP5422062B2 (en) | Medicine package inspection system | |
| CN112001456B (en) | Vehicle positioning method and device, storage medium and electronic equipment | |
| CN110827292B (en) | Video instance segmentation method and device based on convolutional neural network | |
| CN112036462A (en) | Method and device for model training and target detection | |
| CN110612090B (en) | Medicine inspection support device and medicine inspection support method | |
| US11775814B1 (en) | Automated detection of controls in computer applications with region based detectors | |
| CN111210412B (en) | Packaging detection method and device, electronic equipment and storage medium | |
| CN112052702A (en) | Method and device for identifying two-dimensional code | |
| US20220019842A1 (en) | Systematic disposal, classification and dynamic procurement of recyclable resin | |
| CN114440903A (en) | High-precision map construction method and device, storage medium and electronic equipment | |
| CN112861831A (en) | Target object identification method and device, storage medium and electronic equipment | |
| CN112834518A (en) | Particle defect detection method, system, device and medium | |
| CN112950564A (en) | Image detection method and device, storage medium and electronic equipment | |
| CN112712009A (en) | Method and device for detecting obstacle | |
| CN114926437A (en) | Image quality evaluation method and device | |
| CN114255435A (en) | Method and device for detecting abnormality of transport device, electronic apparatus, and storage medium | |
| CN112734851A (en) | Pose determination method and device | |
| CN117506919B (en) | Hand-eye calibration method and device, terminal equipment and storage medium | |
| CN116246774A (en) | Classification method, device and equipment based on information fusion | |
| CN111447366B (en) | Transportation method, transportation device, electronic device, and computer-readable storage medium | |
| CN109509225A (en) | Detection method and device, storage medium and the processor of information | |
| CN117114038A (en) | Automatic checking method and system for medicine packaging machine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |