CN118334639B - Medicine rechecking method and system - Google Patents
Medicine rechecking method and system Download PDFInfo
- Publication number
- CN118334639B CN118334639B CN202410748869.8A CN202410748869A CN118334639B CN 118334639 B CN118334639 B CN 118334639B CN 202410748869 A CN202410748869 A CN 202410748869A CN 118334639 B CN118334639 B CN 118334639B
- Authority
- CN
- China
- Prior art keywords
- data
- character
- recognition
- identification
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000003814 drug Substances 0.000 title claims abstract description 226
- 229940079593 drug Drugs 0.000 title claims abstract description 89
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000009826 distribution Methods 0.000 claims abstract description 41
- 238000012552 review Methods 0.000 claims abstract description 37
- 238000001514 detection method Methods 0.000 claims description 91
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 10
- 238000001914 filtration Methods 0.000 claims description 8
- 238000012216 screening Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000003702 image correction Methods 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims 2
- 238000003860 storage Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 5
- 238000012377 drug delivery Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000012937 correction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 206010010305 Confusional state Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 208000003464 asthenopia Diseases 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000009512 pharmaceutical packaging Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a drug review method and a drug review system, wherein the drug review method comprises the following steps: determining character image data by detecting the image data; analyzing character image data through a preset character recognition model, and determining recognition data of a first character and a second character; calculating a character recognition score of the first recognition data; determining coordinate data of a first area based on a character area of the first character, and determining specification data and color distribution data of the first area; traversing the medicine sample in the database through a preset image recognition model, determining second recognition data, and calculating color recognition scores of the second recognition data; calculating an identification score of the third identification data; and determining the drug identification data by the third identification data with the highest identification score, comparing the drug identification data with the drug data to be taken, and determining the drug rechecking data. According to the invention, the medicine is checked by combining character recognition with color recognition of the area around the recognized character, so that the medicine checking accuracy is improved.
Description
Technical Field
The application relates to the field of medicine detection management, in particular to a medicine rechecking method and system.
Background
The pharmacy and the nurse station are the most frequent workplace of medicine dispensing, and are also service windows for strictly executing the medicine dispensing specification and strictly preventing medicine dispensing errors, the objects of the service are usually ordinary patients, the professional knowledge of the patients is limited, the dispensed medicines cannot be effectively checked, the medicine dispensing errors can directly lead to potential safety hazards of medicine taking, the treatment is failed, and the medicine injury accidents can be caused in serious cases. To prevent drug delivery errors, more advanced and reliable technical measures for drug delivery errors are required.
At present, drug review is basically finished by manpower, such as an outpatient pharmacy, visual fatigue and mental confusion caused by large workload on pharmacists are easy to cause drug delivery errors, such as drug missed delivery, multiple drug delivery, drug misplacement and the like due to similar drug packaging and similar drug name influence.
Therefore, the prior art has defects, and improvement is needed.
Disclosure of Invention
In view of the above problems, the present invention aims to provide a drug review method and system, which can more effectively and more rapidly review drugs and improve the accuracy of drug review.
The first aspect of the invention provides a drug review method, comprising the following steps:
Determining the data of the medicine to be taken according to the medical advice data, and acquiring detection image data according to detection equipment;
performing image recognition on the detection image data to determine character image data of the detection image data;
analyzing the character image data through a preset character recognition model to determine recognition data of a first character and a second character;
determining first identification data according to the identification data of the first character and the second character, and calculating character identification scores of the first identification data;
traversing the character area of the first character, determining coordinate data of the first area, analyzing the detection image data based on the coordinate data of the first area, and determining specification data and color distribution data of the first area;
Inputting the specification data and the color distribution data of the first area into a preset image recognition model, traversing the medicine samples in the database, determining second recognition data of a second area, and calculating the color recognition score of the second recognition data;
Analyzing based on the first identification data and the second identification data, performing weighted calculation on the character identification score and the color identification score of the same identification data, and determining the identification score of the third identification data;
and determining the drug identification data by the third identification data with the highest identification score, comparing the drug identification data with the drug data to be taken, and determining drug rechecking data.
In this aspect, the performing image recognition on the detected image data, determining character image data of the detected image data includes:
Performing image denoising and image correction on the detected image data to obtain preprocessed image data;
performing binarization processing on the preprocessed image data to determine binarized image data;
performing edge extraction on the binarized image data to determine edge characteristics;
and determining character image data of the detected image data by analyzing the edge characteristics.
In this scheme, the analyzing the character image data through a preset character recognition model to determine recognition data of the first character and the second character includes:
comparing the character image data with character samples in a database, and determining the character confidence degrees of the sub-character images and the character samples in the character image data;
Determining a sub-character image with the maximum character confidence coefficient larger than a first preset threshold value as a first character, wherein a character sample corresponding to the maximum character confidence coefficient is identification data of the first character;
and determining the sub-character image with the maximum character confidence coefficient between the first preset threshold value and the second preset threshold value as a second character, and determining a character sample with the character confidence coefficient larger than the second preset threshold value as recognition data of the second character.
In this aspect, the determining the first recognition data according to the recognition data of the first character and the second character, and calculating the character recognition score of the first recognition data includes:
Combining the identification data of the first character and the second character based on the drug information template in the database to determine one or more first identification data;
Accumulating the character confidence coefficient of each character in the first identification data to determine the identification score of each first identification data;
when only one first identification data with the identification score larger than a third preset threshold value exists, determining the first identification data as medicine identification data of detection image data;
and otherwise, filtering the first identification data with the identification score smaller than a fourth preset threshold value, and calculating the character identification score of the residual first identification data according to the character confidence degrees of the first character and the second character.
In this scheme, still include:
the calculation method of the character recognition score of the first recognition data is expressed as:
；
Wherein P 1 is a character recognition score of the first recognition data, P 1(i) is a character confidence of the ith first character in the first recognition data, n is a total number of the first characters in the first recognition data, k 1 is an influence weight of the character confidence of the first character, P 2(j) is a character confidence of the jth second character in the first recognition data, m is a total number of the second characters in the first recognition data, and k 2 is an influence weight of the character confidence of the second character.
In this scheme, the traversing based on the character region of the first character, determining the coordinate data of the first region, analyzing the detected image data based on the coordinate data of the first region, and determining the specification data and the color distribution data of the first region includes:
determining a character area of each character according to the coordinate data of the character;
Randomly generating a traversing point based on the edge of a character area of a first character, traversing clockwise along the edge of the character area of the first character to the periphery, ending traversing when traversing to the edge area of a medicine detection surface or the character areas of other characters except the first character, and determining the traversing terminal point of the current direction;
determining coordinate data of a first area according to the coordinate data of the traversing end point of the character area of the first character in each direction;
Calculating specification data of a first area according to the coordinate data of the first area;
Dividing the detection image data based on the coordinate data of the first area, and determining the first image data of the first area;
analyzing the first image data to determine color distribution data of a first area; the color distribution data includes a color category and a distribution area of each color.
In this scheme, the inputting the specification data and the color distribution data of the first region into a preset image recognition model, traversing the drug sample in the database, determining second recognition data of a second region, and calculating a color recognition score of the second recognition data includes:
screening the medicine samples in the database based on the specification data and the color distribution data of the first area, and filtering the medicine samples with the medicine detection surface smaller than the specification data of the first area;
traversing the medicine detection surface of the residual medicine sample based on the specification data of the first area, and determining a traversed area with the image similarity larger than a fifth preset threshold value as a second area;
Determining a drug sample in which a second region exists as second identification data;
determining a color recognition score of the second recognition data according to the image similarity and the area of each second region in the second recognition data;
The calculation method of the color recognition score of the second recognition data is expressed as the following formula:
；
Wherein p 2 is the color identification score of the second identification data, S (q1)、S(q2) and S (qn) are the image areas of the 1 st, 2 nd and n th second areas in the second identification data, r (q1)、r(q2) and r (qn) are the image similarity of the 1 st, 2 nd and n th second areas in the second identification data, respectively, and S 1 is the image area of the drug detection surface of the second identification data.
In this scheme, still include:
The method for calculating the identification score of the third identification data is expressed as follows:
；
wherein p 3(x) is the recognition score of the third recognition data x, p 1(x) is the character recognition score of the recognition data x, p 2(x) is the color recognition score of the recognition data x, and k p1、kp2 is the influence weight of the character recognition score and the color recognition score, respectively.
In this scheme, still include:
and carrying out auxiliary verification on the medicine identification data through the medicine warehouse-in data.
In a second aspect, the present invention provides a drug review system comprising:
The data acquisition module is used for determining the data of the medicine to be taken through the doctor's advice data and acquiring detection image data through the detection equipment;
the image recognition module is used for carrying out image recognition on the detection image data and determining character image data of the detection image data;
The character recognition module is used for analyzing the character image data through a preset character recognition model and determining recognition data of the first character and the second character; determining first identification data according to the identification data of the first character and the second character, and calculating character identification scores of the first identification data;
The color recognition module is used for traversing the character area of the first character, determining coordinate data of the first area, analyzing the detection image data based on the coordinate data of the first area and determining specification data and color distribution data of the first area; inputting the specification data and the color distribution data of the first area into a preset image recognition model, traversing the medicine samples in the database, determining second recognition data of a second area, and calculating the color recognition score of the second recognition data;
The medicine identification module is used for analyzing the first identification data and the second identification data, carrying out weighted calculation on the character identification score and the color identification score of the same identification data, and determining the identification score of the third identification data;
And the medicine rechecking module is used for determining medicine identification data from the third identification data with the highest identification score, comparing the medicine identification data with the medicine data to be taken, and determining medicine rechecking data.
A third aspect of the present invention provides a computer-readable storage medium having embodied therein a drug review method program which, when executed by a processor, implements the steps of a drug review method as described above.
The invention discloses a drug review method and a drug review system, wherein the drug review method comprises the following steps: determining character image data by detecting the image data; analyzing character image data through a preset character recognition model, and determining recognition data of a first character and a second character; calculating a character recognition score of the first recognition data; determining coordinate data of a first area based on a character area of the first character, and determining specification data and color distribution data of the first area; traversing the medicine sample in the database through a preset image recognition model, determining second recognition data, and calculating color recognition scores of the second recognition data; calculating an identification score of the third identification data; and determining the drug identification data by the third identification data with the highest identification score, comparing the drug identification data with the drug data to be taken, and determining the drug rechecking data. According to the invention, the medicine is checked by combining character recognition with color recognition of the area around the recognized character, so that the medicine checking accuracy is improved.
Drawings
FIG. 1 shows a flow chart of a drug review method provided by the invention;
FIG. 2 is a flow chart showing a method for acquiring identification data of a first character and a second character provided by the invention;
FIG. 3 is a flowchart showing a method for calculating a character recognition score of first recognition data according to the present invention;
fig. 4 shows a block diagram of a drug review system provided by the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will be more clearly understood, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. It should be noted that, without conflict, the embodiments of the present application and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
Fig. 1 shows a flowchart of a drug review method provided by the invention.
As shown in fig. 1, the invention discloses a drug review method, which comprises the following steps:
S102, determining medicine data to be taken through doctor' S advice data, and acquiring detection image data through detection equipment;
s104, carrying out image recognition on the detection image data, and determining character image data of the detection image data;
S106, analyzing the character image data through a preset character recognition model, and determining recognition data of the first character and the second character;
S108, determining first identification data according to the identification data of the first character and the second character, and calculating character identification scores of the first identification data;
s110, traversing the character area based on the first character, determining coordinate data of the first area, analyzing the detection image data based on the coordinate data of the first area, and determining specification data and color distribution data of the first area;
S112, inputting specification data and color distribution data of the first area into a preset image recognition model, traversing a medicine sample in a database, determining second recognition data of a second area, and calculating a color recognition score of the second recognition data;
S114, analyzing the first identification data and the second identification data, performing weighted calculation on the character identification score and the color identification score of the same identification data, and determining the identification score of the third identification data;
s116, determining the drug identification data by the third identification data with the highest identification score, comparing the drug identification data with the drug data to be taken, and determining the drug rechecking data.
According to the embodiment of the invention, the doctor's advice data can be acquired through a plurality of channels, such as doctor information issued by doctors in the associated systems of patient identification card number, doctor's insurance number, hospitalization ID number, bed position and the like. The detection device may be a photographing device such as a GigE industrial camera, and the detection image data of the medicine in a plurality of directions is acquired by a plurality of detection devices provided in the medicine review area.
The method comprises the steps of acquiring historical detection image data of medicines in medicine rechecking and medicine detection processes, training the historical detection image data of the medicines to obtain a preset character recognition model and a preset image recognition model, analyzing the historical detection image data of the medicines, counting medicine recognition data of different medicines, including information such as medicine names, specifications, production lot numbers, shelf lives and the like, establishing corresponding medicine recognition templates, marking each character image, determining character samples, storing the medicine recognition templates and the character samples in a database, and respectively establishing the preset character recognition model and the preset image recognition model.
Image preprocessing such as image denoising and image correction is carried out on the detected image data, gray processing is carried out on the preprocessed image data, a proper gray threshold value is selected for binarization processing, edge extraction is carried out on the binarized image data, and character image data of the detected image data are determined according to edge characteristics.
Due to reasons such as medicine placement and mutual shielding, a complete medicine detection surface cannot be obtained from a single detection direction, detection image data obtained by detection equipment in different directions are converted into the same coordinate system through coordinate conversion, and the detection image data are spliced, so that the complete medicine detection surface is obtained. And meanwhile, the repeated data are subjected to de-duplication treatment, so that the repeated analysis of the same medicine is avoided. The medicine detection surface is detection image data of any one surface of the medicine box, and comprises a front surface, a back surface, a left side surface, a right side surface, an upper surface and a lower surface, detection images of different surfaces can be converted into the same coordinate system through coordinate conversion to be spliced, and the images spliced by the six surfaces are determined to be the medicine detection surface.
Comparing the character image data with character samples in a database based on a preset character recognition model, determining a first character and a second character according to the character confidence degrees of the sub-character images and the character samples in the character image data, and respectively determining the character samples with the character confidence degrees meeting the preset conditions of the system as recognition data of the first character and the second character. Meanwhile, character image data can be recognized by OCR technology, and recognition data of characters can be determined by setting character intervals, character width height ranges, and the like for analysis.
Based on the medicine information templates in the database, the identification data of the first characters and the second characters are combined, one or more pieces of first identification data are determined, and the character identification score of the first identification data is calculated through the character confidence of each character in the first identification data. Traversing based on the character area of the first character, finishing traversing when traversing to the edge area of the medicine detection surface or the character areas of other characters except the first character, determining the traversing end point of the current direction, determining the coordinate data and the first image data of the first area according to the traversing end points of the four directions of the character area of the first character, and determining the specification data and the color distribution data of the first area according to the coordinate data of the first area. Inputting the first image data into a preset image recognition model, screening the medicine samples according to the specification data of the first area, determining the medicine samples with the second area with the color similarity larger than a fifth preset threshold value as second recognition data, and calculating the color recognition score of the second recognition area according to the area of all the second areas in the second recognition area and the relative image similarity. And determining the identification score of the third identification data by carrying out weighted summation on the character identification score and the color identification score of the same identification area, determining the drug identification data by the third identification data with the highest identification score, comparing the drug identification data with the drug data to be taken in the doctor's advice data, and determining the drug review data to finish drug review. And when the drug check fails, restarting the detection equipment to acquire detection image data for analysis, and when the repeated detection times are more than the preset times of the system, carrying out alarm prompt.
In addition, the detection equipment and the detection system which are needed to be used are loaded and initialized when the medicine checking program is started, wherein the loading and the training of the template pictures are included, so that the acquired results can be directly operated by the incoming pictures when photographing and identifying are performed, and the identifying speed is improved; the multi-thread technology is adopted in the drug review process, and multiple drugs can be simultaneously output the identification result, so that the identification speed is improved; the output of the result adopts event registration, namely registering an identification event to finish the processing function, the main interface does not need a direct polling waiting result, the function can be automatically triggered after all identification tools run, the page death waiting can be prevented, and the occupation of a CPU (central processing unit) is reduced; if the medicine rechecking process is finished, the checking completion event is automatically triggered after the medicine is checked, the photographing scanning process is exited, and the main interface does not directly poll whether photographing actions exist.
According to an embodiment of the present invention, performing image recognition on detected image data, determining character image data of the detected image data, includes:
performing image denoising and image correction on the detected image data to obtain preprocessed image data;
Performing binarization processing on the preprocessed image data to determine binarized image data;
performing edge extraction on the binarized image data to determine edge characteristics;
character image data of the detected image data is determined by analyzing the edge features.
It should be noted that, due to the influence of the shooting environment, such as uneven illumination intensity, noise exists in the acquired detection image data, the image quality is reduced, and the subsequent image analysis and recognition are affected, so that the detection image data is subjected to image denoising by a gaussian filtering method and the like. Under the influence of medicine placement, the detected image data can have inclination and other conditions, the detected image data is subjected to inclination correction through a perspective correction method, the coordinate positions of characteristic points such as corner points, edges and boundaries of a medicine detection surface of a medicine box in the detected image data are determined, the inclination direction of an image is detected through algorithms such as Hough circle transformation, a corresponding correction matrix is calculated, affine transformation is carried out on the detected image data, and image correction is completed. The system carries out gray processing on the preprocessed image data, processes different gray values into two gray values with distinct contrast of 0 and 255 by selecting a proper gray threshold value, and respectively represents a character part and a non-character background part in the medicine detection surface to complete binarization processing on the preprocessed image data. And carrying out edge extraction on the binarized image data, and carrying out image segmentation on the binarized image data through edge characteristics to determine character image data of the detection image data.
Fig. 2 shows a flowchart of a method for acquiring identification data of a first character and a second character provided by the invention.
As shown in fig. 2, according to an embodiment of the present invention, analyzing character image data by a preset character recognition model, determining recognition data of a first character and a second character includes:
S202, comparing character image data with character samples in a database, and determining the character confidence of sub-character images and the character samples in the character image data;
S204, determining the sub-character image with the maximum character confidence coefficient larger than a first preset threshold value as a first character, wherein a character sample corresponding to the maximum character confidence coefficient is identification data of the first character;
s206, determining the sub-character image with the maximum character confidence coefficient between the first preset threshold value and the second preset threshold value as a second character, and determining a character sample with the character confidence coefficient larger than the second preset threshold value as recognition data of the second character.
It should be noted that the character image data may be composed of a plurality of character images, the character image data is divided into a plurality of single character sub-character images through a character width height range set by the system, each sub-character image is compared with a character sample in the database, the similarity between the sub-character image and the character sample is calculated, and the recognition result between each sub-character image and the character sample is represented through the character confidence. Wherein, the higher the character confidence is, the higher the recognition accuracy of the sub-character image is. When the maximum character confidence coefficient is larger than a first preset threshold value, the character recognition accuracy meets the preset recognition requirement of the system, a character sample corresponding to the maximum character confidence coefficient is determined to be recognition data of a sub-character image, the sub-character image is determined to be a first character, and a character image recognition result is determined. And when the maximum character confidence is between a first preset threshold value and a second preset threshold value, the final recognition result which cannot be determined as the sub-character image is indicated, the sub-character image is determined as the second character, and a character sample with the character confidence larger than the second preset threshold value is determined as the undetermined recognition result of the sub-character image, namely the recognition data of the second character.
In addition, when a plurality of character confidence degrees larger than a first preset threshold value exist in the sub-character image, the sub-character image is determined to be a second character, and all character samples with the character confidence degrees larger than the second preset threshold value are determined to be identification data of the second character.
Wherein the first preset threshold and the second preset threshold are set by a person skilled in the art according to actual requirements.
Fig. 3 is a flowchart showing a method for calculating a character recognition score of first recognition data according to the present invention.
As shown in fig. 3, according to an embodiment of the present invention, determining first recognition data from recognition data of a first character and a second character, calculating a character recognition score of the first recognition data includes:
S302, based on a medicine information template in a database, combining identification data of the first character and the second character, and determining one or more first identification data;
s304, accumulating the character confidence coefficient of each character in the first identification data, and determining the identification score of each first identification data;
S306, when only one piece of first identification data with the identification score being larger than a third preset threshold value exists, determining the first identification data as medicine identification data of detection image data;
And S308, otherwise, filtering the first recognition data with the recognition score smaller than a fourth preset threshold value, and calculating the character recognition scores of the rest first recognition data according to the character confidence degrees of the first character and the second character.
The medicine information template includes information such as medicine name, specification, production lot number, shelf life, etc., and includes the number of characters, character interval distance, etc. in each item of medicine information, according to the coordinate data of each first character and each second character, the identification data of the first character and the identification data of the second character are combined, the identification data of the first character is a fixed character, the identification data of the second character is a plurality of candidate characters, the identification data corresponding to each coordinate data is determined, and a character string composed of a plurality of characters is generated, so as to obtain the first identification data. In the first recognition data, recognition results of other characters than the first character and the second character are determined by corresponding medicine information templates.
The character confidence coefficient of each character in the first identification data comprises the character confidence coefficient of each character sample in the first character, the second character, other characters and corresponding medicine information templates, the identification score of the first identification data is determined by accumulating the character confidence coefficient of each character and corresponding character sample, when only one first identification data with the identification score being larger than a third preset threshold value exists, the identification data which uniquely meets the system identification requirement is indicated to exist, and the first identification data is output as medicine identification data of the current detection image data; otherwise, if there are a plurality of first identification data with identification scores greater than a third preset threshold value or there are no first identification data with identification scores greater than the third preset threshold value, the character identification score of each first identification data is calculated. The first identification data with the identification score smaller than the fourth preset threshold value can basically judge the medicine identification data which is not the current detection image data, filter the medicine identification data, reduce the data size of subsequent analysis and improve the data analysis speed.
The third preset threshold value and the fourth preset threshold value are set by a person skilled in the art according to actual requirements, and the value of the third preset threshold value is larger than the fourth preset threshold value.
According to an embodiment of the present invention, further comprising:
the calculation method of the character recognition score of the first recognition data is expressed as:
；
Wherein P 1 is a character recognition score of the first recognition data, P 1(i) is a character confidence of the ith first character in the first recognition data, n is a total number of the first characters in the first recognition data, k 1 is an influence weight of the character confidence of the first character, P 2(j) is a character confidence of the jth second character in the first recognition data, m is a total number of the second characters in the first recognition data, and k 2 is an influence weight of the character confidence of the second character.
It should be noted that, the character recognition score of the first recognition data is determined according to the ratio of the sum of the weighted calculation of the confidence degrees of the characters of all the first characters and the second characters in the first recognition data to the number of the first characters and the second characters. The character confidence of each first character is equally weighted, denoted by k 1, and the character confidence of each second character is equally weighted, denoted by k 2.
According to an embodiment of the present invention, traversing is performed based on a character region of a first character, coordinate data of the first region is determined, detected image data is analyzed based on the coordinate data of the first region, specification data and color distribution data of the first region are determined, including:
determining a character area of each character according to the coordinate data of the character;
randomly generating a traversing point based on the edge of the character area of the first character, traversing clockwise along the edge of the character area of the first character to the periphery, ending traversing when traversing to the edge area of the medicine detection surface or the character areas of other characters except the first character, and determining the traversing terminal point of the current direction;
determining coordinate data of a first area according to the coordinate data of the traversing end point of the character area of the first character in each direction;
Calculating specification data of the first region according to the coordinate data of the first region;
dividing the detected image data based on the coordinate data of the first region, and determining the first image data of the first region;
Analyzing the first image data to determine color distribution data of the first area; the color distribution data includes a color category and a distribution area of each color.
The character area is a rectangular area, a coordinate axis is established by taking the central coordinate of the medicine detection surface in the detected image data as the origin of coordinates, the x and y axes respectively represent the length and the width of the medicine detection surface, and the minimum circumscribed rectangle, namely the character area of the character, is determined by the coordinate data of the character.
In the traversing process, the coordinates of pixel points along the edge of the character area of the first character and the coordinates of the pixel points which have been traversed are traversed clockwise to the periphery of the character area of the first character, when coordinates of the traversing points are overlapped with the edge area of the medicine detection surface or the character areas of other characters except the first character, the traversing in the current direction is ended, the last traversing coordinates of the traversing points are determined to be the traversing end point in the current direction, the traversing in the current direction is not carried out in the subsequent traversing process, the traversing is ended after the traversing end points in the upper direction, the lower direction, the left direction and the right direction of the character area of the first character are determined, and the rectangular area enclosed by the coordinates corresponding to the traversing end points in the upper direction, the lower direction, the left direction and the right direction of the character area of the first character is determined to be the first area.
The character area of each first character corresponds to a first area, and the character areas of adjacent first characters can be used as an integral area for analysis to determine the first area.
And dividing and cutting the detected image data according to the coordinate data of each side of the first area, and determining the first image data of the first area. The color data of each pixel point in the first image data is analyzed, the first image data is divided into a plurality of color areas consisting of different colors by setting a color threshold interval, and the representation color of each color area is represented by the intermediate value of the corresponding color threshold interval. The color category is determined according to the number of color areas in the first image data, and the distribution area of each color is determined according to the coordinate data of each color area.
According to an embodiment of the present invention, specification data and color distribution data of a first area are input to a preset image recognition model, a drug sample in a database is traversed, second recognition data of a second area is determined, and a color recognition score of the second recognition data is calculated, including:
Screening the medicine samples in the database based on the specification data and the color distribution data of the first area, and filtering the medicine samples with the medicine detection surface smaller than the specification data of the first area;
traversing the medicine detection surface of the residual medicine sample based on the specification data of the first area, and determining a traversed area with the image similarity larger than a fifth preset threshold value as a second area;
Determining a drug sample in which a second region exists as second identification data;
determining a color recognition score of the second recognition data according to the image similarity and the area of each second region in the second recognition data;
the calculation method of the color recognition score of the second recognition data is expressed as:
；
Wherein p 2 is the color identification score of the second identification data, S (q1)、S(q2) and S (qn) are the image areas of the 1 st, 2 nd and n th second areas in the second identification data, r (q1)、r(q2) and r (qn) are the image similarity of the 1 st, 2 nd and n th second areas in the second identification data, respectively, and S 1 is the image area of the drug detection surface of the second identification data.
The method includes the steps of screening a medicine detection surface of a medicine sample in a database according to specification data of a first area, filtering the medicine sample with the medicine detection surface smaller than the specification data of the first area, reducing the number of samples, generating a traversing frame based on the specification data of the first area, traversing the medicine detection surface of the rest medicine sample by the traversing frame according to a sequence from left to right and from top to bottom on the medicine detection surface of the rest medicine sample according to a traversing pixel distance preset by a system, analyzing color data (color types and distribution areas of each color) of each pixel point in the image data and the first image data in the traversing frame through a preset image recognition model, calculating image similarity of the image data and the first image data, determining a traversing area corresponding to a current traversing frame as a second area when the image similarity is larger than a fifth preset threshold, and determining the currently analyzed medicine sample as second recognition data, namely recognition data determined through the color similarity of a local area. Wherein each first region can only have a corresponding second region on the drug detection surface of the same drug sample.
After all the first areas in the detected image data are analyzed, color recognition scores of the second recognition data are calculated in sequence according to the similarity of the second areas and the corresponding images in the second recognition data.
In addition, when the second identification data includes a plurality of second areas corresponding to different first areas, the second areas can be verified according to the distribution relationship (relative direction, distance, etc.) between the first areas, and if the distribution relationship between the plurality of second areas is the same as the distribution relationship between the corresponding plurality of first areas, the verification is successful; otherwise, the verification fails, and the current second identification data is not the medicine identification data of the detection image data.
According to an embodiment of the present invention, further comprising:
The calculation method of the identification score of the third identification data is expressed as:
；
wherein p 3(x) is the recognition score of the third recognition data x, p 1(x) is the character recognition score of the recognition data x, p 2(x) is the color recognition score of the recognition data x, and k p1、kp2 is the influence weight of the character recognition score and the color recognition score, respectively.
The identification data x includes first identification data and second identification data, and when the character identification score or the color identification score does not exist in the identification data x, the corresponding score is defaulted to 0.
According to an embodiment of the present invention, further comprising:
and carrying out auxiliary verification on the medicine identification data through the medicine warehouse-in data.
The accuracy of the drug identification data is verified by comparing the data such as the production lot number, the production date, the shelf life and the like in the drug storage data with the first character and the second character of the same type of data in the drug identification data. If the two characters are different, the medicine identification data is wrong, the third identification data with the highest identification score is selected to determine the medicine identification data, and verification is performed again until the two characters are identical, and the final medicine identification data is determined.
Fig. 4 shows a block diagram of a drug review system provided by the present invention.
As shown in fig. 4, a second aspect of the present invention provides a drug review system, comprising:
The data acquisition module is used for determining the data of the medicine to be taken through the doctor's advice data and acquiring detection image data through the detection equipment;
the image recognition module is used for carrying out image recognition on the detection image data and determining character image data of the detection image data;
The character recognition module is used for analyzing the character image data through a preset character recognition model and determining recognition data of the first character and the second character; determining first identification data according to the identification data of the first character and the second character, and calculating character identification scores of the first identification data;
The color recognition module is used for traversing the character area of the first character, determining coordinate data of the first area, analyzing the detection image data based on the coordinate data of the first area and determining specification data and color distribution data of the first area; inputting specification data and color distribution data of the first area into a preset image recognition model, traversing a medicine sample in a database, determining second recognition data of a second area, and calculating color recognition scores of the second recognition data;
The medicine identification module is used for analyzing the first identification data and the second identification data, carrying out weighted calculation on the character identification score and the color identification score of the same identification data, and determining the identification score of the third identification data;
And the medicine rechecking module is used for determining medicine identification data according to the third identification data with the highest identification score, comparing the medicine identification data with the medicine data to be taken, and determining medicine rechecking data.
A third aspect of the present invention provides a computer-readable storage medium, the computer-readable storage medium including a drug review method program, which when executed by a processor, implements the steps of a drug review method as described above.
The invention discloses a drug review method and a drug review system, wherein the drug review method comprises the following steps: determining character image data by detecting the image data; analyzing character image data through a preset character recognition model, and determining recognition data of a first character and a second character; calculating a character recognition score of the first recognition data; determining coordinate data of a first area based on a character area of the first character, and determining specification data and color distribution data of the first area; traversing the medicine sample in the database through a preset image recognition model, determining second recognition data, and calculating color recognition scores of the second recognition data; calculating an identification score of the third identification data; and determining the drug identification data by the third identification data with the highest identification score, comparing the drug identification data with the drug data to be taken, and determining the drug rechecking data. According to the invention, the medicine is checked by combining character recognition with color recognition of the area around the recognized character, so that the medicine checking accuracy is improved.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units; can be located in one place or distributed to a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or optical disk, or the like, which can store program codes.
Or the above-described integrated units of the invention may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
Claims (7)
1. A drug review method, comprising:
Determining the data of the medicine to be taken according to the medical advice data, and acquiring detection image data according to detection equipment;
performing image recognition on the detection image data to determine character image data of the detection image data;
analyzing the character image data through a preset character recognition model to determine recognition data of a first character and a second character;
determining first identification data according to the identification data of the first character and the second character, and calculating character identification scores of the first identification data;
traversing the character area of the first character, determining coordinate data of the first area, analyzing the detection image data based on the coordinate data of the first area, and determining specification data and color distribution data of the first area;
Inputting the specification data and the color distribution data of the first area into a preset image recognition model, traversing the medicine samples in the database, determining second recognition data of a second area, and calculating the color recognition score of the second recognition data;
Analyzing based on the first identification data and the second identification data, performing weighted calculation on the character identification score and the color identification score of the same identification data, and determining the identification score of the third identification data;
Determining medicine identification data according to third identification data with highest identification score, comparing the medicine identification data with medicine data to be taken, and determining medicine rechecking data;
the analyzing the character image data through a preset character recognition model to determine the recognition data of the first character and the second character comprises the following steps:
comparing the character image data with character samples in a database, and determining the character confidence degrees of the sub-character images and the character samples in the character image data;
Determining a sub-character image with the maximum character confidence coefficient larger than a first preset threshold value as a first character, wherein a character sample corresponding to the maximum character confidence coefficient is identification data of the first character;
determining a sub-character image with the maximum character confidence coefficient between a first preset threshold value and a second preset threshold value as a second character, and determining a character sample with the character confidence coefficient larger than the second preset threshold value as recognition data of the second character;
the method for determining first recognition data according to the recognition data of the first character and the second character, and calculating the character recognition score of the first recognition data comprises the following steps:
Combining the identification data of the first character and the second character based on the drug information template in the database to determine one or more first identification data;
Accumulating the character confidence coefficient of each character in the first identification data to determine the identification score of each first identification data;
when only one first identification data with the identification score larger than a third preset threshold value exists, determining the first identification data as medicine identification data of detection image data;
otherwise, the first recognition data with the recognition score smaller than a fourth preset threshold value are filtered, and the character recognition scores of the remaining first recognition data are calculated according to the character confidence degrees of the first characters and the second characters;
Inputting the specification data and the color distribution data of the first area into a preset image recognition model, traversing the medicine sample in the database, determining second recognition data of a second area, and calculating the color recognition score of the second recognition data, wherein the method comprises the following steps:
screening the medicine samples in the database based on the specification data and the color distribution data of the first area, and filtering the medicine samples with the medicine detection surface smaller than the specification data of the first area;
traversing the medicine detection surface of the residual medicine sample based on the specification data of the first area, and determining a traversed area with the image similarity larger than a fifth preset threshold value as a second area;
Determining a drug sample in which a second region exists as second identification data;
determining a color recognition score of the second recognition data according to the image similarity and the area of each second region in the second recognition data;
The calculation method of the color recognition score of the second recognition data is expressed as the following formula:
；
Wherein p 2 is the color identification score of the second identification data, S (q1)、S(q2) and S (qn) are the image areas of the 1 st, 2 nd and n th second areas in the second identification data, r (q1)、r(q2) and r (qn) are the image similarity of the 1 st, 2 nd and n th second areas in the second identification data, respectively, and S 1 is the image area of the drug detection surface of the second identification data.
2. The drug review method of claim 1 wherein the performing image recognition on the test image data to determine character image data of the test image data comprises:
Performing image denoising and image correction on the detected image data to obtain preprocessed image data;
performing binarization processing on the preprocessed image data to determine binarized image data;
performing edge extraction on the binarized image data to determine edge characteristics;
and determining character image data of the detected image data by analyzing the edge characteristics.
3. The drug review method of claim 1, further comprising:
the calculation method of the character recognition score of the first recognition data is expressed as:
；
Wherein P 1 is a character recognition score of the first recognition data, P 1(i) is a character confidence of the ith first character in the first recognition data, n is a total number of the first characters in the first recognition data, k 1 is an influence weight of the character confidence of the first character, P 2(j) is a character confidence of the jth second character in the first recognition data, m is a total number of the second characters in the first recognition data, and k 2 is an influence weight of the character confidence of the second character.
4. The drug review method of claim 1 wherein the traversing based on the character region of the first character, determining the coordinate data of the first region, analyzing the detected image data based on the coordinate data of the first region, determining the specification data and the color distribution data of the first region, comprises:
determining a character area of each character according to the coordinate data of the character;
Randomly generating a traversing point based on the edge of a character area of a first character, traversing clockwise along the edge of the character area of the first character to the periphery, ending traversing when traversing to the edge area of a medicine detection surface or the character areas of other characters except the first character, and determining the traversing terminal point of the current direction;
determining coordinate data of a first area according to the coordinate data of the traversing end point of the character area of the first character in each direction;
Calculating specification data of a first area according to the coordinate data of the first area;
Dividing the detection image data based on the coordinate data of the first area, and determining the first image data of the first area;
analyzing the first image data to determine color distribution data of a first area; the color distribution data includes a color category and a distribution area of each color.
5. The drug review method of claim 1, further comprising:
The method for calculating the identification score of the third identification data is expressed as follows:
；
wherein p 3(x) is the recognition score of the third recognition data x, p 1(x) is the character recognition score of the recognition data x, p 2(x) is the color recognition score of the recognition data x, and k p1、kp2 is the influence weight of the character recognition score and the color recognition score, respectively.
6. The drug review method of claim 1, further comprising:
and carrying out auxiliary verification on the medicine identification data through the medicine warehouse-in data.
7. A drug review system for implementing the drug review method of any one of claims 1-6, comprising:
The data acquisition module is used for determining the data of the medicine to be taken through the doctor's advice data and acquiring detection image data through the detection equipment;
the image recognition module is used for carrying out image recognition on the detection image data and determining character image data of the detection image data;
The character recognition module is used for analyzing the character image data through a preset character recognition model and determining recognition data of the first character and the second character; determining first identification data according to the identification data of the first character and the second character, and calculating character identification scores of the first identification data;
The color recognition module is used for traversing the character area of the first character, determining coordinate data of the first area, analyzing the detection image data based on the coordinate data of the first area and determining specification data and color distribution data of the first area; inputting the specification data and the color distribution data of the first area into a preset image recognition model, traversing the medicine samples in the database, determining second recognition data of a second area, and calculating the color recognition score of the second recognition data;
The medicine identification module is used for analyzing the first identification data and the second identification data, carrying out weighted calculation on the character identification score and the color identification score of the same identification data, and determining the identification score of the third identification data;
The medicine rechecking module is used for determining medicine identification data according to third identification data with highest identification score, comparing the medicine identification data with the medicine data to be taken, and determining medicine rechecking data;
the analyzing the character image data through a preset character recognition model to determine the recognition data of the first character and the second character comprises the following steps:
comparing the character image data with character samples in a database, and determining the character confidence degrees of the sub-character images and the character samples in the character image data;
Determining a sub-character image with the maximum character confidence coefficient larger than a first preset threshold value as a first character, wherein a character sample corresponding to the maximum character confidence coefficient is identification data of the first character;
determining a sub-character image with the maximum character confidence coefficient between a first preset threshold value and a second preset threshold value as a second character, and determining a character sample with the character confidence coefficient larger than the second preset threshold value as recognition data of the second character;
the method for determining first recognition data according to the recognition data of the first character and the second character, and calculating the character recognition score of the first recognition data comprises the following steps:
Combining the identification data of the first character and the second character based on the drug information template in the database to determine one or more first identification data;
Accumulating the character confidence coefficient of each character in the first identification data to determine the identification score of each first identification data;
when only one first identification data with the identification score larger than a third preset threshold value exists, determining the first identification data as medicine identification data of detection image data;
otherwise, the first recognition data with the recognition score smaller than a fourth preset threshold value are filtered, and the character recognition scores of the remaining first recognition data are calculated according to the character confidence degrees of the first characters and the second characters;
Inputting the specification data and the color distribution data of the first area into a preset image recognition model, traversing the medicine sample in the database, determining second recognition data of a second area, and calculating the color recognition score of the second recognition data, wherein the method comprises the following steps:
screening the medicine samples in the database based on the specification data and the color distribution data of the first area, and filtering the medicine samples with the medicine detection surface smaller than the specification data of the first area;
traversing the medicine detection surface of the residual medicine sample based on the specification data of the first area, and determining a traversed area with the image similarity larger than a fifth preset threshold value as a second area;
Determining a drug sample in which a second region exists as second identification data;
determining a color recognition score of the second recognition data according to the image similarity and the area of each second region in the second recognition data;
The calculation method of the color recognition score of the second recognition data is expressed as the following formula:
；
Wherein p 2 is the color identification score of the second identification data, S (q1)、S(q2) and S (qn) are the image areas of the 1 st, 2 nd and n th second areas in the second identification data, r (q1)、r(q2) and r (qn) are the image similarity of the 1 st, 2 nd and n th second areas in the second identification data, respectively, and S 1 is the image area of the drug detection surface of the second identification data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410748869.8A CN118334639B (en) | 2024-06-12 | 2024-06-12 | Medicine rechecking method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410748869.8A CN118334639B (en) | 2024-06-12 | 2024-06-12 | Medicine rechecking method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118334639A CN118334639A (en) | 2024-07-12 |
| CN118334639B true CN118334639B (en) | 2024-08-23 |
Family
ID=91771110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410748869.8A Active CN118334639B (en) | 2024-06-12 | 2024-06-12 | Medicine rechecking method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118334639B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102782680A (en) * | 2010-02-26 | 2012-11-14 | 乐天株式会社 | Information processing device, information processing method, and recording medium that has recorded information processing program |
| CN113012783A (en) * | 2021-03-18 | 2021-06-22 | 深圳市瑞意博科技股份有限公司 | Medicine rechecking method and device, computer equipment and storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH08212296A (en) * | 1995-02-08 | 1996-08-20 | Oki Electric Ind Co Ltd | Optical character reader |
| WO2011074067A1 (en) * | 2009-12-15 | 2011-06-23 | 富士通フロンテック株式会社 | Character recognition method, character recognition device, and character recognition program |
| JP6704722B2 (en) * | 2015-12-14 | 2020-06-03 | キヤノン株式会社 | Image processing apparatus and image processing method |
| CN109740606B (en) * | 2018-12-20 | 2021-02-05 | 上海众源网络有限公司 | Image identification method and device |
| US11620839B2 (en) * | 2019-11-14 | 2023-04-04 | Walmart Apollo, Llc | Systems and methods for detecting text in images |
| CN111563495B (en) * | 2020-05-09 | 2023-10-27 | 北京奇艺世纪科技有限公司 | Method and device for recognizing characters in image and electronic equipment |
| CN114168772A (en) * | 2021-12-03 | 2022-03-11 | 杭州睿胜软件有限公司 | Tablet identification method, readable storage medium, and electronic device |
| CN117011855A (en) * | 2023-10-08 | 2023-11-07 | 深圳市豪斯莱科技有限公司 | Character string image cutting and identifying method, system and readable storage medium |
-
2024
- 2024-06-12 CN CN202410748869.8A patent/CN118334639B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102782680A (en) * | 2010-02-26 | 2012-11-14 | 乐天株式会社 | Information processing device, information processing method, and recording medium that has recorded information processing program |
| CN113012783A (en) * | 2021-03-18 | 2021-06-22 | 深圳市瑞意博科技股份有限公司 | Medicine rechecking method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118334639A (en) | 2024-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104933429B (en) | The method and device of information extraction in image | |
| EP2937819A1 (en) | Information code, method for generating information code, device for reading information code, and system for using information code | |
| US10217083B2 (en) | Apparatus, method, and program for managing articles | |
| KR101744123B1 (en) | Apparatus and method for detecting medicinal products | |
| CN113569863B (en) | Document checking method, system, electronic equipment and storage medium | |
| CN113012783A (en) | Medicine rechecking method and device, computer equipment and storage medium | |
| CN114037541A (en) | Fixed-point medicine institution supervision method based on biological feature recognition and related equipment | |
| CN118334639B (en) | Medicine rechecking method and system | |
| CN114494189A (en) | Nursing wound image recognition and detection method and device | |
| CN114168772A (en) | Tablet identification method, readable storage medium, and electronic device | |
| JP7449505B2 (en) | information processing system | |
| CN112364857B (en) | Image recognition method, device and storage medium based on numerical extraction | |
| CN113420684A (en) | Report recognition method and device based on feature extraction, electronic equipment and medium | |
| CN111477289A (en) | Method and device for storing and reading detection report | |
| CN112131919B (en) | Security inspection method, device, equipment and medium | |
| CN111460959A (en) | Document management method and related device | |
| Suksawatchon et al. | Shape Recognition Using Unconstrained Pill Images Based on Deep Convolution Network | |
| KR102598969B1 (en) | Pill identification system | |
| JP6454249B2 (en) | Drug inspection apparatus and method | |
| CN111599436A (en) | Clinical medication dispensing method and dispensing device thereof | |
| CN110751158A (en) | Digital identification method and device in therapeutic bed display and storage medium | |
| CN117557794A (en) | Automatic identification and verification method and system for pharmacy mixed medicines | |
| CN117576711A (en) | Medical record information extraction method and system | |
| WO2023171132A1 (en) | Information processing system | |
| US20170200383A1 (en) | Automated review of forms through augmented reality |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |