TWI781856B - Method for identifying medicine image, computer device and storage medium - Google Patents

Abstract

The present application relates to an image analysis technology, and the present application provides a method for identifying medicine image, a computer device and a storage medium. The method includes obtaining a plurality of medicine images, a test image and a pre-trained medicine detection model, which includes a location detection network, a text-recognition network and a category-identification network; obtaining a plurality of target images by inputting the medicine images into the location detection network, generating a plurality of text characteristic matrices by inputting the target images into the text-recognition network, and generating a plurality of image characteristic matrices by inputting the target images into the category-identification network; generating a reference matrix according to each of the image characteristic matrices and a corresponding text characteristic matrix; generating a test matrix according to the test image and the medicine detection model, generating an identification result according to a similarity between the test matrix and each reference matrix.

Description

Drug image recognition method, computer equipment and storage medium

The present application relates to the field of image processing, in particular to a drug image recognition method, computer equipment and storage media.

In the existing drug image recognition scheme, it takes a lot of manpower to label drug data. When the training data is too small, the recognition accuracy of the training model will be poor. Therefore, how to improve the accuracy of drug image recognition has become an urgent need to solve the technology question.

In view of the above, it is necessary to provide a drug image recognition method, computer equipment and storage The storage medium can solve the technical problem that it is difficult to accurately and efficiently identify drug images.

A drug image recognition method, the drug image recognition method includes: acquiring a plurality of drug images and drug images to be tested; acquiring a pre-trained drug detection model, the drug detection model includes a position detection network, A character recognition network and a category recognition network; input the multiple drug images into the position detection network to obtain multiple target images, each target image contains an image of a single drug; Generate a complex character feature matrix according to the multiple target images and the character recognition network; input the multiple target images into the category recognition network to obtain complex image features feature matrix; generate a reference matrix according to each image feature matrix and corresponding text feature matrix; process the drug image to be tested based on the drug detection model to obtain a test matrix; according to the test matrix and each reference The similarity of the matrix generates the recognition result of the drug image to be tested.

According to an optional embodiment of the present application, before inputting the multiple medicine images into the position detection network to obtain multiple target images, the medicine image recognition method further includes: acquiring position detection learning device and a position image, the position image includes a first image and a second image, and the second image includes a plurality of marked images and a plurality of unmarked images; using the first image pair The position detection learner is trained to obtain a first pre-training network; the first pre-training network is adjusted based on the multiple marked images to obtain a first marked network; the multiple marked images The unmarked image is input into the first marking network to obtain the output image and the predicted probability value of the drug contained in each output image; the output image corresponding to the predicted probability value greater than the preset threshold is paired with the adjusting the first marked network to obtain a second marked network; calculating a first loss value of the second marked network, and performing multiple adjustments to the second marked network based on the first loss value , stop adjusting until the first loss value drops to a minimum, and obtain the position detection network.

According to an optional embodiment of the present application, before generating a complex character feature matrix based on the multiple target images and the character recognition network, the drug image recognition method further includes: acquiring a character image and a character recognition learner , the character image includes a third image and a fourth image; use the third image to train the character recognition learner to obtain a second pre-training network, wherein the second pre-training Networks include convolutional neural network models and recurrent neural network models type; calculate the second loss value of the second pre-training network, and perform backpropagation through the second loss value, and adjust the parameters of the second pre-training network multiple times until the first After the second pre-training model reaches convergence, the adjustment is stopped to obtain the character recognition network.

According to an optional embodiment of the present application, the generating complex character feature matrix according to the multiple target images and the character recognition network includes: performing color conversion on each target image to obtain multiple gray scale images of characters image; Binarize each text grayscale image to obtain multiple binarized images; filter each binarized image to obtain multiple filtered images; locate each filtered image According to the position of the Chinese medicine text, the text position is obtained; the text image is selected from each target image according to the text position; each text image is input to the convolutional neural network model for feature extraction, and the feature is obtained. Sequence; input the feature sequence into the cyclic neural network model to obtain the complex character feature matrix.

According to an optional embodiment of the present application, before inputting the plurality of target images into the class recognition network to obtain a complex image feature matrix, the drug image recognition method further includes: acquiring a class recognition learner, so The soft-max function is used as the activation function in the class identification learner; the third loss value of the class identification learner is calculated, and the class identification learner is adjusted based on the third loss value until the first After the loss value drops to the minimum, the adjustment is stopped, and the activation function is deleted from the adjusted class recognition learner to obtain the class recognition network.

其中，

是指所述第三損失值，2N是指所述多張增廣圖像，i是 指所述多張增廣圖像中的第i張增廣圖像，yi是指所述第i張增廣圖像的類別，j是指與i的類別相同的增廣圖像中的第j張增廣圖像，yj是指所述第j張增廣圖像的類別，N _yi 是指與i的類別相同的所有增廣圖像的數量，∥ _i≠j 為第一指示函數，當且僅當i=j時取零，當i≠j時取1，∥ _yi=yj 為第二指示函數，當且僅當yi=yj時取零，當yi≠yj時取1，∥ _i≠k 為第三指示函數，當且僅當i=k時取零，當i≠k時取1，z _i 是指將i輸入到所述類別辨識網路中得到的單位向量，z _j 是指將j輸入到所述類別辨識網路中得到的單位向量，k是指除了i之外的任意一張增廣圖像，z _k 是指將k輸入到所述類別辨識網路中得到的單位向量，τ為預設的標量調節參數。 According to an optional embodiment of the present application, the calculating the third loss value of the class recognition learner includes: acquiring a plurality of class images, the plurality of class images including plural classes; Perform augmentation processing to obtain multiple augmented images; the formula for determining the third loss value is:

in,

refers to the third loss value, 2 N refers to the multiple augmented images, i refers to the i -th augmented image in the plurality of augmented images, and yi refers to the category of the i -th augmented image, j refers to the j -th augmented image among the augmented images of the same category as i , yj refers to the category of the j -th augmented image, N _yi refers to all augmented images of the same category as i ∥ _{i ≠ j} is the first indicator function, which is zero if and only when i = j , and 1 when i ≠ j , ∥ _{yi = yj} is the second indicator function, which is taken if and only when yi = yj zero, takes 1 when yi ≠ yj , ∥ _{i ≠ k} is the third indicator function, takes zero if and only when i = k , takes 1 when i ≠ k , z _i refers to inputting i into the category The unit vector obtained in the identification network, z _j refers to the unit vector obtained by inputting j into the category identification network, k refers to any augmented image except i , z _k refers to inputting k to the unit vector obtained in the category recognition network, τ is a preset scalar adjustment parameter.

According to an optional embodiment of the present application, the generating the reference matrix according to each image feature matrix and the corresponding text feature matrix includes: adding each image feature matrix and the corresponding text feature matrix to obtain the reference matrix, Wherein, each image feature matrix and the corresponding text feature matrix have the same number of rows and columns.

According to an optional embodiment of the present application, the generating the recognition result of the test drug image according to the similarity between the test matrix and each reference matrix includes: calculating the similarity between the test matrix and each reference matrix degree; determining the reference matrix corresponding to the maximum similarity as the target matrix; performing mapping processing on the target matrix based on a preset label mapping table to obtain the identification result.

The present application provides a computer device, which includes: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the drug image recognition method.

The present application provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in a computer device to implement the drug image recognition method.

It can be seen from the above technical solutions that the position detection network is obtained by using the multiple marked images to adjust the second pre-training network multiple times, because the position detection network has learned the The features of multiple marked images, so the position detection network can mark the position of the image to be tested, and perform augmentation processing on the multiple category images when training the category recognition network , to avoid the problem of too little training data, generate a complex reference matrix according to each image feature matrix and corresponding text feature matrix, generate the test matrix in the same way as the image to be tested, and calculate the test matrix The similarity with each reference matrix, and select the label information of the reference matrix corresponding to the maximum similarity as the identification result, because the matrix to be tested contains both the text features and image features of the image to be tested , so that the matrix to be tested can fully reflect the features of the image to be tested, so that the accuracy of the recognition result is higher.

1: Computer equipment

2: camera device

12: Storage

13: Processor

S10~S17: Steps

FIG. 1 is an application environment diagram of a preferred embodiment of the drug image recognition method of the present application.

FIG. 2 is a flowchart of a preferred embodiment of the drug image recognition method of the present application.

FIG. 3 is a schematic structural diagram of a computer device for implementing a preferred embodiment of the drug image recognition method of the present application.

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in FIG. 1 , it is an application environment diagram of a preferred embodiment of a drug image recognition method of the present application. The drug image recognition method can be applied to one or a plurality of computer equipment 1, and the computer equipment 1 communicates with the camera device 2, and the camera device 2 can be a camera, or can be used to realize the camera For example, the drug to be tested can be photographed through the camera device 2 to obtain an image of the drug to be tested. The drug to be tested can be a capsule or tablet drug, for example, amoxicillin capsule, clarithromycin dispersible tablet and the like.

The computer device 1 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to: microprocessor, application specific integrated circuit (Application Specific Integrated Circuit) , ASIC), programmable gate array (Field-Programmable Gate Array, FPGA), digital signal processor (Digital Signal Processor, DSP), embedded devices, etc.

The computer device 1 can be any computer product capable of man-machine interaction with the user, for example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), a game console, an interactive network TV (Internet Protocol Television, IPTV), smart wearable devices, etc.

The computer equipment 1 may also include network equipment and/or user equipment. Wherein, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on Cloud Computing.

The network where the computer device 1 is located includes but not limited to Internet, wide area network, metropolitan area network, local area network, virtual private network (Virtual Private Network, VPN) and so on.

As shown in FIG. 2 , it is a flowchart of a preferred embodiment of a drug image recognition method of the present application. According to different requirements, the order of each step in the flow chart can be adjusted according to actual detection requirements, and some steps can be omitted. The execution body of the method is a computer device, such as the computer device 1 shown in FIG. 1 .

Step S10, acquiring a plurality of images of the drug and images of the drug to be tested.

In at least one embodiment of the present application, the multiple medicine images refer to medicine images carrying label information, and the multiple medicine images can be used to generate a reference matrix.

The label information may include, but not limited to: the name of the drug, the category of the drug, the usage of the drug, and the like.

In at least one embodiment of the present application, the drug image to be tested refers to a drug image that does not carry the label information, and there are drug characters on the surface of the drug to be tested in the drug image to be tested, the Drug characters can be letters and numbers.

In at least one embodiment of the present application, the computer device acquires the plurality of medicine images and label information corresponding to each medicine image from a preset target database.

In at least one embodiment of the present application, the computer device controls the imaging device 2 to photograph the drug to be tested to obtain an image of the drug to be tested.

Wherein, the camera device 2 may be a camera.

Step S11 , obtaining a pre-trained drug detection model, which includes a location detection network, a text recognition network, and a category recognition network.

In at least one embodiment of the present application, the drug detection model refers to a network model that detects the position of the drug in the image to be tested and each drug image.

In at least one embodiment of the present application, the position detection network is used to select a single image of the drug from each image of the drug.

In at least one embodiment of the present application, the text recognition network can be used to obtain text information in each medicine image.

In at least one embodiment of the present application, the category recognition network can be used to identify the category of medicine in each medicine image.

Step S12, inputting the multiple images of the drug into the position detection network to obtain multiple target images, each of which contains an image of a single drug.

In at least one embodiment of the present application, the target image refers to an image containing a single drug selected from each drug image.

In at least one embodiment of the present application, before inputting the multiple medicine images into the position detection network to obtain multiple target images, the medicine image recognition method further includes: the computer The device acquires a position detection learner and a position image, the position image includes a first image and a second image, and the second image includes a plurality of marked images and a plurality of unmarked images, so The computer device uses the first image to train the position detection learner to obtain a first pre-training network, and adjusts the first pre-training network based on the plurality of marked images to obtain The first marking network, the computer device inputs the multiple unmarked images into the first marking network, obtains the output image and the predicted probability value of the drug contained in each output image, and The output image corresponding to the predicted probability value greater than the preset threshold is adjusted to the first marked network to obtain a second marked network. Further, the computer device calculates the first marked network of the second marked network. loss value, and adjust the second marking network multiple times based on the first loss value until the first loss value drops to a minimum and then stop the adjustment to obtain the position detection network.

Specifically, the computer device adjusts the second marking network multiple times based on the first loss value, and stops the adjustment until the first loss value drops to a minimum, so that the position detection network includes : the computer device inputs the output image corresponding to the predicted probability value greater than the preset threshold to the first labeling network for training, and iteratively updates the weights of the first labeling network until The first marking network achieves convergence to obtain the location detection network.

Wherein, the position detection learner can be a target detector efficientDet, and the position detection learner can be used to accurately locate the position of a single drug in each drug image. The first image refers to the image obtained from the preset first database, which can be an image containing any object, and the first database can be COCO database, ImageNet database and CPTN data database etc. The first image includes images of multiple types of items such as animals (for example, puppies, kittens, etc.), plants (for example, flowers, trees, etc.), and the second image refers to items containing medicines. image, and the second image may be obtained from a preset second database. The multiple marked images refer to images in which the position of the drug in the image has been marked, and the multiple unmarked images refer to images in which the position of the drug in the image has not been marked.

The output image refers to an image that may include a single drug, and the output image can be used for multiple adjustments to the first marking network.

The predicted probability value refers to the probability that a single drug is contained in the output image.

The preset threshold can be set by itself, which is not limited in this application.

The first pre-trained network refers to a network obtained after pre-training using the first image.

The first labeled network refers to a network obtained after retraining the first pre-trained network according to the multiple labeled images, and the first labeled network can be used to Label images for feature labeling.

[0,1]，γ

0。 The calculation formula of the first loss value is: FL ( p _t )=- α _t (1- p _t ) ^γ log( p _t ); wherein, FL ( p _t ) is the first loss value, p _t is the predicted probability value, α _t

[0,1], γ

0.

Through the above implementation, the second labeling network with the smallest first loss value can be selected as the position detection network, because the second labeling network has learned the features of the multiple labeling images, so that The position detection network can accurately mark each drug image.

Step S13, generating complex character feature matrices according to the plurality of target images and the character recognition network.

In at least one embodiment of the present application, the complex text feature matrix refers to a matrix containing text features in the multiple target images, and each text feature matrix can be used to generate the reference matrix.

In at least one embodiment of the present application, before generating a complex character feature matrix based on the multiple target images and the character recognition network, the drug image recognition method further includes: the computer device acquires a character image An image and character recognition learner, the character image includes a third image and a fourth image, and the third image is used to train the character recognition learner to obtain a second pre-training network, wherein, The second pre-training network includes a convolutional neural network model and a recurrent neural network model, and the computer device calculates a second loss value of the second pre-training network, and uses the second loss value to perform Backpropagation, adjusting the parameters of the second pre-training network multiple times until the second pre-training model reaches convergence and then stop the adjustment to obtain the character recognition network.

Wherein, the character recognition learner refers to a learner that recognizes the characters of each drug image, and the character image refers to an image used to train the character recognition learner, and in the character image Contains text attached to any medicine.

The convolutional neural network model may be a VGG16 network, and the convolutional neural network model may be used to extract text features of the fourth image. The cyclic neural network model may be a long short-term memory network (Long Short-Term Memory, LSTM), and the cyclic neural network model may be used to extract time series information of the character features.

The third image can be used to train the weight of the character recognition learner, and the fourth image refers to an image containing any drug characters, and the drug characters can include, but not limited to: letters and numbers. The third image can be obtained from the first database, and the fourth image can be obtained from a pre-set third database, and the third database stores multiple images of drug characters. picture.

The loss function used by the second pre-training network may be a connectionist temporal classification loss function (Connectionist temporal classification, CTC).

Through the above implementation, selecting the second pre-training network corresponding to the lowest second loss value as the text recognition network can improve the reliability and accuracy of the text recognition network, so that the text recognition network can Accurately extract the text features of the target image.

In at least one embodiment of the present application, the computer device generates a complex character feature matrix according to the multiple target images and the character recognition network includes: the computer device performs color conversion on each target image , to obtain multiple text grayscale images, and perform binarization processing on each text grayscale image to obtain multiple binarized images, and filter each binarized image to obtain multiple filtered images image, further, the computer device locates the position of the drug text in each filtered image to obtain the text position, and selects a text image from each target image according to the text position, and further, the The computer device inputs each text image into the convolutional neural network model for feature extraction to obtain a feature sequence, and the computer device inputs the feature sequence into the cyclic neural network model to obtain the complex number Text feature matrix.

Wherein, the feature sequence refers to the convolutional neural network model for each filter image The feature obtained by row extraction, the complex character feature matrix refers to the feature obtained by the cyclic neural network model extracting the feature sequence.

Through the above-mentioned implementation, each target image is processed by color conversion, binarization, filtering, etc., and a clearer filtered image can be obtained. According to the multiple filtered images, character features can be accurately obtained, which is beneficial to Generate the character feature matrix.

Step S14, inputting the plurality of target images into the class recognition network to obtain complex image feature matrices.

In at least one embodiment of the present application, the complex image feature matrix refers to a matrix including image features in the multiple target images.

In at least one embodiment of the present application, before inputting the plurality of target images into the class recognition network to obtain a complex image feature matrix, the drug image recognition method further includes: the computer device acquires A class recognition learner, using a soft-max function as an activation function in the class recognition learner, the computer device calculates a third loss value of the class recognition learner, and further, based on the third loss value pair The class recognition learner is adjusted until the third loss value drops to a minimum, then the adjustment is stopped, and the activation function is deleted from the adjusted class recognition learner to obtain the class recognition network.

Specifically, the computer device acquires a class recognition learner, and using the soft-max function as an activation function in the class recognition learner includes: the computer device builds the class recognition learner based on the resnet50 network, and uses the soft-max function as the activation function. max is used as the activation function.

In at least one embodiment of the present application, the calculation of the third loss value of the class recognition learner by the computer device includes: the computer device acquires a plurality of class images, and the plurality of class images include plural classes, performing augmentation processing on the plurality of category images to obtain a plurality of augmented images, the plurality of augmented images exist in pairs, and each pair of augmented images includes a first augmented image and a second augmented image, The first augmented image and the second augmented image are from the same category image.

The plurality of category images refer to output images corresponding to predicted probability values whose values are greater than a preset threshold.

The augmentation process refers to an image obtained by performing transformations such as rotation and cropping on each category image.

The plural categories include, but are not limited to: antibiotics, vitamins, and the like.

其中，

是指所述第三損失值，2N是指所述多張增廣圖像，i是 指所述多張增廣圖像中的第i張增廣圖像，yi是指所述第i張增廣圖像的類別，j是指與i的類別相同的增廣圖像中的第j張增廣圖像，yj是指所述第j張增廣圖像的類別，N _yi 是指與i的類別相同的所有增廣圖像的數量，∥ _i≠j 為第一指示函數，當且僅當i=j時取零，當i≠j時取1，∥ _yi=yj 為第二指示函數，當且僅當yi=yj時取零，當yi≠yj時取1，∥ _i≠k 為第三指示函數，當且僅當i=k時取零，當i≠k時取1，z _i 是指將i輸入到所述類別辨識網路中得到的單位向量，z _j 是指將j輸入到所述類別辨識網路中得到的單位向量，k是指除了i之外的任意一張增廣圖像，z _k 是指將k輸入到所述類別辨識網路中得到的單位向量，τ為預設的標量調節參數。 The formula for determining the third loss value is:

in,

refers to the third loss value, 2 N refers to the multiple augmented images, i refers to the i -th augmented image in the plurality of augmented images, and yi refers to the category of the i -th augmented image, j refers to the j -th augmented image among the augmented images of the same category as i , yj refers to the category of the j -th augmented image, N _yi refers to all augmented images of the same category as i ∥ _{i ≠ j} is the first indicator function, which is zero if and only when i = j , and 1 when i ≠ j , ∥ _{yi = yj} is the second indicator function, which is taken if and only when yi = yj zero, takes 1 when yi ≠ yj , ∥ _{i ≠ k} is the third indicator function, takes zero if and only when i = k , takes 1 when i ≠ k , z _i refers to inputting i into the category The unit vector obtained in the identification network, z _j refers to the unit vector obtained by inputting j into the category identification network, k refers to any augmented image except i , z _k refers to inputting k to the unit vector obtained in the category recognition network, τ is a preset scalar adjustment parameter.

Through the above implementation manner, the multiple category images are augmented, the training data is expanded, and more training data are used to train the category recognition network, which can improve the recognition accuracy of the category recognition network .

Step S15, generating a reference matrix according to each image feature matrix and the corresponding character feature matrix.

In at least one embodiment of the present application, the reference matrix refers to a matrix that includes the image features and text features of the complex target images, and the reference matrix can be used to indicate the label of the drug image to be tested information.

In at least one embodiment of the present application, the computer device according to each image feature moment Generating the reference matrix by the matrix and the corresponding character feature matrix includes: the computer equipment adds each image feature matrix and the corresponding character feature matrix to obtain the reference matrix, wherein each image feature matrix and the corresponding character matrix The feature matrix has the same number of rows and columns.

In at least one embodiment of the present application, there are other ways to generate the reference matrix, for example: the computer device multiplies each image feature matrix and the corresponding character feature matrix to obtain the reference matrix, or , the computer device subtracts each image feature matrix and the corresponding text feature matrix to obtain the reference matrix.

Through the above embodiments, the image features and text features of the medicine in each medicine image can be extracted, and a reference matrix having both the image features and the text features can be generated.

Step S16, processing the image of the drug to be tested based on the drug detection model to obtain a matrix to be tested.

In at least one embodiment of the present application, the test matrix refers to a matrix including image features and text features of the test image.

Since the process of generating the test matrix is consistent with that of generating the reference matrix, the present application does not repeat it here.

Through the above-mentioned embodiment, the image to be tested is processed by processing the plurality of drug images to obtain the matrix to be tested, so that the matrix to be tested and each reference matrix have the same number of rows and columns It is easier to calculate the similarity between the test matrix and each reference matrix.

Step S17, generating a recognition result of the image of the drug to be tested according to the similarity between the matrix to be tested and each reference matrix.

In at least one embodiment of the present application, the identification result refers to label information corresponding to the drug to be tested.

In at least one embodiment of the present application, the computer device generating the recognition result of the test drug image according to the similarity between the test matrix and each reference matrix includes: the computer device calculating the test The similarity between the matrix and each reference matrix will be maximized The reference matrix corresponding to the similarity is determined as the target matrix, and the target matrix is mapped based on the preset label mapping table to obtain the identification result.

Wherein, the preset label mapping table refers to a mapping table between each reference matrix and corresponding label information, and each reference matrix in the preset label mapping table corresponds to label information one by one.

The similarity may include, but not limited to: cosine similarity and Euclidean distance.

cosine是指所述余弦相似度，n是指所述待測矩陣及任意一個參照矩陣中的所有元素，i指所述待測矩陣及任意一個參照矩陣中的第i個元素，A _i 是指所述待測矩陣中的第i個元素，B _i 是指任意一個參照矩陣中的第i個元素。 The formula for calculating the cosine similarity is:

cosine refers to the cosine similarity, n refers to all elements in the test matrix and any reference matrix, i refers to the i -th element in the test matrix and any reference matrix, A _i refers to The i -th element in the test matrix, B _i refers to the i -th element in any reference matrix.

Specifically, the computer device performs mapping processing on the target matrix based on a preset label mapping table, and obtaining the identification result includes: the computer device determines that the drug to be tested is in the preset label according to the target matrix. corresponding label information in the mapping table, and use the corresponding label information as the identification result.

Through the above-mentioned embodiment, the reference matrix corresponding to the maximum similarity is selected as the target matrix, so that the drug image corresponding to the target matrix is more similar to the drug image to be tested, and the corresponding drug image The label information is used as the label information of the drug to be tested, and according to the one-to-one correspondence between the target matrix and the label information, the label information of the drug to be tested can be quickly found, which improves the identification efficiency of the drug image.

It can be seen from the above technical solutions that the position detection network is obtained by using the multiple marked images to adjust the second pre-training network multiple times, because the position detection network has learned the The features of multiple marked images, so the position detection network can mark the position of the image to be tested, and perform augmentation processing on the multiple category images when training the category recognition network , to avoid the problem of too little training data, generate a complex reference matrix according to each image feature matrix and corresponding text feature matrix, generate the test matrix in the same way as the image to be tested, and calculate the test matrix The similarity with each reference matrix, and select the label of the reference matrix corresponding to the largest similarity information as the recognition result, since the matrix to be tested includes both text features and image features of the image to be tested, the matrix to be tested can fully reflect the features of the image to be tested, Therefore, the accuracy of the identification result is higher.

As shown in FIG. 3 , it is a schematic structural diagram of a computer device for implementing a preferred embodiment of the drug image recognition method of the present application.

In one embodiment of the present application, the computer device 1 includes, but is not limited to, a storage 12, a processor 13, and a computer program stored in the storage 12 and operable on the processor 13 , such as a drug image recognition program.

Those skilled in the art can understand that the schematic diagram is only an example of the computer device 1 and does not constitute a limitation to the computer device 1. It may include more or less components than those shown in the illustration, or combine certain components, or have different Components, for example, the computer device 1 may also include input and output devices, network access devices, bus bars, and the like.

The processor 13 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC) , Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., and the processor 13 is the computing core and control center of the computer device 1, and utilizes various interfaces and lines to connect the entire computer device 1 Each part of the system, and obtain the operating system of the computer device 1 and various installed applications, program codes, etc. For example, the processor 13 may acquire the image of the drug to be tested captured by the camera device 2 through an interface.

The processor 13 acquires the operating system of the computer device 1 and various installed applications. The processor 13 acquires the application program to implement the steps in the above embodiments of the drug image recognition method, for example, the steps shown in FIG. 2 .

Exemplarily, the computer program can be divided into one or multiple modules/units, the one or multiple modules/units are stored in the storage 12 and acquired by the processor 13, to complete this application. The one or multiple modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the acquisition process of the computer program in the computer device 1 .

The storage 12 can be used to store the computer programs and/or modules, and the processor 13 executes or obtains the computer programs and/or modules stored in the storage 12, and calls the computer programs and/or modules stored in the storage 12 to realize various functions of the computer device 1. The storage device 12 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.); The area can store data created according to the use of the computer equipment, etc. In addition, the storage 12 may include non-volatile storage, such as hard disk, memory, plug-in hard disk, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash memory A memory card (Flash Card), at least one magnetic disk storage device, a flash memory device, or other non-volatile solid state storage devices.

The storage 12 may be an external storage and/or an internal storage of the computer device 1 . Further, the storage 12 may be a physical storage, such as a storage stick, a TF card (Trans-flash Card) and the like.

If the integrated modules/units of the computer device 1 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the methods of the above embodiments of the present application can also be completed by controlling related hardware through instructions of a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is acquired by the processor, it can realize the steps of the above-mentioned various method embodiments.

Wherein, the computer program includes computer program code, and the computer program code may be in the form of original code, object code, obtainable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer storage, a read-only memory (ROM, Read- Only Memory).

Referring to FIG. 2 , the storage 12 in the computer device 1 stores multiple instructions to implement a drug image recognition method, and the processor 13 can obtain the multiple instructions to realize: acquiring multiple images of drugs and the drug images to be tested Drug image; obtain a pre-trained drug detection model, the drug detection model includes a position detection network, a character recognition network and a category recognition network; input the multiple drug images to the position In the detection network, multiple target images are obtained, and each target image contains an image of a single drug; according to the multiple target images and the character recognition network, a complex character feature matrix is generated; The plurality of target images are input into the category recognition network to obtain complex image feature matrices; a reference matrix is generated according to each image feature matrix and the corresponding character feature matrix; the drug detection model is used to process the to-be The test drug image is obtained to obtain the test matrix; the identification result of the test drug image is generated according to the similarity between the test matrix and each reference matrix.

Specifically, for the specific implementation method of the above instruction by the processor 13, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 2 , which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or may also be distributed to multiple network units superior. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of hardware plus software function modules.

Therefore, no matter from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the application is defined by the appended claims rather than the above description, so it is intended to All changes that come within the meaning and range of equivalents of the claims are embraced in this application. Any attached reference mark in a claim shall not be deemed to limit the claim to which it relates.

In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or devices stated in this application can also be realized by one unit or device through software or hardware. The terms first, second, etc. are used to denote names and do not imply any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application without limitation. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be Make modifications or equivalent replacements without departing from the spirit and scope of the technical solutions of the present application.

S10~S17: Steps

Claims (8)

A drug image recognition method, which is applied to computer equipment, wherein the drug image recognition method includes: acquiring a plurality of drug images and drug images to be tested; acquiring a pre-trained drug detection model, the drug detection The model includes a position detection network, a character recognition network, and a category recognition network, wherein the obtaining the category recognition network includes: obtaining a category recognition learner, and a soft-max function is used in the category recognition learner as The activation function is used to calculate the third loss value of the category recognition learner, including: acquiring a plurality of category images, the plurality of category images including complex categories, and performing augmentation processing on the plurality of category images to obtain Multiple augmented images, the formula for determining the third loss value is:

in,

refers to the third loss value, 2 N refers to the multiple augmented images, i refers to the i -th augmented image in the plurality of augmented images, and yi refers to the category of the i -th augmented image, j refers to the j -th augmented image among the augmented images of the same category as i , yj refers to the category of the j -th augmented image, N _yi refers to all augmented images of the same category as i ∥ _{i ≠ j} is the first indicator function, which is zero if and only when i = j , and 1 when i ≠ j , ∥ _{yi = yj} is the second indicator function, which is taken if and only when yi = yj zero, takes 1 when yi ≠ yj , ∥ _{i ≠ k} is the third indicator function, takes zero if and only when i = k , takes 1 when i ≠ k , z _i refers to inputting i into the category The unit vector obtained in the identification network, z _j refers to the unit vector obtained by inputting j into the category identification network, k refers to any augmented image except i , z _k refers to inputting k to the unit vector obtained in the class recognition network, τ is a preset scalar adjustment parameter, and the class recognition learner is adjusted based on the third loss value until the third loss value drops to the minimum stop the adjustment, and delete the activation function from the adjusted class recognition learner to obtain the class recognition network; input the multiple drug images into the position detection network to obtain multiple target Image, each target image contains an image of a single drug; According to the multiple target images and the text recognition network, a complex character feature matrix is generated; the multiple target images are input to the In the category recognition network, a complex image feature matrix is obtained; a reference matrix is generated according to each image feature matrix and the corresponding character feature matrix; the drug image to be tested is processed based on the drug detection model to obtain a test matrix; A recognition result of the drug image to be tested is generated according to the similarity between the matrix to be tested and each reference matrix. The medicine image recognition method according to claim 1, wherein, before inputting the multiple medicine images into the position detection network to obtain multiple target images, the medicine image recognition method further includes : Obtain a position detection learner and a position image, the position image includes a first image and a second image, and the second image includes a plurality of marked images and a plurality of unmarked images, wherein, The first image is a plurality of types of item images; using the first image to train the position detection learner to obtain a first pre-training network; The first pre-training network is adjusted to obtain a first marked network; the multiple unlabeled images are input into the first marked network to obtain output images and each output image contains The predicted probability value of the drug; the output image corresponding to the predicted probability value greater than the preset threshold is adjusted to the first marked network to obtain the second marked network; the first marked network of the second marked network is calculated. loss value, and adjust the second marking network multiple times based on the first loss value until the first loss value drops to a minimum and then stop the adjustment to obtain the position detection network. The drug image recognition method according to claim 1, wherein, before generating a complex character feature matrix based on the multiple target images and the character recognition network, the drug image recognition method further includes: obtaining a character image Image and text recognition learner, the text image includes a third image and a fourth image, Wherein, the text image is a text image related to any drug; the text recognition learner is trained using the third image to obtain a second pre-training network, wherein the second pre-training The network includes a convolutional neural network model and a recurrent neural network model; the second loss value of the second pre-training network is calculated, and backpropagation is performed through the second loss value, and the second pre-trained The parameters of the training network are adjusted several times until the second pre-training model reaches convergence, and the adjustment is stopped to obtain the character recognition network. The drug image recognition method according to claim 3, wherein said generating a complex character feature matrix based on said multiple target images and said character recognition network includes: performing color conversion on each target image to obtain multiple gray-scale text images; binarize each gray-scale text image to obtain multiple binarized images; filter each binarized image to obtain multiple filtered images; Locate the position of the drug text in each filtered image to obtain the text position; select a text image from each target image according to the text position; input each text image to the convolutional neural network model performing feature extraction to obtain a feature sequence; inputting the feature sequence into the cyclic neural network model to obtain the complex character feature matrix. The drug image recognition method according to claim 1, wherein the generating the reference matrix according to each image feature matrix and the corresponding text feature matrix includes: adding each image feature matrix and the corresponding text feature matrix, The reference matrix is obtained, wherein each image feature matrix and the corresponding text feature matrix have the same number of rows and columns. The drug image recognition method according to claim 1, wherein the generating the recognition result of the drug image to be tested according to the similarity between the matrix to be tested and each reference matrix includes: calculating the matrix to be tested and the The similarity of each reference matrix; determining the reference matrix corresponding to the maximum similarity as the target matrix; performing mapping processing on the target matrix based on the preset label mapping table to obtain the identification result. A computer device, wherein the computer device includes: a memory storing at least one instruction; and a processor obtaining the instruction stored in the memory to realize the drug image as described in any one of claims 1 to 6 identification method. A computer-readable storage medium, wherein: at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in a computer device to implement any one of claims 1 to 6. drug image recognition method.

Images