WO2021039437A1 - Dispositif de traitement d'image, terminal portable, procédé de traitement d'image, et programme - Google Patents

Dispositif de traitement d'image, terminal portable, procédé de traitement d'image, et programme Download PDF

Info

Publication number
WO2021039437A1
WO2021039437A1 PCT/JP2020/030872 JP2020030872W WO2021039437A1 WO 2021039437 A1 WO2021039437 A1 WO 2021039437A1 JP 2020030872 W JP2020030872 W JP 2020030872W WO 2021039437 A1 WO2021039437 A1 WO 2021039437A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
unit
image processing
drug
recognition
Prior art date
Application number
PCT/JP2020/030872
Other languages
English (en)
Japanese (ja)
Inventor
真司 羽田
Original Assignee
富士フイルム富山化学株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 富士フイルム富山化学株式会社 filed Critical 富士フイルム富山化学株式会社
Priority to JP2021542745A priority Critical patent/JP7225416B2/ja
Publication of WO2021039437A1 publication Critical patent/WO2021039437A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition

Definitions

  • the present invention relates to an image processing device, a mobile terminal, an image processing method and a program, and particularly relates to a technique for recognizing a stamp or print added to a recognition object.
  • Patent Document 1 a drug recognition device capable of accurately recognizing the type of drug with a stamp.
  • the lighting unit capable of illuminating the engraved drug from a plurality of lighting directions surrounding the drug switches the lighting direction for illuminating the drug in order.
  • the imaging unit repeatedly photographs the drug each time the illumination direction of the illumination unit is switched.
  • the feature image extraction unit analyzes the captured image (drug image) for each illumination direction acquired by the photographing unit, and extracts the feature image corresponding to the shadow of the engraving for each drug image.
  • the feature image integration unit integrates the feature images for each illumination direction extracted by the feature image extraction unit to generate an integrated image.
  • the recognition unit recognizes the marking included in the integrated image generated by the feature image integration unit, and recognizes the type of the drug based on the recognition result of the marking.
  • the drug recognition device described in Patent Document 1 can acquire an integrated image in which the engraving is emphasized, but the device becomes large because it requires a plurality of lighting units having different illumination directions for the drug.
  • the present invention has been made in view of such circumstances, and is an image processing device, a mobile terminal, an image processing method, and a program capable of easily acquiring an image in which the marking or printing emphasized on the recognition object is emphasized.
  • the purpose is to provide.
  • the image processing apparatus is a learning data set of a plurality of different recognition objects to which marking or printing is added, and the marking or printing of the recognition target is emphasized.
  • a recognizer that has been machine-learned by a learning data set for learning, which is a set of an unmarked first image and a second image with embossed or printed emphasis, and an arbitrary recognition target with engraved or printed images.
  • An image input unit that causes the recognizer to input a third image that is an image of an object and whose marking or printing is not emphasized, and outputs a recognition result obtained from the recognizer when the third image is input to the recognizer.
  • An image output unit is provided.
  • an image of an arbitrary recognition object to which an engraving or printing is added is input to the recognizer.
  • the recognition result indicating the marking or printing can be output.
  • the recognition result is a fourth image in which the marking or printing added to an arbitrary recognition object is emphasized.
  • an image generation unit that combines a third image and a fourth image to generate a fifth image in which engraving or printing is emphasized.
  • the image output unit outputs the recognition result to the display unit and displays the recognition result on the display unit.
  • the recognition target is a drug.
  • the image output unit outputs the recognition result to the drug recognition device.
  • the recognizer is composed of a convolutional neural network in which the first image of the training data set is used as an input image and the second image is used as an output image for machine learning. Is preferable.
  • the second image included in the training data set is a recognition target based on a plurality of images of the recognition target having different illumination directions of light on the recognition target. It is preferable to include an image that has been subjected to an enhancement process that emphasizes the marking or printing added to the object.
  • the image input unit includes a camera unit that captures an image including an arbitrary recognition object, and a region corresponding to the recognition object from the captured image captured by the camera unit. It is preferable to include an image extraction unit for extracting the image and input the image extracted by the image extraction unit to the recognizer as a third image.
  • the invention according to still another aspect is a mobile terminal provided with the above-mentioned image processing device.
  • the image processing method is a learning data set of a plurality of different recognition objects to which marking or printing is added, and the marking or printing of the recognition target is not emphasized.
  • a step of preparing a learning data set for learning which is a set of a second image in which the marking or printing is emphasized, a step of causing the recognizer to perform machine learning by the learning data set, and a step of marking or printing are added.
  • the recognition result is a fourth image in which the marking or printing added to an arbitrary recognition object is emphasized.
  • the image generation unit includes a step of synthesizing the third image and the fourth image to generate a fifth image in which the marking or printing is emphasized. ..
  • the step of outputting the recognition result is to output the recognition result to the display unit and display the recognition result on the display unit.
  • the object to be recognized is a drug.
  • the recognition result is output to the drug recognition device in the step of outputting the recognition result.
  • the second image included in the training data set is a recognition target based on a plurality of images of the recognition target having different illumination directions of light on the recognition target. It is preferable to include an image that has been subjected to an enhancement process that emphasizes the marking or printing added to the object.
  • the program according to still another aspect of the present invention is installed in a computer to make the computer function as the above-mentioned image processing device.
  • FIG. 1 is a system configuration diagram showing an embodiment of a drug identification system including a mobile terminal according to the present invention.
  • FIG. 2 is an external view of a smartphone constituting the drug identification system shown in FIG.
  • FIG. 3 is a block diagram showing an internal configuration of the smartphone shown in FIG.
  • FIG. 4 is a block diagram showing the electrical configuration of the drug identification system shown in FIG.
  • FIG. 5 is a block diagram showing a hardware configuration of an image processing device including a machine learning device.
  • FIG. 6 is a diagram showing an example of a learning data set stored in the database shown in FIG.
  • FIG. 7 is a functional block diagram showing the functions of the machine learning device, which is a main component of the image processing device shown in FIG. FIG.
  • FIG. 8 is a flowchart showing an embodiment of the image processing method according to the present invention, and in particular, is a diagram showing processing of a learning phase in a machine learning device.
  • FIG. 9 is a flowchart showing an embodiment of the image processing method according to the present invention, and in particular, is a diagram showing processing of a drug recognition phase by a smartphone.
  • FIG. 1 is a system configuration diagram showing an embodiment of a drug identification system including a mobile terminal according to the present invention.
  • the drug identification system is composed of a smartphone 100 which is a mobile terminal with a camera and a server 200 which functions as a drug identification device.
  • the smartphone 100 and the server 200 are connected to the Internet and a LAN (Local Area Network). Is connected so that data communication is possible via the network 2 such as.
  • LAN Local Area Network
  • the smartphone 100 has a camera unit, and the camera unit captures the drug 10 which is a recognition target.
  • the smartphone 100 includes an image processing device according to the present invention that processes an image (third image) of the photographed drug 10, and displays an image (fourth image) after image processing by the image processing device on the display unit. Or, it is transmitted to the server 200 via the network 2. The details of the image processing device will be described later.
  • the server 200 identifies the drug 10 based on the fourth image of the drug 10 uploaded from the smartphone 100, and outputs the identification result (for example, drug identification information consisting of a drug name, a product name, an abbreviation, or a combination thereof). ,
  • the fourth image of the drug 10 is transmitted to the smartphone 100 that has transmitted the image.
  • identification code information for identifying the type of drug is attached to the surface of the drug (tablet).
  • This identification code information is generally attached by engraving or printing (printing).
  • the server 200 can improve the discriminating power of the drug by using the identification code information attached to the drug.
  • the engraving on the drug means that the identification code information is formed by forming a groove, which is a depressed region, on the surface of the drug.
  • the groove is not limited to the one formed by digging the surface, and may be formed by pressing the surface. Further, the engraving may include a marking that does not have an identification function such as a score line.
  • the printing attached to the drug means that the identification code information is formed by applying edible ink or the like to the surface of the drug in contact or non-contact.
  • attachmented by printing is synonymous with “attached by printing.”
  • the smartphone 100 shown in FIG. 2 has a flat-plate housing 102, and a display panel 121 as a display unit and an operation panel 122 as an input unit are integrally formed on one surface of the housing 102.
  • Display unit 120 is provided.
  • the display panel 121 is composed of a liquid crystal panel, and the display unit 120 of this example is a liquid crystal display.
  • the housing 102 includes a speaker 131, a microphone 132, an operation unit 140, and a camera unit 141.
  • the camera unit 141 includes at least one of a camera (in-camera) provided on the same surface side as the display unit 120 and a camera (out-camera (not shown)) provided on the surface side opposite to the display unit 120.
  • FIG. 3 is a block diagram showing the internal configuration of the smartphone 100 shown in FIG.
  • the smartphone 100 has, as main components, a wireless communication unit 110, a display unit 120, a call unit 130, an operation unit 140, a camera unit 141, a storage unit 150, and an external input / output unit. It includes 160 (image output unit), a GPS (global positioning system) receiving unit 170, a motion sensor unit 180, a power supply unit 190, and a main control unit 101. Further, as a main function of the smartphone 100, it is provided with a wireless communication function for performing mobile wireless communication via a base station device and a mobile communication network.
  • a wireless communication function for performing mobile wireless communication via a base station device and a mobile communication network.
  • the wireless communication unit 110 performs wireless communication with the base station device connected to the mobile communication network according to the instruction of the main control unit 101.
  • the wireless communication is used to send and receive various file data such as voice data and image data, e-mail data, and receive web data and streaming data.
  • the display unit 120 is a display with a so-called touch panel provided with an operation panel 122 arranged on the screen of the display panel 121, and displays images (still images and moving images), character information, and the like under the control of the main control unit 101.
  • the information is visually transmitted to the user, and the user operation on the displayed information is detected.
  • the display panel 121 uses an LCD (Liquid Crystal Display) as a display device.
  • the display panel 121 is not limited to the LCD, and may be, for example, an OLED (organic light emission diode).
  • the operation panel 122 is a device provided in a state in which an image displayed on the display surface of the display panel 121 can be visually recognized, and detects one or a plurality of coordinates operated by a user's finger or a stylus.
  • the operation panel 122 outputs a detection signal generated due to the operation to the main control unit 101.
  • the main control unit 101 detects the operation position (coordinates) on the display panel 121 based on the received detection signal.
  • the call unit 130 includes a speaker 131 and a microphone 132, converts a user's voice input through the microphone 132 into voice data that can be processed by the main control unit 101, and outputs the data to the main control unit 101, or a wireless communication unit.
  • the audio data received by the 110 or the external input / output unit 160 is decoded and output from the speaker 131.
  • the operation unit 140 is a hardware key using a key switch or the like, and receives an instruction from the user.
  • the operation unit 140 is mounted on the side surface of the housing 102 of the smartphone 100, and is switched on when pressed with a finger or the like, and switched off by a restoring force such as a spring when the finger is released. It is a push button type switch that is in a state.
  • the storage unit 150 includes the control program and control data of the main control unit 101, address data associated with the name and telephone number of the communication partner, transmitted / received e-mail data, web data downloaded by web browsing, and downloaded contents. Data etc. are stored, and streaming data etc. are temporarily stored.
  • the storage unit 150 is composed of an internal storage unit 151 and an external storage unit 152 having a detachable external memory slot.
  • Each of the internal storage unit 151 and the external storage unit 152 constituting the storage unit 150 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM (Random Access Memory), or a ROM (Read). It is realized by using a storage medium such as Only Memory).
  • the external input / output unit 160 serves as an interface with all external devices connected to the smartphone 100, and is used for communication (for example, USB (Universal Serial Bus), IEEE 1394, etc.) or network (for example, wireless LAN (Local Area)). Connect directly or indirectly to other external devices via Network) or Bluetooth (registered trademark).
  • USB Universal Serial Bus
  • IEEE 1394 IEEE 1394
  • network for example, wireless LAN (Local Area)
  • Bluetooth registered trademark
  • the GPS receiving unit 170 receives GPS signals transmitted from the GPS satellites ST1 and ST2 to STn according to the instruction of the main control unit 101, executes positioning calculation processing based on the received plurality of GPS signals, and executes positioning calculation processing based on the received GPS signals, and the latitude of the smartphone 100. , Acquires position information (GPS information) specified by longitude and altitude.
  • GPS information position information specified by longitude and altitude.
  • the GPS receiving unit 170 can acquire the position information from the wireless communication unit 110 and / or the external input / output unit 160 (for example, wireless LAN), the GPS receiving unit 170 can also detect the position using the position information.
  • the motion sensor unit 180 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 100 according to the instruction of the main control unit 101. By detecting the physical movement of the smartphone 100, the moving direction and acceleration of the smartphone 100 are detected. The result of the detection is output to the main control unit 101.
  • the power supply unit 190 supplies the electric power stored in the battery (not shown) to each unit of the smartphone 100 according to the instruction of the main control unit 101.
  • the main control unit 101 includes a microprocessor, operates according to the control program and control data stored in the storage unit 150, and controls each part of the smartphone 100 in an integrated manner.
  • the main control unit 101 includes a mobile communication control function that controls each unit of the communication system and a software processing function in order to perform voice communication and data communication through the wireless communication unit 110.
  • the software processing function is realized by operating the main control unit 101 according to the software (program) stored in the storage unit 150.
  • the software processing function includes, for example, an e-mail function for sending and receiving e-mail by controlling an external input / output unit 160, a web browsing function for browsing a web page, and an image processing device according to the present invention for the smartphone 100.
  • the software that causes the smartphone 100 to function as the image processing device according to the present invention downloads the corresponding software from the server 200 that functions as the drug identification device or the site of the business operator that operates the server 200. By doing so, it can be installed on the smartphone 100.
  • the main control unit 101 has an image processing function such as displaying an image on the display unit 120 based on image data (still image or moving image data) such as received data or downloaded streaming data.
  • image data still image or moving image data
  • the main control unit 101 executes display control for the display unit 120 and operation detection control for detecting a user operation through the operation unit 140 and the operation panel 122.
  • the camera unit 141 converts the image data obtained by imaging into compressed image data such as JPEG (Joint Photographic Experts Group), and records the image data in the storage unit 150. It can be output through the external input / output unit 160 or the wireless communication unit 110.
  • compressed image data such as JPEG (Joint Photographic Experts Group)
  • the camera unit 141 can be used for various functions of the smartphone 100. In this example, when identifying a drug, it is used for photographing the drug. The image from the camera unit 141 can also be used in the software.
  • FIG. 4 is a block diagram showing the electrical configuration of the drug identification system shown in FIG.
  • a program (application) according to the present invention is installed in the smartphone 100, and the main control unit 101 of the smartphone 100 executes this application to execute the image extraction unit 101A, the recognizer 101B, and the image generation unit 101C. And functions as a communication control unit 101D.
  • the camera unit 141 and the image extraction unit 101A function as an image input unit for inputting an image (third image) of the drug into the recognizer 101B.
  • the photographed image of the drug photographed by the camera unit 141 is input to the main control unit 101.
  • the image extraction unit of the main control unit 101 extracts a region corresponding to the drug, which is a recognition target, from the input captured image, and causes the recognizer 101B to input an image (drug image) of the extracted region.
  • the drug image is preferably extracted (cut out) by detecting the outer shape of the drug and cutting out according to the outer shape of the drug. For example, a rectangular region inscribed by the outer shape of the drug can be cut out.
  • the recognizer 101B can apply a convolutional neural network (CNN: Convolution Neural Network), which is one of the deep learning models.
  • CNN Convolution Neural Network
  • the recognizer 101B is a learning data set of a plurality of different drugs to which the marking or printing is added, and the image (first image) in which the marking or printing of the drug is not emphasized and the marking or printing of the drug are emphasized.
  • Machine learning was performed using a learning data set for learning, which is a set of the image (second image).
  • the recognizer 101B does not need to have a learning function by itself, and is configured as a learned model by acquiring the parameters of a model (CNN) in which machine learning is performed by an external machine learning device. It may be a new one.
  • FIG. 5 is a block diagram showing a hardware configuration of an image processing device 300 including a machine learning device.
  • the image processing device 300 shown in FIG. 5 a personal computer or a workstation can be used.
  • the image processing device 300 of this example mainly includes an image input unit 312, a database 314, a storage unit 316, an operation unit 318, a CPU (Central Processing Unit) 320, a RAM (Random Access Memory) 322, and a ROM ( It is composed of a Read Only Memory) 324 and a display unit 326.
  • a CPU Central Processing Unit
  • RAM Random Access Memory
  • ROM It is composed of a Read Only Memory 324 and a display unit 326.
  • the image input unit 312 is a part for inputting an image of a recognition object (“drug” in this example) to which a stamp or print is added, and inputting a learning data set or the like to be stored in the database 314.
  • Database 314 is a storage unit that stores the learning data set.
  • FIG. 6 is a diagram showing an example of a learning data set stored in the database 314 shown in FIG.
  • the learning data set is a set of images of a plurality of drugs of different types (first image 25) and an image in which the marking or printing of each drug corresponding to the first image 25 is emphasized (second image 27). ing.
  • the first image 25 and the second image 27 are input images and correct answer data used during machine learning of the learning model, respectively.
  • the first image 25 can be collected by photographing the drug. Generally, the marking in the first image 25 is not clearly shown.
  • the second image 27 is an image showing the marking or printing of the drug.
  • the second image 27 can be obtained by displaying the first image 25 on the display unit 326 and the user using the operation unit 318 to fill the engraved portion or the printed portion on the screen of the display unit 326.
  • the second image is not limited to the one created manually, but uses an integrated image (an image with enhancement processing for emphasizing engraving or printing) generated by the drug recognition device described in Patent Document 1 and the like. can do. That is, as the second image 27, it is possible to use an image that has been subjected to an enhancement process that emphasizes the marking or printing added to the drug, based on a plurality of images of the drug having different illumination directions of the light on the drug. it can.
  • FIG. 7 is a functional block diagram showing the functions of the machine learning device 350, which is a main component of the image processing device 300 shown in FIG. 5, and includes the CPU 320, the storage unit 316, the RAM 322, the ROM 324, and the like shown in FIG. It consists of hardware.
  • the machine learning device 350 mainly includes a recognizer 352, a loss value calculation unit 354 and a parameter control unit 356 that function as a learning unit that causes the recognizer 352 to perform machine learning.
  • the CNN model is applied to the recognizer 352 in this example.
  • the recognizer 352 has a plurality of layer structures and holds a plurality of parameters.
  • the recognizer 352 can change from an unlearned model to a trained model by updating the parameters from the initial values to the optimum values.
  • the initial value of the parameter of the recognizer 352 may be an arbitrary value, or for example, the parameter of the trained model of the image system for classifying images may be applied. In the latter case, good machine learning can be performed with a relatively small number of training data sets by performing transfer learning using the learning data set shown in FIG.
  • the recognizer 352 includes an input layer 352A, an intermediate layer 352B having a plurality of sets composed of a convolutional layer and a pooling layer, and an output layer 352C, and a plurality of "nodes” are connected by “edges” in each layer. It has a structure.
  • the first image 25 of the learning data set (FIG. 6) is input to the input layer 352A as an input image.
  • the intermediate layer 352B has a plurality of sets including a convolutional layer and a pooling layer as one set, and is a portion for extracting features from the first image 25 input from the input layer 352A.
  • the convolutional layer filters nearby nodes in the previous layer (performs a convolutional operation using the filter) and acquires a "feature map".
  • the pooling layer reduces the feature map output from the convolution layer to a new feature map.
  • the "convolution layer” plays a role of feature extraction such as edge extraction from an image, and the "pooling layer” plays a role of imparting robustness so that the extracted features are not affected by translation or the like.
  • the intermediate layer 352B is not limited to the case where the convolutional layer and the pooling layer are set as one set, but also includes the case where the convolutional layers are continuous and the normalization layer.
  • the convolution layer in the final stage is a feature map (image) having the same size as the input image, and is a portion that outputs a feature map showing the features (engraving, etc.) of the drug.
  • the output layer 352C is a portion that outputs the recognition result of the recognizer 352 (in this example, an image in which the marking or the like is emphasized).
  • the loss value calculation unit 354 acquires the recognition result (output image) output from the output layer 352C of the recognizer 352 and the second image 27 (correct answer data) paired with the first image 25, and the loss between the two. Calculate the value.
  • a method of calculating the loss value for example, a Jaccard coefficient or a dice coefficient can be used.
  • the parameter control unit 356 minimizes or is similar to the distance between the correct answer data and the output of the recognizer 352 in the feature space by the error back propagation method based on the loss value calculated by the loss value calculation unit 354.
  • the parameters of the recognizer 352 (such as the coefficient of the filter of each convolution layer) are adjusted in order to maximize the degree.
  • the trained recognizer 352 uses an image of an arbitrary drug (third image) acquired by the image input unit 312 as an input image, recognizes the marking of the drug from the input image, and recognizes the recognition result (third). 4 images) is output to the image output unit 360.
  • the recognizer 101B of the smartphone 100 acquires the same parameters as the parameters of the learned recognizer 352 from the machine learning device 350 shown in FIG. 7, and learns by setting the acquired parameters. It has the same recognition function as the existing recognizer 352.
  • the image generation unit 101C is recognized by the recognizer 101B and the third image of the drug (the image of the drug to be recognized whose marking or printing is not emphasized) taken by the camera unit 141 and extracted by the image extraction unit 101A.
  • the recognition result (fourth image) is combined to generate a composite image (fifth image) in which the marking or printing of the drug is emphasized.
  • the fourth image is an image showing only the marking or printing of the drug, like the second image 27 shown in FIG. 6, and is an image having high brightness of the stamped portion or the printed portion. Therefore, the image generation unit 101C can generate a fifth image in which the engraved or printed portion of the drug is highlighted in black by subtracting the fourth image from the third image. In the case of a third image having low brightness (for example, an image obtained by photographing a black drug), the image generation unit 101C adds the fourth image to the third image to emphasize the stamped or printed portion of the drug in white. The fifth image can be generated.
  • the display control unit (not shown) that functions as an image output unit outputs the recognition result (fourth image) by the recognizer 101B or the fifth image including the fourth image to the display unit 120 and displays it on the display unit 120. Let me.
  • the user can display the fourth image or the fifth image on the display unit 120 of the smartphone 100 by photographing the drug with the smartphone 100, and is added to the drug by the fourth image or the fifth image.
  • the engraving or printing can be easily visually recognized.
  • the recognition result (fourth image) by the recognizer 101B or the fifth image including the fourth image emphasizes the marking or printing added to the drug, and is therefore suitable for discriminating or auditing the drug. Is.
  • the communication control unit 101D and the wireless communication unit 110 which function as image output units, transmit the recognition result (fourth image) by the recognizer 101B or the fifth image including the fourth image to the server via the network 2. It is transmitted to the 200, and the identification result of the drug to be identified, which is identified by the server 200 based on the fourth image or the fifth image, is acquired via the network 2.
  • the server 200 shown in FIG. 4 functions as a drug identification device, and is mainly composed of a communication unit 210, a CPU (Central Processing Unit) 220, a drug DB (database) 230, a memory 240, and a drug identification unit 250. There is.
  • the CPU 220 is a part that controls each part of the server 200, and functions the communication unit 210 as an image receiving unit that receives the fourth image or the fifth image of the drug transmitted from the smartphone 100, and receives the fourth image or the fourth image.
  • the drug identification unit 250 executes the drug identification process based on the image.
  • the drug DB 230 is a part for registering and managing drug images (drug images on the front side and back side of the drug) in association with drug identification information such as the name of the drug.
  • the drug image of the drug (registered drug) registered in the drug DB 230 is used as a template image for identifying which of the registered drugs the registered drug corresponds to the drug to be identified.
  • the memory 240 includes a storage unit in which a program for providing a drug identification service is stored, and a portion serving as a work area of the CPU 220.
  • the drug identification unit 250 performs template matching between the image (fourth image or fifth image) of the drug to be identified received via the communication unit 210 and the template image of the registered drug registered in the drug DB 230, and matches them. Acquire identification results such as drug identification information (including images of registered drugs) of the registered drug having the maximum degree or a plurality of registered drugs having a high degree of matching.
  • the CPU 220 transmits the drug identification result by the drug identification unit 250 to the smartphone 100 that transmitted the fourth image or the fifth image via the communication unit 210.
  • the server 200 is provided with the function of the smartphone 100 that generates the fourth image or the fifth image, the fourth image or the fifth image generated by the server 200 is transmitted to the smartphone 100, and the identification result of the drug is transmitted to the smartphone. It may be sent to 100.
  • the smartphone 100 captures an image of the drug to be identified and transmits the captured drug image to the server 200 as it is, thereby acquiring or capturing an image in which the engraving or printing is emphasized from the server 200.
  • the drug recognition result can be obtained from the server 200.
  • ⁇ Image processing method> 8 and 9 are flowcharts showing embodiments of the image processing method according to the present invention, respectively.
  • FIG. 8 shows the processing of the learning phase in the machine learning device 350 shown in FIG. 7.
  • a learning data set of a plurality of different drugs with engraving or printing is prepared (step S10).
  • the learning data set is a learning data set for machine learning that includes a first image 25 in which the marking or printing is not emphasized and a second image 27 in which the marking or printing is emphasized as shown in FIG. Yes, it is stored in database 314 (FIG. 5).
  • the machine learning device 350 causes the recognizer 352 to perform machine learning using the learning data set stored in the database 314 (step S12).
  • FIG. 9 shows the processing of the drug recognition phase by the smartphone 100 shown in FIG. 4 and the like.
  • the smartphone 100 of this example includes a recognizer 101B in which the same parameters as those of the learned recognizer 252 are set.
  • the recognizer 101B has the same recognition function as the learned recognizer 252.
  • an image (third image) of an arbitrary recognition target drug to which a stamp or print is added from the image input unit is input to the recognizer 101B as an input image (step S20). That is, the drug to be recognized is photographed by the camera unit 141 that functions as an image input unit, an image (drug image) of the region corresponding to the drug is extracted from the captured image, and the extracted drug image (third image) is used as a recognizer. Let 101B input.
  • the recognizer 101B outputs an image (fourth image) showing the marking or printing added to the drug to be recognized as the recognition result for the input third image (step S22).
  • the image generation unit 101C synthesizes the third image (drug image) and the fourth image output from the recognizer 101B, and generates a composite image (fifth image) in which the marking or printing of the drug is emphasized (step). S24).
  • the display control unit that functions as an image output unit outputs the fifth image generated in step S24 to the display unit 120, and causes the display unit 120 to display the fifth image in which the marking or printing of the drug to be recognized is emphasized. (Step S26).
  • the user can display the fifth image on the display unit 120 of the smartphone 100 by photographing the drug with the smartphone 100, and easily visually recognizes the marking or printing added to the drug by the fifth image. be able to.
  • the communication control unit 101D and the wireless communication unit 110 that function as the image output unit transmit the fifth image generated in step S24 to the server 200 via the network 2 (step S28).
  • the server 200 acquires identification results such as drug identification information such as the name of the drug to be recognized based on the fifth image, transmits the acquired identification results to the smartphone 100, and the smartphone 100 identifies the drug from the server 200. Receive the result (step S30).
  • the display control unit of the smartphone 100 outputs the drug identification result received from the server 200 to the display unit 120, and displays the drug identification result on the display unit 120 (step S32).
  • the user can display the identification result such as the drug name of the drug on the display unit 120 of the smartphone 100 by photographing the drug with the smartphone 100.
  • the image processing device of the present embodiment can be incorporated into a drug recognition device, whereby the drug recognition device can be miniaturized and inexpensive.
  • the mobile terminal according to the present invention is not limited to a smartphone, but may be a tablet terminal having a camera function, a mobile phone, a PDA (Personal Digital Assistants), or the like.
  • the drug is used as a recognition object, but the present invention is not limited to this, and the present invention can be applied to the recognition of other recognition objects such as metal parts with engravings and precious metals.
  • processors include CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), which are general-purpose processors that execute programs and function as various processing units.
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • FPGA Field Programmable Gate Array
  • a dedicated electric circuit that is a processor having a circuit configuration specially designed to execute a specific process such as a programmable logic device (PLD) or an ASIC (Application Specific Integrated Circuit), which is a processor that can change the CPU. Etc. are included.
  • One processing unit constituting the image processing device may be composed of one of the above-mentioned various processors, or may be composed of two or more processors of the same type or different types.
  • one processing unit may be composed of a plurality of FPGAs or a combination of a CPU and an FPGA.
  • a plurality of processing units may be configured by one processor.
  • a processor As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which this processor functions as a plurality of processing units.
  • SoC System On Chip
  • SoC System On Chip
  • the various processing units are configured by using one or more of the above-mentioned various processors as a hardware structure.
  • the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.
  • the present invention includes a program that causes the computer to function as an image processing device according to the present invention by being installed in the computer, and a storage medium in which this program is recorded.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un dispositif de traitement d'image, un terminal portable, un procédé de traitement d'image et un programme au moyen desquels il est possible d'acquérir facilement une image dans laquelle le marquage ou l'impression ajouté à un objet à reconnaître est mis en évidence. Un téléphone intelligent (100) comprend : un dispositif de reconnaissance (101B) qui a été formé par apprentissage automatique à l'aide d'un ensemble de données d'apprentissage destiné à l'apprentissage d'une pluralité de médicaments différents qui ont été marqués ou sur lesquels l'on a imprimé, un ensemble étant formé par une première image de médicament dans laquelle le marquage ou l'impression n'est pas mis en évidence et une deuxième image de médicament dans laquelle le marquage ou l'impression est mis en évidence ; une unité de caméra (141) fonctionnant en tant qu'unité d'entrée d'image et servant à entrer, dans le dispositif de reconnaissance (101B), une troisième image qui est une image d'un médicament discrétionnaire qui a été marqué ou sur lequel l'on imprimé et dans lequel le marquage ou l'impression n'est pas mis en évidence ; et une unité de sortie d'image servant à fournir le résultat de reconnaissance obtenu du dispositif de reconnaissance (101B) à une unité d'affichage (120) lorsque la troisième image a été entrée dans le dispositif de reconnaissance (101B).
PCT/JP2020/030872 2019-08-27 2020-08-14 Dispositif de traitement d'image, terminal portable, procédé de traitement d'image, et programme WO2021039437A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2021542745A JP7225416B2 (ja) 2019-08-27 2020-08-14 画像処理装置、携帯端末、画像処理方法及びプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-154363 2019-08-27
JP2019154363 2019-08-27

Publications (1)

Publication Number Publication Date
WO2021039437A1 true WO2021039437A1 (fr) 2021-03-04

Family

ID=74684564

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/030872 WO2021039437A1 (fr) 2019-08-27 2020-08-14 Dispositif de traitement d'image, terminal portable, procédé de traitement d'image, et programme

Country Status (2)

Country Link
JP (1) JP7225416B2 (fr)
WO (1) WO2021039437A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017049974A (ja) * 2015-09-04 2017-03-09 キヤノン株式会社 識別器生成装置、良否判定方法、およびプログラム
JP2018027242A (ja) * 2016-08-18 2018-02-22 安川情報システム株式会社 錠剤検知方法、錠剤検知装置および錠剤検知プログラム
US20180260665A1 (en) * 2017-03-07 2018-09-13 Board Of Trustees Of Michigan State University Deep learning system for recognizing pills in images

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3610843A4 (fr) * 2017-04-14 2021-01-27 Yuyama Mfg. Co., Ltd. Dispositif de tri de médicaments, récipient de tri et procédé de retour de médicaments
WO2019039016A1 (fr) * 2017-08-25 2019-02-28 富士フイルム株式会社 Dispositif d'assistance d'inspection de médicament, dispositif de traitement d'image, procédé de traitement d'image et programme

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017049974A (ja) * 2015-09-04 2017-03-09 キヤノン株式会社 識別器生成装置、良否判定方法、およびプログラム
JP2018027242A (ja) * 2016-08-18 2018-02-22 安川情報システム株式会社 錠剤検知方法、錠剤検知装置および錠剤検知プログラム
US20180260665A1 (en) * 2017-03-07 2018-09-13 Board Of Trustees Of Michigan State University Deep learning system for recognizing pills in images

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ZENG,XIAO ET AL.: "MobileDeepPill: A Small-Footprint Mobile Deep Learning System for Recognizing Unconstrained Pill Images", PROCEEDINGS OF THE 15TH ANNUAL INTERNATIONAL CONFERENCE ON MOBILE SYSTEMS,APPLICATIONS,AND SERVICES, 23 June 2017 (2017-06-23), pages 56 - 67, XP058369188, DOI: 10.1145/3081333.3081336 *

Also Published As

Publication number Publication date
JPWO2021039437A1 (fr) 2021-03-04
JP7225416B2 (ja) 2023-02-20

Similar Documents

Publication Publication Date Title
CN109409161B (zh) 图形码识别方法、装置、终端及存储介质
US10780347B2 (en) Print system, server, print method, and program
US8094139B2 (en) Method in electronic pen, computer program product, and electronic pen
JP2018067315A (ja) ユーザー編集イメージを用いた拡張現実コンテンツの提供方法及びシステム
WO2020162262A1 (fr) Système d'identification de médicament, dispositif d'identification de médicament, procédé d'identification de médicament, et programme
US10931880B2 (en) Electronic device and method for providing information thereof
JP7286208B2 (ja) 生体顔検出方法、生体顔検出装置、電子機器、及びコンピュータプログラム
CN109033276A (zh) 贴纸的推送方法、装置、存储介质及电子设备
CN111339737A (zh) 实体链接方法、装置、设备及存储介质
CN110378318B (zh) 文字识别方法、装置、计算机设备及存储介质
CN110442879A (zh) 一种内容翻译的方法和终端
JP5754293B2 (ja) 記入支援システム及びサーバ装置
WO2021039437A1 (fr) Dispositif de traitement d'image, terminal portable, procédé de traitement d'image, et programme
US20230368552A1 (en) Drug identification device, drug identification method and program, drug identification system, drug loading table, illumination device, imaging assistance device, trained model, and learning device
CN112053360A (zh) 图像分割方法、装置、计算机设备及存储介质
CN112818979A (zh) 文本识别方法、装置、设备及存储介质
CN110717060A (zh) 图像mask的过滤方法、装置及存储介质
CN114333029A (zh) 模板图像生成方法、装置及存储介质
CN113298040A (zh) 关键点检测方法、装置、电子设备及计算机可读存储介质
CN111479060B (zh) 图像采集方法、装置、存储介质及电子设备
CN114550185B (zh) 一种文档生成的方法、相关装置、设备以及存储介质
US20240112476A1 (en) Drug identification apparatus, drug identification method and program
JP2023119446A (ja) 医療品情報登録システム、医療品情報登録方法、情報処理装置、医療情報システム及びプログラム
JP2024000578A (ja) 画像補正装置、文字認識装置、画像補正方法、文字認識方法、プログラム及び記録媒体
JP6317714B2 (ja) プリント受付装置、プリント受付サーバ、及びプリント受付方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20858172

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021542745

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20858172

Country of ref document: EP

Kind code of ref document: A1