Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0004—Industrial image inspection
  • G06T7/001—Industrial image inspection using an image reference approach
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
  • G01N21/9508—Capsules; Tablets
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/8806—Specially adapted optical and illumination features
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
  • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10024—Color image

Definitions

• the invention relates to inspecting medicine objects, in particular pouches comprising medicaments, based on hyperspectral imaging, and, in particular, though not exclusively, to methods and systems for inspecting medicine objects based on hyperspectral imaging and a computer program product for executing such methods.
• Patients are provided with medicaments according to a prescription. Especially, people with a chronic disease periodically need to take the same medicines over a long period of time. Often patients need to take a combination of different medicaments, i.e. pills, tablets and/or capsules.
• the medicaments may be packed into pouches, e.g. a transparent plastic pouches, blisters or bags, according to the prescription using an automated packaging system. Incorrect packaging of a prescription may be result in the patient taking the wrong (combination of) medicaments or an incorrect dosage of medicaments, which may be harmful for the health of the patient.
• medicine objects are checked by an inspection system which is configured to inspect medicine objects using an image processing system, wherein medicine objects may represent e.g. pills and/or tablets, capsules, ampules or packets, blisters or pouches comprising medicine objects.
• An example of such inspection system is known from EP2951563.
• other inspection techniques may be considered.
• LIS2014/0319351 describes an example of an inline system for inspecting pills arranged in a blister package, based on near infrared NIR hyperspectral imaging. The inspection system illuminates pills in a blister package with light of a halogen lamp and a hyperspectral image sensor then detects fifteen response values for fifteen bands in the NIR spectrum. The response values are processed to determine parts of the response values belonging to responses of the pills. These parts are then compared to a reference in order to determine if the pills contain the correct composition.
• medicine objects in medicine pouches may include different medicine objects of different size, shape and composition which are spatially distributed in a random order. Medicine objects may be arranged on their side, next to each other or (partly) over each other, while the transparent pouch material may introduce errors in the measured data.
• the NIR response of medicaments is a relatively weak signal because most medicaments largely consist of the same ingredients (coating, binder material, etc.) which often account for a large part of the mass of the pill. Therefore, instead of 15 values as mentioned in the prior art, large numbers, e.g. a few hundred or more, spectral response values per pixel are needed to distinguish different medicaments.
• hyperspectral image data typically includes a block of data (a data stack) of a considerable amount of data, e.g. more than 100 Mbyte per picture, that needs to be analyzed in real-time. Methods in the prior art for processing the hyperspectral data of imaged medicament pouches are not suitable for that purpose.
• aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit,” “module” or “system.” Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.
• the methods, systems, modules, functions and/or algorithms described with reference to the embodiments in this application may be realized in hardware, software, or a combination of hardware and software.
• the methods, systems, modules, functions and/or algorithms may be realized in a centralized fashion in at least one computing system, or in a distributed fashion where different elements are spread across several interconnected computing systems. Any kind of computing system or other apparatus adapted for carrying out the embodiments (or parts thereof) described in this application is suited.
• a typical implementation may comprise one or more digital circuits such as application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), and/or one or more processors (e.g., x86, x64, ARM, PIC, and/or any other suitable processor architecture) and associated supporting circuitry (e.g., storage, DRAM, FLASH, bus interface circuits, etc.).
• ASICs application specific integrated circuits
• FPGAs field programmable gate arrays
• processors e.g., x86, x64, ARM, PIC, and/or any other suitable processor architecture
• associated supporting circuitry e.g., storage, DRAM, FLASH, bus interface circuits, etc.
• Each discrete ASIC, FPGA, processor, or other circuit may be referred to as “chip,” and multiple such circuits may be referred to as a “chipset.”
• the programmable logic devices may be provided with fast RAM, in particular block RAM (BRAM).
• BRAM block RAM
• Another implementation may comprise a non-transitory machine-readable (e.g., computer readable) medium (e.g., FLASH drive, optical disk, magnetic storage disk, or the like) having stored thereon one or more lines of code that, when executed by a machine, cause the machine to perform processes as described in this disclosure.
• a non-transitory machine-readable (e.g., computer readable) medium e.g., FLASH drive, optical disk, magnetic storage disk, or the like
• each block in a flowchart or a block diagrams may represent a module, segment, or portion of code, which may be implemented as software, hardware or a combination of software and hardware.
• the ability to accurately distinguish medications based on the substances (composition) is very important, since a very large number of medications are not visually distinct (very often round, white tablets).
• hyperspectral imaging may include the high spectral resolution (>200 bands instead of the three conventional color bands with RGB multispectral imaging), which allows detection of differences in otherwise similar objects in the visible spectrum. Additionally, it allows recognizing different medications based on the non-visible part (the near infrared part) of the electromagnetic spectrum.
• the invention may relate to a method for inspecting medicine objects comprising: capturing an image of medicine objects, preferably medicaments of different shapes, sizes and/or compositions, randomly arranged in a pouch, the image having a first spatial resolution; capturing hyperspectral image data of the medicine objects in the pouch, the hyperspectral image data having a second spatial resolution smaller than the first spatial resolution; determining blobs of pixels in the image of the first spatial resolution, each of the blobs of pixels representing one of the medicine objects; selecting at least one hyperspectral image data part from the hyperspectral image data based on at least one of the blobs of pixels in the image of the first spatial resolution; determining a hyperspectral fingerprint based on the hyperspectral image data part, the hyperspectral fingerprint being indicative of a spectral response of one or more chemical compounds in a medicine object; and, comparing the hyperspectral fingerprint with one or more reference fingerprints.
• the capturing of the hyperspectral image data may include exposing the medicine object to light having a continuous spectrum, preferably a continuous spectrum in the visible and/or near-infrared region of the electromagnetic spectrum.
• the hyperspectral data may include pixels, each pixel being associated with a plurality of spectral values, preferably the plurality of spectral values including spectral values in the visible and/or the near-infrared region of the electromagnetic spectrum.
• the one or more single or multi-band images may include a 2D grid of pixels, each pixel being associated with one or a few spectral values, preferably a spectral value selected from one or more spectral values, e.g. RGB values and/or an IR value.
• the hyperspectral image data may include line-scan hyperspectral image data, the line-scan hyperspectral image data including lines of pixels.
• the method may further comprise: localizing one or more groups of pixels associated with one or more medicine objects in the image based on a segmentation algorithm.
• selecting one or more hyperspectral image data parts may include: mapping each of the one or more groups of pixels onto the pixels of the hyperspectral image data.
• one or more of the following steps may be executed: removing background pixels (outliers) from the one or more hyperspectral image data using an algorithm, preferably a clustering algorithm; and, removing pixels that are contaminated with specular reflections and/or that are overexposed from the one or more hyperspectral image data.
• an algorithm preferably a clustering algorithm
• the determining one or more hyperspectral fingerprints may further comprise: reducing the dimension of the one or more hyperspectral image data parts, preferably based on a PCA methods; and, determining a fingerprint based on at least one of the one or more reduced hyperspectral image data parts.
• a camera system is used to capture the one or more single or multi-band images and hyperspectral image data, preferably the camera system including a multispectral camera and, optionally, a single or multi-band camera, such as a monochromatic or a color camera.
• the hyperspectral image data may be captured using a hyperspectral line scan camera, wherein during the capturing, the medicine object moves relative to the hyperspectral line scan camera, more preferably the medicine object moves through the field of view of the camera system.
• the invention may relate to a module for controlling a medicine inspection apparatus comprising an camera system, the module comprising a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: capturing an image of medicine objects, preferably medicaments of different shapes, sizes and/or compositions, randomly arranged in a pouch, the image having a first spatial resolution; capturing hyperspectral image data of the medicine objects in the pouch, the hyperspectral image data having a second spatial resolution smaller than the first spatial resolution; determining blobs of pixels in the image of the first spatial resolution, each of the blobs of pixels representing one of the medicine objects; selecting at least one hyperspectral image data part from the hyperspectral image data based on at least one of the blobs of pixels in the image of the first spatial resolution; determining a hyperspectral fingerprint based on the hyper
• the invention may relate to a medicine object inspection apparatus comprising: a camera system, and, a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: capturing an image of medicine objects, preferably medicaments of different shapes, sizes and/or compositions, randomly arranged in a pouch, the image having a first spatial resolution; capturing hyperspectral image data of the medicine objects in the pouch, the hyperspectral image data having a second spatial resolution smaller than the first spatial resolution; determining blobs of pixels in the image of the first spatial resolution, each of the blobs of pixels representing one of the medicine objects; selecting at least one hyperspectral image data part from the hyperspectral image a based on at least one of the blobs of pixels
• the hyperspectral data may be determined using a hyperspectral camera which may be configured to detect the spectral response of an imaged area in the near-infrared (NIR) part of the spectrum.
• the hyperspectral camera may be configured to detect the spectral response of an imaged area in both the visible and NIR part of the spectrum. In that case, the hyperspectral camera may generate image data both in the visible range and in the NIR range. If the hyperspectral camera is configured to generate both NIR and visible spectral values for each pixel.
• a separate multispectral camera e.g. an RGB or RGB/IR camera is no longer needed.
• one or more slices of spectral values at one or more wavelengths in the visible spectrum may be taken from the hyperspectral data stack.
• a single or multi color image may be derived from the hyperspectral image data. Based on this color image medical objects, e.g. pills, may be detected and located using standard image processing algorithms.
• the camera system may include a hyperspectral camera and a lamp for illuminating an imaging area of the hyperspectral camera.
• the lamp may include a housing and an illumination source.
• the housing may include an aperture allowing light to exit the housing and illuminate a medicine object.
• the illumination source may be configured to generate light of a continuous spectrum such as a halogen lamp or the light.
• the housing may include an outlet which may be connected to a cooling system, e.g. an air cooling system. This way, a flow, e.g. an air flow, can be generated wherein heat is transported away from the aperture towards the outlet.
• the invention may also relate to a method of inspecting medicine objects comprising: capturing a single-band image or a multi-band image of medicine objects, preferably medicaments of different shapes, sizes and/or compositions, randomly arranged in a pouch; capturing hyperspectral image data of the medicine objects in the pouch; determining blobs of pixels in the single-band image or a multi-band image, each of the blobs of pixels representing one of the medicine objects; selecting at least one hyperspectral image data part from the hyperspectral image data based on at least one of the blobs of pixels in the single-band image or a multi-band image; determining a hyperspectral fingerprint based on the hyperspectral image data part, the hyperspectral fingerprint being indicative of a spectral response of one or more chemical compounds in a medicine object; and, comparing the hyperspectral fingerprint with one or more reference fingerprints.
• the invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of process steps described above.
• Fig. 1 illustrates a medicine object inspection system according to an embodiment of the invention
• Fig. 2 illustrates a medicine object inspection scheme based on hyperspectral imaging according to an embodiment of the invention
• Fig. 3 depicts a flow diagram of a method for inspecting medicine packets according to an embodiment of the invention
• Fig. 4 depicts a medicine object inspection apparatus according to an embodiment of the invention
• Fig. 5 depicts a system for processing hyperspectral imaging data according to an embodiment of the invention
• Fig. 6 depicts an example of an image of a medicine packet captured by a hyperspectral imaging system
• Fig. 7A-7D depict images processed based on image processing methods according to the embodiments in this application.
• Fig. 8A-8D depict images processed based on image processing methods according to the embodiments in this application;
• Fig. 9 and 10 show images of medicine pouches and fingerprints of medicine objects.
• Fig. 1 illustrates a medicine object inspection system according to an embodiment of the invention.
• the figure depicts an inspection system 100, comprising a transporting system 102 for transporting medicine objects 106, including medicine pouches comprising a plurality of different medicine objects, through an inspection area configured to inspect the medicine objects based on an imaging system.
• the medicine objects may represent e.g. pills and/or tablets, capsules, ampules which may be packaged in packets or pouches and which may be inspected based on an imaging system.
• the imaging system may comprise one or more camera systems 114,116.
• a first camera system 114 may comprise one or more image sensors configured to capture images of a first spatial resolution of the medicine objects based a (limited) number of color channels.
• an image sensor may include RGB pixels for capturing an RGB color image or an image for each color channel.
• an image sensor may include a spectral channel in the non-visible part of the electromagnetic spectrum, e.g. a channel in the near infrared (NIR).
• the first spatial resolution may be a high spatial resolution so that details of the medicaments in a pouch, including shape, contour and letters, can be determined very fast and accurately based on known image processing algorithms.
• a NIR camera may be used to obtain a high spatial resolution (near) infrared image of the medicaments. Such image provides accurate information of the outer contours of the medicaments in the package.
• a color camera may be used to capture high spatial resolution color images of the medicaments Based on these images the location, shape and for example the color of the medicaments in the package may be determined very fast and accurately.
• a second camera system 116 may comprise a hyperspectral camera system, in particular a hyperspectral camera that may be configured to perform hyperspectral imaging on medicine objects.
• Pharmaceutically active compounds in the medicine objects are responsive to near infrared radiation, in particular near infrared radiation in the range between 800 and 1700 nm. This way, hyperspectral imaging may be a valuable tool for inspecting medicaments, such as inspecting pharmaceutically active compounds in a in pill, tablet or capsule.
• a plurality of spectral values may be detected within a predetermined part of the electromagnetic spectrum, for example, the visible band between 400 nm and 800 nm and/or the near infrared NIR band, e.g. between 800 and 1700 nm.
• the hyperspectral camera may produce a spectral image data stack wherein a slice of the spectral image data stack at a wavelength of the spectrum may represent an image of a second spatial resolution of the package including the medicaments, wherein the second spatial resolution is smaller than the first spatial resolution.
• a spectral value of the hyperspectral image data stack may represent a spectral response of an medicament captured by the hyperspectral imaging system.
• an object may be illuminated using an illumination source 122 that is especially suitable for hyperspectral imaging.
• the illumination source may be selected to have a continuous spectrum in the relevant parts of the spectrum, for example a continuous spectrum in the UV, visible and/or near infrared (NIR) range.
• Illumination sources that are suitable for this purpose include incandescent light sources, such as halogen lamps, that are based on a high-temperature heated filament.
• the hyperspectral camera may be configured to detect the spectral response of an imaged area in both the visible and NIR part of the spectrum.
• the hyperspectral camera may generate image data both in the visible range and in the NIR range.
• a separate multispectral camera e.g. an RGB or RGB/IR camera may not be needed if the hyperspectral camera is configured to generate both NIR and visible spectral values for pixels.
• one or more slices of spectral values at one or more wavelengths in the visible spectrum may be taken from the hyperspectral data stack.
• a single-band image e.g. a NIR image
• multi-band image e.g. an RGB or RGBI image
• groups of pixels (blobs) representing medical objects, e.g. pills, may be detected and located using standard image processing algorithms.
• a computer 118 may control the imaging system and the transport of the medicine objects. Further, the computer may comprise one or more image processing modules configured to process the image data generated by the imaging system so that medicine objects can be reliably inspected.
• the image processing module may be configured to execute the image processes as described with reference to the embodiments in this application.
• Fig. 2 illustrates a scheme for inspecting medicine objects based on hyperspectral imaging according to an embodiment of the invention.
• the figure includes a scheme 200, including capturing one or more first images of a first spatial resolution, e.g. one or more RGB and/or IR images, of a medicine pouch 201 comprising medicaments, in this example pills 2011-5, which may be of different shapes, sizes and compositions and which may be randomly arranged in the pouch.
• some of the pills such as pills 2012,3 and pills 2014,5 may be arranged partially next or over each other.
• the one or more first images may be used to localize the pills in the image of a first spatial resolution based on known object detection and segmentation algorithms.
• the medicaments 2011-5 in the image may represent groups of pixels (blobs) in the image (step 202).
• the medicine pouch may be imaged by a hyperspectral camera to create hyperspectral image data, a hyperspectral image data stack, of a second spatial resolution which is lower than the spatial resolution of the one or more first images.
• the hyperspectral camera may be implemented in different ways.
• the camera may be a 2D camera capturing an exposure area that includes the pouch.
• the camera may be a 1 D camera, i.e. a line scanner.
• Such line scan camera may comprise a row of light-sensitive pixels, which constantly scan moving objects at a high line scan frequency.
• a two-dimensional image of an object can be generated with a line-scan camera if the object moves under the camera at a known speed.
• Data generated by a line scanner may be “stitched” together into a 2D image.
• the hyperspectral data acquired by an hyperspectral camera may have the form of a “data cube” 204 having a third dimension representing spectral response at different parts of the spectrum and two other dimensions (in the x and y direction) representing the spatial axis.
• the y-axis may be a time respectively as shown in the figure.
• hyperspectral data associated with pills localized in the one or more first images may be determined (step 205).
• Such hyperspectral blob may contain spectral values 206 for a localized medicament, e.g. a pill. These values may represent a spectrum 208 at a pixel location that is part of a medicine object. Based on the spectrum a fingerprint may be determined which can be compared with a reference fingerprint.
• the high-resolution information in the high-resolution image allows fast and accurate distinction between the different medicaments in a pouch.
• fast and accurate selection of hyperspectral image data associated with that localized medicament can be achieved.
• This information can then be used for selecting the relevant part of the data in the hyperspectral image data which is needed for real-time, high-throughput inspection.
• Fig. 3 depicts a flow diagram of a method for inspecting medicine objects according to an embodiment of the invention.
• the process may include a first step 300 of capturing one or more first images of a first spatial resolution of the medicine pouch.
• a camera system may be used comprising a high-resolution image sensor, e.g. a 1440x1080 pixel image sensor and an optical system providing a spatial resolution of 0.1 mm per pixel (or -256 pixels per inch, PPI), preferably 0.08 mm per pixel (-317 PPI) or less.
• the one or more images may be captured, while exposing the medicine packet to light of one or more parts of the electromagnetic spectrum.
• At least one of the one or more first images may be an image that has a limited number of color channels, e.g. an RGB image.
• at least one of the one or more first images may be an infrared IR or near-infrared NIR image.
• such images may be captured using and RGB camera or a RGBI camera wherein the “I” represents pixels forming an infrared or nearinfrared NIR channel.
• the method may include capturing hyperspectral image data of the medicine packet.
• a hyperspectral pixel of the hyperspectral image data may comprise a plurality of spectral values representing the near-infrared spectral response of the medicine packet at that pixel location (as described above with reference to Fig. 2).
• captured spectral values of associated with one wavelength may form a 2D image of a second spatial resolution, wherein the second resolution is lower than the first resolution.
• the hyperspectral imaging system may have a pixelized image sensor and an optical system that provides a spatial resolution that is at least a factor 2 lower, e.g.
• the medicine packet may be exposed to light of a continuous spectrum in the visible and/or near-infrared (NIR) part of the electromagnetic spectrum.
• NIR near-infrared
• the process may further include determining one or more first blobs of first pixels, representing one or more medicaments, e.g. pills and/or capsules, in the one or more first images of the first spatial resolution (step 304). Then, one or more second blobs of second pixels may be selected from the hyperspectral image data based on the location of the one or more first blobs in the one or more first images (step 306). In step 308 a hyperspectral fingerprint for one of the one or more second pixel groups may be determined, wherein a hyperspectral fingerprint may be indicative of a spectral response of one or more chemical compounds in the medicine object. Thereafter, the hyperspectral fingerprint may be compared with a reference fingerprint to determine if the inspected medicine object can be identified as a medicine object according to the reference fingerprint (step 310).
• the method provides a very fast, efficient and accurate way of inspecting medicine objects based on capturing an image, such as color image, of one or more medicine objects and hyperspectral image data of the one or more medicine objects.
• an image such as color image
• hyperspectral image data parts from the hyperspectral image data may be selected wherein the hyperspectral image data have a second spatial resolution that is lower than the first resolution.
• hyperspectral image data parts are determined based on the hyperspectral image data with high speed and accuracy. This way, hyperspectral pixels may be determined that are related to the medicine objects.
• the one or more hyperspectral image data parts may be subsequently used for determining one or more hyperspectral fingerprints, wherein a hyperspectral fingerprint is indicative of a spectral response of one or more chemical compounds in a medicine object. These one or more hyperspectral fingerprints are used to determine if the one or more medicine objects can be identified based on reference fingerprints.
• Fig. 4 depicts a medicine inspection apparatus comprising a hyperspectral imaging system according to an embodiment of the invention.
• an inspection system 400 comprising an imaging system 401 for imaging one or more medicine objects 402i. n , i.e. one or more pouches comprising medicaments.
• the system may further comprise a transport structure 404 comprising a transporting path 406 for guiding one or more medicine objects through an inspection area of the imaging system.
• the medicine objects may include pills, tablets, capsules, ampules, etc. or a packet or pouch comprising such pills, tablets, capsules, ampules, etc., which are inspected based on image data generated by the imaging system.
• the medicine objects When the inspection system is in use, the medicine objects may be transported over the transport path to the inspection area.
• the medicine objects may be configured as a string of packets that can be unwound from a first (upstream) reel 4082, guided through the inspection area and rewound around a second (downstream) reel 408i.
• the movement of the reels may be controlled by a motor 412.
• the imaging system may comprise one or more camera systems.
• the imaging system may comprise a camera system 414, 416 comprising one or more multi-spectral image sensors which are configured to capture images of the packets, based on a (limited) number of color channels.
• an image system may include RGB pixels for capturing an RGB color image or three images for each color channel.
• the image system may include one or more further spectral channels, e.g. a spectral channel in the near-infrared (NIR).
• NIR near-infrared
• the imaging system may comprise a hyperspectral camera system according to any of the embodiments in this application.
• the hyperspectral camera system may include a hyperspectral camera 418 and a lamp 420 for illuminating an imaging area of the hyperspectral camera.
• the lamp may include a housing 419 and an illumination source 423.
• the housing may include an aperture 421 allowing light to exit the housing and illuminate a medicine object.
• the illumination source may be configured to generate light of a continuous spectrum such as a halogen lamp or the light.
• the housing may include an outlet 425 which may be connected to a cooling system 422, e.g. an air cooling system.
• the inspection system may be controlled by a controller 424, e.g. a computer, that comprises different modules, e.g. software and/or hardware modules, configured to control the processes that are needed for inspecting the medicine objects.
• a controller 424 e.g. a computer, that comprises different modules, e.g. software and/or hardware modules, configured to control the processes that are needed for inspecting the medicine objects.
• the hyperspectral camera may be configured to detect the spectral response of an imaged area in the near-infrared (NIR) part of the spectrum. In some embodiments, the hyperspectral camera may also be configured to detect the spectral response of an imaged area in the visible part of the spectrum. In that case, the hyperspectral camera may generate image data both in the visible range and in the NIR range.
• NIR near-infrared
• each spectral value represents a spectral response of an object, e.g. a medicament, that is imaged by the hyperspectral imaging system.
• Pictures generated by the first and second camera system may be processed by an image processing module that is executed by the controller 424.
• image data of the first camera system e.g. 2D color pictures such as RGB color pictures
• image processing algorithm which is configured to localize and recognize medicine objects in the picture based on features such as shape and/or color.
• image data of the second camera system e.g. a 3D stack of image data comprising spectral information on medicine objects, preferably near infrared spectral information, may be used to determine a fingerprint of a medicine object, which may be compared with reference fingerprints in a database in order to derive information about the composition of the medicine object.
• the hyperspectral camera may be implemented in different ways.
• the camera may be a 2D imager.
• the camera may be implemented as a line scanner.
• the camera may comprise a 2D grid of light sensitive pixels configured to generate 2D hyperspectral image data.
• the 2D hyperspectral image data may include pixels of the imaged area, wherein each pixel is associated with a plurality of spectral response values.
• the camera may comprise a row of light-sensitive pixels, which scans an area at a high line scan frequency to produce 1 D hyperspectral image data for each scan.
• a two-dimensional image of an object can be generated with a line-scan camera if the object moves under the camera at a known speed or if the camera moves over the object at a known speed.
• the 1 D hyperspectral image data (a line of pixel data, wherein each pixel data includes a plurality of spectral values) that is generated by the line-scanner may be “stitched” together into 2D hyperspectral image data that include pixels of the imaged area, wherein each pixels is associated with a plurality of spectral response values.
• the data acquired by the hyperspectral cameras may have the form of a “data cube” having a third dimension representing spectral response at different parts of the spectrum and two other dimensions (in the x and y direction) representing the spatial axis and time, respectively.
• the hyperspectral camera may be configured to generate spectral values in at least the near infrared (NIR) range (wavelengths selected approximately between 900 nm and 1700 nm) of the electromagnetic spectrum.
• the hyperspectral camera may be configured to generate spectral values both in the NIR range and in the visible range or only in the visible range.
• a typical data acquisition of a line-scanner may correspond to a “line” of 600 to 1000 pixels with length approximately between 200 and 300 pm each. The width of the pixel varies according to the field of view of the lens but in our case is approximately between 300 and 600 pm.
• Every such spatial pixel may comprise more than 200 spectral values spread equidistantly in the 900 - 1700 nm bandwidth. It is submitted that this figure is merely a non-limiting example of a hyperspectral imaging system that may be used in a medicine inspection system according to the various embodiments described in this application.
• the motor e.g. a stepper motor, that drives the transport structure (e.g. a conveyor belt) may serve as the triggering mechanism for the camera.
• the camera may be triggered to acquire a line of pixels.
• the conveyor belt may be controlled at a speed of 100-200 mm/sec, which would trigger the hyperspectral camera around 300 times per second, so the object is scanned with 300 fps. That means a maximum of 3.3 ms between the acquisition of two consecutive lines and therefore a maximum exposure time not longer than 3 ms, taking into account the time needed to transport the data.
• the processing of the hyperspectral data may comprise a step of identifying in the hyperspectral image data, data that are related to specular reflections and overexposed areas (at the packet level) and removing the identified hyperspectral data. Then, in a further step hyperspectral fingerprint(s) (at the pill level) may be determined, wherein each detected medicine object (pill, capsule, tablet) may be represented by a blob on the x-y plane of the hyperspectral cube. Overexposed pixels and/or pixels that are contaminated from specular reflections may be detected so that these values can excluded from the computation of hyperspectral fingerprints. The detection of pixel values that have been overexposed during acquisition may be based on threshold values.
• overexposure may be determined if the reflectance signal equals the maximum of the dynamic range of the sensor. These pixels may be filtered out of the raw data easily since their reflectance values are equal to the maximum of the dynamic range across all spectral bands.
• FIG. 6 shows such reflections (white regions as e.g. indicated by references 602 and 604) on a hyperspectral scan of a pouch where the pill inside the pouch is not visible because of reflections of the pouch.
• the reflectance spectrum in those regions may be essentially equivalent to the spectral power distribution of the light source itself (SPD), which is equivalent to the reflection of the total amount of light emitted.
• CEM Constrained Energy Minimization
• Fig. 7A-7D schematically show the process of detection of specular reflections and overexposed pixels and the subsequent removal of these pixels from the hyperspectral image data as shown in Fig. 6.
• specular reflections are detected based on a target detection technique as described above.
• overexposed pixels may be determined based on a threshold value.
• both the pixels affected by specular reflections and overexposure may be used to form a pixel mask as shown in Fig. 7C, identifying pixels (and associated spectral values) that should be removed from the spectral image data.
• Fig. 7D depicts the result wherein the pixel mask is applied to the hyperspectral image data. Based on these data hyperspectral fingerprints may be determined.
• the extraction of a hyperspectral fingerprint of individual medicine objects inside a pouch may comprise a first step of localization of a medicament, e.g. a pill, in one or more high resolution images of a medicine pouch.
• the image processing of these images that precedes the hyperspectral processing may already provide a robust pill detection and segmentation.
• the contours of a detected blob representing a medicament may be used to localize medicine objects inside the pouch.
• the resolution and the pixel size of the high- resolutin image may be different compared to those of the hyperspectral image, so the contour coordinates need to be scaled so that it can be used to localize blobs of pixels in the hyperspectral data (hyperspectral blobs) representing medicine objects.
• the scaling coefficients may be constant for every pouch which results in a very fast computation of the coordinates of the tablet on the x-y plane of the hyperspectral image.
• outliers may be removed from in the hyperspectral blobs.
• the hyperspectral blobs may comprise background pixels because the mapping of coordinates from the high resolution image to the hyperspectral image may not be exact. Additionally, the position of a pouch or a medicine object in the pouch may change slightly when being transported from the color camera exposure area to the exposure area of the hyperspectral camera. In such cases using all the pixels designated by this mapping would result in some background pixels being taken into account in the computation of the medication fingerprint.
• selected hyperspectral image data may be clustered in two groups according to their spectral characteristics.
• the centroids of the two clusters may be defined as the spectral mean of the whole pouch, representing the background cluster and the center of mass of the mapped blob, representing the medicine objects.
• the pixels assigned to the medication cluster may be used for all subsequent computations.
• a further step relates to the de-noising and normalization of pixels in the hyperspectral blob.
• the thermal noise of the camera may be subtracted. This may be realized based on the raw reflectance values. This noise is essentially the signal received by the sensor when the shutter of the camera is closed (complete absence of light).
• plurality of scans with the shutter closed may be taken and the values for each wavelength may be averaged. The thus obtained average noise profile may be subtracted from the reflectance of each individual pixel.
• spectral characteristics of the light source may be removed. This is done to ensure that only the reflectance characteristics of the medicine objects are used in the determination of a fingerprint.
• a logarithmic derivative may be computed to make the hyperspectral fingerprints invariant to the light intensity.
• the logarithmic derivative of a spectrum p at the spectral band i can be computed as: where e is a small positive constant that ensures that division by zero does not occur.
• This form of derivative is called logarithmic because it uses the ratio between consecutive spectra instead of their difference.
• the logarithmic derivative may accentuate small structural differences between nearly identical spectra.
• the log-derivatives of the spectra may be smoothed with a filter, e.g.
• a Savitzky-Golay filter that performs a piece-by-piece fitting of a polynomial function, e.g. second degree polynomial function to the input signal.
• the mean of the smoothed logarithmic derivatives of all the valid pixels for each spectral bin may be computed, thus reducing the data to a single reflectance spectrum per medication and averaging out noise.
• a medication object may be represented by a vector of predetermined dimensions, e.g. 150 dimensions of more.
• Each dimension may correspond to a different wavelength in the range 930 - 1630 nm and it may be possible that a number of wavelengths carry no significant discriminative power among different medicine objects.
• Such redundant dimensions do not contribute anything to successfully matching medications and in fact they often reduce the performance of a matching algorithm
• a dimensionality reduction algorithm such as a PCA dimensionality reduction algorithm may be used.
• Such algorithm may be used to detect the non-linear structures in the original data and unfolds them to linearly separable projections.
• a cosine kernel may be used, which essentially means that the data is projected to a new feature space based on the matrix of pairwise cosine distances among the hyperspectral profiles in a reference set. This step may require to define a set of reference pouches beforehand, as it is this set that is used to compute the Kernel PCA transformation.
• Fig. 5 depicts a method for processing hyperspectral image data according to an embodiment of the invention. Examples of images during the image processing are depicted in Fig. 8A-8D and Fig. 9 and Fig. 10.
• this figure depicts a method for processing hyperspectral image data based on the steps as described above.
• the method may include a step to capture an image of a first spatial resolution of a medicine packet and localize one or more medicine objects in the image and to capture hyperspectral image data from the medicine packet (step 500). Then, a number of image processing steps may be applied to the hyperspectral data. These steps may include removal of background pixels (outliers) from the one or more hyperspectral image data parts using an algorithm, such as a clustering algorithm (step 502). Further, the method may comprise a step of removing pixels that are contaminated with specular reflections and/or that are overexposed from the one or more hyperspectral image data (step 504).
• Fig. 8A depicts an example of a localized pill in a color image.
• Fig. 8B depicts a hyperspectral image of the pill
• Fig. 8C depicts an image in which pixels comprising specular reflections and overexposure are removed.
• one or more hyperspectral image data parts may be determined by mapping the one or more localized medicine objects in the image onto the hyperspectral image data (step 506). This step is illustrated by Fig. 8D which depicts the selection of a blob of pixels from the hyperspectral image data based on the pill that is localized in the color image.
• the dimension of the one or more hyperspectral image data parts may be reduced, preferably based on a PCA method (step 508).
• a fingerprint may be determined based on at least one of the one or more reduced hyperspectral image data parts (step 510).
• Fig. 9 and 10 depict examples of fingerprints of two pills of the same pharmaceutical composition, wherein the fingerprints are computed based on the data processing steps described with reference to the embodiments in this disclosure.
• the techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set).
• IC integrated circuit
• a set of ICs e.g., a chip set.
• Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.

Landscapes

• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Immunology (AREA)
• Health & Medical Sciences (AREA)
• Biochemistry (AREA)
• General Health & Medical Sciences (AREA)
• Chemical & Material Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Pathology (AREA)
• Analytical Chemistry (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Theoretical Computer Science (AREA)
• Signal Processing (AREA)
• Quality & Reliability (AREA)
• Investigating Or Analysing Materials By Optical Means (AREA)
• Investigating Or Analysing Biological Materials (AREA)
• Image Processing (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=79024589

Family Applications (1)

Country Status (6)

Families Citing this family (1)

Family Cites Families (8)

• 2021
  • 2021-12-03 WO PCT/EP2021/084272 patent/WO2022117874A1/en active Application Filing
  • 2021-12-03 CA CA3200248A patent/CA3200248A1/en active Pending
  • 2021-12-03 KR KR1020237021710A patent/KR20230127227A/ko unknown
  • 2021-12-03 US US18/255,286 patent/US20240029237A1/en active Pending
  • 2021-12-03 EP EP21830973.0A patent/EP4256312A1/en active Pending
  • 2021-12-03 JP JP2023533831A patent/JP2023551558A/ja active Pending

Also Published As

Similar Documents

Legal Events