WO2022244075A1 - 検査システム - Google Patents

検査システム Download PDF

Info

Publication number
WO2022244075A1
WO2022244075A1 PCT/JP2021/018652 JP2021018652W WO2022244075A1 WO 2022244075 A1 WO2022244075 A1 WO 2022244075A1 JP 2021018652 W JP2021018652 W JP 2021018652W WO 2022244075 A1 WO2022244075 A1 WO 2022244075A1
Authority
WO
WIPO (PCT)
Prior art keywords
time
observation
model
movement trajectory
series data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/018652
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
あずさ 澤田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to PCT/JP2021/018652 priority Critical patent/WO2022244075A1/ja
Priority to JP2023522022A priority patent/JP7529151B2/ja
Priority to US18/559,634 priority patent/US12555251B2/en
Publication of WO2022244075A1 publication Critical patent/WO2022244075A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/90—Investigating the presence of flaws or contamination in a container or its contents
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Definitions

  • the present invention relates to an inspection system, an inspection method, and a recording medium.
  • An inspection system has been proposed that inspects the presence or absence of foreign matter in a liquid sealed in a transparent or translucent container.
  • Non-Patent Document 1 a method and apparatus for predicting the certainty of the identification result output by the identification model configured by the deep neural network for image identification has been proposed (see, for example, Non-Patent Document 1). Specifically, when an image is input to a trained discriminant model, intermediate feature values extracted from the discriminative model and true class probability (TCP: True Class Probability) are used as teacher data, and the input is obtained from the discriminative model. A machine-learned confidence prediction model is used so that the output is the confidence of the discrimination result of the discrimination model.
  • TCP True Class Probability
  • the identification results of models that identify types of objects are not always 100% reliable and may be incorrect. There is a possibility that It is important to be able to predict the certainty of the identification results of the identification model, especially in applications where failures can have a serious impact, such as the inspection of foreign substances in liquid pharmaceuticals such as injectable preparations.
  • Non-Patent Document 1 when an image to be identified is input to a trained identification model, intermediate feature amounts and true class probabilities extracted from the identification model are used as teacher data, and the confidence prediction model to learn.
  • a configuration that trains a confidence prediction model using input images (intermediate features of them) as training data it is difficult to differentiate the confidences of multiple results estimated from multiple similar input images. be. Therefore, in the method described in Non-Patent Document 1, a difference is added to the confidence of the result estimated from time-series data representing mutually similar movement trajectories obtained by insufficient observation from a plurality of different types of objects. It is difficult.
  • the present invention is to provide an inspection system that solves the above problems.
  • An inspection system includes: Time-series data representing the movement trajectory of an object obtained by observation and the type of the object are used as first teacher data, and the time-series data representing the movement trajectory of the object obtained by observation are used to determine the object.
  • a discriminative model learning means for learning a discriminative model for estimating the type of A time series representing the movement trajectory of an object obtained by observation using time-series data representing the movement trajectory of an object obtained by observation, its observation specifications, and the type of the object as second teacher data.
  • a certainty prediction model learning means for learning a certainty prediction model for predicting the certainty of an estimation result of the discriminative model from data observation specifications; Using the learned discrimination model, estimating the type of the object from time series data representing the movement trajectory of the object obtained by observation, and using the learned confidence prediction model, the time series determination means for predicting the confidence level of the estimation result of the discriminative model from observation data data; is configured to include
  • an inspection method includes: Time-series data representing the movement trajectory of an object obtained by observation and the type of the object are used as first teacher data, and the time-series data representing the movement trajectory of the object obtained by observation are used to determine the object. learn a discriminant model that estimates the type of A time series representing the movement trajectory of an object obtained by observation using time-series data representing the movement trajectory of an object obtained by observation, its observation specifications, and the type of the object as second teacher data.
  • a computer-readable recording medium includes to the computer, Time-series data representing the movement trajectory of an object obtained by observation and the type of the object are used as first teacher data, and the time-series data representing the movement trajectory of the object obtained by observation are used to determine the object.
  • a process of learning a discriminative model that estimates the type of A time series representing the movement trajectory of an object obtained by observation using time-series data representing the movement trajectory of an object obtained by observation, its observation specifications, and the type of the object as second teacher data.
  • the present invention by having the configuration as described above, even when time-series data representing a plurality of mutually similar movement trajectories obtained by observation from a plurality of objects of different types, the observation specifications are different. If so, we can differentiate between the confidences of the results estimated from the time-series data.
  • FIG. 1 is a block diagram of an inspection system according to a first embodiment of the present invention
  • FIG. It is a block diagram showing an example of an inspection device in a 1st embodiment of the present invention.
  • 4 is a diagram showing a configuration example of image information according to the first embodiment of the present invention.
  • FIG. It is a figure which shows the structural example of the tracking information in the 1st Embodiment of this invention. It is a figure which shows the structural example of the inspection result information in the 1st Embodiment of this invention.
  • FIG. 4 is a schematic diagram showing an example of a method of creating teacher data used for machine learning of a certainty prediction model according to the first embodiment of the present invention
  • 4 is a flow chart showing an example of operation in a learning phase according to the first embodiment of the present invention
  • FIG. 4 is a flow chart showing an example of operation in an inspection phase in the first embodiment of the present invention
  • FIG. 7 is a schematic diagram showing another example of a method of creating teacher data used for machine learning of a certainty prediction model according to the first embodiment of the present invention
  • FIG. 11 is a schematic diagram showing an example of a method of learning a discriminant model according to Modification 3 of the first embodiment of the present invention
  • FIG. 10 is a schematic diagram showing an example of a method of creating teacher data used for machine learning of a certainty prediction model in Modification 4 of the first embodiment of the present invention.
  • FIG. 11 is a schematic diagram showing an example of a method of creating teacher data used for machine learning of a certainty prediction model in Modification 5 of the first embodiment of the present invention.
  • FIG. 11 is a schematic diagram showing an example of a discrimination model used in modification 6 of the first embodiment of the present invention;
  • FIG. 4 is a block diagram of an inspection system according to a second embodiment of the present invention; FIG.
  • FIG. 1 is a block diagram of an inspection system 100 according to the first embodiment of the invention.
  • an inspection system 100 is a system for inspecting the presence or absence of foreign matter in a liquid enclosed in a container 400 .
  • the inspection system 100 includes a grasping device 110, an illumination device 120, a camera device 130, an inspection device 200, and a display device 300 as main components.
  • a container 400 is a transparent or translucent container such as a glass bottle or a PET bottle.
  • the inside of the container 400 is sealed and filled with a liquid such as a medicine or water.
  • the liquid enclosed in the container 400 may contain foreign matter. Examples of foreign matter include pieces of glass, pieces of plastic, pieces of rubber, hair, pieces of fiber, soot, and the like.
  • the gripping device 110 is configured to grip the container 400 in a predetermined posture.
  • the predetermined posture is arbitrary.
  • the predetermined posture may be the posture when the container 400 is upright.
  • the predetermined posture may be a posture in which the container 400 is tilted at a predetermined angle from the upright posture.
  • the upright posture of the container 400 is assumed to be the predetermined posture.
  • a mechanism for holding the container 400 in an upright position is optional.
  • the gripping mechanism includes a pedestal on which the container 400 is placed in an upright position, and a member that presses the upper surface of the cap 401 that is the top of the container 400 placed on the pedestal. It's okay.
  • the gripping device 110 is configured to tilt, swing, or rotate the container 400 in a predetermined direction from an upright position while gripping the container 400 .
  • a mechanism for tilting, swinging, and rotating the container 400 is arbitrary.
  • the mechanism for tilting, swinging, and rotating may include a motor that tilts, swings, and rotates the entire gripping mechanism while gripping the container 400 .
  • the gripping device 110 is connected to the inspection device 200 by wire or wirelessly.
  • the gripping device 110 tilts, swings, and rotates the container 400 in a predetermined direction from an upright posture while gripping the container 400 .
  • the gripping device 110 stops tilting, swinging, and rotating the container 400, and returns to gripping the container 400 in an upright posture.
  • the illumination device 120 is configured to irradiate the liquid sealed in the container 400 with illumination light.
  • the illumination device 120 is, for example, a surface light source having a size corresponding to the size of the container 400 .
  • the illumination device 120 is installed on the opposite side of the container 400 to the side where the camera device 130 is installed. That is, illumination by the illumination device 120 is transmitted illumination.
  • the position of the illumination device 120 is not limited to this.
  • the camera device 130 is a photographing device that continuously photographs the liquid in the container 400 at a predetermined frame rate from a predetermined position on the opposite side of the container 400 where the lighting device 120 is installed.
  • the camera device 130 may include, for example, a color camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor having a pixel capacity of several million pixels.
  • Camera device 130 is connected to inspection device 200 by wire or wirelessly.
  • the camera device 130 is configured to transmit time-series images obtained by photographing to the inspection device 200 together with information indicating photographing times.
  • the display device 300 is a display device such as an LCD (Liquid Crystal Display).
  • the display device 300 is connected to the inspection device 200 by wire or wirelessly.
  • the display device 300 is configured to display the inspection result of the container 400 performed by the inspection device 200 and the like.
  • the inspection device 200 is an information processing device that performs image processing on time-series images captured by the camera device 130 and inspects the presence or absence of foreign matter in the liquid enclosed in the container 400 . Inspection device 200 is connected to grasping device 110, camera device 130, and display device 300 by wire or wirelessly.
  • FIG. 2 is a block diagram showing an example of the inspection device 200.
  • the inspection device 200 includes a communication I/F section 210 , an operation input section 220 , a storage section 230 and an arithmetic processing section 240 .
  • the communication I/F unit 210 is composed of a data communication circuit, and is configured to perform wired or wireless data communication with the gripping device 110, the camera device 130, the display device 300, and other external devices (not shown). ing.
  • the operation input unit 220 is composed of an operation input device such as a keyboard and a mouse, and is configured to detect an operator's operation and output it to the arithmetic processing unit 240 .
  • the storage unit 230 is composed of one or more types of storage devices such as hard disks and memories, and is configured to store processing information and programs 231 necessary for various processes in the arithmetic processing unit 240 .
  • the program 231 is a program that realizes various processing units by being read and executed by the arithmetic processing unit 240, and is transmitted from an external device (not shown) or a recording medium via a data input/output function such as the communication I/F unit 210. It is read in advance and stored in the storage unit 230 .
  • Main processing information stored in the storage unit 230 includes image information 232 , tracking information 233 , identification model 234 , confidence prediction model 235 , and inspection result information 236 .
  • the image information 232 includes time-series images obtained by continuously photographing the liquid in the container 400 with the camera device 130 . If floating matter exists in the liquid in the container 400, the image information 232 shows an image of the floating matter.
  • FIG. 3 shows a configuration example of the image information 232.
  • FIG. The image information 232 in this example is composed of an entry consisting of a set of a container ID 2321 , photographing time 2322 and frame image 2323 .
  • An ID for uniquely identifying the container 400 to be inspected is set in the container ID 2321 item.
  • As the container ID 2321 a serial number assigned to the container 400, a barcode attached to the container 400, fingerprint information collected from the cap 401 of the container 400, or the like can be considered.
  • the shooting time 2322 and the frame image 2323 are set with the shooting time and the frame image.
  • the photographing time 2322 is set to a precision (for example, in units of milliseconds) that can be distinguished from other frame images with the same container ID.
  • the photographing time 2322 for example, the elapsed time from when the container 400 stopped tilting, swinging, or rotating may be used.
  • the container ID 2321 is associated with each frame image 2323 in the example of FIG. 3, the container ID 2321 may be associated with each group of a plurality of frame images 2323 .
  • the tracking information 233 includes time-series data representing the moving trajectory of the suspended matter detected and tracked by detecting the image of the suspended matter present in the liquid in the container 400 shown in the image information 232 and its observation specifications.
  • Observation specifications include, for example, the length of the observed trajectory, the size of the observed floating object, the start time of the observed trajectory, the location in the container 400 where the observed trajectory existed, and the trajectory.
  • the trace information 233 in this example is composed of entries of a container ID 2331, a set of a trace ID 2332, a pointer 2333-1 and a pointer 2333-2.
  • An ID that uniquely identifies the container 400 is set in the container ID 2331 entry.
  • An entry consisting of a set of tracking ID 2332, pointer 2333-1 and pointer 2333-2 is provided for each floating object to be tracked.
  • An ID for distinguishing the tracked floating matter from other floating matter in the same container 400 is set in the tracking ID 2332 item.
  • a pointer to the movement locus information 2334 of the floating object to be tracked is set in the pointer 2333-1 item.
  • a pointer to the observation item list 2335 of the moving trajectory information of the floating object to be tracked is set in the pointer 2333-2 item.
  • the movement trajectory information 2334 consists of an entry consisting of a set of time 23341, position information 23342, size 23343, color 23344, and shape 23345.
  • the items of time 23341, position information 23342, size 23343, color 23344, and shape 23345 include the photographing time, coordinate values indicating the position of the tracked floating object at the photographing time, the size of the floating object, the color of the floating object, and the like.
  • the shape of the float is set.
  • the coordinate values may be, for example, coordinate values in a predetermined coordinate system.
  • the predetermined coordinate system may be a camera coordinate system centered on the camera, or a world coordinate system centered on a certain position in space.
  • Entries in the trajectory information 2334 are arranged in order of time 23341 .
  • the time 23341 of the top entry is the tracking start time.
  • the time 23341 of the last entry is the tracking end time.
  • the times 23341 of entries other than the first and last entries are tracking intermediate times.
  • the observation item list 2335 is a list of observation items considered to be related to the degree of certainty of the type of floating matter estimated from the movement trajectory information 2334 .
  • the observation specification list 2335 of this example is composed of entries each consisting of a set of a tracking length 23351, a floating object size 23352, a tracking start time 23353, a tracking area 23354, and a moving trajectory information quality 23355 related to the trajectory information 2334. It is
  • the length of the trajectory indicated by the trajectory information 2334 is set in the tracking length 23351 item.
  • the length of the movement trajectory may be the number of entries (that is, the number of frame images) forming the movement trajectory information 2334, or the length of time from the tracking start time to the tracking end time. It is considered that the longer the movement trajectory of a floating object is observed, the higher the probability that the movement corresponding to the type of the floating object appears in the movement trajectory. On the other hand, it is considered that a floating object with a short moving trajectory has a low probability that a movement corresponding to the type of the floating object appears in the moving trajectory. Therefore, the tracking length 23351 can be one of the observation parameters related to the degree of certainty of the type of floating matter estimated from the moving trajectory.
  • a value obtained by statistically processing the size 23343 included in the movement trajectory information 2334 (for example, average value, maximum value, minimum value, median value) is set in the item of the size 23352 of the floating object.
  • Large-sized foreign matter tends to settle early after the tilting, rocking, and rotation of the container 400 are stopped. Therefore, the size 23352 of the floating object can be one of the observation parameters related to the certainty of the type of floating object estimated from the movement trajectory.
  • the tracking start time of the movement trajectory information 2334 is set in the tracking start time 23353 item.
  • the tracking start time is a value representing the length of elapsed time from the time when the container 400 stops tilting, rocking, or rotating to the time when tracking of the movement trajectory information 2334 is started. It is considered that the earlier the tracking start time 23353 is, the more likely it is to be affected by the flow of the liquid, so it will take longer for the movement corresponding to the type of floating matter to appear in the movement trajectory. On the other hand, if the tracking start time 23353 is delayed, the influence of the liquid flow is reduced, so it is considered that the probability that movement corresponding to the type of floating matter will appear in the movement trajectory increases. Therefore, the tracking start time 23353 can be one of the observation parameters related to the certainty of the type of floating matter estimated from the movement trajectory.
  • a value indicating in which area within the container 400 the movement trajectory indicated by the movement trajectory information 2334 is set is set in the item of the tracking region 23354 .
  • Tracking regions are also called observation locations.
  • the tracking area 23354 may be, for example, a value specifying the circumscribing rectangle of the movement trajectory (for example, a coordinate value of the vertex of the circumscribing rectangle), or the shortest distance from the circumscribing rectangle to the liquid surface, wall surface, or bottom surface of the container 400. It may be a value representing Near the liquid surface of the container 400, it is difficult to correctly detect foreign matter due to the influence of air bubbles floating on the liquid surface. Moreover, it is not easy to correctly detect floating matter near the wall surface of the container 400 due to the lens effect.
  • the region in the container 400 in which the movement trajectory is located affects the reliability of the movement trajectory and the degree of certainty of the type of floating matter estimated from the movement trajectory.
  • the quality of the trajectory information 2334 is set in the trajectory information quality 23355 item.
  • the quality of the trajectory information 2334 may be determined, for example, based on the discontinuity of the position information 23342 included in the trajectory information 2334 and the amount of variation in the size 23343/color 23344/shape 23345 .
  • the movement trajectory information 2334 with excessive size variation and positional discontinuity that are expected to result in uncertain detection and tracking has poor reliability as a movement trajectory resulting from tracking the same floating object. Therefore, the quality 23355 of trajectory information can be one of the observation parameters related to the certainty of the type of floating object estimated from the trajectory.
  • observation specifications used in the present invention are not limited to the above. Confidence in the types of floating objects estimated from their movement trajectories, such as conditions that make it difficult to correctly evaluate features even though they are not directly included in the feature values of the discriminative model, and conditions that may increase exceptional errors due to failures in the observation itself. Any other observational dimension may be used as long as it is related to .
  • the observation specifications to be used may be determined from the characteristics of the observation (such as conditions under which the assumptions assumed in the detection/tracking process collapse), or conditions estimated from actual errors (such as conditions where the basis for identification cannot be clearly read). It's okay.
  • the identification model 234 is a model for estimating the type of floating matter from the time-series data representing the moving trajectory of the floating matter.
  • the discriminative model 234 may be configured using, for example, a recursive structure of neural networks such as RNN and LSTM.
  • the identification model 234 may use padding, pooling and resizing to result in identification of fixed length data.
  • the certainty prediction model 235 is a model that predicts the certainty of the result estimated by the discriminative model 234 based on the observation specifications of the time series data representing the movement trajectory of the floating object based on the time series data related to the observation specifications. is.
  • the certainty prediction model 235 may be constructed using, for example, a neural network.
  • confidence prediction model 235 may be a linear discriminator, decision tree, or the like.
  • the inspection result information 236 includes information according to the result of inspection for the presence or absence of foreign matter in the liquid sealed in the container 400 .
  • the inspection results include the estimation result of the type of floating matter calculated by the discrimination model 234 and the confidence factor of the estimation result of the discrimination model 234 calculated by the confidence prediction model 235 .
  • FIG. 5 shows a configuration example of the inspection result information 236.
  • the inspection result information 236 in this example is obtained from entries of a container ID 2361, an inspection result 2362, the number of detected foreign matter 2363, the number of detected air bubbles 2364, a set of a detected foreign matter ID 2365 and a pointer 2366, and a set of a detected air bubble ID 2367 and a pointer 2368. It is configured.
  • the entry of the container ID 2361 is set with an ID that uniquely identifies the container 400 to be inspected.
  • the entry of the inspection result 2362 is set with an inspection result of either OK (inspection passed) or NG (inspection unsuccessful).
  • the entry of the number of detected foreign matter 2363 is set with the total number of detected foreign matter.
  • the total number of detected bubbles is set in the entry of the number of detected bubbles 2364 .
  • the identification result may include an aggregate of components in the liquid in addition to air bubbles and foreign matter.
  • An entry of a set of detected foreign object ID 2365 and pointer 2366 is provided for each detected foreign object.
  • An ID for distinguishing the detected foreign matter from other foreign matter in the same container 400 is set in the item of the detected foreign matter ID 2365 .
  • a pointer to the detected foreign matter information 2369 of the detected foreign matter is set in the pointer 2366 item.
  • An entry of a pair of detected bubble ID 2367 and pointer 2368 is provided for each detected bubble.
  • An ID for distinguishing the detected bubble from other bubbles in the same container 400 is set in the item of detected bubble ID 2367 .
  • a pointer to the detected bubble information 2370 of the detected bubble is set in the pointer 2368 item.
  • the detected foreign matter information 2369 is composed of a set of a tracking ID 23691, a pointer 23692-1 and a pointer 23692-2, a judgment result 23693, a degree of certainty 23694, and a visualized image 23695.
  • the trace ID 23691 field contains the trace ID 2332 of the detected foreign matter.
  • a pointer to the movement locus information 23696 of the detected foreign object is set in the pointer 23692-1 item.
  • Movement trajectory information 23696 is a copy of movement trajectory information 2334 during tracking of the detected foreign object.
  • a pointer to an observation specification list 23697 related to the movement trajectory information 23696 of the detected foreign object is set.
  • the observation item list 23697 is a copy of the observation item list 2335 related to the movement trajectory information 2334 during tracking of the detected foreign object.
  • text indicating that the determination result is "foreign matter” is set.
  • the entry of the degree of certainty 23694 is set with the degree of certainty, which is an index representing the certainty of the determination result 23693 .
  • At least one image that visualizes the movement trajectory information 23696 of the detected foreign matter is set in the entry of the visualized image 23695 .
  • the detected air bubble information 2370 consists of a set of tracking ID 23701, pointer 23702-1 and pointer 23702-2, determination result 23703, certainty 23704, and visualized image 23705.
  • the tracking ID 23701 field contains the tracking ID 2332 of the detected bubble.
  • Pointer 23702-1 is set with a pointer to movement trajectory information 23706 of the detected bubble.
  • Trajectory information 23706 is a copy of trajectory information 2334 when tracking the detected bubble.
  • a pointer to the observation item list 23707 related to the movement locus information 23706 of the detected bubble is set in the pointer 23702-2 item.
  • the observation item list 23707 is a copy of the observation item list 2335 related to the movement trajectory information 2334 of the detected bubble tracking information.
  • the determination result 23703 entry is set with text indicating that the determination result is "bubble".
  • the entry of the degree of certainty 23704 is set with the degree of certainty, which is an index representing the certainty of the determination result 23703 .
  • At least one image that visualizes the movement trajectory information 23706 of the detected bubbles is set in the entry of the visualized image 23705 .
  • the arithmetic processing unit 240 has a microprocessor such as an MPU and its peripheral circuits, and reads and executes the program 231 from the storage unit 230 to cooperate with the hardware and the program 231. It is configured so as to realize various processing units.
  • Main processing units implemented by the arithmetic processing unit 240 include an acquisition unit 241 , a discrimination model learning unit 242 , a certainty prediction model learning unit 243 , and a determination unit 244 .
  • the acquisition unit 241 is configured to control the gripping device 110 and the camera device 130 to acquire image information 232 representing an image of floating matter present in the liquid enclosed in the container 400 . Further, the acquisition unit 241 is configured to acquire tracking information 233 including time-series data representing the movement trajectory of floating matter and its observation specifications by analyzing the image information 232 . Details of the acquisition unit 241 will be described below.
  • the acquisition unit 241 first activates the gripping device 110 that grips the container 400 to be inspected in an upright posture, thereby tilting, swinging, and rotating the container 400 to be inspected. Next, after a certain period of time has passed since the acquisition unit 241 was started, the acquisition unit 241 stops the gripping device 110, thereby causing the container 400 to stand still in a predetermined posture. By tilting, swinging, and rotating the container 400 for a certain period of time and then standing still, a state in which the liquid flows by inertia within the stationary container 400 is obtained. Next, the acquisition unit 241 starts an operation of continuously photographing the liquid in the container 400 to be inspected at a predetermined frame rate with the camera device 130 under transmitted illumination by the illumination device 120 . That is, if the time Ts is the time when the container 400 stops after being tilted, swung, and rotated, the acquisition unit 241 starts the photographing operation from time Ts.
  • the acquisition unit 241 continuously captures images of the liquid in the container 400 with the camera device 130 from time Ts to time Te when the predetermined time Tw elapses.
  • the predetermined time Tw is set so that, for example, assuming that all floating substances floating in the liquid are bubbles, all the bubbles move upward in the container 400 and are no longer considered to move downward. It may be set longer than the time required to obtain an accurate movement trajectory (hereinafter referred to as the minimum photographing time length).
  • the minimum imaging time length may be determined in advance by experiments or the like and fixedly set in the acquisition unit 241 .
  • the obtaining unit 241 may immediately stop the photographing by the camera device 130 when the time Te is reached, or may continue the photographing by the camera device 130 .
  • the acquisition unit 241 adds the shooting time and the container ID to each of the time-series frame images acquired from the camera device 130 and stores them in the storage unit 230 as image information 232 .
  • the acquiring unit 241 detects shadows of floating matter in the liquid in the container 400 from each of the frame images. For example, the acquisition unit 241 detects the shadow of floating matter in the liquid by the method described below. However, the obtaining unit 241 may detect the shadow of floating matter in the liquid by a method other than the method described below.
  • the acquisition unit 241 performs binarization processing on each frame image to create a binarized frame image.
  • the acquiring unit 241 detects shadows of floating objects from each of the binarized frame images as follows.
  • the acquisition unit 241 first sets the binarized frame image, which is the target for detecting the shadow of floating matter, as the binarized frame image of interest.
  • a differential image is generated between the binarized frame image of interest and the binarized frame image photographed after ⁇ t.
  • ⁇ t is set to such a time that the same floating matter partially overlaps in the two images, or appears at very close positions even if they do not overlap. Therefore, the time difference ⁇ t is determined according to the properties and flow conditions of the liquid and the foreign matter.
  • the matching image portions in the two binarized frame images are erased, leaving only the different image portions.
  • the obtaining unit 241 detects the shadow of the binarized frame image of interest corresponding to the portion where the shadow appears in the difference image as the shadow of the floating matter present in the binarized frame image of interest.
  • the acquisition unit 241 tracks the detected floating matter in time-series images and creates tracking information 233 according to the tracking results.
  • the acquisition unit 241 initializes the tracking information 233 .
  • the container ID of the container 400 to be inspected is set in the entry of container ID 2331 in FIG.
  • the acquisition unit 241 tracks the floating object in the time-series images by the method described below, and according to the tracking result, the tracking ID 2332 and the pointer in FIG. 2333-1 and pointer 233-2 pair entry, movement trajectory information 2334, and observation specification list 2335 are created.
  • the acquisition unit 241 focuses on the binarized frame image with the earliest shooting time in the time series of the binarized frame images created above.
  • the acquisition unit 241 assigns a unique tracking ID to each floating object detected in the binarized frame image of interest.
  • the acquisition unit 241 sets the tracking ID given to the floating matter detected in the binarized frame image of interest to the item of the tracking ID 2332 in FIG.
  • the photographing time of the binarized frame image under consideration is set in the time 23341 item of the first entry of the movement trajectory information 2334 indicated by 2333-1, and the position information 23342, size 23343, color 23344, and shape 23345 items are set. Set the coordinate value, size, color and shape of the floating object in the binarized frame image of interest.
  • the acquisition unit 241 shifts attention to a binarized frame image one frame after the binarized frame image of interest.
  • the acquisition unit 241 focuses on one floating object detected in the binarized frame image of interest.
  • the acquiring unit 241 compares the position of the floating object under consideration with the position of the floating object detected in the binarized frame image one frame earlier (hereinafter referred to as the preceding binarized frame image). If there is a floating matter within a predetermined threshold distance from the floating matter of interest, it is determined that the floating matter of interest and the floating matter present within the threshold distance are the same floating matter. In this case, the acquisition unit 241 assigns, to the floating matter of interest, the tracking ID assigned to the floating matter determined to be the same floating matter.
  • the acquiring unit 241 secures a new entry in the moving track information 2334 pointed to by the pointer 2333-1 of the entry of the tracking information 233 to which the assigned tracking ID 2332 is set, and acquires the time 23341 and position information of the secured entry.
  • the shooting time of the binarized frame image of interest and the coordinate values, size, color, and shape of the floating matter of interest are set.
  • the acquisition unit 241 determines that the interested floating matter is a new floating matter and assigns a new tracking ID. .
  • the acquisition unit 241 sets the tracking ID assigned to the currently focused floating object to the item of the tracking ID 2332 in FIG. , set the shooting time of the binarized frame image under consideration in the item of the time 23341 of the first entry of , and set the coordinate value, size, and color of the floating object under consideration in the items of the position information 23342, the size 23343, the color 23344, and the shape 23345. and shape.
  • the acquisition unit 241 shifts attention to the next floating matter detected in the focused binarized frame image, and repeats the same processing as described above. After paying attention to all floating substances detected in the binarized frame image of interest, the acquiring unit 241 shifts its attention to the next frame image, and repeats the same processing as described above. Then, when the acquisition unit 241 finishes paying attention to the last frame image in the image information 232, it ends the tracking process.
  • the acquisition unit 241 performs tracking based on the distance between floating objects in two adjacent frame images.
  • the acquisition unit 241 may perform tracking based on the distance between floating objects in two frame images adjacent to each other across n frames (n is a positive integer equal to or greater than 1).
  • the acquisition unit 241 obtains the tracking result obtained by tracking based on the distance between floating objects in two frame images adjacent to each other with m frames (m is a positive integer equal to or greater than 0) and the m+j frames (j is a positive integer Tracking may be performed by comprehensively determining the tracking results based on the distance between floating objects in two frame images adjacent to each other with a positive integer of 1 or more interposed therebetween.
  • the acquisition unit 241 creates an observation specification list 2335 for each movement trajectory information 2334 created as described above.
  • the acquisition unit 241 focuses on one piece of the movement track information 2334 .
  • the acquisition unit 241 creates an initial state observation item list 2335 related to the currently focused movement trajectory information 2334 in the area pointed to by the pointer set in the pointer 2333-2.
  • the acquisition unit 241 sets the length of the movement trajectory represented by the movement trajectory information 2334 of interest in the tracking length 23351 item.
  • the acquisition unit 241 sets a value obtained by statistically processing the size 23343 included in the movement trajectory information 2334 of interest in the item of the size 23352 of the floating object.
  • the acquisition unit 241 sets the tracking start time of the movement trajectory information 2334 of interest in the item of the tracking start time 23353 .
  • the acquisition unit 241 sets a value indicating in which region within the container 400 the movement trajectory represented by the movement trajectory information 2334 of interest is in the item of the tracking region 23354 .
  • the acquisition unit 241 sets a numerical value representing the quality of the moving track information of interest 2334 in the item of moving track information quality 23355 .
  • the acquisition unit 241 shifts attention to one of the remaining trajectory information 2334 and repeats the same processing as described above. This process is repeated until attention is paid to all movement trajectory information 2334 .
  • the discriminative model learning unit 242 is configured to generate the discriminative model 234 by machine learning.
  • the discriminative model learning unit 242 uses the time-series data representing the movement trajectory of the floating object and the type of the floating object as teacher data (hereinafter also referred to as first teacher data).
  • teacher data hereinafter also referred to as first teacher data.
  • the time-series data representing the movement trajectory of the floating object for example, the movement trajectory information 2334 shown in FIG. 4 may be used.
  • the time-series data representing the movement trajectory of the floating object is obtained by removing one, two, or all of the size 23343, the color 23344, and the shape 23345 from the movement trajectory information 2334 shown in FIG. information.
  • the type of floating matter may be a label value representing either foreign matter or air bubbles.
  • the first training data includes time-series data representing the movement trajectory of the floating object and labels representing the types of the floating object.
  • Such first teacher data can be created, for example, through interactive processing with the user.
  • the identification model learning unit 242 displays the movement trajectory information 2334 acquired by the acquisition unit 241 on the screen of the display device 300 and receives the label of the movement trajectory information 2334 from the user through the operation input unit 220 . Then, the discriminant model learning unit 242 creates a set of the displayed trajectory information 2334 and the received label as one piece of first teacher data.
  • the method of creating the first training data is not limited to the above.
  • the discriminative model learning unit 242 uses the above-described first teacher data, inputs time-series data representing the movement trajectory of floating matter (foreign matter or air bubbles), and outputs the discriminative model 234 of the type of floating matter. It is configured to be generated by machine learning.
  • the certainty predictive model learning unit 243 is configured to generate the certainty predictive model 235 by machine learning.
  • FIG. 6 is a schematic diagram showing an example of a method for creating teacher data used for machine learning of the certainty prediction model 235.
  • each of the teacher data 250 includes time-series data 2501 representing the movement trajectory of floating matter, the type 2502 of the floating matter, and its observation data 2503 .
  • the time-series data 2501 for example, the movement track information 2334 shown in FIG. 4 may be used.
  • the time-series data 2501 may be, for example, remaining information obtained by removing one, two, or all of the size 23343, color 23344, and shape 23345 from the movement trajectory information 2334 shown in FIG. .
  • the observation item list 2335 of the trajectory information 2334 shown in FIG. 4 may be used.
  • the floating matter type 2502 may be a label value representing either foreign matter or air bubbles.
  • label values can be created, for example, through interactive processing with the user.
  • the certainty prediction model learning unit 243 displays the movement trajectory information 2334 acquired by the acquisition unit 241 on the screen of the display device 300 and receives the label of the movement trajectory information 2334 from the user through the operation input unit 220 . Then, the certainty prediction model learning unit 243 creates a set of the displayed movement trajectory information 2334 and the received label and observation specification list 2335 of the movement trajectory information 2334 as one teacher data.
  • the method of creating teacher data is not limited to the above.
  • the certainty prediction model learning unit 243 creates one new teacher data 252 from one teacher data 250 as follows. First, the certainty prediction model learning unit 243 inputs the time-series data 2501 in the teacher data 250 to the learned discrimination model 234, and estimates the type of floating objects finally output from the discrimination model 234. get. Next, the certainty prediction model learning unit 243 compares the type of floating matter represented by the estimation result of the discrimination model 234 with the type of floating matter in the teacher data 250 (block 251). Next, the certainty predictive model learning unit 243 creates a pair of the certainty 2521 set to the value corresponding to the comparison result and the observation specification 2503 in the teacher data 250 as the teacher data 252 .
  • a large value (for example, 1 or a value closer to 1) may be set when both match (that is, when the estimation result of the identification model 234 is correct).
  • This value may be a predetermined fixed value (eg, 1), or the softmax value (TCP) of the correct class of the discriminative model 234 .
  • a small value (for example, 0 or a value close to 0) may be used.
  • This value may be a predetermined fixed value (eg, 0), or the softmax value (TCP) of the correct class of the discriminative model 234 .
  • the certainty prediction model learning unit 243 uses the teacher data 252 created as described above, the input is the observation specification of the time series data representing the movement trajectory of the floating object obtained by observation, and the output is the observation specification It is configured to generate a certainty prediction model 235, which is the certainty of the estimation result of the discriminative model 234 estimated from the original time-series data, by machine learning.
  • the determination unit 244 uses the learned identification model 234 to estimate the type of floating matter from the time-series data representing the movement trajectory of the suspended matter in the liquid enclosed in the container 400 acquired by the acquiring unit 241. is configured to The determining unit 244 is configured to predict the certainty of the estimation result of the discriminant model 234 from the observation specifications acquired by the acquiring unit 241 using the learned certainty prediction model 235 . In addition, the determination unit 244 creates inspection result information 236 including the type of floating matter estimated using the discrimination model 234 and the certainty of the estimation result of the discrimination model 234 registered using the certainty prediction model 235. is configured as
  • the determination unit 244 reads out the tracking information 233 from the storage unit 230, and for each tracking ID included in the tracking information 233, the movement trajectory information 2334 representing the movement trajectory of the floating object is used as time-series data to identify the learned identification model 234. , it is determined whether the floating substance with the tracking ID is a foreign substance or an air bubble.
  • the determination unit 244 uses the identification model 234 by inputting the observation specification list 2335 of the movement trajectory of the floating object to the learned confidence prediction model 235 for each tracking ID included in the tracking information 233. Predict the degree of certainty of the determination result of the type of suspended matter that has been determined. Then, the determination unit 244 creates inspection result information 236 according to the determination result and stores it in the storage unit 230 . Further, the determination unit 244 displays the inspection result information 236 on the display device 300 and/or transmits it to an external device through the communication I/F unit 210 .
  • phases of the inspection system 100 are roughly divided into a learning phase and an inspection phase.
  • the learning phase is a phase in which the identification model 234 and the confidence prediction model 235 are created by machine learning.
  • the inspection phase is a phase in which the presence or absence of foreign matter in the liquid enclosed in the container 400 is inspected using the learned discrimination model 234 and confidence prediction model 235 .
  • FIG. 7 is a flow chart showing an example of the operation of the learning phase.
  • the acquiring unit 241 first acquires image information 232 representing an image of floating matter present in the liquid enclosed in the container 400 by controlling the gripping device 110 and the camera device 130 (step S1).
  • the acquisition unit 241 acquires the tracking information 233 including the time-series data representing the moving trajectory of the floating matter and its observation specifications (step S2).
  • the discriminative model learning unit 242 creates first teacher data used for machine learning of the discriminative model 234 (step S3).
  • the discriminative model learning unit 242 uses the created first teacher data to generate the discriminative model 234 by machine learning, with the input as time-series data representing the movement trajectory of the floating object and the output as the type of the floating object. (step S4).
  • the certainty predictive model learning unit 243 creates second teacher data used for machine learning of the certainty predictive model 235 (step S5).
  • the certainty prediction model learning unit 243 uses the created second teacher data, the input is the observation specification of the time series data representing the movement trajectory of the floating object obtained by observation, and the output is the observation specification
  • a certainty factor prediction model 235 is generated by machine learning as the certainty factor of the estimation result of the discrimination model 234 estimated from the original time-series data (step S6).
  • FIG. 8 is a flow chart showing an example of the operation of the inspection phase.
  • the acquisition unit 241 acquires image information 232 representing an image of floating matter present in the liquid enclosed in the container 400 by controlling the gripping device 110 and the camera device 130 (step S11).
  • the acquisition unit 241 acquires the tracking information 233 including the time-series data representing the movement trajectory of the floating matter and its observation specifications (step S12).
  • the determination unit 244 uses the learned identification model 234 to estimate the type of floating matter from the time-series data representing the movement trajectory of the floating matter included in the tracking information 233 (step S13).
  • the determination unit 244 uses the learned confidence prediction model 235 to extract the result of the estimation by the identification model 234 from the observation specification list of the time-series data representing the movement trajectory of the floating object included in the tracking information 233. is predicted (step S14).
  • the determination unit 244 creates inspection result information 236 based on the estimated type of floating matter and the predicted certainty of the estimation result (step S15).
  • the certainty prediction model learning unit 243 uses source data composed of a set of time-series data representing the movement trajectory of the floating object obtained by the acquiring unit 241, its observation specifications, and the type of floating object.
  • the determination unit 244 acquires observation specifications related to the time-series data representing the movement trajectory of the object obtained by the acquisition unit 241, and uses the learned confidence prediction model 235 to use the acquired observation specifications. This is for outputting the confidence of the estimation result of the discriminant model 234 estimated from the original.
  • the determination unit 244 may modify or correct the estimation result by the discrimination model 234 based on the confidence predicted by the confidence prediction model 235 .
  • the determination unit 244 predicts from the observation specifications of the time-series data using the certainty prediction model 235.
  • the type of floating matter may be corrected to air bubbles instead of foreign matter.
  • the determination unit 244 predicts from the observation specifications of the time-series data using the certainty prediction model 235
  • the degree of certainty of the estimation result may be calculated as a foreign matter likelihood score.
  • the determination unit 244 determines the foreign matter likelihood score by the identification model 234 (probability of foreign matter output by the identification model 234). , may be corrected using the certainty factor of the estimation result predicted from the observation specifications of the time-series data using the certainty factor prediction model 235 .
  • the certainty predictive model learning unit 243 may use a predetermined output of the discrimination model 234 for learning the certainty predictive model 235 .
  • the predetermined output of the discriminative model 234 may be, for example, a feature quantity output from the intermediate layer of the discriminative model 234 .
  • FIG. 9 is a schematic diagram showing another example of a method for creating teacher data used for machine learning of the certainty prediction model 235.
  • 252A is teacher data
  • 2522 is a predetermined output of the discrimination model 234.
  • the certainty prediction model learning unit 243 creates one new teacher data 252A from one teacher data 250 as follows.
  • the certainty prediction model learning unit 243 inputs the time series data 2501 in the teacher data 250 to the learned discrimination model 234, and the classification result of the floating object type finally output from the discrimination model 234. A predetermined output 2522 is obtained.
  • the confidence prediction model learning unit 243 compares the type of floating matter represented by the estimation result of the discrimination model 234 with the type of floating matter in the teacher data 250. 2521 (block 251). Then, the certainty prediction model learning unit 243 creates a set of the certainty 2521, the observation specification 2503 in the teacher data 250, and the predetermined output 2522 as the teacher data 252A.
  • the certainty prediction model learning unit 243 uses the teacher data 252A created as described above to identify the observation specifications of the time-series data representing the movement trajectory of the floating matter obtained by observing the input and the time-series data. Confidence as a set with a predetermined output 2522 output from the discriminative model 234 when input to the model 234, and the output as the certainty of the estimation result of the discriminative model 234 estimated from the time series data related to the above observation specifications It is configured to generate the degree prediction model 235 by machine learning.
  • the determination unit 244 may use the predetermined output of the discrimination model 234 for confidence prediction. For example, the determination unit 244 assigns the observation specification list of the time-series data representing the movement trajectory of the floating object included in the tracking information 233 and the time-series data to the learned confidence prediction model 235 as the identification model 234. A combination with a predetermined output 2522 output from the discriminative model 234 at the time of input is input, and the certainty factor of the estimation result of the discriminative model 234 is obtained.
  • the predetermined output of the discriminative model 234 is the feature quantity output from the intermediate layer of the discriminative model 234.
  • the predetermined output of discriminative model 234 is not limited to the above.
  • a given output of discriminative model 234 may be the final output of discriminative model 234 .
  • the discriminative model learning unit 242 may further learn the discriminative model 234 generated in step S4 at any time after the certainty prediction model 235 is generated in step S6 of FIG. In that case, the discriminative model learning unit 242 may control the learning of the discriminative model 243 based on the certainty predicted by the learned certainty prediction model 235 .
  • FIG. 10 is a schematic diagram showing an example of a learning method for the discriminative model 234 according to Modification 3.
  • teacher data 260 includes time-series data 2601 representing the movement trajectory of a floating object and the type 2602 of the floating object.
  • Observation specifications 2603 of time-series data 2601 are prepared for each teacher data 260 .
  • the discriminative model learning unit 242 learns the discriminative model 234 using the teacher data 260
  • the discriminative model learning unit 242 inputs the observation specifications 2603 paired with the teacher data 260 to the learned confidence prediction model 235, and the confidence prediction model 235
  • the learning of the discriminative model 234 is controlled by the confidence output from .
  • the discriminant model learning unit 242 reduces the learning weight as the certainty is lower. As a result, the identification accuracy of the identification model 234 can be improved.
  • the certainty prediction model learning unit 243 divides the time-series data representing the movement trajectory of floating matter obtained by observation into several partial time-series data, and observes the individual partial time-series data. It is configured to perform machine learning of the confidence predictive model 235 using the originals.
  • FIG. 11 is a schematic diagram showing an example of a method for creating teacher data used for machine learning of the certainty prediction model 235 in modification 4.
  • each of the teacher data 250 includes time-series data 2501, types of floating matter 2502, and observation specifications 2503 as already described with reference to FIG.
  • the time-series data 2501 may be, for example, the trajectory information 2334 in FIG.
  • the observation specification 2503 may be, for example, the observation specification list 2335 of FIG.
  • the certainty prediction model learning unit 243 has a data conversion unit 2431 that converts each of the teacher data 250 into two new teacher data 250-1 and 250-2.
  • the teacher data 250-1 consists of time-series data 2501-1, types of floating matter 2502-1, and observation specifications 2503-1.
  • the teacher data 250-2 consists of time-series data 2501-2, types of floating matter 2502-2, and observation specifications 2503-2. Although one teacher data 250 is converted into two teacher data in the example of FIG. 11, it may be converted into three or more teacher data.
  • the data conversion unit 2431 converts the teacher data 250 into teacher data 250-1 and 250-2 by, for example, the following method. First, the data conversion unit 2431 calculates an intermediate time between the tracking start time and the tracking end time of the time-series data 2501 in the teacher data 250 . Next, the data conversion unit 2431 converts the time-series data 2501 into time-series data 2501-1 from the tracking start time to the intermediate time and time-series data 2501-2 from the intermediate time to the tracking end time. . Next, the data conversion unit 2431 creates floating matter types 2502 - 1 and 2502 - 2 having the same contents as the floating matter type 2502 of the teacher data 250 . Next, data conversion section 2431 creates observation specifications 2503-1 and 2503-2 from time-series data 2501-1 and 2501-2.
  • the data conversion unit 2431 creates observation specifications 2503-1 from the time-series data 2501-1 by the following method. First, the data conversion unit 2431 determines the number of frame images constituting the time-series data 2501-1, or the length of time from the shooting time of the first frame image in the time-series data 2501-1 to the shooting time of the last frame image. and set it as the tracking length of the time-series data 2501-1. Next, the data conversion unit 2431 calculates values (for example, average value, maximum value, minimum value, median value) obtained by statistically processing the sizes 23343 included in the time-series data 2501-1, and converts them to the sizes of floating objects. Satoru.
  • values for example, average value, maximum value, minimum value, median value
  • the data conversion unit 2431 acquires the photographing time of the first frame image of the time-series data 2501-1, and uses it as the tracking start time.
  • the data conversion unit 2431 calculates values (for example, coordinate values of apexes of the circumscribing rectangle) specifying the circumscribing rectangle of the movement trajectory represented by the time-series data 2501-1, and defines them as the tracking area.
  • the data conversion unit 2431 calculates the quality of the movement trajectory information based on the discontinuity of the position information 23342 included in the time-series data 2501-1 and the amount of variation in the size 23343, color 23344, and shape 23345.
  • the data conversion unit 2431 converts the tracking length calculated as described above, the size of the floating object, the tracking start time, the tracking area, and the observation specification 2503-1 composed of the collection of quality of the moving trajectory information. to create Using a similar method, the data conversion unit 2431 creates observation specifications 2503-2 from the time-series data 2501-2.
  • the certainty prediction model learning unit 243 creates one new teacher data 252-1 from one teacher data 250-1 as follows. First, the certainty prediction model learning unit 243 inputs the time-series data 2501-1 in the teacher data 250-1 to the learned discrimination model 234, and the type of floating matter finally output from the discrimination model 234. Get the estimation result of . Next, confidence prediction model learning unit 243 compares the type of floating matter represented by the estimation result of discrimination model 234 with the type of floating matter 2502-1 in teacher data 250-1 (block 251). Next, the certainty predictive model learning unit 243 sets the certainty 2521-1 set to the value according to the comparison result and the observation specification 2503-1 in the teacher data 250-1 as teacher data 252-1. create.
  • Confidence prediction model learning unit 243 creates one new teacher data from teacher data 250-2 by a similar method. As a result, 2 ⁇ n pieces of teacher data 252 - 1 and the like are finally generated from n pieces of teacher data 250 .
  • the certainty prediction model learning unit 243 uses the teacher data 252-1 etc. created as described above, the input is the observation specification of the time series data representing the movement trajectory of the floating object obtained by observation, and the output is A certainty prediction model 235 is generated by machine learning as the certainty of the estimation result of the discriminative model 234 estimated from the time-series data related to the observation specifications.
  • the number of teacher data used for learning the confidence prediction model 235 can be increased.
  • the number of time-series data representing movement trajectories of foreign matter is small.
  • the number of incorrect time-series data time-series data in which a bubble is identified as a foreign matter, and time-series data in which a foreign matter is identified as a bubble
  • a large amount of teacher data can be created from such a small number of time-series data.
  • FIG. 12 is a schematic diagram showing an example of a method of creating teacher data used for machine learning of the certainty prediction model 235 in modification 5. As shown in FIG. 12, the same reference numerals as in FIG. 11 denote the same parts, 252-1A is teacher data, and 2522-1 is the discrimination result of the discrimination model 234. In FIG. Referring to FIG. 12, certainty prediction model learning unit 243 creates one new teacher data 252-1A from one teacher data 250-1 as follows.
  • the certainty prediction model learning unit 243 inputs the time-series data 2501-1 in the teacher data 250-1 to the learned discrimination model 234, and acquires the estimation result 2522-1 output from the discrimination model 234. do.
  • the confidence prediction model learning unit 243 creates a confidence 2521-1 according to the result of comparing the type of floating matter represented by the estimation result 2522-1 and the type of floating matter in the teacher data 250-1. (block 251). Confidence predictive model learning unit 243 creates a set of confidence 2521-1, observation specification 2503-1 in teacher data 250-1, and estimation result 2522-1 as teacher data 252-1A.
  • the certainty prediction model learning unit 243 uses the teacher data 252-1A and the like created as described above, and inputs the observation specifications of the time series data representing the movement trajectory of the floating object obtained by observation and the time series The data is paired with the estimation result 2522-1 output from the discrimination model 234 when the data is input to the discrimination model 234, and the output is the confidence of the estimation result of the discrimination model 234 estimated from the time series data related to the above observation specifications. It is configured to generate a certainty prediction model 235 by machine learning.
  • the determination unit 244 may use the estimation result 2522-1 of the discriminative model 234 for confidence prediction. For example, the determination unit 244 assigns the observation specification list of the time-series data representing the movement trajectory of the floating object included in the tracking information 233 and the time-series data to the learned confidence prediction model 235 as the identification model 234. A set with the estimation result 2522-1 output from the discriminative model 234 when input is input, and the certainty factor of the estimation result of the discriminative model 234 outputted from the certainty factor prediction model 235 is obtained.
  • the confidence prediction model learning unit 243 obtains the estimation result 2522-1 by inputting the time series data 2501-1 to the learned discrimination model 234.
  • the method of obtaining estimation result 2522-1 estimated from partial time-series data 2501-1 is not limited to the above.
  • the certainty prediction model 235 by configuring the certainty prediction model 235 so as to output the identification result based on the feature amount up to the time series data in the middle, the entire time series data 2501 including the time series data 2501-1 can be generated.
  • An estimation result 2522-1 estimated from the partial time-series data corresponding to the time-series data 2501-1 may be obtained from the discrimination model 234 by inputting it to the trained discrimination model 234.
  • the identification model 234 is composed of, for example, an LSTM, and can output the identification result from the final stage as indicated by the solid line arrow, and can output the intermediate frame as indicated by the broken line arrow from the middle stage. It is configured to be able to output identification results with features up to.
  • FIG. 14 is a block diagram of an inspection system 500 according to a second embodiment of the invention.
  • the inspection system 500 comprises discriminant model learning means 501 , certainty prediction model learning means 502 and determination means 503 .
  • the discriminant model learning means 501 uses time-series data representing the movement trajectory of the object obtained by observation and the type of the object as first teacher data, and uses time series data representing the movement trajectory of the object obtained by observation. It is configured to learn a discriminative model for estimating object types from series data.
  • the discriminant model learning means 501 can be configured, for example, in the same manner as the discriminative model learning unit 242 in FIG. 2, but is not limited thereto.
  • Confidence prediction model learning means 502 uses time-series data representing the movement trajectory of the object obtained by observation, its observation specifications, and the type of object as second teacher data, and the object obtained by observation It is configured to learn a certainty prediction model that predicts the certainty of the estimation result of the discriminative model from the observed specifications of the time-series data representing the movement trajectory of the object.
  • the certainty predictive model learning means 502 can be configured, for example, in the same manner as the certainty predictive model learning unit 243 in FIG. 2, but is not limited thereto.
  • the determination means 503 is configured to estimate the type of object from the time-series data representing the movement trajectory of the object obtained by observation using the learned discrimination model. Further, the determination means 503 is configured to predict the confidence of the estimation result of the discriminant model from the observed data of the time-series data using the learned confidence prediction model.
  • the determination means 503 can be configured, for example, in the same manner as the determination section 244 in FIG. 2, but is not limited thereto.
  • the inspection system 500 configured as above operates as follows. That is, first, the discriminant model learning means 501 uses the time-series data representing the movement trajectory of the object obtained by observation and the type of the object as first teacher data, and uses the movement of the object obtained by observation. A discriminant model is learned to estimate the type of object from the time-series data representing the trajectory. Next, the certainty prediction model learning means 502 uses the time-series data representing the movement trajectory of the object obtained by observation, its observation specifications, and the type of object as second teacher data, and obtains by observation. It learns a certainty prediction model that predicts the certainty of the estimation result of the discriminative model from the observation data of the time-series data representing the movement trajectory of the object.
  • the determination means 503 uses the learned discrimination model to estimate the type of the object from the time-series data representing the movement trajectory of the object obtained by observation, and the learned confidence prediction model is used to predict the confidence of the estimation result of the discriminative model from the observed parameters of the time-series data.
  • the inspection system 500 configured and operated as described above, even if the time-series data representing a plurality of mutually similar movement trajectories obtained by observation from a plurality of different types of objects, the observation specifications are different. If so, we can differentiate between the confidences of the results estimated from the time-series data.
  • the certainty prediction model learning means 502 uses time-series data representing the movement trajectory of the object obtained by observation, its observation specifications, and the type of object as the second teacher data, This is for learning a certainty prediction model for predicting the certainty of the estimation result of the discriminative model from the observation specifications of the obtained time-series data representing the moving trajectory of the object.
  • the determining means 503 is for predicting the confidence of the estimation result of the discriminative model from the observed data of the time-series data using the learned confidence prediction model.
  • the present invention has been described above with several embodiments and modifications, the present invention is not limited to the above embodiments and modifications, and various additions and modifications are possible.
  • the present invention can also combine the embodiments and modifications described above.
  • the operation of performing identification and confidence prediction using the learned discrimination model and confidence model described in the first embodiment, and the learned discrimination model and confidence prediction model described in any of the modifications is an inspection system that concurrently or alternately performs identification and confidence prediction using .
  • the present invention can be used for inspection systems in general that estimate the type of an object from time-series data representing the movement trajectory of the object obtained by observation.
  • the present invention can be applied to an inspection system that inspects the presence or absence of foreign matter in a liquid enclosed in a container.
  • the present invention can be applied to a preclinical test system for investigating the safety of pharmaceuticals by determining the presence or absence of abnormalities in mice or the like from time-series data representing movement trajectories of mice or the like.
  • Time-series data representing the movement trajectory of an object obtained by observation and the type of the object are used as first teacher data, and the time-series data representing the movement trajectory of the object obtained by observation are used to determine the object.
  • a certainty prediction model learning means for learning a certainty prediction model for predicting the certainty of an estimation result of the discriminative model from data observation specifications; Using the learned discrimination model, estimating the type of the object from time series data representing the movement trajectory of the object obtained by observation, and using the learned confidence prediction model, the time series determination means for predicting the confidence level of the estimation result of the discriminative model from observation data data; inspection system.
  • the certainty prediction model learning means is configured such that the type of the object estimated from the time-series data in the second teacher data using the learned discrimination model is the object in the second teacher data. If it does not match the type, acquire a certainty that is lower than when it matches, and use the acquired certainty and the observation specifications in the second teacher data as third teacher data.
  • the inspection system learns the confidence prediction model by The inspection system according to Appendix 1.
  • the observation specifications include at least one of the length of the movement trajectory, the size of the target object, the start time of the movement trajectory, the observation location of the movement trajectory, and the quality of the time-series data. , 3.
  • the determination means corrects the determination result of the type of the object based on the determination result of the certainty factor. 4.
  • the inspection system according to any one of Appendices 1 to 3.
  • the confidence prediction model learning means learns the confidence prediction model using a predetermined output obtained by inputting the time-series data in the second teacher data to the learned discrimination model, 5.
  • the inspection system according to any one of Appendices 1 to 4.
  • the discriminant model learning means further learns the learned discriminant model using the confidence predicted by the learned confidence prediction model for learning control. 6.
  • the certainty prediction model learning means converts the second teacher data into a plurality of new teacher data, and each new teacher data is a plurality of new teacher data obtained by replacing the time-series data in the second teacher data. One time-series data after conversion to time-series data, the type of object in the second teacher data, and the observation specifications of the one partial time-series data,
  • the confidence prediction model learning means uses the new teacher data to learn the confidence prediction model. 7.
  • the inspection system according to any one of Appendices 1 to 6.
  • the confidence prediction model learning means machine-learns the confidence prediction model using a discrimination result obtained by inputting time-series data in the new teacher data to the learned discrimination model.
  • the inspection system according to Appendix 7.
  • Time-series data representing the movement trajectory of an object obtained by observation and the type of the object are used as first teacher data, and the time-series data representing the movement trajectory of the object obtained by observation are used to determine the object. learn a discriminant model that estimates the type of A time series representing the movement trajectory of an object obtained by observation using time-series data representing the movement trajectory of an object obtained by observation, its observation specifications, and the type of the object as second teacher data.
  • Time-series data representing the movement trajectory of an object obtained by observation and the type of the object are used as first teacher data, and the time-series data representing the movement trajectory of the object obtained by observation are used to determine the object.
  • a process of learning a discriminative model that estimates the type of A time series representing the movement trajectory of an object obtained by observation using time series data representing the movement trajectory of an object obtained by observation, its observation specifications, and the type of the object as second teacher data.
  • a process of learning a confidence prediction model that predicts the confidence of the estimation result of the discriminative model from the observation specifications of the data;
  • the type of the object is estimated from time series data representing the movement trajectory of the object obtained by observation, and using the learned confidence prediction model, the time series a process of predicting the degree of certainty of the estimation result of the discriminative model from the observed data data;
  • a computer-readable recording medium that records a program for performing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Chemical & Material Sciences (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Biochemistry (AREA)
  • Analytical Chemistry (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Image Analysis (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
  • Image Processing (AREA)
PCT/JP2021/018652 2021-05-17 2021-05-17 検査システム Ceased WO2022244075A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/JP2021/018652 WO2022244075A1 (ja) 2021-05-17 2021-05-17 検査システム
JP2023522022A JP7529151B2 (ja) 2021-05-17 2021-05-17 検査システム
US18/559,634 US12555251B2 (en) 2021-05-17 2021-05-17 Inspection system for decision making support

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/018652 WO2022244075A1 (ja) 2021-05-17 2021-05-17 検査システム

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US18/559,634 A-371-Of-International US12555251B2 (en) 2021-05-17 2021-05-17 Inspection system for decision making support
US19/428,833 Continuation US20260120297A1 (en) 2025-12-22 Inspection system for decision making support

Publications (1)

Publication Number Publication Date
WO2022244075A1 true WO2022244075A1 (ja) 2022-11-24

Family

ID=84141336

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/018652 Ceased WO2022244075A1 (ja) 2021-05-17 2021-05-17 検査システム

Country Status (3)

Country Link
US (1) US12555251B2 (https=)
JP (1) JP7529151B2 (https=)
WO (1) WO2022244075A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102723094B1 (ko) * 2023-05-11 2024-10-31 (주)빅아이 제품 검사장치
JP7659257B1 (ja) 2024-10-04 2025-04-09 ＡＩ ｉｎｓｉｄｅ株式会社 情報処理装置、方法、及びプログラム
WO2026070859A1 (ja) * 2024-09-26 2026-04-02 ジオインサイト合同会社 容器内に封入された液体内の異物の有無を判定するためのシステム及び方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116630947B (zh) * 2023-03-08 2026-01-20 京东方科技集团股份有限公司 异物检测方法及装置、非瞬态计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011153944A (ja) * 2010-01-28 2011-08-11 Hitachi High-Technologies Corp 自動分析装置
CN104215584A (zh) * 2014-08-29 2014-12-17 华南理工大学 一种基于高光谱图像技术区分大米产地的检测方法
US20180322660A1 (en) * 2017-05-02 2018-11-08 Techcyte, Inc. Machine learning classification and training for digital microscopy images
JP2019174346A (ja) * 2018-03-29 2019-10-10 富士通株式会社 検査方法、検査装置、及び検査プログラム
JP2019529882A (ja) * 2016-08-22 2019-10-17 アイリス インターナショナル， インコーポレイテッド 生体粒子の分類のシステム及び方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4724182B2 (ja) * 2004-07-30 2011-07-13 イーグル・ヴィジョン・システムズ・ベスローテン・フェンノートシャップ 容器を検査する方法及び装置
TWI648532B (zh) 2011-08-29 2019-01-21 美商安美基公司 用於非破壞性檢測-流體中未溶解粒子之方法及裝置
EP3475692A1 (en) * 2016-06-28 2019-05-01 H. Hoffnabb-La Roche Ag Inspection device
WO2019180868A1 (ja) * 2018-03-22 2019-09-26 日本電気株式会社 画像生成装置、画像生成方法および画像生成プログラム
WO2021214994A1 (ja) * 2020-04-24 2021-10-28 日本電気株式会社 検査システム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011153944A (ja) * 2010-01-28 2011-08-11 Hitachi High-Technologies Corp 自動分析装置
CN104215584A (zh) * 2014-08-29 2014-12-17 华南理工大学 一种基于高光谱图像技术区分大米产地的检测方法
JP2019529882A (ja) * 2016-08-22 2019-10-17 アイリス インターナショナル， インコーポレイテッド 生体粒子の分類のシステム及び方法
US20180322660A1 (en) * 2017-05-02 2018-11-08 Techcyte, Inc. Machine learning classification and training for digital microscopy images
JP2019174346A (ja) * 2018-03-29 2019-10-10 富士通株式会社 検査方法、検査装置、及び検査プログラム

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102723094B1 (ko) * 2023-05-11 2024-10-31 (주)빅아이 제품 검사장치
WO2026070859A1 (ja) * 2024-09-26 2026-04-02 ジオインサイト合同会社 容器内に封入された液体内の異物の有無を判定するためのシステム及び方法
JP7659257B1 (ja) 2024-10-04 2025-04-09 ＡＩ ｉｎｓｉｄｅ株式会社 情報処理装置、方法、及びプログラム

Also Published As

Publication number Publication date
JP7529151B2 (ja) 2024-08-06
JPWO2022244075A1 (https=) 2022-11-24
US12555251B2 (en) 2026-02-17
US20250078289A1 (en) 2025-03-06

Similar Documents

Publication Publication Date Title
WO2022244075A1 (ja) 検査システム
US10746665B2 (en) Methods and apparatus for classifying an artifact in a specimen
JP7359297B2 (ja) 検査システム
CA2976769C (en) Model-based methods and apparatus for classifying an interferent in specimens
US11035870B2 (en) Systems, methods and apparatus for identifying a specimen container cap
US11650197B2 (en) Methods and apparatus adapted to quantify a specimen from multiple lateral views
JP7234502B2 (ja) 方法、装置、及びプログラム
US20250321241A1 (en) Methods and apparatus for determining a viewpoint for inspecting a sample within a sample container
US20230031937A1 (en) Recycling system with discrete conveyor bins for improved accuracy
US20260120297A1 (en) Inspection system for decision making support
JP7848862B2 (ja) 異物検査装置
US20240394340A1 (en) Learning device
CN114599981B (zh) 用于在自动化诊断分析系统中试样的hiln确定中使用的训练图像的散列和检索的方法和设备
WO2020144881A1 (ja) 検品支援方法、及び検品支援システム
WO2025027717A1 (ja) 検査装置
US20260098817A1 (en) Inspection system
US20260086040A1 (en) Inspection system
WO2025154143A1 (ja) 検査装置
WO2025154144A1 (ja) 検査装置
EP4446730A1 (en) Foreign matter inspecting device
HK40068395A (en) Methods and apparatus for hashing and retrieval of training images used in hiln determinations of specimens in automated diagnostic analysis systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21940695

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023522022

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18559634

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21940695

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18559634

Country of ref document: US

WWG Wipo information: grant in national office

Ref document number: 18559634

Country of ref document: US