WO2023053728A1 - 表示処理装置、表示処理方法、及び、表示処理プログラム - Google Patents

表示処理装置、表示処理方法、及び、表示処理プログラム Download PDF

Info

Publication number
WO2023053728A1
WO2023053728A1 PCT/JP2022/030414 JP2022030414W WO2023053728A1 WO 2023053728 A1 WO2023053728 A1 WO 2023053728A1 JP 2022030414 W JP2022030414 W JP 2022030414W WO 2023053728 A1 WO2023053728 A1 WO 2023053728A1
Authority
WO
WIPO (PCT)
Prior art keywords
detection target
target
display processing
display
relevance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/030414
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
遼 池田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to EP22875603.7A priority Critical patent/EP4411356A4/en
Priority to CN202280061993.6A priority patent/CN117957439A/zh
Priority to JP2023550428A priority patent/JP7828357B2/ja
Publication of WO2023053728A1 publication Critical patent/WO2023053728A1/ja
Priority to US18/609,290 priority patent/US20240221343A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N23/00—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00
    • G01N23/02—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material
    • G01N23/04—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material and forming images of the material
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N23/00—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00
    • G01N23/02—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material
    • G01N23/06—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by transmitting the radiation through the material and measuring the absorption
    • G01N23/18—Investigating the presence of flaws defects or foreign matter
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/20—Drawing from basic elements
    • G06T11/23—Drawing from basic elements using straight lines or curves
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/06—Recognition of objects for industrial automation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Definitions

  • the present invention relates to a display processing device, a display processing method, and a display processing program, and more particularly, to a display processing device, a display processing method, and a display that display a detection target using a segmentation image determined using segmentation. It relates to a processing program.
  • non-destructive inspection As a method of inspecting defects in industrial products to be inspected, there is a non-destructive inspection that involves irradiating the industrial product with light or radiation.
  • non-destructive inspection the image obtained by irradiating the industrial product to be inspected with light rays or radiation is judged to be defective or non-defective for each pixel, called segmentation, and the image is divided into different areas. are being divided into
  • the area determined to be defective that is, the location where the defect is detected, is displayed by filling the defective area with a color corresponding to the defect type in advance. It is On the other hand, saving and passing all the coordinate information for painting to the display viewer used at the inspection site poses a problem of increasing the file capacity.
  • Patent Document 1 describes a defect detection method that properly combines defects and uses the smallest rectangle that covers the entire circumscribing rectangle for all detected defects after the combination.
  • a rectangle (bounding box) of a fixed size centered on the barycentric coordinates is defined for the continuous area detected by segmentation, and one rectangle is used for each defect area. Surrounding is done.
  • the present invention has been made in view of such circumstances, and provides a display processing device, a display processing method, and a display processing program for displaying an image in which it is easy to confirm the relevance of a plurality of detection target areas. With the goal.
  • a display processing apparatus discriminates an object to be detected and other objects for each pixel of a photographed image based on a photographed image of the object.
  • an acquisition unit that acquires a segmentation result
  • an extraction unit that extracts a detection target region from the segmentation result, and, when a plurality of discontinuous detection target regions are extracted, for determining the relevance between the detection target regions
  • a measurement unit that measures the feature amount of the area
  • a relevance determination unit that determines the relevance of a plurality of detection target areas based on the feature amount, and a plurality of detections based on the evaluation results determined by the relevance determination unit
  • An area determining unit that determines a target area to be displayed in an integrated display format among the target areas, and a drawing unit that draws the target area in the display format.
  • a display control unit that passes information necessary for displaying the target area to the display means and displays the target area.
  • the captured image is preferably a transmitted image captured by allowing radiation to pass through the object.
  • the display format is preferably at least one of a frame surrounding the target area, highlighting of the target area and other areas, and a mark indicating the target area.
  • the detection target is preferably a defect.
  • the relevance determination unit determines relevance for detection target regions having defects of the same type.
  • the feature amount is preferably the distance between detection target areas.
  • the relevance determination unit has a distance threshold determined from the characteristics of the target object, the past detection history, the pass/fail criteria for the detection target, and the type of the detection target. It is preferable to determine the relevance from the feature amount measured in the section.
  • the feature amount is preferably position information of the detection target.
  • the feature amount is preferably the regularity of the detection target area.
  • the area determination unit determines a target area and a sub-target area within the target area in which the generation density distribution of the detection target is different from that of other areas inside the target area, and draws the sub-target area.
  • the unit renders the target area in the first display format and renders the sub-target area in the second display format.
  • the target area includes different types of defects
  • the drawing unit may add information indicating that the target area includes different types of defects. preferable.
  • the information is preferably given by at least one of the display color of the frame surrounding the target area, the line type, and the mark.
  • a display processing method discriminates an object to be detected and other objects for each pixel of a photographed image based on a photographed image of the object. , an acquisition step of acquiring a segmentation result, an extraction step of extracting a detection target region from the segmentation result, and when a plurality of discontinuous detection target regions are extracted, to determine the relevance between the detection target regions A measurement step of measuring the feature amount of, a relevance determination step of determining the relevance of a plurality of detection target regions based on the feature amount, and a plurality of detections based on the evaluation results determined in the relevance determination step A region determination step of determining a target region to be displayed in an integrated display format, and a drawing step of drawing the target region in a display format.
  • a display processing method acquires information of a detection target area included in a segmentation result obtained by distinguishing a detection target and other targets for each pixel of an object.
  • an acquisition step a measurement step of measuring a feature amount for judging the relevance between the detection target areas when there are a plurality of discontinuous detection target areas; a relevance determination step of determining relevance of the regions; and a region determination step of determining a target region in which the plurality of detection target regions should be displayed in an integrated display format based on the evaluation result determined in the relevance determination step. And prepare.
  • a display processing program discriminates an object to be detected and other objects for each pixel of a photographed image based on a photographed image of the object.
  • an acquisition function that acquires the segmentation result, an extraction function that extracts the detection target area from the segmentation result, and when multiple non-contiguous detection target areas are extracted, to determine the relevance between the detection target areas
  • a measurement function that measures the feature amount of each area, a relevance determination function that determines the relevance of multiple detection target areas based on the feature amount, and a plurality of detections based on the evaluation results determined by the relevance determination function.
  • a computer is made to implement a region determination function for determining a target region to be displayed in an integrated display format and a drawing function for drawing the target region in a display format.
  • FIG. 1 is a block diagram showing a defect display processing device according to one embodiment of the present invention.
  • FIG. 2 is a block diagram showing an example of the image processing section 24.
  • FIG. 3 is a block diagram showing an example of object image data.
  • FIG. 4 is a block diagram showing an example of product data.
  • FIG. 5 is a block diagram showing an example of an imaging system.
  • FIG. 6 is a flowchart illustrating a defect indication processing method according to one embodiment of the present invention.
  • FIG. 7 is a diagram showing an example of display processing according to one embodiment of the present invention.
  • FIG. 8 is a diagram showing another example of display processing according to one embodiment of the present invention.
  • FIG. 9 is a diagram showing still another example of display processing according to one embodiment of the present invention.
  • FIG. 10 is a diagram showing still another example of display processing according to one embodiment of the present invention.
  • FIG. 11 is a diagram showing still another example of display processing according to one embodiment of the present invention.
  • a display processing device, a display processing method, and a display processing program according to the present invention will be described below with reference to the accompanying drawings.
  • a defect display processing apparatus, a defect display processing method, and a defect display processing program will be described below as an embodiment of the present invention.
  • FIG. 1 is a block diagram showing a defect display processing device (display processing device) 10 according to one embodiment of the present invention.
  • the defect display processing device 10 is a device that performs defect display processing from a photographed image of an industrial product of a target, and is a device for assisting an inspector in diagnosing defects in the target.
  • the defect display processing apparatus 10 includes a control unit 12, an operation unit 14, an input/output interface (hereinafter referred to as an I/F (interface)) 16, a display unit 18, a buffer memory. 20 , an image recognition unit 22 , an image processing unit 24 and a recording unit 26 .
  • the control section 12 includes a CPU (Central Processing Unit) that controls the operation of each section of the defect display processing device 10 .
  • the control unit 12 receives an operation input from the inspector via the operation unit 14, and transmits a control signal corresponding to the operation input to each unit of the defect display processing device 10 to control the operation of each unit.
  • CPU Central Processing Unit
  • the operation unit 14 is an input device that receives operation input from the inspector, and includes a keyboard for character input, a pointer displayed on the display unit 18, and a pointing device (mouse, trackball, etc.) for operating icons and the like. contains.
  • a touch panel may be provided on the surface of the display unit 18 instead of or in addition to the means enumerated above.
  • the I/F 16 is means for communicating with external devices via the network NW.
  • wired communication eg, LAN (Local Area Network), WAN (Wide Area Network), Internet connection, etc.
  • wireless communication eg, LAN, WAN, Internet connection, etc.
  • the defect display processing device 10 can receive input of object photographed data D100 including photographed image data of the object OBJ photographed by the photographing system 100 via the I/F 16 .
  • the method of inputting the object image data D100 from the image capturing system 100 to the defect display processing device 10 is not limited to communication via the network NW listed above.
  • the defect display processing device 10 and the imaging system 100 may be connected by a USB (Universal Serial Bus) cable, Bluetooth (registered trademark), infrared communication, etc., and the object photographing data D100 is sent to the defect display processing device 10.
  • the photographed image data may be stored in a detachable and readable memory card, and the photographed image data may be input to the defect display processing device 10 via this memory card.
  • the defect display processing device 10 can communicate with a product database (product DB (database)) 200 via the network NW.
  • product DB 200 stores product data D200 for each target industrial product.
  • the control unit 12 retrieves and reads object specifying information for specifying an object from the photographed image data of the object OBJ acquired from the photographing system 100, and outputs product data D200 corresponding to the read object specifying information to the product. It is possible to acquire from DB200. By using this product data D200, it becomes possible to detect defects according to the type or characteristics of the object OBJ.
  • product DB 200 may be installed on the network NW as in the present embodiment so that the manufacturer or the like can update the product data D200, or may be installed in the defect display processing device 10.
  • the display unit (display means) 18 is a device for displaying captured images and segmentation images (segmentation results).
  • a liquid crystal monitor can be used as the display unit 18.
  • the buffer memory 20 is used as a work area for the control unit 12 and as an area for temporarily storing image data output to the display unit 18 .
  • the recording unit 26 is means for storing data including control programs used by the control unit 12 .
  • a device including a magnetic disk such as HDD (Hard Disk Drive), a device including flash memory such as eMMC (embedded Multi Media Card), SSD (Solid State Drive), etc. can be used.
  • the recording unit 26 stores object photographed data D100 and product data D200.
  • the image recognition unit 22 (acquisition unit) identifies the type of object in the captured image using the defect type identification model.
  • the defect type identification model is a model that uses an object image as input data and a segmentation image (segmentation result) that is a defect type identification result as output data.
  • the obtained segmentation image is stored in the recording unit 26 .
  • the segmentation image can be used to identify the types of defects in the image of the object (e.g., foreign matter, cracks, scratches, air bubbles, gas defects, wear, rust, and welding defects (overfill, poor penetration, spatter, undercut), etc.). ) is an image that has been identified on a pixel-by-pixel basis as indicated by different colors.
  • a technique for determining the types of defects in such an image on a pixel-by-pixel basis is called semantic segmentation.
  • a machine learning model that performs segmentation includes, for example, a U-shaped convolutional neural network (U-Net; U-Shaped Neural Network).
  • the image processing unit 24 reads the segmentation image from the image recognition unit 22 or the recording unit 26, and extracts the defect area (detection target area). Then, a feature amount for judging the relationship between the extracted defect areas is measured, and the relationship between the defect areas is judged based on this feature amount. Furthermore, based on the evaluation result of determining the relevance, a target area to be displayed in an integrated display format is determined, and a display format indicating the target region is drawn. The image processing section 24 outputs these results and information to the buffer memory 20 .
  • the control unit 12 uses the data output to the buffer memory 20 to create a display image in which a plurality of defective areas are displayed in an integrated display format on the segmentation image, and displays this display image on the display unit 18. Let This enables the inspector to interpret the image displayed on the display unit 18 and inspect the object OBJ.
  • FIG. 2 is a block diagram showing an example of the image processing unit 24.
  • the image processing section 24 includes an extraction section 240 , a measurement section 242 , a relevance determination section 244 , an area determination section 246 and a drawing section 248 .
  • the extraction unit 240 extracts defects of the object OBJ by detecting different colors from the segmentation image. Thereby, the position and shape of the defect are specified.
  • the measurement unit 242 measures feature amounts for determining the relevance between the defect areas.
  • the feature amounts include the distance (pixels) between defect areas for judging the relevance, the positional information of the defect areas, and the regularity of the distribution of the defect areas.
  • the center of the defect area means the center of the circle or ellipse that circumscribes the defect area.
  • the positional information between defects includes the presence or absence of a luminance level difference (edge) between defect areas, the difference in texture around the defect areas, and the like.
  • the regularity of the distribution of the defect areas includes the fact that two or more defect areas are distributed at equal intervals, the directionality of the defect areas is linear or curved, and the like.
  • the relevance determination unit 244 determines relevance between defect areas based on the feature amount. When determining the relevance based on the distance between defect areas, the relevance determination unit 244 holds a distance threshold determined from the characteristics of the object, the past detection history, the acceptance/rejection criteria for defects, the type of defect, and the like. Based on the result of the threshold value and the distance between the measured defect areas, the presence or absence of relevance is determined. Also, if a difference in brightness is observed between the defects and if a difference in texture is observed, it is determined that there is no relationship. In addition, when the shape of the object is known, such as a 3D model, it is determined that multiple layers of parts overlap in the depth direction near the defect based on the correspondence with the position information of the defect in the segmentation image.
  • the related determined to be a potential defect if two or more defect areas are distributed at equal intervals, and if the directionality of the defect areas is linear or curvilinear, the related determined to be a potential defect. Furthermore, the shape information of the defect area, for example, the directionality considering the length may be taken into consideration. It can be judged as defective. On the other hand, if two defects face each other along their short axes, they can be determined to be unrelated and separate defects.
  • the area determination unit 246 determines a target area in which a plurality of defect areas should be displayed in an integrated display format. Defect areas of the same type and determined to be related by the relevance determining unit 244 are determined as target areas to be displayed in an integrated display format. In addition, although the defects are of different types, if each defect is displayed in a single display format, the display may overlap, making it difficult to confirm the defect area on the display unit 18 in some cases. At this time, different types of defects are determined as target areas to be displayed in an integrated display format.
  • the drawing unit 248 draws the target area determined by the area determination unit 246 in an integrated display format.
  • Examples of the integrated display format include surrounding the target area with a frame, changing the brightness of the target area and areas other than the target area to highlight them, and displaying them with marks such as arrows.
  • FIG. 3 is a block diagram showing an example of object photographed data.
  • the object photographing data D100 includes object specifying information, photographed image data, photographing condition data, and illumination condition data.
  • the object specifying information is information for specifying the object OBJ, and includes, for example, information indicating the product name, product number, ID (identification) information, manufacturer name, and technical classification of the object OBJ.
  • the photographed image data is image data obtained by photographing the object OBJ (for example, an X-ray transmission image or a visible light image), and includes image resolution or resolution information.
  • the shooting condition data is stored for each captured image data of the object OBJ, and includes the shooting date and time of each shot image data, the shooting target location, the distance between the object OBJ and the imaging device at the time of shooting, and the angle with respect to the imaging device. contains information indicating
  • the illumination condition data indicates the type of radiation used for imaging the object OBJ (for example, X-ray, visible light, transmitted light, or reflected light), irradiation intensity, irradiation angle, and parameters of tube current and tube voltage. contains information.
  • FIG. 4 is a block diagram showing an example of product data.
  • product data D200 includes product identification information, product attribute information, and inspection area designation information.
  • the product data D200 may be recorded in the recording unit 26 in association with the object image data D100 via the object specifying information and the product specifying information, or may be acquired from the product DB 200 each time a defect inspection is performed. You may do so.
  • Product identification information is information for identifying a product, and includes, for example, information indicating the product name, product number, manufacturer name, and technical classification.
  • Product attribute information includes, for example, information indicating the materials, dimensions, and usage of each part of the product.
  • the information indicating the use of the product includes, for example, information on the name, type, processing state, and mounting method (for example, joints, welds, screwing, fitting, and soldering) of equipment to which the product is attached.
  • the product attribute information includes defect occurrence information.
  • the defect occurrence information includes, for example, the date and time of the past inspection, the material of the object OBJ, the type of defect that occurred in the past (e.g., foreign matter, crack, scratch, air bubble, welding gas defect, wear, rust, etc.), and position information. , shape, size, depth, location of occurrence (coordinates of location, thickness of material, processing state (e.g. joints, welds, etc.)), frequency information on defect occurrence frequency, at least one defect capture image contains information.
  • the inspection area specification information is information indicating the inspection area specified by the manufacturer of each product (for example, information including the position of the inspection area, whether or not defects have occurred in the past, and defect occurrence frequency information such as defect occurrence frequency). based on information).
  • the inspection area designation information is created, for example, by specifying locations where defects are likely to occur statistically and structurally, based on information obtained when the manufacturer or the like repaired the product in the past.
  • the defect occurrence information includes, for example, at least one of the past inspection date and time, the material of the object OBJ, the type, shape, size, depth, occurrence location, and defect capture image of the defect that occurred in the past.
  • FIG. 5 is a block diagram showing an example of an imaging system.
  • the imaging system 100 is for imaging an object OBJ placed in an imaging room 114, and as shown in FIG. and radiation sources 110,112.
  • the imaging control unit 102 includes a CPU (Central Processing Unit) that controls the operation of each unit of the imaging system 100 .
  • the imaging control unit 102 receives an operation input from an operator (photographer) via the imaging operation unit 104, and transmits a control signal corresponding to the operation input to each unit of the imaging system 100 to control the operation of each unit.
  • the imaging operation unit 104 is an input device that receives operation inputs from the operator, and includes a keyboard for character input, a pointer displayed on the display unit 18, and a pointing device (mouse, trackball, etc.) for operating icons and the like. contains.
  • the operator inputs information about the object OBJ and commands the imaging device 108 to execute imaging (setting of imaging conditions such as exposure time, focal length, and aperture, imaging angle, imaging location, etc.) via the imaging operation unit 104.
  • the image recording unit 106 records image data (received light image) of the object OBJ photographed by the imaging device 108 .
  • Information for identifying the object OBJ is recorded in the image recording unit 106 in association with the image data.
  • the imaging device 108 and radiation sources 110 and 112 are arranged inside an imaging room 114 .
  • the radiation sources 110 and 112 are, for example, X-ray sources, and the partition walls and doorways between the imaging room 114 and the outside are X-ray protected by X-ray protective materials (for example, lead, concrete, etc.). there is It should be noted that there is no need to use the protected photography room 114 when photographing the object OBJ by irradiating it with visible light.
  • the radiation sources 110 and 112 irradiate the object OBJ placed in the imaging room 114 with radiation according to instructions from the imaging control unit 102 .
  • the imaging apparatus 108 receives radiation emitted from the radiation source 110 to the object OBJ and reflected by the object OBJ, or the radiation emitted from the radiation source 112 to the object OBJ in accordance with an instruction to execute imaging from the imaging control unit 102 .
  • the object OBJ is photographed by receiving the radiation transmitted through the object OBJ.
  • a light receiving panel can be used when irradiating the object OJT with an X-ray source, and a camera can be used when irradiating visible light.
  • the object OBJ is held in the imaging room 114 by a holding member (for example, a manipulator, a mounting table, a movable mounting table) (not shown), and the object OBJ is attached to the imaging device 108 and the radiation sources 110 and 112. Distance and angle are adjustable.
  • the operator can control the relative positions of the object OBJ, the imaging device 108, and the radiation sources 110 and 112 via the imaging control unit 102, and can image a desired portion of the object OBJ. .
  • the radiation sources 110 and 112 finish irradiating the object OBJ with radiation in synchronization with the end of the execution of imaging by the imaging device 108 .
  • the imaging device 108 is arranged inside the photography room 114, but the imaging device 108 can be arranged outside if the object OBJ in the photography room 114 can be photographed. may be
  • one imaging device 108 and two radiation sources 110 and 112 are provided, but the number of imaging devices and radiation sources is not limited to this. For example, there may be multiple imaging devices and radiation sources, or there may be one each.
  • FIG. 6 is a flowchart illustrating a defect indication processing method according to one embodiment of the present invention.
  • the defect display processing device 10 acquires object photographed data D100 including photographed image data (photographed image) of the object OBJ from the photographing system 100 via the I/F 16 .
  • the image recognition unit 22 acquires a segmentation image (segmentation result), which is a defect type identification result, from the acquired captured image data using the defect type identification model (step S12: acquisition step).
  • the extraction unit 240 of the image processing unit 24 extracts the defect area from the segmentation image (step S14: extraction step).
  • the segmentation image shows different types of defects with different colors, and by detecting different colors, the defect area is extracted.
  • the measurement unit 242 of the image processing unit 24 measures feature amounts for determining the relevance between the defect areas extracted in the extraction step (step S14) (step S16: measurement step).
  • the distance (pixels) between the defect areas for determining the relevance is measured.
  • the distance between the defect areas the distance between the centers of the defect areas, the distance between the centers of gravity, or the shortest distance between the ends is measured.
  • the relevance determination unit 244 of the image processing unit 24 determines the relevance between the defect areas based on the feature amount (distance between the defect areas) measured in the measurement step (step S16) (step S18: relevance determination step).
  • the relevance determination unit 244 holds a distance threshold determined from the characteristics of the target object, the past detection history, the pass/fail criteria for defects, and the like. Object characteristics, past detection history, acceptance/rejection criteria for defects, and the like can be obtained from the object identification information of the object photographing data D100 and the product attribute information of the product data D200.
  • the reference threshold can be determined according to the type of defect. For example, considering that air bubble defects (called porosity) may occur in clusters over a wide area, in the case of air bubble defects, the distance between defect areas that are determined to be relevant can be lengthened.
  • FIG. 7 is a diagram showing an example of display processing according to an embodiment of the present invention.
  • 700A in FIG. 7 is a diagram in which two defects are determined to be related, and 700B in FIG.
  • FIG. 10 is a diagram in which two defects are determined to be irrelevant;
  • 700A is a view with defect regions 302, 304 showing two circular defects with diameters of 15 px on the segmentation image 300, and 10 px from the edges of the defect regions 302, 304 from each other.
  • 700B is a view of the defect regions 306, 308 showing two circular defects with a diameter of 2 px on the segmentation image 300, 10 px from the edges of the defect regions 306, 308 from each other. Even if the distance between the defect areas is the same, the ratio of the distance to the size of the defect itself is different, so it can be determined that the defect areas of 600A are related. Also, it can be determined that the defect areas of 600B are not related to each other.
  • the region determination unit 246 of the image processing unit 24 determines a target region in which a plurality of defect regions should be displayed in an integrated display format based on the evaluation result determined in the relationship determination step (step S18).
  • Step S20 area determination step.
  • the target area to be displayed in an integrated display format is an area including a plurality of defective areas determined to be related in the relatedness determining step (step S18).
  • defect areas 302 and 304 are determined to be relevant defects, and the area containing defect areas 302 and 304 is determined as target area 309 .
  • the defect area 306 and the defect area 308 are defects determined to be unrelated defects, and the area including the defect area 306 and the area including the defect area 308 are each the target Areas 311 and 313 are determined.
  • the drawing unit 248 of the image processing unit 24 draws each of the target areas 309, 311, and 313 determined in the area determination process (step S20) in an integrated display format (step S22: drawing process).
  • the target region 309 is drawn with a rectangular frame 310 as a display format to indicate the target region 309.
  • the defect areas 306 and 308 are defects determined to have no relationship in the relevance determination step (step S18).
  • 313 are drawn with rectangular frames 312, 314 to indicate the regions of interest.
  • the target area is indicated by enclosing it with a rectangular frame as an integrated display format for drawing the target area, but it is not limited to this.
  • the target area can be indicated by highlighting by maintaining the brightness of the target area and reducing the brightness around the target area.
  • the region of interest can be indicated by an arrow.
  • the shape of the frame is not limited to a rectangle, and may be other shapes such as a circle and an ellipse.
  • the line type is not limited, for example, the line type of the frame is indicated by a dashed line.
  • the control unit 12 determines the segmentation image acquired in the acquisition step (step S12), the target region determined in the region determination step (step S20), and the rectangular frames 310, 312, and 314 drawn in the drawing step (step S22). Based on the position information, a display image (see FIG. 7) showing the target area on the segmentation image is created, and this information is passed to the display unit 18. FIG. As a result, an inspection image in which the target area including the related defects is displayed is displayed on the display unit 18 .
  • FIG. 8 is a diagram showing another example of display processing according to an embodiment of the present invention, and is a diagram in which relevance is determined based on the regularity of defect areas.
  • 800A in FIG. 8 is a diagram in which two defects are determined to be relevant
  • 800B in FIG. 8 is a diagram in which two defects are determined to be irrelevant.
  • 800A is a diagram in which defect areas 322 and 324 representing two elliptical defects with a major axis of 10 px and a minor axis of 3 px are arranged continuously in the major axis direction with a center distance of 8 px on the segmentation image 300 .
  • defect areas 326 and 328 representing two elliptical defects having a major axis of 10 px and a minor axis of 3 px are continuously arranged at a center distance of 8 px on the segmentation image 300. Regions 326, 328 are positioned with minor axes facing each other. Even if the distance between defect areas is the same, defect areas 322 and 324 that are continuous along the long axis direction can be determined as related defects, and defect areas 326 and 328 that face each short axis can be determined as related defects. There is no difference between them and they can be determined as separate defects.
  • the defect areas 322 and 324 have been determined to be relevant defects, the area including the defect areas 322 and 324 is determined as the target area 329 , and the target area 329 is indicated by a frame 330 . Moreover, the defect areas 326 and 328 are determined to be irrelevant defects, and the area including the defect area 326 and the area including the defect area 328 are determined as target areas 331 and 333, respectively. , the region of interest is indicated by frame 334 .
  • FIG. 9 is a diagram showing still another example of display processing according to one embodiment of the present invention.
  • the segmentation image 300 shown in FIG. 9 has a defect area 342 showing 30 or more defects.
  • the plurality of defect regions 342 are oriented in the same direction in the longitudinal direction, and have curved continuity in the lateral direction. Therefore, these defect areas 342 are a series of defects (flaws) and can be determined to be related defects. Accordingly, the area containing the series of defect areas 342 is determined as the area of interest 344 , and the area of interest 344 is indicated by a frame 346 .
  • defect position information can be used as a feature quantity for determining the relevance of defect areas.
  • the positional information of defects if a difference in luminance is observed between defect areas on the segmentation image, it is determined that the defects are separate defects separated in the depth direction of the captured image, and unrelated defects are identified. It can be determined that Moreover, even when the textures around the defect area are different, the defects generated in different areas can be determined as separate defects, and can be determined as unrelated defects.
  • the shape of the object is known in advance from the 3D model of the object, etc., and it is determined that multiple layers of parts overlap in the depth direction near the defect from the correspondence with the position information of the defect in the segmentation image, , can be determined to be separate defects and can be determined to be unrelated defects.
  • FIG. 10 is a diagram showing still another example of display processing according to an embodiment of the present invention, and is a diagram showing an image in which defects of clusters of bubbles (porosity) have occurred.
  • Defective areas 362, which indicate air bubble defects, may occur in clusters over a wide area.
  • the defect areas 362 indicating these defects of air bubbles are determined to be related defects.
  • regions including these defect regions 362 are determined as target regions 364.
  • regions where the defect occurrence density distribution differs from other regions inside the target region 364 are determined as sub-target regions 366 and 368.
  • a region with a high defect density distribution is defined as a first sub-target region 366, and a region with a low defect density distribution is defined as a second sub-region 368.
  • the drawing step (step S22) draws the target region 364 in a rectangular first frame (first display format) 370, and renders the first sub-target region 366 and the second sub-target region 368 respectively in a rectangular second frame. are drawn in frames (second display format) 372 and 374 .
  • the first display format and the second display format are not limited to frames, and can be drawn using the above-described highlight display or marks.
  • FIG. 10 shows an example of a two-level hierarchical structure having a first sub-target area and a second sub-target area within the target area. It is also possible to have a hierarchical structure of three or more stages.
  • FIG. 11 is a diagram showing still another example of display processing according to one embodiment of the present invention.
  • FIG. 11 is a diagram in which regions containing different types of defects are determined as target regions.
  • the defect area 382 and the defect areas 384 and 386 are distinguished by different colors and are different defect types. Moreover, although the defect area 384 and the defect area 386 are of the same type of defect, the defect area 382 exists between the defect area 384 and the defect area 386, so the defects are not related to each other.
  • areas including each of the defect areas 382, 384, and 386 are determined as target areas, and the target areas are displayed by drawing each of them in an individual display format.
  • the defect areas 382, 384, and 386 are close to each other, there is a concern that if the display formats are drawn individually, they will overlap, making it difficult to understand the defects.
  • the defect regions 382 and 384 including different types of defects and unrelated defects are determined in the region determination step (step S20).
  • 386 is determined as a target area 388, and the target area 388 is drawn in an integral display format in the drawing step (step S22).
  • 1100A in FIG. 11 is a diagram showing a display format in which the target area 388 is displayed with a frame divided by a plurality of display colors.
  • a frame 390 surrounding the target area 388 is displayed by a side 390A given the first color and a side 390B given the second color.
  • the first and second colors correspond to the colors of the defect area 382 and the defect areas 384 and 386 used to distinguish the defect types in the segmentation image 300 . Thereby, the type of defect can be read from the display of the frame 390 .
  • 1100B of FIG. 11 is a diagram showing a display format in which the target area 388 is displayed with a double frame 391.
  • a frame 391 surrounding the target area 388 is displayed with an inner frame 391A given the first color and an outer frame 391B given the second color.
  • the first and second colors correspond to the colors of the defect area 382 and the defect areas 384 and 386 used to distinguish the defect types in the segmentation image 300 .
  • the type of defect can be read from the display in frame 389 .
  • 1100C of FIG. 11 is a diagram showing a display format in which the target area 388 is displayed with a dashed frame 392.
  • the present embodiment it is possible to confirm the detailed shape and distribution of the detection target and extract information necessary for inspection from the detailed discrimination results for each pixel between the detection target and other targets by segmentation. .
  • multiple inspection objects can be grouped and drawn in an integrated display format. can be made
  • the detection target is described as a defect, but the detection target is not limited to a defect.
  • the detection target is not limited to a defect. For example, it is possible to detect minute flaws that meet the product standard but have deformation.
  • the present invention is not limited to inspection applications, and can be used for applications other than inspection, such as object analysis, product classification, and evaluation.
  • the present invention can also be implemented as a program (display processing program) that causes a computer to implement the above processing, or as a non-temporary recording medium or program product that stores such a program.
  • a display processing program By applying such a display processing program to a computer, the computation means, recording means, etc. of the computer can be replaced with the acquisition function, extraction function, measurement function, relationship determination function, area determination function, and drawing function of the display processing program. It is possible to function as
  • defect display processing device 12 control unit 14 operation unit 16 input/output interface (I/F) 18 display unit 20 buffer memory 22 image recognition unit 24 image processing unit 26 recording unit 100 imaging system 102 imaging control unit 104 imaging operation unit 106 image recording unit 108 imaging device 110 radiation source 112 radiation source 114 imaging room 200 product database (product DB ) 240 extraction unit 242 measurement unit 244 relevance determination unit 246 region determination unit 248 drawing unit 300 segmentation image 302 defect region 304 defect region 306 defect region 308 defect region 309 target region 310 frame 311 target region 312 frame 313 target region 314 frame 322 defect Area 324 Defect area 326 Defect area 328 Defect area 329 Target area 330 Frame 331 Target area 332 Frame 333 Target area 334 Frame 342 Defect area 344 Target area 346 Frame 362 Defect area 364 Target area 366 Sub-target area (first sub-target area ) 368 sub-region of interest (second sub-region of interest) 370 First frame 372 Second frame 374 Second frame 382 Defective area 384 De

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Pathology (AREA)
  • Immunology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biochemistry (AREA)
  • Analytical Chemistry (AREA)
  • Chemical & Material Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)
PCT/JP2022/030414 2021-09-29 2022-08-09 表示処理装置、表示処理方法、及び、表示処理プログラム Ceased WO2023053728A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP22875603.7A EP4411356A4 (en) 2021-09-29 2022-08-09 DISPLAY PROCESSING DEVICE, DISPLAY PROCESSING METHOD, AND DISPLAY PROCESSING PROGRAM
CN202280061993.6A CN117957439A (zh) 2021-09-29 2022-08-09 显示处理装置、显示处理方法及显示处理程序
JP2023550428A JP7828357B2 (ja) 2021-09-29 2022-08-09 表示処理装置、表示処理方法、及び、表示処理プログラム
US18/609,290 US20240221343A1 (en) 2021-09-29 2024-03-19 Display processing device, display processing method, and display processing program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-159740 2021-09-29
JP2021159740 2021-09-29

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/609,290 Continuation US20240221343A1 (en) 2021-09-29 2024-03-19 Display processing device, display processing method, and display processing program

Publications (1)

Publication Number Publication Date
WO2023053728A1 true WO2023053728A1 (ja) 2023-04-06

Family

ID=85782287

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/030414 Ceased WO2023053728A1 (ja) 2021-09-29 2022-08-09 表示処理装置、表示処理方法、及び、表示処理プログラム

Country Status (5)

Country Link
US (1) US20240221343A1 (https=)
EP (1) EP4411356A4 (https=)
JP (1) JP7828357B2 (https=)
CN (1) CN117957439A (https=)
WO (1) WO2023053728A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025023191A1 (ja) * 2023-07-21 2025-01-30 富士フイルム株式会社 制御装置、制御方法、及び制御プログラム

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118318157A (zh) * 2022-10-21 2024-07-09 株式会社东芝 判断装置

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0563897A1 (en) * 1992-03-30 1993-10-06 Fuji Photo Film Co., Ltd. Defect inspection system
JP2004012177A (ja) * 2002-06-04 2004-01-15 Dainippon Screen Mfg Co Ltd 欠陥検出装置、欠陥検出方法、輪郭抽出方法、輪郭平滑化方法、領域結合方法およびプログラム
JP2007322256A (ja) * 2006-06-01 2007-12-13 Dainippon Screen Mfg Co Ltd ムラ検査方法、ムラ検査装置およびプログラム
JP2008241298A (ja) 2007-03-26 2008-10-09 Dainippon Screen Mfg Co Ltd 欠陥検出方法
JP2010164318A (ja) * 2009-01-13 2010-07-29 Nippon Steel Corp 欠陥検査方法および欠陥検査装置
JP2011058940A (ja) * 2009-09-09 2011-03-24 Panasonic Electric Works Co Ltd 外観検査装置、外観検査方法
JP2015040827A (ja) * 2013-08-23 2015-03-02 シャープ株式会社 欠陥判定装置および欠陥判定方法
JP2016200444A (ja) * 2015-04-08 2016-12-01 一般財団法人電力中央研究所 ボイドの計測方法、計測装置、及び計測プログラム
CN111462113A (zh) * 2020-04-24 2020-07-28 上海精测半导体技术有限公司 无图形晶圆的复检方法
JP2020144688A (ja) * 2019-03-07 2020-09-10 株式会社Ihi 学習方法、学習装置、及び当該学習装置を備える溶接欠陥検出装置

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4211092B2 (ja) * 1998-08-27 2009-01-21 株式会社Ｉｈｉ 放射線透過検査における溶接欠陥自動検出法
JP2008170256A (ja) * 2007-01-11 2008-07-24 Seiko Epson Corp 欠陥検出方法、欠陥検出プログラムおよび検査装置
JP4262297B2 (ja) * 2008-07-25 2009-05-13 大日本スクリーン製造株式会社 クラスタ生成装置、欠陥分類装置、クラスタ生成方法およびプログラム
JP5943722B2 (ja) * 2012-06-08 2016-07-05 三菱重工業株式会社 欠陥判定装置、放射線撮像システム、及び欠陥判定方法
JP6654649B2 (ja) * 2015-12-14 2020-02-26 株式会社ニコン・トリンブル 欠陥検出装置及びプログラム
US10325363B2 (en) * 2016-01-15 2019-06-18 Instrumental, Inc. Methods for automatically generating a common measurement across multiple assembly units
KR102814354B1 (ko) * 2019-12-05 2025-05-30 차나안 브라이트 사이트 컴퍼니 리미티드 문자 분할 방법, 장치 및 컴퓨터 판독 가능 저장 매체
WO2022116109A1 (en) * 2020-12-03 2022-06-09 Boe Technology Group Co., Ltd. Computer-implemented method for defect analysis, apparatus for defect analysis, computer-program product, and intelligent defect analysis system

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0563897A1 (en) * 1992-03-30 1993-10-06 Fuji Photo Film Co., Ltd. Defect inspection system
JP2004012177A (ja) * 2002-06-04 2004-01-15 Dainippon Screen Mfg Co Ltd 欠陥検出装置、欠陥検出方法、輪郭抽出方法、輪郭平滑化方法、領域結合方法およびプログラム
JP2007322256A (ja) * 2006-06-01 2007-12-13 Dainippon Screen Mfg Co Ltd ムラ検査方法、ムラ検査装置およびプログラム
JP2008241298A (ja) 2007-03-26 2008-10-09 Dainippon Screen Mfg Co Ltd 欠陥検出方法
JP2010164318A (ja) * 2009-01-13 2010-07-29 Nippon Steel Corp 欠陥検査方法および欠陥検査装置
JP2011058940A (ja) * 2009-09-09 2011-03-24 Panasonic Electric Works Co Ltd 外観検査装置、外観検査方法
JP2015040827A (ja) * 2013-08-23 2015-03-02 シャープ株式会社 欠陥判定装置および欠陥判定方法
JP2016200444A (ja) * 2015-04-08 2016-12-01 一般財団法人電力中央研究所 ボイドの計測方法、計測装置、及び計測プログラム
JP2020144688A (ja) * 2019-03-07 2020-09-10 株式会社Ihi 学習方法、学習装置、及び当該学習装置を備える溶接欠陥検出装置
CN111462113A (zh) * 2020-04-24 2020-07-28 上海精测半导体技术有限公司 无图形晶圆的复检方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4411356A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025023191A1 (ja) * 2023-07-21 2025-01-30 富士フイルム株式会社 制御装置、制御方法、及び制御プログラム

Also Published As

Publication number Publication date
EP4411356A1 (en) 2024-08-07
JPWO2023053728A1 (https=) 2023-04-06
JP7828357B2 (ja) 2026-03-11
EP4411356A4 (en) 2025-01-22
US20240221343A1 (en) 2024-07-04
CN117957439A (zh) 2024-04-30

Similar Documents

Publication Publication Date Title
JP6767490B2 (ja) 欠陥検査装置、欠陥検査方法、およびプログラム
JP6560220B2 (ja) 表面欠陥の検出のための検査システムを点検する方法及び装置
US10989672B2 (en) Defect inspection device, defect inspection method, and program
JP6556266B2 (ja) 欠陥検査装置、方法およびプログラム
JP6930926B2 (ja) 欠陥検査装置、方法およびプログラム
JP6422573B2 (ja) 画像処理装置及び画像処理方法及びプログラム
US20240221343A1 (en) Display processing device, display processing method, and display processing program
WO2020003917A1 (ja) 欠陥表示装置及び方法
US20210295491A1 (en) System and method for automated visual inspection
US20240331339A1 (en) Information processing apparatus, information processing method, and program
JPWO2023053728A5 (https=)
US20240362888A1 (en) Information processing apparatus, information processing method, and information processing program
JP2009264882A (ja) 外観検査装置
JP4981433B2 (ja) 検査装置、検査方法、検査プログラムおよび検査システム
WO2024043190A1 (ja) 検査装置、検査システムおよび検査方法
JP2563865B2 (ja) 画像認識装置及び方法
JP2005147665A (ja) ワイヤ検出プログラム及びワイヤ検査装置
JP2006275977A (ja) 超音波検査判断の支援装置
EP4465027A1 (en) Information processing device, information processing method, and program
JP2008134074A (ja) タイヤ検査方法
JP2007005089A (ja) 陰極線管の検査装置、及びこれを用いた陰極線管の製造方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22875603

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280061993.6

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2023550428

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022875603

Country of ref document: EP

Effective date: 20240429