WO2023175831A1 - Image confirmation device and image confirmation method - Google Patents

Image confirmation device and image confirmation method Download PDF

Info

Publication number
WO2023175831A1
WO2023175831A1 PCT/JP2022/012247 JP2022012247W WO2023175831A1 WO 2023175831 A1 WO2023175831 A1 WO 2023175831A1 JP 2022012247 W JP2022012247 W JP 2022012247W WO 2023175831 A1 WO2023175831 A1 WO 2023175831A1
Authority
WO
WIPO (PCT)
Prior art keywords
board
image
work
work image
product
Prior art date
Application number
PCT/JP2022/012247
Other languages
French (fr)
Japanese (ja)
Inventor
智也 藤本
雄哉 稲浦
貴紘 小林
Original Assignee
株式会社Fuji
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社Fuji filed Critical 株式会社Fuji
Priority to PCT/JP2022/012247 priority Critical patent/WO2023175831A1/en
Publication of WO2023175831A1 publication Critical patent/WO2023175831A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
    • G01N21/956—Inspecting patterns on the surface of objects
    • H—ELECTRICITY
    • H05—ELECTRIC TECHNIQUES NOT OTHERWISE PROVIDED FOR
    • H05K—PRINTED CIRCUITS; CASINGS OR CONSTRUCTIONAL DETAILS OF ELECTRIC APPARATUS; MANUFACTURE OF ASSEMBLAGES OF ELECTRICAL COMPONENTS
    • H05K3/00—Apparatus or processes for manufacturing printed circuits
    • H05K3/30—Assembling printed circuits with electric components, e.g. with resistor
    • H05K3/32—Assembling printed circuits with electric components, e.g. with resistor electrically connecting electric components or wires to printed circuits
    • H05K3/34—Assembling printed circuits with electric components, e.g. with resistor electrically connecting electric components or wires to printed circuits by soldering

Definitions

  • This specification discloses a technique related to an image confirmation device and an image confirmation method.
  • the quality determination device described in Patent Document 1 includes a distribution acquisition section and a quality determination section.
  • the distribution acquisition unit extracts a reference feature amount, which is a feature amount of the entire image, for each image of the reference image, and obtains a feature amount distribution, which is a distribution of a plurality of extracted reference feature amounts.
  • the pass/fail judgment unit extracts the target feature amount, which is the feature amount of the entire image, for each image of the target image, and acquires the target image based on the degree of deviation of the target feature amount from the feature area defined by the feature amount distribution. Judge the quality of the board work when doing so.
  • a teacher image of a board product used for machine learning is prepared before production, and the teacher image is used to inspect the board product during production. can do.
  • not all teacher images are appropriate images. If an inappropriate teacher image obtained when the board product produced by the board-to-board work machine is defective is included, the inspection accuracy of the board product may be reduced.
  • there are a large number of teacher images and it is difficult for an operator to check each teacher image one by one and discover inappropriate teacher images.
  • the present specification discloses an image confirmation device and an image confirmation method that are capable of determining whether or not a board product produced by a board-facing work machine is likely to be a defective product.
  • an image confirmation device that includes an acquisition unit and a determination unit.
  • the acquisition unit includes a pre-work image of the board before a target object is provided on the board by a board-to-board work machine that performs a predetermined board-to-board work, and a pre-work image of the board before the target object is provided by the board-to-board work machine.
  • the determination unit determines whether or not there is a possibility that the board product produced by the board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image acquired by the acquisition unit. do.
  • an image confirmation method that includes an acquisition step and a determination step.
  • the acquisition step includes a pre-work image of the board before a target object is provided on the board by a board-to-board work machine that performs a predetermined board-to-board work, and a pre-work image of the board before the target object is provided by the board-to-board work machine.
  • the determination step determines whether or not there is a possibility that the board product produced by the board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image acquired in the acquisition step. do.
  • the image confirmation device includes an acquisition unit and a determination unit. Thereby, the image confirmation device can determine whether or not there is a possibility that the board product produced by the board-to-board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image. What has been described above regarding the image confirmation device also applies to the image confirmation method.
  • FIG. 1 is a configuration diagram showing an example of a production line.
  • FIG. 2 is a plan view showing an example of a component mounting machine.
  • FIG. 3 is a schematic diagram showing an example of a teacher image and an after-work image. It is a block diagram showing an example of a control block of an image confirmation device. 3 is a flowchart illustrating an example of a control procedure by the image confirmation device. It is a schematic diagram which shows an example of an image before work. It is a schematic diagram which shows the example of a display of the image after a work by an image confirmation apparatus.
  • Embodiment 1-1 Configuration Example of Production Line WL0
  • FIG. 1 shows an example of a production line WL0 to which the image confirmation device 50 is applied.
  • the board-to-board work machine WM0 performs a predetermined board-to-board work on the board 90.
  • the type and number of board-facing work machines WM0 are not limited.
  • the production line WL0 of the embodiment includes a plurality of (five) board-to-board working machines WM0, including a printing machine WM1, a print inspection machine WM2, a component mounting machine WM3, a reflow oven WM4, and an appearance inspection machine WM5.
  • the substrates 90 are transported in the above order by the substrate transport device.
  • the printing machine WM1 prints solder 90h at the mounting position of the component 91 on the board 90.
  • the print inspection machine WM2 inspects the printing state of the solder 90h printed by the printing machine WM1.
  • the component mounting machine WM3 mounts a plurality of components 91 onto the board 90 on which the solder 90h has been printed by the printing machine WM1.
  • the number of component mounting machines WM3 may be one or more. When a plurality of component mounting machines WM3 are provided, the plurality of component mounting machines WM3 can share the task of mounting a plurality of components 91.
  • the reflow furnace WM4 heats the board 90 on which the component 91 is mounted by the component mounting machine WM3, melts the solder 90h, and performs soldering.
  • the appearance inspection machine WM5 inspects the mounting state of the component 91 mounted by the component mounting machine WM3. In this way, the production line WL0 can produce the board product 900 by sequentially transporting the boards 90 and performing production processing including inspection processing using a plurality of (five) board working machines WM0. .
  • the production line WL0 may be equipped with substrate-related working machines WM0 such as a function inspection machine, a buffer device, a substrate supply device, a substrate reversing device, a shield mounting device, an adhesive coating device, an ultraviolet irradiation device, etc., as necessary. You can also do it.
  • substrate-related working machines WM0 such as a function inspection machine, a buffer device, a substrate supply device, a substrate reversing device, a shield mounting device, an adhesive coating device, an ultraviolet irradiation device, etc., as necessary. You can also do it.
  • a plurality of (five) board-facing work machines WM0 and management device HC0 are communicably connected by a wired or wireless communication unit.
  • a local area network (LAN) is configured by a plurality of (five) board-oriented work machines WM0 and a management device HC0.
  • the plurality of (five) board-oriented working machines WM0 can communicate with each other via the communication section.
  • the plurality (five) of board-oriented work machines WM0 can communicate with the management device HC0 via the communication section.
  • the management device HC0 controls a plurality of (five) board-facing working machines WM0 that constitute the production line WL0, and monitors the operating status of the production line WL0.
  • the management device HC0 stores various control data for controlling a plurality of (five) board-oriented working machines WM0.
  • the management device HC0 transmits control data to each of the plurality of (five) board-facing work machines WM0. Further, each of the plurality of (five) board-oriented work machines WM0 transmits the operating status and production status to the management device HC0.
  • a data server 70 is provided in the management device HC0.
  • the data server 70 can store, for example, acquired data obtained by the board-related work machine WM0 regarding the board-related work.
  • image data of an image captured by the substrate-facing work machine WM0 is included in the acquired data.
  • a teacher image 40, a pre-work image 41, and a post-work image 42, which will be described later, are included in the acquired data. Records (log data) of operating conditions acquired by the board-oriented work machine WM0 are included in the acquired data.
  • the data server 70 can also store various production information regarding the production of the board product 900.
  • component data such as information regarding the shape of each type of component 91, information regarding the electrical characteristics of component 91, and information regarding how to handle component 91 is included in the production information.
  • inspection results by inspection machines such as the print inspection machine WM2 and the visual inspection machine WM5 are included in the production information.
  • the component mounting machine WM3 mounts a component 91 onto a board 90.
  • the component mounting machine WM3 of the embodiment includes a board transfer device 11, a component supply device 12, a component transfer device 13, a component camera 14, a board camera 15, and a control device 16.
  • the substrate conveyance device 11 is constituted by, for example, a belt conveyor, and conveys the substrate 90 in the conveyance direction (X-axis direction).
  • the substrate 90 is a circuit board on which an electronic circuit, an electric circuit, a magnetic circuit, etc. are formed.
  • the board transport device 11 carries the board 90 into the component mounting machine WM3 and positions the board 90 at a predetermined position inside the machine.
  • the board transport device 11 carries the board 90 out of the component mounting machine WM3 after the mounting process of the component 91 by the component mounting machine WM3 is completed.
  • the component supply device 12 supplies components 91 to be mounted on the board 90.
  • the component supply device 12 includes a feeder 12a provided along the conveyance direction (X-axis direction) of the substrate 90.
  • the feeder 12a pitch-feeds a carrier tape containing a plurality of parts 91, and supplies the parts 91 so that they can be collected at a supply position located on the tip side of the feeder 12a.
  • the component supply device 12 can also supply electronic components (for example, lead components) that are relatively large compared to chip components and the like while being arranged on a tray.
  • the component transfer device 13 includes a head drive device 13a and a moving table 13b.
  • the head drive device 13a is configured to be able to move the movable table 13b in the X-axis direction and the Y-axis direction (direction orthogonal to the X-axis direction in the horizontal plane) using a linear motion mechanism.
  • a mounting head 20 is removably (replaceably) provided on the moving table 13b using a clamp member.
  • the mounting head 20 uses at least one holding member 30 to pick up and hold the component 91 supplied by the component supply device 12, and mounts the component 91 onto the substrate 90 positioned by the substrate transfer device 11.
  • a suction nozzle, a chuck, etc. can be used as the holding member 30.
  • the component camera 14 is fixed to the base of the component mounting machine WM3 so that its optical axis faces upward in the Z-axis direction (vertical direction perpendicular to the X-axis direction and the Y-axis direction).
  • the component camera 14 can image the component 91 held by the holding member 30 from below.
  • the board camera 15 is provided on the movable table 13b of the component transfer device 13 so that its optical axis points downward in the Z-axis direction.
  • the board camera 15 can image the board 90 from above.
  • the component camera 14 and the board camera 15 can use known imaging devices, and perform imaging based on control signals sent from the control device 16. Image data of images captured by the component camera 14 and the board camera 15 are transmitted to the control device 16.
  • the control device 16 includes a known arithmetic unit and a storage device, and constitutes a control circuit. Information, image data, etc. output from various sensors provided in the component mounting machine WM3 are input to the control device 16. The control device 16 sends control signals to each device based on a control program and predetermined mounting conditions set in advance.
  • control device 16 causes the substrate camera 15 to image the substrate 90 positioned by the substrate transport device 11.
  • the control device 16 processes the image captured by the board camera 15 and recognizes the positioning state of the board 90.
  • the control device 16 causes the holding member 30 to collect and hold the component 91 supplied by the component supply device 12, and causes the component camera 14 to take an image of the component 91 held by the holding member 30.
  • the control device 16 processes the image captured by the component camera 14 to recognize the presence or absence of the component 91, the suitability of the component 91, the holding posture of the component 91, and the like.
  • the control device 16 moves the holding member 30 upward from a scheduled mounting position that is preset by a control program or the like. Further, the control device 16 corrects the scheduled mounting position based on the positioning state of the board 90, the holding posture of the component 91, etc., and sets the mounting position where the component 91 is actually mounted.
  • the scheduled mounting position and the mounting position include the rotation angle in addition to the position (X-axis coordinate and Y-axis coordinate).
  • the control device 16 corrects the target position (X-axis coordinate and Y-axis coordinate) and rotation angle of the holding member 30 according to the mounting position.
  • the control device 16 lowers the holding member 30 at the corrected rotation angle at the corrected target position, and mounts the component 91 on the board 90.
  • the control device 16 executes a mounting process for mounting a plurality of components 91 on the board 90 by repeating the above pick-and-place cycle.
  • FIG. 3 shows an example of the teacher image 40.
  • one component 91 target object 91t provided on the board 90
  • the component mounting machine WM3 is captured.
  • the teacher image 40 shown in the figure can be imaged from above the board 90 by an imaging device 80 such as a camera provided outside the board camera 15, the visual inspection machine WM5, and the board work machine WM0.
  • an imaging device 80 such as a camera provided outside the board camera 15, the visual inspection machine WM5, and the board work machine WM0.
  • an operator determines whether an image captured by the imaging device 80 is appropriate as the teacher image 40.
  • the operator may mistakenly register an inappropriate image as the teacher image 40 (for example, an image in which the appropriate part 91 is not properly attached to the predetermined area 90t of the board 90) as the teacher image 40. .
  • an image confirmation device 50 is provided.
  • the image confirmation device 50 determines whether there is a possibility that the board product 900 produced by the board work machine WM0 is a defective product.
  • the image confirmation device 50 includes an acquisition section 51 and a determination section 52.
  • the image confirmation device 50 can include an extraction section 53.
  • the image confirmation device 50 can also include a guide section 54.
  • the acquisition unit 51, the determination unit 52, the extraction unit 53, and the guide unit 54 can be provided in various arithmetic devices, control devices, and the like.
  • at least one of the acquisition unit 51, the determination unit 52, the extraction unit 53, and the guide unit 54 can be provided in the management device HC0.
  • At least one of the acquisition unit 51, the determination unit 52, the extraction unit 53, and the guide unit 54 can also be formed on the cloud.
  • the image confirmation device 50 of the embodiment includes an acquisition section 51, a determination section 52, an extraction section 53, and a guide section 54.
  • the image confirmation device 50 is provided in the management device HC0. Further, the image confirmation device 50 of the embodiment executes control according to the flowchart shown in FIG.
  • the acquisition unit 51 performs the process shown in step S11.
  • the determining unit 52 makes the determination shown in step S12.
  • the extraction unit 53 performs the process shown in step S13.
  • the guide unit 54 performs the process shown in step S14.
  • the acquisition unit 51 acquires both the before-work image 41 and the after-work image 42 (step S11 shown in FIG. 5).
  • the pre-work image 41 is an image of the board 90 before the object 91t is provided thereon by the board-to-board work machine WM0 that performs a predetermined board-to-board work on the board 90.
  • the post-work image 42 is an image of the board 90 on which the target object 91t is expected to be provided by the board-facing work machine WM0.
  • FIG. 6 shows an example of the pre-work image 41.
  • This figure shows an example of the state of the board 90 before the component 91, which is the target object 91t, is mounted by the component mounting machine WM3, which is the board-to-board working machine WM0.
  • the pre-work image 41 captures a predetermined region 90t on the board 90 where a component 91, which is a target object 91t, is to be provided.
  • a pair of land portions 90r are provided in the predetermined region 90t.
  • Solder 90h is printed on each of the pair of land portions 90r.
  • the pair of land portions 90r are made of, for example, copper foil, and are electrically connected to the electrode portions of the component 91 when the component 91 is mounted on the board 90.
  • FIG. 3 shows an example of the post-work image 42.
  • This figure shows an example of a state in which a component 91 is mounted in the predetermined area 90t shown in FIG. 6 by the component mounting machine WM3.
  • the component 91 is a chip component such as a chip resistor or a chip capacitor
  • the component 91 includes an electrode area AR11 and an electrode area AR12, which are electrode area areas, and a main body area AR13, which is a main body area.
  • the electrode region AR11 is electrically connected to one of the pair of land portions 90r.
  • the electrode region AR12 is electrically connected to the other land portion 90r of the pair of land portions 90r.
  • the after-work image 42 shown in FIG. 3 is an appropriate image in which the appropriate parts 91 are properly mounted on the predetermined area 90t of the board 90.
  • the post-work image 42 may include an inappropriate image in which the appropriate component 91 is not properly attached to the predetermined area 90t of the board 90.
  • an image taken of the board 90 without the component 91 attached to the predetermined area 90t, or an image taken of the board 90 with the component 91 attached with the component 91 protruding from the predetermined area 90t may be inappropriate. included in the image.
  • the before-work image 41 and the after-work image 42 can be generated by capturing an image from above the board 90 using an imaging device 80 such as a camera provided outside the board camera 15, the visual inspection machine WM5, and the board-facing working machine WM0. can.
  • the imaging device 80 can generate the before-work image 41 and the after-work image 42 under the same type of imaging conditions.
  • the imaging conditions include, for example, the type of light source, the direction of light irradiation, the exposure time, and the aperture value. Note that, since it is difficult to completely match the imaging conditions due to the influence of natural light, etc., the imaging conditions may be conditions that can be defined by the imaging device 80.
  • the imaging device 80 can also generate the before-work image 41 and the after-work image 42 under different imaging conditions.
  • the imaging device 80 generates the pre-work image 41 under imaging conditions suitable for imaging the solder 90h printed on the board 90, and is suitable for imaging the component 91 mounted on the board 90.
  • the post-work image 42 can also be generated under different imaging conditions.
  • the imaging device 80 generates the pre-work image 41 and the post-work image 42 during production preparation before the inspection of the board product 900 using the teacher image 40 is started.
  • the imaging device 80 can also generate the pre-work image 41 and the post-work image 42 by capturing an image of the board product 900 that is produced after the inspection of the board product 900 using the teacher image 40 is started.
  • the before-work image 41 and the after-work image 42 are stored in the data server 70 shown in FIG.
  • the acquisition unit 51 can acquire the before-work image 41 and the after-work image 42 from the data server 70 .
  • Judgment section 52 For example, in the predetermined area 90t, the after-work image 42 in which the board 90 on which the component 91 is not mounted is the same as the before-work image 41, and the similarity between the before-work image 41 and the after-work image 42 is becomes higher. Conversely, the after-work image 42 in which the appropriate parts 91 are properly attached to the predetermined area 90t of the board 90 is significantly different from the before-work image 41, and the similarity between the before-work image 41 and the after-work image 42 is low. Become.
  • the determination unit 52 determines the possibility that the board product 900 produced by the board work machine WM0 is a defective product based on the degree of similarity between the before-work image 41 and the after-work image 42 acquired by the acquisition unit 51.
  • the presence or absence is determined (step S12 shown in FIG. 5).
  • the determining unit 52 may take various forms as long as it can determine whether there is a possibility that the board product 900 is a defective product based on the degree of similarity between the before-work image 41 and the after-work image 42.
  • the electrode area AR11 and the electrode area AR12 of the component 91 shown in FIG. 3 are silver (metallic color).
  • the main body area AR13 on the front surface side of the component 91 (the surface that is visible when the component 91 is properly attached to the board 90) is black, and the main body area AR13 on the back surface (bottom surface) side of the component 91 is black. , is assumed to be white.
  • the imaging device 80 images the electrode area AR11 and the electrode area AR12 (silver) and the main body area AR13 (black) on the front side of the component 91.
  • the imaging device 80 If the holding member 30 mistakenly attracts the back side of the component 91 and the component 91 is attached to the substrate 90 with the front side and the back side of the component 91 reversed, the imaging device 80 Then, the electrode area AR12 (silver color) and the main body area AR13 (white color) on the back side of the component 91 are imaged.
  • the difference in brightness between the electrode area AR11 and the electrode area AR12 and the main body area AR13 of the component 91 is as follows. The size is smaller than when the component 91 is properly attached to the board 90.
  • the inappropriate after-work image 42 (for example, an image in which the appropriate part 91 is not properly attached to the predetermined area 90t of the board 90) is different from the inappropriate after-work image 42 (for example, an image in which the appropriate part 91 is not properly mounted on the board).
  • the feature amount of the entire image changes compared to the image in which the image is properly attached to the predetermined area 90t.
  • the brightness of a plurality of pixels constituting an image is included in the feature amount.
  • the predetermined area 90t corresponds to the target area on the board 90 where the component 91 is to be mounted.
  • the target area 91s corresponds to a mounting area on the board 90 where the component 91 is actually mounted.
  • the post-work image 42 in which the substrate 90 has an object 91t different from the object 91t that should be provided on the substrate 90 is inappropriate.
  • the object 91t to be provided on the board 90 corresponds to the component 91 to be mounted on the board 90.
  • the area of the target region 91s may increase or decrease compared to the case where the target object 91t to be provided is provided on the substrate 90.
  • the index (for example, circularity) indicating the shape of the target object 91t may increase or decrease compared to the case where the target object 91t to be provided is provided on the substrate 90.
  • the square component 91 has a higher degree of circularity than the rectangular component 91, which is one of the indicators for determining whether the object 91t should be provided on the substrate 90.
  • the determining unit 52 determines that the smaller the Euclidean distance between the feature amount of the entire image of the before-work image 41 and the feature amount of the entire image of the after-work image 42, the higher the degree of similarity between the before-work image 41 and the after-work image 42. It can be determined that Further, the feature amounts include the luminance of a plurality of pixels constituting the image, the center of gravity of the target region 91s in which the target object 91t is provided on the substrate 90, the area of the target region 91s, and an index indicating the shape of the target object 91t. At least one of these is preferred.
  • the determination unit 52 determines the degree of similarity between the before-work image 41 and the after-work image 42 using the above-mentioned feature amounts.
  • the brightness of a plurality of pixels constituting an image can be expressed by Mahalanobis distance.
  • the determining unit 52 normalizes the Mahalanobis distance representing the brightness of a plurality of pixels constituting the image before work 41 from 0 (minimum value) to 1 (maximum value), and The Mahalanobis distance representing the brightness of the pixel is normalized from 0 (minimum value) to 1 (maximum value).
  • the determination unit 52 normalizes the center of gravity of the target region 91s, the area of the target region 91s, and the index indicating the shape of the target object 91t.
  • the determination unit 52 calculates the difference between the normalized brightness value of the before-work image 41 and the normalized brightness value of the after-work image 42. Similarly, the determination unit 52 calculates the difference between the center of gravity of the target area 91s normalized for the pre-work image 41 and the center of gravity of the target area 91s normalized for the post-work image 42. The determining unit 52 calculates the difference between the area of the target area 91s normalized for the pre-work image 41 and the area of the target area 91s normalized for the post-work image 42. The determining unit 52 calculates the difference between an index indicating the shape of the target object 91t normalized in the pre-work image 41 and an index indicating the shape of the target object 91t normalized in the post-work image 42.
  • the determining unit 52 calculates the sum of squares of the above differences.
  • the determining unit 52 determines that the closer the sum of squares is to 0 (zero), the smaller the Euclidean distance between the feature amount of the entire image of the pre-work image 41 and the feature amount of the entire image of the post-work image 42; It can be determined that the similarity of the subsequent image 42 is high.
  • the determining unit 52 determines that as the sum of squares increases, the Euclidean distance between the feature amount of the entire image of the before-work image 41 and the feature amount of the entire image of the after-work image 42 increases, and the difference between the before-work image 41 and the after-work image 42 increases. It can be determined that the similarity of 42 is low.
  • the determining unit 52 can determine that the higher the similarity between the before-work image 41 and the after-work image 42, the more likely the board product 900 produced by the board-to-board working machine WM0 is a defective product. Conversely, the lower the degree of similarity between the before-work image 41 and the after-work image 42, the less likely the judgment unit 52 is to judge that the board product 900 produced by the board work machine WM0 is unlikely to be a defective product. can.
  • the determining unit 52 can set a threshold value for determining whether there is a possibility that the board product 900 is a defective product to a designated value designated by the operator or a predefined fixed value. For example, the operator can input the above specified value on the display device 60, which will be described later. For example, if the determining unit 52 determines that there is no possibility that the board product 900 is a defective product, but it is confirmed that the board product 900 is a defective product, the operator may A threshold value can be input as a specified value.
  • the operator may A threshold value can be input as a specified value.
  • the determination unit 52 can obtain the above-mentioned fixed value in advance through simulation, verification using an actual machine, or the like. In either case, if the sum of squares of the differences is smaller than the threshold, the determining unit 52 can determine that the degree of similarity between the before-work image 41 and the after-work image 42 is high, and the board product 900 is defective. It can be determined that the product may be of good quality. Further, when the sum of squares of the above-mentioned differences is greater than or equal to the threshold value, the determining unit 52 can determine that the degree of similarity between the before-work image 41 and the after-work image 42 is low, and it is possible that the board product 900 is a defective product. It can be determined that there is no gender.
  • Extraction section 53 and guide section 54 When the determining unit 52 determines that the board product 900 may be defective (Yes in step S12 shown in FIG. 5), the extracting unit 53 extracts the after-work image 42 of the board product 900. Extract (step S13). For example, the post-work image 42 extracted by the extraction unit 53 can be deleted from the data server 70. As a result, the post-work image 42 is no longer used as the teacher image 40.
  • the post-work images 42 extracted by the extraction unit 53 may include images that can be used as the teacher images 40. Therefore, the guide section 54 can guide the worker through the post-work image 42 extracted by the extraction section 53.
  • the guide section 54 only needs to be able to guide the worker through the post-work image 42 extracted by the extraction section 53, and can take various forms.
  • the guide unit 54 can guide the after-work image 42 by various methods such as displaying the after-work image 42 and providing audio guidance of information (for example, file name of image data) that can identify the after-work image 42 .
  • the guide unit 54 causes the display device 60 to display the after-work image 42 extracted by the extraction unit 53 (step S14 shown in FIG. 5).
  • the display device 60 only needs to be able to display the post-work image 42, and various known display devices can be used.
  • FIG. 7 shows an example of the display of the post-work image 42 by the image confirmation device 50. Specifically, the figure shows an example of a display screen of the display device 60.
  • the guide unit 54 causes the display device 60 to display the after-work images 42 in order of the similarity between the before-work image 41 and the after-work image 42. Thereby, the worker can check the post-work images 42 on the display device 60 in order from the post-work image 42 in which the board product 900 that is likely to be a defective product is captured.
  • the post-work image 42 of the No. 1 area AR21 shown in the figure is an image of the board 90 in which the component 91 is not attached to the predetermined area 90t.
  • the after-work image 42 of the first region AR21 has the smallest Euclidean distance described above, and the similarity between the before-work image 41 and the after-work image 42 is the highest. Therefore, it is most likely that the board product 900 is a defective product.
  • the after-work image 42 of the second area AR22 is an image of the board 90 on which the component 91 is mounted, with the component 91 protruding from the predetermined area 90t.
  • the after-work image 42 of the second area AR22 has a larger Euclidean distance as described above, and the similarity between the before-work image 41 and the after-work image 42 is low.
  • the board product 900 captured in the after-work image 42 of the first area AR21 the board product 900 is most likely to be a defective product.
  • the after-work image 42 of the third area AR23 is an image of the board 90 on which the component 91 is mounted, with the component 91 protruding from the predetermined area 90t.
  • the proportion of parts 91 protruding from the predetermined area 90t is smaller than in the after-work image 42 of the No. 2 area AR22.
  • the after-work image 42 of the third area AR23 has a larger Euclidean distance as described above, and the similarity between the before-work image 41 and the after-work image 42 is low. .
  • the board product 900 is most likely to be a defective product.
  • the guide unit 54 can change the display method of the after-work image 42 to be displayed on the display device 60 according to the classification of similarity between the before-work image 41 and the after-work image 42.
  • the guide unit 54 can arbitrarily set classifications for the degree of similarity between the before-work image 41 and the after-work image 42.
  • the guide section 54 can take various display methods. For example, the guide unit 54 changes the display color (including shading) of the background of the post-work image 42 in at least one of the first area AR21, the second area AR22, and the third area AR23. be able to.
  • the guide unit 54 causes the display device 60 to display the background of the first area AR21 in a predetermined color (for example, red).
  • the guide unit 54 causes the display device 60 to display the background of the second area AR22 in another predetermined color (for example, pink).
  • the guide unit 54 causes the display device 60 to display the background of the third area AR23 in another predetermined color (for example, white).
  • the guide unit 54 displays the background of the after-work image 42 on the display device 60 in a display color that is easier to alert the worker in the classification where the degree of similarity between the before-work image 41 and the after-work image 42 is higher. It can be displayed.
  • the guide section 54 can also cause blinking display in at least one of the first area AR21, the second area AR22, and the third area AR23.
  • the guide unit 54 can shorten the blinking cycle for the classification in which the pre-work image 41 and the post-work image 42 have a higher degree of similarity.
  • the guide unit 54 can lengthen the blinking cycle for a category in which the degree of similarity between the before-work image 41 and the after-work image 42 is lower.
  • the guide unit 54 can also change the display method when displaying the background of the post-work image 42 on the display device 60 by combining display colors and blinking displays.
  • the guide unit 54 can have the operator check the quality of the board product 900 based on the post-work image 42 displayed on the display device 60. For example, on the display screen shown in FIG. 7, the displayed image in area AR31 indicates that mounting of component 91 may have failed, and an instruction to input the determination result after visual confirmation is displayed.
  • the display screen shown in FIG. 7 is composed of a touch panel.
  • the operator touches the operation unit BA1 or BA2 in the area where the quality of the board product 900 is to be confirmed. For example, when the operator confirms that the board product 900 captured in the post-work image 42 of the first area AR21 is a good product (Pass), the operator touches the operation unit BA1 of the first area AR21. .
  • Pass good product
  • the operator Conversely, if the operator confirms that the board product 900 captured in the post-work image 42 of the No. 1 area AR21 is a defective product (Fail), the operator operates the operation unit BA2 of the No. 1 area AR21. touch. In the figure, a defective product (Fail) has been selected by the operator, and it has been confirmed that the board product 900 is a defective product (Fail). What has been described above regarding the first area AR21 is also true for the other areas. However, other areas are in a state before being checked by the operator.
  • the worker touches and selects at least one of the first area AR21, the second area AR22, and the third area AR23, and the after-work image 42 of the selected area is captured.
  • the board product 900 is a good product (Pass)
  • the worker touches and selects at least one of the first area AR21, the second area AR22, and the third area AR23, and the after-work image 42 of the selected area is captured.
  • the operation section BB2 can also be touched.
  • the operator can also select all of the first area AR21, the second area AR22, and the third area AR23 by touching the operation part BB3.
  • the operator can also deselect the area by touching the operation unit BB3 again.
  • the worker can display the next display screen (the after-work image 42 in which the degree of similarity between the before-work image 41 and the after-work image 42 is lower than the currently displayed after-work image 42) by touching the operation unit BC1. It can be displayed.
  • the worker displays the previous display screen (the after-work image 42 in which the similarity between the before-work image 41 and the after-work image 42 is higher than the currently displayed after-work image 42) by touching the operation unit BC2. be able to.
  • the guide unit 54 can display various information such as the degree of similarity between the before-work image 41 and the after-work image 42 and image information regarding the after-work image 42 on the display device 60 together with the after-work image 42.
  • the image information regarding the post-work image 42 is not limited.
  • the image information regarding the after-work image 42 includes information regarding the date and time when the after-work image 42 was acquired, information regarding the imaging device 80, and imaging conditions (for example, the exposure time and aperture value when the imaging device 80 captured the after-work image 42). , type of light source, and direction of light irradiation).
  • the determining unit 52 can cause at least one of the two post-work images 42 shown below to be used as the teacher image 40 for machine learning.
  • One of the two after-work images 42 is the after-work image 42 of the board product 900 when it is determined that there is no possibility that the board product 900 is a defective product.
  • the other one of the two after-work images 42 is an image in which it is determined that the board product 900 may be a defective product, but the operator confirms that the board product 900 is a good product. This is an image 42 of the board product 900 after work.
  • the determination unit 52 causes the two post-work images 42 described above to be used as the teacher images 40 for machine learning.
  • the teacher image 40 can be used for various known machine learning methods.
  • the teacher image 40 can be used for various types of machine learning such as support vector machine and regression analysis.
  • the substrate-to-board working machine WM0 is a component mounting machine WM3 that mounts a component 91, which is a target object 91t, onto a board 90.
  • the component mounting machine WM3 can use the teacher image 40 to perform a component presence/absence test to check whether the component 91 is mounted on the predetermined area 90t of the board 90 by the component mounting machine WM3.
  • the component mounting machine WM3 also uses the teacher image 40 to determine the center of gravity of a predetermined region 90t on the board 90 where the component 91 is to be mounted and the center of gravity of a target region 91s on the board 90 where the component 91 is mounted. It is also possible to perform a positional shift inspection of the component 91 to be inspected for deviation.
  • the determining unit 52 determines that there is a possibility that the board product 900 is a defective product, and the after-work image of the board product 900 when the operator confirms that the board product 900 is a defective product. 42 is not used as the teacher image 40 for machine learning. Specifically, the determination unit 52 can delete from the data server 70 the post-work images 42 that are not allowed to be used as the teacher images 40. The deletion of the post-work image 42 is reflected in the board-facing work machine WM0 that uses the post-work image 42.
  • an after-work image 42 is shown for one component 91 among the plurality of components 91 mounted on the board 90 by the component mounting machine WM3.
  • the guide unit 54 can similarly guide the post-work images 42 for other parts 91 as well.
  • the determining unit 52 determines that there is no possibility that the board product 900 is a defective product (No in step S12 shown in FIG. 5)
  • the process shown in step S13 and step S14 is not executed, and the image is The control by the confirmation device 50 ends once.
  • the board-to-board working machine WM0 is not limited to the component mounting machine WM3.
  • the substrate work machine WM0 may be a printing machine WM1 that prints the solder 90h on the substrate 90.
  • the solder 90h printed on the substrate 90 is included in the target object 91t.
  • the pre-work image 41 is an image of the board 90 before the solder 90h is printed.
  • the post-work image 42 is an image of the board 90 on which the solder 90h is expected to be printed.
  • the predetermined area 90t on the substrate 90 where the target object 91t is to be provided corresponds to the target area on the substrate 90 where the solder 90h is to be printed.
  • the target area 91s on the board 90 where the target object 91t is provided corresponds to the printing area on the board 90 where the solder 90h is actually printed.
  • the board-facing work machine WM0 can perform various inspections using machine learning.
  • the inspection of the board product 900 may be a solder presence inspection.
  • the inspection of the board product 900 involves inspecting the deviation between the center of gravity of a predetermined area 90t on the board 90 where the solder 90h is to be printed and the center of gravity of the target area 91s on the board 90 where the solder 90h is printed. It may be a 90 hour positional deviation inspection.
  • the image confirmation method includes an acquisition step and a determination step.
  • the acquisition process corresponds to control performed by the acquisition unit 51.
  • the determination process corresponds to the control performed by the determination unit 52.
  • the image confirmation method can include an extraction step.
  • the extraction process corresponds to the control performed by the extraction unit 53.
  • the image confirmation method can also include a guiding step.
  • the guiding process corresponds to the control performed by the guiding section 54.
  • the image confirmation device 50 includes an acquisition section 51 and a determination section 52. Thereby, the image confirmation device 50 determines whether or not there is a possibility that the board product 900 produced by the board work machine WM0 is a defective product based on the degree of similarity between the before-work image 41 and the after-work image 42. I can do it. What has been described above regarding the image confirmation device 50 also applies to the image confirmation method.

Abstract

Provided is an image confirmation device comprising an acquiring unit and a determination unit. The acquiring unit acquires both of a pre-work image, in which a substrate is imaged before an object is provided by a substrate-working machine that performs prescribed substrate work on the substrate, and a post-work image, in which a substrate on which an object is expected to be provided by the substrate-working machine is imaged. The determination unit determines whether the substrate product produced by the substrate-working machine is defective on the basis of the degree of similarity between the pre-work image and the post-work image acquired by the acquiring unit.

Description

画像確認装置および画像確認方法Image confirmation device and image confirmation method
　本明細書は、画像確認装置および画像確認方法に関する技術を開示する。 This specification discloses a technique related to an image confirmation device and an image confirmation method.
　特許文献１に記載の良否判定装置は、分布取得部と、良否判断部とを備えている。分布取得部は、基準画像の各画像について画像全体の特徴量である基準特徴量を抽出して、抽出された複数の基準特徴量の分布である特徴量分布を取得する。良否判断部は、対象画像の各画像について画像全体の特徴量である対象特徴量を抽出して、特徴量分布によって規定される特徴領域に対する対象特徴量の外れ度合いに基づいて、対象画像を取得したときの対基板作業の良否を判断する。 The quality determination device described in Patent Document 1 includes a distribution acquisition section and a quality determination section. The distribution acquisition unit extracts a reference feature amount, which is a feature amount of the entire image, for each image of the reference image, and obtains a feature amount distribution, which is a distribution of a plurality of extracted reference feature amounts. The pass/fail judgment unit extracts the target feature amount, which is the feature amount of the entire image, for each image of the target image, and acquires the target image based on the degree of deviation of the target feature amount from the feature area defined by the feature amount distribution. Judge the quality of the board work when doing so.
国際公開第２０２０／１８３７３４号International Publication No. 2020/183734
　特許文献１に記載の良否判定装置のように、対基板作業機では、生産前に機械学習に使用される基板製品の教師画像を用意しておき、生産時に教師画像を用いて基板製品を検査することができる。しかしながら、全ての教師画像が適切な画像であるとは限らない。対基板作業機によって生産された基板製品が不良品である場合に取得された不適切な教師画像が含まれていると、基板製品の検査精度が低下する可能性がある。また、教師画像は、大量に存在し、作業者が一枚ずつ教師画像を確認して、不適切な教師画像を発見することは困難である。 As with the quality determination device described in Patent Document 1, in a board work machine, a teacher image of a board product used for machine learning is prepared before production, and the teacher image is used to inspect the board product during production. can do. However, not all teacher images are appropriate images. If an inappropriate teacher image obtained when the board product produced by the board-to-board work machine is defective is included, the inspection accuracy of the board product may be reduced. Furthermore, there are a large number of teacher images, and it is difficult for an operator to check each teacher image one by one and discover inappropriate teacher images.
　このような事情に鑑みて、本明細書は、対基板作業機によって生産された基板製品が不良品である可能性の有無を判断することが可能な画像確認装置および画像確認方法を開示する。 In view of these circumstances, the present specification discloses an image confirmation device and an image confirmation method that are capable of determining whether or not a board product produced by a board-facing work machine is likely to be a defective product.
　本明細書は、取得部と、判断部とを備える画像確認装置を開示する。前記取得部は、基板に所定の対基板作業を行う対基板作業機によって対象物が設けられる前の前記基板が撮像されている作業前画像、および、前記対基板作業機によって前記対象物が設けられたことが見込まれる前記基板が撮像されている作業後画像の両方を取得する。前記判断部は、前記取得部によって取得された前記作業前画像および前記作業後画像の類似度に基づいて、前記対基板作業機によって生産された基板製品が不良品である可能性の有無を判断する。 This specification discloses an image confirmation device that includes an acquisition unit and a determination unit. The acquisition unit includes a pre-work image of the board before a target object is provided on the board by a board-to-board work machine that performs a predetermined board-to-board work, and a pre-work image of the board before the target object is provided by the board-to-board work machine. Obtain both post-work images of the board that is expected to have been removed. The determination unit determines whether or not there is a possibility that the board product produced by the board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image acquired by the acquisition unit. do.
　また、本明細書は、取得工程と、判断工程とを備える画像確認方法を開示する。前記取得工程は、基板に所定の対基板作業を行う対基板作業機によって対象物が設けられる前の前記基板が撮像されている作業前画像、および、前記対基板作業機によって前記対象物が設けられたことが見込まれる前記基板が撮像されている作業後画像の両方を取得する。前記判断工程は、前記取得工程によって取得された前記作業前画像および前記作業後画像の類似度に基づいて、前記対基板作業機によって生産された基板製品が不良品である可能性の有無を判断する。 Additionally, this specification discloses an image confirmation method that includes an acquisition step and a determination step. The acquisition step includes a pre-work image of the board before a target object is provided on the board by a board-to-board work machine that performs a predetermined board-to-board work, and a pre-work image of the board before the target object is provided by the board-to-board work machine. Obtain both post-work images of the board that is expected to have been removed. The determination step determines whether or not there is a possibility that the board product produced by the board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image acquired in the acquisition step. do.
　上記の画像確認装置によれば、取得部および判断部を備えている。これにより、画像確認装置は、作業前画像および作業後画像の類似度に基づいて、対基板作業機によって生産された基板製品が不良品である可能性の有無を判断することができる。画像確認装置について上述されていることは、画像確認方法についても同様に言える。 According to the above image confirmation device, the image confirmation device includes an acquisition unit and a determination unit. Thereby, the image confirmation device can determine whether or not there is a possibility that the board product produced by the board-to-board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image. What has been described above regarding the image confirmation device also applies to the image confirmation method.
生産ラインの一例を示す構成図である。FIG. 1 is a configuration diagram showing an example of a production line. 部品装着機の一例を示す平面図である。FIG. 2 is a plan view showing an example of a component mounting machine. 教師画像および作業後画像の一例を示す模式図である。FIG. 3 is a schematic diagram showing an example of a teacher image and an after-work image. 画像確認装置の制御ブロックの一例を示すブロック図である。It is a block diagram showing an example of a control block of an image confirmation device. 画像確認装置による制御手順の一例を示すフローチャートである。3 is a flowchart illustrating an example of a control procedure by the image confirmation device. 作業前画像の一例を示す模式図である。It is a schematic diagram which shows an example of an image before work. 画像確認装置による作業後画像の表示例を示す模式図である。It is a schematic diagram which shows the example of a display of the image after a work by an image confirmation apparatus.
　１．実施形態
　１－１．生産ラインＷＬ０の構成例
　図１は、画像確認装置５０が適用される生産ラインＷＬ０の一例を示している。生産ラインＷＬ０では、対基板作業機ＷＭ０が基板９０に所定の対基板作業を行う。対基板作業機ＷＭ０の種類および数は、限定されない。図１に示すように、実施形態の生産ラインＷＬ０は、印刷機ＷＭ１、印刷検査機ＷＭ２、部品装着機ＷＭ３、リフロー炉ＷＭ４および外観検査機ＷＭ５の複数（５つ）の対基板作業機ＷＭ０を備えており、基板９０は、基板搬送装置によって、上記の順に搬送される。 1. Embodiment 1-1. Configuration Example of Production Line WL0 FIG. 1 shows an example of a production line WL0 to which the image confirmation device 50 is applied. In the production line WL0, the board-to-board work machine WM0 performs a predetermined board-to-board work on the board 90. The type and number of board-facing work machines WM0 are not limited. As shown in FIG. 1, the production line WL0 of the embodiment includes a plurality of (five) board-to-board working machines WM0, including a printing machine WM1, a print inspection machine WM2, a component mounting machine WM3, a reflow oven WM4, and an appearance inspection machine WM5. The substrates 90 are transported in the above order by the substrate transport device.
　印刷機ＷＭ１は、基板９０の部品９１の装着位置に、はんだ９０ｈを印刷する。印刷検査機ＷＭ２は、印刷機ＷＭ１によって印刷されたはんだ９０ｈの印刷状態を検査する。部品装着機ＷＭ３は、印刷機ＷＭ１によってはんだ９０ｈが印刷された基板９０に複数の部品９１を装着する。部品装着機ＷＭ３は、一つであっても良く、複数であっても良い。部品装着機ＷＭ３が複数設けられる場合は、複数の部品装着機ＷＭ３が分担して、複数の部品９１を装着することができる。 The printing machine WM1 prints solder 90h at the mounting position of the component 91 on the board 90. The print inspection machine WM2 inspects the printing state of the solder 90h printed by the printing machine WM1. The component mounting machine WM3 mounts a plurality of components 91 onto the board 90 on which the solder 90h has been printed by the printing machine WM1. The number of component mounting machines WM3 may be one or more. When a plurality of component mounting machines WM3 are provided, the plurality of component mounting machines WM3 can share the task of mounting a plurality of components 91.
　リフロー炉ＷＭ４は、部品装着機ＷＭ３によって部品９１が装着された基板９０を加熱し、はんだ９０ｈを溶融させて、はんだ付けを行う。外観検査機ＷＭ５は、部品装着機ＷＭ３によって装着された部品９１の装着状態などを検査する。このように、生産ラインＷＬ０は、複数（５つ）の対基板作業機ＷＭ０を用いて、基板９０を順に搬送し、検査処理を含む生産処理を実行して基板製品９００を生産することができる。なお、生産ラインＷＬ０は、例えば、機能検査機、バッファ装置、基板供給装置、基板反転装置、シールド装着装置、接着剤塗布装置、紫外線照射装置などの対基板作業機ＷＭ０を必要に応じて備えることもできる。 The reflow furnace WM4 heats the board 90 on which the component 91 is mounted by the component mounting machine WM3, melts the solder 90h, and performs soldering. The appearance inspection machine WM5 inspects the mounting state of the component 91 mounted by the component mounting machine WM3. In this way, the production line WL0 can produce the board product 900 by sequentially transporting the boards 90 and performing production processing including inspection processing using a plurality of (five) board working machines WM0. . Note that the production line WL0 may be equipped with substrate-related working machines WM0 such as a function inspection machine, a buffer device, a substrate supply device, a substrate reversing device, a shield mounting device, an adhesive coating device, an ultraviolet irradiation device, etc., as necessary. You can also do it.
　複数（５つ）の対基板作業機ＷＭ０および管理装置ＨＣ０は、有線または無線の通信部によって通信可能に接続されている。実施形態では、複数（５つ）の対基板作業機ＷＭ０および管理装置ＨＣ０によって、構内情報通信網（ＬＡＮ：Ｌｏｃａｌ　Ａｒｅａ　Ｎｅｔｗｏｒｋ）が構成されている。これにより、複数（５つ）の対基板作業機ＷＭ０は、通信部を介して、互いに通信することができる。また、複数（５つ）の対基板作業機ＷＭ０は、通信部を介して、管理装置ＨＣ０と通信することができる。 A plurality of (five) board-facing work machines WM0 and management device HC0 are communicably connected by a wired or wireless communication unit. In the embodiment, a local area network (LAN) is configured by a plurality of (five) board-oriented work machines WM0 and a management device HC0. Thereby, the plurality of (five) board-oriented working machines WM0 can communicate with each other via the communication section. Further, the plurality (five) of board-oriented work machines WM0 can communicate with the management device HC0 via the communication section.
　管理装置ＨＣ０は、生産ラインＷＬ０を構成する複数（５つ）の対基板作業機ＷＭ０の制御を行い、生産ラインＷＬ０の動作状況を監視する。管理装置ＨＣ０には、複数（５つ）の対基板作業機ＷＭ０を制御する種々の制御データが記憶されている。管理装置ＨＣ０は、複数（５つ）の対基板作業機ＷＭ０の各々に制御データを送信する。また、複数（５つ）の対基板作業機ＷＭ０の各々は、管理装置ＨＣ０に動作状況および生産状況を送信する。 The management device HC0 controls a plurality of (five) board-facing working machines WM0 that constitute the production line WL0, and monitors the operating status of the production line WL0. The management device HC0 stores various control data for controlling a plurality of (five) board-oriented working machines WM0. The management device HC0 transmits control data to each of the plurality of (five) board-facing work machines WM0. Further, each of the plurality of (five) board-oriented work machines WM0 transmits the operating status and production status to the management device HC0.
　管理装置ＨＣ０には、データサーバ７０が設けられている。データサーバ７０は、例えば、対基板作業機ＷＭ０が対基板作業に関して取得した取得データを保存することができる。例えば、対基板作業機ＷＭ０によって撮像された画像の画像データは、取得データに含まれる。後述されている教師画像４０、作業前画像４１および作業後画像４２は、取得データに含まれる。対基板作業機ＷＭ０によって取得された稼働状況の記録（ログデータ）などは、取得データに含まれる。 A data server 70 is provided in the management device HC0. The data server 70 can store, for example, acquired data obtained by the board-related work machine WM0 regarding the board-related work. For example, image data of an image captured by the substrate-facing work machine WM0 is included in the acquired data. A teacher image 40, a pre-work image 41, and a post-work image 42, which will be described later, are included in the acquired data. Records (log data) of operating conditions acquired by the board-oriented work machine WM0 are included in the acquired data.
　また、データサーバ７０は、基板製品９００の生産に関する種々の生産情報を保存することもできる。例えば、部品９１の種類ごとの形状に関する情報、部品９１の電気的特性に関する情報、部品９１の取り扱い方法に関する情報などの部品データは、生産情報に含まれる。また、印刷検査機ＷＭ２、外観検査機ＷＭ５などの検査機による検査結果は、生産情報に含まれる。 Additionally, the data server 70 can also store various production information regarding the production of the board product 900. For example, component data such as information regarding the shape of each type of component 91, information regarding the electrical characteristics of component 91, and information regarding how to handle component 91 is included in the production information. In addition, inspection results by inspection machines such as the print inspection machine WM2 and the visual inspection machine WM5 are included in the production information.
　１－２．部品装着機ＷＭ３の構成例
　部品装着機ＷＭ３は、基板９０に部品９１を装着する。図２に示すように、実施形態の部品装着機ＷＭ３は、基板搬送装置１１、部品供給装置１２、部品移載装置１３、部品カメラ１４、基板カメラ１５および制御装置１６を備えている。 1-2. Configuration Example of Component Mounting Machine WM3 The component mounting machine WM3 mounts a component 91 onto a board 90. As shown in FIG. 2, the component mounting machine WM3 of the embodiment includes a board transfer device 11, a component supply device 12, a component transfer device 13, a component camera 14, a board camera 15, and a control device 16.
　基板搬送装置１１は、例えば、ベルトコンベアなどによって構成され、基板９０を搬送方向（Ｘ軸方向）に搬送する。基板９０は、回路基板であり、電子回路、電気回路、磁気回路などが形成される。基板搬送装置１１は、部品装着機ＷＭ３の機内に基板９０を搬入し、機内の所定位置に基板９０を位置決めする。基板搬送装置１１は、部品装着機ＷＭ３による部品９１の装着処理が終了した後に、基板９０を部品装着機ＷＭ３の機外に搬出する。 The substrate conveyance device 11 is constituted by, for example, a belt conveyor, and conveys the substrate 90 in the conveyance direction (X-axis direction). The substrate 90 is a circuit board on which an electronic circuit, an electric circuit, a magnetic circuit, etc. are formed. The board transport device 11 carries the board 90 into the component mounting machine WM3 and positions the board 90 at a predetermined position inside the machine. The board transport device 11 carries the board 90 out of the component mounting machine WM3 after the mounting process of the component 91 by the component mounting machine WM3 is completed.
　部品供給装置１２は、基板９０に装着される部品９１を供給する。部品供給装置１２は、基板９０の搬送方向（Ｘ軸方向）に沿って設けられるフィーダ１２ａを備えている。フィーダ１２ａは、複数の部品９１が収納されているキャリアテープをピッチ送りさせて、フィーダ１２ａの先端側に位置する供給位置において部品９１を採取可能に供給する。また、部品供給装置１２は、チップ部品などと比べて比較的大型の電子部品（例えば、リード部品など）を、トレイ上に配置した状態で供給することもできる。 The component supply device 12 supplies components 91 to be mounted on the board 90. The component supply device 12 includes a feeder 12a provided along the conveyance direction (X-axis direction) of the substrate 90. The feeder 12a pitch-feeds a carrier tape containing a plurality of parts 91, and supplies the parts 91 so that they can be collected at a supply position located on the tip side of the feeder 12a. Further, the component supply device 12 can also supply electronic components (for example, lead components) that are relatively large compared to chip components and the like while being arranged on a tray.
　部品移載装置１３は、ヘッド駆動装置１３ａおよび移動台１３ｂを備えている。ヘッド駆動装置１３ａは、直動機構によって移動台１３ｂを、Ｘ軸方向およびＹ軸方向（水平面においてＸ軸方向と直交する方向）に移動可能に構成されている。移動台１３ｂには、クランプ部材によって装着ヘッド２０が着脱可能（交換可能）に設けられている。装着ヘッド２０は、少なくとも一つの保持部材３０を用いて、部品供給装置１２によって供給された部品９１を採取し保持して、基板搬送装置１１によって位置決めされた基板９０に部品９１を装着する。保持部材３０は、例えば、吸着ノズル、チャックなどを用いることができる。 The component transfer device 13 includes a head drive device 13a and a moving table 13b. The head drive device 13a is configured to be able to move the movable table 13b in the X-axis direction and the Y-axis direction (direction orthogonal to the X-axis direction in the horizontal plane) using a linear motion mechanism. A mounting head 20 is removably (replaceably) provided on the moving table 13b using a clamp member. The mounting head 20 uses at least one holding member 30 to pick up and hold the component 91 supplied by the component supply device 12, and mounts the component 91 onto the substrate 90 positioned by the substrate transfer device 11. For example, a suction nozzle, a chuck, etc. can be used as the holding member 30.
　部品カメラ１４は、光軸がＺ軸方向（Ｘ軸方向およびＹ軸方向と直交する鉛直方向）の上向きになるように、部品装着機ＷＭ３の基台に固定されている。部品カメラ１４は、保持部材３０に保持されている部品９１を下方から撮像することができる。基板カメラ１５は、光軸がＺ軸方向の下向きになるように、部品移載装置１３の移動台１３ｂに設けられている。基板カメラ１５は、基板９０を上方から撮像することができる。部品カメラ１４および基板カメラ１５は、公知の撮像装置を用いることができ、制御装置１６から送出される制御信号に基づいて撮像を行う。部品カメラ１４および基板カメラ１５によって撮像された画像の画像データは、制御装置１６に送信される。 The component camera 14 is fixed to the base of the component mounting machine WM3 so that its optical axis faces upward in the Z-axis direction (vertical direction perpendicular to the X-axis direction and the Y-axis direction). The component camera 14 can image the component 91 held by the holding member 30 from below. The board camera 15 is provided on the movable table 13b of the component transfer device 13 so that its optical axis points downward in the Z-axis direction. The board camera 15 can image the board 90 from above. The component camera 14 and the board camera 15 can use known imaging devices, and perform imaging based on control signals sent from the control device 16. Image data of images captured by the component camera 14 and the board camera 15 are transmitted to the control device 16.
　制御装置１６は、公知の演算装置および記憶装置を備えており、制御回路が構成されている。制御装置１６には、部品装着機ＷＭ３に設けられる各種センサから出力される情報、画像データなどが入力される。制御装置１６は、制御プログラムおよび予め設定されている所定の装着条件などに基づいて、各装置に対して制御信号を送出する。 The control device 16 includes a known arithmetic unit and a storage device, and constitutes a control circuit. Information, image data, etc. output from various sensors provided in the component mounting machine WM3 are input to the control device 16. The control device 16 sends control signals to each device based on a control program and predetermined mounting conditions set in advance.
　例えば、制御装置１６は、基板搬送装置１１によって位置決めされた基板９０を基板カメラ１５に撮像させる。制御装置１６は、基板カメラ１５によって撮像された画像を画像処理して、基板９０の位置決め状態を認識する。また、制御装置１６は、部品供給装置１２によって供給された部品９１を保持部材３０に採取させ保持させて、保持部材３０に保持されている部品９１を部品カメラ１４に撮像させる。制御装置１６は、部品カメラ１４によって撮像された画像を画像処理して、部品９１の有無、部品９１の適否、部品９１の保持姿勢などを認識する。 For example, the control device 16 causes the substrate camera 15 to image the substrate 90 positioned by the substrate transport device 11. The control device 16 processes the image captured by the board camera 15 and recognizes the positioning state of the board 90. Further, the control device 16 causes the holding member 30 to collect and hold the component 91 supplied by the component supply device 12, and causes the component camera 14 to take an image of the component 91 held by the holding member 30. The control device 16 processes the image captured by the component camera 14 to recognize the presence or absence of the component 91, the suitability of the component 91, the holding posture of the component 91, and the like.
　制御装置１６は、制御プログラムなどによって予め設定される装着予定位置の上方に向かって、保持部材３０を移動させる。また、制御装置１６は、基板９０の位置決め状態、部品９１の保持姿勢などに基づいて、装着予定位置を補正して、実際に部品９１を装着する装着位置を設定する。装着予定位置および装着位置は、位置（Ｘ軸座標およびＹ軸座標）の他に回転角度を含む。 The control device 16 moves the holding member 30 upward from a scheduled mounting position that is preset by a control program or the like. Further, the control device 16 corrects the scheduled mounting position based on the positioning state of the board 90, the holding posture of the component 91, etc., and sets the mounting position where the component 91 is actually mounted. The scheduled mounting position and the mounting position include the rotation angle in addition to the position (X-axis coordinate and Y-axis coordinate).
　制御装置１６は、装着位置に合わせて、保持部材３０の目標位置（Ｘ軸座標およびＹ軸座標）および回転角度を補正する。制御装置１６は、補正された目標位置において補正された回転角度で保持部材３０を下降させて、基板９０に部品９１を装着する。制御装置１６は、上記のピックアンドプレースサイクルを繰り返すことによって、基板９０に複数の部品９１を装着する装着処理を実行する。 The control device 16 corrects the target position (X-axis coordinate and Y-axis coordinate) and rotation angle of the holding member 30 according to the mounting position. The control device 16 lowers the holding member 30 at the corrected rotation angle at the corrected target position, and mounts the component 91 on the board 90. The control device 16 executes a mounting process for mounting a plurality of components 91 on the board 90 by repeating the above pick-and-place cycle.
　１－３．画像確認装置５０の構成例
　対基板作業機ＷＭ０では、生産前に機械学習に使用される基板製品９００の教師画像４０を用意しておき、生産時に教師画像４０を用いて基板製品９００を検査することができる。しかしながら、全ての教師画像４０が適切な画像であるとは限らない。図３は、教師画像４０の一例を示している。同図に示す教師画像４０には、部品装着機ＷＭ３によって基板９０に装着された複数の部品９１のうちの一つの部品９１（基板９０に設けられた対象物９１ｔ）が撮像されている。 1-3. Configuration example of the image confirmation device 50 In the board work machine WM0, a teacher image 40 of the board product 900 used for machine learning is prepared before production, and the teacher image 40 is used to inspect the board product 900 during production. be able to. However, not all teacher images 40 are appropriate images. FIG. 3 shows an example of the teacher image 40. In the teacher image 40 shown in the figure, one component 91 (target object 91t provided on the board 90) out of the plurality of components 91 mounted on the board 90 by the component mounting machine WM3 is captured.
　同図に示す教師画像４０は、基板カメラ１５、外観検査機ＷＭ５、対基板作業機ＷＭ０の外部に設けられるカメラなどの撮像装置８０によって、基板９０の上方から撮像することができる。例えば、撮像装置８０によって撮像された画像が教師画像４０として適切であるか否かを作業者が判断する場合を想定する。この場合、教師画像４０として不適切な画像（例えば、適切な部品９１が基板９０の所定領域９０ｔに適正に装着されていない画像）を作業者が誤って教師画像４０に登録する可能性がある。 The teacher image 40 shown in the figure can be imaged from above the board 90 by an imaging device 80 such as a camera provided outside the board camera 15, the visual inspection machine WM5, and the board work machine WM0. For example, assume that an operator determines whether an image captured by the imaging device 80 is appropriate as the teacher image 40. In this case, there is a possibility that the operator may mistakenly register an inappropriate image as the teacher image 40 (for example, an image in which the appropriate part 91 is not properly attached to the predetermined area 90t of the board 90) as the teacher image 40. .
　このように、対基板作業機ＷＭ０によって生産された基板製品９００が不良品である場合に取得された不適切な教師画像４０が含まれていると、基板製品９００の検査精度が低下する可能性がある。また、教師画像４０は、大量に存在し、作業者が一枚ずつ教師画像４０を確認して、不適切な教師画像４０を発見することは困難である。そこで、実施形態では、画像確認装置５０が設けられている。画像確認装置５０は、対基板作業機ＷＭ０によって生産された基板製品９００が不良品である可能性の有無を判断する。 In this way, if the inappropriate teacher image 40 acquired when the board product 900 produced by the board work machine WM0 is defective is included, the inspection accuracy of the board product 900 may be reduced. There is. Further, there are a large number of teacher images 40, and it is difficult for the operator to check the teacher images 40 one by one and discover inappropriate teacher images 40. Therefore, in the embodiment, an image confirmation device 50 is provided. The image confirmation device 50 determines whether there is a possibility that the board product 900 produced by the board work machine WM0 is a defective product.
　画像確認装置５０は、制御ブロックとして捉えると、取得部５１と、判断部５２とを備えている。画像確認装置５０は、抽出部５３を備えることができる。画像確認装置５０は、案内部５４を備えることもできる。取得部５１、判断部５２、抽出部５３および案内部５４は、種々の演算装置、制御装置などに設けることができる。例えば、取得部５１、判断部５２、抽出部５３および案内部５４のうちの少なくとも一つは、管理装置ＨＣ０に設けることができる。取得部５１、判断部５２、抽出部５３および案内部５４のうちの少なくとも一つは、クラウド上に形成することもできる。 Viewed as a control block, the image confirmation device 50 includes an acquisition section 51 and a determination section 52. The image confirmation device 50 can include an extraction section 53. The image confirmation device 50 can also include a guide section 54. The acquisition unit 51, the determination unit 52, the extraction unit 53, and the guide unit 54 can be provided in various arithmetic devices, control devices, and the like. For example, at least one of the acquisition unit 51, the determination unit 52, the extraction unit 53, and the guide unit 54 can be provided in the management device HC0. At least one of the acquisition unit 51, the determination unit 52, the extraction unit 53, and the guide unit 54 can also be formed on the cloud.
　図４に示すように、実施形態の画像確認装置５０は、取得部５１、判断部５２、抽出部５３および案内部５４を備えている。画像確認装置５０は、管理装置ＨＣ０に設けられている。また、実施形態の画像確認装置５０は、図５に示すフローチャートに従って、制御を実行する。取得部５１は、ステップＳ１１に示す処理を行う。判断部５２は、ステップＳ１２に示す判断を行う。抽出部５３は、ステップＳ１３に示す処理を行う。案内部５４は、ステップＳ１４に示す処理を行う。 As shown in FIG. 4, the image confirmation device 50 of the embodiment includes an acquisition section 51, a determination section 52, an extraction section 53, and a guide section 54. The image confirmation device 50 is provided in the management device HC0. Further, the image confirmation device 50 of the embodiment executes control according to the flowchart shown in FIG. The acquisition unit 51 performs the process shown in step S11. The determining unit 52 makes the determination shown in step S12. The extraction unit 53 performs the process shown in step S13. The guide unit 54 performs the process shown in step S14.
　１－３－１．取得部５１
　取得部５１は、作業前画像４１および作業後画像４２の両方を取得する（図５に示すステップＳ１１）。作業前画像４１は、基板９０に所定の対基板作業を行う対基板作業機ＷＭ０によって対象物９１ｔが設けられる前の基板９０が撮像されている画像をいう。作業後画像４２は、対基板作業機ＷＭ０によって対象物９１ｔが設けられたことが見込まれる基板９０が撮像されている画像をいう。 1-3-1. Acquisition unit 51
The acquisition unit 51 acquires both the before-work image 41 and the after-work image 42 (step S11 shown in FIG. 5). The pre-work image 41 is an image of the board 90 before the object 91t is provided thereon by the board-to-board work machine WM0 that performs a predetermined board-to-board work on the board 90. The post-work image 42 is an image of the board 90 on which the target object 91t is expected to be provided by the board-facing work machine WM0.
　図６は、作業前画像４１の一例を示している。同図は、対基板作業機ＷＭ０である部品装着機ＷＭ３によって、対象物９１ｔである部品９１が設けられる前の基板９０の状態の一例を示している。具体的には、作業前画像４１には、基板９０において対象物９１ｔである部品９１が設けられるべき領域である所定領域９０ｔが撮像されている。所定領域９０ｔには、一対のランド部９０ｒが設けられている。一対のランド部９０ｒの各々には、はんだ９０ｈが印刷されている。一対のランド部９０ｒは、例えば、銅箔で形成されており、部品９１が基板９０に装着された場合に、部品９１の電極部と電気的に接続される。 FIG. 6 shows an example of the pre-work image 41. This figure shows an example of the state of the board 90 before the component 91, which is the target object 91t, is mounted by the component mounting machine WM3, which is the board-to-board working machine WM0. Specifically, the pre-work image 41 captures a predetermined region 90t on the board 90 where a component 91, which is a target object 91t, is to be provided. A pair of land portions 90r are provided in the predetermined region 90t. Solder 90h is printed on each of the pair of land portions 90r. The pair of land portions 90r are made of, for example, copper foil, and are electrically connected to the electrode portions of the component 91 when the component 91 is mounted on the board 90.
　図３は、作業後画像４２の一例を示している。同図は、部品装着機ＷＭ３によって、図６に示す所定領域９０ｔに部品９１が装着された状態の一例を示している。例えば、部品９１がチップ抵抗器、チップコンデンサなどのチップ部品の場合、部品９１は、電極部の領域である電極領域ＡＲ１１および電極領域ＡＲ１２と、本体部の領域である本体領域ＡＲ１３とを備えている。電極領域ＡＲ１１は、一対のランド部９０ｒのうちの一のランド部９０ｒと電気的に接続されている。電極領域ＡＲ１２は、一対のランド部９０ｒのうちの他の一のランド部９０ｒと電気的に接続されている。 FIG. 3 shows an example of the post-work image 42. This figure shows an example of a state in which a component 91 is mounted in the predetermined area 90t shown in FIG. 6 by the component mounting machine WM3. For example, when the component 91 is a chip component such as a chip resistor or a chip capacitor, the component 91 includes an electrode area AR11 and an electrode area AR12, which are electrode area areas, and a main body area AR13, which is a main body area. There is. The electrode region AR11 is electrically connected to one of the pair of land portions 90r. The electrode region AR12 is electrically connected to the other land portion 90r of the pair of land portions 90r.
　このように、図３に示す作業後画像４２は、適切な部品９１が基板９０の所定領域９０ｔに適正に装着されている適切な画像である。しかしながら、作業後画像４２には、適切な部品９１が基板９０の所定領域９０ｔに適正に装着されていない不適切な画像が含まれる可能性がある。例えば、部品９１が所定領域９０ｔに装着されていない基板９０を撮像した画像、所定領域９０ｔから部品９１がはみ出した状態で、部品９１が装着されている基板９０を撮像した画像は、不適切な画像に含まれる。 In this way, the after-work image 42 shown in FIG. 3 is an appropriate image in which the appropriate parts 91 are properly mounted on the predetermined area 90t of the board 90. However, the post-work image 42 may include an inappropriate image in which the appropriate component 91 is not properly attached to the predetermined area 90t of the board 90. For example, an image taken of the board 90 without the component 91 attached to the predetermined area 90t, or an image taken of the board 90 with the component 91 attached with the component 91 protruding from the predetermined area 90t may be inappropriate. included in the image.
　作業前画像４１および作業後画像４２は、基板カメラ１５、外観検査機ＷＭ５、対基板作業機ＷＭ０の外部に設けられるカメラなどの撮像装置８０によって、基板９０の上方から撮像して生成することができる。撮像装置８０は、同種の撮像条件で、作業前画像４１および作業後画像４２を生成することができる。撮像条件には、例えば、光源の種類、光の照射方向、露光時間、絞り値などが含まれる。なお、自然光などの影響によって撮像条件を完全に一致させることは困難であるので、撮像条件は、撮像装置８０によって規定可能な条件であれば良い。 The before-work image 41 and the after-work image 42 can be generated by capturing an image from above the board 90 using an imaging device 80 such as a camera provided outside the board camera 15, the visual inspection machine WM5, and the board-facing working machine WM0. can. The imaging device 80 can generate the before-work image 41 and the after-work image 42 under the same type of imaging conditions. The imaging conditions include, for example, the type of light source, the direction of light irradiation, the exposure time, and the aperture value. Note that, since it is difficult to completely match the imaging conditions due to the influence of natural light, etc., the imaging conditions may be conditions that can be defined by the imaging device 80.
　撮像装置８０は、異なる撮像条件で、作業前画像４１および作業後画像４２を生成することもできる。例えば、撮像装置８０は、基板９０に印刷されているはんだ９０ｈを撮像するのに適した撮像条件で、作業前画像４１を生成し、基板９０に装着されている部品９１を撮像するのに適した撮像条件で、作業後画像４２を生成することもできる。 The imaging device 80 can also generate the before-work image 41 and the after-work image 42 under different imaging conditions. For example, the imaging device 80 generates the pre-work image 41 under imaging conditions suitable for imaging the solder 90h printed on the board 90, and is suitable for imaging the component 91 mounted on the board 90. The post-work image 42 can also be generated under different imaging conditions.
　なお、実施形態では、撮像装置８０は、教師画像４０を使用した基板製品９００の検査が開始される前の生産準備において、作業前画像４１および作業後画像４２を生成する。撮像装置８０は、教師画像４０を使用した基板製品９００の検査が開始された後に生産される基板製品９００を撮像して、作業前画像４１および作業後画像４２を生成することもできる。いずれの場合も、作業前画像４１および作業後画像４２は、図１に示すデータサーバ７０に保存される。取得部５１は、データサーバ７０から作業前画像４１および作業後画像４２を取得することができる。 Note that in the embodiment, the imaging device 80 generates the pre-work image 41 and the post-work image 42 during production preparation before the inspection of the board product 900 using the teacher image 40 is started. The imaging device 80 can also generate the pre-work image 41 and the post-work image 42 by capturing an image of the board product 900 that is produced after the inspection of the board product 900 using the teacher image 40 is started. In either case, the before-work image 41 and the after-work image 42 are stored in the data server 70 shown in FIG. The acquisition unit 51 can acquire the before-work image 41 and the after-work image 42 from the data server 70 .
　１－３－２．判断部５２
　例えば、所定領域９０ｔにおいて、部品９１が装着されていない基板９０が撮像されている作業後画像４２は、作業前画像４１と同様の画像になり、作業前画像４１および作業後画像４２の類似度が高くなる。逆に、適切な部品９１が基板９０の所定領域９０ｔに適正に装着されている作業後画像４２は、作業前画像４１と顕著に異なり、作業前画像４１および作業後画像４２の類似度が低くなる。 1-3-2. Judgment section 52
For example, in the predetermined area 90t, the after-work image 42 in which the board 90 on which the component 91 is not mounted is the same as the before-work image 41, and the similarity between the before-work image 41 and the after-work image 42 is becomes higher. Conversely, the after-work image 42 in which the appropriate parts 91 are properly attached to the predetermined area 90t of the board 90 is significantly different from the before-work image 41, and the similarity between the before-work image 41 and the after-work image 42 is low. Become.
　そこで、判断部５２は、取得部５１によって取得された作業前画像４１および作業後画像４２の類似度に基づいて、対基板作業機ＷＭ０によって生産された基板製品９００が不良品である可能性の有無を判断する（図５に示すステップＳ１２）。判断部５２は、作業前画像４１および作業後画像４２の類似度に基づいて、基板製品９００が不良品である可能性の有無を判断することができれば良く、種々の形態をとり得る。 Therefore, the determination unit 52 determines the possibility that the board product 900 produced by the board work machine WM0 is a defective product based on the degree of similarity between the before-work image 41 and the after-work image 42 acquired by the acquisition unit 51. The presence or absence is determined (step S12 shown in FIG. 5). The determining unit 52 may take various forms as long as it can determine whether there is a possibility that the board product 900 is a defective product based on the degree of similarity between the before-work image 41 and the after-work image 42.
　例えば、図３に示す部品９１の電極領域ＡＲ１１および電極領域ＡＲ１２は、銀色（金属色）である。また、部品９１の表面側（部品９１が適正に基板９０に装着されている場合に視認可能な面）の本体領域ＡＲ１３は、黒色であり、部品９１の裏面（底面）側の本体領域ＡＲ１３は、白色であると仮定する。部品９１が適正に基板９０に装着されている場合、撮像装置８０は、電極領域ＡＲ１１および電極領域ＡＲ１２（銀色）と、部品９１の表面側の本体領域ＡＲ１３（黒色）とを撮像する。 For example, the electrode area AR11 and the electrode area AR12 of the component 91 shown in FIG. 3 are silver (metallic color). Further, the main body area AR13 on the front surface side of the component 91 (the surface that is visible when the component 91 is properly attached to the board 90) is black, and the main body area AR13 on the back surface (bottom surface) side of the component 91 is black. , is assumed to be white. When the component 91 is properly attached to the substrate 90, the imaging device 80 images the electrode area AR11 and the electrode area AR12 (silver) and the main body area AR13 (black) on the front side of the component 91.
　保持部材３０が部品９１の裏面側を誤って吸着して、部品９１の表面側と裏面側とが反転した状態で部品９１が基板９０に装着されている場合、撮像装置８０は、電極領域ＡＲ１１および電極領域ＡＲ１２（銀色）と、部品９１の裏面側の本体領域ＡＲ１３（白色）とを撮像する。部品９１の表面側と裏面側とが反転した状態で部品９１が基板９０に装着されている場合、例えば、電極領域ＡＲ１１および電極領域ＡＲ１２と、部品９１の本体領域ＡＲ１３との輝度の差は、部品９１が適正に基板９０に装着されている場合と比べて、小さくなる。 If the holding member 30 mistakenly attracts the back side of the component 91 and the component 91 is attached to the substrate 90 with the front side and the back side of the component 91 reversed, the imaging device 80 Then, the electrode area AR12 (silver color) and the main body area AR13 (white color) on the back side of the component 91 are imaged. When the component 91 is attached to the substrate 90 with the front side and the back side of the component 91 reversed, for example, the difference in brightness between the electrode area AR11 and the electrode area AR12 and the main body area AR13 of the component 91 is as follows. The size is smaller than when the component 91 is properly attached to the board 90.
　このように、不適切な作業後画像４２（例えば、適切な部品９１が基板９０の所定領域９０ｔに適正に装着されていない画像）は、適切な作業後画像４２（適切な部品９１が基板９０の所定領域９０ｔに適正に装着されている画像）と比べて、画像全体の特徴量が変化する。画像を構成する複数の画素の輝度は、特徴量に含まれる。 In this way, the inappropriate after-work image 42 (for example, an image in which the appropriate part 91 is not properly attached to the predetermined area 90t of the board 90) is different from the inappropriate after-work image 42 (for example, an image in which the appropriate part 91 is not properly mounted on the board The feature amount of the entire image changes compared to the image in which the image is properly attached to the predetermined area 90t. The brightness of a plurality of pixels constituting an image is included in the feature amount.
　また、基板９０において対象物９１ｔが設けられるべき所定領域９０ｔの重心と、基板９０において対象物９１ｔが設けられている対象領域９１ｓの重心との偏差が大きくなるほど、作業後画像４２は、不適切である。既述されている例では、所定領域９０ｔは、基板９０において部品９１が装着されるべき目標領域に相当する。対象領域９１ｓは、基板９０において部品９１が実際に装着されている装着領域に相当する。 Furthermore, the greater the deviation between the center of gravity of the predetermined region 90t on the board 90 where the object 91t is to be provided and the center of gravity of the target region 91s on the board 90 where the object 91t is provided, the more inappropriate the post-work image 42 becomes. It is. In the example already described, the predetermined area 90t corresponds to the target area on the board 90 where the component 91 is to be mounted. The target area 91s corresponds to a mounting area on the board 90 where the component 91 is actually mounted.
　さらに、基板９０に設けられるべき対象物９１ｔと異なる対象物９１ｔが基板９０に設けられている作業後画像４２は、不適切である。既述されている例では、基板９０に設けられるべき対象物９１ｔは、基板９０に装着されるべき部品９１に相当する。この場合、対象領域９１ｓの面積は、設けられるべき対象物９１ｔが基板９０に設けられている場合と比べて、増減する可能性がある。 Further, the post-work image 42 in which the substrate 90 has an object 91t different from the object 91t that should be provided on the substrate 90 is inappropriate. In the example already described, the object 91t to be provided on the board 90 corresponds to the component 91 to be mounted on the board 90. In this case, the area of the target region 91s may increase or decrease compared to the case where the target object 91t to be provided is provided on the substrate 90.
　また、対象物９１ｔの形状を示す指標（例えば、真円度）は、設けられるべき対象物９１ｔが基板９０に設けられている場合と比べて、増減する可能性がある。例えば、正方形状の部品９１は、矩形状の部品９１と比べて真円度が高く、基板９０に設けられるべき対象物９１ｔであるか否かを判断する際の指標の一つになる。 Further, the index (for example, circularity) indicating the shape of the target object 91t may increase or decrease compared to the case where the target object 91t to be provided is provided on the substrate 90. For example, the square component 91 has a higher degree of circularity than the rectangular component 91, which is one of the indicators for determining whether the object 91t should be provided on the substrate 90.
　そこで、判断部５２は、作業前画像４１の画像全体の特徴量と作業後画像４２の画像全体の特徴量とのユークリッド距離が小さいほど、作業前画像４１および作業後画像４２の類似度が高いと判断することができる。また、特徴量は、画像を構成する複数の画素の輝度、基板９０において対象物９１ｔが設けられている対象領域９１ｓの重心、対象領域９１ｓの面積、および、対象物９１ｔの形状を示す指標のうちの少なくとも一つであると好適である。 Therefore, the determining unit 52 determines that the smaller the Euclidean distance between the feature amount of the entire image of the before-work image 41 and the feature amount of the entire image of the after-work image 42, the higher the degree of similarity between the before-work image 41 and the after-work image 42. It can be determined that Further, the feature amounts include the luminance of a plurality of pixels constituting the image, the center of gravity of the target region 91s in which the target object 91t is provided on the substrate 90, the area of the target region 91s, and an index indicating the shape of the target object 91t. At least one of these is preferred.
　実施形態では、判断部５２は、上記の特徴量を用いて、作業前画像４１および作業後画像４２の類似度を判断する。例えば、画像を構成する複数の画素の輝度は、マハラノビス距離によって表すことができる。判断部５２は、作業前画像４１の画像を構成する複数の画素の輝度を表すマハラノビス距離を、０（最小値）から１（最大値）で正規化し、作業後画像４２の画像を構成する複数の画素の輝度を表すマハラノビス距離を、０（最小値）から１（最大値）で正規化する。同様に、判断部５２は、対象領域９１ｓの重心、対象領域９１ｓの面積、および、対象物９１ｔの形状を示す指標について、正規化する。 In the embodiment, the determination unit 52 determines the degree of similarity between the before-work image 41 and the after-work image 42 using the above-mentioned feature amounts. For example, the brightness of a plurality of pixels constituting an image can be expressed by Mahalanobis distance. The determining unit 52 normalizes the Mahalanobis distance representing the brightness of a plurality of pixels constituting the image before work 41 from 0 (minimum value) to 1 (maximum value), and The Mahalanobis distance representing the brightness of the pixel is normalized from 0 (minimum value) to 1 (maximum value). Similarly, the determination unit 52 normalizes the center of gravity of the target region 91s, the area of the target region 91s, and the index indicating the shape of the target object 91t.
　判断部５２は、作業前画像４１について正規化された輝度値と、作業後画像４２について正規化された輝度値との差分を算出する。同様に、判断部５２は、作業前画像４１について正規化された対象領域９１ｓの重心と、作業後画像４２について正規化された対象領域９１ｓの重心との差分を算出する。判断部５２は、作業前画像４１について正規化された対象領域９１ｓの面積と、作業後画像４２について正規化された対象領域９１ｓの面積との差分を算出する。判断部５２は、作業前画像４１について正規化された対象物９１ｔの形状を示す指標と、作業後画像４２について正規化された対象物９１ｔの形状を示す指標との差分を算出する。 The determination unit 52 calculates the difference between the normalized brightness value of the before-work image 41 and the normalized brightness value of the after-work image 42. Similarly, the determination unit 52 calculates the difference between the center of gravity of the target area 91s normalized for the pre-work image 41 and the center of gravity of the target area 91s normalized for the post-work image 42. The determining unit 52 calculates the difference between the area of the target area 91s normalized for the pre-work image 41 and the area of the target area 91s normalized for the post-work image 42. The determining unit 52 calculates the difference between an index indicating the shape of the target object 91t normalized in the pre-work image 41 and an index indicating the shape of the target object 91t normalized in the post-work image 42.
　そして、判断部５２は、上記の差分の二乗和を算出する。判断部５２は、二乗和が０（ゼロ）に近いほど、作業前画像４１の画像全体の特徴量と作業後画像４２の画像全体の特徴量とのユークリッド距離が小さく、作業前画像４１および作業後画像４２の類似度が高いと判断することができる。逆に、判断部５２は、二乗和が大きくなるほど、作業前画像４１の画像全体の特徴量と作業後画像４２の画像全体の特徴量とのユークリッド距離が大きく、作業前画像４１および作業後画像４２の類似度が低いと判断することができる。 Then, the determining unit 52 calculates the sum of squares of the above differences. The determining unit 52 determines that the closer the sum of squares is to 0 (zero), the smaller the Euclidean distance between the feature amount of the entire image of the pre-work image 41 and the feature amount of the entire image of the post-work image 42; It can be determined that the similarity of the subsequent image 42 is high. Conversely, the determining unit 52 determines that as the sum of squares increases, the Euclidean distance between the feature amount of the entire image of the before-work image 41 and the feature amount of the entire image of the after-work image 42 increases, and the difference between the before-work image 41 and the after-work image 42 increases. It can be determined that the similarity of 42 is low.
　判断部５２は、作業前画像４１および作業後画像４２の類似度が高いほど、対基板作業機ＷＭ０によって生産された基板製品９００が不良品である可能性があると判断することができる。逆に、判断部５２は、作業前画像４１および作業後画像４２の類似度が低いほど、対基板作業機ＷＭ０によって生産された基板製品９００が不良品である可能性がないと判断することができる。 The determining unit 52 can determine that the higher the similarity between the before-work image 41 and the after-work image 42, the more likely the board product 900 produced by the board-to-board working machine WM0 is a defective product. Conversely, the lower the degree of similarity between the before-work image 41 and the after-work image 42, the less likely the judgment unit 52 is to judge that the board product 900 produced by the board work machine WM0 is unlikely to be a defective product. can.
　判断部５２は、基板製品９００が不良品である可能性があるか否かを判断する閾値を、作業者によって指定された指定値または予め規定されている固定値に設定することができる。例えば、作業者は、後述される表示装置６０において、上記の指定値を入力することができる。例えば、判断部５２は、基板製品９００が不良品である可能性がないと判断したが、当該基板製品９００が不良品であることが確認された場合、作業者は、現在の閾値よりも小さい閾値を指定値として入力することができる。逆に、判断部５２は、基板製品９００が不良品である可能性があると判断したが、当該基板製品９００が良品であることが確認された場合、作業者は、現在の閾値よりも大きい閾値を指定値として入力することができる。 The determining unit 52 can set a threshold value for determining whether there is a possibility that the board product 900 is a defective product to a designated value designated by the operator or a predefined fixed value. For example, the operator can input the above specified value on the display device 60, which will be described later. For example, if the determining unit 52 determines that there is no possibility that the board product 900 is a defective product, but it is confirmed that the board product 900 is a defective product, the operator may A threshold value can be input as a specified value. Conversely, if the determining unit 52 determines that there is a possibility that the board product 900 is a defective product, but it is confirmed that the board product 900 is a non-defective product, the operator may A threshold value can be input as a specified value.
　また、判断部５２は、シミュレーション、実機による検証などによって、予め上記の固定値を取得しておくことができる。いずれの場合も、判断部５２は、上記の差分の二乗和が閾値よりも小さい場合に、作業前画像４１および作業後画像４２の類似度が高いと判断することができ、基板製品９００が不良品である可能性があると判断することができる。また、判断部５２は、上記の差分の二乗和が閾値以上の場合に、作業前画像４１および作業後画像４２の類似度が低いと判断することができ、基板製品９００が不良品である可能性がないと判断することができる。 Further, the determination unit 52 can obtain the above-mentioned fixed value in advance through simulation, verification using an actual machine, or the like. In either case, if the sum of squares of the differences is smaller than the threshold, the determining unit 52 can determine that the degree of similarity between the before-work image 41 and the after-work image 42 is high, and the board product 900 is defective. It can be determined that the product may be of good quality. Further, when the sum of squares of the above-mentioned differences is greater than or equal to the threshold value, the determining unit 52 can determine that the degree of similarity between the before-work image 41 and the after-work image 42 is low, and it is possible that the board product 900 is a defective product. It can be determined that there is no gender.
　１－３－３．抽出部５３および案内部５４
　抽出部５３は、判断部５２によって基板製品９００が不良品である可能性があると判断された場合（図５に示すステップＳ１２においてＹｅｓの場合）に、当該基板製品９００の作業後画像４２を抽出する（ステップＳ１３）。例えば、抽出部５３によって抽出された作業後画像４２は、データサーバ７０から削除することができる。これにより、当該作業後画像４２は、教師画像４０として使用されなくなる。 1-3-3. Extraction section 53 and guide section 54
When the determining unit 52 determines that the board product 900 may be defective (Yes in step S12 shown in FIG. 5), the extracting unit 53 extracts the after-work image 42 of the board product 900. Extract (step S13). For example, the post-work image 42 extracted by the extraction unit 53 can be deleted from the data server 70. As a result, the post-work image 42 is no longer used as the teacher image 40.
　また、抽出部５３によって抽出された作業後画像４２の中には、教師画像４０として使用可能な画像が含まれている可能がある。そこで、案内部５４は、抽出部５３によって抽出された作業後画像４２を作業者に案内することができる。案内部５４は、抽出部５３によって抽出された作業後画像４２を作業者に案内することができれば良く、種々の形態をとり得る。案内部５４は、作業後画像４２の表示、作業後画像４２を特定可能な情報（例えば、画像データのファイル名）の音声案内など種々の方法によって、作業後画像４２を案内することができる。 Further, the post-work images 42 extracted by the extraction unit 53 may include images that can be used as the teacher images 40. Therefore, the guide section 54 can guide the worker through the post-work image 42 extracted by the extraction section 53. The guide section 54 only needs to be able to guide the worker through the post-work image 42 extracted by the extraction section 53, and can take various forms. The guide unit 54 can guide the after-work image 42 by various methods such as displaying the after-work image 42 and providing audio guidance of information (for example, file name of image data) that can identify the after-work image 42 .
　実施形態では、案内部５４は、抽出部５３によって抽出された作業後画像４２を表示装置６０に表示させる（図５に示すステップＳ１４）。表示装置６０は、作業後画像４２を表示することができれば良く、公知の種々の表示装置を用いることができる。図７は、画像確認装置５０による作業後画像４２の表示例を示している。具体的には、同図は、表示装置６０の表示画面の一例を示している。 In the embodiment, the guide unit 54 causes the display device 60 to display the after-work image 42 extracted by the extraction unit 53 (step S14 shown in FIG. 5). The display device 60 only needs to be able to display the post-work image 42, and various known display devices can be used. FIG. 7 shows an example of the display of the post-work image 42 by the image confirmation device 50. Specifically, the figure shows an example of a display screen of the display device 60.
　例えば、案内部５４は、作業前画像４１および作業後画像４２の類似度が高い順に、表示装置６０に作業後画像４２を表示させる。これにより、作業者は、表示装置６０において、不良品である可能性が高い基板製品９００が撮像されている作業後画像４２から順に、作業後画像４２を確認することができる。 For example, the guide unit 54 causes the display device 60 to display the after-work images 42 in order of the similarity between the before-work image 41 and the after-work image 42. Thereby, the worker can check the post-work images 42 on the display device 60 in order from the post-work image 42 in which the board product 900 that is likely to be a defective product is captured.
　同図に示す１番の領域ＡＲ２１の作業後画像４２は、部品９１が所定領域９０ｔに装着されていない基板９０を撮像した画像である。１番の領域ＡＲ２１の作業後画像４２は、既述されているユークリッド距離が最も小さく、作業前画像４１および作業後画像４２の類似度が最も高い。よって、基板製品９００が不良品である可能性が最も高い。 The post-work image 42 of the No. 1 area AR21 shown in the figure is an image of the board 90 in which the component 91 is not attached to the predetermined area 90t. The after-work image 42 of the first region AR21 has the smallest Euclidean distance described above, and the similarity between the before-work image 41 and the after-work image 42 is the highest. Therefore, it is most likely that the board product 900 is a defective product.
　また、２番の領域ＡＲ２２の作業後画像４２は、所定領域９０ｔから部品９１がはみ出した状態で、部品９１が装着されている基板９０を撮像した画像である。２番の領域ＡＲ２２の作業後画像４２は、１番の領域ＡＲ２１の作業後画像４２と比べて、既述されているユークリッド距離が大きく、作業前画像４１および作業後画像４２の類似度が低い。１番の領域ＡＲ２１の作業後画像４２に撮像されている基板製品９００の次に、基板製品９００が不良品である可能性が高い。 Further, the after-work image 42 of the second area AR22 is an image of the board 90 on which the component 91 is mounted, with the component 91 protruding from the predetermined area 90t. Compared to the after-work image 42 of the first area AR21, the after-work image 42 of the second area AR22 has a larger Euclidean distance as described above, and the similarity between the before-work image 41 and the after-work image 42 is low. . Next to the board product 900 captured in the after-work image 42 of the first area AR21, the board product 900 is most likely to be a defective product.
　同様に、３番の領域ＡＲ２３の作業後画像４２は、所定領域９０ｔから部品９１がはみ出した状態で、部品９１が装着されている基板９０を撮像した画像である。但し、３番の領域ＡＲ２３の作業後画像４２は、２番の領域ＡＲ２２の作業後画像４２と比べて、所定領域９０ｔから部品９１がはみ出している割合が少ない。３番の領域ＡＲ２３の作業後画像４２は、２番の領域ＡＲ２２の作業後画像４２と比べて、既述されているユークリッド距離が大きく、作業前画像４１および作業後画像４２の類似度が低い。２番の領域ＡＲ２２の作業後画像４２に撮像されている基板製品９００の次に、基板製品９００が不良品である可能性が高い。 Similarly, the after-work image 42 of the third area AR23 is an image of the board 90 on which the component 91 is mounted, with the component 91 protruding from the predetermined area 90t. However, in the after-work image 42 of the No. 3 area AR23, the proportion of parts 91 protruding from the predetermined area 90t is smaller than in the after-work image 42 of the No. 2 area AR22. Compared to the after-work image 42 of the second area AR22, the after-work image 42 of the third area AR23 has a larger Euclidean distance as described above, and the similarity between the before-work image 41 and the after-work image 42 is low. . Next to the board product 900 captured in the after-work image 42 of the second area AR22, the board product 900 is most likely to be a defective product.
　また、案内部５４は、作業前画像４１および作業後画像４２の類似度の区分に応じて、表示装置６０に表示させる作業後画像４２の表示方法を変更することができる。案内部５４は、作業前画像４１および作業後画像４２の類似度について、任意に区分を設定することができる。また、案内部５４は、種々の表示方法をとり得る。例えば、案内部５４は、１番の領域ＡＲ２１、２番の領域ＡＲ２２および３番の領域ＡＲ２３のうちの少なくとも一つの領域において、作業後画像４２の背景の表示色（濃淡を含む）を変更することができる。 Further, the guide unit 54 can change the display method of the after-work image 42 to be displayed on the display device 60 according to the classification of similarity between the before-work image 41 and the after-work image 42. The guide unit 54 can arbitrarily set classifications for the degree of similarity between the before-work image 41 and the after-work image 42. Further, the guide section 54 can take various display methods. For example, the guide unit 54 changes the display color (including shading) of the background of the post-work image 42 in at least one of the first area AR21, the second area AR22, and the third area AR23. be able to.
　例えば、案内部５４は、１番の領域ＡＲ２１の背景を所定色（例えば、赤色）で表示装置６０に表示させる。案内部５４は、２番の領域ＡＲ２２の背景を他の所定色（例えば、ピンク色）で表示装置６０に表示させる。案内部５４は、３番の領域ＡＲ２３の背景を他の所定色（例えば、白色）で表示装置６０に表示させる。このように、案内部５４は、作業前画像４１および作業後画像４２の類似度が高い区分ほど、作業者に対する注意喚起が容易な表示色で、表示装置６０に作業後画像４２の背景などを表示させることができる。 For example, the guide unit 54 causes the display device 60 to display the background of the first area AR21 in a predetermined color (for example, red). The guide unit 54 causes the display device 60 to display the background of the second area AR22 in another predetermined color (for example, pink). The guide unit 54 causes the display device 60 to display the background of the third area AR23 in another predetermined color (for example, white). In this way, the guide unit 54 displays the background of the after-work image 42 on the display device 60 in a display color that is easier to alert the worker in the classification where the degree of similarity between the before-work image 41 and the after-work image 42 is higher. It can be displayed.
　また、案内部５４は、１番の領域ＡＲ２１、２番の領域ＡＲ２２および３番の領域ＡＲ２３のうちの少なくとも一つの領域において、点滅表示させることもできる。この場合、案内部５４は、作業前画像４１および作業後画像４２の類似度が高い区分ほど、点滅周期を短くすることができる。逆に、案内部５４は、作業前画像４１および作業後画像４２の類似度が低い区分ほど、点滅周期を長くすることができる。さらに、案内部５４は、表示色および点滅表示を組み合わせて、表示装置６０に作業後画像４２の背景などを表示させる際の表示方法を変更することもできる。 Furthermore, the guide section 54 can also cause blinking display in at least one of the first area AR21, the second area AR22, and the third area AR23. In this case, the guide unit 54 can shorten the blinking cycle for the classification in which the pre-work image 41 and the post-work image 42 have a higher degree of similarity. Conversely, the guide unit 54 can lengthen the blinking cycle for a category in which the degree of similarity between the before-work image 41 and the after-work image 42 is lower. Furthermore, the guide unit 54 can also change the display method when displaying the background of the post-work image 42 on the display device 60 by combining display colors and blinking displays.
　また、案内部５４は、表示装置６０に表示されている作業後画像４２に基づいて、作業者に基板製品９００の良否を確認させることができる。例えば、図７に示す表示画面では、領域ＡＲ３１において、表示されている画像は、部品９１の装着に失敗した可能性があり、目視確認後に判定結果を入力する旨の指示が表示されている。 Further, the guide unit 54 can have the operator check the quality of the board product 900 based on the post-work image 42 displayed on the display device 60. For example, on the display screen shown in FIG. 7, the displayed image in area AR31 indicates that mounting of component 91 may have failed, and an instruction to input the determination result after visual confirmation is displayed.
　例えば、図７に示す表示画面は、タッチパネルにより構成されている。作業者は、基板製品９００の良否を確認しようとする領域において、操作部ＢＡ１または操作部ＢＡ２をタッチする。例えば、作業者は、１番の領域ＡＲ２１の作業後画像４２に撮像されている基板製品９００が良品（Ｐａｓｓ）であることを確認した場合に、１番の領域ＡＲ２１の操作部ＢＡ１をタッチする。 For example, the display screen shown in FIG. 7 is composed of a touch panel. The operator touches the operation unit BA1 or BA2 in the area where the quality of the board product 900 is to be confirmed. For example, when the operator confirms that the board product 900 captured in the post-work image 42 of the first area AR21 is a good product (Pass), the operator touches the operation unit BA1 of the first area AR21. .
　逆に、作業者は、１番の領域ＡＲ２１の作業後画像４２に撮像されている基板製品９００が不良品（Ｆａｉｌ）であることを確認した場合に、１番の領域ＡＲ２１の操作部ＢＡ２をタッチする。同図では、作業者によって不良品（Ｆａｉｌ）が選択されており、当該基板製品９００が不良品（Ｆａｉｌ）であることが確認されている。１番の領域ＡＲ２１について上述されていることは、他の領域についても同様に言える。但し、他の領域は、作業者による確認前の状態である。 Conversely, if the operator confirms that the board product 900 captured in the post-work image 42 of the No. 1 area AR21 is a defective product (Fail), the operator operates the operation unit BA2 of the No. 1 area AR21. touch. In the figure, a defective product (Fail) has been selected by the operator, and it has been confirmed that the board product 900 is a defective product (Fail). What has been described above regarding the first area AR21 is also true for the other areas. However, other areas are in a state before being checked by the operator.
　また、作業者は、１番の領域ＡＲ２１、２番の領域ＡＲ２２および３番の領域ＡＲ２３のうちの少なくとも一つの領域をタッチして選択し、選択した領域の作業後画像４２に撮像されている基板製品９００が良品（Ｐａｓｓ）であることを確認した場合に、操作部ＢＢ１をタッチすることもできる。また、作業者は、１番の領域ＡＲ２１、２番の領域ＡＲ２２および３番の領域ＡＲ２３のうちの少なくとも一つの領域をタッチして選択し、選択した領域の作業後画像４２に撮像されている基板製品９００が不良品（Ｆａｉｌ）であることを確認した場合に、操作部ＢＢ２をタッチすることもできる。 Further, the worker touches and selects at least one of the first area AR21, the second area AR22, and the third area AR23, and the after-work image 42 of the selected area is captured. When it is confirmed that the board product 900 is a good product (Pass), it is also possible to touch the operation part BB1. Further, the worker touches and selects at least one of the first area AR21, the second area AR22, and the third area AR23, and the after-work image 42 of the selected area is captured. When it is confirmed that the board product 900 is a defective product (Fail), the operation section BB2 can also be touched.
　さらに、作業者は、操作部ＢＢ３をタッチすることにより、１番の領域ＡＲ２１、２番の領域ＡＲ２２および３番の領域ＡＲ２３の全ての領域を選択することもできる。作業者は、操作部ＢＢ３を再度タッチすることにより、領域の選択を解除することもできる。また、作業者は、操作部ＢＣ１をタッチすることにより、次の表示画面（作業前画像４１および作業後画像４２の類似度が現在表示されている作業後画像４２より低い作業後画像４２）を表示させることができる。作業者は、操作部ＢＣ２をタッチすることにより、前の表示画面（作業前画像４１および作業後画像４２の類似度が現在表示されている作業後画像４２より高い作業後画像４２）を表示させることができる。 Furthermore, the operator can also select all of the first area AR21, the second area AR22, and the third area AR23 by touching the operation part BB3. The operator can also deselect the area by touching the operation unit BB3 again. In addition, the worker can display the next display screen (the after-work image 42 in which the degree of similarity between the before-work image 41 and the after-work image 42 is lower than the currently displayed after-work image 42) by touching the operation unit BC1. It can be displayed. The worker displays the previous display screen (the after-work image 42 in which the similarity between the before-work image 41 and the after-work image 42 is higher than the currently displayed after-work image 42) by touching the operation unit BC2. be able to.
　なお、案内部５４は、作業前画像４１および作業後画像４２の類似度、作業後画像４２に関する画像情報などの種々の情報を、作業後画像４２と共に表示装置６０に表示させることができる。作業後画像４２に関する画像情報は、限定されない。作業後画像４２に関する画像情報は、作業後画像４２が取得された日時に関する情報、撮像装置８０に関する情報、撮像条件（例えば、撮像装置８０が作業後画像４２を撮像したときの露光時間、絞り値、光源の種類、光の照射方向）に関する情報などの種々の情報を含み得る。 Note that the guide unit 54 can display various information such as the degree of similarity between the before-work image 41 and the after-work image 42 and image information regarding the after-work image 42 on the display device 60 together with the after-work image 42. The image information regarding the post-work image 42 is not limited. The image information regarding the after-work image 42 includes information regarding the date and time when the after-work image 42 was acquired, information regarding the imaging device 80, and imaging conditions (for example, the exposure time and aperture value when the imaging device 80 captured the after-work image 42). , type of light source, and direction of light irradiation).
　判断部５２は、以下に示す二つの作業後画像４２のうちの少なくとも一つを、機械学習の教師画像４０として使用させることができる。二つの作業後画像４２のうちの一つは、基板製品９００が不良品である可能性がないと判断した場合の当該基板製品９００の作業後画像４２である。二つの作業後画像４２のうちの他の一つは、基板製品９００が不良品である可能性があると判断したが作業者によって当該基板製品９００が良品であることが確認された場合の当該基板製品９００の作業後画像４２である。 The determining unit 52 can cause at least one of the two post-work images 42 shown below to be used as the teacher image 40 for machine learning. One of the two after-work images 42 is the after-work image 42 of the board product 900 when it is determined that there is no possibility that the board product 900 is a defective product. The other one of the two after-work images 42 is an image in which it is determined that the board product 900 may be a defective product, but the operator confirms that the board product 900 is a good product. This is an image 42 of the board product 900 after work.
　実施形態では、判断部５２は、上記の二つの作業後画像４２を、機械学習の教師画像４０として使用させる。教師画像４０は、公知の種々の機械学習に使用することができる。例えば、教師画像４０は、サポートベクターマシン、回帰分析などの種々の機械学習に使用することができる。実施形態では、対基板作業機ＷＭ０は、対象物９１ｔである部品９１を基板９０に装着する部品装着機ＷＭ３である。 In the embodiment, the determination unit 52 causes the two post-work images 42 described above to be used as the teacher images 40 for machine learning. The teacher image 40 can be used for various known machine learning methods. For example, the teacher image 40 can be used for various types of machine learning such as support vector machine and regression analysis. In the embodiment, the substrate-to-board working machine WM0 is a component mounting machine WM3 that mounts a component 91, which is a target object 91t, onto a board 90.
　例えば、部品装着機ＷＭ３は、教師画像４０を使用して、部品装着機ＷＭ３によって基板９０の所定領域９０ｔに部品９１が装着されているか否かを検査する部品有無検査を行うことができる。また、部品装着機ＷＭ３は、教師画像４０を使用して、基板９０において部品９１が装着されるべき所定領域９０ｔの重心と、基板９０において部品９１が装着されている対象領域９１ｓの重心との偏差を検査する部品９１の位置ずれ検査を行うこともできる。 For example, the component mounting machine WM3 can use the teacher image 40 to perform a component presence/absence test to check whether the component 91 is mounted on the predetermined area 90t of the board 90 by the component mounting machine WM3. The component mounting machine WM3 also uses the teacher image 40 to determine the center of gravity of a predetermined region 90t on the board 90 where the component 91 is to be mounted and the center of gravity of a target region 91s on the board 90 where the component 91 is mounted. It is also possible to perform a positional shift inspection of the component 91 to be inspected for deviation.
　なお、判断部５２は、基板製品９００が不良品である可能性があると判断し、作業者によって当該基板製品９００が不良品であることが確認された場合の当該基板製品９００の作業後画像４２を、機械学習の教師画像４０として使用させない。具体的には、判断部５２は、教師画像４０として使用させない作業後画像４２をデータサーバ７０から削除することができる。作業後画像４２の削除は、当該作業後画像４２を使用する対基板作業機ＷＭ０において反映される。 Note that the determining unit 52 determines that there is a possibility that the board product 900 is a defective product, and the after-work image of the board product 900 when the operator confirms that the board product 900 is a defective product. 42 is not used as the teacher image 40 for machine learning. Specifically, the determination unit 52 can delete from the data server 70 the post-work images 42 that are not allowed to be used as the teacher images 40. The deletion of the post-work image 42 is reflected in the board-facing work machine WM0 that uses the post-work image 42.
　また、図７では、部品装着機ＷＭ３によって基板９０に装着された複数の部品９１のうちの一つの部品９１について、作業後画像４２が案内されている。案内部５４は、同様にして、他の部品９１についても作業後画像４２を案内することができる。また、判断部５２によって基板製品９００が不良品である可能性がないと判断された場合（図５に示すステップＳ１２でＮｏの場合）、ステップＳ１３およびステップＳ１４に示す処理を実行しないで、画像確認装置５０による制御は、一旦、終了する。 Further, in FIG. 7, an after-work image 42 is shown for one component 91 among the plurality of components 91 mounted on the board 90 by the component mounting machine WM3. The guide unit 54 can similarly guide the post-work images 42 for other parts 91 as well. Further, if the determining unit 52 determines that there is no possibility that the board product 900 is a defective product (No in step S12 shown in FIG. 5), the process shown in step S13 and step S14 is not executed, and the image is The control by the confirmation device 50 ends once.
　２．他の形態
　対基板作業機ＷＭ０は、部品装着機ＷＭ３に限定されない。例えば、対基板作業機ＷＭ０は、基板９０にはんだ９０ｈを印刷する印刷機ＷＭ１であっても良い。この場合、基板９０に印刷されるはんだ９０ｈは、対象物９１ｔに含まれる。作業前画像４１は、はんだ９０ｈが印刷される前の基板９０が撮像されている画像である。作業後画像４２は、はんだ９０ｈが印刷されたことが見込まれる基板９０が撮像されている画像である。また、基板９０において対象物９１ｔが設けられるべき所定領域９０ｔは、基板９０において、はんだ９０ｈが印刷されるべき目標領域に相当する。基板９０において対象物９１ｔが設けられている対象領域９１ｓは、基板９０において、はんだ９０ｈが実際に印刷されている印刷領域に相当する。 2. Other forms The board-to-board working machine WM0 is not limited to the component mounting machine WM3. For example, the substrate work machine WM0 may be a printing machine WM1 that prints the solder 90h on the substrate 90. In this case, the solder 90h printed on the substrate 90 is included in the target object 91t. The pre-work image 41 is an image of the board 90 before the solder 90h is printed. The post-work image 42 is an image of the board 90 on which the solder 90h is expected to be printed. Furthermore, the predetermined area 90t on the substrate 90 where the target object 91t is to be provided corresponds to the target area on the substrate 90 where the solder 90h is to be printed. The target area 91s on the board 90 where the target object 91t is provided corresponds to the printing area on the board 90 where the solder 90h is actually printed.
　さらに、対基板作業機ＷＭ０は、機械学習による種々の検査を行うことができる。例えば、基板製品９００の検査は、はんだ有無検査であっても良い。また、基板製品９００の検査は、基板９０において、はんだ９０ｈが印刷されるべき所定領域９０ｔの重心と、基板９０において、はんだ９０ｈが印刷されている対象領域９１ｓの重心との偏差を検査するはんだ９０ｈの位置ずれ検査であっても良い。 Furthermore, the board-facing work machine WM0 can perform various inspections using machine learning. For example, the inspection of the board product 900 may be a solder presence inspection. In addition, the inspection of the board product 900 involves inspecting the deviation between the center of gravity of a predetermined area 90t on the board 90 where the solder 90h is to be printed and the center of gravity of the target area 91s on the board 90 where the solder 90h is printed. It may be a 90 hour positional deviation inspection.
　３．画像確認方法
　画像確認装置５０について既述されていることは、画像確認方法についても同様に言える。具体的には、画像確認方法は、取得工程と、判断工程とを備える。取得工程は、取得部５１が行う制御に相当する。判断工程は、判断部５２が行う制御に相当する。画像確認方法は、抽出工程を備えることができる。抽出工程は、抽出部５３が行う制御に相当する。画像確認方法は、案内工程を備えることもできる。案内工程は、案内部５４が行う制御に相当する。 3. Image Verification Method What has already been described regarding the image verification device 50 is also applicable to the image verification method. Specifically, the image confirmation method includes an acquisition step and a determination step. The acquisition process corresponds to control performed by the acquisition unit 51. The determination process corresponds to the control performed by the determination unit 52. The image confirmation method can include an extraction step. The extraction process corresponds to the control performed by the extraction unit 53. The image confirmation method can also include a guiding step. The guiding process corresponds to the control performed by the guiding section 54.
　４．実施形態の効果の一例
　画像確認装置５０によれば、取得部５１および判断部５２を備えている。これにより、画像確認装置５０は、作業前画像４１および作業後画像４２の類似度に基づいて、対基板作業機ＷＭ０によって生産された基板製品９００が不良品である可能性の有無を判断することができる。画像確認装置５０について上述されていることは、画像確認方法についても同様に言える。 4. Example of Effect of Embodiment The image confirmation device 50 includes an acquisition section 51 and a determination section 52. Thereby, the image confirmation device 50 determines whether or not there is a possibility that the board product 900 produced by the board work machine WM0 is a defective product based on the degree of similarity between the before-work image 41 and the after-work image 42. I can do it. What has been described above regarding the image confirmation device 50 also applies to the image confirmation method.
４０：教師画像、４１：作業前画像、４２：作業後画像、
５０：画像確認装置、５１：取得部、５２：判断部、５３：抽出部、
５４：案内部、６０：表示装置、９０：基板、９１ｓ：対象領域、
９１ｔ：対象物、９００：基板製品、ＷＭ０：対基板作業機。 40: Teacher image, 41: Before work image, 42: After work image,
50: Image confirmation device, 51: Acquisition unit, 52: Judgment unit, 53: Extraction unit,
54: guide section, 60: display device, 90: substrate, 91s: target area,
91t: Target object, 900: Board product, WM0: Board working machine.

Claims (11)

  1. 　基板に所定の対基板作業を行う対基板作業機によって対象物が設けられる前の前記基板が撮像されている作業前画像、および、前記対基板作業機によって前記対象物が設けられたことが見込まれる前記基板が撮像されている作業後画像の両方を取得する取得部と、
    　前記取得部によって取得された前記作業前画像および前記作業後画像の類似度に基づいて、前記対基板作業機によって生産された基板製品が不良品である可能性の有無を判断する判断部と、
    を備える画像確認装置。     A pre-work image of the board before a target object is provided on the board by a board-to-board working machine that performs a predetermined board-to-board work, and a pre-work image in which the board is imaged before the target object is provided by the board-to-board working machine, and a pre-work image in which it is expected that the target object is provided by the board-to-board working machine. an acquisition unit that acquires both post-work images in which the board is imaged;
    a determination unit that determines whether or not there is a possibility that the board product produced by the board-to-board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image acquired by the acquisition unit;
    An image confirmation device comprising:
  2. 　前記判断部によって前記基板製品が不良品である可能性があると判断された場合に、当該基板製品の前記作業後画像を抽出する抽出部を備える請求項１に記載の画像確認装置。 The image confirmation device according to claim 1, further comprising an extraction unit that extracts the post-work image of the board product when the judgment unit determines that the board product may be defective.
  3. 　前記抽出部によって抽出された前記作業後画像を表示装置に表示させる案内部を備える請求項２に記載の画像確認装置。 The image confirmation device according to claim 2, further comprising a guide section that causes a display device to display the after-work image extracted by the extraction section.
  4. 　前記案内部は、前記作業前画像および前記作業後画像の前記類似度が高い順に前記表示装置に前記作業後画像を表示させる請求項３に記載の画像確認装置。 The image confirmation device according to claim 3, wherein the guide unit displays the after-work images on the display device in order of the similarity between the before-work image and the after-work image.
  5. 　前記案内部は、前記作業前画像および前記作業後画像の前記類似度の区分に応じて、前記表示装置に表示させる前記作業後画像の表示方法を変更する請求項３または請求項４に記載の画像確認装置。 5. The guide unit according to claim 3, wherein the guide unit changes a display method of the after-work image to be displayed on the display device according to the similarity classification of the before-work image and the after-work image. Image confirmation device.
  6. 　前記案内部は、前記表示装置に表示されている前記作業後画像に基づいて、作業者に前記基板製品の良否を確認させる請求項３～請求項５のいずれか一項に記載の画像確認装置。 The image confirmation device according to any one of claims 3 to 5, wherein the guide section allows a worker to confirm the quality of the board product based on the after-work image displayed on the display device. .
  7. 　前記判断部は、前記作業前画像の画像全体の特徴量と前記作業後画像の画像全体の特徴量とのユークリッド距離が小さいほど、前記作業前画像および前記作業後画像の前記類似度が高いと判断する請求項１～請求項６のいずれか一項に記載の画像確認装置。 The determination unit determines that the smaller the Euclidean distance between the feature amount of the entire image of the before-work image and the feature amount of the entire image of the after-work image, the higher the similarity between the before-work image and the after-work image. The image confirmation device according to any one of claims 1 to 6, which makes a determination.
  8. 　前記特徴量は、画像を構成する複数の画素の輝度、前記基板において前記対象物が設けられている対象領域の重心、前記対象領域の面積、および、前記対象物の形状を示す指標のうちの少なくとも一つである請求項７に記載の画像確認装置。 The feature amount is one of the brightness of a plurality of pixels constituting an image, the center of gravity of a target area in which the target object is provided on the substrate, the area of the target area, and an index indicating the shape of the target object. The image confirmation device according to claim 7, which is at least one image confirmation device.
  9. 　前記判断部は、前記基板製品が不良品である可能性があるか否かを判断する閾値を、作業者によって指定された指定値または予め規定されている固定値に設定する請求項１～請求項８のいずれか一項に記載の画像確認装置。 The determining unit sets a threshold value for determining whether or not there is a possibility that the board product is a defective product to a designated value designated by an operator or a predefined fixed value. The image confirmation device according to any one of Item 8.
  10. 　前記判断部は、前記基板製品が不良品である可能性がないと判断した場合の当該基板製品の前記作業後画像、および、前記基板製品が不良品である可能性があると判断したが作業者によって当該基板製品が良品であることが確認された場合の当該基板製品の前記作業後画像のうちの少なくとも一つを、機械学習の教師画像として使用させる請求項１～請求項９のいずれか一項に記載の画像確認装置。 The determining unit includes the after-work image of the board product when it is determined that there is no possibility that the board product is a defective product, and the after-work image of the board product when it is determined that there is no possibility that the board product is a defective product. Any one of claims 1 to 9, wherein at least one of the post-work images of the board product that has been confirmed by a person to be a good product is used as a teacher image for machine learning. The image confirmation device according to item 1.
  11. 　基板に所定の対基板作業を行う対基板作業機によって対象物が設けられる前の前記基板が撮像されている作業前画像、および、前記対基板作業機によって前記対象物が設けられたことが見込まれる前記基板が撮像されている作業後画像の両方を取得する取得工程と、
    　前記取得工程によって取得された前記作業前画像および前記作業後画像の類似度に基づいて、前記対基板作業機によって生産された基板製品が不良品である可能性の有無を判断する判断工程と、
    を備える画像確認方法。     A pre-work image of the board before a target object is provided on the board by a board-to-board working machine that performs a predetermined board-to-board work, and a pre-work image in which the board is imaged before the target object is provided by the board-to-board working machine, and a pre-work image in which it is expected that the target object is provided by the board-to-board working machine. an acquisition step of acquiring both post-work images in which the substrate is imaged;
    a determination step of determining whether or not there is a possibility that the board product produced by the board-to-board work machine is a defective product based on the degree of similarity between the before-work image and the after-work image acquired in the acquisition step;
    An image confirmation method comprising:
PCT/JP2022/012247 2022-03-17 2022-03-17 Image confirmation device and image confirmation method WO2023175831A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/012247 WO2023175831A1 (en) 2022-03-17 2022-03-17 Image confirmation device and image confirmation method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/012247 WO2023175831A1 (en) 2022-03-17 2022-03-17 Image confirmation device and image confirmation method

Publications (1)

Publication Number Publication Date
WO2023175831A1 true WO2023175831A1 (en) 2023-09-21

Family

ID=88022590

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/012247 WO2023175831A1 (en) 2022-03-17 2022-03-17 Image confirmation device and image confirmation method

Country Status (1)

Country Link
WO (1) WO2023175831A1 (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1117400A (en) * 1997-06-23 1999-01-22 Oki Electric Ind Co Ltd Packaging part/inspecting device
JP2009192282A (en) * 2008-02-13 2009-08-27 Ckd Corp Solder printing inspection apparatus and component mounting system
WO2011074183A1 (en) * 2009-12-16 2011-06-23 株式会社日立ハイテクノロジーズ Defect observation method and defect observation device
JP2012013644A (en) * 2010-07-05 2012-01-19 Sharp Corp Macro inspection method and inspection system, program for having computer execute the macro inspection method, and computer readable recording medium recording the program
JP2018096908A (en) * 2016-12-15 2018-06-21 大日本印刷株式会社 Inspection device and inspection method
WO2020187473A1 (en) * 2019-03-20 2020-09-24 Asml Netherlands B.V. A substrate container, a lithographic apparatus and a method using a lithographic apparatus
JP2021039476A (en) * 2019-09-02 2021-03-11 株式会社東芝 Defect inspection device, defect inspection method, and program
JP2021140739A (en) * 2020-02-28 2021-09-16 株式会社Pros Cons Program, method for creating learned model, information processing method and information processing device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1117400A (en) * 1997-06-23 1999-01-22 Oki Electric Ind Co Ltd Packaging part/inspecting device
JP2009192282A (en) * 2008-02-13 2009-08-27 Ckd Corp Solder printing inspection apparatus and component mounting system
WO2011074183A1 (en) * 2009-12-16 2011-06-23 株式会社日立ハイテクノロジーズ Defect observation method and defect observation device
JP2012013644A (en) * 2010-07-05 2012-01-19 Sharp Corp Macro inspection method and inspection system, program for having computer execute the macro inspection method, and computer readable recording medium recording the program
JP2018096908A (en) * 2016-12-15 2018-06-21 大日本印刷株式会社 Inspection device and inspection method
WO2020187473A1 (en) * 2019-03-20 2020-09-24 Asml Netherlands B.V. A substrate container, a lithographic apparatus and a method using a lithographic apparatus
JP2021039476A (en) * 2019-09-02 2021-03-11 株式会社東芝 Defect inspection device, defect inspection method, and program
JP2021140739A (en) * 2020-02-28 2021-09-16 株式会社Pros Cons Program, method for creating learned model, information processing method and information processing device

Similar Documents

Publication Publication Date Title
US8849442B2 (en) Component mounting line and component mounting method
KR101337881B1 (en) Method for inspecting and generating job data of pcb inspection system
JP4767995B2 (en) Component mounting method, component mounting machine, mounting condition determining method, mounting condition determining apparatus, and program
JP4629584B2 (en) Mounting system and electronic component mounting method
JP2014038045A (en) Inspection device, illumination, inspection method, program and substrate producing method
US20230262903A1 (en) Inspection and production of printed circuit board assemblies
KR20030081176A (en) Control method for mounting parts, mounting tester and mounting system
JP6037580B2 (en) Component mounter
JP2012160627A (en) Substrate processing apparatus
JP2013247228A (en) Substrate inspection method, inspection program, and inspection device
WO2023175831A1 (en) Image confirmation device and image confirmation method
CN207663008U (en) A kind of full-automatic wiring board check machine
US20220151120A1 (en) Correction amount calculating device and correction amount calculating method
WO2023162142A1 (en) Image confirmation device and image confirmation method
JP2004235582A (en) Method for inspection of mounting error, and substrate inspection apparatus using this method
KR101379324B1 (en) Defect position display apparatus of printed circuit board
JP4388423B2 (en) Electronic component mounting device
US20220404816A1 (en) Operation state display device and operation state display method
WO2017072908A1 (en) Base plate position searching device and component mounting machine
JP4263519B2 (en) Inspection method of automatic component mounting equipment
JP7349596B2 (en) Substrate processing system
KR20170012911A (en) PCB Inspection device with AOI
JP7473735B2 (en) Foreign object detection device and foreign object detection method
WO2024033961A1 (en) Foreign matter detection device and foreign matter detection method
US20230110169A1 (en) Data management device and data management method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22932107

Country of ref document: EP

Kind code of ref document: A1