JP2024016646A - Secondary appearance inspection equipment, appearance inspection system, and secondary appearance inspection method - Google Patents

Abstract

An object of the present invention is to suppress the occurrence of over-determination when a machine performs a secondary determination on an object. A primary determination result R indicating a defective area An determined as defective in the primary determination among a plurality of inspection areas A(N) is obtained (step S201). The degree of abnormality indicating the abnormality included in the solder image Is of the solder S (object) in the defective area An indicated by the primary determination result R, that is, the degree of abnormality of the defective area image In, is calculated, and based on the abnormality degree, the solder S The state of is determined (secondary determination). [Selection diagram] Figure 5

Description

The present invention further performs a secondary determination to determine the condition of an object that has been determined to be defective by a primary determination of the condition of the object in each of a plurality of inspection areas. This invention relates to technology for accurately determining the state of an object.

2. Description of the Related Art Appearance inspection devices are known that determine the condition of solder that joins a component to a board based on a captured image of the solder. In such a visual inspection apparatus, in order to determine the state of solder in detail, determination may be performed for each of a plurality of different inspection areas. For example, three inspection areas with different distances from the electrodes of the component are set. In the inspection area closest to the electrode, it is determined whether there is a defective solder shape (unsoldered) where the solder is recessed in front of the electrode of the component. In the inspection area second closest to the electrode, it is determined whether there is a solder shortage defect (small solder) in which the amount of solder is insufficient. In addition, in the inspection area third closest to the electrode, it is determined whether there is a metal foil exposure defect (red eye) in which metal foil (copper foil) that should be covered with solder is exposed.

However, with such a visual inspection device, there are cases where the product is incorrectly determined to be defective even though the product is actually good. Therefore, the user appropriately performs a secondary determination to supplement the determination (primary determination) made by the visual inspection device. That is, in the primary determination of the visual inspection apparatus, if any of the plurality of inspection areas is determined to be defective, an image of the solder that is the target of the primary determination is displayed on the display. This allows the user to make a final judgment based on visual observation on the display (secondary judgment).

Japanese Patent Application Publication No. 2010-8159

By the way, it is possible to save labor by having a machine perform the secondary determination that was previously performed by the user. For example, Patent Document 1 proposes a neural network that determines the state of an object based on abnormalities in pixels included in an image of the object. However, when a machine performs a secondary judgment to supplement the primary judgment made by the visual inspection device, there are the following problems.

In other words, the purpose of the above-mentioned secondary determination by the user is to confirm the state of the inspection area that is determined to be defective in the primary determination among the plurality of inspection areas. On the other hand, when an image of an object includes multiple inspection areas, as a result of the machine calculating the abnormalities included in the image, the inspection areas that were not determined to be defective in the first judgment may be judged as defective. Judgment may occur.

This invention has been made in view of the above-mentioned problems, and it further determines the condition of an object that has been determined to be defective by a primary judgment that determines the condition of the object in each of a plurality of inspection areas. It is an object of the present invention to suppress the occurrence of over-determination when performing secondary determination by a machine.

The secondary appearance inspection device according to the present invention indicates a defective area determined to be defective among the plurality of inspection areas in the primary judgment of determining the state of the object in each of the plurality of inspection areas that are different from each other. an information acquisition unit that acquires defective area information; an abnormality data calculation unit that calculates defective area abnormality data indicating an abnormality included in the image of the object in the defective area indicated by the defective area information; and a secondary determination execution unit that executes a secondary determination to determine the state of the object.

The visual inspection system according to the present invention executes a primary judgment for determining the state of an object in each of a plurality of different inspection areas, and determines which of the plurality of inspection areas is defective in the primary judgment. The present invention includes a primary visual inspection device that outputs defective area information indicating a defective area that has been inspected, and the above-mentioned secondary visual inspection device, and an information acquisition unit acquires defective area information output from the primary visual inspection device.

The secondary appearance inspection method according to the present invention indicates a defective area that is determined to be defective among a plurality of inspection areas in a primary judgment of determining the state of an object in each of a plurality of inspection areas that are different from each other. A step of acquiring defective area information, a step of calculating defective area abnormality data indicating an abnormality included in the image of the object in the defective area indicated by the defective area information, and determining the state of the object based on the defective area abnormality data. and performing a secondary determination.

In the present invention (secondary visual inspection device, visual inspection system, and secondary visual inspection method) configured as described above, defective area information indicating a defective area determined to be defective in the first judgment among a plurality of inspection areas is acquired. be done. Then, defective area abnormality data indicating an abnormality included in the image of the object in the defective area indicated by the defective area information is calculated, and the state of the object is determined based on the defective area abnormality data (secondary determination). . In other words, the inspection areas that are not determined to be defective in the primary determination are excluded from the criteria for determining the state of the object in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which an inspection area that was not determined to be defective in the primary determination is determined to be defective.

The abnormality data calculation unit also includes an image acquisition unit that acquires an object image that is an image of the object in an imaging area that includes a plurality of inspection areas, which was imaged in the primary determination, and an image acquisition unit that acquires an object image corresponding to the plurality of inspection areas. It has a data calculation unit that calculates defective area abnormal data using a plurality of machine learning models provided and a corresponding model corresponding to the defective area among the plurality of machine learning models. Each outputs inspection area abnormality data indicating an abnormality included in an image captured in the corresponding inspection area, and the data calculation unit extracts an image included in the defective area from the target object image as a defective area image. The secondary visual inspection device may be configured to calculate the inspection area abnormality data that the corresponding model outputs regarding the defective area image as the defective area abnormality data. In such a configuration, an inspection area that is not determined to be defective in the primary determination is excluded from the criteria for determining the state of the object in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which an inspection area that was not determined to be defective in the primary determination is determined to be defective.

The abnormal data calculation unit also includes an image acquisition unit that acquires an object image that is an image of the object in an imaging area that includes a plurality of inspection areas that was imaged in the primary determination, and an object image that is included in the image that is captured in the imaging area. Defective area abnormal data is calculated by extracting data corresponding to the defective area from the machine learning model that outputs imaging area abnormality data indicating the abnormality that occurs, and the imaging area abnormality data output by the machine learning model for the target image. The secondary visual inspection device may be configured to include a data calculation unit that performs the following steps. In such a configuration, an inspection area that is not determined to be defective in the primary determination is excluded from the criteria for determining the state of the object in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which an inspection area that was not determined to be defective in the primary determination is determined to be defective. Furthermore, instead of providing a plurality of machine learning models corresponding to a plurality of inspection areas, it is sufficient to provide one machine learning model for an imaging area including a plurality of inspection areas. Therefore, the user does not need to manage multiple machine learning models, making it possible to reduce the user's management burden.

Further, the secondary visual inspection device may be configured to further include a user interface and a display control unit that causes the user interface to display visualized abnormality data including at least defective area abnormality data. With this configuration, the user can visually confirm the defective area abnormal data included in the visualized abnormal data.

In addition, there is an automatic judgment mode in which the quality of the object is determined by a secondary judgment performed by a secondary judgment execution unit, and an automatic judgment mode in which the quality of the object is judged based on operations performed by the user on the user interface while displaying abnormal data on the user interface. The secondary visual inspection device may be configured to further include a mode selection unit that selectively executes the manual determination mode. In this configuration, by selecting the automatic determination mode, the burden on the user in the secondary determination can be reduced, and by selecting the manual determination mode, the secondary determination can be performed visually by the user.

Moreover, the mode selection unit may configure the secondary visual inspection apparatus to execute one mode selected by a user's operation on the user interface out of the automatic determination mode and the manual determination mode. With such a configuration, it is possible to execute a mode according to a user's request among the automatic determination mode and the manual determination mode.

As described above, according to the present invention, for an object that is determined to be defective by the primary determination that determines the condition of the object in each of a plurality of inspection areas, the secondary inspection that further determines the condition of the object When the determination is performed by a machine, it is possible to suppress the occurrence of excessive determination.

1 is a block diagram showing an example of a visual inspection system according to the present invention. 2 is a flowchart showing an example of a primary determination performed by the primary visual inspection apparatus of FIG. 1. FIG. FIG. 3 is a diagram schematically showing operations performed in the primary determination of FIG. 2; FIG. 1 is a block diagram showing a first example of a secondary visual inspection device according to the present invention. 5 is a flowchart showing a secondary determination performed by the first example of the secondary visual inspection device of FIG. 4. FIG. FIG. 6 is a diagram schematically showing operations performed in the secondary determination of FIG. 5; FIG. 2 is a block diagram showing a second example of a secondary appearance inspection device according to the present invention. 8 is a flowchart showing secondary determination performed by the second example of the secondary visual inspection device of FIG. 7. FIG.

FIG. 1 is a block diagram showing an example of a visual inspection system according to the present invention. The visual inspection system 1 shown in FIG. 1 includes a primary visual inspection device 2 and a secondary visual inspection device 4, and inspects the condition of solder S that joins a component E such as a capacitor, a resistor, or an integrated circuit to a substrate B. The primary appearance inspection device 2 has a light irradiation unit 21 and an imaging camera 22, and while irradiating light from the light irradiation unit 21 to a predetermined imaging area Ac including the solder S that is an object to be inspected, the imaging area Ac By capturing the image with the imaging camera 22, a solder image Is showing the solder S is captured. Further, the primary visual inspection device 2 includes a communication section 28 that communicates with the secondary visual inspection device 4, and a controller 29 that controls the light irradiation section 21, the imaging camera 22, and the communication section 28. The controller 29 obtains a primary determination result R that is the result of determining the quality of the solder S based on the solder image Is. Further, the communication unit 28 transmits the solder image Is and the primary determination result R to the secondary appearance inspection device 4.

As a specific configuration of the primary visual inspection device 2, for example, the device configuration disclosed in Japanese Patent Laid-Open No. 2010-071844 can be adopted. In the device disclosed in the publication, light with different wavelengths (infrared, red, green, and blue) is irradiated onto the solder at different angles, and a camera facing the board captures the light reflected by the solder, creating an image. is obtained. The quality of the solder is then determined based on this image. Note that the specific configuration of the primary visual inspection device 2 is not limited to this example, and for example, the device configuration described in WO2018/163278 can also be adopted.

FIG. 2 is a flowchart showing an example of the primary determination performed by the primary visual inspection apparatus of FIG. 1, and FIG. 3 is a diagram schematically showing the operation performed in the primary determination of FIG. In FIG. 3, only component E is shown, and solder S is not shown.

In the primary determination of FIG. 2, a plurality of mutually different inspection areas A(N) (N=1, 2, 3) shown in FIG. 3 are set. Inspection area A (1) is set to determine whether there is a defective solder shape (unsoldered) where solder S is recessed in front of the electrode of component E, and inspection area A (2) is set to determine whether there is an insufficient amount of solder S. Inspection area A (3) is set to determine whether there is a solder shortage defect (small solder), and the inspection area A (3) is set to determine whether there is a metal foil exposure defect (red eye) where the metal foil (copper foil) that should be covered by solder S is exposed. It is set to determine the

As shown in FIG. 2, in step S101, the controller 29 irradiates a predetermined imaging area Ac including the solder S with light from the light irradiation unit 21 and images the imaging area Ac with the imaging camera 22. A solder image Is is captured. This solder image Is is image data that indicates the brightness of each of a plurality of pixels included in the imaging area Ac. In step S102, the controller 29 resets the count value N for identifying the inspection area A(N) to zero, and in step S103, the controller 29 increments the count value N by one.

Then, the controller 29 determines the quality of the inspection area A(N) based on the image included in the inspection area A(N) among the solder images Is (step S104). Further, the controller 29 stores the determination result (good/bad) for the inspection area A(N) (step S105). In step S106, it is checked whether the count value N has reached the maximum count value Nx (=3). The maximum count value Nx corresponds to the number (=3) of inspection areas A(N). Then, steps S104 and S105 are repeated while incrementing the count value N until the count value N reaches the maximum count value Nx. As a result, determination results are obtained for each of the plurality of inspection areas A(N) (N=1, 2, 3).

In step S107, the controller 29 checks whether there is a defective determination among the determination results for each of the plurality of inspection areas A(N). If there is a defective determination (“YES” in step S107), the controller 29 outputs a primary determination result R indicating a defective area An in which a defective determination has been made among the plurality of inspection areas A(N); The solder image Is captured in step S101 is output to the secondary visual inspection device 4 by the communication unit 28.

FIG. 4 is a block diagram showing a first example of a secondary visual inspection device according to the present invention. The secondary visual inspection device 4 is a computer including a calculation section 41, a storage section 42, a communication section 43, and a UI (User Interface) 44. The calculation unit 41 is a processor such as a CPU (Central Processing Unit), and the storage unit 42 is a storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive). The communication unit 43 communicates with the communication unit 28 of the primary visual inspection device 2, receives, for example, the primary determination result R and solder image Is output from the communication unit 28, and stores them in the storage unit 42. The UI 44 includes input devices such as a mouse and keyboard, and output devices such as a display. Note that the input device and output device of the UI 44 do not need to be configured separately, and may be configured integrally using a touch panel display.

The calculation unit 41 configures an information acquisition unit 411, an image acquisition unit 412, a data calculation unit 413, an automatic determination execution unit 414, a manual determination control unit 415, and a mode selection unit 416 by executing a predetermined secondary determination program. do. The storage unit 42 also stores a plurality of off-machine learning models M(N) provided corresponding to the plurality of inspection areas A(N) (N=1, 2, 3). The out-of-machine learning model M(N) is an inspection area image Ia(N) (FIG. 3), which is an image within the inspection area A(N) corresponding to the out-of-machine learning model M(N) out of the solder image Is. and the abnormality degree of the inspection area image Ia(N), and when the inspection area image Ia(N) is input, the abnormality degree of the inspection area image Ia(N) is output. That is, the ex-machine learning model M(1) outputs the degree of abnormality of the inspection area A(1), and can be used to determine the presence or absence of defective solder shape (unsoldered). The ex-machine learning model M(2) outputs the degree of abnormality of the inspection area A(2), and can be used to determine the presence or absence of a solder shortage defect (small solder). Further, the ex-machine learning model M(3) outputs the degree of abnormality of the inspection area A(3), and can be used to determine the presence or absence of poor metal foil exposure (red eye).

Note that a well-known technique can be adopted as a learning method for a machine learning model that outputs the degree of abnormality of an image. For example, a feature vector of the intermediate layer of an existing machine learning model obtained when a good product image is input to the machine learning model is stored as a reference. Then, the distance between the feature vector obtained when the image captured by the primary visual inspection device 2 is input to this machine learning model and the reference feature vector can be calculated as the degree of abnormality. This degree of abnormality can be calculated for each of a plurality of pixels that make up the image.

FIG. 5 is a flowchart showing the secondary judgment performed by the first example of the secondary visual inspection apparatus shown in FIG. 4, and FIG. 6 is a diagram schematically showing the operation performed in the secondary judgment shown in FIG. . Note that in FIG. 6, only the component E is shown, and the solder S is not shown.

As shown in FIG. 5, the information acquisition unit 411 acquires the primary judgment result R received by the communication unit 43 from the primary visual inspection device 2 (step S201), and the image acquisition unit 412 The solder image Is received from the device 2 is acquired (step S202). As illustrated in the column of step S202 in FIG. 6, this solder image Is shows solder S imaged in an imaging area Ac that includes a plurality of inspection areas A(N) set in the primary determination. Note that the order of execution of steps S201 and S202 is not limited to this example.

In step S203, the data calculation unit 413 extracts a defective area image In, which is an image in a defective area An indicated by the primary determination result R, from the solder image Is. In the example shown in the column of step S203 in FIG. 6, among the plurality of inspection areas A(1), A(2), and A(3), inspection area A(2) corresponds to the defective area An, and the defective area A defective area image In of An is extracted from the solder image Is. Furthermore, the data calculation unit 413 calculates which of the out-of-machine learning models M(1), M(2), and M(3) corresponding to the plurality of inspection areas A(1), A(2), and A(3), respectively. , the off-machine learning model M(2) corresponding to the defective area An (ie, inspection area A(2)) is selected (step S204). Then, the data calculation unit 413 calculates the degree of abnormality of the defective area image In by inputting the defective area image In to the ex-machine learning model M(2) selected in step S204.

In step S206, the mode selection unit 416 selects whether the automatic determination execution unit 414 executes an automatic determination mode in which secondary determination is automatically performed, or based on the result of the manual determination control unit 415 determining the user's input operation to the UI 44. Check whether to execute manual judgment mode that performs secondary judgment. Specifically, the user can set, in the mode selection unit 416, which mode to execute, automatic determination mode or manual determination mode, by operating the UI 44.

If the automatic determination mode is set (“YES” in step S206), the automatic determination execution unit 414 determines the quality of the solder S based on the degree of abnormality of the defective area image In calculated in step S205. Determination is made (step S207). Specifically, when the degree of abnormality of each pixel constituting the defective area image In is below the threshold value ("YES" in step S207), the state of the solder S shown in the solder image Is is good. The automatic determination execution unit 414 obtains a non-defective determination indicating that the product is a good product as a result of the secondary determination (step S208). On the other hand, if there is a pixel whose degree of abnormality is greater than the threshold value among the pixels constituting the defective area image In ("NO" in step S207), the state of the solder S shown in the solder image Is is defective. A defective determination indicating that there is a defect is obtained as a result of the secondary determination by the automatic determination execution unit 414 (step S209). The results of the secondary determination obtained in steps S208 and S209 are displayed on the display of the UI 44, for example.

If the manual determination mode is set (“NO” in step S206), the manual determination control unit 415 displays an abnormality degree map (heat map) that visualizes the abnormality degree of the defective area image In on the display of the UI 44. (Step S210). Then, the manual determination control unit 415 determines whether the user's determination input into the UI 44 is good (step S211). If the user's judgment input into the UI 44 is good (“YES” in step S211), the non-defective judgment indicating that the state of the solder S shown in the solder image Is is good is the secondary judgment. The result is obtained by the manual determination control unit 415 (step S208). On the other hand, if the user's judgment input into the UI 44 is bad (“NO” in step S211), the bad judgment indicating that the state of the solder S shown in the solder image Is is bad is secondary. The manual determination control unit 415 obtains the determination result (step S209). The results of the secondary determination obtained in steps S208 and S209 are displayed on the display of the UI 44, for example.

In the embodiment described above, the primary determination result R (defective area information) indicating the defective area An (inspection area A(2)) that is determined to be defective in the primary determination among the plurality of inspection areas A(N) is obtained. (Step S201). Then, the degree of abnormality (defect area abnormality data) indicating the abnormality included in the solder image Is of the solder S (object) in the defective area An indicated by the primary determination result R, that is, the degree of abnormality of the defective area image In is calculated ( In step S205) and steps S206 to S209, the state of the solder S is determined based on the degree of abnormality (secondary determination). That is, the inspection areas A(1) and A(3), which were not determined to be defective in the primary determination, are excluded from the criteria for determining the state of the solder S in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which inspection areas A(1) and A(3), which were not determined to be defective in the primary determination, are determined to be defective.

In particular, the information acquisition unit 411 acquires the solder image Is (object image), which is an image of the solder S in the imaging area Ac including the plurality of inspection areas A(N), which was imaged in the primary determination (step S202). . On the other hand, a plurality of ex-machine learning models M(N) respectively provided corresponding to a plurality of inspection areas A(N) are stored in the storage unit 42. Then, using the ex-machine learning model M(2) (corresponding model) corresponding to the defective area image In (inspection area A(2)) among the plurality of ex-machine learning models M(N), the defective area image In The degree of abnormality (defective area abnormality data) is calculated by the data calculation unit 413 (steps S203 to S205). In other words, each of the plurality of ex-machine learning models M(N) outputs the degree of abnormality (inspection area abnormality data) included in the inspection area image Ia(N) captured in the corresponding inspection area A(N). It is composed of Therefore, the data calculation unit 413 extracts the image included in the defective area An (inspection area A(2)) from the solder image Is as the defective area image In (step S203), and extracts the image included in the defective area An (inspection area A(2)) as the defective area image In. The degree of abnormality (inspection area abnormality data) output by the external learning model M(2) (corresponding model) for the defective area image In is calculated as the degree of abnormality (defective area abnormality data) of the defective area image In (steps S204, S205 ). In this configuration, the inspection areas A(1) and A(3), which were not determined to be defective in the primary determination, are excluded from the criteria for determining the state of the solder S in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which inspection areas A(1) and A(3), which were not determined to be defective in the primary determination, are determined to be defective.

Further, the UI 44 is provided with a manual determination control unit 415 (display control unit) that causes the UI 44 to display an abnormality degree map (visualized abnormality data) that includes at least the abnormality degree (defective area abnormality data) in the defective area image In. ing. With this configuration, the user can visually check the degree of abnormality in the defective area image In included in the degree of abnormality map.

In addition, there is an automatic judgment mode (steps S207, S208, S209) in which the quality of the solder S is judged by a secondary judgment by the automatic judgment execution unit 414 (secondary judgment execution unit), and an automatic judgment mode (steps S207, S208, S209) in which the user can A mode selection unit 416 is provided that selectively executes a manual determination mode (steps SS210, S211, S208, and S209) in which the quality of the solder S is determined based on operations performed on the UI 44. In this configuration, by selecting the automatic determination mode, the burden on the user in the secondary determination can be reduced, and by selecting the manual determination mode, the secondary determination can be performed visually by the user.

Furthermore, the mode selection unit 416 executes one mode selected by the user's operation on the UI 44 out of the automatic determination mode and the manual determination mode (step S206). With such a configuration, it is possible to execute a mode according to a user's request among the automatic determination mode and the manual determination mode.

FIG. 7 is a block diagram showing a second example of the secondary visual inspection device according to the present invention, and FIG. 8 is a flowchart showing the secondary determination performed by the second example of the secondary visual inspection device of FIG. 7. . In the following, the explanation will focus on the differences from the first example, and the common points with the first example will be given corresponding symbols and the explanation will be omitted as appropriate. However, it goes without saying that by providing the same configuration as the first example, the same effects as the first example can be achieved.

In the storage unit 42 of the primary visual inspection apparatus 2 shown in FIG. 7, an ex-machine learning model Mc is stored. This ex-machine learning model Mc is provided corresponding to an imaging area Ac including a plurality of inspection areas A(N). In other words, the external learning model Mc has already learned the relationship between the solder image Is captured in the imaging area Ac and the degree of abnormality of the solder image Is, and when the solder image Is is input, the solder image Is becomes Output the degree of abnormality.

As shown in FIG. 8, the information acquisition unit 411 acquires the primary judgment result R received by the communication unit 43 from the primary visual inspection device 2 (step S301), and the image acquisition unit 412 Acquire the solder image Is received from the device 2 (step S302) Note that the order of execution of steps S301 and S302 is not limited to this example.

In step S303, the data calculation unit 413 calculates the degree of abnormality of the solder image Is by inputting the solder image Is to the ex-machine learning model Mc. Further, the data calculation unit 413 extracts the abnormality degree in the defective area An indicated by the primary determination result R from among the abnormality degrees calculated in step S303 (step S304).

In step S306, the mode selection unit 416 confirms whether automatic determination mode or manual determination mode is set. If the automatic determination mode is set (“YES” in step S306), the automatic determination execution unit 414 determines the quality of the solder S based on the degree of abnormality in the defective area An extracted in step S304. (Step S307). In other words, if the degree of abnormality of each pixel included in the defective area An is less than or equal to the threshold value ("YES" in step S307), it indicates that the solder S shown in the solder image Is is in good condition. The determination is obtained by the automatic determination execution unit 414 as a result of the secondary determination (step S308). On the other hand, if there is a pixel whose degree of abnormality is greater than the threshold value among the pixels included in the defective area An ("NO" in step S307), the state of the solder S shown in the solder image Is is defective. The automatic determination execution unit 414 obtains a defective determination indicating this as a result of the secondary determination (step S309). The results of the secondary determination obtained in steps S308 and S309 are displayed on the display of the UI 44, for example.

If the manual determination mode is set (“NO” in step S306), steps S310, S311, S308, and S309 are executed in the same manner as steps S210, S211, S208, and S209 in the first example. Ru.

In the embodiment described above, the primary determination result R (defective area information) indicating the defective area An (inspection area A(2)) that is determined to be defective in the primary determination among the plurality of inspection areas A(N) is obtained. (Step S301). Then, the degree of image abnormality (defective area abnormality data) in the defective area An indicated by the primary determination result R in the solder image Is is calculated (steps S303, S304). Then, in steps S307 to S309, the state of the solder S is determined based on the degree of abnormality of the image in the defective area An (defective area abnormality data) (secondary determination). That is, the inspection areas A(1) and S(3), which were not determined to be defective in the primary determination, are excluded from the criteria for determining the state of the solder S in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which inspection areas A(1) and S(3), which were not determined to be defective in the primary determination, are determined to be defective.

In particular, the image acquisition unit 412 acquires the solder image Is (object image), which is an image of the solder S in the imaging area Ac including the plurality of inspection areas A(N), which was imaged in the primary determination. On the other hand, an extra-machine learning model Mc that outputs the degree of abnormality of the image captured in the imaging area Ac (imaging area abnormality data) is stored in the storage unit 42. Then, the data calculation unit 413 extracts the degree of abnormality (data) corresponding to the defective area An from the degree of abnormality (imaging area abnormality data) output by the ex-machine learning model Mc for the solder image Is. The degree of abnormality (defective area abnormal data) is calculated (steps S303, S304). In this configuration, the inspection areas A(1) and A(3), which were not determined to be defective in the primary determination, are excluded from the criteria for determining the state of the solder S in the secondary determination. As a result, it is possible to suppress the occurrence of overdetermination in which inspection areas A(1) and A(3), which were not determined to be defective in the primary determination, are determined to be defective. Moreover, instead of providing multiple out-of-machine learning models M(1), M(2), and M(3) corresponding to multiple inspection areas A(1), A(2), and A(3), It is sufficient to provide one ex-machine learning model Mc for the imaging area Ac including the plurality of inspection areas A(1), A(2), and A(3). Therefore, the user does not need to manage multiple machine learning models M(1), M(2), and M(3), making it possible to reduce the user's management burden.

As explained above, in this embodiment, the visual inspection system 1 corresponds to an example of the "visual inspection system" of the present invention, and the primary visual inspection device 2 corresponds to an example of the "primary visual inspection device" of the present invention. , the secondary visual inspection device 4 corresponds to an example of the "secondary visual inspection device" of the present invention, the information acquisition section 411 corresponds to an example of the "information acquisition section" of the present invention, and the image acquisition section 412 corresponds to an example of the "information acquisition section" of the present invention. The learning models M(N), Mc, and the data calculation unit 413 work together to function as the “abnormal data calculation unit” of the present invention, and the automatic determination execution unit 414 is an example of the “secondary determination execution unit” of the present invention. Correspondingly, the inspection area A(N) corresponds to an example of the "inspection area" of the present invention, the defective area An corresponds to an example of the "defective area" of the present invention, and the degree of abnormality in the defective area An of the solder image Is is corresponds to an example of the "defective area abnormal data" of the present invention, the primary determination result R corresponds to an example of the "defective area information" of the present invention, and the solder S corresponds to an example of the "object" of the present invention. .

Note that the present invention is not limited to the above-described embodiments, and various changes can be made to the above-described embodiments without departing from the spirit thereof. For example, the specific setting manner of the plurality of inspection areas A(N) in the primary determination is not limited to the above example. Therefore, two inspection areas A(N) may be set among inspection areas A(1), A(2), and A(3), or two inspection areas A(N) may be set among inspection areas A(1), A(2), and A(3). An inspection area different from A(3) may be set.

Further, the object of the primary determination and the secondary determination is not limited to the solder S described above.

Moreover, it is not necessarily necessary to configure the primary visual inspection device 2 and the secondary visual inspection device 4 separately, and they may be configured integrally.

Further, the manual determination mode described above is not essential, and it is not necessary to provide the manual determination mode in the secondary determination.

1... Appearance inspection system 2... Primary appearance inspection device 4... Secondary appearance inspection device 411... Information acquisition section 412... Image acquisition section (abnormal data calculation section)
413...Data calculation unit (abnormal data calculation unit)
414...Automatic judgment execution unit (secondary judgment execution unit)
A(N)...Inspection area An...Defective area Is...Solder image M(N)...External learning model (abnormal data calculation unit)
Mc…External learning model (abnormal data calculation unit)
R…Primary judgment result (defective area information)
S...Solder (object)

Claims (8)

an information acquisition unit that acquires defective area information indicating a defective area determined to be defective among the plurality of inspection areas in a primary judgment of determining the state of an object in each of the plurality of inspection areas that are different from each other; ,
an abnormality data calculation unit that calculates defective area abnormality data indicating an abnormality included in the image of the object in the defective area indicated by the defective area information;
A secondary visual inspection device comprising: a secondary determination execution unit that executes a secondary determination to determine the state of the object based on the defective area abnormality data. The abnormal data calculation unit includes:
an image acquisition unit that acquires an object image that is an image of the object in an imaging area that includes the plurality of inspection areas that was imaged in the primary determination;
a plurality of machine learning models respectively provided corresponding to the plurality of inspection areas;
a data calculation unit that calculates the defective area abnormal data using a corresponding model corresponding to the defective area among the plurality of machine learning models;
Each of the plurality of machine learning models outputs inspection area abnormality data indicating an abnormality included in an image captured in the corresponding inspection area,
The data calculation unit extracts an image included in the defective area from the object image as a defective area image, and converts the inspection area abnormality data outputted by the corresponding model regarding the defective area image into the defective area abnormality. The secondary appearance inspection device according to claim 1, wherein the secondary appearance inspection device is calculated as data. The abnormal data calculation unit includes:
an image acquisition unit that acquires an object image that is an image of the object in an imaging area that includes the plurality of inspection areas that was imaged in the primary determination;
a machine learning model that outputs imaging region abnormality data indicating abnormalities included in images captured in the imaging region;
2. The method according to claim 1, further comprising a data calculation unit that calculates the defective area abnormal data by extracting data corresponding to the defective area from the imaging area abnormal data output by the machine learning model for the object image. The secondary appearance inspection device described. a user interface;
The secondary appearance inspection apparatus according to claim 1, further comprising a display control unit that causes the user interface to display visualized abnormality data including at least the defective area abnormality data. an automatic judgment mode in which the quality of the object is determined by the secondary judgment by the secondary judgment execution unit; and an automatic judgment mode in which the object is judged to be good or bad based on the operation performed by the user on the user interface while displaying the abnormality data on the user interface. 5. The secondary visual inspection apparatus according to claim 4, further comprising a mode selection unit that selectively executes a manual determination mode for determining the quality of the product. The secondary visual inspection apparatus according to claim 5, wherein the mode selection unit executes one mode selected by a user's operation on the user interface from among the automatic determination mode and the manual determination mode. Defective area information indicating a defective area determined to be defective among the plurality of inspection areas in the primary judgment by performing a primary judgment to determine the state of the object in each of the plurality of inspection areas that are different from each other. a primary visual inspection device that outputs
A secondary appearance inspection device according to any one of claims 1 to 6,
The information acquisition unit is a visual inspection system that acquires the defective area information output from the primary visual inspection device. a step of obtaining defective area information indicating a defective area determined to be defective among the plurality of inspection areas in a primary determination of determining the state of the object in each of the plurality of inspection areas that are different from each other;
calculating defective area abnormality data indicating an abnormality included in the image of the object in the defective area indicated by the defective area information;
A secondary appearance inspection method comprising: executing a secondary determination to determine the state of the object based on the defective area abnormality data.

Images