JP2023138330A - Information processor and information processing method - Google Patents

Abstract

To provide an information processor and an information processing method that enable detection of a defect in a manufactured product before components are joined to a base by reflow in manufacturing a product by soldering.SOLUTION: In an information processor including an image processing unit and a determination unit, the determination unit uses a machine learning model that outputs an inspection result of an inspection after reflow from input of image data based on an image before reflow, and determines pass or fail in an inspection scheduled to be performed after reflow from image data based on the image before reflow acquired in real time.SELECTED DRAWING: Figure 3

Description

Embodiments of the present invention relate to an information processing device and an information processing method.

When joining components to a substrate such as a printed wiring board or a printed circuit board by soldering, normal temperature solder and components are mounted on the substrate, which is a base, in this order. Then, by performing reflow with the solder and components mounted on the board, the solder is melted and the components are joined to the board by soldering. In addition, when a printed circuit board with components joined to the board by soldering is manufactured as a product, after reflow, the manufactured printed circuit board is subjected to visual inspections, electrical tests, etc., and defective products are marketed. This prevents leakage to.

As mentioned above, when manufacturing products such as printed circuit boards by soldering, it is important to reduce product There is a need to be able to detect defects before reflow. That is, it is required to be able to detect defects in manufactured products before parts are bonded to a base by reflow.

Japanese Patent Application Publication No. 2020-161749 Japanese Patent Application Publication No. 2020-161750 JP 2019-110257 Publication

The problem to be solved by the present invention is to provide an information processing device and an information processing method that can detect defects in manufactured products before parts are joined to a base by reflow in manufacturing products by soldering. It is about providing.

According to the embodiment, an information processing device related to soldering of components to a base is provided. The information processing device includes a determination unit, and the determination unit inputs image data based on images before reflow and uses a machine learning model to output inspection results in an inspection after reflow. Based on the image data based on the image before reflow, it is determined whether the test is good or bad in an inspection scheduled to be performed after reflow.

FIG. 1 is a schematic diagram showing a manufacturing system according to the first embodiment as an example of a manufacturing system that manufactures printed circuit boards as products. FIG. 2 is a block diagram schematically showing the configuration of the information processing device of the manufacturing system according to the first embodiment. FIG. 3 is a schematic diagram illustrating an example of processing by the image processing unit and determination unit of the information processing device according to the first embodiment. FIG. 4 is a flowchart schematically showing an example of image processing performed by the image processing unit of the information processing apparatus according to the first embodiment. FIG. 5 is a flowchart schematically showing another example of image processing, different from that shown in FIG. 4, performed by the image processing unit of the information processing apparatus according to the first embodiment. FIG. 6 is a schematic diagram illustrating an example of a machine learning model used for determination by the determination unit of the information processing apparatus according to the first embodiment. FIG. 7 is a flowchart schematically showing an example of a machine learning model generation process performed by the learning model generation unit of the information processing apparatus according to the first embodiment. FIG. 8 is a block diagram schematically showing the configuration of an information processing device of a manufacturing system according to the second embodiment. FIG. 9 is a flowchart schematically showing an example of a defect cause identification process performed by the cause identification unit of the information processing apparatus according to the second embodiment. FIG. 10 is a schematic diagram illustrating an example of a process for extracting a node having a high degree of contribution to a defect determination, which is performed by the cause identification unit of the information processing apparatus according to the second embodiment. FIG. 11 is a schematic diagram illustrating an example of a notification process of a portion of image data identified as a cause of a defect, which is performed by the cause identification unit of the information processing apparatus according to the second embodiment. FIG. 12 is a schematic diagram showing a manufacturing system according to the third embodiment as an example of a manufacturing system that manufactures semiconductor devices as products. FIG. 13 is a schematic diagram illustrating an example of image data generated by image processing by the image processing unit of the information processing apparatus according to the third embodiment.

Embodiments and the like will be described below with reference to the drawings.

(First embodiment)
First, a first embodiment will be described as an example of the embodiment. FIG. 1 shows a manufacturing system 1 according to a first embodiment as an example of a manufacturing system that manufactures printed circuit boards as products. As shown in FIG. 1 and the like, the manufacturing system 1 includes a solder printing device 2, a component mounting device 3, a reflow device 4, an inspection device 5, and an information processing device 6. In the manufacturing system 1, a manufacturing line for printed circuit boards as products is formed, and in the manufacturing line, a solder printing device 2, a component mounting device 3, a reflow device 4, and an inspection device 5 are arranged in this order from the upstream side.

In the manufacturing line, a substrate serving as a base body is carried into a solder printing device 2 . The board may be a printed wiring board, or may be a printed circuit board (printed circuit board) in which parts such as chip parts are already attached to the printed wiring board. The solder printing device 2 mounts solder on the substrate by, for example, printing solder on pads or lands formed on the surface of the substrate. In one example, solder may be applied to a printed circuit board by printing the solder onto the surface of a component already attached to the printed circuit board. In the solder printing device 2, solder at room temperature is mounted on the board, and the solder is mounted on the board in an unmolten state. Note that the method for mounting the solder on the substrate is not limited to printing. In one example, solder may be applied to the substrate by either dispensing solder, applying solder sheets, or the like.

A board on which solder is mounted is carried into the component mounting device 3. The component mounting device 3 mounts a new component to be attached to the board. Examples of the components mounted on the board include chip components and IC packages. When the component is mounted, the solder is sandwiched between the board and the component. A board on which solder and components are mounted is carried into the reflow apparatus 4 . The reflow device 4 performs soldering by reflow. By reflowing, the solder mounted on the board is melted, and the newly mounted component is bonded to the board. As a result, a printed circuit board with the mounted components attached to the board is formed. In one example, a newly mounted component is bonded to a pad, land, or the like formed on the surface of a substrate by soldering. In another example, a newly mounted component is joined to a component that has already been mounted on the board by soldering.

A printed circuit board manufactured as a product, that is, a printed circuit board with components joined to the board by reflowing in the reflow device 4 is carried into the inspection device 5 . The inspection device 5 inspects the manufactured printed circuit board and determines whether the manufactured printed circuit board is good or defective. Only printed circuit boards that are determined to be good through the inspection by the inspection device 5 are released onto the market as distributed products. For example, the inspection device 5 performs a visual inspection, an electrical test, etc. on the manufactured printed circuit board. In the visual inspection, an image of a printed circuit board with components joined to the board by reflow is acquired by photographing or the like, and based on the acquired image of the printed circuit board, it is determined whether the manufactured printed circuit board is good or defective. In the electrical test, electricity is passed through the manufactured printed circuit board and the amount of electricity is measured by a tester, thereby determining whether the printed circuit board is good or defective. The inspection device 5 transmits information indicating the test results of the actually performed test to the information processing device 6.

Further, the manufacturing system 1 is provided with photographing devices 11 to 13. Each of the photographing devices 11 to 13 is, for example, a camera or a video camera. The photographing device 11 is arranged upstream of the solder printing device 2 and photographs an image of only the board. The photographing device 12 is arranged between the solder printing device 2 and the component mounting device 3, and photographs an image of only solder being mounted on the board. The photographing device 13 is arranged between the component mounting device 3 and the reflow device 4, and photographs an image of the solder and components mounted on the board. As a result, in the manufacturing system 1, the imaging devices 11 to 13 capture three types of images: an image of only the board, an image of only solder mounted on the board, and an image of solder and components mounted on the board. The image is taken as an image before reflow by the device 4.

Each of the photographing devices 11 to 13 transmits a photographed image (image data) to the information processing device 6. Therefore, the information processing device 6 acquires the three types of images described above as images before reflow. Further, each of the photographing devices 11 to 13 performs photographing in real time each time one substrate is carried into the manufacturing line as a base body. For this reason, the information processing device 6 acquires in real time three types of images taken by the imaging devices 11 to 13 before reflow, every time one board is carried into the manufacturing line.

FIG. 2 shows an example of the configuration of the information processing device 6 according to this embodiment. As shown in FIG. 2, the information processing device 6 includes a processing execution section 21, a storage section 22, a communication interface 23, and a user interface 25. The processing execution section 21 includes an image processing section 31, a determination section 32, a learning model generation section 33, and a learning model updating section 35. The image processing section 31, the determination section 32, the learning model generation section 33, and the learning model updating section 35 each perform a part of the processing performed by the processing execution section 21.

In one example, the information processing device 6 is composed of a computer or the like, and a processor or an integrated circuit of the computer functions as the processing execution unit 21. The processor or integrated circuit of a computer is either a CPU (Central Processing Unit), ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), microcomputer, FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), etc. including. The computer serving as the information processing device 6 may include only one integrated circuit or the like, or may include a plurality of integrated circuits or the like.

Further, in the computer serving as the information processing device 6, a storage medium of the computer functions as the storage unit 22. The storage medium may include an auxiliary storage device in addition to a main storage device such as a memory. Examples of storage media include magnetic disks, optical disks (CD-ROM, CD-R, DVD, etc.), magneto-optical disks (MO, etc.), semiconductor memories, and the like. The computer serving as the information processing device 6 may be provided with only one storage medium or the like, or may be provided with a plurality of storage media and the like.

In the computer serving as the information processing device 6, a processor or an integrated circuit executes a program stored in a storage medium or the like, so that the processing execution unit 21 performs the processing described below. In one example, in a computer serving as the information processing device 6, a program executed by a processor or the like may be stored in a computer (server) connected via a network such as the Internet, or a server in a cloud environment. . In this case, the processor downloads the program via the network. Further, in one example, the information processing device 6 is composed of a plurality of computers that are separate from each other. In this case, the processing described below by the processing execution unit 21 is performed by processors or integrated circuits of a plurality of computers.

Further, in one example, the information processing device 6 is configured from a server in a cloud environment. The infrastructure of the cloud environment is composed of virtual processors such as virtual CPUs and cloud memory. In the server in the cloud environment serving as the information processing device 6, a virtual processor or the like functions as the processing execution unit 21, and the processing described below by the processing execution unit 21 is performed by the virtual processor or the like. The cloud memory then functions as the storage unit 22.

In the information processing device 6, the communication interface 23 includes an interface for accessing external devices such as the inspection device 5 and the imaging devices 11 to 13. The information processing device 6 can communicate with an external device via the communication interface 23 by wire or wirelessly. Therefore, the information processing device 6 acquires, via the communication interface 23, information indicating the test results of the test device 5, images taken by the photographing devices 11 to 13, and the like.

Further, in the information processing device 6, various operations and the like are inputted through the user interface 25 by the user of the manufacturing system 1 and the like. The user interface 25 is provided with any one of a button, a switch, a touch panel, etc. as an operation member through which an operation is input by a user of the manufacturing system 1 or the like. Further, the user interface 25 notifies users of the manufacturing system 1 of various types of information. The information is announced by, for example, displaying the information on a screen or transmitting a voice. In one example, the user interface 25 is provided as an external device to the information processing device 6, and is provided separately from the computer and the like that constitute the information processing device 6.

The processing performed by the processing execution unit 21 will be described below. FIG. 3 shows an example of processing by the image processing section 31 and the determination section 32. As shown in FIG. 3 and the like, the image processing unit 31 performs image processing as preprocessing for the processing in the determination unit 32 (S101). The image processing unit 31 performs image processing using three types of images before reflow, namely, an image of only the board, an image of only solder mounted on the board, and an image of solder and components mounted on the board. I do. The image processing unit 31 performs image processing to generate image data based on three types of images before reflow as image data used for processing in the determination unit 32. Moreover, the image processing unit 31 generates image data based on three types of images before reflow acquired in real time every time one substrate is carried into the manufacturing line.

FIG. 4 shows an example of image processing by the image processing section 31. The example process shown in FIG. 4 is performed each time one substrate is brought into the manufacturing line, that is, each time three types of images before reflow are acquired (photographed) in real time. When the process of FIG. 4 is started, the image processing unit 31 converts each of the three types of images into grayscale (S111). Then, the image processing unit 31 aligns the three types of images with respect to each other (S112). As a result, positional deviations between the three types of images relative to each other are corrected.

The image processing unit 31 then assigns different color information to each of the three types of grayscaled images (S113). In this case, in one example, an image of only the board has blue color information, an image of only solder mounted on the board has green color information, and an image of the board with solder and components mounted has red color information. Color information is given to each. Then, the image processing unit 31 combines three types of images to which different color information is added (S114). As a result, image data composed of a composite image of three types of images is generated as image data used for processing in the determination unit 32. Therefore, in the example of FIG. 4, after aligning the three types of images and adding different color information to the three types of images, the determination unit 32 combines the three types of images. Image data used for processing is generated.

FIG. 5 shows another example of image processing by the image processing unit 31, which is different from that shown in FIG. The process shown in the example of FIG. 5 is also performed every time one substrate is brought into the manufacturing line, that is, every time three types of images before reflow are acquired (photographed) in real time. When the process of FIG. 4 is started, the image processing unit 31 arranges three types of images (S115). As a result, image data in which three types of images are arranged is generated as image data used for processing in the determination unit 32. Therefore, in the example of FIG. 5, the image data used for processing in the determination unit 32 is composed of three types of images arranged relative to each other.

The storage unit 22 stores a machine learning model generated (constructed) as described below. As shown in FIG. 3, etc., the determination unit 32 uses a machine learning model to determine the results of the inspection scheduled to be performed by the inspection device 5 after reflow from the image data generated by the image processing unit 31 as described above. It is determined whether the product is good or bad (S102). In other words, the determination unit 32 determines whether the inspection to be performed after reflow is good or bad based on image data based on three types of images before reflow acquired in real time. The determination unit 32 inputs the image data generated by the image processing unit 31 to the machine learning model. Then, the determining unit 32 causes the machine learning model to output whether the test is good or bad, and uses the output result from the machine learning model as the judgment result regarding the good or bad test.

FIG. 6 shows an example of a machine learning model used for determination by the determination unit 32. As shown in FIG. 6 and the like, it is composed of an input layer 41, an output layer 42, and an intermediate layer (hidden layer) 43 between the input layer 41 and the output layer 42. When the above-mentioned image data is input to the machine learning model, pixel information such as RGB values for each pixel is input to the input layer 41 for all images forming the image data. For example, when a composite image is generated as image data as in an example such as FIG. 4, pixel information about the composite image is input to the input layer 41 for each pixel. Furthermore, when image data in which three types of images are arranged is generated as in an example such as FIG. Ru.

In an example machine learning model such as that shown in FIG. 6, the input layer 41 is configured from the same number of nodes as the total number of pixels in all images that make up the image data. In an example such as FIG. 6, the total number of pixels in all images constituting the image data is k, and pixel information x1 to xk is input to the input layer 41. Further, in the example machine learning model shown in FIG. 6 and the like, the output layer 42 is composed of a node that outputs a positive result in the test and a node that outputs a defective result in the test.

Further, the intermediate layer 43 includes a convolution layer, a pooling layer, a fully connected layer, and the like. In the convolution layer, filter processing is performed for each pixel to extract characteristic parts in the image. Further, in the pooling layer, the image is reduced while maintaining the characteristic parts of the image. Through processing in the pooling layer, positional deviations of characteristic parts in images that constitute image data input to the machine learning model are absorbed. In the intermediate layer 43, image features are recognized through processing in multiple convolutional layers and multiple pooling layers. Further, the fully connected layer converts the image data whose characteristic parts have been recognized by the convolution layer and the pooling layer into one-dimensional data. By performing the processing as described above, the intermediate layer 43 of the machine learning model recognizes the characteristic parts in the image, and converts the recognized characteristic parts into information used for determining whether the inspection is good or bad. Ru.

A machine learning model used for determination by the determination unit 32 is generated (constructed) by a learning model generation unit 33. In generating a machine learning model, learning data consisting of a large number of data sets is used. In each dataset of learning data, image data based on the above-mentioned three types of images before reflow is shown for printed circuit boards inspected in the past. The image data shown in each of the data sets is generated using three types of images before reflow, for example, in a manner similar to either the example of FIG. 4 or the example of FIG. 5.

Furthermore, in each data set of learning data, the test results of tests actually performed after reflow are shown for printed circuit boards that have been tested in the past. In each of the data sets, the above-mentioned image data and inspection results may be shown for printed circuit boards inspected in the past in the manufacturing system 1, and the image data and inspection results may be shown for printed circuit boards inspected in the past in another manufacturing system having the same configuration as the manufacturing system 1. The above-mentioned image data and inspection results may be shown for the printed circuit board. Since the training data is configured as described above, each of the many datasets of training data includes image data based on images before reflow and image data based on images actually performed after reflow for printed circuit boards inspected in the past. The test results from the test are associated with each other. It should be noted that among the multiple data sets, the inspected printed circuit boards are different from each other.

FIG. 7 shows an example of a machine learning model generation process (construction process) by the learning model generation unit 33. When the process of FIG. 7 is started, the learning model generation unit 33 classifies a large number of datasets of learning data into training datasets and evaluation datasets (S121). In one example, a large number of data sets are classified such that the ratio of training data sets to evaluation data sets is 9.

Then, the learning model generation unit 33 trains the model by deep learning using the training data set of the learning model (S122). At this time, for example, a neural network is trained as a model. In addition, the model is trained by supervised learning in which the test results shown in each training data set are given as correct answers. Through deep learning, the model learns the features included in the image data of the training data set, for example, the features in the image data when the test result is bad.

When learning using the training data set is completed, the learning model generation unit 33 evaluates the learned model using the evaluation data set (S123). In model evaluation, image data based on images before reflow is input to the learned model for each of the evaluation data sets. Then, for each of the evaluation data sets, the output results from the learned model and the test results from the tests actually performed are compared. Then, as an index for evaluating the learned model, the correct answer rate of the output results from the model for the test results of the tests actually performed is calculated.

When the evaluation using the evaluation data set is completed, the learning model generation unit 33 determines whether the accuracy rate calculated as an index for evaluating the learned model is equal to or higher than the reference level (S124). In one example, based on the fact that the correct answer rate is 90% or more, it is determined that the correct answer rate is equal to or higher than the reference level. If the accuracy rate is equal to or higher than the reference level (S124-Yes), the learning model generation unit 33 stores the learned model in the storage unit 22 as the machine learning model used for the determination by the determination unit 32 (S125). ).

If the accuracy rate is lower than the reference level (S124-No), the learning model generation unit 33 adds a data set to the learning data (S126). Then, the process returns to S121, and the learning model generation unit 33 sequentially performs the processes from S121 onwards. Thereby, learning of the model using the added data set and evaluation of the model using the added data set are performed. Note that the learning model generation unit 33 does not need to be provided in the information processing device 6. In one example, the above-described machine learning model generation process (construction process) is performed by a computer or the like different from the information processing device 6.

Further, in the information processing device 6, the learning model updating unit 35 updates the machine learning model by relearning the machine learning model. The learning model updating unit 35 stores the updated machine learning model in the storage unit 22. In one example, the machine learning model is retrained every time a situation occurs in which the determination result by the determination unit 32 differs from the test result in an actually performed test. In another example, the machine learning model is retrained based on the fact that the correct answer rate of the determination result by the determination unit 32 for the test result of the actually performed test has become lower than the reference level. In this case, for example, based on the above-mentioned accuracy rate being lower than 90%, it is determined that the accuracy rate has become lower than the reference level. In another example, the machine learning model is periodically retrained at predetermined intervals, regardless of the above-mentioned accuracy rate and the like.

Updating the machine learning model is also performed in the same way as generating the machine learning model. In other words, a large number of datasets are used to update the machine learning model, and each of the datasets used includes image data based on the image before reflow and image data based on the image actually performed after reflow of the inspected printed circuit board. The test results from the test are associated with each other. Then, the learning model updating unit 35 classifies the dataset into a training dataset and an evaluation dataset, and uses the training dataset to model a machine learning model in the same way as model learning in generating a machine learning model. Re-learn. Then, when the relearning is completed, the retrained machine learning model is evaluated in the same manner as the evaluation of the learned model in generating the machine learning model.

By updating the machine learning model as described above, in the updated machine learning model, the accuracy rate of the output result from the machine learning model with respect to the test result of the actually performed test becomes equal to or higher than the reference level. In updating the machine learning model, in any one of a large number of data sets used for relearning, the actual judgment result in the judgment part 32 is different from the test result in the test actually performed after reflow. The test results of the tests conducted on the following day are shown. Therefore, the learning model updating unit 35 performs machine learning using the test results of the actually performed test when the determination result of the determination unit 32 is different from the test result of the test actually performed after reflow. Retrain the model.

In one example, when the determination unit 32 determines that the test result is good but the actual test result is poor, image data based on the image before reflow and the test actually performed after reflow are used. Results are presented in any one of a number of data sets. That is, in the case where a defect is newly discovered in an inspection, the above-mentioned image data and inspection results are shown in one of the data sets. In this case, during relearning, the machine learning model learns the characteristics of image data for a data set in which image data and inspection results are newly discovered to be defective in an inspection. Note that the learning model updating unit 35 does not need to be provided in the information processing device 6. In one example, the above-described machine learning model updating process is performed by a computer or the like different from the information processing device 6.

As described above, in this embodiment, a machine learning model is used that outputs the inspection results of the inspection after reflow from the input of image data based on the image before reflow, and the image before reflow is acquired in real time. Based on the image data based on , it is determined whether the test is good or bad in the inspection scheduled to be performed after reflow. Therefore, in manufacturing a printed circuit board by soldering, defects in the printed circuit board manufactured as a product can be detected before the components are joined to the base board by reflow.

By making it possible to detect defects in an inspection performed after reflow before reflow, the occurrence of defective printed circuit boards (products) can be effectively suppressed, and component loss can be appropriately reduced. Further, by suppressing the occurrence of printed circuit boards that become defective products, an increase in the number of steps due to repair of defective products, etc. in manufacturing printed circuit boards can be effectively suppressed. Furthermore, by being able to detect defects before reflow, it becomes possible to investigate the cause of defects from the state before the component is joined to the base substrate by reflow. Therefore, even in investigating the cause of defects, loss of parts is appropriately reduced and an increase in the number of steps is effectively suppressed.

Furthermore, in the present embodiment, image data based on images before reflow is input into a machine learning model, and it is determined whether the product is good or bad in an inspection performed after reflow. Because image data based on images before reflow is used as input to the machine learning model, variations in the state of the product after reflow from one printed circuit board to another, and variations between each printed circuit board or printed circuit board at each step of the manufacturing process. Passing or defective inspection is determined by taking into consideration the variation in each item. Therefore, it is possible to more appropriately determine pass or fail in the inspection scheduled to be performed after reflow.

In addition, in this embodiment, in generating a machine learning model, deep learning The model is trained by Therefore, a machine learning model is generated that appropriately outputs the inspection results after the reflow from the input of image data based on the image before the reflow.

In addition, in this embodiment, when updating the machine learning model, the machine learning Retrain the model. Since the machine learning model is updated in the manner described above, a decrease in the accuracy rate of the determination result with respect to the test result in the actual test is effectively suppressed.

Further, in an example such as FIG. 4, the three types of images before reflow are aligned with each other, and the three types of images before reflow are given different color information with respect to each other. . Then, after alignment and color information are added, the three types of images are combined to generate image data to be input to the machine learning model. When a synthetic image is input to a machine learning model, the complexity of the machine learning model is suppressed, and the processing speed of the machine learning model is improved. Therefore, it is possible to more quickly determine whether the product is good or bad in the inspection scheduled to be performed after reflow.

Further, in an example such as FIG. 5, image data input to the machine learning model is generated by arranging the three types of images described above before reflow. When image data consisting of three types of images is input to a machine learning model, more detailed data analysis is performed in the machine learning model, which improves the appropriateness of the output results from the machine learning model. Therefore, it is possible to more appropriately determine pass or fail in the inspection scheduled to be performed after reflow.

(Second embodiment)
Next, a second embodiment will be described as a modification of the first embodiment. In the second embodiment as well, the manufacturing system 1 includes a solder printing device 2, a component mounting device 3, a reflow device 4, an inspection device 5, an information processing device 6, and photographing devices 11 to 13. A printed circuit board is manufactured as a product using the device 3, the reflow device 4, and the inspection device 5 in the same manner as in the embodiments described above.

FIG. 8 shows an example of the information processing device 6 according to this embodiment. As shown in FIG. 8 and the like, in this embodiment as well, the information processing device 6 includes a processing execution unit 21, a storage unit 22, a communication interface 23, and a user interface 25. The processing execution unit 21 includes an image processing unit 31, a determination unit 32, a learning model generation unit 33, and a learning model updating unit 35. Also in this embodiment, the photographing devices 11 to 13 photograph images before reflow in the same manner as in the above-described embodiments, and the image processing section 31, determination section 32, learning model generation section 33, and learning model updating section Each of 35 performs the same processing as in the above-described embodiment.

In this embodiment, the process execution unit 21 further includes a cause identification unit 36, and the cause identification unit 36 performs a part of the processing performed by the process execution unit 21. The cause specifying unit 36 specifies the cause of the defect when the determining unit 32 determines that the product will be defective in an inspection scheduled to be performed after reflow. At this time, the cause identification unit 36 uses XAI (explainable artificial intelligence) technology to identify the cause of the defect in the image data input to the machine learning model. As a result, the cause of the defect is identified in the image data based on the three types of images before reflow. Note that when identifying the cause of a defect as in this embodiment, image data in which three types of images are arranged is input to a machine learning model, as shown in an example of FIG. It is preferable that

FIG. 9 shows an example of the process of identifying the cause of a defect by the cause identifying unit 36. The example process in FIG. 9 is performed when the determination unit 32 determines that the product is defective in the inspection after reflow. When the process of FIG. 9 is started, the cause identifying unit 36 extracts nodes that have a high degree of contribution to the defect determination in the intermediate layer 43 and input layer 41 of the machine learning model (S131). That is, nodes that have a high degree of contribution to the output result from the node that outputs a defect in the output layer 42 are extracted from the intermediate layer 43 and the input layer 41.

Then, the cause identifying unit 36 identifies, in the image data input to the machine learning model, a portion related to the node extracted as a node having a high degree of contribution to the defect determination (S132). As a result, a portion of the image data related to the extracted node is identified as the cause of the defect. Then, the cause specifying unit 36 uses the user interface 25 or the like to notify the portion of the image data that has been specified as the cause of the defect (S133). At this time, in one example, the portion identified as the cause of the defect in the image constituting the image data is displayed in a different color from other portions to notify the portion identified as the cause of the defect.

FIG. 10 explains an example of the process of extracting nodes that have a high degree of contribution to the defect determination, which is performed in the cause identification unit 36. In the example of FIG. 10, a plurality of nodes including nodes N2 and N3 are extracted from the intermediate layer 43 as nodes that have a high degree of contribution to the output result from the node N1 which outputs a defect in the output layer 42. Then, the node N4 is extracted from the input layer 41 as a node that has a high degree of contribution to the output result from the node N1 which outputs that it is defective in the output layer 42. Then, in the image data input to the machine learning model, that is, in any of the three types of images before reflow that make up the image data, the part related to node N4 is identified as the cause of the defect, and the pixel information The portion corresponding to the x3 pixel is identified as the cause of the defect. Note that in FIG. 10, the node N1 that outputs that it is defective is shown in black, and the nodes N2 to N4, which are extracted as nodes with a high degree of contribution to the defect determination, are shown in diagonal hatching.

FIG. 11 illustrates an example of a process performed by the cause identification unit 36 to notify a portion of image data identified as the cause of a defect. In an example of FIG. 11, an image I1 of only the substrate 51 as a base body, an image I2 of only the solders 52A and 52B mounted on the substrate 51, and an image I3 of the substrate 51 mounted with the solders 52A and 52B and a component 53 are shown. The three types of images I1-I3 before reflow are aligned with respect to each other to generate image data that is input to the machine learning model. Further, in the example of FIG. 11, pads 55A and 55B are formed on the substrate 51, and the component 53 is an LED chip including an LED 56.

In the example of FIG. 11, the cause identifying unit 36 displays three types of images I1 to I3 that constitute the image data input to the machine learning model on the user interface 25 or the like in notifying the portion identified as the cause of the defect. let Then, the cause identifying unit 36 displays in red the portion identified as the cause of the defect in the images I1 to I3. In the example of FIG. 11, the pad 55A in the image I1 of only the board 51 and the LED 56 of the component 53 in the image I3 in which the solders 52A, 52B and the component 53 are mounted on the board 51 are displayed in red. Based on the images I1 to I3 displayed as an example of FIG. 11 as a result of identifying the cause of the defect, the user of the manufacturing system 1 can confirm that the pad 55A formed on the substrate 51 is thin and that the component mounting It is recognized that the fact that the mounting position of the component 53 by the device 3 deviates from the proper position is the cause of the determination as being defective.

In this embodiment as well, image data based on the image before reflow acquired in real time is generated using a machine learning model that outputs the inspection results of the inspection after reflow from the input of image data based on the image before reflow. From this, it is determined whether the product passes or fails in the inspection scheduled to be performed after reflow. Therefore, in this embodiment, as in the first embodiment, when manufacturing a printed circuit board by soldering, the printed circuit board to be manufactured as a product is defects can be detected. Therefore, this embodiment also has the same functions and effects as the first embodiment and the like.

Furthermore, in the present embodiment, when it is determined that a defect occurs in an inspection scheduled to be performed after reflow, the cause of the defect is identified in image data based on an image before reflow. This allows the user of the manufacturing system 1 to recognize which part of the process performed before reflow caused the defect. That is, by performing the above-described process of specifying the cause of the determination as defective in the information processing device 6, the cause of the defect can be quickly and appropriately discovered.

(Modifications of the first embodiment and the second embodiment)
In the above embodiment, three types of images are input to the machine learning model: an image of only the base board, an image of only solder mounted on the board, and an image of solder and components mounted on the board. However, the image data is not limited to this. In one variation, only two of the aforementioned three types of images before reflow are used to generate the image data input to the machine learning model. In this case, in one example, image data input to the machine learning model is generated using two types of images: an image of only the board and an image of solder and components mounted on the board, and in another example Here, image data to be input to a machine learning model is generated using two types of images: an image in which only solder is mounted on a board, and an image in which solder and components are mounted on a board.

Therefore, in the embodiments, image data to be input to the machine learning model may be generated by using a plurality of types of images among the three types of images described above as images before reflow. In this case, in one example, similar to the example in FIG. 4, a plurality of types of images are aligned with each other, and different color information is given to the plurality of types of images with respect to each other. Then, after the above-described alignment and color information have been added, image data to be input to the machine learning model is generated by composing a plurality of types of images. In another example, image data input to a machine learning model is generated by arranging multiple types of images, similar to the example in FIG. 5 and the like.

(Third embodiment)
Next, a third embodiment will be described as a modification of the first embodiment and the like. FIG. 12 shows a manufacturing system 1 according to a third embodiment as an example of a manufacturing system that manufactures semiconductor devices as products. As shown in FIG. 12 and the like, the manufacturing system 1 of this embodiment also includes a reflow device 4, an inspection device 5, and an information processing device 6, similarly to the above-described embodiments. However, in this embodiment, the manufacturing system 1 is provided with a solder printing device 101, a chip mounting device 102, a solder printing device 103, and a connector mounting device 104 instead of the solder printing device 2 and the component mounting device 3. In the manufacturing system 1 of this embodiment, a manufacturing line for semiconductor devices as products is formed, and the manufacturing line includes a solder printing device 101, a chip mounting device 102, a solder printing device 103, a connector mounting device 104, a reflow device 4, and an inspection device. The devices 5 are arranged in order from the upstream side.

On the manufacturing line, a lead frame, which is a base, is carried into a solder printing device 101. The solder printing device 101 mounts solder on the lead frame by printing solder on the surface of the lead frame. At this time, the solder at room temperature is mounted on the lead frame, and the solder is mounted on the lead frame in an unmolten state. Note that the method for mounting the solder on the lead frame is not limited to printing, and the solder may be mounted on the lead frame by dispensing (coating) solder, mounting a solder sheet, or the like.

A lead frame with solder mounted thereon is carried into the chip mounting apparatus 102 . The chip mounting device 102 mounts a chip as a new component on a lead frame. When the chip is mounted, the solder mounted by the solder printing device 101 is sandwiched between the lead frame, which is the base, and the chip, which is the component.

A lead frame with solder and chips mounted thereon is carried into the solder printing apparatus 103 . The solder printing device 103 mounts solder on the lead frame and the chip by printing solder on the surface of the lead frame and the chip. At this time, the solder at room temperature is mounted on the lead frame and the chip, and the solder is mounted on the lead frame and the chip in an unmolten state. Note that the method for mounting solder onto lead frames and chips is not limited to printing, and solder may be mounted onto lead frames and chips by either dispensing (coating) solder or mounting solder sheets. good.

A lead frame with solder mounted on the lead frame and chip in the solder printing apparatus 103 is carried into the connector mounting apparatus 104 . The connector mounting device 104 mounts a connector as a new component to be attached to a lead frame and a chip. A connector is a component mounted on a lead frame and is a component separate from a chip. By mounting the connector, the solder mounted by the solder printing device 103 is sandwiched between the lead frame and the connector or between the chip and the connector.

In this embodiment, components such as a chip, a connector, and a lead frame on which solder is mounted are carried into the reflow apparatus 4 . Also in this embodiment, the reflow device 4 performs soldering by reflow. The reflow melts the solder mounted by the solder printing devices 101, 103, etc. A chip, which is a newly mounted component, is joined to the lead frame, and a connector, which is a newly mounted component, is joined to the lead frame and the chip. As a result, a semiconductor device is formed in which the mounted chips, connectors, and other components are attached to the lead frame that is the base.

A semiconductor device manufactured as a product, that is, a semiconductor device in which components such as a chip and a connector are bonded to a lead frame as a base by reflowing in the reflow device 4 is carried into the inspection device 5 . The inspection device 5 inspects the manufactured semiconductor device and determines whether the manufactured semiconductor device is good or defective. Only semiconductor devices that are determined to be good through the inspection by the inspection device 5 are released onto the market as distributed products. The inspection device 5 performs, for example, a visual inspection and an electrical test on the manufactured semiconductor device. Note that the visual inspection, electrical test, etc. are performed as described above.

In the manufacturing system 1 of this embodiment, photographing devices 111 to 115 such as cameras or video cameras are provided. The photographing device 111 is arranged upstream of the solder printing device 101 and photographs a first image of only the lead frame, which is the base. The photographing device 112 is disposed between the solder printing device 101 and the chip mounting device 102, and photographs a second image showing only solder being mounted on the lead frame. The photographing device 113 is disposed between the chip mounting device 102 and the solder printing device 103, and photographs a third image in which the chip as a component is mounted on the lead frame from the state of the second image. Further, the photographing device 114 is arranged between the solder printing device 103 and the connector mounting device 104, and photographs a fourth image in which solder is further mounted on the lead frame and the chip from the state of the third image. Then, the photographing device 115 is placed between the connector mounting device 104 and the reflow device 4, and from the state of the fourth image, a fifth image in which a connector, which is a component other than the chip, is mounted on the lead frame and the chip. Take an image.

Therefore, in the manufacturing system 1 of the present embodiment, the photographing devices 111 to 115 photograph five types of images, the first image to the fifth image, as images before reflow by the reflow device 4. Each of the photographing devices 111 to 115 transmits a photographed image (image data) to the information processing device 6. Then, the information processing device 6 acquires the images photographed by the photographing devices 111 to 115 in the same way as acquiring the images photographed by the photographing devices 11 to 13 in the above-described embodiments. Therefore, every time one lead frame is brought into the manufacturing line, the information processing device 6 selects the first to fifth images as five types of images taken by the imaging devices 111 to 115 before reflow. obtained in real time.

In this embodiment, the information processing device 6 includes a processing execution unit 21, a storage unit 22, a communication interface 23, and a user interface 25, as in the first embodiment and the like (see FIG. 2). The processing execution unit 21 includes an image processing unit 31, a determination unit 32, a learning model generation unit 33, and a learning model updating unit 35.

Also in this embodiment, the image processing section 31 performs image processing as pre-processing for the processing of the determination section 32, similar to the above-described embodiments. However, in this embodiment, the image processing unit 31 performs image processing using five types of images, the above-mentioned first image to fifth image, as images before reflow. The image processing unit 31 performs image processing to generate image data based on five types of images before reflow, as image data used for processing in the determination unit 32. Moreover, the image processing unit 31 generates image data based on five types of images before reflow acquired in real time every time one lead frame is carried into the manufacturing line.

In one example, similar to the example in FIG. 4, the image processing unit 31 aligns five types of images before reflow and adds different color information to the five types of images. Thereafter, by combining the five types of images, image data used for processing in the determination unit 32 is generated. In another example, image data used for processing in the determination unit 32 is generated by arranging five types of images before reflow.

FIG. 13 shows an example of image data generated by image processing by the image processing unit 31 in this embodiment. In the example of FIG. 13, the determination is made by arranging five types of images before reflow: a first image Ia1, a second image Ia2, a third image Ia3, a fourth image Ia4, and a fifth image Ia5. Image data used for processing in section 32 is generated. In the first image Ia1, only the lead frame 151, which is the base, is shown. Further, the second image Ia2 shows a state in which only the solder 152 is mounted on the lead frame 151. The third image Ia3 shows a state in which the chip 153, which is a component, is mounted on the lead frame 151 from the state in the second image Ia2. Furthermore, the fourth image Ia4 shows a state in which solders 154A, 154B, and 154C are further mounted on the lead frame 151 and solders 154D and 154E are further mounted on the chip 153 from the state in the third image Ia3. It will be done. The fifth image Ia5 shows a state in which connectors 155A and 155B, which are parts different from the chip 153, are mounted on the lead frame 151 and the chip 153 from the state of the fourth image Ia4.

In the present embodiment as well, the determination unit 32 uses the machine learning model stored in the storage unit 22 etc. to perform a process in the inspection device 5 after reflow from the image data generated by the image processing unit 31 as described above. Determine whether the scheduled inspection is good or bad. Therefore, in this embodiment as well, the determining unit 32 determines whether the test is good or bad in the inspection scheduled to be performed after reflow, based on image data based on images before reflow acquired in real time. Also in this embodiment, the determination unit 32 inputs the image data generated by the image processing unit 31 to the machine learning model. Then, the determining unit 32 causes the machine learning model to output whether the test is good or bad, and uses the output result from the machine learning model as the judgment result regarding the good or bad test.

Also in this embodiment, the machine learning model used for the determination by the determination unit 32 is composed of an input layer 41, an output layer 42, and an intermediate layer (hidden layer) 43 between the input layer 41 and the output layer 42. . Then, when the above-mentioned image data is input to the machine learning model, pixel information such as RGB values for each pixel is input to the input layer 41 for all images forming the image data. Further, in the machine learning model, similarly to the above-described embodiments, the output layer 42 is composed of a node that outputs a positive result in the test and a node that outputs a defective result in the test. The intermediate layer 43 is composed of a convolution layer, a pooling layer, a fully connected layer, etc., as in the above-described embodiments.

Further, in this embodiment, the machine learning model used for the determination by the determination unit 32 is generated by the learning model generation unit 33 using learning data composed of a large number of data sets, as in the above-described embodiments. Generated (constructed). However, in this embodiment, in each dataset of learning data, image data based on the aforementioned five types of images before reflow is shown for semiconductor devices inspected in the past. Further, in the present embodiment, the learning model updating unit 35 updates the machine learning model by relearning the machine learning model in the same manner as in the above-described embodiment.

In this embodiment as well, image data based on the image before reflow acquired in real time is generated using a machine learning model that outputs the inspection results of the inspection after reflow from the input of image data based on the image before reflow. From this, it is determined whether the product passes or fails in the inspection scheduled to be performed after reflow. Therefore, in manufacturing a semiconductor device by soldering, defects in the semiconductor device manufactured as a product can be detected before parts such as chips and connectors are joined to a lead frame as a base by reflow. As described above, this embodiment in which a semiconductor device is manufactured as a product by soldering also has the same operation and effect as the above-described embodiment in which a printed circuit board is manufactured by soldering.

(Modification of third embodiment)
Note that in a modified example of the third embodiment, etc., in the manufacturing system 1 that manufactures semiconductor devices as products as described above, the processing execution unit 21 of the information processing device 6 is configured as in the second embodiment, etc. , further includes a cause identification unit 36. Similarly to the second embodiment and the like, the cause identifying unit 36 identifies the cause of the defect when the determining unit 32 determines that the inspection scheduled to be performed after reflow results in a defect. At this time, the cause identification unit 36 uses XAI (explainable artificial intelligence) technology to identify the cause of the defect in the image data input to the machine learning model. As a result, the cause of the defect is identified in the image data based on the five types of images before reflow.

In this modification, as in the second embodiment, when the determination unit 32 determines that the inspection after reflow results in a defect, the cause identification unit 36 determines that the intermediate layer 43 and input layer 41 of the machine learning model , extract nodes that have a high degree of contribution to defectiveness determination. Then, the cause identification unit 36 identifies, in the image data input to the machine learning model, a portion related to the node extracted as a node having a high degree of contribution to the defect determination. As a result, a portion of the image data related to the extracted node is identified as the cause of the defect. Also in this modification, the cause identification unit 36 uses the user interface 25 or the like to notify the portion of the image data that has been identified as the cause of the defect.

In this modification, a machine learning model is used that outputs the inspection results from the inspection after reflow from the input of image data based on the image before reflow, and the image data based on the image before reflow is acquired in real time. From this, it is determined whether the product passes or fails in the inspection scheduled to be performed after reflow. Therefore, in this modification, as in the third embodiment, when manufacturing a semiconductor device by soldering, before parts such as chips and connectors are joined to a lead frame as a base by reflow, as a product. Defects in manufactured semiconductor devices can be detected. Therefore, this modification also has the same functions and effects as the above-described embodiments.

Furthermore, in this modification, when it is determined that the product is defective in an inspection scheduled to be performed after reflow, the cause of the defect is identified in the image data based on the image before reflow. As a result, similarly to the second embodiment and the like, the user of the manufacturing system 1 can recognize which part of the process performed before reflow caused the defect. That is, by performing the above-described process of specifying the cause of the determination as defective in the information processing device 6, the cause of the defect can be quickly and appropriately discovered.

Furthermore, in the third embodiment and its modifications described above, the image data input to the machine learning model is generated using five types of images before reflow, which are the first image to the fifth image. However, it is not limited to this. In one variation, wire bonding is performed in the manufacture of semiconductor devices instead of mounting the connector as a component. In this case, the manufacturing system 1 for manufacturing semiconductor devices is not provided with the solder printing device 103 and the connector mounting device 104. In this modification, when the solder printing device 101 loads the solder onto the lead frame and the chip mounting device 102 loads the chip onto the lead frame, the reflow device 4 does not mount the connector or the like. reflow is performed.

Since the semiconductor device is manufactured as described above, in this modification, only the photographing devices 111 to 113 are provided, and the photographing devices 114 and 115 are not provided. Therefore, in this modification, only three types of images, the first image to the third image, are photographed as images before reflow, and the fourth image and the fifth image are not photographed. In this modification, three types of images, the first image to the third image, are used as images before reflow to generate image data to be input to the machine learning model. Therefore, in the third embodiment and its modified examples, the machine learning model It is only necessary that the image data inputted to the image data be generated.

(Other variations)
In a modification, in addition to the image data based on the image before reflow, it is determined whether the product is good or bad in an inspection scheduled to be performed after reflow, based on information indicating conditions for reflow. In this case as well, the machine learning model described above is used to determine whether the test is good or bad, and the test is judged to be good or bad in the same manner as in the embodiments described above. The information indicating the conditions for reflow includes spec information of the reflow apparatus 4, the environmental temperature of the environment in which reflow is performed, and the like.

Also, in some variations, solder thickness information may be input to the machine learning model in addition to image data based on images before reflow. In addition to the above-mentioned image data based on images before reflow, position information in the height direction (thickness direction of the substrate, solder, and components) for each of the images before reflow is input into the machine learning model. It's okay. The positional information in the height direction in each of the images before reflow is, for example, an image of the height distribution in each of the images before reflow. In this modification as well, the above-described machine learning model is used to determine whether the test is good or bad, and the test is judged to be good or bad in the same manner as in the embodiments described above.

According to at least one of these embodiments or examples, the image data is obtained in real time using a machine learning model that outputs an inspection result in an inspection after reflow from an input of image data based on an image before reflow. Based on the image data based on the image before reflow, it is determined whether the test is good or bad in an inspection scheduled to be performed after reflow. As a result, it is possible to provide an information processing device and an information processing method that can detect defects in manufactured products before components are joined to a base by reflow in manufacturing products by soldering.

Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

DESCRIPTION OF SYMBOLS 1...Manufacturing system, 2...Solder printing device, 3...Component mounting device, 4...Reflow device, 5...Inspection device, 6...Information processing device, 11-13...Photographing device, 21...Process execution unit, 22...Storage unit , 31... Image processing section, 32... Judgment section, 33... Learning model generation section, 35... Learning model updating section, 36... Cause identification section, 41... Input layer, 42... Output layer, 43... Intermediate layer (hidden layer) , 101, 103...Solder printing device, 102...Chip mounting device, 104...Connector mounting device, 111-115...Photographing device.

Claims (11)

An information processing device related to soldering of components to a base,
Using a machine learning model that outputs the inspection results of the inspection after the reflow from the input of image data based on the image before the reflow, the image data based on the image before the reflow acquired in real time is An information processing device comprising: a determination unit that determines whether the product is good or bad in an inspection scheduled to be performed after the test. If the determination unit determines that the inspection scheduled to be performed after the reflow results in a defect, the method further includes a cause identification unit that identifies the cause of the failure in the image data based on the image before the reflow. , The information processing device according to claim 1. The cause identification unit extracts a node having a high degree of contribution to the defect determination in the machine learning model,
The cause identifying unit identifies a portion related to the extracted node in the image data based on the image before the reflow as the cause of the defect.
The information processing device according to claim 2. A model is trained by deep learning to generate the machine learning model using learning data in which image data based on the image before reflow is associated with inspection results from an inspection actually performed after the reflow. The information processing device according to any one of claims 1 to 3, further comprising a learning model generation section. retraining the machine learning model using the test result of the test actually performed when the judgment result of the judgment unit is different from the test result of the test actually performed after the reflow; The information processing apparatus according to any one of claims 1 to 3, further comprising a learning model updating section that updates the machine learning model by updating the machine learning model. The information according to any one of claims 1 to 3, further comprising an image processing unit that uses a plurality of types of images as the images before the reflow to generate the image data to be input to the machine learning model. Processing equipment. The information processing apparatus according to claim 6, wherein the image processing unit generates the image data to be input to the machine learning model by arranging the plurality of types of images. The image processing unit aligns the plurality of types of images with respect to each other, and provides color information different from each other to the plurality of types of images,
The image processing unit generates the image data to be input to the machine learning model by synthesizing the plurality of types of images after alignment and adding the color information.
The information processing device according to claim 6. The image processing unit generates multiple types of images among three types: an image of only the substrate as the base body, an image of only the solder mounted on the substrate, and an image of the solder and components mounted on the board. 7. The information processing apparatus according to claim 6, wherein the image is used to generate the image data input to the machine learning model. The image processing unit generates a first image of only the lead frame as the base body, a second image of only the solder mounted on the lead frame, and a second image of the component on the lead frame based on the state of the second image. a third image in which the chip is mounted; a fourth image in which solder is further mounted on the lead frame and the chip; and a fourth image in which the lead frame and the chip are further mounted in the state in the fourth image; A plurality of types of images among the five types of images of a fifth image in which a connector, which is a component different from the chip, is mounted on the chip, are used to generate the image data that is input to the machine learning model. , The information processing device according to claim 6. An information processing method regarding soldering of components to a base, the method comprising:
Using a machine learning model that outputs the inspection results of the inspection after the reflow from the input of image data based on the image before the reflow, the image data based on the image before the reflow acquired in real time is An information processing method comprising determining pass or fail in an inspection scheduled to be performed after.