JP7339051B2 - Inspection support system, server device, inspection support method, and inspection support program - Google Patents

Description

The present disclosure relates to an inspection support system, a server device, an inspection support method, and an inspection support program.

A company that manufactures and sells products (including parts and intermediate products in the manufacturing process) may visually inspect the product and prepare a report. It takes time and effort for an inspector to visually inspect a product, determine the presence or absence of defects, dimensions, conditions, etc., and fill in a report. In addition, there is variation in inspection ability among inspectors, and oversights during inspection are likely to occur. A similar problem may occur in inspections during maintenance after the product is sold.

Therefore, in recent years, there has been proposed an apparatus for determining defects in a product based on an image of the product to be inspected and creating a report. For example, Patent Literature 1 discloses a report creation device that acquires an image of a part of a wind power generation facility, determines damage (defect) included in the image, and creates a report. there is The report creation device is configured to perform judgment by machine learning a large number of damage (defect) images in advance.

JP 2018-181235 A

However, even if a large number of damage images are machine-learned in advance and defects are determined based thereon, the determination accuracy may not be sufficient. Therefore, it is necessary for the inspector to make the final determination. The report creation device of Patent Document 1 also has a configuration for receiving input of observations and special remarks by inspectors.

In this way, there is a risk that the accuracy of judgment will be insufficient only by machine-learning images of damage, and the burden of inspection work on the inspector will not be sufficiently reduced. If operation continues under these conditions, the discrepancy between the results of visual inspections by inspectors (users) and the results of judgments based on machine learning will not decrease from the start of operation, making it impossible to reduce the burden of inspection work. The problem of insufficient determination accuracy may remain. Therefore, convenience is low.

In view of the circumstances described above, it is an object of at least one embodiment of the present invention to provide an inspection support system or the like capable of improving convenience.

An inspection support system according to at least one embodiment of the present invention includes:
An inspection support system for supporting product inspection work,
an image acquisition unit for acquiring a product image obtained by imaging a product to be inspected;
The product to be inspected included in the product image acquired by the image acquiring unit using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. a determination unit configured to determine the defect of
an input information acquisition unit for acquiring input information required by the user for the judgment result by the judgment unit;
a model updating unit configured to perform machine learning by adding the input information to existing teacher data and update the defect determination model;
Prepare.

According to this configuration, the defect determination model used by the determination unit is updated based on the input information input by the inspector (user) with respect to, for example, an erroneous determination result by the determination unit. In this case, when the operation is continued, the difference between the result of the visual inspection by the user and the determination result by the determination unit is reduced compared to when the operation is started. As a result, the accuracy of determination is improved, and the burden of inspection work is reduced. Therefore, convenience can be improved.

In some embodiments, the input information is information indicating addition or correction of any one or more of the presence/absence, position, area, area, size, and type of the defect.

According to such a configuration, machine learning is performed using input information that is highly relevant to the content of defect determination, so it is possible to more reliably improve determination accuracy.

In some embodiments, the inspection support system includes:
a display unit for displaying information about the defect;
an evaluation unit for evaluating the determination reliability for each type of the defect with respect to the determination result;
a reliability information storage unit for storing reliability information that associates the type of the product to be inspected, the type of the defect, and one or more thresholds of the judgment reliability;
The content to be displayed on the display unit is controlled based on the evaluation result of the determination reliability of the evaluation unit and the one or more threshold values included in the reliability information corresponding to the evaluation result. a display control unit;
Prepare.

According to this configuration, depending on whether the evaluation result of the determination reliability exceeds the threshold, the determination result of the defect of that type can be displayed conspicuously or inconspicuously. becomes. Therefore, the visibility of the determination result can be improved.

In some embodiments, the one or more thresholds include a first threshold, and the display control unit controls, among the determination results, items for which the evaluation result of the determination reliability is lower than the first threshold is controlled so as not to be displayed on the display unit.

According to such a configuration, it is possible to display the determination results excluding items with low determination reliability among the determination results. Therefore, unnecessary check items for the user can be reduced.

In some embodiments, the one or more thresholds include a second threshold, and the display control unit controls, among the determination results, the items for which the evaluation result of the determination reliability exceeds the second threshold. is displayed on the display section as a fixed item.

According to such a configuration, it is possible to display items with high determination reliability as fixed items, thereby reducing the number of check items that require final determination by the user.

In some embodiments, the one or more thresholds include a first threshold and a second threshold, and the display control unit determines that, among the determination results, the evaluation result of the determination reliability is the first Items that are equal to or greater than the threshold value and equal to or less than the second threshold value are controlled to be displayed on the display unit as undetermined items requiring final determination by the user.

According to such a configuration, since an item whose determination reliability is incomplete is finalized by the user's final determination, it is possible to ensure the final determination accuracy.

In some embodiments, the inspection support system includes the one or more thresholds included in the reliability information stored in the reliability information storage unit based on the input frequency or the input content of the input information. further comprising a setting changer configured to change the setting of the

According to such a configuration, for example, the second threshold is lowered for items with low input frequency of input information, the first threshold is raised for items with high input frequency of input information for excluding detected defects, and detection It is possible to lower the first threshold value for items with a high input frequency of input information for adding defects that have not been detected. As a result, the amount of checking by the user can be reduced.

In some embodiments, the inspection support system includes:
a display unit for displaying information about the defect;
a display control unit configured to control content to be displayed on the display unit;
a drawing information storage unit for storing drawing data of a plurality of products;
with
The display control unit extracts the drawing data corresponding to the product image acquired by the image acquisition unit from the drawing information storage unit, and depicts the defect included in the determination result on the extracted drawing data. An image is displayed on the display unit.

According to such a configuration, the user can confirm the recognition result of the position, area, etc. of the defect by the inspection support system on the drawing, thereby improving the visibility.

In some embodiments, the inspection support system includes:
A condition information storage unit for storing condition information indicating conditions of the defect when the product does not conform to the inspection criteria,
The determination unit determines whether or not the product to be inspected is nonconforming by comparing the determination result and the condition information.

According to such a configuration, the inspection support system can also determine whether or not the product to be inspected conforms to the inspection standard. In this case, even if the user does not know the inspection standard, the system side can determine it. Therefore, the user's burden can be reduced.

In some embodiments, the inspection support system further includes a report creation unit that creates a report based on the determination result confirmed by the user's operation and the determination result of nonconformity. .

According to such a configuration, since the inspection support system creates a report reflecting the determination result and the determination result, the user's workload for creating the report is reduced.

A server device according to at least one embodiment of the present invention comprises:
A server device for supporting product inspection work,
an image acquisition unit for acquiring a product image obtained by imaging a product to be inspected;
The product to be inspected included in the product image acquired by the image acquiring unit using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. a determination unit configured to determine the defect of
an input information acquisition unit for acquiring input information required by the user for the judgment result by the judgment unit;
a model updating unit configured to perform machine learning by adding the input information to existing teacher data and update the defect determination model;
Prepare.

According to this configuration, the defect determination model used by the determination unit is updated based on the input information input by the inspector (user) with respect to, for example, an erroneous determination result by the determination unit. In this case, when the operation is continued, the difference between the result of the visual inspection by the user and the determination result by the determination unit is reduced compared to when the operation is started. As a result, the accuracy of determination is improved, and the burden of inspection work is reduced. Therefore, convenience can be improved.

An inspection support method according to at least one embodiment of the present invention comprises:
An inspection support method for supporting product inspection work, comprising:
an image acquisition step of acquiring a product image of the product to be inspected;
The product to be inspected included in the product image acquired in the image acquisition step, using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. a determination step of determining the defect of
an input information acquisition step of acquiring input information required by the user for the determination result of the determination step;
a model updating step of performing machine learning by adding the input information to existing teacher data to update the defect determination model;
Prepare.

According to this method, the defect determination model is updated based on the input information input by the inspector (user) for the erroneous determination result in the determination step, for example. In this case, when the operation is continued, the difference between the result of the visual inspection by the user and the determination result in the determination step is reduced compared to when the operation is started. As a result, the accuracy of determination is improved, and the burden of inspection work is reduced. Therefore, convenience can be improved.

An inspection support program according to at least one embodiment of the present invention,
to the computer,
an image acquisition procedure for acquiring a product image obtained by imaging the product to be inspected;
The product to be inspected included in the product image acquired in the image acquisition procedure, using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. Determination procedure for determining defects in
an input information acquisition procedure for acquiring input information required by the user for the judgment result in the judgment procedure;
A model updating procedure for performing machine learning by adding the input information to existing teacher data and updating the defect determination model;
to run.

According to this program, the defect determination model is updated based on the input information input by the inspector (user) with respect to, for example, an erroneous determination result in the determination procedure. In this case, when the operation is continued, the difference between the result of the visual inspection by the user and the determination result of the determination procedure is reduced compared to when the operation is started. As a result, the accuracy of determination is improved, and the burden of inspection work is reduced. Therefore, convenience can be improved.

According to at least one embodiment of the present invention, it is possible to provide an inspection support system and the like that can improve convenience.

It is a block diagram showing roughly composition of an inspection support system concerning one embodiment. It is a block diagram showing roughly the composition of the server device concerning one embodiment. 1 is a block diagram schematically showing the configuration of a communication terminal according to one embodiment; FIG. It is a figure which shows an example of the reliability information which the inspection assistance system which concerns on one Embodiment uses. FIG. 3 is a diagram illustrating an example of an image depicting defects displayed by an inspection support system according to one embodiment; It is a figure which shows an example of the input information with respect to the determination result of the inspection assistance system which concerns on one Embodiment. 7 is a flowchart showing an example of determination processing of the inspection support system according to one embodiment; 6 is a flowchart showing an example of update processing of the inspection support system according to one embodiment;

Several embodiments of the present invention will now be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of the components described as embodiments or shown in the drawings are not intended to limit the scope of the present invention, and are merely illustrative examples. do not have.
For example, expressions such as "identical", "equal", and "homogeneous", which express that things are in the same state, not only express the state of being strictly equal, but also have tolerances or differences to the extent that the same function can be obtained. It shall also represent the existing state.
On the other hand, the expressions "comprising", "comprising", "having", "including", or "having" one component are not exclusive expressions excluding the presence of other components.

(Overall configuration of inspection support system)
The overall configuration of the inspection support system 1 according to one embodiment will be described below. FIG. 1 is a block diagram schematically showing the configuration of an inspection support system 1 according to one embodiment.

Note that the inspection support system 1 according to the embodiment described below is configured to achieve its functions through a cooperative relationship between the server device 100 and the communication terminal 200 . However, the inspection support system 1 is not limited to such a configuration, and may be a single device or may be composed of more devices.

As shown in FIG. 1 , the inspection support system 1 includes a server device 100 and one or more communication terminals 200 communicably connected to the server device 100 via a network 2 . The network 2 may be a WAN (World Area Network) or a LAN (Local Area Network).

The inspection support system 1 is a system for supporting product inspection work. A user of the inspection support system 1 may be an inspector who inspects a product, or a user who uses the product. Defects determined by the inspection support system 1 are defects that can be determined from the appearance, and the types thereof include, for example, cracks, dents, scratches, poor coating, oxidation, thinning, dirt, and the like.

The product to be inspected may be the entire product, or may be a part (for example, mechanical part) that constitutes the product instead of the entire product. The product to be inspected may be a new product, a repaired product, or an existing facility. The products to be inspected may be, for example, rotor blades, stator blades, split rings, combustors, etc. of gas turbines.

The server device 100 is, for example, a server device managed by a product manufacturer, an inspection support service provider, or the like. The server device 100 uses the defect determination model to perform determination regarding defects, which will be described later.

The communication terminal 200 is, for example, a device having a communication function such as a personal computer, a tablet terminal, or a smart phone. The communication terminal 200 is used by a user at an inspection location, and is used to provide the server device 100 with a product image obtained by imaging a product to be inspected. Note that the communication terminal 200 itself may be configured to have an imaging function, or may be configured to be connected to an imaging device such as a camera without an imaging function and capable of acquiring image data.

The configuration of each device (server device 100 and communication terminal 200) will be described below. FIG. 2 is a block diagram schematically showing the configuration of the server device 100 according to one embodiment. FIG. 3 is a block diagram schematically showing the configuration of communication terminal 200 according to one embodiment.

(Configuration of server device)
The configuration of the server device 100 will be described below. As shown in FIG. 2, the server device 100 includes a communication unit 110 that communicates with other devices, a storage unit 120 that stores various data, and an input unit that receives input from a user (for example, an administrator of the server device 100). It includes a unit 130, an output unit 140 that outputs information to a user (for example, an administrator of the server device 100), and a control unit 150 that controls the entire device. These components are interconnected by bus lines 160 .

The communication unit 110 is a communication interface including a NIC (Network Interface Card controller) for performing wired communication or wireless communication. The communication unit 110 communicates with another device (for example, the communication terminal 200) via the network 2. FIG.

The storage unit 120 includes a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The storage unit 120 stores programs for executing various control processes (e.g., a defect determination model for determining defects, an evaluation program for evaluating determination reliability, etc.) and various data (e.g., acquired input information, created reports, judgment results, etc.).

The storage unit 120 includes a reliability information storage unit 121 for storing reliability information, a drawing information storage unit 122 for storing drawing data of a plurality of products, and condition information for storing condition information. and storage unit 123 . Note that the storage unit 120 may be configured by one storage device, or may be configured by a plurality of storage devices.

The input unit 130 includes input devices such as operation buttons, a keyboard, a pointing device, and a microphone. The input unit 130 is an input interface used by the user to input instructions.

The output unit 140 includes an output device such as an LCD (Liquid Crystal Display), an EL (Electroluminescence) display, and a speaker. The output unit 140 is an output interface for presenting various information to the user.

The control unit 150 includes processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Control unit 150 controls the operation of the entire apparatus by executing programs stored in storage unit 120 .

A functional configuration of the control unit 150 will be described below. The control unit 150 functions as an image acquisition unit 151 , a determination unit 152 , an input information acquisition unit 153 , a model update unit 154 , an evaluation unit 155 , a display control unit 156 , a setting change unit 157 and a report creation unit 158 .

The image acquisition unit 151 acquires a product image obtained by imaging the product to be inspected. Specifically, the image acquisition unit 151 acquires the product image by receiving the product image from the communication terminal 200 via the communication unit 110 .

The determination unit 152 uses the defect determination model to determine defects in the product to be inspected included in the product image acquired by the image acquisition unit 151 . Specifically, the determination unit 152 reads the defect determination model from the storage unit 120, inputs the product image to the defect determination model, and performs determination regarding the defect. Determination regarding defects includes determination regarding any one or more of the presence or absence of defects, the types of defects, and details such as the location, area, area, and dimensions (length, width, and depth) of defects. .

Here, the defect determination model will be described. The defect determination model is a model that is configured to perform determination regarding defects in the product to be inspected included in the input product image. Specifically, the defect determination model analyzes RGB information indicating numerical values of red, green, and blue components of each pixel of an image, and determines defects by pattern classification of contrast and hue. be done.

As will be described later, the defect determination model performs machine learning based on user input information. Defects can be identified by estimating dimensions to a high degree of accuracy (eg, to the nearest 0.5 mm).

The defect determination model may be a trained model obtained by learning images of products containing defects and images of products without defects as learning data before the user starts using the model, or may be a learned model based on the knowledge of the manufacturer. It may be a program whose parameters are set based on. In any case, after the user starts using the defect determination model, machine learning is performed using the input information as teacher data, so the defect determination model becomes a learned model.

Further, the determination unit 152 determines whether or not the product to be inspected is nonconforming by comparing the determination result and the condition information. The condition information is information indicating defect conditions when the product does not conform to the inspection criteria.

Cases that do not conform to inspection standards include, for example, cases in which replacement or repair is required in maintenance inspections after sales, cases in which shipment is not possible in the manufacturing process of new or repaired products, or in inspections before shipment. is. The condition information is, for example, information indicating conditions of defects such as a scratch with a length of 5 mm or more and a dent with an area of 1 cm ² or more. Specifically, the determination unit 152 refers to the condition information stored in the condition information storage unit 123, and depending on whether the determination result satisfies the conditions indicated in the condition information, the product to be inspected is nonconforming. Determine whether or not

The input information acquisition unit 153 acquires input information input by the user (inspector) with respect to the determination result by the determination unit 152 . Specifically, the input information acquisition unit 153 acquires the input information by receiving the input information from the communication terminal 200 via the communication unit 110 . The input information is information indicating the content of addition or correction of any one or more of the presence/absence, position, area, area, size, and type of defect that has been input with respect to the determination result. The input information acquisition unit 153 may be configured to acquire input information required by the user instead of input information input by the user in response to the determination result by the determination unit 152 . In other words, the input information acquisition unit 153 may acquire input information when no input such as correction is required for the determination result.

The model update unit 154 performs machine learning by adding the input information acquired by the input information acquisition unit 153 to the existing teacher data, and updates the defect determination model. If the defect determination model is updated each time input information is acquired, the amount of processing increases. Therefore, it is preferable that the control unit 150 causes the storage unit 120 to store the input information acquired by the input information acquiring unit 153, and the model updating unit 154 reads them at the first update timing to update the defect determination model. . The first update timing may be, for example, a regular timing (for example, 12:00 pm) or a timing when the amount of input information stored in the storage unit 120 reaches a predetermined amount.

The evaluation unit 155 evaluates the determination reliability for each defect type of the determination result by the determination unit 152 . The determination reliability is an index (for example, correct answer rate) that quantifies the certainty of the determination result. The determination reliability means, for example, that the larger the numerical value, the higher the reliability, and the evaluation result may be represented by a numerical value within the range of 0-1. An evaluation program for evaluating determination reliability may be updated based on the input information. For example, if there are many cases where there are no corrections or additions by the user to the judgment result regarding the type of defect in a certain product, the parameters of the evaluation program are set so as to highly evaluate the judgment reliability for the type of defect in the product. and the arithmetic expression may be modified. Note that the method of updating the evaluation program is not limited to such an example, and can be changed as appropriate.

The display control unit 156 controls the content displayed on the display unit 240 . The display control unit 156 according to one embodiment is configured to control what is displayed on the display unit 240 of the communication terminal 200 by transmitting a control signal to the communication terminal 200 via the communication unit 110 . Note that the display control unit 156 may be provided in the communication terminal 200 instead of the server device 100 . In this case, the display control unit 156 may be configured to control the content to be displayed on the display unit 240 according to the control signal received from the server device 100 .

Here, the reliability information will be explained. FIG. 4 is a diagram showing an example of reliability information used by the inspection support system 1 according to one embodiment. As shown in FIG. 4, the reliability information includes a plurality of product types (for example, product A and product B) and defect types for each product type (for example, defect 1, defect 2, and defect 3). , and one or more thresholds (for example, a first threshold and a second threshold) of the determination reliability of each of the product types and defect types.

In FIG. 4, for example, product A is a gas turbine rotor blade, and product B is a gas turbine stator blade. For example, defect 1 is a crack, defect 2 is a dent, and defect 3 is a scratch. The first threshold and the second threshold are set within a numerical range (for example, within a range of 0 to 1) that the determination reliability can take. The first threshold is set to a value smaller than the second threshold.

The first threshold is a threshold that serves as a criterion for determining whether or not the display unit 240 should display the determination result of the defect type of the product to be inspected. For example, if the product to be inspected is product A and the determination result includes defect 1 (crack), the determination reliability of the determination result is a value (for example, 0.2) that is lower than the first threshold value (0.3). , the determination item for defect 1 is excluded from the determination results and is not displayed on the display unit 240 . On the other hand, if the determination reliability of the determination result is equal to or greater than the first threshold value (for example, 0.5), the determination item for defect 1 is included in the determination result and displayed on the display unit 240 .

The second threshold is a threshold used as a criterion for determining whether or not the determination result of the defect type of the product to be inspected should be displayed on the display unit 240 as a confirmation item. A fixed item is an item that does not require a final judgment by the user (inspector), and an unfixed item is an item that requires a final judgment by the user. For example, if the product to be inspected is product A and the determination result includes defect 1 (crack), the determination reliability of the determination result is a value (for example, 0.7) that exceeds the second threshold (0.5). , the determination result of the defect 1 is displayed on the display unit 240 as a confirmed item. On the other hand, when the determination reliability of the determination result is a value (for example, 0.2) equal to or less than the second threshold, the determination result of defect 1 is displayed on the display section 240 as an undetermined item.

Note that the reliability information is not limited to the example shown in FIG. For example, the one or more thresholds may be only one of the first threshold and the second threshold. One or more thresholds may be further subdivided to include three or more thresholds, and several thresholds may be used to determine the priority of the content to be displayed on the display unit 240 . Product type and defect type may contain more information. When using both the first and second thresholds, by default the first threshold is set to a small value (e.g., 0.01) and the second threshold is set to a large value (e.g., 0.99). , and the difference between the first threshold value and the second threshold value is preferably reduced by changing the settings thereafter.

The display control unit 156 controls the content to be displayed on the display unit 240 of the communication terminal 200 based on such reliability information. Specifically, the display control unit 156 displays the evaluation result of the determination reliability of the evaluation unit 155 and one or more corresponding to the evaluation result included in the reliability information stored in the reliability information storage unit 121. The content to be displayed on the display unit 240 is controlled by comparing with the threshold.

For example, the display control unit 156 controls the display unit 240 not to display items for which evaluation results of determination reliability are below the first threshold among the determination results. The display control unit 156 controls the display unit 240 to display items for which evaluation results of determination reliability exceed the second threshold among the determination results as fixed items. The display control unit 156 controls the display unit 240 to display items for which evaluation results of determination reliability are equal to or greater than the first threshold value and equal to or less than the second threshold value among the determination results as undetermined items.

In addition, the display control unit 156 extracts the product drawing for inspection from the drawing information storage unit 122 as drawing data corresponding to the product image acquired by the image acquiring unit 151, and the extracted product drawing for inspection is included in the determination result. An image automatically depicting the defect is displayed on the display unit 240 . For example, the display control unit 156 realizes such processing by transmitting to the communication terminal 200 via the communication unit 110 an image depicting defects included in the determination result on the extracted drawing data.

The setting changing unit 157 changes the setting of one or more threshold values included in the reliability information stored in the reliability information storage unit 121 based on the input frequency or input content of the input information. For example, the second threshold is lowered for items with low frequency of input information, and the detected defects are excluded. The first threshold is raised for items with high input information, and undetected defects are added The setting is changed so as to lower the first threshold value for items with a high input frequency of input information.

Such change in threshold setting is performed at the second update timing. The second update timing may be, for example, a regular timing (for example, 12:00 pm) or a timing when the amount of input information stored in the storage unit 120 reaches a predetermined amount. Note that, as described above, the setting change unit 157 may be configured to modify the evaluation program instead of the threshold and change the reliability evaluation method.

The report creation unit 158 creates a report based on the determination result regarding the defect determined by the operation of the user (inspector) and the determination result as to whether or not the defect is nonconforming. Specifically, when the report creation unit 158 acquires information indicating that the user (inspector) performed an operation to confirm the determination result and the determination result from the communication terminal 200 via the communication unit 110, Create a report based on the content of the determination results and discrimination results. The created report may be stored in the storage unit 120, or may be transferred to another device (for example, the communication terminal 200 used by the user or a server device that manages the report).

(Configuration of communication terminal)
The configuration of communication terminal 200 will be described below. As shown in FIG. 3, the communication terminal 200 includes a communication unit 210 that communicates with other devices, a storage unit 220 that stores various data, an input unit 230 that receives input from a user (inspector), and a It comprises a display unit 240 for displaying information, an imaging unit 250 for imaging the product to be inspected, and a control unit 260 for controlling the entire apparatus. These components are interconnected by bus lines 270 .

The communication unit 210 is a communication interface including an NIC for performing wired communication or wireless communication. The communication unit 210 communicates with another device (for example, the server device 100) via the network 2. FIG.

The storage unit 220 is composed of RAM, ROM, and the like. The storage unit 220 stores programs for executing various control processes and various data.

The input unit 230 includes input devices such as operation buttons, a keyboard, a touch panel, a pointing device, and a microphone. The input unit 230 is an input interface used by the user to input instructions. The input information described above is input via the input unit 230 .

The display unit 240 is configured by a display device such as an LCD or an EL display, for example. The display unit 240 displays information about defects such as the determination result received from the server device 100, the evaluation result of the determination reliability, and the determination result of nonconformity. The display unit 240 also displays an input acceptance screen or the like for the user to input information.

The imaging unit 250 is configured by, for example, a camera including a lens, an image sensor, an integrated circuit, and the like. The camera is configured to be capable of capturing still images or moving images of visible light. The imaging unit 250 is used for imaging the appearance of the product to be inspected. Note that the imaging unit 250 may be replaced with a configuration capable of acquiring the defect state more precisely by a measurement technique using ultrasonic waves, lasers, infrared rays, or the like.

The control unit 260 is composed of a processor such as a CPU or GPU. Control unit 260 controls the operation of the entire apparatus by executing programs stored in storage unit 220 . Control unit 260 also receives user operations via input unit 230 . Accordingly, the control unit 260 controls the imaging unit 250 to capture an image of the product to be inspected, and transmits the product image to the server terminal 100 via the communication unit 210 . Also, the control unit 260 transmits input information received through the input unit 230 to the server terminal 100 through the communication unit 210 .

(Contents displayed on the communication terminal)
A specific example of the display screen displayed on the communication terminal 200 of the inspection support system 1 will be described below. FIG. 5 is a diagram showing an example of an image depicting defects displayed by the inspection support system 1 according to one embodiment. FIG. 6 is a diagram showing an example of input information for determination results of the inspection support system 1 according to one embodiment.

As shown in FIG. 5, the display control unit 156 displays an inspection product drawing in which defects included in the determination result are automatically drawn on the drawing data (inspection product drawing) corresponding to the extracted product image on the display unit 240. to display. In this example, the product images are images of the stator blades of the gas turbine captured from multiple directions. Defects are superimposed on inspection product drawings corresponding to those acquired images. Specifically, in the image on the dorsal side, a scratch is indicated by a line L1 as a wing defect. Also, in the image of the ventral side, a dent is shown in the area A1 as a wing defect. The image showing these defects is preferably colored so as to be distinguishable from the image of the product drawing for inspection.

Note that the length and width dimensions may be estimated from the product image. The dimension in the depth direction (for example, the depth of a scratch or the depth of a dent) may be added by the user or estimated from the product image. Although not shown in FIG. 5, detailed information of the determination result (evaluation result of determination reliability and information such as defect dimensions) is displayed on another screen.

Here, a case where the user corrects such determination results will be described. As shown in FIG. 6, when the area A1 is different from the area recognized by the user in the actual visual inspection as a result of the dent determination, the user recognizes the area A1 as the area A2 indicated by the dotted line. It is possible to make an input to modify the For example, the user may fill in area A2 on the inspection product drawing displayed on the touch panel. This input information is transmitted to the server device 100, and the server device 100 acquires the input information. As a result, the input information is used for machine learning.

(Determination process flow)
A specific example of determination processing of the inspection support system 1 will be described below. This process is executed when a user (inspector) performs an inspection. FIG. 7 is a flowchart showing an example of determination processing of the inspection support system 1 according to one embodiment.

As shown in FIG. 7, the communication terminal 200 receives an operation by a user who is an inspector, controls the imaging unit 250, and images the product to be inspected (step S1). The communication terminal 200 transmits the product image of the product to be inspected to the server device 100 via the communication unit 210 (step S2). Thereby, the image acquiring unit 151 of the server device 100 acquires the product image (step S3). A dotted arrow from S2 to S3 represents an instruction by information transmission/reception.

The determination unit 152 of the server device 100 uses the defect determination model to determine defects in the inspection target product included in the acquired product image (step S4). In this case, the evaluation unit 155 of the server device 100 evaluates the reliability of the determination result.

Server device 100 transmits the determination result to communication terminal 200 via communication unit 110 (step S5).

Communication terminal 200 receives the determination result and the reliability evaluation result transmitted from server device 100 (step S6). Communication terminal 200 causes display unit 240 to display the determination result (step S7). Note that the display content is controlled by the display control unit 156 of the server device 100 . By such control, for example, defects are displayed on the inspection product drawing on the display unit 240, and the determination result regarding the defect is displayed or not displayed according to the reliability evaluation result and the threshold value.

Here, communication terminal 200 displays an input information acceptance screen for accepting user input for the determination result. Here, the control unit of communication terminal 200 determines whether or not there is input information (step S8). If the user inputs a correction or additional addition to the judgment result, it is determined that there is input information (step S8; Yes), and the communication terminal 200 transmits the input information to the server device 100 (step S9). .

On the other hand, when the user does not input anything for the determination result and performs an operation to confirm the determination result, communication terminal 200 determines that there is no input information (step S8; No), and displays Display the judgment result that is present as the final judgment result. In this case, communication terminal 200 notifies server apparatus 100 of determination of the determination result, and server apparatus 100 stores the determination result as final information (step S15).

After transmitting the determination result, the server device 100 receives and acquires the input information from the communication terminal 200 (step S10).

Then, the determination unit 152 of the server device 100 performs determination reflecting the input information (step S11). The server apparatus 100 transmits the determination result as the final determination result to the communication terminal 200 (step S12).
Communication terminal 200 receives the final determination result transmitted from server device 100 (step S13). Then, display unit 240 of communication terminal 200 displays the final determination result (step S14).

The control unit 150 of the server device 100 stores the final determination result and the acquired input information in the storage unit 120 as final information (step S15). Note that the input information stored in the storage unit 120 is used for machine learning.

Then, communication terminal 200 accepts a user's operation and instructs server device 100 to create a report (step S16). The server device 100 that has received the instruction compares the condition information for evaluating the conformity of the product to the inspection criteria with the final judgment result, and determines the conformity of the product (step S17). The server device 100 transmits the determination result to the communication terminal 200 (step S18), and creates a report (step S19). On the other hand, the communication terminal 200 that has received the determination result (step S20) displays the determination result (step S21).

(Update process flow)
A specific example of the update process of the inspection support system 1 will be described below. This process may be periodically executed by the server device 100 . FIG. 8 is a flowchart showing an example of update processing of the inspection support system 1 according to one embodiment.

As shown in FIG. 8, when starting the update process, the control unit 150 of the server device 100 determines whether or not it is the first update timing (step S31). Here, if it is determined that it is not the first update timing (step S31; No), the control unit 150 of the server device 100 skips the subsequent processes and ends the update process. On the other hand, when it is determined that it is the first update timing (step S31; Yes), the model update unit 154 of the server device 100 adds the input information to the existing teacher data and executes machine learning (step S32). Also, the model updating unit 154 of the server device 100 updates the defect determination model by the machine learning (step S33).

Next, the control unit 150 of the server device 100 determines whether or not it is the second update timing (step S34). Here, if it is determined that it is not the second update timing (step S34; No), the control unit 150 of the server device 100 skips the subsequent processes and ends the update process. On the other hand, if it is determined that it is the second update timing (step S34; Yes), the setting change unit 157 of the server device 100 changes the setting of one or more thresholds based on the input information, and the reliability information storage unit 121 is updated (step S35).

As described above, for example, as shown in FIGS. 1 to 3, the inspection support system 1 according to one embodiment includes an image acquisition unit 151 for acquiring a product image of a product to be inspected, and an input Using a defect determination model for determining defects in the product to be inspected included in the product image acquired by the image acquisition unit 151, determination regarding defects in the product to be inspected included in the product image acquired by the image acquisition unit 151 is performed. an input information acquisition unit 153 for acquiring input information required by the user for the determination result of the determination unit 152; and machine learning in which the input information is added to existing teacher data. and a model updater 154 configured to update the defect determination model.

According to this configuration, the defect determination model used by the determination unit 152 is updated based on the input information input by the inspector (user) with respect to an erroneous determination result by the determination unit 152, for example. In this case, when the operation is continued, the deviation between the result of the visual inspection by the user and the determination result by the determination unit 152 is reduced compared to when the operation is started. As a result, the accuracy of determination is improved, and the burden of inspection work is reduced. Therefore, convenience can be improved.

In some embodiments, the input information is information indicating addition or correction of any one or more of presence/absence, location, area, area, size, and type of defect.

According to such a configuration, machine learning is performed using input information that is highly relevant to the content of defect determination, so it is possible to more reliably improve determination accuracy.

In some embodiments, for example, as shown in FIGS. 1 to 3, the inspection support system 1 includes a display unit 240 for displaying information about defects, and determination reliability for each defect type for determination results. An evaluation unit 155 for evaluation, a reliability information storage unit 121 for storing reliability information that associates the type of product to be inspected, the type of defect, and one or more thresholds of determination reliability, and the evaluation unit. A display control unit 156 configured to control the content to be displayed on the display unit 240 based on the evaluation result of the determination reliability of 155 and one or more thresholds included in the reliability information corresponding to the evaluation result. And prepare.

According to this configuration, depending on whether the evaluation result of the determination reliability exceeds the threshold, the determination result of the defect of that type can be displayed conspicuously or inconspicuously. becomes. Therefore, the visibility of the determination result can be improved.

In some embodiments, the one or more thresholds include a first threshold, and the display control unit 156 causes the display unit 240 to display the item for which the evaluation result of the determination reliability is lower than the first threshold among the determination results. Control so that it is not displayed.

According to such a configuration, it is possible to display the determination results excluding items with low determination reliability among the determination results. Therefore, unnecessary check items for the user can be reduced.

In some embodiments, the one or more thresholds include a second threshold, and the display control unit 156 displays, among the determination results, items for which the evaluation result of determination reliability exceeds the second threshold as confirmed items. It is controlled to display on the unit 240 .

According to such a configuration, it is possible to display items with high determination reliability as fixed items, thereby reducing the number of check items that require final determination by the user.

In some embodiments, the one or more thresholds include a first threshold and a second threshold, and the display control unit 156 determines that the evaluation result of the determination reliability is equal to or greater than the first threshold and the second threshold among the determination results. Items below the threshold are controlled to be displayed on the display unit 240 as undetermined items requiring final judgment by the user.

According to such a configuration, since an item whose determination reliability is incomplete is finalized by the user's final determination, it is possible to ensure the final determination accuracy.

In some embodiments, for example, as shown in FIGS. 1 and 2 , the inspection support system 1 uses the reliability level stored in the reliability level information storage unit 121 based on the input frequency or the input content of the input information. It further comprises a setting changer 157 configured to change settings of one or more thresholds contained in the information (see FIG. 4).

According to such a configuration, for example, the second threshold is lowered for items with low input frequency of input information, the first threshold is raised for items with high input frequency of input information for excluding detected defects, and detection It is possible to lower the first threshold value for items with a high input frequency of input information for adding defects that have not been detected. As a result, the amount of checking by the user can be reduced.

In some embodiments, for example, as shown in FIGS. 1 to 3 and 5, the inspection support system 1 controls the display unit 240 for displaying information about defects and the content to be displayed on the display unit 240. and a drawing information storage unit 122 for storing drawing data of a plurality of products. Further, the display control unit 156 extracts drawing data corresponding to the product image acquired by the image acquisition unit 151 from the drawing information storage unit 122, and displays an image depicting defects included in the determination result on the extracted drawing data. 240 is caused to display.

According to such a configuration, the user can check the recognition result of the position, area, etc. of the defect by the inspection support system 1 on the drawing, thereby improving the visibility.

In some embodiments, for example, as shown in FIGS. 1 and 2, the inspection support system 1 includes condition information storage for storing condition information indicating defect conditions when the product does not conform to inspection standards. The determination unit 152 includes a unit 123 and determines whether or not the product to be inspected does not conform to the inspection criteria by comparing the determination result with the condition information.

According to this configuration, the inspection support system 1 can also determine whether or not the product to be inspected conforms to the inspection standard. In this case, even if the user does not know the inspection standard, the system side can determine it. Therefore, the user's burden can be reduced.

In some embodiments, for example, as shown in FIGS. 1 and 2, the inspection support system 1 prepares a report based on the determination result confirmed by the user's operation and the determination result of nonconformity. It further includes a report creation unit 158 that creates the report.

According to such a configuration, since the inspection support system 1 creates a report reflecting the determination result and the determination result, the user's work load for creating the report is reduced.

The server device 100 according to one embodiment is, for example, as shown in FIG. The inspection target product included in the product image acquired by the image acquisition unit 151 using the unit 151 and a defect determination model that performs determination regarding the defect of the inspection target product included in the product image for the input product image. an input information acquisition unit 153 for acquiring input information that the user needs for the determination result of the determination unit 152; and a model updater 154 configured to perform machine learning in addition to and update the defect determination model.

According to this configuration, the defect determination model used by the determination unit 152 is updated based on the input information input by the inspector (user) with respect to an erroneous determination result by the determination unit 152, for example. In this case, when the operation is continued, the deviation between the result of the visual inspection by the user and the determination result by the determination unit 152 is reduced compared to when the operation is started. As a result, the accuracy of determination is improved, and the burden of inspection work is reduced. Therefore, convenience can be improved.

The present invention is not limited to the above-described embodiments, and includes modifications of the above-described embodiments and modes in which these modes are combined as appropriate.

(Modification)
In the above-described embodiment, for example, as shown in FIGS. 1 to 3, the example in which the server device 100 mainly executes processing such as determination processing and update processing has been described. However, the inspection support system 1 is not limited to such a configuration. For example, image acquisition unit 151, determination unit 152, input information acquisition unit 153, model update unit 154, evaluation unit 155, display control unit 156, setting change unit 157, report creation unit 158, reliability information storage unit 121, drawing One or more of the information storage unit 122 and the condition information storage unit 123 may be provided in the communication terminal 200 instead of the server device 100 . That is, each function in the inspection support system 1 may be implemented by the server device 100 or by the communication terminal 200 as long as there is no technical contradiction.

Moreover, the inspection support system 1 is not limited to a configuration in which a plurality of devices work together, and may be a single device. That is, the inspection support system 1 is used at the inspection place, and on the spot, the execution of imaging, the judgment processing, the judgment of nonconformity, the creation of a report, the update of the defect judgment model, etc. are executed. good too. In this case, the inspection support system 1 may be configured without the communication units 110 and 210 because data transmission/reception is unnecessary. Also, the two storage units 120, 220, the two control units 150, 260, and the two input units 130, 230 may be integrated.

The imaging unit 250 may be an imaging device independent of the communication terminal 200 , and the storage units 120 and 220 may be external storage devices independent of the server device 100 or the communication terminal 200 . That is, the configurations of the server device 100 and the communication terminal 200 are not limited to the examples shown in FIGS.

The order of determination processing and update processing executed by the inspection support system 1 is not limited to the examples shown in FIGS. 7 and 8, and can be changed as appropriate. Also, some processing may be omitted. For example, if the inspection support system 1 is a single device, steps S2, S5, S9, S10, S12, and S13 shown in FIG. 7 can be omitted. Moreover, although the example shown in FIG. 7 shows the case where the input information is obtained only once, the display of the determination result and the acquisition of the input information corresponding thereto may be executed a plurality of times. When not creating a report, step S19 may be omitted. Also, the order of steps S12 and S15 may be reversed.

In the example shown in FIG. 8, the reliability information updating process of steps S34 and S35 is executed only when the defect determination model updating process shown in steps S31 to S33 is executed. However, these processes may be processed separately and in parallel. Further, even when the defect determination model update process is not performed, the reliability information update process may be performed. The order of steps S31 to S33 and steps S34 and S35 may be reversed.

Moreover, the inspection support system 1 according to the present disclosure may realize its functions by executing a program (inspection support program) by a computer, for example, without using the above-described device. The inspection support program for realizing the function of the inspection support system 1 can be stored in USB (Universal Serial Bus) memory, CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), HDD (Hard Disc Drive), etc. computer-readable recording medium, or downloaded to a computer via the network 2.

1 inspection support system 2 network 100 server device 110, 210 communication unit 120, 220 storage unit 121 reliability information storage unit 122 drawing information storage unit 123 condition information storage unit 130, 230 input unit 140 output unit 150, 260 control unit 151 image Acquisition unit 152 Determination unit 153 Input information acquisition unit 154 Model update unit 155 Evaluation unit 156 Display control unit 157 Setting change unit 158 Report creation unit 160, 270 Bus line 200 Communication terminal 240 Display unit 250 Imaging unit

Claims (13)

An inspection support system for supporting product inspection work,
an image acquisition unit for acquiring a product image obtained by imaging a product to be inspected;
The product to be inspected included in the product image acquired by the image acquiring unit using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. a determination unit configured to determine the defect of
an input information acquisition unit for acquiring input information required by the user for the judgment result by the judgment unit;
a model updating unit configured to perform machine learning by adding the input information to existing teacher data and update the defect determination model;
a display unit for displaying information about the defect;
an evaluation unit for evaluating the determination reliability for each type of the defect with respect to the determination result;
a reliability information storage unit for storing reliability information that associates the type of the product to be inspected, the type of the defect, and one or more thresholds of the judgment reliability;
The content to be displayed on the display unit is controlled based on the evaluation result of the determination reliability of the evaluation unit and the one or more threshold values included in the reliability information corresponding to the evaluation result. and a display control unit . 2. The inspection support system according to claim 1, wherein said input information is information indicating the content of addition or content of correction of any one or more of the presence/absence, position, area, area, size and type of said defect. The evaluation unit is configured to update the determination reliability based on the input information added to the existing training data.
The inspection support system according to claim 1 or 2. the one or more thresholds includes a first threshold;
4. The inspection support system according to claim 3, wherein the display control unit controls not to display items among the determination results in which the evaluation result of the determination reliability is lower than the first threshold value on the display unit. the one or more thresholds includes a second threshold;
5. The display control unit according to claim 3 or 4, wherein the display control unit performs control to display items, among the determination results, for which the evaluation result of the determination reliability exceeds the second threshold as confirmed items on the display unit. inspection support system. the one or more thresholds include a first threshold and a second threshold;
The display control unit, among the determination results, regards an item for which the evaluation result of the determination reliability is equal to or greater than the first threshold value and equal to or less than the second threshold value as an undetermined item requiring final determination by the user. 6. The inspection support system according to any one of claims 3 to 5, wherein control is performed to display on a display unit. a setting change unit configured to change settings of the one or more threshold values included in the reliability information stored in the reliability information storage unit based on the input frequency or input content of the input information; The inspection support system according to any one of claims 3 to 6, further comprising. a display unit for displaying information about the defect;
a display control unit configured to control content to be displayed on the display unit;
a drawing information storage unit for storing drawing data of a plurality of products;
with
The display control unit extracts the drawing data corresponding to the product image acquired by the image acquisition unit from the drawing information storage unit, and depicts the defect included in the determination result on the extracted drawing data. 8. The inspection support system according to any one of claims 1 to 7, wherein an image is displayed on said display unit. A condition information storage unit for storing condition information indicating conditions of the defect when the product does not conform to the inspection criteria,
The inspection support system according to any one of claims 1 to 8, wherein the determination unit determines whether or not the product to be inspected is the nonconforming product by comparing the determination result and the condition information. 10. The inspection support system according to claim 9, further comprising a report creation unit that creates a report based on the determination result confirmed by the user's operation and the determination result of nonconformity. A server device for supporting product inspection work,
an image acquisition unit for acquiring a product image obtained by imaging a product to be inspected;
The product to be inspected included in the product image acquired by the image acquiring unit using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. a determination unit configured to determine the defect of
an input information acquisition unit for acquiring input information required by the user for the judgment result by the judgment unit;
a model updating unit configured to perform machine learning by adding the input information to existing teacher data and update the defect determination model;
an evaluation unit for evaluating the determination reliability for each type of the defect with respect to the determination result;
a reliability information storage unit for storing reliability information that associates the type of the product to be inspected, the type of the defect, and one or more thresholds of the judgment reliability;
causing a display unit to display information about the defect based on the evaluation result of the determination reliability of the evaluation unit and the one or more thresholds included in the reliability information corresponding to the evaluation result a display control configured to control content;
A server device comprising An inspection support method for supporting product inspection work, comprising:
an image acquisition step of acquiring a product image of the product to be inspected;
The product to be inspected included in the product image acquired in the image acquisition step, using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. a determination step of determining the defect of
an input information acquisition step of acquiring input information required by the user for the determination result of the determination step;
a model updating step of performing machine learning by adding the input information to existing teacher data to update the defect determination model;
an evaluation step of evaluating the determination reliability for each type of the defect with respect to the determination result;
The determination of the evaluation step using a reliability information storage unit for storing reliability information that associates the type of the product to be inspected, the type of the defect, and one or more thresholds of the determination reliability. Display control for controlling the content to be displayed on a display unit for displaying information about the defect based on the reliability evaluation result and the one or more thresholds included in the reliability information corresponding to the evaluation result. a step;
An inspection support method comprising: to the computer,
an image acquisition procedure for acquiring a product image obtained by imaging the product to be inspected;
The product to be inspected included in the product image acquired in the image acquisition procedure, using a defect determination model for determining defects in the product to be inspected included in the product image that has been input. Determination procedure for determining defects in
an input information acquisition procedure for acquiring input information required by the user for the judgment result in the judgment procedure;
A model updating procedure for performing machine learning by adding the input information to existing teacher data and updating the defect determination model;
An evaluation procedure for evaluating the determination reliability for each type of defect with respect to the determination result;
The determination of the evaluation procedure using a reliability information storage unit for storing reliability information that associates the type of the product to be inspected, the type of the defect, and one or more thresholds of the determination reliability. Display control for controlling the content to be displayed on a display unit for displaying information about the defect based on the reliability evaluation result and the one or more thresholds included in the reliability information corresponding to the evaluation result. procedure,
Inspection support program for executing

Images