JP7322560B2 - Program, information processing method and information processing apparatus - Google Patents

Description

The present invention relates to a program, an information processing method, and an information processing apparatus.

In recent years, automation of production lines using image recognition technology and the like has been promoted. Patent Literature 1 discloses an automatic operation device based on image recognition that realizes automatic operation by recording user operations on a GUI (Graphical User Interface) of an operation target application running on a computer and reproducing the recorded operations. It is

Japanese Patent No. 5931806

However, the invention according to Patent Document 1 has a problem that it is impossible to analyze an image or the like in a graph area displayed on an operation screen.

One aspect is to provide a program or the like capable of analyzing an image or the like in a graph area displayed on an operation screen.

A program according to one aspect acquires screen information obtained by capturing a monitor screen that displays information about a device, extracts an image of an object or an image within a graph area in the acquired screen information, and extracts the extracted object A computer connected to the apparatus is caused to execute processing for determining whether or not there is an abnormality in the object based on the image of the object or the image within the graph area.

In one aspect, it is possible to analyze an image or the like in a graph area displayed on the operation screen.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an explanatory diagram showing an outline of an automation system for an inspection device; It is a block diagram which shows the structural example of a server. FIG. 11 is an explanatory diagram showing an example of a record layout of an inspection result DB; It is a block diagram which shows the structural example of an inspection apparatus. FIG. 4 is an explanatory diagram showing an example of a record layout of a measurement data DB; FIG. 4 is an explanatory diagram for explaining the operation of determining the presence/absence of an abnormality with respect to an inspection object; FIG. 4 is an explanatory diagram for explaining an image in a graph area extracted from screen information; 4 is a flow chart showing a processing procedure when determining whether or not there is an abnormality in an inspection object; It is explanatory drawing which outputs the recognition result of the presence or absence of abnormality using a pass/fail judgment model. FIG. 8 is a block diagram showing a configuration example of a server according to Embodiment 2; FIG. 10 is an explanatory diagram for outputting a recognition result of presence/absence of an abnormality using a second pass/fail judgment model; FIG. 12 is a block diagram showing a configuration example of a server according to Embodiment 3; FIG. 4 is an explanatory diagram showing an example of a record layout of a button layout DB; 4 is a flow chart showing a processing procedure when operating an inspection apparatus; 4 is a flow chart showing a processing procedure when operating an inspection apparatus; It is an image figure which shows the monitor screen of an inspection apparatus. FIG. 12 is a block diagram showing a configuration example of an inspection apparatus according to Embodiment 4; It is explanatory drawing which shows an example of the record layout of message DB. FIG. 21 is an explanatory diagram showing an example of a record layout of a button layout DB according to Embodiment 4; FIG. 10 is a flow chart showing a processing procedure when identifying a button to be operated based on the contents of an alert message; FIG. FIG. 4 is an image diagram showing a monitor screen of an inspection device displaying an alert message; FIG. 12 is an explanatory diagram showing an outline of an automation system for an inspection apparatus according to Embodiment 5; 4 is a flowchart showing a processing procedure for outputting screen information and a captured image; FIG. 4 is an image diagram of a screen displaying screen information and a captured image; FIG. 12 is a block diagram showing a configuration example of a server according to Embodiment 6; FIG. 4 is an explanatory diagram showing an example of a record layout of material information DB; FIG. 4 is an explanatory diagram showing an example of a record layout of an operating condition DB; FIG. 10 is an explanatory diagram showing an example of a score DB record layout; FIG. 4 is an explanatory diagram for outputting set values of operating conditions using a set value learning model; 4 is a flowchart showing a processing procedure when outputting an operation signal including setting values of operating conditions; FIG. 10 is an explanatory diagram of outputting setting values of a production apparatus using a second setting value learning model;

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.

(Embodiment 1)
The first embodiment relates to a form of determining whether or not there is an abnormality in an inspection object based on screen information obtained by capturing a monitor screen displaying inspection results of an inspection apparatus. In addition, although this embodiment demonstrates using an inspection apparatus as an example, it does not restrict to this. For example, it may be a device such as a production device, a manufacturing device, a measurement device, a test device, or various other custom devices. FIG. 1 is an explanatory diagram showing an outline of an automation system for an inspection apparatus. The system of this embodiment includes an information processing device 1 and an inspection device 2, and each device transmits and receives information via a network N such as the Internet.

The information processing device 1 is an information processing device that processes, stores, and transmits/receives various information. The information processing device 1 is, for example, a server device, a personal computer, or the like. In the present embodiment, the information processing device 1 is assumed to be a server device, which is replaced with the server 1 for the sake of brevity.

The inspection device 2 is a device that inspects the state, shape, direction, size, height, etc. of an object to be inspected in a production line. The object to be inspected is, for example, a printed circuit board (electronic circuit board), an optical film or optical sheet used in a liquid crystal display device, or a pattern printed matter (a sheet-shaped object such as paper or film on which a pattern is printed). Also good. The inspection device 2 may be, for example, an inspection device that carries a workpiece into a processing facility, measures the height of the workpiece when the workpiece is transferred between processes, and inspects the workpiece for abnormalities. A work is an object to be processed (workpiece) in a production line. A production line holds a single workpiece and assembles it through multiple processes to complete a finished product.

The server 1 according to this embodiment is an information processing device having a remote operation function such as a remote desktop. The server 1 uses the remote control function to capture a monitor screen that displays the inspection result of the inspection device 2, and acquires screen information from the captured monitor screen. In this embodiment, an example in which the monitor and the inspection device 2 are integrated is shown, but the monitor is not limited to this, and a monitor of a personal computer connected to the inspection device 2 may be used.

The server 1 extracts an image of an object to be inspected (for example, a photographed image) in the acquired screen information, or an image within a graph (for example, waveform data) indicating an inspection result. The server 1 determines whether or not there is an abnormality in the inspection object based on the extracted image of the inspection object or the image within the graph area. The server 1 recognizes a button (for example, "OK" or "NG" button) for inputting the presence or absence of an abnormality from the acquired screen information.

The server 1 transmits (outputs) an operation signal for a corresponding button to the inspection device 2 according to the result of determining whether or not there is an abnormality in the inspection object. The inspection device 2 executes the corresponding operation according to the button operation signal received from the server 1 . For example, in the inspection of non-defective products or defective products, when the server 1 outputs the operation signal of the "NG" button to the inspection device 2, the inspection device 2 may perform an operation to discharge the work to the defective product line. This eliminates the need for an operator's operation, modification of the inspection apparatus 2 (for example, introduction of software for automatic operation), or the like. The remote server 1 analyzes the screen information of the inspection device 2 and remotely outputs an operation instruction to automatically operate the inspection device 2 .

FIG. 2 is a block diagram showing a configuration example of the server 1. As shown in FIG. The server 1 includes a control section 11 , a storage section 12 , a communication section 13 , an input section 14 , a display section 15 , a reading section 16 and a mass storage section 17 . Each configuration is connected by a bus B.

The control unit 11 includes arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). , various information processing, control processing, etc. related to the server 1 are performed. Note that although FIG. 2 illustrates the controller 11 as a single processor, it may be a multiprocessor.

The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data necessary for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data and the like necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the inspection device 2 via the network N.

The input unit 14 is an input device such as a mouse, keyboard, touch panel, buttons, etc., and outputs received operation information to the control unit 11 . The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11 .

The reader 16 reads a portable storage medium 1a including CD (Compact Disc)-ROM or DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the large-capacity storage unit 17 . Alternatively, the control unit 11 may download the control program 1P from another computer via the network N or the like and store it in the large-capacity storage unit 17 . Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

The large-capacity storage unit 17 includes a recording medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive). The large-capacity storage unit 17 includes an inspection result DB 171 and a pass/fail judgment model 172 . The inspection result DB 171 stores inspection results obtained by determining whether or not there is an abnormality in an inspection object. The pass/fail determination model 172 is a detector that detects (determines) the presence or absence of an abnormality from the image of the object to be inspected or the image within the graph area, and is a learned model (image recognition model) generated by machine learning.

In addition, in this embodiment, the storage unit 12 and the large-capacity storage unit 17 may be configured as an integrated storage device. Also, the large-capacity storage unit 17 may be composed of a plurality of storage devices. Furthermore, the large-capacity storage unit 17 may be an external storage device connected to the server 1 .

In this embodiment, the server 1 is described as a single information processing device, but it may be distributed to a plurality of devices, or may be composed of virtual machines.

FIG. 3 is an explanatory diagram showing an example of the record layout of the inspection result DB 171. As shown in FIG. The inspection result DB 171 includes a management ID column, a monitor screen column, an inspection result column, a detail column, and an inspection date and time column. The management ID column stores a unique inspection result ID to identify each inspection result. The monitor screen column stores screen information obtained by capturing the monitor screen displaying the inspection result of the inspection apparatus 2 . The inspection result column stores inspection results obtained by determining whether or not there is an abnormality in the inspection object based on the captured screen information. The detail column stores detailed descriptive information for inspection results. The inspection date and time column stores date and time information when the inspection object was inspected.

FIG. 4 is a block diagram showing a configuration example of the inspection device 2. As shown in FIG. The inspection device 2 includes a control section 21 , a storage section 22 , a position detection section 23 , an input section 24 , a display section 25 , a communication section 26 , a transport mechanism 27 , a reading section 28 and a large capacity storage section 29 . Each piece of hardware is connected by a bus B.

The control unit 21 includes an arithmetic processing unit such as a CPU, MPU, GPU, etc., and reads and executes the control program 2P stored in the storage unit 22 to perform various information processing, control processing, and inspection processing related to the inspection apparatus 2. etc. In addition, although FIG. 4 illustrates the control unit 21 as a single processor, it may be a multiprocessor. The storage unit 22 includes memory elements such as RAM and ROM, and stores the control program 2P or data necessary for the control unit 21 to execute processing. The storage unit 22 also temporarily stores data and the like necessary for the control unit 21 to perform arithmetic processing.

The position detection unit 23 detects the traveling distance, traveling direction, angle, and the like of the inspection object from a predetermined reference position. For example, positional information may be detected by using a distance sensor for detecting distance, an axis angle sensor for detecting horizontal orientation, and a tilt sensor for detecting vertical tilt angle.

The input unit 24 is an input device such as a mouse, keyboard, touch panel, buttons, etc., and outputs received operation information to the control unit 21 . The display unit 25 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 21 . The communication unit 26 is a communication module for performing processing related to communication, and transmits/receives information to/from the server 1 or the like via the network N.

The transport mechanism 27 includes a transport conveyor 271 , a conveyor motor 272 and a sorting device 273 and is driven by the controller 21 . Specifically, the control unit 21 obtains power supply from a battery (not shown) in response to various operation signals and executes movement (transportation) control. The transport conveyor 271 transports the work along the transport rails to inspect the work on the production line. The conveyor motor 272 is a motor control circuit that drives the transport conveyor 271 and controls the transport speed of the transport conveyor 271 and the like. The sorting device 273 discharges the inspection objects that have been determined to be abnormal by the control unit 21 among the inspection objects that have come out from the conveyer 271 to the defective product line.

The reader 28 reads the portable storage medium 2a including CD-ROM or DVD-ROM. The control unit 21 may read the control program 2P from the portable storage medium 2a via the reading unit 28 and store it in the large-capacity storage unit 29 . Alternatively, the control unit 21 may download the control program 2P from another computer via the network N or the like and store it in the large-capacity storage unit 29 . Furthermore, the control unit 21 may read the control program 2P from the semiconductor memory 2b.

The large-capacity storage unit 29 includes a recording medium such as an HDD, SSD, or the like. A measurement data DB 291 is stored in the large-capacity storage unit 29 . The measurement data DB 291 stores various inspection data for determining the presence or absence of an abnormality in the inspection object.

Note that the storage unit 22 and the large-capacity storage unit 29 may be configured as an integrated storage device in this embodiment. Also, the large-capacity storage unit 29 may be composed of a plurality of storage devices. Furthermore, the large-capacity storage unit 29 may be an external storage device connected to the inspection device 2 .

FIG. 5 is an explanatory diagram showing an example of the record layout of the measurement data DB 291. As shown in FIG. The measurement data DB 291 includes a measurement ID column, a work ID column, a measurement item column, and a measurement date/time column. The measurement ID column stores a unique measurement data ID to identify each measurement data. The work ID column stores unique work IDs for identifying each work. The measurement item column includes a height column, an orientation column and an inclination angle column. The height column stores the measured height of the workpiece. The orientation column stores orientation data of the measured workpiece (for example, azimuth angle values). The tilt angle column stores the measured tilt angle values of the workpiece. The measurement date and time column stores information on the date and time when the work was measured.

FIG. 6 is an explanatory diagram for explaining the operation of determining the presence/absence of an abnormality in an inspection object. The control unit 11 of the server 1 connects to the inspection device 2 via a network N (for example, the Internet) using a remote control function such as remote desktop. The control unit 11 captures the monitor screen displaying the inspection result of the inspection device 2 connected by the remote desktop via the communication unit 13, and acquires screen information from the captured monitor screen.

The control unit 11 extracts the image of the inspection object in the acquired screen information or the image in the graph area using a local feature quantity extraction method such as A-KAZE (Accelerated KAZE). If the graph display area is known in advance, a pixel area corresponding to the graph display area may be extracted. The image of the inspection object may be, for example, a photographed image (photograph) of the inspection object. The image in the graph area is a graph showing the inspection results, and may be, for example, a graph of waveform data obtained by measuring the height of the inspection object.

The control unit 11 determines whether or not there is an abnormality in the inspection object based on the extracted image of the inspection object or the image within the graph area. Specifically, the control unit 11 inputs the extracted image of the object to be inspected or the image within the graph area to the pass/fail determination model 172 constructed by deep learning. The control unit 11 uses the pass/fail determination model 172 to output the recognition result of the presence/absence of abnormality with respect to the inspection object. In the present embodiment, an image of a target area (for example, a graph area) is extracted from screen information and the image of the extracted target area is input to the pass/fail determination model 172, but the present invention is not limited to this. For example, the entire captured image of the monitor screen may be input to the pass/fail judgment model 172 as it is. The processing for determining whether or not there is an abnormality using the pass/fail determination model 172 will be described in detail in the second embodiment.

The control unit 11 stores the inspection results obtained by determining whether or not there is an abnormality in the inspection object in the inspection result DB 171 of the large-capacity storage unit 17 . Specifically, the control unit 11 assigns a management ID, and stores the monitor screen, the inspection result, the detailed explanation information for the inspection result, and the inspection date and time as one record in the inspection result DB 171 .

Based on the acquired screen information, the control unit 11 recognizes a button for inputting whether or not there is an abnormality. The buttons for inputting the presence or absence of abnormality are, for example, "OK" and "NG" buttons, and are buttons for receiving corresponding operation instructions according to the result of determining the presence or absence of abnormality. As for the button recognition processing, for example, a well-known image analysis method such as extracting an edge (portion with a large change in pixel value) from a captured image of the monitor screen to specify the shape of the button may be used. When the monitor screen includes a plurality of buttons, the control unit 11 recognizes all of these buttons and identifies the function assigned to each recognized button.

Specifically, the control unit 11, for example, recognizes the shape of each button and the positional relationship between each button, and the shape and each pattern of each button pattern arranged in the operation panel configuration diagram stored in the storage unit 12 in advance. Compare the positional relationship between Based on the comparison result of the positional relationship, the control unit 11 identifies which of the button patterns arranged in the operation panel configuration diagram corresponds to each recognized button. Further, when a button ID (identifier) is assigned to each button arranged in the operation panel configuration diagram as described above, the correspondence relationship between each recognized button and each button pattern arranged in the operation panel configuration diagram is If it can be specified, the button ID of each recognized button can be determined.

The control unit 11 identifies the corresponding button from the plurality of recognized buttons according to the result of determining whether or not there is an abnormality in the inspection object. For example, when determining that an abnormality has occurred in the inspection object, the control unit 11 may specify an "NG" button corresponding to the abnormality. The control unit 11 transmits an operation signal of the specified button to the inspection device 2 via the communication unit 13 . The operation signal may be, for example, a button ID (serial number) or coordinate information (for example, 10 pixels to the right and 5 pixels down from the upper left corner of the screen).

The control unit 21 of the inspection device 2 receives a button operation signal from the server 1 via the communication unit 26, and outputs an operation command (for example, reexamination, etc.) according to the received operation signal. Specifically, based on the received operation signal, the control unit 21 moves the mouse cursor to the corresponding button to generate a click (touch) event.

FIG. 7 is an explanatory diagram for explaining an image in a graph area extracted from screen information. Although an example of an image showing a waveform graph will be described with reference to FIG. 7, the present invention is not limited to this. For example, a line graph, bar graph, or the like may be used.

FIG. 7A shows a normal waveform in which no abnormality has occurred in the inspection object. The horizontal axis indicates the measurement position (horizontal direction) of the work to be inspected, and the vertical axis indicates the height of the work in millimeters (mm). When the height of the workpiece is within the range of normal values, waves of the same width are repeated periodically, and the peak value of the waves is suppressed within a predetermined fluctuation range of the threshold value (for example, threshold value ±1 mm).

FIG. 7B shows a bad waveform in the presence of large waves. The control unit 11 of the server 1 extracts the peak value (crest value) of the wave and compares the extracted peak value with the fluctuation range of the threshold. As shown in the figure, there is a peak value that exceeds the variation range of the threshold, so the control unit 11 determines that an abnormality has occurred with respect to the workpiece.

FIG. 7C shows a tilted fault waveform. The control unit 11 extracts the peak value of the wave and calculates the slope of the approximate straight line formed by the extracted peak value. When the controller 11 determines that the calculated inclination exceeds a predetermined threshold value, it determines that an abnormality has occurred with respect to the workpiece.

FIG. 7D shows a defective waveform with uneven peak spacing. The control unit 11 acquires peak intervals (periods) and determines whether or not the acquired intervals are uniform. When the control unit 11 determines that the peak intervals are uneven, it determines that an abnormality has occurred in the workpiece.

In the second embodiment, which will be described later, the image of the waveform graph described above can be input to the learned pass/fail determination model 172, and the recognition result of recognizing (determining) the presence or absence of an abnormality in the inspection object can be output. Details of the processing will be described later.

FIG. 8 is a flow chart showing a processing procedure for judging whether or not an inspection object has an abnormality. The control unit 11 of the server 1 connects to the inspection apparatus 2 using a remote desktop via the network N (step S101). The control unit 11 captures a monitor screen displaying the inspection result of the inspection device 2 connected by remote desktop via the communication unit 13 (step S102). The control unit 11 acquires screen information from the captured monitor screen (step S103).

The control unit 11 extracts an image within the graph area within the acquired screen information using a local feature amount extraction method such as A-KAZE (step S104). Although an example of an image within the graph area has been described in FIG. 8, the present invention is not limited to this. For example, when an image of an object to be inspected is displayed on a monitor screen, the image of the object to be inspected may be extracted.

The control unit 11 determines whether or not there is an abnormality in the inspection object based on the extracted image in the graph area (step S105). For example, when the image in the extracted graph area is an image of a bad waveform when there is a large wave as shown in FIG. . When the waveform is inclined as shown in FIG. 7C, the control unit 11 extracts the peak value of the wave and calculates the inclination of the approximate straight line formed by the extracted peak value. The control unit 11 compares the calculated tilt with a predetermined threshold value. When the controller 11 determines that the tilt exceeds the threshold, it determines that an abnormality has occurred in the inspection object. The control unit 11 stores the inspection result obtained by determining whether or not there is an abnormality in the inspection object in the inspection result DB 171 of the large-capacity storage unit 17 (step S106).

Based on the acquired screen information, the control unit 11 recognizes each button on the monitor screen, for example, by using an image analysis technique for extracting edges and specifying the shape of the button (step S107). The control unit 11 inputs the presence or absence of an abnormality according to the shape of each button pattern arranged in the operation panel configuration diagram stored in the storage unit 12 in advance, the positional relationship between the patterns, and the determination result of the presence or absence of an abnormality. A button (for example, "OK" or "NG") is specified (step S108).

The control unit 11 transmits an operation signal of the specified button to the inspection device 2 via the communication unit 13 (step S109). The control unit 21 of the inspection device 2 receives a button operation signal from the server 1 via the communication unit 26 (step S201). The control unit 21 outputs an operation command (for example, reexamination, etc.) according to the received operation signal (step S202), and ends the process.

According to this embodiment, the remote server 1 can automatically operate the inspection device 2 by analyzing the monitor screen of the inspection device 2 and outputting an operation signal.

(Embodiment 2)
The second embodiment relates to a mode of outputting a recognition (judgment) result when an image within a graph area is input using the pass/fail judgment model 172 . In addition, description is abbreviate|omitted about the content which overlaps with Embodiment 1. FIG.

FIG. 9 is an explanatory diagram for outputting the result of recognition of the presence or absence of abnormality using the pass/fail judgment model 172. As shown in FIG. The pass/fail judgment model 172 is constructed by deep learning and used as a program module that is part of artificial intelligence software. The pass/fail determination model 172 is an extractor that has already constructed (generated) a neural network that receives an image in the graph area as an input and outputs a recognition result of recognizing the presence or absence of an abnormality in the inspection object.

The control unit 11 of the server 1 constructs (generates) the pass/fail judgment model 172 by performing deep learning for learning the feature amount of the waveform graph in the image within the graph area. For example, the pass/fail judgment model 172 is a CNN (Convolution Neural Network), and includes an input layer that receives an input of an image within a graph area, an output layer that outputs the result of recognizing the presence or absence of an abnormality, and a trained model by back propagation. and an intermediate layer.

The input layer has a plurality of neurons that receive input of pixel values of pixels included in the image within the graph area, and passes the input pixel values to the intermediate layer. The intermediate layer has a plurality of neurons for extracting image features in the graph area, and passes the extracted image features to the output layer. For example, a case where the pass/fail judgment model 172 is CNN will be described as an example. The intermediate layer consists of a convolution layer that convolves the pixel values of each pixel input from the input layer and a pooling layer that maps the pixel values convolved in the convolution layer, which are alternately connected. While compressing the pixel information of the image, the feature amount of the image is finally extracted. After that, the hidden layer predicts whether or not the image in the graph area has an abnormality with respect to the inspection target by the fully connected layer whose parameters have been learned by back propagation. The prediction results are output to an output layer having multiple neurons.

Note that the image in the graph area may be input to the input layer after passing through alternately connected convolution layers and pooling layers to extract feature amounts.

Instead of CNN, using any object detection algorithm such as RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), or YOLO (You Only Look Once) Also good.

The control unit 11 of the server 1 learns using teacher data in which images in a plurality of graph regions (for example, graph images of normal waveforms and graph images of abnormal waveforms) are associated with the presence or absence of abnormality in each image. I do. The training data is data in which the presence or absence of abnormality is labeled with respect to the image within the graph area. The control unit 11 generates teacher data for learning the presence or absence of abnormality by labeling the presence or absence of abnormality for images in a plurality of graph regions. Specifically, the control unit 11 generates teacher data that associates a unique name indicating the presence or absence of abnormality (for example, normal or abnormal) with the image in the graph area.

The control unit 11 inputs an image in the graph area, which is teacher data, to the input layer, performs arithmetic processing in the intermediate layer, and acquires the identification result indicating the presence or absence of abnormality from the output layer. The identification result output from the output layer may be a value discretely indicating the presence or absence of an abnormality (for example, a value of "0" or "1"), or a continuous probability value (for example, from "0" to " value in the range up to 1”).

The control unit 11 compares the identification result output from the output layer with the information labeled for the image in the graph area in the teacher data, that is, the correct value, so that the output value from the output layer approaches the correct value. Second, optimize the parameters used for arithmetic processing in the intermediate layer. The parameters are, for example, weights (coupling coefficients) between neurons, coefficients of activation functions used in each neuron, and the like. Although the parameter optimization method is not particularly limited, for example, the control unit 11 optimizes various parameters using the error backpropagation method.

The control unit 11 performs the above processing on the image in each graph area included in the teacher data, and generates the pass/fail determination model 172 . As a result, for example, the control unit 11 learns the pass/fail determination model 172 using the teacher data, thereby constructing a model capable of recognizing the presence or absence of an abnormality in the inspection object.

When acquiring an image within the graph area, the control unit 11 inputs the acquired image to the pass/fail determination model 172 . The control unit 11 performs arithmetic processing for extracting the feature amount of the image in the intermediate layer of the pass/fail judgment model 172, inputs the extracted feature amount into the output layer of the pass/fail judgment model 172, and detects an abnormality in the inspection object. The identification result indicating the presence or absence of is obtained as an output. Based on the identification result output from the output layer of the pass/fail judgment model 172, the control unit 11 judges whether or not there is an abnormality in the inspection object. For example, when the normality probability value (for example, 0.97) is equal to or greater than a predetermined threshold value (for example, 0.95), the control unit 11 may determine that no abnormality has occurred.

It should be noted that the above-described machine learning is not limited to the process of determining the presence or absence of an abnormality in an inspection object. For example, A-KAZE (Accelerated KAZE), SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), HOG (Histograms of Oriented Gradients) and other local feature extraction methods may be used to determine the presence or absence of an abnormality.

In this embodiment, an example in which an image within the graph area is input to the pass/fail determination model 172 has been described, but the present invention is not limited to this. The entire captured image of the monitor screen may be input to the pass/fail determination model 172 as it is.

<Modification 1>
Next, a process for outputting recognition results classified into abnormal types will be described. FIG. 10 is a block diagram showing a configuration example of the server 1 of the second embodiment. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 2, and description is abbreviate|omitted.

A second pass/fail judgment model 173 is stored in the large-capacity storage unit 17 . The second pass/fail judgment model 173 is constructed by deep learning and used as a program module that is part of artificial intelligence software. The second pass/fail judgment model 173 is an extractor that has already constructed (generated) a neural network that receives an image in the graph area as an input and outputs a recognition result in which the type of abnormality is classified for the object to be inspected. The abnormality type may be, for example, abnormality level information classified according to the waveform of the image in the graph area (for example, wave width, peak interval, inclination angle, etc.).

FIG. 11 is an explanatory diagram for outputting the result of recognition of the presence/absence of abnormality using the second pass/fail judgment model 173. As shown in FIG. Note that the description of the content that overlaps with FIG. 9 will be omitted. The control unit 11 of the server 1 provides teacher data in which a plurality of graph images of normal waveforms (images within the graph area) and graph images of various abnormal waveforms are associated with the presence or absence of abnormality in each image and information on the level of abnormality. learning using The teacher data is data in which the presence or absence of abnormality and abnormality level information are labeled with respect to the image in the graph area.

The control unit 11 labels a plurality of graph images of normal waveforms and graph images of various abnormal waveforms with information on the presence or absence of anomalies and information on the level of anomalies, thereby providing teacher data for learning the presence or absence of anomalies and the anomaly levels. Generate. Specifically, the control unit 11 generates teacher data that associates the presence or absence of an abnormality with the graph image described above and the unique names of the abnormality levels (for example, normal, abnormality level 1, abnormality level 2, etc.).

The control unit 11 constructs (generates) the second pass/fail judgment model 173 by performing deep learning for learning the feature amount of the teacher image using the generated teacher data. Note that the process of constructing the second pass/fail judgment model 173 using the teacher data is the same as the process of constructing the pass/fail judgment model 172, so the explanation is omitted.

The input layer of the second pass/fail judgment model 173 accepts the input of the image in the graph area, and then the intermediate layer converts the image in the graph area to the inspection target by the fully connected layer whose parameters have been learned by back propagation. Presence or absence of abnormality and abnormality level information are predicted. A recognition result including the predicted presence/absence of abnormality and abnormality level information is output to an output layer having a plurality of neurons. As shown in the figure, when an inclined defective waveform is input to the second pass/fail determination model 173, a recognition result including the presence/absence of an abnormality and an abnormality level is output.

The recognition result is, for example, a normal probability value, an abnormal level 1 probability value, an abnormal level 2 probability value, and an abnormal level 3 probability value. Based on the identification result output from the output layer of the second pass/fail judgment model 173, the control unit 11 judges the presence or absence of an abnormality in the object to be inspected and the level of abnormality when the abnormality occurs. As shown in the figure, when the probability value (for example, 0.88) of abnormality level 3 is equal to or greater than a predetermined threshold value (for example, 0.80), the control unit 11 determines that the abnormality type is abnormality level 3. good.

In this embodiment, an example of an image (waveform data) in a graph area representing the height of an inspection object will be described, but the same applies to an image of an inspection object. For example, when measuring the hole diameter of an object to be inspected, an image (photograph) of the object to be inspected may be input to the third pass/fail judgment model, and a recognition result recognizing whether or not there is an abnormality in the hole diameter may be output. . The control unit 11 of the server 1 generates teacher data for learning the presence or absence of an abnormality by labeling the images of a plurality of inspection objects with the presence or absence of an abnormality. Specifically, the control unit 11 generates teacher data in which proper names of normal holes and abnormal holes (for example, normal holes, abnormal holes, etc.) are associated with the image of the inspection object. The control unit 11 constructs a third pass/fail judgment model by performing deep learning for learning the feature amount of the teacher image using the generated teacher data.

The control unit 11 uses the constructed third pass/fail judgment model to judge the presence/absence of an abnormality in the inspection object. Specifically, the input layer of the third pass/fail judgment model receives the input of the image of the inspection object, and then the intermediate layer calculates the hole diameter of the inspection object by the fully connected layer whose parameters are learned by back propagation Predict the presence or absence of abnormalities for A recognition result including the presence or absence of the predicted abnormality is output to an output layer having a plurality of neurons.

According to this embodiment, it is possible to output the recognition result of the presence or absence of an abnormality using the learned pass/fail judgment model 172 that outputs the recognition result when the image of the inspection object or the image within the graph area is input. becomes.

According to this embodiment, the learned second pass/fail judgment model 173 that outputs the recognition result when the image of the inspection object or the image in the graph area is input is used to output the recognition result classified by the type of abnormality. It becomes possible to

According to this embodiment, it is possible to improve the accuracy of determining the presence or absence of an abnormality in an inspection target using a pass/fail determination model (image recognition model) constructed by deep learning.

(Embodiment 3)
The third embodiment relates to a form of recognizing operation buttons in each stage of inspection of the inspection device 2 from screen information obtained by capturing the monitor screen of the inspection device 2 . Note that the description of the contents overlapping those of the first and second embodiments will be omitted.

Note that the inspection device 2 in this embodiment is a device operated in four steps of "carrying in work", "executing measurement", "judgment of measurement result" and "ejection of work", but it is limited to this. not a thing For example, an operation step may be provided according to a line production method for assembling a product, arranging workers, and attaching parts.

FIG. 12 is a block diagram showing a configuration example of the server 1 of the third embodiment. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 10, and description is abbreviate|omitted. A button layout DB 174 is stored in the large-capacity storage unit 17 . The button arrangement DB 174 stores information such as the shape of each button arranged on the monitor screen and the positional relationship between the buttons.

FIG. 13 is an explanatory diagram showing an example of the record layout of the button layout DB 174. As shown in FIG. The button layout DB 174 includes a button ID column, name column, coordinate column, relative position column and shape column. The button ID column stores a unique button ID to identify each button. The name column stores the name of the button. The coordinate column stores coordinate values indicating the positions of buttons on the monitor screen. The relative position column stores the positional relationship and the like between each button. The shape column stores the shape of the button.

14 and 15 are flow charts showing the processing procedure when the inspection device 2 is operated. The control unit 11 of the server 1 captures the monitor screen displaying the inspection results of the inspection device 2 connected by remote desktop via the communication unit 13, and acquires screen information from the captured monitor screen (step S111). .

The control unit 11 recognizes each button on the screen based on the acquired screen information (step S112). Specifically, the control unit 11 uses a well-known image analysis method to extract edges (portions with large changes in pixel values) from the screen information. Note that when the screen transitions, the control unit 11 may re-recognize each button on the screen after the transition. The control unit 11 identifies the shape, coordinates, relative position, etc. of the button based on the extracted edges. The control unit 11 collates the button layout DB 174 of the large-capacity storage unit 17 based on the shape, coordinates, relative position, etc. of the specified button, and acquires each button ID. Note that the control unit 11 may recognize characters on the button and extract a button ID having a name matching the recognized characters from the button layout DB 174 .

The control unit 11 identifies the carry-in button by the acquired button ID (step S113). The control unit 11 transmits an operation signal of the specified carry-in button to the inspection apparatus 2 via the communication unit 13 (step S114). The control unit 21 of the inspection apparatus 2 receives the carry-in button operation signal transmitted from the server 1 via the communication unit 26 (step S211), and outputs a transport command to the transport conveyor 271 (step S212).

The control unit 11 of the server 1 identifies the measurement button based on the obtained button ID (step S115). The control unit 11 transmits the operation signal of the identified measurement button to the inspection device 2 via the communication unit 13 (step S116). The control unit 21 of the inspection device 2 receives the measurement button operation signal transmitted from the server 1 via the communication unit 26 (step S213), and outputs a measurement command for measuring the workpiece (step S214). The control section 21 stores the measurement data for each workpiece in the measurement data DB 291 of the large-capacity storage section 29 . Specifically, the control unit 21 assigns a measurement ID, and stores the work ID, the measurement item (for example, height), and the measurement date and time as one record in the measurement data DB 291 .

The control unit 11 of the server 1 extracts an image within the graph area from the acquired screen information (step S117), and determines whether or not there is an abnormality in the inspection object (step S118). Note that the process of extracting an image in the graph area and the process of determining the presence or absence of an abnormality are the same as those in the first or second embodiment, so description thereof will be omitted.

If the control unit 11 determines that an abnormality has occurred in the inspection object (YES in step S119), the NG button is specified by the acquired button ID (step S120). The control unit 11 transmits an operation signal of the specified NG button to the inspection device 2 via the communication unit 13 (step S121). The control unit 21 of the inspection apparatus 2 receives the NG button operation signal transmitted from the server 1 via the communication unit 26 (step S216). The control unit 21 outputs a command to the sorting device 273 to discharge the work to the defective product line (step S217).

If the control unit 11 determines that no abnormality has occurred in the inspection object (NO in step S119), it identifies the OK button based on the acquired button ID (step S122). The control unit 11 transmits an operation signal of the identified OK button to the inspection device 2 via the communication unit 13 (step S123). The control unit 21 of the inspection device 2 receives the OK button operation signal transmitted from the server 1 via the communication unit 26 (step S218). The control unit 21 outputs a command to the sorting device 273 to discharge the work to the non-defective line (step S219), and ends the process.

FIG. 16 is an image diagram showing the monitor screen of the inspection device 2. As shown in FIG. The "carry in" button 16a accepts a workpiece carry-in command. The "measurement" button 16b accepts a workpiece measurement command. The "OK" button 16c accepts a command to discharge the work to the non-defective line. The "NG" button 16d accepts a command to discharge the work to the defective product line. The image display area 16e displays a photographed image of the workpiece or a graph image showing the inspection result.

The control unit 21 of the inspection apparatus acquires, for example, the measured work height data from the measurement data DB 291 of the large-capacity storage unit 29 . The control unit 21 generates a waveform graph based on the obtained work height data, and causes the display unit 25 to display the generated waveform graph in the image display area 16e.

When receiving a button operation signal from the server 1 via the communication unit 26, the control unit 21 outputs an operation command corresponding to the received operation signal. Specifically, the control unit 21 outputs a transport command to the transport conveyor 271 when receiving the operation signal of the “carry in” button 16a. When the controller 21 receives the operation signal of the "measurement" button 16b, it outputs a measurement command to measure the workpiece.

When receiving the operation signal of the "OK" button 16c, the control unit 21 outputs a command to the sorting device 273 to discharge the works to the non-defective line. When receiving the operation signal of the "NG" button 16d, the control unit 21 outputs a command to the sorting device 273 to discharge the work to the defective product line.

In this embodiment, an example in which the operation signal for each button is received from the server 1 has been described, but the present invention is not limited to this. For example, an operation signal for each button may be received by an operator's touch (click) operation.

According to this embodiment, it is possible to recognize the operation buttons at each stage in the inspection of the inspection apparatus 2 .

According to this embodiment, by using the remote operation function to output the operation signal of each button, the operator's operation becomes unnecessary, and it is possible to realize a reduction in labor costs.

(Embodiment 4)
The fourth embodiment relates to a form of specifying a button to be operated according to the contents of an alert message extracted from screen information obtained by capturing the monitor screen of the inspection device 2 . Note that the description of the contents overlapping those of the first to third embodiments will be omitted.

FIG. 17 is a block diagram showing a configuration example of the inspection device 2 of the fourth embodiment. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 4, and description is abbreviate|omitted. A message DB 292 is stored in the large-capacity storage unit 29 . The message DB 292 stores various alert messages in inspections of the inspection device 2 .

FIG. 18 is an explanatory diagram showing an example of the record layout of the message DB 292. As shown in FIG. The message DB 292 includes message ID columns and message columns. The message ID column stores unique message IDs for identifying each message. The message column stores the contents of alert messages. Alert messages include error messages, warning messages, notification messages, and the like.

FIG. 19 is an explanatory diagram showing an example of the record layout of the button layout DB 174 according to the fourth embodiment. The button layout DB 174 includes message ID columns and message columns. The message ID column stores alert message IDs associated with button IDs. A plurality of message IDs may be stored in the message ID column. That is, the same operation can be performed according to different message contents. The message column stores the contents of the message.

The control unit 11 of the server 1 captures the monitor screen displaying the inspection results of the inspection device 2 connected by remote desktop via the communication unit 13, and acquires screen information from the captured monitor screen. The control unit 11 recognizes each button based on the acquired screen information. Note that the button recognition processing is the same as that of the third embodiment, so the description is omitted.

The control unit 11 extracts the image of the alert message display area from the acquired screen information. Specifically, for example, the control unit 11 may extract using a local feature extraction method such as SIFT. The control unit 11 extracts the alert message from the extracted image of the alert message display area. Regarding the alert message extraction process, for example, using optical character recognition (OCR) technology, the image of the alert message display area is converted into a string of character codes, and the alert message can be extracted. good.

The control unit 11 collates the button layout DB 174 of the large-capacity storage unit 17 according to the content of the extracted alert message, and acquires the button ID corresponding to the alert message. Based on the acquired button ID, the control unit 11 transmits an operation signal of the button to the inspection device 2 via the communication unit 13 . The control unit 21 of the inspection apparatus 2 receives the button operation signal transmitted from the server 1 via the communication unit 26, and outputs an operation command (for example, remeasurement) according to the received operation signal.

FIG. 20 is a flow chart showing a processing procedure for identifying a button to be operated based on the contents of an alert message. The control unit 11 of the server 1 captures the monitor screen displaying the inspection result of the inspection device 2 connected by the remote desktop, and acquires screen information from the captured monitor screen (step S131). The control unit 11 recognizes each button based on the acquired screen information (step S132).

The control unit 11 extracts the image of the alert message display area from the acquired screen information using a local feature amount extraction method such as A-KAZE (step S133). The control unit 11 uses the optical character recognition technology to extract the alert message from the image of the extracted alert message display area (step S134). The control unit 11 collates the button layout DB 174 of the large-capacity storage unit 17 according to the content of the extracted alert message, and acquires the button ID corresponding to the alert message (step S135).

Based on the obtained button ID, the control unit 11 transmits an operation signal of the button to the inspection device 2 via the communication unit 13 (step S136). The control unit 21 of the inspection apparatus 2 receives the button operation signal transmitted from the server 1 via the communication unit 26 (step S231), and outputs an operation command according to the received operation signal (step S232). .

FIG. 21 is an image diagram showing a monitor screen of the inspection device 2 displaying an alert message. 16 are assigned the same reference numerals, and description thereof will be omitted. 21a is a display area for displaying an alert message. When the control unit 21 of the inspection device 2 detects a notification or an error, it acquires a corresponding alert message from the message DB 292 of the large-capacity storage unit 29 . The control unit 21 displays the acquired alert message on the monitor screen via the display unit 25 .

According to this embodiment, it is possible to extract an alert message from the captured monitor screen of the inspection device 2 using the remote control function.

According to this embodiment, by outputting the operation signal of the button to be operated to the inspection device 2 according to the content of the alert message, it is possible to automatically recover from the failure without intervention of the operator. .

(Embodiment 5)
The fifth embodiment relates to a mode of outputting screen information obtained by capturing a monitor screen and a captured image obtained by an imaging device that captures an image of the inspection device 2 . Note that descriptions of the contents overlapping those of the first to fourth embodiments will be omitted.

FIG. 22 is an explanatory diagram showing an outline of an automation system for an inspection apparatus according to the fifth embodiment. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 1, and description is abbreviate|omitted. In this embodiment, an imaging device 3 for imaging the inspection device 2 is included.

The imaging device 3 captures an image of the inspection device obliquely above the transport conveyor 271 to generate an image. Note that the installation position of the imaging device 3 is not limited to the positions described above. For example, the imaging device 3 may be installed above or beside the transport conveyor 271 . The imaging device 3 of this embodiment includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing processing related to communication, and transmits and receives captured images to and from the server 1 and the like via the network N.

FIG. 23 is a flow chart showing a processing procedure for outputting screen information and a captured image. The control unit 11 of the server 1 captures the monitor screen displaying the inspection results of the inspection device 2 connected by remote desktop via the communication unit 13, and acquires screen information from the captured monitor screen (step S141). . The control unit 11 acquires a captured image of the inspection device 2 from the imaging device 3 via the communication unit 13 (step S142). The control unit 11 causes the display unit 15 to display the acquired screen information and the captured image (step S143), and terminates the process.

FIG. 24 is an image diagram of a screen displaying screen information and a captured image. 16 are assigned the same reference numerals, and description thereof will be omitted. 24 a is a display area for displaying a captured image acquired from an imaging device that captures an image of the inspection device 2 . The control unit 11 of the server 1 acquires the captured image of the inspection object captured by the imaging device 3 via the communication unit 13 in real time. The control unit 11 causes the display unit 15 to display the acquired captured image.

The control unit 11 simultaneously displays the captured monitor screen, the captured image (still image) of the inspection apparatus at the time of capturing the monitor screen, and the captured image (moving image) of the inspection object in real time. 15 may be displayed.

According to this embodiment, it is possible to simultaneously output the screen information obtained by capturing the monitor screen and the captured image obtained by the imaging device that captures the inspection device 2 .

(Embodiment 6)
Embodiment 6 relates to a form of outputting setting values of operating conditions of the inspection apparatus 2 based on information on materials to be inspected. Note that the description of the contents overlapping those of the first to fifth embodiments will be omitted.

FIG. 25 is a block diagram showing a configuration example of the server 1 of the sixth embodiment. 12 are assigned the same reference numerals, and description thereof will be omitted. A set value learning model 175 , a material information DB 176 , an operating condition DB 177 and a score DB 178 are stored in the large-capacity storage unit 17 .

The set value learning model 175 is a trained model generated by reinforcement learning. Reinforcement learning will be described later. The material information DB 176 stores information on materials to be inspected. For example, when the material is glass, information such as the size, thickness, and product standard of the glass may be stored in the material information DB 176 . The operating condition DB 177 stores setting values of operating conditions of the inspection device 2 . The score DB 178 is a table that defines rewards (scores) given in reinforcement learning, which will be described later, and associates product (work) evaluation actions with rewards given when each action is taken for product evaluation. It is a table that stores

FIG. 26 is an explanatory diagram showing an example of the record layout of the material information DB 176. As shown in FIG. The material information DB 176 includes a material ID column, a type column, a size column, a thickness column, a product name column and a product specification column. The material ID column stores a unique material ID to identify each material. The type column stores the type of material (for example, glass). The size column stores the size of the material. The thickness column stores the thickness of the material. The product name column stores the name of the product (work) produced using the material. The product standard column stores standard information that defines the shape, size, material, composition, quality, performance, durability, safety, function, etc. of the product (work) produced using the material.

FIG. 27 is an explanatory diagram showing an example of the record layout of the operating condition DB 177. As shown in FIG. The operating condition DB 177 includes an operating condition ID column, a measurement position column, a measurement acceptance range column, a measurement speed column, a measurement accuracy column, and a measurement pressure column. The operating condition ID column stores an ID of a uniquely specified operating condition in order to identify each operating condition. The measurement position column stores measurement positions with respect to the inspection object. The pass range column of measurement stores the upper and lower variation ranges (errors) of the pass threshold. The measurement speed column stores the speed at which the inspection object is measured. The measurement accuracy column stores the accuracy when measuring the inspection object. The measured pressure column stores the pressure used when measuring the inspection object.

FIG. 28 is an explanatory diagram showing an example of the record layout of the score DB 178. As shown in FIG. Score DB 178 includes a status column and a score column. The status column stores product evaluation behaviors (statuses) as criteria for rewarding in reinforcement learning. In the present embodiment, the status of the product (eg, glass bottle) evaluation as a criterion for rewarding includes the weight, thickness, shape, size, heat resistance, durability, and safety of the product. You have defined a status. For example, two levels for heat resistance may be provided. If the product achieves level 1 of heat resistance, it receives a score of 50, and if it achieves level 2 of heat resistance, it receives a score of 80. The score column stores scores as reward values for reinforcement learning in association with product evaluation statuses.

Note that the storage form of each DB described above is an example, and other storage forms may be used as long as the relationship between data is maintained.

In this embodiment, the control unit 11 of the server 1 uses a reinforcement learning method to set operating conditions for the inspection device 2 based on the material information to be inspected stored in the material information DB 176 of the large-capacity storage unit 17. perform processing to learn (change) the priority of

Reinforcement learning (DQN: Deep Q-Network) is one of the machine learning methods, and is a learning method for selecting actions (a; Actions) to be taken based on the current state (s; States). . More specifically, reinforcement learning evaluates the policy from the obtained reward (r; Reward) when taking an action in a certain state based on a certain policy (π; Policy), and It is a method to improve the policy so as to maximize the cumulative value of the reward given. In reinforcement learning, for example, the following formulas (1) and (2) are used to evaluate and improve a policy.

Equation (1) is the state value function, the function that defines the value of state s under policy π. Rt is a function (Reward Function) that expresses the final accumulated reward, and is expressed by a reward r at a certain point in time t and a discount factor γ (discount factor) for discounting and evaluating the reward r obtained in the distant future. be done. As shown in Equation (1), the state value function is expressed by the expected value of the accumulated reward Rt that would be obtained in a certain state s.

Equation (2) is the action-value function, which defines the value of taking action a in state s under policy π. Like the state value function, the action value function is also expressed by the expected value of the cumulative reward Rt.

In reinforcement learning, each function represented by equations (1) and (2) is learned to be maximized (optimized) as shown by the following equations (3) and (4).

Putting together the equations (3) and (4), the following equation (5) is obtained.

In reinforcement learning, expression (5) is used as an objective function, and the objective function is optimized. Specifically, the objective function is optimized using, for example, the Monte Carlo method, Q-learning, policy gradient method, or the like. Since these optimization algorithms are well-known, further detailed description will be omitted.

The control unit 11 of the server 1 uses the above-described reinforcement learning to perform processing for learning the priority of the setting values of the operating conditions for the inspection device 2 . Specifically, the control unit 11 sets the material information to be inspected stored in the material information DB 176 of the large-capacity storage unit 17 to the state s, and sets the operating conditions for outputting various set values to the inspection device 2 to the action a. , reinforcement learning based on rewards (scores) defined in the score DB 178 is performed.

The control unit 11 of the server 1 inputs the material information to be inspected as the state s to the objective function represented by the formula (5). The control unit 11 refers to the score DB 178 and calculates a reward value based on the evaluation behavior for the product (work) inspected using the inspection device 2 . The score DB 178 prescribes the reward r to be given when the evaluation action of each status of the product is taken, using the evaluation action of the product inspected using the inspection device 2 as a criterion. Specifically, as described with reference to FIG. 28, the score DB 178 stores a score as a reward r for reinforcement learning in association with each status representing product evaluation behavior.

In the score DB 178, for example, two statuses of "Achieved thickness level 1" and "Achieved thickness level 2" for product thickness may be defined. A score of "30" is defined for "achieved thickness level 1" and a score of "45" is defined for "achieved thickness level 2". That is, when the product thickness reaches thickness level 1 (eg, 3 mm to 3.10 mm), 30 points are given as a reward r. If the product thickness does not reach both thickness level 1 and thickness level 2 (the product thickness is outside the reference range), 0 points are given as reward r. Alternatively, minus points (eg, -10 points) may be given as a reward r in accordance with a low evaluation. Accordingly, the cumulative reward Rt is calculated based on the evaluation behavior of the product corresponding to the status column of the score DB 178.

The control unit 11 optimizes the objective function based on the reward calculated as described above. That is, the control unit 11 sets the operating condition for outputting various set values for the inspection device 2 as action a, the material information to be inspected as state s, and the cumulative reward Rt calculated based on the evaluation action for the product. Then, optimization of the Q function represented by Equation (5) is performed.

As described above, the control unit 11 uses the material information (state s) to be inspected as input values, and obtains the operating conditions (behavior a) for outputting various set values for the inspection device 2 as output values. A set value learning model 175 is generated.

FIG. 29 is an explanatory diagram for outputting setting values of operating conditions using the setting value learning model 175. FIG. When the material information (state s) to be inspected is input to the set value learning model 175 that has undergone reinforcement learning, the control unit 11 outputs the operating conditions (behavior a) of the inspection device 2 . As shown, the state s is determined by the size, thickness, shape and composition of the material under inspection. The set value learning model 175 outputs a Q value corresponding to each action a (eg, a1, a2, a3) from state s. The output action a is, for example, set values of measurement position, measurement speed, measurement accuracy, and measurement pressure. It should be noted that environmental information such as humidity and temperature around the inspection apparatus may be included in the state s for learning.

The control unit 11 extracts the Q value having the highest value, and transmits (outputs) an operation signal including a setting value corresponding to the extracted Q value to the inspection device 2 . For example, the control unit 11 transmits an operation signal including the measurement speed to the inspection device 2 via the communication unit 13 . The control unit 21 of the inspection device 2 receives the operation signal transmitted from the server 1 via the communication unit 26 and outputs an operation command based on the received operation signal.

FIG. 30 is a flow chart showing a processing procedure when outputting an operation signal including setting values of operating conditions. The control unit 11 of the server 1 acquires material information from the inspection device 2 via the communication unit 13 (step S151). The control unit 11 stores the acquired material information in the material information DB 176 of the large-capacity storage unit 17 (step S152). Specifically, the control unit 11 assigns a material ID, and stores the type, size, thickness, product name, and product standard as one record in the material information DB 176 .

The control unit 11 inputs the acquired material information to the setting value learning model 175 (step S153). The control unit 11 acquires the setting values of the operating conditions of the inspection device 2 (step S154). Specifically, as described above, the control unit 11 uses the set value learning model 175 that is learned to output the set values of the operating conditions of the inspection device 2 when the material information is input using the reinforcement learning technique. do. The control unit 11 inputs information about the material to be inspected to the set value learning model 175 and outputs set values for operating conditions of the inspection device 2 .

The control unit 11 transmits an operation signal including the acquired setting values of the operating conditions to the inspection device 2 (step S155). The control unit 21 of the inspection device 2 receives the operation signal transmitted from the server 1 via the communication unit 26 (step S251). The control unit 21 outputs an operation command according to the received operation signal (step S252), and ends the process.

Further, the above-described processing is also applied to a production device to be inspected attached to the inspection device 2 . FIG. 31 is an explanatory diagram of outputting the setting values of the production equipment using the second setting value learning model. The server 1 inputs the production material information to a second setting value learning model that is learned to output the setting values of the production equipment when the production material information to be inspected is input, and outputs the setting values of the production equipment.

As shown, state s is determined by production material size, production material material, production humidity and production temperature. A second setpoint learning model outputs a Q value corresponding to each action a (eg, a4, a5, a6) from state s. The output action a is, for example, the set values of the operating time, production speed and press pressure of the production equipment. Note that the optimization processing of the Q value of the second set value learning model is the same as the optimization processing of the Q value of the set value learning model 175, so the description is omitted. The server 1 extracts the Q value having the highest value, and transmits (outputs) an operation signal including the set value corresponding to the extracted Q value to the production apparatus.

According to this embodiment, it is possible to perform a process of learning the priority of the setting values of the operating conditions for the inspection device 2 based on the information of the material to be inspected using the reinforcement learning technique.

According to this embodiment, it is possible to output an operation signal including various setting values for an inspection apparatus according to an object to be inspected.

According to this embodiment, it is possible to output an operation signal including various setting values for a production apparatus to be inspected attached to the inspection apparatus.

According to this embodiment, it is possible to improve productivity by outputting the set values of the optimum operating conditions.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 Information processing device (server)
REFERENCE SIGNS LIST 11 control unit 12 storage unit 13 communication unit 14 input unit 15 display unit 16 reading unit 17 large-capacity storage unit 171 inspection result DB
172 Pass/Fail Judgment Model 173 Second Pass/Fail Judgment Model 174 Button Arrangement DB
175 Set value learning model 176 Material information DB
177 Operating condition DB
178 Score DB
1a portable storage medium 1b semiconductor memory 1P control program 2 inspection device 21 control unit 22 storage unit 23 position detection unit 24 input unit 25 display unit 26 communication unit 27 transport mechanism 271 transport conveyor 272 conveyor motor 273 sorting device 28 reading unit 29 Mass storage unit 291 Measurement data DB
292 message database
2a portable storage medium 2b semiconductor memory 2P control program 3 imaging device

Claims (14)

Acquire screen information that captures the monitor screen that displays information about the device,
extracting an image of an object in the acquired screen information or an image in a graph area;
A program that causes a computer connected to the device to execute a process of determining whether or not there is an abnormality in the object based on the extracted image of the object or the image within the graph area. Acquire screen information that captures the monitor screen that displays the inspection result of the inspection device,
extracting an image of an object to be inspected in the acquired screen information or an image in a graph area;
2. The program according to claim 1, which determines whether or not there is an abnormality in the inspection object based on the extracted image of the inspection object or the image within the graph area. Based on the screen information, recognizing the button for inputting the presence or absence of abnormality,
3. The program according to claim 1 or 2, for executing a process of outputting the recognized operation signal of the button to the device according to the result of determining the presence or absence of the abnormality. 4. The program according to claim 2 or 3, which executes a process of judging the presence or absence of an abnormality using a trained image recognition model that outputs a recognition result when an image of an object to be inspected or an image within a graph area is input. . Any one of claims 2 to 4, wherein a process of classifying the type of abnormality is executed using a trained second image recognition model that outputs a recognition result when an image of an inspection object or an image within a graph area is input. Program described in one. 6. The program according to any one of claims 2 to 5, which executes a process of recognizing an operation button in each stage of inspection of the inspection apparatus from the screen information. 7. The program according to any one of claims 1 to 6, causing execution of a process of extracting an alert message from said screen information. Identifying a button to be operated according to the content of the extracted alert message,
8. The program according to claim 7, which causes execution of a process of outputting an operation signal of the specified button to the device . Acquiring a captured image from an imaging device that captures an image of an inspection device,
9. The program according to any one of claims 2 to 8, wherein the acquired captured image and the screen information are output to a display unit . Accepts various setting values for inspection equipment according to inspection objects,
10. The program according to any one of claims 2 to 9, for executing a process of outputting an operation signal including the received set value to the inspection device . Receiving various setting values for the production equipment to be inspected attached to the inspection equipment,
11. The program according to any one of claims 2 to 10, for executing a process of outputting an operation signal including the received set value to the production apparatus . inputting the material information into a learning model trained to output set values of operating conditions of an inspection device when inputting material information to be inspected, and outputting set values of the operating conditions;
11. The program according to claim 10 , which causes execution of a process of outputting an operation signal including the output set value to the inspection apparatus . Acquire screen information that captures the monitor screen that displays information about the device,
extracting an image of an object in the acquired screen information or an image in a graph area;
An information processing method that causes a computer connected to the apparatus to execute a process of determining whether or not there is an abnormality in the object based on the extracted image of the object or the image within the graph area. an acquisition unit that acquires screen information obtained by capturing a monitor screen that displays information about the device;
an extraction unit that extracts an image of an object in the acquired screen information or an image in a graph area;
An information processing apparatus connected to the apparatus, comprising: a determination unit that determines whether or not there is an abnormality in the object based on the extracted image of the object or the image within the graph area.

Images