WO2022158060A1

WO2022158060A1 - Machining surface determination device, machining surface determination program, machining surface determination method, machining system, inference device, and machine learning device

Info

Publication number: WO2022158060A1
Application number: PCT/JP2021/038549
Authority: WO
Inventors: 知行内村; 智哉坂井; 健太郎織田
Original assignee: 株式会社荏原製作所
Priority date: 2021-01-25
Filing date: 2021-10-19
Publication date: 2022-07-28
Also published as: CN116724224A; JP2022113345A

Abstract

A machining surface determination device (7) provided with: a classification result acquisition unit (70A) that, regarding multiple subimage regions (430) obtained by dividing a determination image region (420) contained in a determination image (42), acquires for each subimage region (430) a classification result in which the state of a machining surface (100) is classified as one of multiple machining states; and a determination result inference unit (71) that infers a determination result for the determination image (42) by inputting the classification results for the multiple subimage regions (430) into a determination learning model (2) that has machine-learned relationships between classification results for multiple learning image regions (410) corresponding to the multiple subimage regions (430) and determination results in which the states of machining surfaces (100) in the multiple learning image regions (410) are determined on the basis of the classification results.

Description

Machined surface determination device, machined surface determination program, machined surface determination method, machining system, reasoning device, and machine learning device

The present invention relates to a machined surface determination device, a machined surface determination program, a machined surface determination method, a machining system, an inference device, and a machine learning device.

In recent years, in the manufacturing process of manufacturing various products, instead of visually determining product quality by workers, there has been progress in the development of devices that automatically determine product quality using various sensors. ing. For example, Patent Literature 1 discloses an inspection apparatus that inspects the outer shape of an impeller by performing a binarization process or the like on an image of an impeller, which is an inspection object.

JP 2008-51664 A

One of the indicators for judging product quality is, for example, the state of the processed surface after various processing processes such as polishing, grinding, cutting, or casting have been performed. The state of the machined surface includes, for example, various judgment items such as roughness, unevenness, undulation, warp, pattern, streaks, and waviness.

However, although the inspection apparatus disclosed in Patent Document 1 inspects the outer shape of the inspection object, it cannot determine the state of the machined surface of the inspection object. In addition, if workers are to judge the state of the machined surface, it depends on the skill and experience (including tacit knowledge) of the workers, so individual differences between workers will increase, and product quality will not be guaranteed. It is difficult to

In view of the problems described above, the present invention provides a machined surface determination device, a machined surface determination program, a machined surface determination method, a machining system, and an inference device that enable automatic determination of the state of a machined surface of an object to be determined. , and to provide a machine learning device.

In order to achieve the above object, a machined surface determination device according to one aspect of the present invention includes:
A machined surface determination device that determines the state of the machined surface based on a judgment image in which the machined surface of the object to be judged is captured,
Classification for obtaining a classification result for each small image area when the state of the processed surface is classified into one of a plurality of processing states for a plurality of small image areas obtained by dividing the determination image area of the determination image. a result acquisition unit;
The processing surface in the plurality of learning image areas based on the classification results of the plurality of small image areas and the classification results of the plurality of learning image areas corresponding to the plurality of small image areas. a determination result inference unit that infers the determination result for the determination image by inputting a correlation with the determination result when the state of is determined to a determination learning model subjected to machine learning.

According to the machined surface determination device according to the present invention, the determination result inference unit divides the determination image region of the determination image into a plurality of small image regions, and classifies the classification result for each small image region into the determination learning model. By inputting, the determination result for the image for determination is inferred. Therefore, it is possible to automatically determine the state of the machined surface of the object to be determined.

Problems, configurations, and effects other than the above will be clarified in the mode for carrying out the invention described later.

1 is a schematic configuration diagram showing an example of a machining system 1 including a machined surface determination device 7 according to a first embodiment; FIG. 2 is a hardware configuration diagram showing an example of a computer 200 that constitutes a machine learning device 6 and a machined surface determination device 7. FIG. 1 is a block diagram showing an example of a machine learning device 6 according to a first embodiment; FIG. FIG. 4 is a data configuration diagram showing an example of first classification learning data; It is a data block diagram which shows an example of the data for determination learning. FIG. 2 is a schematic diagram showing an example of an inference model 20 applied to a first classification learning model 2A; 2 is a schematic diagram showing an example of an inference model 20 applied to a judgment learning model 2; FIG. 1 is a block diagram showing an example of a machined surface determination device 7 according to a first embodiment; FIG. FIG. 11 is a function explanatory diagram showing an example of a classification result acquisition process by a classification result acquisition unit 70A; FIG. 7 is a functional explanatory diagram showing an example of determination result inference processing by a determination result inference unit 71; 5 is a flowchart showing an example of a machined surface determination method by the machined surface determination device 7 according to the first embodiment; It is a block diagram which shows an example of the machine-learning apparatus 6 which concerns on 2nd Embodiment. FIG. 10 is a data configuration diagram showing an example of second classification learning data; FIG. 4 is a schematic diagram showing an example of an inference model 20B applied to a second classification learning model 2B; FIG. 7 is a block diagram showing an example of a machined surface determination device 7 according to a second embodiment; FIG. 11 is a function explanatory diagram showing an example of a classification result acquisition process by a classification result acquisition unit 70B; 9 is a flow chart showing an example of a machined surface determination method by the machined surface determination device 7 according to the second embodiment.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the following, the range necessary for the description to achieve the object of the present invention is schematically shown, and the range necessary for the description of the relevant part of the present invention is mainly described. It shall be by technology.

(First embodiment)
FIG. 1 is a schematic configuration diagram showing an example of a machining system 1 including a machined surface determination device 7 according to the first embodiment.

The processing system 1 uses a processing unit 3 that processes the determination target object 10, an imaging unit 4 that captures an image of the processed surface 100 of the determination target object 10, the first classification learning model 2A, and the determination learning model 2. A machined surface determination device 7 that determines the state of the machined surface 100 of the determination object 10 , and a control device 5 that controls the processing unit 3 , the imaging unit 4 , and the machined surface determination device 7 . The processing system 1 also includes a machine learning device 6 that generates the first learning model for classification 2A and the learning model for determination 2 as an additional configuration.

The determination target object 10 is an arbitrary article to be processed by the processing unit 3, made of any material such as metal, resin, ceramics, or the like. The determination target 10 is, as a specific example, a fluid machine or a fluid component that constitutes a fluid machine. The three-dimensional shape, surface properties, color, size, etc. of the determination object 10 are not particularly limited.

The processed surface 100 is, for example, the surface of the determination target object 10 when the determination target object 10 is processed by the processing unit 3 . The processing surface 100 may be any surface of the determination target object 10 , and may be the entire surface of the determination target object 10 or a portion thereof.

The processing unit 3 is composed of various robot manipulators that operate using electric power, fluid pressure, etc. as drive sources, processing mechanism units of machine tools, and the like. The processing unit 3 performs processing steps such as polishing, grinding, cutting, casting, etc. based on control commands from the control device 5 . The processing unit 3 may perform any processing process as long as it processes or forms the surface of the determination target object 10, and may perform a combination of a plurality of processing steps.

In the processing system 1 shown in FIG. 1, the processing section 3 is composed of a robot manipulator with a replaceable grindstone attached to its tip, and performs the grinding process. Further, the determination target 10 is an impeller having a plurality of blades as a fluid component constituting a pump, and the processed surface 100 is the surface of each blade processed by the grinding process by the processing unit 3 .

The imaging unit 4 is a camera that images the processing surface 100, and is composed of an image sensor such as a CMOS sensor or a CCD sensor, for example. The imaging unit 4 is attached at a predetermined position where the processing surface 100 can be imaged. For example, when the processing unit 3 is composed of a robot manipulator, the imaging unit 4 may be attached to the tip of the robot manipulator, or may be a mounting table (including a movable type) on which the determination object 10 is mounted. ) may be fixed above. Further, when the processing unit 3 is configured by, for example, a processing mechanism unit of a machine tool, the imaging unit 4 may be attached inside a safety cover of the machine tool, or may be a separate unit from the machine tool. It may be fixed above the workbench.

The imaging unit 4 is attached to the predetermined position as described above, and its position and orientation are adjusted so that the processed surface 100 fits within the angle of view of the imaging unit 4 . In addition, as shown in FIG. 1, the imaging unit 4 may be separately provided with the imaging unit 4 connected to the machine learning device 6 and the imaging unit 4 connected to the machined surface determination device 7, One imaging unit 4 may be connected to both the machine learning device 6 and the machined surface determination device 7 and shared. In addition, the imaging unit 4 may have pan/tilt/zoom functions. Furthermore, the imaging unit 4 is not limited to imaging the processing surface 100 with one camera, and may be imaging with a plurality of cameras.

The control device 5 includes, for example, a control panel 50 composed of a general-purpose or dedicated computer (see FIG. 2 described later), a microcontroller, etc., and an operation display panel 51 composed of a touch panel display, switches, buttons, etc. .

The control panel 50 is connected to actuators and sensors (both not shown) of the machining unit 3, and sends control commands to the actuators according to machining operation parameters for carrying out the machining process and detection signals of the sensors. It controls the machining process by the machining unit 3 . The control panel 50 sends an image capturing command to the image capturing unit 4 and receives a captured image captured by the image capturing unit 4 as a result. The control panel 50 sends the captured image as a determination image to the machined surface determination device 7 , and as a result, receives the state of the machined surface 100 determined by the machined surface determination device 7 . Note that the control panel 50 may send the captured image to the machine learning device 6 .

The operation display panel 51 accepts operator's operations and outputs various information by display and sound.

The machine learning device 6 operates as the subject of the learning phase in machine learning. The machine learning device 6 acquires learning data based on the captured image captured by the imaging unit 4, and generates the first classification learning model 2A and the determination learning model 2 based on the learning data. The machine learning device 6 provides the machined surface determination device 7 with the learned first classification learning model 2A and determination learning model 2 via an arbitrary communication network, recording medium, or the like. Details of the machine learning device 6 will be described later.

The machined surface determination device 7 operates as the subject of the inference phase in machine learning. The machined surface determination device 7 judges the image of the machined surface 100 captured by the imaging unit 4 using the first learning model 2A for classification and the learning model 2 for determination generated by the machine learning device 6. The state of the processing surface 100 of the determination object 10 is determined as the image for determination. Details of the machined surface determination device 7 will be described later.

In addition, each component of the machining system 1 may be configured as, for example, one machine tool by being incorporated in one housing. In that case, the machine learning device 6 and the machined surface determination device 7 may be incorporated in the control device 5. Further, each component of the processing system 1 may be configured by a processing device including the processing unit 3 and an inspection device including the imaging unit 4 and the processing surface determination device 7. In that case, the control device 5 Functionality may be distributed between processing equipment and inspection equipment. Furthermore, each component of the machining system 1 is connected by a wireless or wired network, so that at least one of the machine learning device 6 and the machined surface determination device 7 can perform machining in which the machining unit 3 and the imaging unit 4 are installed. It may be installed at a place away from the work site, and in that case, the control device 5 may be installed at the work site or at another place.

FIG. 2 is a hardware configuration diagram showing an example of a computer 200 that constitutes the machine learning device 6 and the machined surface determination device 7. As shown in FIG.

Each of the machine learning device 6 and the machined surface determination device 7 is configured by a general-purpose or dedicated computer 200 . As shown in FIG. 2, the computer 200 includes, as its main components, a bus 210, a processor 212, a memory 214, an input device 216, a display device 218, a storage device 220, a communication I/F (interface) section 222, an external A device I/F section 224 , an I/O (input/output) device I/F section 226 , and a media input/output section 228 are provided. Note that the above components may be omitted as appropriate depending on the application in which the computer 200 is used.

The processor 212 is composed of one or more arithmetic processing units (CPU, MPU, GPU, DSP, etc.) and operates as a control unit that controls the computer 200 as a whole. The memory 214 stores various data and programs 230, and is composed of, for example, a volatile memory (DRAM, SRAM, etc.) functioning as a main memory and a non-volatile memory (ROM, flash memory, etc.).

The input device 216 is composed of, for example, a keyboard, mouse, numeric keypad, electronic pen, and the like. The display device 218 is composed of, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, or the like. The input device 216 and the display device 218 may be configured integrally like a touch panel display. The storage device 220 is composed of, for example, an HDD, SSD, etc., and stores various data necessary for executing the operating system and programs 230 .

The communication I/F unit 222 is wired or wirelessly connected to a network 240 such as the Internet or an intranet, and transmits and receives data to and from other computers according to a predetermined communication standard. The external device I/F unit 224 is connected to an external device 250 such as a printer or a scanner by wire or wirelessly, and transmits and receives data to and from the external device 250 according to a predetermined communication standard. The I/O device I/F unit 226 is connected to I/O devices 260 such as various sensors and actuators, and exchanges with the I/O devices 260, for example, detection signals from sensors and control signals to actuators. Sends and receives various signals and data. The media input/output unit 228 is composed of, for example, a drive device such as a DVD drive and a CD drive, and reads and writes data with respect to media 270 such as a DVD and a CD.

In the computer 200 having the above configuration, the processor 212 calls the program 230 to the work memory area of the memory 214 and executes it, and controls each part of the computer 200 via the bus 210 . Note that the program 230 may be stored in the storage device 220 instead of the memory 214 . The program 230 may be recorded in a non-temporary recording medium such as a CD or DVD in an installable file format or executable file format and provided to the computer 200 via the media input/output unit 228 . Program 230 may be provided to computer 200 by downloading via network 240 via communication I/F section 222 . Further, the computer 200 may implement various functions realized by the processor 212 executing the program 230 by hardware such as FPGA and ASIC, for example.

The computer 200 is, for example, a stationary computer or a portable computer, and is an arbitrary form of electronic equipment. The computer 200 may be a client computer, a server computer, or a cloud computer. Computer 200 may be applied to devices other than machine learning device 6 and machined surface determination device 7 .

(Machine learning device 6)
FIG. 3 is a block diagram showing an example of the machine learning device 6 according to the first embodiment.

The machine learning device 6 includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a trained model storage unit 63. The machine learning device 6 is composed of, for example, a computer 200 shown in FIG. In that case, the learning data acquisition unit 60 is composed of the communication I/F unit 222 or the I/O device I/F unit 226 and the processor 212, and the machine learning unit 62 is composed of the processor 212, and the learning data The storage unit 61 and the learned model storage unit 63 are configured by the storage device 220 .

The learning data acquisition unit 60 is an interface unit that is connected to various external devices via a communication network and acquires learning data in which input data and output data are associated. The external devices are, for example, the imaging unit 4, the machined surface determination device 7, and the worker terminal 8 used by the worker.

The learning data storage unit 61 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 60 . The learning data includes first classification learning data for generating the first classification learning model 2A and determination learning data for generating the determination learning model 2A. Note that the specific configuration of the database that constitutes the learning data storage unit 61 may be appropriately designed.

The machine learning unit 62 performs machine learning using the learning data stored in the learning data storage unit 61. That is, the machine learning unit 62 inputs a plurality of sets of the first classification learning data to the first classification learning model 2A, thereby determining the correlation between the input data and the output data included in the first classification learning data. The relationship is machine-learned by the first learning model for classification 2A to generate the first learning model for classification 2A. In addition, the machine learning unit 62 inputs a plurality of sets of data for determination learning to the learning model 2 for determination, so that the correlation between the input data and the output data included in the data for determination learning is machine-learned into the learning model 2 for determination. The learning model 2 for determination is generated by learning.

The learned model storage unit 63 is a database that stores the first classification learning model 2A and the determination learning model 2 generated by the machine learning unit 62. The first classification learning model 2A and the determination learning model 2 stored in the trained model storage unit 63 are provided to the actual system (for example, the machined surface determination device 7) via any communication network, recording medium, or the like. be done. Note that the first learning model for classification 2A and the learning model for determination 2 may be provided to an external computer (for example, a server computer or a cloud computer) and stored in a storage unit of the external computer. In addition, although the learning data storage unit 61 and the trained model storage unit 63 are shown as separate storage units in FIG. 3, they may be configured as a single storage unit.

FIG. 4 is a data configuration diagram showing an example of the first classification learning data.

The first classification learning data includes the learning image 41 as input data, and the classification results obtained by classifying the state of the machined surface 100 included in the learning image 41 into one of a plurality of machining states as output data. , these input data and output data are associated with each other.

A learning image 41 as input data is generated by dividing a captured image 40 having a predetermined captured image region 400 in which the processing surface 100 of the determination object 10 is captured by the imaging unit 4 into learning image regions 410 . each of a plurality of images.

A captured image area 400 of the captured image 40 is an area captured by the imaging unit 4 and is determined by the angle of view of the imaging unit 4 . A captured image area 400 shown in FIG. 4 is set so as to include a portion of one blade of the impeller, which is the determination target 10 . In the captured image 40 shown in FIG. 4, not only the processed surface 100 but also the background 110 are captured. However, the captured image area 400 may be set so that the background 110 is not captured.

As shown in FIG. 4, the learning image area 410 of the learning image 41 is obtained by dividing the captured image area 400 of the captured image 40 into a lattice so that each of the learning image areas 410 has a square shape. be. The number of images, the shape, the size, and the aspect ratio of the learning image area 410 may be changed as appropriate. For example, the shape may be rectangular or other shapes. Also, the method of dividing the captured image area 400 into the learning image areas 410 may be changed as appropriate.

The classification results as output data are called, for example, teacher data or correct labels in supervised learning. If, for example, two classes of "good" and "bad" are used as a plurality of machining states, the classification result is represented by either "good" or "bad". When adopting three classes of "good", "acceptable" and "bad" as a plurality of machining states, the classification result is represented by one of "good", "acceptable" and "bad". Note that the plurality of machining states when classifying the states of the machined surface 100 are not limited to the classes described above, and may be classified into, for example, four or more classes, or may be classified from other viewpoints. .

Furthermore, when the edge of the processing surface 100 or the background 110 other than the processing surface 100 exists in the learning image area 410, it is possible to add a class of "not subject to determination" for classification. In the example of the above two classes, the classification result in the case of classification as "not subject to determination" for the reason that the edge of the processing surface 100 or the background 110 other than the processing surface 100 exists in the learning image region 410 is " In the example of the above three classes, as shown in FIG. It is represented by either. Note that when both the processing surface 100 and the background 110 are captured in the learning image 41, for example, when the ratio of the background 110 is higher than a predetermined ratio, it is classified into the class of "out of determination". Alternatively, it may be arranged such that it is not always classified into the class of "non-judgment target".

FIG. 5 is a data configuration diagram showing an example of determination learning data.

The determination learning data is input data obtained by classifying the state of the processing surface 100 into one of a plurality of processing states for each of the plurality of learning image regions 410. Based on the classification results, a plurality of learning processes are performed. Each output data includes determination results when determining the state of the processing surface 100 in the image area 410 for processing, and these input data and output data are associated with each other.

The classification results for the plurality of learning image regions 410 as input data are, for example, when the state of the processing surface 100 is classified into one of “good”, “acceptable”, “bad”, and “not subject to judgment”. is represented by the integer values "0", "1", "2" and "3".

The judgment results as output data are called, for example, teacher data or correct labels in supervised learning. The determination result is obtained by determining the state of the entire processing surface 100 with respect to a plurality of learning image regions 410, that is, the captured image region 400 before being divided into a plurality of learning image regions 410. FIG.

The determination result is, as the state of the machined surface 100, the necessity of re-machining in which the same machining process as when the machined surface 100 was machined, or the necessity of another machining in which the machining process different from when the machined surface 100 was machined is required. , at least one of the necessity of finish machining in which an operator performs finish machining on the machined surface 100, and the machining range for which re-machining, another machining, or finishing machining is performed on the machined surface 100 is determined. is. Alternatively or additionally, the determination result may be one of a plurality of machining states including at least "good" and "bad" with respect to the entire machined surface 100.

The learning data acquisition unit 60 can employ various methods as a method for acquiring the first classification learning data and the determination learning data. For example, the learning data acquisition unit 60 acquires a captured image 40 captured by the imaging unit 4 of the determination object 10 after the processing process has been performed by the processing unit 3, and divides the captured captured image 40. By doing so, a plurality of learning images 41 are generated. Next, the learning data acquisition unit 60 superimposes a frame line forming each learning image region 410 on the captured image 40, so that the plurality of learning images 41 can be distinguished from each other. displayed on the display screen of the user terminal 8.

A result (classification result ) is input, and the result (determination result) of determining the state of the processing surface 100 included in the captured image 40 is input via the operator terminal 8 . Then, the learning data acquisition unit 60 accepts the operator's input operation, and acquires the learning image 41 (input data) and the classification result (output data) of the input operation to the learning image 41. A plurality of first classification learning data are acquired by associating. In addition, the learning data acquisition unit 60 obtains the classification results (input data) for the plurality of learning image regions 410 of each of the learning images 41 and the determination results (output data ) are associated with each other to acquire data for judgment learning.

Therefore, the learning data acquisition unit 60 can acquire a number of first classification learning data corresponding to the number of divisions when one captured image 40 is divided into a plurality of learning images 41. Furthermore, by repeating the above operations, a desired number of first classification learning data can be acquired. In addition, the learning data acquisition unit 60 can acquire determination learning data in conjunction with acquiring the first classification learning data. Therefore, the first data for classification learning and the data for judgment learning can be easily collected.

FIG. 6 is a schematic diagram showing an example of an inference model 20A applied to the first classification learning model 2A.

The inference model 20A employs a convolutional neural network (CNN) as a specific method of machine learning. The inference model 20A includes an input layer 21, an intermediate layer 22, and an output layer 23.

The input layer 21 has a number of neurons corresponding to the number of pixels in the learning image 41 as input data, and the pixel value of each pixel is input to each neuron.

The intermediate layer 22 is composed of a convolutional layer 22a, a pooling layer 22b and a fully connected layer 22c. For example, a plurality of convolution layers 22a and pooling layers 22b are provided alternately. The convolution layer 22a and the pooling layer 22b extract features from the image input via the input layer 21. FIG. The fully connected layer 22c converts the feature amount extracted from the image by the convolution layer 22a and the pooling layer 22b, for example, by an activation function, and outputs it as a feature vector. It should be noted that the total bonding layer 22c may be provided in a plurality of layers.

The output layer 23 outputs output data including classification results based on the feature vectors output from the fully connected layer 22c. Note that the output data may include, for example, a score indicating the reliability of the classification result in addition to the classification result.

Between each layer of the inference model 20A, a synapse connecting each neuron between the layers is set, and a weight is associated with each synapse of the convolution layer 22a and the fully connected layer 22c of the intermediate layer 22.

The machine learning unit 62 inputs the first classification learning data to the inference model 20A, and causes the inference model 20A to machine-learn the correlation between the learning image 41 and the classification result. Specifically, the machine learning unit 62 inputs the learning image 41 constituting the first classification learning data to the input layer 21 of the inference model 20A as input data. Note that the machine learning unit 62 applies predetermined image adjustments (eg, image format, image size, image filter, image mask, etc.) to the learning image 41 as preprocessing when inputting the learning image 41 to the input layer 21 . may be applied to

The machine learning unit 62 uses an error function that compares the classification result (inference result) indicated by the output data output from the output layer 23 and the classification result (teacher data) that constitutes the first data for classification learning. Then, the weight associated with each synapse is adjusted (back promotion) so that the evaluation value of the error function becomes smaller. Then, when the machine learning unit 62 determines that a predetermined learning end condition is satisfied, such as repeating the series of processes described above a predetermined number of times or that the evaluation value of the error function becomes smaller than an allowable value, terminates the machine learning and stores the inference model 20A (all weights associated with each synapse) at that time in the trained model storage unit 63 as the first classification learning model 2A.

FIG. 7 is a schematic diagram showing an example of the inference model 20 applied to the learning model 2 for judgment.

The inference model 20 employs a convolutional neural network as a specific method of machine learning, similar to the inference model 20A shown in FIG. In the following, the inference model 20 will be described, focusing on the differences from the inference model 20A shown in FIG.

The input layer 21 has a number of neurons corresponding to the number of divisions when the captured image area 400 is divided into a plurality of learning image areas 410, and the classification results (for example, 0, 1, Integer values of 2 and 3) are input to each neuron, respectively.

The output layer 23 outputs output data including determination results based on the feature vectors output from the fully connected layer 22c. In addition to the determination result, the output data may include, for example, a score indicating the reliability of the determination result.

The machine learning unit 62 inputs the judgment learning data to the inference model 20 and causes the inference model 20 to machine-learn the correlation between the classification results and the judgment results for the plurality of learning image regions 410 . Specifically, the machine learning unit 62 inputs the classification results for the plurality of learning image regions 410 constituting the determination learning data to the input layer 21 of the inference model 20 as input data.

The machine learning unit 62 uses an error function that compares the determination result (inference result) indicated by the output data output from the output layer 23 with the determination result (teacher data) that constitutes the determination learning data, and calculates the error It repeats adjusting the weight associated with each synapse (back promotion) so that the evaluation value of the function becomes smaller. Then, when the machine learning unit 62 determines that a predetermined learning end condition is satisfied, such as repeating the series of processes described above a predetermined number of times or that the evaluation value of the error function becomes smaller than an allowable value, terminates the machine learning and stores the inference model 20 (all weights associated with each synapse) at that time in the learned model storage unit 63 as the learning model 2 for determination.

(machined surface determination device 7)
FIG. 8 is a block diagram showing an example of the machined surface determination device 7 according to the first embodiment.

The machined surface determination device 7 includes a classification result acquisition unit 70A, a determination result inference unit 71, a learned model storage unit 72, and an output processing unit 73. The machined surface determination device 7 is composed of, for example, a computer 200 shown in FIG. In that case, the classification result acquisition unit 70A is composed of the communication I/F unit 222 or the I/O device I/F unit 226 and the processor 212, and the determination result inference unit 71 and the output processing unit 73 are composed of the processor 212. , and the trained model storage unit 72 is configured by the storage device 220 .

The classification result acquisition unit 70A obtains the classification result when the state of the processing surface 100 is classified into one of a plurality of processing states for a plurality of small image regions 430 obtained by dividing the judgment image region 420 included in the judgment image 42. A classification result acquisition process (see FIG. 9 described later) for acquiring in units of small image areas 430 is performed.

As a specific configuration, the classification result acquisition unit 70A is connected to the imaging unit 4, and converts the captured image obtained by capturing the processing surface 100 of the determination target object 10 by the imaging unit 4 into a determination image area 420 having a determination image area 420. a small image generating unit 701 that generates a plurality of small images 43 from the judgment image 42 by dividing the judgment image region 420 into a plurality of small image regions 430; A first classification result inference unit 702A for inferring classification results for a plurality of small image regions 430 by inputting the small images 43 into the first classification learning model 2A in units of small image regions 430 .

The first classification result inference unit 702A determines the positional relationship of each small image region 430 with respect to the judgment image region 420 so that the judgment image 42 before division can be reconstructed from the plurality of small images 43, for example, the small image 43 additional information.

The determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 acquired by the classification result acquisition unit 70A to the determination learning model 2, thereby inferring the determination result for the determination image region 420. Inference processing (see FIG. 10 described later) is performed.

The determination results inferred by the determination result inference unit 71 are the necessity of remachining, the necessity of separate machining, the necessity of finish machining, and the target of remachining, separate machining, or finish machining of the machined surface 100. At least one of the processing ranges to be determined. Instead of or in addition to the above, the determination result may be one of a plurality of machining states including at least "good" and "bad" with respect to the entire machined surface 100.

A part or all of the classification result acquisition unit 70A and the determination result inference unit 71 may be replaced by a processor of an external computer (for example, a server computer or a cloud computer). Part or all of the acquisition processing and determination result inference processing by the determination result inference unit 71 may be executed by an external computer.

The trained model storage unit 72 stores the trained first classification learning model 2A used in the inference processing of the classification result acquisition unit 70A and the learned judgment model used in the inference processing of the judgment result inference unit 71. It is a database that stores the learning model 2. The number of the first classification learning model 2A and the judgment learning model 2 stored in the learned model storage unit 72 is not limited to one. A plurality of trained models with different conditions such as the determination target 10 may be stored and selectively used. Also, the trained model storage unit 72 may be replaced by a storage unit of an external computer (for example, a server computer or a cloud computer). In that case, the classification result acquisition unit 70A and the determination result inference unit 71 , the above-described classification result acquisition processing and determination result inference processing may be performed by accessing the external computer.

The output processing unit 73 performs output processing for outputting the determination result inferred by the determination result inference unit 71 . Various means can be adopted as specific output means for outputting the determination result. For example, according to the determination result, the output processing unit 73 transmits an operation command for reprocessing or another processing to the processing unit 3 via the control panel 50, or via the operation display panel 51 or the operator terminal 8. The execution of the finishing process may be notified to the operator by display or sound, or may be stored in the storage means of the control panel 50 as the operation history of the processing section 3 . Note that the output processing unit 73 may simply output (transmit, notify, or store) the determination result from the determination result inference unit 71, or may output (transmit, notify, or store) the determination result from the determination result inference unit 71, and may also output a plurality of data from the classification result acquisition unit 70A. The classification result for the small image area 430 may be further output (transmitted, notified, or stored).

FIG. 9 is a functional explanatory diagram showing an example of the classification result acquisition process by the classification result acquisition section 70A.

A judgment image area 420 of the judgment image 42 is an area captured by the imaging unit 4 and is determined by the angle of view of the imaging unit 4 . A determination image area 420 shown in FIG. 9 is set to include a portion of one blade of the impeller, which is the determination target 10, similarly to the captured image area 400 shown in FIG. Note that the judgment image area 420 may be set at a position different from that of the captured image area 400, and the number, shape, size, and aspect ratio of both images may be different.

As shown in FIG. 9, the small image regions 430 of the small image 43 are obtained by dividing the judgment image region 420 of the judgment image 42 into a grid so that each of the small image regions 430 has a square shape. . A small image region 430 of the small image 43 corresponds to the learning image region 410 of the learning image 41 when the first classification learning model 2A is generated by the machine learning device 6. , shape, size and aspect ratio are preferably the same or comparable.

Therefore, if the number, shape, size, and aspect ratio of the small image area 430 correspond to the number, shape, size, and aspect ratio of the image area for learning 410, the image area for determination 420 is set to the small image area. The method of dividing into the regions 430 may be changed as appropriate. For example, division may be performed in a zigzag pattern, or division may be performed according to other criteria. At this time, the dividing method for dividing the determination image region 420 into the small image regions 430 may be the same as or different from the dividing method for dividing the captured image region 400 into the learning image regions 410 .

Here, the first classification learning model 2A divides the state of the learning image 41 having the learning image region 410 corresponding to the small image region 430 and the processing surface 100 included in the learning image 41 into a plurality of processing states. The machine learning device 6 performs machine learning on the correlation with the classification results classified into any of the above. Therefore, the first classification result inference unit 702A inputs the plurality of small images 43 to the first classification learning model 2A in units of small image regions 430, thereby obtaining the state of the processing surface 100 in the small image regions 430. It functions as a classifier that classifies into one of multiple processing states. When adopting two classes (good, bad) as the plurality of machining states, the classification result is represented by two classes (good, bad), and three classes (good, fair, bad) as the plurality of machining states. is adopted, the classification result is represented by three classes (good, fair, and bad).

The first classification learning model 2A includes a learning image 41 in which at least one of the processed surface 100 and the background 110 other than the processed surface 100 is captured, and the state of the processed surface 100 captured in the learning image 41. is classified into one of a plurality of processing states, or is excluded from the determination target for the reason that the edge of the processing surface 100 or the background 110 other than the processing surface 100 exists in the small image area 430 of the learning image 41 The machine learning device 6 may perform machine learning on the correlation with the classification result when classified. In this case, the first classification result inference unit 702A of the classification result acquisition unit 70A inputs the plurality of small images 43 to the first classification learning model 2A in units of small image regions 430, thereby The state of the processed surface 100 is classified into one of a plurality of processed states, or the edge of the processed surface 100 or the background 110 other than the processed surface 100 exists in the small image area 430. function as a classifier for In the classification results, there are a plurality of machining states, as well as non-judgment conditions. not covered).

It should be noted that the classification result for the small image region 430 may include the score (reliability) for each class. In this case, assuming that the classification results are represented by four classes (good, acceptable, bad, not subject to judgment), the score for each class for the specific small image region 430 is, for example, "0.02", " 0.10", "0.95", and "0.31". Any method may be adopted as the method of using the score. For example, the class with the highest score (in the above example, the defective score of "0.95") may be used as the classification result, or the score of a predetermined class may be used. exceeds a predetermined score reference value (in the above example, if the score “0.95” of the “bad” class exceeds the score reference value “0.80”), the class is classified as a result may be

Further, the classification results for the small image region 430 are preferably stored in the learned model storage unit 72 or another storage device (not shown), and the past classification results are stored in the learned first classification learning, for example. In order to further improve the inference accuracy of the model 2A, it can be used as first classification learning data used for online learning and re-learning.

FIG. 10 is a functional explanatory diagram showing an example of determination result inference processing by the determination result inference unit 71. FIG. In the following description, it is assumed that one judgment image 42 is divided into 60 small images 43 as shown in FIG. 10 by dividing the judgment image region 420 into 60 small image regions 430. I will assume and explain.

The determination learning model 2 determines the state of the processing surface 100 in the plurality of learning image regions 410 based on the classification results for the plurality of learning image regions 410 corresponding to the plurality of small image regions 430 and the classification results. This is a result of machine learning of the correlation with the judgment result of time. Therefore, the determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 acquired by the classification result acquisition unit 70A to the learning model 2 for determination, thereby processing the processing surface 100 in the plurality of small image regions 430. , that is, the determination result for the processing surface 100 in the determination image area 420 is inferred.

When adopting the necessity of remachining, the necessity of another machining, and the necessity of finish machining as the state of the machined surface 100, the determination result is, for example, a real value having a value range of 0 to 1, The closer to "0", the more "no", and the closer to 1, the more "required". Furthermore, when a processing range is adopted as the state of the processing surface 100, a range including at least the small image regions 430 classified as "defective" as a classification result for the plurality of small image regions 430, for example, is used as the processing range. be judged.

Note that the determination result inference unit 71 may perform predetermined post-processing on the determination result inferred by the determination learning model 2 as described above. For example, as post-processing, the determination result inference unit 71 compares the value of the determination result regarding the necessity of reprocessing, the value of the determination result regarding the necessity of separate processing, and the value of the determination result regarding the necessity of finishing processing. , the machining with the largest determination result value may be selected as the final determination result.

(Processing surface determination method)
FIG. 11 is a flow chart showing an example of a machined surface determination method by the machined surface determination device 7 according to the first embodiment. 11 is repeatedly executed by the machined surface determination device 7 at predetermined timings. The predetermined timing may be arbitrary timing, for example, it may be after the processing process by the processing unit 3 is completed, it may be in the middle of the processing process, or when a predetermined event occurs (at the time of operator operation, production control system at the time of an instruction from, etc.). A case will be described below in which the machined surface determination method is executed on the determination target object 10 machined by the machining process after the machining process by the machining unit 3 is completed.

First, in step S100, when the processing process by the processing unit 3 is completed, the processing surface 100 of the determination object 10 processed by the processing process is imaged by the imaging unit 4, and the captured image is transmitted via the control device 5. The image acquisition unit 700 of the classification result acquisition unit 70A acquires the captured image as the determination image 42 by sending it to the machined surface determination device 7 .

Next, in step S110, the small image generation unit 701 divides the judgment image region 420 of the judgment image 42 into a plurality of small image regions 430 as preprocessing for the judgment image 42, thereby dividing the judgment image 42 into a plurality of small image regions 430. A plurality of small images 43 are generated.

Next, in steps S120 to S128, the first classification result inference unit 702A assigns serial numbers (1≦n≦K) to the plurality of small images 43, where K is the division number of the plurality of small images 43. , the loop process is executed by incrementing the variable i from "1" to "K".

Specifically, in step S120, the first classification result inference unit 702A initializes the variable i to "1". Next, in step S122, the first classification result inference unit 702A selects the i-th small image 43 and inputs it to the input layer 21 of the first classification learning model 2A, thereby performing the first classification Infer the classification results output from the output layer 23 of the learning model 2A.

Next, in step S126, the variable i is incremented, and in step S128, it is determined whether or not the variable i exceeds the division number K. Then, the first classification result inference unit 702A obtains the classification results for the plurality of small image regions 430 by repeating steps S122 and S126 until the variable i exceeds the number of divisions K. FIG.

Next, in step S130, the determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 to the input layer 21 of the determination learning model 2, and from the output layer 23 of the determination learning model 2, The output determination result (for example, necessity of reprocessing, necessity of another processing, necessity of finish processing, processing range, etc.) is inferred.

Next, in step S140, the output processing unit 73 outputs information corresponding to the determination result inferred by the determination result inference unit 71 to output means (eg, the control device 5, the worker terminal 8, etc.). Then, a series of the machined surface determination method shown in FIG. 11 ends. In the machined surface determination method, step S100 corresponds to an image obtaining step, steps S100 to S128 correspond to a classification result obtaining step, step S130 corresponds to a determination result inference step, and step S140 corresponds to an output processing step.

As described above, according to the processed surface determination device 7 and the processed surface determination method according to the present embodiment, the classification result acquisition unit 70A divides the determination image region 420 into the small image regions 430, thereby dividing the determination image 42 into the small image regions 430. By inputting each of the plurality of small images 43 generated from , into the first learning model 2A for classification, the classification results for the plurality of small image regions 430 are inferred. Then, the determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 to the determination learning model 2, thereby inferring the state of the machined surface 100 as the determination result.

Therefore, the classification result by the first classification learning model 2A is inferred in units of small image regions 430 by inputting each of the plurality of small images 43 into which the judgment image 42 is divided. To facilitate the collection of learning data necessary for machine learning and to improve the accuracy of the first classification learning model 2A as compared with the case of inputting the determination image 42 to the first classification learning model 2A. can be done. Then, the state of the machined surface 100 included in the determination image 42 is determined by inputting the classification results for the plurality of small image regions 430 by the first classification learning model 2A to the determination learning model 2. . Therefore, the state of the processing surface 100 of the determination target 10 can be automatically determined.

(Second embodiment)
In the processing system 1 according to the first embodiment, the case where the first learning model for classification 2A and the learning model for judgment 2 are employed in the learning phase and the inference phase of machine learning has been described. On the other hand, in the processing system 1 according to the second embodiment, a case where the second learning model for classification 2B and the learning model for determination 2 are adopted will be described. Since the basic configuration and operation of the processing system 1 according to the second embodiment are the same as those in the first embodiment, the second classification, which is the difference from the first embodiment, will be described below. The description will focus on the part related to the learning model 2B.

(Machine learning device 6)
FIG. 12 is a block diagram showing an example of the machine learning device 6 according to the second embodiment.

The machine learning device 6 includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a trained model storage unit 63, as in the first embodiment.

The learning data acquisition unit 60 is an interface unit that is connected to various external devices via a communication network and acquires learning data. The learning data storage unit 61 is a database that stores a plurality of sets of learning data acquired by the learning data acquisition unit 60 . The learning data includes second classification learning data for generating the second classification learning model 2B and judgment learning data similar to the first embodiment.

By inputting a plurality of sets of second classification learning data to the second classification learning model 2B, the machine learning unit 62 calculates the correlation between the input data and the output data included in the second classification learning data. The second classification learning model 2B is generated by performing machine learning on the second classification learning model 2B. Further, the machine learning unit 62 generates the determination learning model 2 using the determination learning data, as in the first embodiment.

The learned model storage unit 63 is a database that stores the second classification learning model 2B and the determination learning model 2 generated by the machine learning unit 62.

FIG. 13 is a data configuration diagram showing an example of the second classification learning data.

The second classification learning data uses the pixel classification results for the plurality of learning pixel regions 411 acquired from the learning image 41 as input data, and the state of the processing surface 100 included in the learning image 41 is processed by a plurality of processing methods. The classification results classified into any of the states are included as output data, and these input data and output data are associated with each other.

The pixel classification results for the plurality of learning pixel regions 411 as input data are obtained based on the pixel values in the learning pixel regions 411 for the plurality of learning pixel regions 411 forming the learning image 41 . The pixel classification result indicating the classification result for is obtained in units of learning pixel regions 411 .

The learning pixel area 411 is an area corresponding to one pixel, and pixel values in the learning pixel area 411 are represented by, for example, RGB values, grayscale values, luminance values, and the like. For example, when four classes of "good", "acceptable", "bad", and "out of determination" are adopted as the plurality of processing states, the pixel classification result is, for example, the pixel value in the learning pixel region 411. and three predetermined thresholds (third threshold < second threshold < first threshold), and if the pixel value is equal to or greater than the first threshold, it is classified as “good” (0). As a result, if the pixel value is less than the first threshold and greater than or equal to the second threshold, the classification result of "good" (1) is: If the pixel value is less than the second threshold and greater than or equal to the third threshold, is assigned a classification result of "defective" (2), and a classification result of "out of determination" (3) when the pixel value is less than the third threshold.

As in the first embodiment, the classification results as output data are, for example, "good", "acceptable", and "bad" as shown in FIG. ” and “not subject to determination”.

The learning data acquisition unit 60 can employ various methods as a method for acquiring the second classification learning data and the determination learning data. For example, as in the first embodiment, the learning data acquisition unit 60 acquires the captured image 40 captured by the imaging unit 4 of the determination object 10 after the processing step has been performed by the processing unit 3, A plurality of learning images 41 are generated by dividing the captured image 40 , and the plurality of learning images 41 are displayed on the display screen of the operator terminal 8 .

A result (classification result ) is input, and the result (determination result) of determining the state of the processing surface 100 included in the captured image 40 is input via the operator terminal 8 . Then, the learning data acquisition unit 60 accepts the operator's input operation, and the pixel classification results (input data) for the plurality of learning pixel regions 411 acquired from the learning image 41 and the learning image 41 A plurality of second classification learning data are acquired by associating the classification result (output data) input-operated with respect to . In addition, the learning data acquisition unit 60 obtains the classification results (input data) for the plurality of learning image regions 410 of each of the learning images 41 and the determination results (output data ) are associated with each other to acquire data for judgment learning.

Therefore, the learning data acquisition unit 60 can acquire a number of second classification learning data corresponding to the number of divisions when one captured image 40 is divided into a plurality of learning images 41. Furthermore, by repeating the above operation, a desired number of second classification learning data can be acquired. In addition, the learning data acquisition unit 60 can acquire determination learning data in conjunction with acquiring the second classification learning data. Therefore, it is possible to easily collect the second data for classification learning and the data for judgment learning.

FIG. 14 is a schematic diagram showing an example of the inference model 20B applied to the second classification learning model 2B.

The inference model 20B employs a convolutional neural network as a specific method of machine learning, similar to the inference model 20A shown in FIG. In the following, the inference model 20B will be described, focusing on the differences from the inference model 20A shown in FIG.

The input layer 21 has a number of neurons corresponding to the number of pixels in the learning image 41 as input data, and pixel classification results for a plurality of learning pixel regions 411 are input to each neuron.

The machine learning unit 62 inputs the second classification learning data to the inference model 20B, and causes the inference model 20B to machine-learn the correlation between the pixel classification results for the plurality of learning pixel regions 411 and the classification results. Specifically, the machine learning unit 62 inputs the pixel classification results for the plurality of learning pixel regions 411 constituting the second classification learning data to the input layer 21 of the inference model 20B as input data.

The machine learning unit 62 uses an error function that compares the classification result (inference result) indicated by the output data output from the output layer 23 and the classification result (teacher data) that constitutes the second classification learning data. Then, the weight associated with each synapse is adjusted (back promotion) so that the evaluation value of the error function becomes smaller. Then, when the machine learning unit 62 determines that a predetermined learning termination condition is satisfied, such as repeating the series of processes described above a predetermined number of times or the evaluation value of the error function being smaller than the allowable value, terminates the machine learning and stores the inference model 20B (all weights associated with each synapse) at that time in the trained model storage unit 63 as the second classification learning model 2B.

(machined surface determination device 7)
FIG. 15 is a block diagram showing an example of the machined surface determination device 7 according to the second embodiment.

The machined surface determination device 7 includes a classification result acquisition unit 70B, a determination result inference unit 71, a learned model storage unit 72, and an output processing unit 73, as in the first embodiment.

The classification result acquisition unit 70B obtains the classification result when the state of the processing surface 100 is classified into one of a plurality of processing states for a plurality of small image regions 430 obtained by dividing the judgment image region 420 included in the judgment image 42. A classification result acquisition process (see FIG. 16 described later) for acquiring in units of small image areas 430 is performed.

The classification result acquisition unit 70B includes an image acquisition unit 700 and a small image generation unit 701 similar to those in the first embodiment, and a plurality of pixel regions that form each of the plurality of small images 43. a pixel classification result obtaining unit 703 for obtaining, in units of pixel areas, pixel classification results indicating the classification results for pixel areas based on the pixel classification results; and a second classification result inference unit 702B for inferring classification results for the plurality of small image regions 430 by inputting .

The determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 acquired by the classification result acquisition unit 70B to the determination learning model 2, thereby inferring the determination result for the determination image region 420. Perform inference processing.

The trained model storage unit 72 stores the trained second classification learning model 2B used in the inference processing of the classification result acquisition unit 70B and the learned judgment model 2B used in the inference processing of the judgment result inference unit 71. It is a database that stores the learning model 2.

FIG. 16 is a functional explanatory diagram showing an example of the classification result acquisition process by the classification result acquisition section 70B.

A judgment image area 420 of the judgment image 42 is an area captured by the image pickup unit 4 as in the first embodiment, and a small image area 430 of the small image 43 is a judgment image of the judgment image 42. The region 420 is divided into grids. A small image region 430 of the small image 43 corresponds to the learning image region 410 of the learning image 41, and a plurality of pixel regions 431 forming the small image 43 correspond to a plurality of learning pixel regions forming the learning image 41. 411 equivalent.

Here, the second classification learning model 2B performs pixel classification results for the plurality of learning pixel regions 411 corresponding to the plurality of pixel regions 431 and processing in the plurality of learning pixel regions 411 based on the pixel classification results. Machine learning is performed on the correlation with the classification result when the state of the surface 100 is classified into one of a plurality of machining states. Therefore, the second classification result inference unit 702B inputs the pixel classification results for the plurality of pixel regions 431 forming each of the plurality of small images 43 to the second classification learning model 2B in units of small image regions 430. functions as a classifier that classifies the state of the processing surface 100 in the small image area 430 into one of a plurality of processing states.

(Processing surface determination method)
FIG. 17 is a flowchart showing an example of a machined surface determination method by the machined surface determination device 7 according to the second embodiment.

First, in step S100, the image acquisition unit 700 of the classification result acquisition unit 70B acquires the determination image 42.

Next, in step S110, the small image generating unit 701 divides the judgment image region 420 of the judgment image 42 into a plurality of small image regions 430 as preprocessing for the judgment image 42, thereby dividing the judgment image 42 into a plurality of small image regions 430. A plurality of small images 43 are generated.

Then, in step S112, the pixel classification result acquisition unit 703 acquires the pixel classification result for the pixel regions 431 based on the pixel values in the pixel regions 431 for the plurality of pixel regions 431 forming each of the plurality of small images 43. Acquired in units of 431 areas.

Next, in steps S120 to S128, the second classification result inference unit 702B assigns a serial number (1≦n≦K) to each of the plurality of small images 43, where K is the division number of the plurality of small images 43. , the loop process is executed by incrementing the variable i from "1" to "K".

Specifically, in step S120, the second classification result inference unit 702B initializes the variable i to "1". Next, in step S124, the second classification result inference unit 702B selects the i-th small image 43, and applies the pixel classification results for the plurality of pixel regions 431 forming the small image 43 to the second classification learning. By inputting to the input layer 21 of the model 2B, the classification result output from the output layer 23 of the second learning model for classification 2B is inferred.

Next, in step S126, the variable i is incremented, and in step S128, it is determined whether or not the variable i exceeds the division number K. Then, the second classification result inference unit 702B acquires the classification results for the plurality of small image regions 430 by repeating steps S124 and S126 until the variable i exceeds the number of divisions K. FIG.

Next, in step S140, the output processing unit 73 outputs information corresponding to the determination result inferred by the determination result inference unit 71 to output means (eg, the control device 5, the worker terminal 8, etc.). Then, the series of the machined surface determination method shown in FIG. 17 ends. In the machined surface determination method, step S100 corresponds to an image obtaining step, steps S100 to S128 correspond to a classification result obtaining step, step S130 corresponds to a determination result inference step, and step S140 corresponds to an output processing step.

As described above, according to the processed surface determination device 7 and the processed surface determination method according to the present embodiment, the classification result acquisition unit 70B divides the determination image region 420 into the small image regions 430, thereby dividing the determination image 42 into the small image regions 430. a plurality of small image regions 43 are generated from the plurality of small image regions 43, and the pixel classification results for the plurality of pixel regions 431 constituting each of the plurality of small image regions 43 are input to the second classification learning model 2B to obtain a plurality of small image regions Infer the classification result for 430. Then, the determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 to the determination learning model 2, thereby inferring the state of the machined surface 100 as the determination result.

Therefore, the classification result by the second classification learning model 2B is inferred in units of small image regions 430 by inputting each of the plurality of small images 43 into which the judgment image 42 is divided. To facilitate collection of learning data necessary for machine learning and to improve the accuracy of the second classification learning model 2B as compared with the case of inputting the judgment image 42 to the second classification learning model 2B. can be done. Then, the state of the machined surface 100 included in the determination image 42 is determined by inputting the classification results for the plurality of small image regions 430 by the second classification learning model 2B to the determination learning model 2. . Therefore, the state of the processing surface 100 of the determination target 10 can be automatically determined.

(Other embodiments)
The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention. All of them are included in the technical idea of the present invention.

For example, in the above embodiment, the judgment image area 420 is set to include a part of one blade of the impeller of the judgment object 10 as the processing surface 100 to be judged. On the other hand, the determination image area 420 may be set so as to include the plurality of blades of the impeller as the plurality of processing surfaces 100 to be determined by expanding the entire impeller. That is, when the determination target object 10 has a plurality of processed surfaces 100 that have been processed through different processing steps by the processing unit 3, the determination image region 420 is set to include the plurality of processed surfaces 100. may

In this case, the classification result acquisition unit 70B acquires the determination images 42 in which the plurality of processed surfaces 100 are captured, and performs determination for each processed surface so that the determination images 42 are separated at the boundaries of the plurality of processed surfaces 100. set the image area 420 for The boundary of the processing surface 100 may be set in advance, or may be set by image processing on the determination image 42 . Then, the classification result acquisition unit 70B acquires the classification result for each small image area 430 for a plurality of small image areas 430 obtained by dividing the determination image area 420 for each processing surface. Next, the determination result inference unit 71 inputs the classification results for the plurality of small image regions 430 to the determination learning model 2 for each processed surface, thereby inferring the determination result for the determination image 42 for each processed surface. .

Further, in the above-described embodiment, the case where CNN (see FIGS. 6 and 7) is adopted as a specific method of machine learning by the machine learning unit 62 has been described, but the machine learning unit 62 can be any other machine A method of learning may be employed. Other machine learning methods include, for example, tree types such as decision trees and regression trees, ensemble learning such as bagging and boosting, neural network types such as recurrent neural networks and convolutional neural networks (including deep learning), Hierarchical clustering, non-hierarchical clustering, k-neighbor method, clustering type such as k-means method, principal component analysis, factor analysis, multivariate analysis such as logistic regression, support vector machine, and the like.

(machined surface determination program)
The present invention can be provided in the form of a program (machined surface determination program) 230 that causes the computer 200 shown in FIG. 2 to function as each unit included in the machined surface determination device 7 according to the above embodiment. The present invention can also be provided in the form of a program (machined surface determination program) 230 for causing the computer 200 shown in FIG. 2 to execute each step included in the machined surface determination method according to the above embodiment.

(Inference Apparatus, Inference Method and Inference Program)
The present invention is not only based on the aspect of the machined surface determination device 7 (machined surface determination method or machined surface determination program) according to the above embodiment, but also an inference device (inference method) used to determine the state of the machined surface 100 or an inference program). In that case, the inference device (inference method or inference program) may include a memory and a processor, and the processor of these may execute a series of processes. The series of processing refers to classification results obtained when the state of the processed surface 100 is classified into one of a plurality of processed states for a plurality of small image regions 430 obtained by dividing the determination image region 420 included in the determination image 42. When the classification result acquisition process (classification result acquisition process) for acquiring each small image area 430 and the classification results for a plurality of small image areas 430 are acquired in the classification result acquisition process, the determination result for the determination image 42 is obtained as a determination image. a judgment result inference process (judgment result inference process) for inferring the state of the machined surface 100 included in 42;

By providing it in the form of an inference device (inference method or inference program), it can be applied to various devices more easily than when the machined surface determination device 7 is implemented. When the inference device (inference method or inference program) infers the state of the machined surface 100, the machined surface judgment device 7 uses the learned judgment learning model 2 generated by the machine learning device 6 according to the above embodiment. It should be understood by those skilled in the art that the inference method performed by the determination result inference unit 71 may be applied.

The present invention can be used for a machined surface determination device, a machined surface determination program, a machined surface determination method, a machining system, an inference device, and a machine learning device.

1... Machining system, 2... Learning model for judgment,
2A... First learning model for classification, 2B... Second learning model for classification 3... Processing unit, 4... Imaging unit, 5... Control device, 6... Machine learning device,
7... Machined surface determination device, 8... Terminal for operator, 10... Object to be determined,
20, 20A, 20B... inference model, 21... input layer, 22... intermediate layer,
22a... convolution layer, 22b... pooling layer, 22c... fully connected layer, 23... output layer,
40... Captured image, 41... Learning image, 42... Judging image, 43... Small image,
50... Control panel 51... Operation display panel 60... Learning data acquisition unit 61... Learning data storage unit 62... Machine learning unit 63...

Model storage units

70A, 70B... Classification result acquisition unit 71... Judgment result inference Unit 72... Model storage unit 73... Output processing unit 100... Machining surface 110... Background 200... Computer,
400... Captured image area 410... Learning image area 411... Learning pixel area 420... Determining image area 430... Small image area 431... Pixel area 700... Image acquiring unit 701... Small image generating unit
702A First classification result inference unit 702B Second classification result inference unit 703 Pixel classification result acquisition unit

Claims

A machined surface determination device that determines the state of the machined surface based on a judgment image in which the machined surface of the object to be judged is captured,
Classification for obtaining a classification result for each small image area when the state of the processed surface is classified into one of a plurality of processing states for a plurality of small image areas obtained by dividing the determination image area of the determination image. a result acquisition unit;
The processing surface in the plurality of learning image areas based on the classification results of the plurality of small image areas and the classification results of the plurality of learning image areas corresponding to the plurality of small image areas. a determination result inference unit that infers the determination result for the determination image by inputting the correlation with the determination result when determining the state of the determination learning model that has undergone machine learning,
Machining surface determination device.
The classification result acquisition unit
an image acquiring unit that acquires the determination image having the determination image area;
a small image generation unit that generates a plurality of the small images from the judgment image by dividing the judgment image region into a plurality of the small image regions;
Correlation between the plurality of small images and the classification result when classifying the state of the machined surface included in the learning image having the learning image area and the state of the machined surface included in the learning image into one of the plurality of machining states a first classification result inference unit that infers the classification results for a plurality of the small image regions by inputting the relationship into the first classification learning model that has undergone machine learning in units of the small image regions;
The machined surface determination device according to claim 1.
The classification result acquisition unit
an image acquiring unit that acquires the determination image having the determination image area;
a small image generation unit that generates a plurality of the small images from the judgment image by dividing the judgment image region into a plurality of the small image regions;
Acquisition of pixel classification results for each of a plurality of pixel areas constituting each of the plurality of small images, the pixel classification result indicating the classification results for the pixel areas based on the pixel values in the pixel areas. Department and
The pixel classification results for the plurality of pixel regions are processed in the plurality of learning pixel regions based on the pixel classification results for the plurality of learning pixel regions corresponding to the plurality of pixel regions and the pixel classification results. By inputting the correlation with the classification result when the state of the surface is classified into one of the plurality of processing states in units of the small image area into the second classification learning model that has undergone machine learning, a second classification result inference unit that infers the classification result for the small image region;
The machined surface determination device according to claim 1.
The classification result acquisition unit
Acquiring the classification result for each of the small image regions for a plurality of the small image regions obtained by dividing the image region for judgment for each of the processing surfaces included in the image for judgment in which the plurality of processing surfaces are imaged;
The determination result inference unit
inferring the determination result for the determination image for each processed surface by inputting the classification results for the plurality of small image regions into the determination learning model for each processed surface;
The machined surface determination device according to any one of claims 1 to 3.
The classification result acquisition unit
For each of the plurality of small image regions, the state of the processing surface in the small image region is classified into one of a plurality of processing states including at least good and bad, or the processing surface is classified into the small image region. Obtaining the classification result in units of the small image area when classified as not to be determined due to the presence of an edge of or a background other than the processed surface.
The machined surface determination device according to any one of claims 1 to 4.
The determination result inference unit
As the determination result,
Necessity of reprocessing to repeat the same processing steps as when the processing surface was processed,
Necessity of another processing to perform a processing process different from that when processing the processing surface,
Necessity of finish processing in which the operator performs finishing processing on the processing surface, and
A processing range targeted for the reprocessing, the separate processing, or the finishing processing of the processing surface,
infer at least one of
The machined surface determination device according to any one of claims 1 to 5.
The processed surface is the surface of the determination target when the determination target is processed by a polishing process, a grinding process, a cutting process, or a casting process.
The machined surface determination device according to any one of claims 1 to 6.
The object to be determined is a fluid machine or a fluid component that constitutes the fluid machine,
The machined surface determination device according to any one of claims 1 to 7.
causing a computer to function as the machined surface determination device according to any one of claims 1 to 8,
Machining surface judgment program.
A machined surface determination method for determining the state of the machined surface based on a judgment image in which the machined surface of the object to be judged is captured,
Classification for obtaining a classification result for each small image area when the state of the processed surface is classified into one of a plurality of processing states for a plurality of small image areas obtained by dividing the determination image area of the determination image. a result acquisition step;
The processing surface in the plurality of learning image areas based on the classification results of the plurality of small image areas and the classification results of the plurality of learning image areas corresponding to the plurality of small image areas. a judgment result inference step of inferring the judgment result for the judgment image by inputting the correlation with the judgment result when judging the state of the judgment learning model machine-learned,
Machining surface judgment method.
A machined surface determination device according to any one of claims 1 to 8;
a processing unit that processes the determination object;
an imaging unit that captures an image of the processed surface of the determination target;
A control unit that controls the machined surface determination device, the processing unit, and the imaging unit,
processing system.
An inference device used to determine the state of the processing surface based on a determination image in which the processing surface of a determination target is captured,
The reasoning device comprises a memory and a processor,
The processor
Classification for obtaining a classification result for each small image area when the state of the processed surface is classified into one of a plurality of processing states for a plurality of small image areas obtained by dividing the determination image area of the determination image. a result acquisition process;
a judgment result inference process for inferring the state of the machined surface included in the judgment image as a judgment result for the judgment image when the classification results for the plurality of small image regions are acquired in the classification result acquisition process; run the
reasoning device.
A machine learning device for generating a determination learning model for use in a machined surface determination device that determines the state of a machined surface based on a judgment image in which a machined surface of an object to be judged is captured,
A classification result when the state of the machined surface is classified into one of a plurality of machining states for each of a plurality of learning image areas corresponding to a plurality of small image areas obtained by dividing the judgment image area of the judgment image. is input data, and a plurality of sets of learning data each including, as output data, determination results when determining the state of the processing surface in the plurality of learning image regions based on the classification results are stored. Department and
a machine learning unit that learns the determination learning model for inferring the correlation between the input data and the output data by receiving a plurality of sets of the learning data;
a learned model storage unit that stores the determination learning model learned by the machine learning unit;
Machine learning device.