JP6177649B2 - Data processing apparatus, length measurement system, defect inspection system, image tracking system, and data processing method - Google Patents

Description

  The present invention relates to a data processing device, a length measurement system, a defect inspection system, an image tracking system, and a data processing method, and, for example, relates to a technique for optimizing a data processing algorithm.

  2. Description of the Related Art Inspection devices and measuring devices based on various image processing, including inspection devices and medical diagnosis related to semiconductor manufacturing, have been proposed and used. In general, these images detect a defective part or the like in an image taken with an electron microscope, an image taken with an X-ray, or an image obtained with an optical camera, and a part to be measured is extracted and specified in the image. Is.

  Inspection devices and measuring devices try to automatically identify parts by image processing, but it is not possible to construct and optimize image processing algorithms according to the global characteristics of the image and the characteristics of the target to be identified. It's not easy. The algorithm is expected to be effective for a large number of image samples in general. However, it is necessary to verify whether a combination of various adjustment parameter values and processing functions is effective for a large number of image samples. Needed.

  In order to improve the above problems, for example, Patent Document 1 discloses a method of automatically selecting an effective algorithm for a part of images selected by a user from many images and displaying the processing results of all the images by the algorithm. It is disclosed.

JP 2010-262350 A

  However, the method disclosed in Patent Document 1 is based on the premise that the user selects an image and confirms the processing results of all images, and requires knowledge and skill of the user. For this reason, a user who does not have sufficient knowledge and experience may not be able to appropriately select an image necessary for automatically selecting an algorithm.

  In general, an algorithm reviewer selects a small number of images that are likely to be representative from a large number of image samples, constructs an effective algorithm for these representative images, and verifies the effect. The procedure of verifying the effectiveness with the remaining many image samples often increases the efficiency of algorithm optimization. This representative image can be selected by classifying images that look similar to each other and selecting a part of them, or by predicting an algorithm that seems to work effectively in many cases. It is to select from the viewpoint of whether or not it is a sample.

  However, such a method is subjective and intuitive and requires knowledge and skill in order to be able to select with improved efficiency. If the selection of image samples is inappropriate, even if an effective algorithm can be constructed for the selected samples, there are many failures when verifying the effectiveness of the remaining many image samples later. It will be started again.

  The present invention has been made in view of such circumstances, and provides a technique for automatically and rapidly constructing and optimizing a general-purpose effective image processing algorithm for a large number of image samples. It is.

  In order to solve the above problems, in the data processing apparatus according to the present invention, the processor executes a plurality of prepared data processing algorithms for all of the plurality of teaching data, and supports the plurality of data processing algorithms. A first evaluation process that evaluates all of the plurality of teaching data and identifies a top-level data processing algorithm for all the teaching data, and all of the plurality of teaching data corresponding to the plurality of data processing algorithms A teaching data selection process for classifying a plurality of teaching data into a plurality of confirmed groups based on the evaluation result, selecting one teaching data from each confirmed group, and generating selected teaching data; and a plurality of data processing Based on all the evaluation results of a plurality of teaching data corresponding to the algorithm, the data processing is executed on the selected teaching data. A plurality of improved data processing algorithms, including a ranking process for ranking the data processing algorithms, a top-level data algorithm for all teaching data, and a data processing algorithm for which the evaluation result is determined to be the top in the ranking process. A second evaluation process for generating a data processing algorithm, executing the plurality of improved data processing algorithms on all of the plurality of teaching data, and evaluating the plurality of improved data processing algorithms; And an algorithm determination process for determining an optimum data processing algorithm based on the result of the evaluation process.

  Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. The embodiments of the present invention can be achieved and realized by elements and combinations of various elements and the following detailed description and appended claims.

  According to the present invention, it is possible to automatically and rapidly construct and optimize a general-purpose effective image processing algorithm for a large number of image samples. Thereby, the cost of development and adjustment of a data processing algorithm can be suppressed, and the accuracy of inspection and measurement using the algorithm can be stabilized.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the hardware constitutions of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the keyboard, mouse, display, and stylus pen as correct data setting means. It is a flowchart for demonstrating the image processing algorithm optimization process by the 1st Embodiment of this invention. It is a figure for demonstrating progress of the image processing algorithm optimization process by the 1st Embodiment of this invention. It is a figure which shows progress for the long time of the image processing algorithm optimization process by the 1st Embodiment of this invention. It is a conceptual diagram of teaching data selection. It is a flowchart for demonstrating the teaching data selection process by embodiment of this invention. It is a figure which shows the example of the table | surface of the evaluation value by the algorithm group for every teaching data. It is a figure for demonstrating the concept of a level | step difference value. It is a figure which shows the example of the tree diagram based on the distance between the evaluation value vectors of teaching data. It is a figure which shows the example of the template image used for pattern matching. It is a figure which shows the example of the setting of the to-be-searched image used for pattern matching, and correct answer data. It is a figure which shows an example which calculates an evaluation value from the result data of pattern matching. It is an example of a table of evaluation values according to a pattern matching algorithm group. It is a figure which shows the relationship between the teaching data of pattern matching, and an algorithm group. It is a figure which shows the example of a display of the clustering result of teaching data. It is a figure which shows the mode of the test | inspection image and correct data setting of a semiconductor. It is a figure which shows an example which calculates an evaluation value from the result data of defect classification. It is a figure which shows the example of the table | surface of the evaluation value by a classification algorithm group. It is a figure which shows the relationship between the teaching data of defect classification, and an algorithm group. It is a figure which shows an example which calculates an evaluation value from the result data of tracking. It is a figure which shows the example of the table | surface of the evaluation value by a tracking algorithm group. It is a figure which shows the relationship between the teaching data of tracking, and an algorithm group. It is a figure which shows an example which calculates an evaluation value from the result data of speech recognition. It is a figure which shows the example of the table | surface of the evaluation value by a speech recognition algorithm group. It is a figure which shows the relationship between the teaching data of speech recognition, and an algorithm group. It is a flowchart for demonstrating the image processing algorithm optimization process by the 2nd Embodiment of this invention. It is a flowchart for demonstrating the image processing algorithm optimization process by the 3rd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be denoted by the same numbers. The attached drawings show specific embodiments and implementation examples based on the principle of the present invention, but these are for understanding the present invention and are not intended to limit the present invention. Not used.

  This embodiment has been described in sufficient detail for those skilled in the art to practice the present invention, but other implementations and configurations are possible without departing from the scope and spirit of the technical idea of the present invention. It is necessary to understand that the configuration and structure can be changed and various elements can be replaced. Therefore, the following description should not be interpreted as being limited to this.

  Furthermore, as will be described later, the embodiment of the present invention may be implemented by software running on a general-purpose computer, or may be implemented by dedicated hardware or a combination of software and hardware.

  In the following description, each information of the present invention will be described in a “table” format. However, the information does not necessarily have to be expressed in a data structure by a table, such as a data structure such as a list, a DB, a queue, or the like. It may be expressed as Therefore, “table”, “list”, “DB”, “queue”, etc. may be simply referred to as “information” to indicate that they do not depend on the data structure.

  In addition, when explaining the contents of each information, the expressions “identification information”, “identifier”, “name”, “name”, “ID” can be used, and these can be replaced with each other. It is.

(1) First Embodiment <Configuration of Information Processing System>
(I) Functional Block Diagram FIG. 1 is a functional block diagram of a data processing system (image processing system) according to an embodiment of the present invention. In this embodiment, an example of an inspection system that automatically inspects a product by image processing will be described as an information processing system. However, the present invention is not limited to this, and various data may be used as will be described later. It can be applied to processing.

  The image processing system 100 stores a correct answer setting unit 101, a correct answer data storage unit 102, an evaluation unit 103, an imaging unit 104, an image data storage unit 105, an image processing algorithm execution unit 106, and an image processing result. A result data storage unit 107, an automatic optimization unit 108, an algorithm setting data storage unit 109, an evaluation value storage unit 110, and a selection function unit 111.

  The imaging unit 104 acquires an image of a processing target (inspection target) and outputs the acquired image data to the image data storage unit 105. Specifically, an electron microscope, an optical microscope, an X-ray imaging apparatus, or a camera using another optical image sensor is used. For the imaging unit 104, sensitivity, exposure time, distortion correction, and the like are adjusted by setting data from the algorithm setting data storage unit 109.

  The image data acquired from the imaging unit 104 is image information stored in the image data storage unit 105 configured by a storage unit such as a hard disk or a memory.

  The image processing algorithm execution unit 106 receives the image data from the image data storage unit 105, performs predetermined image processing (for example, defect inspection processing), and outputs the image processing result to the result data storage unit 107. The image processing algorithm execution unit 106 can select and change various image processing function combinations and parameters according to the algorithm setting data 109, and can be realized by a computer or a program. Similarly, other components can be realized by, for example, a computer or a program including a microprocessor, a hard disk, a memory, a display, and the like.

  Note that the imaging unit 104 and the image processing algorithm execution unit 106 adjust imaging conditions and processing contents based on the algorithm setting data. The setting data for the imaging unit 104 is, for example, exposure time, squeezing, focal length, and the like. The setting data for the image processing algorithm execution unit 106 includes, for example, filter coefficients, template sizes, and detailed contents of a matching method (for example, two-stage processing: template matching with reduced images, and based on the result, the original size image Template matching method).

  The result data storage unit 107 stores an image indicating a defect area of a product and coordinate information of a matching result for alignment.

  The correct answer setting unit 101 generates correct answer data expected to be output to the image processing algorithm execution unit 106.

  The evaluation unit 103 compares the correct data from the correct data storage unit 102 with the result data from the result data storage unit 107 and outputs an evaluation value as an evaluation result to the evaluation value storage unit 110. An evaluation value becomes so high that both data correspond.

  The correct answer data is not always necessary, and the correct answer setting unit 101 and the correct answer data storage unit 102 can be omitted. That is, in order to obtain the evaluation value, a certain evaluation criterion is determined instead of the correct answer data, and the image data may be evaluated according to the criterion. In this case, an evaluation value obtained as a result of evaluating the image data according to a certain standard is stored in the evaluation value storage unit 110, and becomes a parameter of an evaluation value vector described later. For example, when the algorithm is optimized to reduce the variation in the dimension measurement value of the semiconductor pattern, correct data is not given and the variation amount of the dimension measurement value is adopted as an alternative evaluation standard.

  The automatic optimization unit 108 receives the evaluation value from the evaluation value storage unit 110 and changes the algorithm setting data based on this evaluation value. Thereby, the processing in the image processing algorithm execution unit 106 is optimized. Details of the algorithm optimization process will be described later (see FIG. 3 and the like).

  In addition, the automatic optimization unit 108 controls the selection function unit 111 so that only one selected from a large number of image data is input to the image processing algorithm execution unit 106 and acquires an evaluation value. Details of image data selection processing by the automatic optimization unit 108 and the selection function unit 111 will be described later (see FIGS. 6 to 9).

(Ii) Hardware Configuration Example FIG. 2A is a diagram illustrating a hardware configuration example of the image processing system corresponding to FIG. As illustrated, the image processing system 100 can be implemented using a computer.

  2A, an image processing system 100 includes a processor 201 such as a CPU (Central Processing Unit) and MPU (Micro-Processing Unit) that executes processing of various programs, a memory / HDD 202 for storing data, a keyboard, An input device 203 configured with a mouse, a stylus pen, or the like, an output device 204 configured with a printer, a display, or the like, and an imaging device 205 corresponding to the imaging unit 104 in FIG.

  The processor 201 includes an image processing algorithm execution program 2011 corresponding to the image processing algorithm execution unit 106 in FIG. 1, a correct answer setting program 2012 corresponding to the correct answer setting unit 101, an evaluation program 2013 corresponding to the evaluation unit 103, and an automatic optimization unit 108. Each of the automatic optimization program 2014 corresponding to, and the image selection program 2015 corresponding to the selection function unit 111 is executed.

  The memory / HDD 202 includes image data 2021 acquired by the imaging apparatus 205, correct data 2022 set by the correct setting program 2012, result data 2023 that is an execution result of the image processing algorithm execution program 2011, and an evaluation program 2013. An evaluation value 2024 that is an evaluation result and algorithm setting data 2025 that is setting data of the imaging device 205 and setting data of the image processing algorithm execution program 2011 are stored. In FIG. 2A, each data and information is stored in one memory / HDD, but may be stored in a separate memory or HDD.

  The imaging device 205 corresponds to an electron microscope, an optical microscope, an X-ray imaging device, a camera using another optical image sensor, or the like. The imaging device 205 may capture a moving image as well as a still image.

  FIG. 2B is a diagram showing an example of specific means (input device 203) for inputting data given to the correct answer setting program 2012 (correct answer setting unit 101). Examples of the input device 203 include a keyboard 2031, a mouse 2032, a stylus pen 2033, and the like. The keyboard 2031 is used to input the position coordinates in the image data and the type of product defect as numerical values. The mouse 2032 and the stylus pen 2033 are used to designate position coordinates in the image data on the screen of the display 204 that is an output device. The mouse 2032 and the stylus pen 2033 are examples of what is generally called a pointing device, and coordinates may be specified by other pointing devices such as a trackball and a touch pad. Correct data can be created by these means.

<Image processing algorithm optimization processing>
FIG. 3 is a flowchart for explaining the contents of the image processing algorithm optimization processing executed by the automatic optimization unit 108 (automatic optimization program 2014). In the following, the operation subject is the automatic optimization program 2014, but it may be replaced with the automatic optimization unit 108 or the processor 201.

(I) Step 301
First, the automatic optimization program 2014 generates an initial algorithm group (step 301). Here, the algorithm group is a conceptual term. In practice, a plurality of types of algorithm setting data for specifying the processing of the image processing algorithm execution unit 106 are prepared, and different procedures and contents regarding processing in the execution unit 106 are provided. You may prepare more than one. A plurality of setting data corresponding to an algorithm that has a past record and can be expected to be effective may be designated in advance, or a large number of setting data may be generated based on random numbers. In the present embodiment, 50 constant algorithm groups are generated. In other words, in the present invention, a plurality of algorithms is a concept including at least one of an algorithm having the same algorithm contents such as a procedure but different setting parameters and an algorithm having the same setting parameters but different algorithm contents.

(Ii) Step 303
The automatic optimization program 2014 evaluates all prepared teaching data using the image processing algorithm execution program 2011 and the evaluation program 2013 (step 303). Here, the teaching data means data represented by a set of input image data and corresponding correct data. In other words, it is a set of input information and output expected value used for optimizing the algorithm, and is a word meaning that the optimization is regarded as learning and corresponds to a test question and a prepared model answer. If there is no correct answer data, the teaching data means the input image data itself.

  Specifically, first, the image processing algorithm execution program 2011 inputs a large number of sample image data 2021 (image data stored in the image data storage unit 105), and uses all the algorithm groups generated in step 301. The processing of the image processing algorithm execution program 2011 (image processing algorithm execution unit 106) is executed. When the image pickup unit 104 is adjusted, the image data 2021 is acquired by the image pickup unit. Then, the evaluation program 2013 (evaluation unit 103) evaluates all the output result data 2023 to obtain an evaluation value 2024. The total number of evaluation values 2024 obtained in this step is (number of samples of image data 2021) × (number of algorithms). Examples of evaluation values include those shown in FIGS. 12, 17, 20, and 23 described later.

(Iii) Step 304
The automatic optimization program 2014 ranks the algorithms based on the evaluation results of all teaching data obtained in step 303 (step 304). The higher the evaluation value 2024 obtained in step 303, the higher the ranking of each algorithm. Evaluation values 2024 are obtained as many as the number of teaching data for one algorithm, but ranking may be based on an average value (or sum) or a ratio of evaluation values reaching a target value. If the average value is used, the ranking may be higher even if there are significant variations in the evaluation value. However, if the ranking is based on the percentage of the evaluation values that have reached the target value, a more accurate ranking is executed. Will be able to.

(Iv) Step 305
The automatic optimization program 2014 determines whether to end the optimization process (step 305). If the highest ranked algorithm satisfies the user's expectation, the process ends (step 306). Specifically, the user may set an evaluation value average target in advance, and the determination may be made by reaching the target value. Further, it is possible to display the transition graph of the result data and the average evaluation value so that the user can monitor, and the user can judge at any time to issue an end instruction from the keyboard, and the instruction can be ended by detecting the instruction at this step. In the automatic optimization process according to the embodiment of the present invention, since the algorithm having the highest evaluation value is always left in the next algorithm group, the transition graph of the evaluation average increases.

(V) Step 307
If it is determined that the optimization algorithm cannot be terminated because the algorithm that is considered to be the highest-order algorithm in step 304 does not satisfy the user's expectation (No in step 305), the automatic optimization program In step 2014, teaching data is selected using the evaluation results of all teaching data obtained in step 303 (step 307). This is a preparation for optimizing that only a representative one is selected in order to suppress the cost of repeating the evaluation using a large number of image data and correct answer data, and the evaluation is repeated using only the selected one. Details of this step will be described later (see FIGS. 6 to 9).

(Vi) Step 309
The automatic optimization program 2014 ranks the 50 algorithms based on the evaluation result of the teaching data selected in step 307 (the evaluation value has already been calculated in step 303 for the selected teaching data) (step 309). ). In step 309, the higher the evaluation value 2024, the higher the ranking of the algorithms. For one algorithm, evaluation values 2024 are obtained as many as the number of selected teaching data, but ranking may be based on the average value (or sum) or the ratio of evaluation values reaching the target value. This is the same as step 304 described above.

(Vii) Step 310
The automatic optimization program 2014 determines whether or not to end the optimization processing with the selected teaching data (step 310). If the highest ranked algorithm satisfies the user's expectation, the process goes to the loop for evaluating all teaching data (step 302). Whether or not the user's expectation is satisfied may be determined by reaching the target value of the average evaluation value set in advance by the user, as in step 305. However, this is not appropriate when it is not always possible to achieve this. In this embodiment, an upper limit value of the number of loop iterations of optimization is set in the selected teaching data, and determination is made based on whether or not the upper limit value has been reached. I am going to do that.

(Viii) Step 311
The automatic optimization program 2014 changes the algorithm group when the optimization processing with the selected teaching data is not completed (step 311). Specifically, the automatic optimization program 2014 refers to the evaluation value 2024 of the algorithm group, adds a change that is expected to be improved (here, any change is assumed to be a change that is expected to be improved), and the changed algorithm group Is generated. It is only necessary to select an algorithm (setting data) having a high average of selected teaching data evaluation values, and to add a combination of image processing functions processed by the image processing algorithm execution program 2011 and to make partial small changes to parameter values. For example, according to a technique called genetic algorithm or genetic programming, a new algorithm is generated by combining two algorithms selected stochastically (crossover), or an algorithm modified by random numbers is generated (mutation). . Alternatively, if it is known that there is no such thing as a local solution, a large number of algorithms may be generated by selecting a changed portion with a random number based on the highest-order algorithm and adding small changes. In order to save the best algorithm, the algorithm that was highest in the ranking in step 309 is added to the changed algorithm group without being changed. Further, the total number of algorithms in the new changed algorithm group may be 50 in the present embodiment, as in step 301.

(Ix) Step 308
The automatic optimization program 2014 instructs the evaluation program 2013 to evaluate the changed algorithm group, and the evaluation program 2013 evaluates the changed algorithm group using the teaching data selected in step 307 in response to the instruction. (Step 308). Although the process is the same as that in step 303, only the selected teaching data needs to be evaluated and can be performed at a lower cost than in step 303. After the process of step 308, the process proceeds to step 309, and the processes of steps 309 to 311 are repeated.

(X) Step 302
If the optimization is completed in step 310 (Yes in step 310), the process proceeds to step 302.

  In step 302, the automatic optimization program 2014 forms a new algorithm group. The new algorithm group in step 302 was finally generated in step 311 of the process of performing algorithm evaluation using the teaching data selected in step 307 (optimization loop (S309 → S310 → S311 → S308...)). It is the algorithm group plus the algorithm that was the highest in step 304. In order to make the total number of algorithms 50, it was generated in step 311 instead of adding the algorithm that was made the highest in S304. The lowest algorithm in the modified algorithm group is excluded.

(Xi) Steps 303 to 305
Subsequently, the automatic optimization program 2014 uses the evaluation program 2013 to evaluate all teaching data in step 303 and repeat the entire processing in steps 304 and 305. That is, it is verified whether there is an optimal algorithm for the algorithm group generated in step 302.

  As described above, if optimization is performed by combining evaluation using all teaching data and evaluation using automatically selected teaching data, only the evaluation using all teaching data is repeated without requiring user knowledge and skill. It is possible to reduce the total number of algorithm evaluations and complete the optimization earlier.

  In the present embodiment, the total number of algorithms prepared as an algorithm group is 50, but in general, the larger the number, the better. However, if the number of combinations of image processing functions processed by the image processing algorithm execution unit 106 and the range of parameter numerical values are extremely limited, it is sufficient that the number is small. Also, the method of changing the algorithm group in step 311 is only illustrated and does not limit the present invention, and other policies may be used as long as the algorithm group can be improved.

<Time series of optimization processing>
FIG. 4 (upper part) is a diagram for explaining the image processing algorithm optimization processing shown in the flowchart of FIG. 3 in time series. Moreover, FIG. 4 (lower part) also includes what showed the flow of a change of an algorithm group typically. In FIG. 4, the lower algorithm group is associated with a dotted line as a result of the upper process.

First, an initial algorithm group 401 is generated by the processing of step 301.
Then, the evaluation in step 303 is executed using all the teaching data 400 and the algorithm group 401. Algorithm ranking processing (S304) is executed based on the evaluation result. The result of this ranking processing is the algorithm group 402. The highest-order algorithm in the algorithm group 402 is added to the process of forming a new algorithm group (S302).

  The teaching data selection process (S307) is performed without ending the optimization process. Then, when ranking processing (S309) is performed based on the evaluation value of the selected teaching data, an algorithm group 403 is generated.

  Subsequently, algorithm group change processing (S311) is performed, and a change algorithm group 404 is generated.

Further, the changed algorithm group 404 is evaluated using the selected teaching data 406 (S308), and the ranked algorithm group 405 is generated.
Thereafter, the same processing is repeated and the algorithm is optimized.

  FIG. 5 is a diagram showing processing in a time series that is longer than that in FIG. In FIG. 5, the number of optimization end determination processes (S310) using the selected teaching data (the number of loops of S309 → S310 → S311 → S308) is set to 4, and the algorithm group change process (S311) and selection are performed. If the taught data evaluation (S308) is performed four times, the process proceeds to a new algorithm group formation process (S302). Thus, the entire loop of the optimization process is repeated, and the new algorithm group formation process (S302) is performed as the first time, the second time, the third time, and so on. After the n-th new algorithm group formation process (S302) is finally performed, all teaching data evaluation process (S303) and ranking process (S304) are executed. As a result, the highest-level algorithm satisfies the user's expectation. If so, the optimization process ends. The final algorithm output 501 corresponds to the highest-order algorithm at that time.

<Details of teaching data selection process>
Details of the teaching data selection process (S307) will be described with reference to FIGS.
(A) Outline of Teaching Data Selection Process FIG. 6 is a diagram showing an outline of the teaching data selection process (S307).
First, all teaching data 400 is divided and classified into several groups.
Next, one teaching data is selected as a representative from each classification. As many selection teaching data as the number of groups thus classified are determined.
Roughly, if teaching data is selected in this way, representative teaching data can be selected evenly from all teaching data with little deviation in properties.

(B) Details of Teaching Data Selection Processing FIG. 7 is a flowchart for explaining details of teaching data selection processing (S307).

(I) Step 701
The automatic optimization program 2014 obtains an evaluation value vector for each teaching data from the evaluation value for each teaching data by each algorithm (S701). In the all teaching data evaluation process (S303), evaluation values are obtained as many as the number of all algorithms for a certain teaching data, but these evaluation values are used as elements and treated as one vector (evaluation value vector). That is. The element arrangement (corresponding to the algorithm) is unified for all evaluation value vectors.

  FIG. 8 is a diagram illustrating an example of the evaluation value vector. There are N pieces of all teaching data, and each teaching data is d1 to dN. The total number of algorithm groups is M, and the individual algorithms are a1 to aM. And the evaluation value of each algorithm about each teaching data is v11-vMN. At this time, the evaluation value vectors Vd1 to VdN of the respective teaching data are (v11, v21,..., VM1) to (v1N, v2N,..., VMN).

(Ii) Step 702
The automatic optimization program 2014 establishes a link by regarding the two teaching data with the shortest distance between the evaluation value vectors as having similarity (step 702). This includes not only individual teaching data but also a distance to a group described later and a distance between groups. The Manhattan distance is used as the distance between vectors. That is, for example, the distance between the evaluation value vector Vd1 and the evaluation value vector Vd2 is (| v11-v12 | + | v21-v22 | + | v31-v32 | + ... + | vM1-vM2 |) It can be expressed as.

  An evaluation value vector composed of evaluation values of a plurality of algorithms indicates the nature of the teaching data. Therefore, in the embodiment of the present invention, the shorter the distance between the evaluation value vectors of the two teaching data, the more similar the properties are, and these are classified into one group for processing.

(Iii) Step 703
The automatic optimization program 2014 averages the evaluation values of the two teaching data linked in step 702 and all the teaching data included in the group to obtain an evaluation value vector for the new group. Thereafter, the new group is treated as equivalent to one teaching data (step 703). As an example, in the previous evaluation value vector Vd1 and evaluation value vector Vd2, the evaluation value vector Vd (1,2) of the new group is ((v11 + v12) / 2, (v21 + v22) / 2, (v31 + v32) / 2, ..., (vM1 + vM2) / 2).

  Here, handling a new group equivalent to one teaching data means that two teaching data having the shortest distance are to be searched at step 702. On the other hand, the original teaching data and the original group with the link are excluded from the target.

(Iv) Step 704
The automatic optimization program 2014 has a low evaluation value in the difference between the two teaching data linked in step 702 and the evaluation value (one of the elements of the evaluation value vector in this embodiment) of each group by the highest-order algorithm. A value obtained by multiplying the teaching data or group by the number of teaching data is set as the “step value” in the link (step 704). The highest-order algorithm used here is an algorithm that is ranked highest when ranking is performed based on the evaluation results of all teaching data (S304). The step value is a guide for grouping that prioritizes optimization.

  An example of the step value will be described with an evaluation value vector A of one teaching data and an evaluation value vector B of a group including three teaching data. Assuming that the evaluation value of the highest-order algorithm is the first element of the evaluation value vector, when the first element A0 of the vector A is smaller than the first element B0 of the vector B, the step value is (| A0−B0 | × 1) Become. Conversely, when B0 is smaller than A0, the step value is (| A0−B0 | × 3).

  Further, consider each group of teaching data taking an evaluation value as shown in FIG. 8B as an example. Here, the average of the evaluation values of each group is AV1 for group 1, AV2 for group 2, and AV3 for group 3. In this case, it is verified whether there is a step separating the groups between the group 1 and the group 2 and between the group 2 and the group 3 (whether the step value is large). The step values of groups 1 and 2 are expressed as | AV1-AV2 | × (number of teaching data of group 1) = area of S1 portion. Further, the step value between groups 2 and 3 is expressed as | AV2−AV3 | × (number of teaching data of group 3) = area of S2 portion. The difference value of the average evaluation value is larger in the difference between the groups 2 and 3 than the difference in the groups 1 and 2, but the step value is larger between the groups 1 and 2. Therefore, in the example as shown in FIG. 8B, there is a step between the groups 1 and 2, and it can be understood that efficient teaching data can be selected by treating the group 1 as an independent group. Thus, the step value (step evaluation value) is a standard for discriminating the teaching data group that is expected to be improved by the highest algorithm. With this step value, it is possible to identify the group that should be improved most in order to increase the average evaluation value in the entire teaching data. In the above example, it is specified that the teaching data group included in the group 1 is handled as an independent group by calculating the step value.

(V) Step 705
The automatic optimization program 2014 determines whether all teaching data is contained in one group (step 705). For example, if the total number of teaching data is N, it can be determined that the number of linked links is within one group if the total number of linked links is N-1.

  If all teaching data does not fit in one group (No in step 705), the automatic optimization program 2014 repeats the processing from step 702.

  If all the teaching data are contained in one group (Yes in step 705), the process proceeds to step 706.

(Vi) Step 706
The automatic optimization program 2014 searches for the link having the largest step value, and determines the classification based on the group group formed before the link is established (step 706). It is determined that the link with a large step value should be divided to make the teaching data and group independent. This process completes the classification.

(Vii) Step 707
Finally, the automatic optimization program 2014 uses the image selection program 2015 to randomly select one teaching data for each classification (step 707).
Thus, the teaching data selection process (S307) is completed.

(C) Example of Teaching Data Selection Process FIG. 9 is a diagram for explaining a state (example) in which the teaching data d1 to dN shown in FIG. 8 are linked and contained in one group.

  The first link is extended to the teaching data d1 and d2, and the second link is extended to d5 and d6. The third link extends to a group consisting of d1 and d2 and d3. Further, the fourth link is extended to the groups d1 to d3 and the groups d5 to d6. Continuing to link, and finally, at the (N-1) th time, all teaching data is connected by a link to complete one group. Here, if the step value of the fourth link is maximum, the classification is determined by the group formed by the links up to the third link, and {d1, d2, d3}, {d4}, {d5, d6}, {d7},. And one teaching data is selected from each group, for example, {d2, d4, d5, d7, ...} becomes the selected teaching data.

  If teaching data is selected as described above, representative teaching data can be selected evenly from all teaching data with little deviation in properties. Then, optimization can be advanced with less teaching data than all teaching data.

  Note that the above teaching data selection method is given as an example, and the present invention is not construed as being limited thereto. Instead of hierarchically associating teaching data, for example, if the distance between evaluation vectors is less than a certain value set in advance, it is possible to reduce the number of teaching data and proceed with optimization even if the group is treated as a group. Can do. Alternatively, instead of a fixed value, a method of investigating the distribution of distances between evaluation vectors and grouping those having a certain ratio or less may be used.

  These are simple methods, although the validity of classification is lower than the hierarchical method. In addition, the distance between groups may not be measured by the average of the evaluation value vectors, but the minimum or maximum distance between individual teaching data belonging to the group may be used.

  Further, the distance between the evaluation value vectors may be the Euclidean distance instead of the Manhattan distance in accordance with the target image processing algorithm and the evaluation value calculation method.

  In addition, the link having the largest step value is not necessarily selected and the classification is determined, but the link having a larger step value may be selected. This increases the possibility that the optimization can be advanced even when the evaluation value calculation method lacks appropriateness. Alternatively, links may be selected stochastically after the step value is multiplied by the weight of the link order so that the number of classification groups is reduced as much as possible to reduce the number of selected teaching data. Alternatively, the number of classification groups may be set in advance, and the link location that matches the number may be selected.

  Furthermore, the cost of distance calculation may be suppressed by reducing the number of elements (the number of dimensions of the vector) by limiting the elements of the evaluation value vector instead of using all the evaluation values of the algorithm group. For example, in two algorithms, if the evaluation values for all the teaching data are the same, the evaluation value of only one algorithm may be an element of the evaluation value vector, or the evaluation values are the same for all the teaching data. The evaluation value of need not be an element. Alternatively, a limiting method of selecting an algorithm that obtains the maximum evaluation value for each teaching data and making it an element of the evaluation value vector, or a limiting method of simply configuring an evaluation value vector from the highest level by a certain number of algorithms You may depend on.

  In step 707, one teaching data is not randomly selected for each classification, but a high or low evaluation value is added in the group, or a probabilistic selection method according to the evaluation value is added. You may choose from within the group.

<Application example of the present invention>
Hereinafter, the present invention will be described with more specific application examples.
(A) Application Example to Template Matching FIGS. 10 to 14 are diagrams showing an example in which the present invention is applied to template matching.

(I) Example of Template Image FIG. 10 shows an example of the template image 1001. The template matching is image processing for searching for an image similar to the alignment mark 1002 (cross shape) in the image to be searched and obtaining the position as an output result to detect the position.

(Ii) Example of Searched Image FIG. 11 is a diagram illustrating an example of a corresponding searched image. The screen shown in FIG. 11 is a window displayed on the display 204, and the inside of the area 1101 is a searched image. The searched image 1101 has an alignment mark 1102, and an area corresponding to the template image 1001 is a position 1103 indicated by a dotted line. As correct data, the user sets coordinates 1104 with a stylus pen 2033 or the like. Alternatively, a numerical value is input using the keyboard 2031 (1105).
As described above, in template matching, the template image, the image to be searched, and the position coordinates of the matching result as correct data correspond to the teaching data.

(Iii) Template Matching Evaluation FIG. 12 illustrates an example of an evaluation value calculation method performed by the evaluation unit 103 in template matching.

  When the template image 1101 and the searched image 1201 are given as input image data for image processing for template matching, it is assumed that a plurality of position coordinates are output as result data with ranking as a result. .

  As an example of the result data, it is assumed that three coordinates of candidate first rank data 1202, candidate second rank data 1203, and candidate third rank data 1204 are obtained. Further, in this example, the correct answer data is set as the position coordinates of the candidate second rank data 1203 and is not selected as the candidate first rank.

  Here, as an example of the evaluation value calculation method, 100 / (candidate order of correct position) is given. That is, since the correct answer position is the second place in the result data here, 100/2 = 50 is the evaluation value. If the candidate position of the correct answer position is first, 100/1 = 100 is the maximum. If the correct answer is not listed as a candidate, the evaluation value may be set to zero.

(Iv) Example of Evaluation Value Vector FIG. 13 is a diagram illustrating an example of evaluation value vectors for each of a large number of teaching data obtained by executing a plurality of template matching using the above-described evaluation value calculation method. . There are 60 pieces of all teaching data, and each teaching data is p1 to p60.

  The total number of algorithm groups is 50, and the individual algorithms are ap1 to ap50. And the evaluation value of each algorithm about each teaching data is the value of 0-100 of each column of a table | surface. At this time, the evaluation value vectors Vd1 to Vd60 of the respective teaching data are obtained by using the evaluation values arranged side by side as elements.

  Using these evaluation value vectors, the distance between the vectors may be calculated to select teaching data.

(V) Example of template matching algorithm to be evaluated FIG. 14 is a diagram illustrating an example of a plurality of template matching algorithms generated by combining preprocessing filters.

  As shown in FIG. 14, various filter processes 1401 are executed as a preprocess on the input image data before the basic template matching process 1402. The image processing filter is well-known image quality adjustment processing such as edge enhancement, noise reduction processing, and contrast adjustment. Combinations of these filters and variations of parameters of each filter are used as a plurality of algorithms ap1 to ap50. Alternatively, various parameter variations may be used for the template matching processing 1402.

  As described above, correct data may be set, and an appropriate evaluation value may be calculated to select teaching data. If the template matching algorithm is optimized according to the present invention, the total number of algorithm evaluations can be reduced and the optimization can be completed at an early stage rather than repeating only the evaluation with all teaching data.

(Vi) Classification of Teaching Data FIG. 15 is a diagram showing a display example of the classification process, and shows the formation of the hierarchical group described with reference to FIG. 7 for each searched image. By providing a function for displaying such a group-formed image, it becomes easier for the person who examines the algorithm to understand what classification tendency exists as the image characteristic. For example, it is easy to obtain hints even when the combination of the prepared image processing filters does not give a satisfactory result.

(B) Application Example to Defect Classification FIGS. 16 to 19 are diagrams showing an example in which the present invention is applied to defect classification. Defect classification is a process of automatically classifying whether a defect is, for example, a foreign substance or a defect in a manufacturing process when a defect of the product is detected in an apparatus for inspecting the product.

(I) Example of Defect Image FIG. 16 is a diagram showing an example of an inspection image that captures a defective portion of a product (semiconductor device).

  An area 1601 is a window displayed on the display 204, and an inspection image is displayed inside the area. The inspection image includes a foreign matter adhesion 1603 and a defective manufacturing portion 1605. On the other hand, as correct answer data, the user sets the coordinates 1604 for the foreign matter adhesion 1603 and the coordinates 1606 for the defective manufacturing portion 1605 with the stylus pen 2033. Further, the defect classification type and the like are selected and entered numerically using the keyboard 2031 (see area 1602).

  As described above, in the defect classification, the inspection image, the position coordinates of the defective portion, which is correct data, the classification type, and the like correspond to the teaching data.

(Ii) Defect Evaluation FIG. 17 is a diagram illustrating an example of an evaluation value calculation method performed by the evaluation unit 103 in defect classification.

  When an inspection image 1701 that captures a part to which a foreign substance is attached or an inspection image 1702 that captures a part where the wiring is shorted or disconnected is given as input image data for defect classification image processing, one or more results are obtained. The position information of the result part, the classification type and the size of the defect at the position are output as result data.

  Here, as an example of the result data, there are three major classifications and individual minor classifications. The major classification is the type of scratches (scratches) that may have the above-mentioned foreign matter adhesion and manufacturing defects, and possibly external factors. For foreign matter adhesion, the type of the foreign matter is a metal or chemical substance is output as a finer subcategory. For manufacturing defects, the types of short, disconnection, and deformation are output as a small classification. In addition to these large and small types, the size of the defective part is output.

  As an example of the evaluation value calculation method, (large classification accuracy x 40) + (small classification accuracy x 30) + (size accuracy x 30). The degree of correct correct classification is 1 if it matches the correct answer data prepared in advance, and 0 if it does not match. The minor classification accuracy is 0 when the major classification accuracy is 0, but is determined when the major classification accuracy is 1, and is 1 when matching with correct data, and 0 when not matching. It is assumed that the size accuracy is 1 if it matches the size of the correct data, and is closer to 0 as the difference is larger.

(Iii) Example of Evaluation Vector FIG. 18 is a diagram showing an example of evaluation value vectors for each of a large number of teaching data obtained when a plurality of defect classification algorithms are evaluated by the evaluation value calculation method shown in FIG. .

  In this example, there are 33 pieces of all teaching data, and each teaching data is represented by p1 to p33. In this example, the total number of algorithm groups is 40, and the individual algorithms are represented by ar1 to ar40. And the evaluation value of each algorithm about each teaching data is a value of 12-100 of each column of a table. At this time, the evaluation value vectors Vd1 to Vd33 of the respective teaching data are elements having evaluation values arranged horizontally. Using these evaluation value vectors, the distance between the vectors may be calculated to select teaching data.

(Iv) Example of defect classification algorithm to be evaluated FIG. 19 is a diagram illustrating an example of a plurality of defect classification algorithms for performing a combination operation of various feature values. Here, the various feature values indicate shape features in image processing such as complexity, unevenness, and moment features, texture features such as roughness, linearity, and regularity, or if the image data is color. For example, color characteristics or spectral distribution characteristics.

  An algorithm that combines one operation 1902 so that a threshold value is given as a parameter by using one or more of these feature values 1901, giving a threshold value as a basic operation such as four arithmetic operations, integerization, and absolute value, and as a parameter. Configure. These various feature values, combinations of calculations, and variations of calculation parameters are used as a plurality of algorithms ar1 to ar40. Although not shown or described, the calculation of the size of the defective part is also included in the algorithm.

  As described above, correct data may be set, and an appropriate evaluation value may be calculated to select teaching data. If the defect classification algorithm is optimized according to the present invention, it is possible to reduce the total number of algorithm evaluations and complete the optimization at an early stage, rather than repeating only the evaluation with all teaching data.

(C) Application Example to Tracking Processing FIGS. 20 to 22 are diagrams showing an example in which the present invention is applied to tracking processing (also called motion capture or the like). This is processing for detecting a position when a specific object moves in a moving image composed of a number of time-series images. This technique is used in a wide range of fields, such as high-speed and high-precision positioning in various inspection apparatuses and cell observation in the medical field. The tracking algorithm itself is originally expected to be usable in real time and has been improved in speed, but the moving image to be evaluated is various and must be long. For this reason, there is a problem that the optimization process of the algorithm is costly.

  Therefore, if the present invention is applied to the tracking process and the algorithm is optimized quickly, the cost can be suppressed.

(I) Example of Time Series Image Group and Example of Evaluation Method FIG. 20 is a diagram illustrating an example of a series of time series image group 2001. In FIG. 20, as an easy-to-understand example, an image of a situation where insects fly around is shown in a sequence of frames one by one.

  In FIG. 20, the insect is moving little by little, and the position in each image is different. The correct position of the insect is set as correct data 2003 for each of these images. On the other hand, the detection position result 2004 by the tracking process may be slightly shifted. The smaller the deviation, the better the tracking process. The correct data 2003 “double circle” and the tracking result 2004 “cross” are shown in the lower series of image areas 2002 so that they can be easily discriminated. However, the image area group 2002 and the time series image group 2001 are the same.

  When the evaluation value is calculated by applying the present invention, it is not appropriate to simply evaluate the image data one by one like the template matching described above. This is because, in the case of tracking processing, the position is estimated by associating the preceding and succeeding images in many cases, so that the evaluation is not correct. Therefore, in accordance with the content of the tracking process, a plurality of consecutive images are selected with some overlapping and used for evaluation. FIG. 20 shows that the evaluation value of the tracking process is obtained in the section where every three images, one of the front and back are overlapped.

  As described above, the evaluation value of the section composed of a plurality of image data may be calculated as, for example, (Σ (100 / (1+ [position deviation amount]))) / [number of section images].

(Ii) Example of Evaluation Value Vector FIG. 21 shows an example of evaluation value vectors for each of many sections obtained by evaluating a plurality of tracking processing algorithms using the evaluation value calculation method shown in FIG. FIG. In this example, the entire teaching data section is set to 1 to End. Further, each algorithm is assumed to be at1 to at20. The evaluation value of each algorithm for each section corresponds to the value in each column of the table.

  At this time, the evaluation value vectors Vd1 to VdE for each section of the teaching data are obtained by using the evaluation values arranged side by side as elements. Using these evaluation value vectors, the distance between the vectors may be calculated to select the teaching data section.

(Iii) Example of tracking processing algorithm to be evaluated FIG. 22 is a diagram illustrating an example of a plurality of tracking processing algorithms generated by a combination of preprocessing filters.

  As shown in FIG. 22, various filter processes 2201 are combined as a pre-process before the basic tracking process 2202. Combinations of these filters and variations of parameters of each filter are used as a plurality of algorithms at1 to at20. Alternatively, various parameter variations may be used for the tracking process 2202.

  As described above, it is only necessary to select sections of teaching data by setting sections and calculating an appropriate evaluation value for each section. If the tracking algorithm is optimized according to the present invention, the total algorithm evaluation time can be reduced and the optimization can be completed at an early stage rather than repeating only the evaluation with a long moving image.

(D) Application Example to Speech Recognition FIGS. 23 to 25 are diagrams showing an example in which the present invention is applied to speech recognition. The present invention is not limited to the image processing described so far and can be widely applied to a wide variety of data processing. Thus, according to the present invention, the evaluation cost (cost for optimization) of various data processing algorithms can be suppressed.

(I) Example of Data and Example of Evaluation Method FIG. 23 is a diagram illustrating an example of data to be evaluated and an evaluation method. In FIG. 23, a frame 2301 shows a state in which an operator of the inspection apparatus gives a stop instruction (“Stop”) of the apparatus by voice while noise is surrounding. The stereo microphone 2302 converts the sound into an electric signal instead of the imaging unit 104 so far, and sets the sound data 2303 as a processing target instead of the image data.

  The speech recognition algorithm 2304 converts speech data into a language word character string and outputs the result data, and corresponds to the image processing algorithm execution unit 106 so far.

  As an example of the evaluation value calculation method, calculation is performed by 100 / (1+ [difference from correct answer]). The “difference from correct answer” is, for example, the number of characters in which the character string of the voice recognition result data is incorrect with respect to the character string of the correct data. The correct answer data for the voice instruction “Stop” in the example of FIG. 23 is naturally “Stop”, but if the result data is “Stop”, there is no error and the difference from the correct answer is 0, and the result data is “Top”. If there is one character difference, the difference from the correct answer is 1, and if the result data is “Tap”, it is 2. In this way, the evaluation value represents a good or bad result with 100 being the perfect score.

(Ii) Examples of Evaluation Value Vectors FIG. 24 is an example of evaluation value vectors for a large number of speech data obtained by evaluating a plurality of speech recognition algorithms using the evaluation value calculation method shown in FIG. FIG. In FIG. 24, all the teaching data are speech 1 to speech 93, and the individual algorithms are aa1 to aa40. Then, it is assumed that the evaluation value of each algorithm for each audio data is obtained as the value in each column of the table. At this time, the evaluation value vectors Vd1 to Vd93 for each section of the teaching data are obtained by using evaluation values arranged horizontally. Using these evaluation value vectors, the distance between the vectors may be calculated to select the teaching data.

(Iii) Example of speech recognition algorithm to be evaluated FIG. 25 is a diagram illustrating an example of a plurality of speech recognition algorithms generated by combining a plurality of types of filters.

  As shown in FIG. 25, a combination of various filter processes (2502 to 2504) is provided as a pre-process before the basic speech recognition process 2505. The filter related to the audio data 2501 is, for example, a high-pass filter (HPF), a low-pass filter (LPF), or a directional filter based on stereo information. Combinations of these filters and variations of parameters of each filter are used as a plurality of algorithms aa1 to aa40. Alternatively, various parameter variations may be used for the speech recognition processing 2505.

  As described above, the teaching data may be selected by calculating an appropriate evaluation value. When the optimization according to the present invention is performed, the total algorithm evaluation time can be reduced and the optimization can be completed at an early stage, rather than repeating the evaluation only with a large number of all voice data.

(2) Second Embodiment FIG. 26 is a flowchart for explaining algorithm optimization processing according to the second embodiment of the present invention. The optimization process is a modification of part of the process shown in FIG. 3, and the part of “Teaching data uniform selection” (S2601) and the determination “First time?” (S2602) are different from FIG. Is different.

(I) Step 2601
After performing the first algorithm group generation (S301), the automatic optimization program 2014 performs uniform selection of teaching data (S2601). The uniform selection of teaching data is a process of randomly selecting teaching data having a smaller number than all teaching data from all teaching data by a predetermined method without duplication. The predetermined method may be, for example, that the user gives a fixed number, or a predetermined ratio (for example, 20%) of all the teaching data may be selected at random.

  Next, the automatic optimization program 2014 uses the evaluation program 2013 to perform uniformly selected teaching data evaluation (S308), and thereafter the optimization loop with the selected teaching data (S308 → S309). → S310 → S311 → S308...

(Ii) Step 2602
When the optimization using the selected teaching data is completed, the automatic optimization program 2014 determines whether or not the determination in step 2602 is “first time”. In other words, it is determined whether or not the evaluation of all teaching data (S303) has already been performed once. If it is determined in step 2602 that it is “first time” (Yes in step 2602), the process proceeds to step 303, and the new algorithm group formation process (S302) is not performed, and all teaching data evaluation (S303). Is executed. On the other hand, if it is determined in step 2602 that it is not “first time” (No in step 2602), the process proceeds to step 302, and after executing the new algorithm group formation process, the process proceeds to step 303. The subsequent processing is the same as in FIG.

  In FIG. 3, since all teaching data evaluation processing (S303) is performed after the initial algorithm group generation processing (S301), highly reliable teaching data is selected as a representative in the first teaching data selection processing (S307). On the other hand, in FIG. 26, since the teaching data uniformly selected in step 2601 is used, it cannot be said that representative teaching data is necessarily selected. However, since the total number of teaching data evaluations can be reduced by the first time, the optimization of the algorithm can be completed at an early stage when the total number of teaching data is very large.

(3) Third Embodiment FIG. 27 is a flowchart for explaining algorithm optimization processing according to the third embodiment of the present invention. This is a method of selecting teaching data using only the evaluation value of the highest-order algorithm as an element of the evaluation value vector. The optimization process is also a part of the process of FIG. 3, and step 302 is replaced with step 2701 and step 307 is replaced with step 2703. Also, step 2702 is newly added.

  In FIG. 3, teaching data selection processing (S307) is performed based on the evaluation results of all teaching data, but in step 2703, processing for selecting teaching data having a poor evaluation result of the highest algorithm is performed. This process corresponds to a method of selecting teaching data using only the evaluation value of the highest algorithm.

  Specifically, the automatic optimization program 2014 includes teaching data (teaching data within a predetermined distance) that is close to teaching data having the worst evaluation value among the evaluation results obtained in step 304 and teachings other than those. The data is classified and the teaching data of the group with the lower evaluation value is selected. In the flowchart of FIG. 3, representative teaching data is selected from all teaching data, whereas in the third embodiment, only a part of specific teaching data is selected. In addition, the optimization is advanced by making two other changes.

  The first place is processing (S302) in which a new algorithm group is formed by adding an algorithm optimized using the selected teaching data. In FIG. 3, the highest algorithm among the evaluation results of all teaching data is added to the algorithm group optimized using the selected teaching data. However, in FIG. A new algorithm group is formed by combining a large number of algorithms and a part of the higher-order algorithm from the algorithm group optimized using the selected teaching data (S2701). This is because if optimization is performed only with teaching data having a poor evaluation result, optimization does not proceed for all teaching data.

  The second point is that an algorithm group change process (S2702) is further added. In step 2702, the algorithm group is changed based on the evaluation result of all teaching data. By this processing, the algorithm optimized using the selected teaching data having a poor evaluation result and the algorithm having a comparatively good evaluation result are performed on all teaching data.

  In the process of the flowchart of FIG. 27 described above, optimization may be difficult to complete early compared to the process of FIG. However, selection of teaching data is simple, and the total number of algorithm evaluations can be reduced and optimization can be completed earlier than when only evaluation with all teaching data is repeated.

(4) Summary (i) In the present invention, a plurality of prepared data processing algorithms are executed for all of the plurality of teaching data, and all of the plurality of teaching data corresponding to the plurality of data processing algorithms are executed. Is evaluated (S303 in FIGS. 3, 26 and 27). Thereby, all the evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms are obtained. Based on the evaluation result, a plurality of algorithms are ranked, and the highest-order algorithm is specified for all teaching data (S304 in the figure). If a desired result is not obtained in the highest-order algorithm, teaching data selection processing is performed (S307 and S2703 in the figure). In the teaching data selection process, a plurality of teaching data is classified into a plurality of confirmed groups using the evaluation result of all teaching data, and one teaching data is selected from each confirmed group, and selected teaching data is generated. The Then, the prepared plurality of data processing algorithms are ranked again based on the evaluation result of the selected teaching data (included in the evaluation result of all teaching data), and it is determined whether a desired result is obtained. (S309 and S310 in the figure). Next, a new data processing algorithm group including the data processing algorithm for which the evaluation result is determined to be the highest in this ranking processing is generated (S311 in the figure). Other than the algorithm at the top in S309, for example, the setting parameter may be determined using a random number. Subsequently, a new data processing algorithm group (a plurality of improved data processing algorithms) including the highest-order algorithm in S309 and the highest-order algorithm in S304 is applied to all of the plurality of teaching data. (S302 → S303 in the figure), and the improved data processing algorithm group is evaluated (S304 in the figure). Based on this evaluation result, an optimum data processing algorithm is determined. By doing so, the data processing algorithm can be optimized automatically and at high speed even if the user has no knowledge or knowledge about the data processing algorithm to be used. become.

  The details of the teaching data selection process (S307) are as follows. That is, two teaching data with the smallest distance between evaluation value vectors (see FIG. 8A) are classified as belonging to one group, and a plurality of temporary groups are generated (Vd (1, 2 in FIG. 9). ), Vd (5,6) etc.). Then, representative evaluation value vectors of the plurality of temporary groups are generated using the evaluation value vectors of the teaching data belonging to the plurality of temporary groups. The representative evaluation value vector is obtained, for example, by averaging the evaluation value vectors of the teaching data belonging to the temporary group. Next, among the representative evaluation value vectors of a plurality of temporary groups, or the representative evaluation value vectors of each of the plurality of temporary groups and the evaluation value vectors of teaching data not belonging to the plurality of temporary groups, the temporary distance between the vectors is minimized. Further temporary groups are generated with the group or the teaching data belonging to one group (see Vd (1,2,3), Vd (1,2,3,5,6) in FIG. 9). The process of generating a further temporary group is repeated until all the teaching data are within one group (see Vd (all) in FIG. 9). Then, the provisional difference in which the difference (step value) between the average value of the evaluation values of the teaching data included in the temporary group or the further temporary group and the number of teaching data included in the temporary group or the further temporary group is maximized. A group or further temporary group is identified (see FIG. 8B). Further, the provisional group and the further provisional group that are generated before the process of creating the specified provisional group or the further provisional group are set as a plurality of confirmed groups. Finally, one teaching data is selected from a plurality of confirmed groups, and the selected teaching data is generated. In this way, since the concept of the step value is introduced and the teaching data to be processed is selected, it becomes possible to identify the group that should be improved most in order to increase the average evaluation value in the entire teaching data, and efficiently. Data processing algorithm optimization can be performed.

(Ii) As a second embodiment of the present invention, at least a part of a plurality of teaching data is uniformly selected (S303 in FIG. 26) before all teaching data is evaluated (S303 in FIG. 26). S2601). Then, a plurality of data processing algorithms prepared for the uniformly selected teaching data are executed, and the teaching data uniformly selected corresponding to the plurality of data processing algorithms are evaluated (prior evaluation) (FIG. 26). S308 → S309 → S310). Based on the result of this prior evaluation, all teaching data are evaluated (S303 in FIG. 26). In this way, when the input data (image data) to be processed has little variation and it is not necessary to first evaluate all the teaching data (combination of input data and correct answer data), It becomes possible to efficiently optimize the data processing algorithm.

(Iii) As the third embodiment of the present invention, as the details of the teaching data selection process (S307), the teaching having the worst evaluation value among all the evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms Classification data is classified into teaching data within a predetermined distance from the data and teaching data outside the predetermined distance to generate a confirmed group, and the teaching data within the predetermined distance is selected as teaching data (FIG. 27 S2703). By doing so, the total number of teaching data to be evaluated can be reduced, so that the optimization of the data processing algorithm can be executed quickly.

(Iv) The present invention can also be realized by software program codes that implement the functions of the embodiments. In this case, a storage medium in which the program code is recorded is provided to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention. As a storage medium for supplying such program code, for example, a flexible disk, CD-ROM, DVD-ROM, hard disk, optical disk, magneto-optical disk, CD-R, magnetic tape, nonvolatile memory card, ROM Etc. are used.

  Also, based on the instruction of the program code, an OS (operating system) running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. May be. Further, after the program code read from the storage medium is written in the memory on the computer, the computer CPU or the like performs part or all of the actual processing based on the instruction of the program code. Thus, the functions of the above-described embodiments may be realized.

  Further, by distributing the program code of the software that realizes the functions of the embodiment via a network, it is stored in a storage means such as a hard disk or memory of a system or apparatus, or a storage medium such as a CD-RW or CD-R And the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage means or the storage medium when used.

  Finally, it should be understood that the processes and techniques described herein are not inherently related to any particular apparatus, and can be implemented by any suitable combination of components. In addition, various types of devices for general purpose can be used in accordance with the teachings described herein. It may prove useful to build a dedicated device to perform the method steps described herein. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined. Although the present invention has been described with reference to specific examples, these are in all respects illustrative rather than restrictive. Those skilled in the art will appreciate that there are numerous combinations of hardware, software, and firmware that are suitable for implementing the present invention. For example, the described software can be implemented in a wide range of programs or script languages such as assembler, C / C ++, perl, shell, PHP, Java (registered trademark).

  Furthermore, in the above-described embodiment, control lines and information lines are those that are considered necessary for explanation, and not all control lines and information lines on the product are necessarily shown. All the components may be connected to each other.

  In addition, other implementations of the invention will be apparent to those skilled in the art from consideration of the specification and embodiments of the invention disclosed herein. The specification and specific examples are merely exemplary, and the scope and spirit of the invention are indicated in the following claims.

DESCRIPTION OF SYMBOLS 100 ... Image processing system 101 ... Correct answer setting part 102 ... Correct answer data storage part 103 ... Evaluation part 104 ... Imaging part 105 ... Image data storage part 106 ... Image processing algorithm execution Unit 107 ... result data storage unit 108 ... automatic optimization unit 109 ... algorithm setting data storage unit 110 ... evaluation value storage unit 111 ... selection function unit 201 ... processor 202 ... Memory / HDD
203... Input device 204... Output device 205.

Claims (18)

A data processing apparatus for determining a data processing algorithm for processing input data,
A memory for storing a plurality of teaching data;
A processor for optimizing a data processing algorithm so that each evaluation result for the plurality of teaching data falls within a predetermined value range,
The processor is
A plurality of prepared data processing algorithms are executed for all of the plurality of teaching data, all of the plurality of teaching data are evaluated corresponding to the plurality of data processing algorithms, and the evaluation result is obtained. A first evaluation process that identifies a top-level data processing algorithm for all teaching data evaluations based on:
Based on all evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms, the plurality of teaching data are classified into a plurality of confirmed groups, and at least one teaching data is selected from each confirmed group. Teaching data selection processing,
A ranking process that ranks the plurality of data processing algorithms that performed data processing on the selected teaching data;
A plurality of improved data processing algorithms including a data processing algorithm whose evaluation result is determined to be the highest in the ranking processing is generated, and the improved plurality of data processing algorithms are all included in the plurality of teaching data. And a second evaluation process for evaluating the improved plurality of data processing algorithms;
An algorithm determination process for determining an optimal data processing algorithm based on the result of the second evaluation process;
A data processing apparatus characterized by executing In claim 1,
The teaching data includes image data, and the data processing algorithm is an image processing algorithm. In claim 2,
In the teaching data selection process, the processor uses an evaluation result of each teaching data corresponding to each of the plurality of data processing algorithms as an evaluation value vector, and classifies the teaching data based on a distance between the evaluation value vectors. A data processing apparatus. In claim 3,
The processor, in the teaching data selection process,
A process of classifying two teaching data having the smallest distance between the evaluation value vectors as belonging to one group, and generating a plurality of temporary groups;
A process of generating representative evaluation value vectors of the plurality of temporary groups using evaluation value vectors of teaching data belonging to the plurality of temporary groups;
The temporary evaluation between the representative evaluation value vectors of the plurality of temporary groups, or the representative evaluation value vector of each of the plurality of temporary groups and the evaluation value vector of teaching data that does not belong to the plurality of temporary groups has the smallest inter-vector distance. A process of generating a further temporary group with the group or teaching data belonging to one group;
A process of repeating the process of generating the further temporary group until all the teaching data fit in one group;
A temporary group in which a difference between multiplication values of an average value of teaching data included in the temporary group or the further temporary group and the number of teaching data included in the temporary group or the further temporary group is maximized, or The process of identifying additional temporary groups;
Processing the temporary group and the further temporary group generated before executing the process of generating the specified temporary group or further temporary group as the plurality of confirmed groups;
A process of selecting at least one teaching data from the plurality of confirmed groups;
A data processing device for executing. In claim 1,
The processor uniformly selects at least a part of a plurality of teaching data, executes the prepared plurality of data processing algorithms on the uniformly selected teaching data, and the plurality of data processing algorithms Is evaluated by the uniformly selected teaching data, and the first evaluation is performed by a plurality of data processing algorithms including a data processing algorithm in which the evaluation result is determined to be the highest in the prior evaluation processing. A data processing apparatus that executes an evaluation process. In claim 1,
The processor is
In the teaching data selection processing, teaching data within a predetermined distance from teaching data having the worst evaluation value among all evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms, and the predetermined data A data processing apparatus, characterized in that the determined group is generated by classifying the data into teaching data outside the distance, and the teaching data within the predetermined distance is used as the selected teaching data. In claim 1,
The memory stores a plurality of teaching data composed of a combination of a plurality of target data and a plurality of correct data indicating a processing portion in each of the plurality of target data,
The processor optimizes a data processing algorithm so that a processing result of the plurality of target data and a respective evaluation result for the plurality of correct data are within a predetermined value range. Processing equipment. In claim 1,
In the ranking processing, the processor performs data processing on the selected teaching data based on all evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms. Ranking the data processing algorithms,
In the second evaluation process, the processor has been improved by including a top-level data processing algorithm for all the teaching data evaluations and a data processing algorithm in which the evaluation result is determined to be the top in the ranking process. A data processing apparatus that generates a plurality of data processing algorithms. A data processing device according to claim 2;
An electronic microscope device that acquires image data as the teaching data and provides the image data to the data processing device;
The image processing algorithm is a template matching algorithm,
The data processing apparatus executes a pattern matching process using the determined template matching algorithm. A data processing device according to claim 2;
An electronic microscope device that acquires image data as the teaching data and provides the image data to the data processing device;
The image processing algorithm is a defect inspection processing algorithm,
The said data processing apparatus performs the defect classification | category process of a product using the determined said defect inspection process algorithm, The defect inspection system characterized by the above-mentioned. A data processing device according to claim 2;
An image pickup device that acquires image data as the teaching data and provides the image data to the data processing device;
The image processing algorithm is a tracking algorithm,
The image processing system, wherein the data processing device executes a tracking process using the determined tracking algorithm. A data processing method for determining a data processing algorithm for processing teaching data input to a data processing device,
The data processing device includes a memory for storing a plurality of teaching data, a processor for optimizing a data processing algorithm so that each evaluation result for the plurality of teaching data is within a predetermined value range, Have
The data processing method includes:
The processor executes a plurality of prepared data processing algorithms for all of the plurality of teaching data, and evaluates all of the plurality of teaching data corresponding to the plurality of data processing algorithms. 1 evaluation step;
The processor classifies the plurality of teaching data into a plurality of confirmed groups based on all evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms, and at least one teaching from each confirmed group Teaching data selection step for selecting data; and
A ranking step in which the processor ranks the plurality of data processing algorithms that have performed data processing on the selected teaching data;
The processor generates a plurality of data processing algorithms including a data processing algorithm in which the evaluation result is determined to be the highest in the ranking processing, and the plurality of data processing algorithms are applied to all of the plurality of teaching data. And executing a second evaluation step for evaluating a plurality of data processing algorithms including the data processing algorithm determined to be the highest level,
An algorithm determining step in which the processor determines an optimal data processing algorithm based on an evaluation result of the second evaluation step;
A data processing method comprising: In claim 12,
The teaching data is image data, and the plurality of data processing algorithms are image processing algorithms,
In the teaching data selection step, the processor classifies the teaching data on the basis of a distance between the evaluation value vectors by using an evaluation result of each teaching data corresponding to each of the plurality of image processing algorithms as an evaluation value vector. A data processing method. In claim 13,
The teaching data selection step includes:
The processor classifying two pieces of teaching data having the smallest distance between the evaluation value vectors as belonging to one group, and generating a plurality of temporary groups;
The processor generates representative evaluation value vectors of the plurality of temporary groups using evaluation value vectors of teaching data belonging to the plurality of temporary groups;
The processor has a vector-to-vector distance among representative evaluation value vectors of the plurality of temporary groups, or among representative evaluation value vectors of the plurality of temporary groups and evaluation value vectors of teaching data not belonging to the plurality of temporary groups. Generating a temporary group or a temporary group that is the smallest temporary data or teaching data belonging to one group;
The processor repeating the process of generating the further temporary group until all the teaching data fit into one group;
The processor has a maximum difference in the multiplication value between the average value of the teaching data included in the temporary group or the further temporary group and the number of teaching data included in the temporary group or the further temporary group. Identifying a temporary group or a further temporary group,
The processor sets the temporary group and the further temporary group generated before executing the process of generating the specified temporary group or the further temporary group as the plurality of confirmed groups;
The processor selecting at least one teaching data from the plurality of confirmed groups;
A data processing method comprising: The claim 12, further comprising:
The processor uniformly selects at least a part of a plurality of teaching data, executes the plurality of prepared data processing algorithms on the uniformly selected teaching data, and the plurality of data processing algorithms Performing a pre-evaluation process that evaluates the uniformly selected teaching data,
The data processing method, wherein the processor executes the first evaluation step by a plurality of data processing algorithms including a data processing algorithm in which the evaluation result is determined to be the highest in the prior evaluation processing. In claim 12,
In the teaching data selection step, the processor includes teaching data within a predetermined distance from teaching data having the worst evaluation value among all the evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms. The data processing method is characterized in that the determined group is generated by classifying the data into teaching data outside the predetermined distance and the teaching data within the predetermined distance is used as the selected teaching data. In claim 12,
The memory stores a plurality of teaching data composed of a combination of a plurality of target data and a plurality of correct data indicating a processing portion in each of the plurality of target data,
The processor optimizes a data processing algorithm so that a processing result of the plurality of target data and a respective evaluation result for the plurality of correct data are within a predetermined value range. Processing method. In claim 12,
In the ranking step, the processor performs data processing on the selected teaching data based on all evaluation results of the plurality of teaching data corresponding to the plurality of data processing algorithms. Ranking the data processing algorithms,
And a data processing method.

Images