JP2019200628A - Arithmetic apparatus, data processing method, and data processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for selecting data representative of an attribute from a plurality of data having the same attribute. An arithmetic unit sets, in a plurality of clusters, a set of points in a vector space indicating the similarity of two data having the same attribute selected from a first plurality of data having any of a plurality of attributes. Division processing for division, first selection processing for selecting at least one point from each of the plurality of clusters, and second plurality of data serving as a basis for the plurality of points selected from each of the plurality of clusters When the same data as any of the above is included in the second plurality of data, a second selection process of selecting the same data is executed. [Selection diagram] Fig. 5

Description

  The present invention relates to an arithmetic device, a data processing method, and a data processing system. Specifically, the present invention relates to a technique for selecting data representative of an attribute from a plurality of data having the same attribute.

  In recent years, in order to effectively utilize unlabeled data that has not been given a label indicating the attribute, the attribute of unlabeled data is based on the similarity between unlabeled data and labeled data to which a label has been assigned. May be determined (see, for example, Patent Document 1).

US Patent Application No. 2017/0140035

  By the way, a plurality of labeled data having a predetermined attribute may include data that is the same as data included in other attributes or data that is difficult to distinguish. For example, the data “Aoyama” with the label indicating the attribute “Last name” is the same as the data “Aoyama” with the label indicating the attribute “Place name”. In general, it is preferable to remove such data in advance so that the computer can determine the attribute of the data while reducing the amount of data handled by the computer.

  Therefore, an object of the present invention is to provide a technique for selecting data representing an attribute from a plurality of data having the same attribute.

  The arithmetic device of the present invention that solves the above-described problem is a set of points in a vector space indicating the similarity of two data having the same attribute selected from the first plurality of data having any one of the plurality of attributes. A division process for dividing the plurality of clusters; a first selection process for selecting at least one point from each of the plurality of clusters; and a second basis for a plurality of points selected from each of the plurality of clusters. When the same data as any of the plurality of data is included in the second plurality of data, a second selection process for selecting the same data is performed.

  According to the present invention, data representing an attribute can be selected from a plurality of data having the same attribute.

1 is a diagram showing a configuration of a labeling system 10. FIG. It is a figure which shows an example of training data DB30. It is a figure which shows an example of standard data DB31. It is a figure which shows the functional block implement | achieved by the arithmetic unit. It is a flowchart which shows the process which produces | generates standard data DB31. It is a figure which shows the set 300 of the point which shows the similarity of two data with the same attribute. It is a figure showing the state where point set 300 was divided into a plurality of clusters. It is a figure which shows an example of the some data contained in the selected point. It is a figure which shows the functional block implement | achieved by the arithmetic unit. It is a flowchart which shows the process which produces | generates the learned model 32. FIG. It is a figure which shows the calculation result of the calculation part 110 in vector space. FIG. 4 is a diagram showing functional blocks implemented in the arithmetic device 60. It is a flowchart which shows the process which provides a label. It is a figure which shows the identification information I and the probability P corresponding to the point of vector space.

--- Labeling system 10 ---
FIG. 1 is a diagram showing the configuration of the labeling system 10. The label assignment system 10 (data processing system) is a system that determines an attribute of unlabeled data to which no label indicating an attribute is assigned, and assigns a label indicating the attribute to the unlabeled data. The labeling system 10 includes a storage device 20, a data processing device 21, and a labeling device 22.

  The storage device 20 is a non-volatile storage means area such as a hard disk, and stores various information such as programs and databases. The storage device 20 stores a training data DB (database) 30, a standard data DB 31, and a learned model 32.

The training data DB 30 (first plurality of data) is a database in which a plurality of data is classified for each attribute of the data. FIG. 2 is a diagram illustrating an example of the training data DB 30. The training data DB 30 is created in advance by a system administrator or the like and stored in the storage device 20. The first column of the training data DB 30 includes “Alice”, “Bob”, and “John”, which are data (hereinafter, also referred to as data having an attribute) with a label indicating an attribute of “name”. . Here, for example, information “a0” is added before “Alice” and described as “a0: Alice”, but “a0” is a symbol that simply indicates “Alice”. . “A1” to “c2” and the like are the same as “a0”.

  The second column of the training data DB 30 includes “Hokkaido”, “Okinawa”, and “Tokyo”, which are data having an attribute of “address”, and the third column is “30” that is data having an attribute of “age”. Includes “generation”, “20s”, and “50s”. The training data DB 30 includes three or more attributes, but is omitted for convenience in FIG. 2 and only three attributes are shown. Further, the row of the training data DB 30 indicates information related to the person in the first column. For example, the information on the second line is the “address” and “age” of the “name” person named “Alice”.

  The standard data DB 31 (fourth data) is a database in which data (standard data, canonical data) obtained by removing data that does not represent attributes from the training data DB 30 is stored. FIG. 3 is a diagram illustrating an example of the standard data DB 31. Here, among “Alice”, “Bob”, and “John” included in the first column of the training data DB 30, “Bob” and “John” are excluded as data that does not represent the attribute of “name”. Yes. That is, here, “Alice” is selected as data representing the attribute of “name”. A method for generating the standard data DB 31 will be described later.

  The learned model 32 is a model that outputs, based on the similarity (similarity) of two data, identification information I indicating whether or not these data have the same attribute, and a probability P having the same attribute. It is. Details of the learned model 32 will be described later.

  The data processing device 21 is a device that generates a standard data DB 31 and a learned model 32 by executing a predetermined program. The data processing device 21 includes an arithmetic device 50, a memory 51, a storage device 52, an input device 53, a display device 54, and a communication device 55.

  The arithmetic device 50 (first arithmetic device) realizes various functions by executing programs stored in the memory 51 and the storage device 52.

  The memory 51 is, for example, a RAM (Random Access Memory) or the like, and is used as a temporary storage area for programs and data.

  The storage device 52 is a non-volatile storage means area such as a hard disk and stores various information such as programs and databases. The storage device 52 of the present embodiment stores a program 70 that is executed when the standard data DB 31 is generated and a program 71 that is executed when the learned model 32 is generated.

  The input device 53 is, for example, a touch panel or a keyboard, and is a device that receives a user operation result or input. The display device 54 is a display, for example, and displays operation results, processing results, and the like.

  The communication device 55 is a communication unit such as a network interface, and transmits and receives data to and from the storage device 20 and the labeling device 22 via the network.

  The label assigning device 22 is a device that determines the attribute of unlabeled data and assigns a label indicating the attribute to unlabeled data. The label assigning device 22 includes an arithmetic device 60 (second arithmetic device), a memory 61, a storage device 62, an input device 63, a display device 64, and a communication device 65. Here, each device such as the arithmetic device 60 included in the label applying device 22 is the same as the arithmetic device 50 included in the data processing device 21. For this reason, detailed description of the arithmetic unit 60 etc. is abbreviate | omitted. Note that the storage device 62 of this embodiment stores a program 75 that is executed when a label is assigned to unlabeled data.

--- Generation of standard data DB 31 ---
<< Functional Block of Computing Device 50 >>
FIG. 4 is a diagram illustrating functional blocks implemented in the arithmetic device 50 when the arithmetic device 50 executes the program 70 for generating the standard data DB 31.

  In the arithmetic device 50, a calculation unit 100, a division unit 101, selection units 102 and 103, and an update unit 104 are realized.

  The calculation unit 100 calculates a set of similarity vectors between two pieces of data in the training data DB 30. Specifically, the calculation unit 100 calculates the Jaccard similarity and the TF-IDF cosine similarity between the data “x” having the same attribute and the data “y”. Output a vector. Here, the Jaccard similarity between the data “x” and the data “y” is expressed as sim1 (x, y), and the TF-IDF cosine similarity between the data “x” and the data “y”. The degree is represented by sim2 (x, y). The calculation unit 100 calculates a similarity vector (a point in a vector space indicating similarity) between all two pieces of data having the same attribute using the following equation (1).

s (x, y) = (sim1 (x, y), sim2 (x, y)) (1)
Here, (x, y) is (a0, a1), (a1, a2), (b0, b1).
(C1, c2) and the like. In addition, the calculation unit 100 calculates, for example, the Jaccard similarity and the TF-IDF cosine similarity, but may be other similarities (for example, a similarity based on the Levenshtein distance). Accordingly, s (x, y) may be a vector having three or more dimensions.

  The dividing unit 101 divides the set of similarity vectors indicating the calculation result of Expression (1) into n clusters.

  The selection unit 102 selects m vectors closest to the center of each of the n clusters for each of the n clusters.

  Based on the m × n points (first plurality of points) selected by the selection unit 102, the selection unit 103 selects, from the training data DB 30, the standard data DB 31 from which data that does not represent attributes is removed. . Although the details of the selection unit 103 will be described later, the selection unit 103 includes k (predetermined number) or more of the same data in a plurality of data (second plurality of data) based on m × n points. If so, select that data. Further, the selection unit 103 selects all the data (third plurality of data) having an attribute different from the attribute of the k pieces of the same data in the training data DB 30.

  The update unit 104 accesses the storage device 20 and updates information in the standard data DB 31 stored in the storage device 20.

<< Process S100 for Generating Standard Data DB31 >>
The process S100 for generating the standard data DB 31 will be described. Here, the dividing unit 101 divides the set of points into seven (n = 7) clusters, and the selecting unit 102 selects two (m = 2) points from each cluster. To do. Further, if the same data is included in two pieces (k = 2), the selection unit 103 sets the data as a representative of the attribute.

  First, the calculation unit 100 selects two data (for example, (a0, a1)) having the same attribute from the training data DB 30 (S200). Then, the calculation unit 100 calculates a similarity vector indicating the similarity between the two selected data using Expression (1) (S201). As a result, for example, s (a0, a1) = (sim1 (a0, a1), sim2 (a0, a1)) is calculated. Then, the calculation unit 100 determines whether or not all pairs having the same attribute in the training data DB 30 have been selected (S202). When all the pairs have not been selected (S202: No), the process S200 is executed. On the other hand, when all the pairs have been selected (S202: Yes), the point set 300 (first point set) corresponding to each of all the pairs is represented in the vector space shown in FIG. become. In FIG. 6, only the point corresponding to s (a0, a1) is assigned the symbol “s (a0, a1)”, but each of the other points is also in another pair (for example, (b0, b1)). And (c10, c11)). As described above, the training data DB 30 in FIG. 2 describes three pieces of data as data having respective attributes, but actually includes a large number. For this reason, many points are described also in the vector space of FIG.

  If the calculation unit 100 determines in step S202 that all pairs have been selected (S202: Yes), the dividing unit 101 converts a set of vector space points 300 into seven clusters 310 to 310 as shown in FIG. The process is divided into 316 (S203: division process).

The selection unit 102 selects, for each cluster, two points that are closest to the center point of each of the seven clusters 310 to 316 (S204: first selection process). For example, if the center point of the cluster 310 is the point P0 and the points closest to the point P0 are the points P1 and P2, the selection unit 102 selects the points P1 and P2 in the cluster 310. Since the selection unit 102 performs the same processing as that of the cluster 310 for each of the clusters 311 to 316, in the process S204, as a result, 14 points (= 7 × 2) are selected. . Here, each of the 14 points indicates the similarity between two pieces of data (for example, (a0, a1)) having the same attribute. Therefore, for example, when 14 points are selected, 28 data are selected.

  The selection unit 103 determines whether or not two pieces of the same data are included in the 28 pieces of data (second plurality of pieces of data) that are the basis of the 14 points (S205). If two pieces of the same data are included (S205: Yes), the selection unit 103 differs from the training data DB 30 in that the data that includes the two pieces of the same data is different from the attributes of the two pieces of the same data. All data (third plurality of data) of the attribute are selected (S206: second selection process). On the other hand, if two identical data are not included (S205: No), the selection unit 103 selects, for example, all of the training data DB 30 (S207).

  Here, the details of the process executed by the selection unit 103 will be described with reference to FIG. FIG. 8 is an example of 28 data that is the basis of 14 points. For example, when 14 points include (a0, a1), (a0, a2), (b1, b2), etc., the number of data “a0” is 2 as shown in FIG. The number of data “a1”, “a2”, “b1”, and “b2” is one. In FIG. 8, for convenience, only some of the 28 data are shown, and others are omitted. In such a case, the selection unit 103 selects all the data having the attributes “address” and “age” different from the attribute “name” of the data “a0” and the two pieces of the same data in the training data DB 30. . By executing such processing, representative data “a0” can be selected from the data having the “name” attribute. In FIG. 8, it is assumed that there are two pieces of data “a0”. For example, when there are two pieces of the same data out of 28 pieces of data (S205: No), the selection unit 103 selects the training data DB 30. (S207).

  Then, the update unit 104 stores the data selected in step S206 in the storage device 20 as the standard data DB 31 in order to update the information stored in the storage device 20 (S208: update process). As a result, for example, the standard data DB 31 shown in FIG. 3 in which only “a0” is selected in the “name” attribute is obtained. In FIG. 3, for convenience, the data having the attribute of “address” or “age” describes only “b0” to “b2” and “c0” to “c2”, but for example, “b3” , “C3” and the like. In step S206, all data (third plurality of data) having the attributes “address” and “age” are selected.

  For example, if two pieces of data “a1” are included in addition to data “a0” in step S205, data “a0” and “a1” are selected in step S206.

--- About generation of learned model 32 ---
<< Functional Block of Computing Device 50 >>
FIG. 9 is a diagram illustrating functional blocks implemented in the arithmetic device 50 when the arithmetic device 50 executes the program 71 for generating the learned model 32.

  In the arithmetic device 50, a calculation unit 110, an identification information adding unit 111, a determination unit 112, and a training unit 113 are realized.

The calculation unit 110 calculates a set of similarity vectors (points in a vector space indicating similarity) between two pieces of data in the training data DB 30. Specifically, the calculation unit 110 selects two data (“x”, “y”) from the training data DB 30, and calculates the similarity vector between the two data using the above-described equation (1). To do. Note that the calculation unit 110 performs the calculation of Expression (1) for all data pairs included in the training data DB 30 regardless of whether the attributes are the same.

  The identification information adding unit 111 adds identification information I indicating whether the calculation result of the calculation unit 110 is based on two data having the same attribute or based on two data having different attributes. For example, when the similarity between two data “a0” and “a1” having the same attribute is calculated, the identification information adding unit 111 determines the identification information I “True” (hereinafter, the identification information I) from the calculation result. “T”). On the other hand, when the similarity between two data “a0” and “b1” having different attributes is calculated, the identification information adding unit 111 determines the identification information I “False” (hereinafter, the identification information I) from the calculation result. "F").

  The determination unit 112 determines whether all data pairs included in the training data DB 30 have been selected and the calculation has been performed.

  The training unit 113 calculates the calculation result with the identification information I (for example, s (a0, a1) = (1, 35) “T”, s (a1, a2) = (5, 58) “T”, s (A0, b0) = (2, 58) “F” etc.) is used as learning data (teacher data), and a function (learned model 32) for reproducing these learning data is obtained.

<< Process S110 for Generating Learned Model 32 >>
Here, the processing S110 for generating the learned model 32 will be described with reference to FIG. First, the calculation unit 110 selects two pieces of data from the training data DB 30 (S210). Then, the calculation unit 110 calculates the similarity vector of the selected two pieces of data using Expression (1) (S211). For example, when the two selected data are data “a1” and “b1”, s (a1, b1) = (sim1 (a1, b1), sim2 (a1, b1)) is calculated in the process S210. Will be.

  Then, the identification information adding unit 111 refers to the attributes of the two data used by the calculation unit 110, and adds identification information I to the calculation result (S212). For example, when “s (a1, b1)” indicating similarity vectors of data “a1” and “b1” having different attributes is calculated in the process S211, the identification information adding unit 111 sets “s (a1, b1) The identification information I “F” is given to the calculation result of “)”.

  FIG. 11 is a diagram illustrating a calculation result of the calculation unit 110 in the vector space. Here, each of the points included in the set 350 (first point set) indicates a calculation result to which the identification information I “T” is attached, and is included in the set 351 (second point set). Each indicates a calculation result to which identification information I “F” is attached.

Further, the determination unit 112 determines whether or not all data pairs included in the training data DB 30 have been calculated (S213). Here, when calculation of all data pairs has not been performed (S213: No), processing S210 is executed, and a new pair of data is selected. On the other hand, when calculation of all data pairs is performed (S213: Yes),
The training unit 113 executes training using the calculation result with the identification information I as learning data (S214). Specifically, the training unit 113 uses the points shown in the vector space in FIG. And if the training part 113 calculates | requires the learned model 32 using all the learning data obtained from training data DB30, it will store the learned model 32 in the memory | storage device 20 (S215). Note that the learned model 32 of the present embodiment outputs the identification information I indicating whether or not the two data have the same attribute together with the probability P based on the calculation result indicating the similarity between the two data. For example, if the learned model 32 determines that, for example, the (10, 35) point in the vector space is a calculation result based on data having the same attribute and the probability is 70%, the learned model 32 is identified. Information I “T” and probability P “70%” are output.

--- About labeling process ---
<< Functional Block of Computing Device 60 >>
FIG. 12 is a diagram illustrating functional blocks implemented in the arithmetic device 60 when the arithmetic device 60 of the label attaching device 22 executes the program 75 for assigning labels.

  In the arithmetic device 60, a calculation unit 150, an output unit 151, a determination unit 152, and a label assignment unit 153 are realized.

  The calculation unit 150 calculates a plurality of similarity vectors (second plurality of points) between each piece of data in the standard data DB 31 and unlabeled input data “x” based on Expression (1).

  The output unit 151 outputs the identification information I and the probability P using the learned model 32 for each of the plurality of points in the calculated vector space. Specifically, the output unit 151 uses the learned model 32 and indicates whether or not the identification information I has the same attribute as the data paired with the input data “x” and the probability of the identification information I. Probability P (probability of “T” or “F”) is output.

  The determination unit 152 determines the attribute of the input data “x” based on the highest probability P among the probabilities P output from the output unit 151.

  The label attaching unit 153 assigns and outputs a label indicating the attribute determined by the determining unit 152 to the input data “x”.

<< Labeling Process S120 >>
Here, with reference to FIG. 13, the process S120 for assigning a label to unlabeled input data “x” will be described. First, the calculation unit 150 calculates a plurality of similarity vectors between each piece of data in the standard data DB 31 and the input data “x” using Expression (1) (S220). Here, as shown in FIG. 3, since the standard data DB 31 includes data “a0”, “b1”, and the like, the calculation unit 150 calculates s (x, a0), s (x, b0), and the like. As a result, as shown in FIG. 14, a plurality of similarity vectors (second) in a vector space such as s (x, a0) = (77.8, 5), s (x, b0) = (2, 59), etc. A plurality of points).

The output unit 151 outputs the identification information I and the probability P using the learned model 32 for each of the plurality of calculated points (for example, (77.8,5), (2,59)). To do. For example, when the learned model 32 is used for (77.8,5) which is the calculation result of s (x, a0), the learned model 32 has the identification information I “F” and the probability P “70. % ”Is output. That is, the learned model 32 outputs information that the data “x” and the data “a0” have different attributes and the probability is “70%”. Then, the determination unit 152 determines the attribute of the input data “x” based on the highest probability P among the probabilities P output from the output unit 151 (S222). In the example shown in FIG. 14, the probability P based on the calculation result of s (x, b2) is the highest at 95%. In such a case, the determination unit 152 determines that the input data “x” is data having “address” which is an attribute of the data “b2”. The label assigning unit 153 assigns and outputs a label indicating the attribute determined by the determining unit 152 to the input data “x” (S223). Specifically, the label assigning unit 153 attaches a label indicating “address” to the input data “x” and outputs it. By executing such processing, a correct label indicating the attribute of unlabeled data is given to unlabeled data with a high probability. In the process S220, the standard data DB 31 having a smaller data amount than the training data DB 30 is used. For this reason, in the label provision process S120, the load on the arithmetic device 60 is reduced.

---- Summary ---
The label application system 10 according to the present embodiment has been described above. The arithmetic device 50 according to this embodiment selects the same data as any data included in the set from the set of two data having the same attribute. Since the data selected in this way generally includes many feature quantities of the attributes of the data, according to the present embodiment, data representing the attributes can be selected from a plurality of data having the same attributes.

  Moreover, at least the following will be made clear by the description of the present specification. That is, the first selection process may be a process of selecting a point closest to the center point of each of the plurality of clusters from each of the plurality of clusters.

  In the present embodiment, the points included in each of the clusters 310 to 316 shown in FIG. 7 may be selected at random. However, by selecting a point close to the center point of each of the clusters 310 to 316, it is possible to select a plurality of data pairs that are greatly different in similarity between data. As a result, data representing the attribute can be selected from a plurality of data having the same attribute with higher accuracy.

  The second selection process may be a process of selecting the same data when the second plurality of data includes a predetermined number of two or more of the same data.

  When any set of two or more pieces of data is included in a set of two data having the same attribute, such data is generally data including a large amount of feature values of the attribute. For this reason, according to the present embodiment, data representing the attribute can be selected with higher accuracy from a plurality of data having the same attribute.

  In the second selection process, the third plurality of data having an attribute different from the attribute of the same data among the first plurality of data and the same data as the fourth plurality of data. It is good also as processing to choose.

  By executing such processing, the standard data DB 31 can be generated from the training data DB 30.

  Further, the arithmetic device may execute an update process for updating information in the storage device so that the selected fourth plurality of data is stored in the storage device corresponding to each attribute. .

  By executing such processing, the information in the standard data DB 31 is updated, so that the label assignment system 10 can use the latest standard data DB 31.

Moreover, the label assignment system 10 including the arithmetic devices 50 and 60 of the present embodiment can assign a correct label indicating the attribute of unlabeled data to unlabeled data with high probability. Further, in the label assignment system 10, the standard data DB 31 having a smaller data amount than the training data DB 30 is used when assigning labels. For this reason, compared with the case where a label is provided using the training data DB 30, the calculation amount of the arithmetic device 60 can be reduced.

  In addition, the said Example is for making an understanding of this invention easy, and is not for limiting and interpreting this invention. The present invention can be changed and improved without departing from the gist thereof, and the present invention includes equivalents thereof.

  For example, in the process S100 for generating the standard data DB 31 in FIG. 5, the calculation unit 100 calculates the similarity of pairs having the same attribute, and acquires information on the point set 300 (processes S200 to S202). However, the set 300 is the same as the point set 350 (FIG. 11) obtained in the process S110 for generating the learned model 32 of FIG. For this reason, when the point set 350 is stored in the storage device 20, for example, the dividing unit 101 may divide the point set 350 into a plurality of clusters. In this case, the process performed by the calculation unit 100 can be omitted.

  In the present embodiment, m points closest to the center point of the cluster are selected. However, for example, a point close to the center of gravity point of the cluster may be selected.

  The data handled in the present embodiment is text data such as “Alice” and “Hokkaido”, but is not limited thereto, and may be image data, for example.

  In this embodiment, the dividing unit 101 divides the set of points into seven (n = 7) clusters, and the selecting unit 102 assigns two (m = 2) points from each cluster. Although it was decided to select, it is not restricted to this. For example, if the number of clusters (n) and the number of points (m) to be selected from each cluster are increased, data representing attributes can be selected with higher accuracy.

  The first column of the training data DB 30 of the present embodiment is, for example, data related to “person”, but is not limited thereto. For example, “facility”, “organization”, “thing”, and the like may be used. In addition, regarding “age”, for example, if the information in the first column is “facility”, information such as “aged age” may be used. Thus, the content of the training data DB 30 of this embodiment is an example, and any data may be used as long as it constitutes the database.

In addition, a non-transitory computer readable medium with non-transitory computer readable media stored therein.
It is also possible to supply a program to a computer using an executable program thereon). Examples of non-transitory computer readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), CD-ROMs (Read Only Memory), and the like.

10 Labeling system 20, 52, 62 Storage device 21 Data processing device 22 Labeling device 30 Training data DB
31 Standard data DB
32 learned model 50, 60 arithmetic device 51, 61 memory 53, 63 input device 54, 64 display device 55, 65 communication device 70, 71, 75 program 100, 110, 150 calculation unit 101 division unit 102, 103 selection unit 104 Update unit 111 Identification information giving unit 112,152 Judgment unit 113 Training unit 151 Output unit 153 Label giving unit 310-316 Cluster 300, 350, 351

Claims (7)

A division process of dividing a set of points in a vector space indicating the similarity of two data having the same attribute selected from the first plurality of data having any of the plurality of attributes into a plurality of clusters;
A first selection process for selecting at least one point from each of the plurality of clusters;
When the same data as any of the second plurality of data that is the basis of the plurality of points selected from each of the plurality of clusters is included in the second plurality of data, the second data that selects the same data Selection process,
An arithmetic unit characterized by executing The arithmetic device according to claim 1,
The first selection process is a process of selecting a point closest to a center point of each of the plurality of clusters from each of the plurality of clusters;
An arithmetic unit characterized by the above. The arithmetic device according to claim 1,
The second selection process is a process of selecting the same data when the second plurality of data includes a predetermined number of two or more of the same data;
An arithmetic unit characterized by the above. The arithmetic device according to any one of claims 1 to 3,
The second selection processing selects, as the fourth plurality of data, the third plurality of data having an attribute different from the attribute of the same data among the plurality of first data and the same data. Processing,
An arithmetic unit characterized by the above. The arithmetic device according to claim 4,
The arithmetic device executes an update process for updating information in the storage device so that the selected fourth plurality of data is stored in the storage device corresponding to each attribute,
An arithmetic unit characterized by the above. Dividing a set of points in a vector space indicating the similarity of two data having the same attribute selected from the first plurality of data having any of the plurality of attributes into a plurality of clusters;
Selecting at least one point from each of the plurality of clusters;
When the same data as any of the second plurality of data that is the basis of the plurality of points selected from each of the plurality of clusters is included in the second plurality of data, the same data is selected.
A data processing method. A data processing system including a first arithmetic device and a second arithmetic device,
The first arithmetic unit includes:
A division process for dividing a first set of points in a vector space indicating similarity between two data having the same attribute selected from the first plurality of data having any of the plurality of attributes into a plurality of clusters; ,
A first selection process for selecting at least one point from each of the plurality of clusters;
The same data as any of the second plurality of data that is the basis of the first plurality of points selected from each of the plurality of clusters is different from the attribute of the same data among the first plurality of data A second selection process of selecting the third plurality of data having attributes as the fourth plurality of data;
Run
The second arithmetic unit is
A calculation process for calculating a second plurality of points in the vector space indicating similarity between the input data not having attribute information and each of the fourth plurality of data;
Each point included in the second set of points and the first set of points in the vector space showing the similarity of two data having different attributes selected from the first plurality of data And a learned model that outputs identification information indicating whether the points in the vector space are points based on two data having the same attribute or points based on two data having different attributes, obtained as learning data An output process for outputting the identification information for each of the second plurality of points;
Executing a determination process for determining an attribute of the input data based on the identification information of each of the second plurality of points;
A data processing system.

Images