JPWO2019181594A1 - Parameter setting units, arithmetic units, their methods, programs, and recording media - Google Patents

Abstract

Efficiently performs arithmetic processing on a text file containing one or more records, each record containing one or more cells of arbitrary length, and each cell containing an arbitrary number of characters. Parameter setting apparatus performs maximum value S _csv and minimum value s _csv size of one record of a string attribute information as an _input, the maximum value S _enc total size of the encoding _information, the specific operation on the encoding information obtained set the total size _{S ref} of the maximum value _{S ss,} and reference information of the total size of the calculated value is to obtain a function value of _{C / (S csv + S enc} + S ref) as a record number to be encoded and arithmetic processing units , The function value _{of f 0} / I · r · _Scsv is obtained as the number of parallels in the arithmetic processing. However, C is a function value of the cache memory size, M is the main memory size, _{f 0} is _{s csv · M / (s csv} + S enc + max (S ref, S ss)).

Description

The present invention relates to an arithmetic technique for a character string in a text file.

The format of a text file is known that contains one or more records, each record contains one or more cells of arbitrary length (sometimes called a "field"), and each cell contains any number of characters. (See, for example, Non-Patent Document 1 and the like). When a specific arithmetic process (see, for example, Non-Patent Documents 2, 3 and the like) is processed in parallel for the value described in each cell of such a text file, the number of records and the number of parallel processes handled in one unit process. Need to be identified. At this time, in order to efficiently perform the arithmetic processing, not only the main memory size and the cache memory size of the arithmetic unit performing the arithmetic processing, but also the position and length of each record and each cell of the input text file are determined. Must be considered.

Y. Shafranovich, “RFC4180: Common Format and MIME Type for Comma-Separated Values (CSV) Files Status of This Memo,” [online], October, 2005, SolidMatrix Technologies, Inc., [Search January 6, 2018 ], Internet <http://www.ietf.org/rfc/rfc4180.txt> Dai Igarashi, Koji Chida, Hiroki Hamada, Katsumi Takahashi, "Secure Database Operations Using An Improved 3-party Veriable Secure Function Evaluation," InSCIS2011,201. A. Shamir, "How to Share a Secret", Communications of the ACM, November 1979, Volume 22, Number 11, pp.612-613.

However, the length of each cell of such a text file is arbitrary, and the text file often does not contain information indicating the position and length of each cell. Therefore, in order to specify the position and length of each cell, the input text file character strings must be read in order from the beginning. Therefore, it is not easy to specify the number of records handled in one unit processing and the number of parallels in the arithmetic processing in consideration of the position and length of each record and each cell in the text file, and perform efficient arithmetic processing. ..

The present invention has been made in view of these respects, and for a text file containing one or more records, each record containing one or more cells of arbitrary length, and each cell containing any number of characters. The purpose is to perform arithmetic processing efficiently.

In order to solve the above problems, a parameter setting device for arithmetic processing on a character string of a text file is provided. However, the text file contains W records, each of the records contains G cells of arbitrary length, and each of the cells contains any number of characters. W and G are integers of 1 or more, and G cells correspond to the attribute information. C is the cache memory size and M is the main memory size. The parameter setting device has a maximum size setting unit, a minimum size setting unit, an encoding size setting unit, a calculation size setting unit, a reference size setting unit, a processing unit calculation unit, and a parallel number calculation unit. _{The maximum size setting unit sets the maximum value Scsv} of the size of the character string for one record of the text file by inputting the attribute information. _{The minimum size setting unit sets the minimum value s csv} of the size of the character string for one record by inputting the attribute information. _{The encoding size setting unit sets the maximum value Sensor} of the total size of the encoding information obtained by encoding the character string for one record based on a predetermined finite set. _{The calculation size setting unit sets the maximum value S ss} of the total size of the calculation values obtained by performing a specific calculation on the encoding information for one record. However, the encoding and the operation are processes executed for each processing unit character string which is a character string for r records of a text file. _{The reference size setting unit sets the total size Sref} of the reference information representing the position and length of each cell for one record. The processing unit calculation unit obtains a function value _{of C / (S csv} + _Senc + _Sref ) as the number of records r. The parallel number calculation unit obtains a function value of f ₀ / Ir r _{Scsv as the parallel number n p in the arithmetic processing.} However, the maximum value encoding and the number of repetitions of calculation I are performed for each processing unit character _{string, S ref} ≧ _{S ss} when _{max (S ref, S ss)} = a _{S ref, S ref <S} when _{ss max (S ref, S ss} ) a _{= S ss,} is a function value of _{f 0} is _{s csv · M / (s csv} + S enc + max (S ref, S ss)).

As described above, it is possible to efficiently perform arithmetic processing on a text file containing one or more records, each record containing one or more cells having an arbitrary length, and each cell containing an arbitrary number of characters.

FIG. 1 is a block diagram illustrating an arithmetic system of an embodiment. FIG. 2 is a block diagram illustrating the functional configuration of the parameter setting device of the embodiment. FIG. 3 is a block diagram illustrating the functional configuration of the server device of the embodiment. FIG. 4 is a block diagram illustrating the functional configuration of the processing unit of the embodiment. FIG. 5 is a flow chart for exemplifying the parameter setting process of the embodiment. FIG. 6 is a flow chart for exemplifying the arithmetic processing of the embodiment. FIG. 7 is a flow chart for exemplifying the processing of the thread i of the embodiment. FIG. 8 is a conceptual diagram for exemplifying the processing of each thread of the embodiment. FIG. 9 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 10 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 11 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 12 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 13 is a flow chart for exemplifying the processing of the thread i of the embodiment. FIG. 14 is a conceptual diagram for exemplifying the processing of each thread of the embodiment. FIG. 15 is a conceptual diagram for exemplifying the processing of the thread i of the embodiment. FIG. 16 is a conceptual diagram for exemplifying the processing of the thread i of the embodiment. FIG. 17 is a conceptual diagram for exemplifying the processing of the thread i of the embodiment. FIG. 18 is a conceptual diagram for exemplifying the processing of the thread i of the embodiment.

Hereinafter, embodiments of the present invention will be described.
[Overview]
First, an outline will be described.
<Text file>
In each embodiment, arithmetic processing is performed on the character string of the text file. This text file contains W records, each of which contains G cells of arbitrary length, and each of the cells contains any number of characters. However, the length of each cell has an upper limit according to the attribute of each cell. W and G are integers greater than or equal to 1. For example, at least one of W and G is an integer greater than or equal to 2. W may be an integer of 2 or more, G may be an integer of 2 or more, and both W and G may be an integer of 2 or more. When W is an integer of 2 or more, there is information for specifying the record delimiter between the records adjacent to each other. For example, there is a line break between records adjacent to each other, and a plurality of records are separated from each other by a line break. Further, when G is an integer of 2 or more, there is information for specifying the delimiter between cells between cells adjacent to each other. For example, there is a delimiter or newline between cells adjacent to each other, and multiple cells are separated from each other by the delimiter or newline. An example of a delimiter is the comma ",". As another example, there may be tabs or line breaks between cells adjacent to each other, and half-width spaces or line breaks may exist between cells adjacent to each other. When W is an integer of 2 or more, the number G of cells included in each record is the same as each other. The G cells of each record correspond to the attribute information (also called "schema"). The attribute information represents what kind of attribute information each cell has, and is information for specifying or estimating at least the maximum value and the minimum value of the size (data amount) of the character string represented by each cell. Includes. For example, attribute information contains information indicating what kind of finite set of elements a cell represents. For example, the attribute information may represent "a cell represents a remainder (mod p) modulo p (p is a positive integer)" or "a predetermined number of cells (for example, 10)". it may represent "things is an element string represented by the finite field (for example, extension field GF (2 ^8)), the integer" cell is a predetermined integer (e.g., signed 32-bit integer ) Is a character string. ” Each of the G attribute information may have a one-to-one correspondence with each of the G cells of each record (that is, one attribute information may represent the attribute of one cell). One attribute information may correspond to a plurality of cells (for example, G cells) of each record (that is, one attribute information may represent an attribute of a plurality of cells). In the former case, the attributes of a plurality of cells belonging to one record may be different from each other or may be the same as each other. When W is an integer of 2 or more, the "set of G attributes" corresponding to G cells of all records are the same as each other. That is, the attribute att (g) of the gth cell (where g = 1, ..., G) possessed by all the records is the same as each other. In addition, the attribute information may represent the type of information represented by the cell. Further, the attribute information may or may not be included in the text file (for example, the header of the text file may include the attribute information). Examples of text files are CSV (Comma-Separated Values) files, TSV (tab-separated values) files, SSVs (space-separated values), and the like. These are collectively referred to as CSV (character-separated values) files and DSV (delimiter-separated values) files.

<Parameter setting device>
The parameter setting device sets and outputs parameters for "arithmetic processing" for the character string of the text file. This "arithmetic processing" may be anything. Examples of "arithmetic processing" include secret sharing processing, secret calculation processing (see, for example, Non-Patent Documents 1 and 2), encryption processing, signature generation processing, and the like. The parameters set by the parameter setting device are the number of records handled in one unit processing and the number of parallels in the arithmetic processing. Preferably, the parameter setting device also sets the file buffer size of the data that is collectively read from the text file. In the following, the cache memory size (cache memory storage capacity) of the cache memory of the arithmetic unit that performs arithmetic processing on the character string of the text file is expressed as C, and the main memory size (main memory storage capacity) of the main memory is M. Notated as.

The parameter setting device has a maximum size setting unit, a minimum size setting unit, an encoding size setting unit, a calculation size setting unit, a reference size setting unit, a processing unit calculation unit, and a parallel number calculation unit. When the file buffer size is also set, the parameter setting device also has a buffer size calculation unit.

The maximum size setting unit sets and outputs _{the maximum value Scsv} of the size of the character string for one record of the text file by inputting the attribute information. The maximum value S _csv is a record size in which the size of the character string of each record of the text file is overestimated. _{That is, the maximum value Scsv} is the sum of the maximum value (or its estimated value) of the size of each cell represented by the attribute information for one record. As described above, the attribute information includes information for specifying or estimating the maximum value of the size of the character string represented by each cell, and the maximum size setting unit sets the maximum value S _csv using this information. For example, if the attribute information indicates that the cell is a string representing a signed 32-bit integer, the maximum size of the cell is 11 bytes (1 byte for sign + 10 for 11 digit integer). Byte).

The minimum size setting unit sets and outputs _{the minimum value s csv} of the size of the character string for one record by inputting the attribute information. The minimum value s _csv is the record size in which the size of the character string of each record in the text file is underestimated. _{That is, the minimum value s csv} is the sum of the minimum value (or its estimated value) of the size of each cell represented by the attribute information for one record. As described above, the attribute information includes information for specifying or estimating the minimum value of the size of the character string represented by each cell, and the minimum size setting unit sets the minimum value s _csv using this information. For example, when the attribute information indicates "the cell is a character string representing a signed 32-bit integer", the minimum value of the size of the cell is 1 byte.

The encoding size setting unit sets and outputs _{the maximum value Sensor} of the total size of the encoding information obtained by encoding (converting) the character string for one record based on a predetermined finite set. The maximum value _Sec is a record size in which the total size of the encoding information for one record is overestimated. Examples of a "predetermined finite set" to which the encoding information belongs are a finite set represented by a remainder (mod p) modulo p, a finite set of values represented by a predetermined bit, and a predetermined integer with a predetermined number of bits. A finite set represented by an integer of type. This "predetermined finite set" is predetermined according to the content of the "operation" described below. The maximum value _Sec is specified from, for example, a predetermined finite set to which the encoding information belongs and attribute information. For example, the attribute information represents a "that represents a remainder cells modulo ^{2 61} (mod ^{2 61)",} in the remainder of the string of this cell modulo ^{2 61} (mod ^{2 61)} When encoded in the expressed encoding information, the maximum size of the encoding information corresponding to the cell is 8 bytes. The predetermined finite set to which the encoding information belongs is, for example, predetermined.

The calculation size setting unit sets and outputs _{the maximum value S ss} of the total size of the calculation values obtained by performing a specific "calculation" on the encoding information for one record. The maximum value S _ss is a record size in which the total size of the calculated values for one record is overestimated. Examples of this "calculation" are secret sharing, secret calculation, encryption, signature generation, and the like. The "operation" may be performed with each cell as a show operator, or may be performed with a plurality of cells as a show operator. The maximum value S _ss is specified from, for example, a predetermined finite set to which the encoding information belongs, the content of the "operation", and the attribute information. For example, the attribute information represents a "that represents a remainder cells modulo ^{2 61} (mod ^{2 61)",} in the remainder of the string of this cell modulo ^{2 61} (mod ^{2 61)} It is encoded in the encoded information to be expressed, and the "operation" secretly shares the value of each cell to N (however, N is a positive integer) party by the Shamir secret sharing method (for example, Non-Patent Document 3). If, the maximum size of the calculated value corresponding to the cell is 8 Nbytes.

The reference size setting unit sets and outputs _{the total size Sref} of the reference information indicating the position and length of each cell for one record in the text file. The "cell position" may be, for example, the position of the first character of the cell, the position of the last character of the cell, or the position of a character in another cell. The "information representing the cell position" may be, for example, the number of characters from the first character of the character string of the text file to the character of the "cell position", or may be a function value of this number of characters. The "information representing the cell length" may be, for example, the number of characters in the cell or a function value of this number of characters. Reference information is specified from, for example, attribute information. This is because the number of cells belonging to one record can be specified by the attribute information. The data size required to represent the position and length of each cell is determined by the format in which it is represented. For example, when the position and length of each cell are represented by an unsigned 64-bit integer, the reference information of each cell is 16 bytes.

The above-mentioned "encoding" and "operation" are executed for each processing unit character string which is a character string for r records of a text file. Processing The processing executed for each unit character string is called "unit processing". The processing unit calculating section represents the record number r to be processed in unit process once to obtain the _"C / function value of _{(S csv + S enc + S} ref) " _{(C / (S} csv ₊ S as a record number r _enc + S Obtain the function value of _{ref) and output it.} The cache memory size C may be a predetermined one or may be an input one. The maximum value S _csv is obtained by the maximum size setting unit, the maximum value _Senc is obtained by the encoding size setting unit, and the total size S _ref is obtained by the reference size setting unit. .. The "function value of α" may be α itself or another value corresponding to α. Examples of "function values of α" are the smallest integer greater than or equal to α, the largest integer less than or equal to α, and the integer closest to α. For example, r = C / (S _csv + S _enc + S _ref ), r = ROUNDUP (C / (S _csv + S _enc + S _ref )), or r = ROUNDDOWN It may be (C / (S _csv + S _enc + S _ref )) or r = ROUND (C / (S _csv + S _enc + S _ref )). However, ROUNDUP (α) is a round-up function that rounds up α to an integer value, ROUNDDOWN (α) is a round-down function that rounds down α to an integer value, and ROUND (α) is a rounded integer that rounds α to the function closest to α. Is. Here, in S _csv + _Senc + _Sref , until the processing unit reads the character string for one record from the text file, encodes it into the encoded information while referring to the reference information, and performs "calculation" such as secret sharing. Represents the memory size required for the processing (hereinafter referred to as "a series of processing for one record"). If this memory size is less than or equal to the cache memory size, a series of processes for one record can be executed at high speed without reading data from the main memory on the way. C / ( _Scsv + _Senc + _{Sref ) indicates how many times the memory size (Scsv} + _Senc + _Sref ) required for "a series of processing for one record" can be secured in the cache memory. By using the character string of the number of records r corresponding to C / (S _csv + _Senc + _Sref ) as the processing unit character string, the number of accesses to the main memory when processing r records is reduced and the speed is increased. Can perform operations.

The parallel number calculation unit obtains the "function value of _{f 0} / I · r · S _csv " representing the _{parallel number n p} in the arithmetic processing (the function value of f ₀ / I · r · S _{csv as the parallel number n p} ). Get) output. For example, it may be a _{n p = f 0 / I ·} r · S csv, may be a _{n p = ROUNDUP (f 0 /} I · r · S csv), n p = ROUNDDOWN (f 0 / I · r · _Scsv ) or n _p = ROUND (f ₀ / I · r · _Scsv ). f ₀ is a function value of _{s csv} · M / (s _csv + _Senc + max ( _Sref , _Sss)). For example, f ₀ = s _csv · M / (s _csv + S _enc + max (S _ref , S _ss )) or f ₀ = ROUNDUP (s _csv · M / (s _csv + S _enc +) max (S _ref , S _ss ))) or f ₀ = ROUNDDOWN (s _csv · M / (s _csv + S _enc + max (S _ref , S _ss )))) , F ₀ = ROUND (s _csv · M / (s _csv + S _enc + max (S _ref , S _ss ))). _However, when the _{S ref ≧ S ss max (S} ref, S ss) a _{= S ref, S ref <When S ss max (S ref, S} ss) = a _{S ss.} I is the maximum number of repetitions of "encoding" and "operation" executed for each processing unit character string. For example, I is the number of repetitions of "encoding" and "operation" executed for each processing unit character string. The main memory size M may be a predetermined one or may be an input one. The minimum value s _csv is set by the minimum size setting unit, the maximum value _Sec is set by the encoding size setting unit, and the maximum value S _ss is set by the calculation size setting unit. , The total size _Sref is set by the reference size setting unit, and the maximum value I of the number of repetitions is predetermined. Preferably, the maximum value I of the number of repetitions is the total processing amount of preprocessing (of the number of operations) with respect to the total processing amount for performing "encoding" and "calculation" of the processing unit character string which is a character string for r records. The ratio of total) is set to be less than or equal to the predetermined value. r may be obtained from the processing unit calculation unit, or may be obtained from _{C, Scsv} , _Senc , and _Sref. That _is, if the function value of _{f 0 / I · r · S} csv is obtained, not necessarily _{f 0 / I · r · S} r obtained by the processing unit calculating unit to generate the _csv function value is used You may. Here, X = (s _csv + _Senc + max ( _Sref , S _ss )) / s _csv is a character string for one record read from a text file, in which the memory size required for a series of processing for one record is Indicates the maximum number of times the memory size of. Therefore, S _csv / X represents the maximum value of the memory size required for a series of processing for one record, and I / r / S _csv / X is the memory required for repeating the series of processing for r records I times. Represents the maximum size. f ₀ is _{s csv · M / (s csv} + S enc + max (S ref, S ss)) for = is a function value of _{M / X, f 0 / I} · r · S csv , the main memory size M is " It indicates how many times the memory size required for the process of repeating a series of processes for r records (a process of performing a series of processes for one record for r records) I times. Therefore, the buffer overflow of the main memory can be suppressed by setting the value corresponding to _{f 0} / I · r · _{Scsv to the number of parallel np.} Since reference information is required for encoding, it is necessary to secure the _{Sref area in the main memory.} On the other hand, reference information is not required for "operation" such as secret sharing after encoding, but it is necessary to secure an _{S ss area for storing the obtained operation value in the main memory.} That is, _{both S ref} and S _ss regions are not required at the same time. It is sufficient if the max (S _ref , S _ss ) area can be secured in the main memory.

_{The buffer size calculation unit obtains a "function value of f 0} / n _p " representing the file buffer size f of the data collectively read from the character string of the text file during arithmetic processing _{(f 0} as the file buffer size f). (Obtain the function value of _{/ np) and output.} For example, f = f ₀ / n _p , f = ROUNDUP (f ₀ / n _p ), f = ROUNDDOWN (f ₀ / n _p ), or It may be f = ROUND (f ₀ / n _p). f ₀ is as described above, and n _p may be obtained by the parallel number calculation unit or may be obtained from _{f 0} , I, r, _Scsv. That is, if the function value of _{f 0} / n _{p can be obtained, the n p} obtained by the parallel number calculation unit does not necessarily have to be used for _{the generation of the function value of f 0} / n _p. Here, if n _p = f ₀ / I · r · S _csv , then f ₀ / n _p = I · r · S _csv . This corresponds to the file buffer size f of the character string read from the text file in order to repeat the processing for r records I times. The larger the file buffer size f, the closer to the sequential accelerator and the faster the speed, but the larger the storage capacity of the main memory required. The file buffer size f obtained as described above realizes high-speed processing within the constraints of the predetermined main memory size M.

<Arithmetic logic unit (general version)>
The arithmetic unit performs arithmetic processing on the input text file character string. The arithmetic unit uses the number of records r and the number of parallels _np obtained by the parameter setting apparatus in this arithmetic processing. When the parameter setting device further obtains the file buffer size f, the arithmetic unit uses the file buffer size f further obtained by the parameter setting device in this arithmetic processing. If the file buffer size f cannot be obtained by the parameter setting device, the arithmetic unit may use a predetermined value as the file buffer size f, or may use another file buffer size f corresponding to the attribute information. Good.

Text files may contain characters that are not subject to "calculations" such as secret sharing. For example, a comma in a CSV file is a character that represents a cell delimiter and is not the target of "calculation". In a CSV file, a cell character string may be enclosed in double quotation marks "" "" "", but double quotation marks are also not subject to "calculation". Characters that represent line breaks (for example, \ n) are also not subject to "calculation" (these double quotation marks "" "" "are not included in the cell). Characters that are not the target of such "calculations" are called "special characters". There is also a format in which an escape character is added before a special character in a cell to allow the special character to be treated as the character to be "operated". In such a case, it may not be possible to determine whether or not each character in the cell represents the target of "calculation", and it may not be possible to determine the delimiter between cells. For example, when the double quotation mark "" "is used as the target of the" operation ", there is a format in which" "" is further added as an escape character. For example, when the value "123" 456 "of a cell is the target of" calculation ", the cell may be described as" 123 "" 456 ". In such a case, the cell "123" and "456" must be read in order from the beginning to determine the division between cells. For example, when only the latter half "456" is read, it cannot be determined whether this is one cell representing "456" or a part of the cell representing a value including "456". In addition, when the character "\ n" representing a line feed is used as the target of "calculation", there is also a format in which "\" is further added as an escape character. For example, when the value "123 \ n456" of a cell is the target of "calculation", the cell may be described as "123 \ \ n456". In such a case, the cell "123 \\ n456" must be read in order from the beginning to determine the division between cells. In such a case, the process of specifying the position and length of each cell of the text file cannot be performed in parallel, and this process must be performed in order from the beginning of the text file.

An arithmetic unit capable of handling a text file of such a format has a main memory, a cache memory, and a plurality of processing units. Each processing unit has a reading unit, a file reading unlocking unit, a parsing unit, a buffer boundary unlocking unit, an encoding unit, an arithmetic unit, and a parallelism unlocking unit. These plurality of processing units are assigned to the processing of any thread. The processing unit that processes thread i performs the following processing. Note that i represents each thread, i ∈ {0, ..., T-1}, T is a positive integer representing the number of threads corresponding to the size TS of the character string of the text file, and 1 ≦ n _p ≦. It is T. _{For example, for the size TS i} of the character string read from the text file in each thread i, TS = TS ₀ + ... + TS _T-1 or TS ≤ TS ₀ + ... + TS _T-1 is satisfied, or f · T ≥ TS. Fulfill. Further, it is assumed that the file read lock and buffer boundary lock of thread 0 and _{the concurrency lock of threads 0, ..., N p -1 are released in the initial state.}

Reading unit, after the file read lock and parallelism locking thread i is released, stored in the main memory by reading the string S _i can be stored from a string in a region of the file buffer size f of the text file. When i = 0, the character string S ₀ is a "character string of the amount of data that can be stored in the area of the file buffer size f" starting from the first character of the text file. When i ≧ 1, the character string S _i is the “amount of data that can be stored in the area of the file buffer size f” starting from the character immediately after the _{last character of the character string S i-1} read by the thread i-1. Character string ". The "character string that can be stored in the area of the file buffer size f" may be, for example, the longest character string that can be stored in the area of the file buffer size f, or the size obtained by subtracting a constant from the file buffer size f. It may be the longest character string that can be stored in the area.

After the character string S _i is stored in the main memory, the file read unlock unit releases the file read lock of thread i + 1. As a result, it is possible to prevent the access to the main memory by a plurality of threads from competing with each other. However, the thread corresponding to i + 1> T does not exist, and the file read lock of the thread that does not exist is not released.

After the buffer boundary locking thread i is released, stored in the main memory by calculating a reference information representative of the location and length of each cell parser is included in the string S _i. For example, parser identifies each cell string S _i, stored in the main memory to calculate the reference information of each cell identified. For example, the parsing unit identifies each cell based on the information located at the cell boundary (for example, a delimiter or a line feed), calculates the reference information of each identified cell, and stores it in the main memory. When i = 0, the _{terminal character included in the character string S 0} may be the end of the cell or not the end of the cell. If the terminating character contained in the character string S ₀ is not the terminating character of the cell, thread 0 cannot identify the cell containing the terminating character and cannot calculate the reference information. For i ≧ 1, the start of the character string S _i is a cell may not be the beginning of when the cell is a beginning, end in the string S _i characters if the cell is the end of the cell It may not be the end. When starting character of the string S _i is not a beginning of a cell, can not identify the cell containing only string S _i of the start of the string S _i characters. In this case, the parsing part uses the characters of the character string S _i-1 that are not included in the cell specified by the thread i-1, and the character string S _i, and the cell containing the character at the beginning of the character _{string S i} To identify. If the terminating character contained in the character string S _i is not the terminating character of the cell, the thread i cannot identify the cell containing the terminating character and cannot calculate the reference information. The terminal character included in _{the character string ST-1} at the end of the text data is the end of the cell. By using the reference information corresponding to the specified cell, the record to which the cell belongs and the attribute corresponding to the cell (for example, information indicating the number of the attribute from the beginning of the record) can be specified. When the area for storing the reference information is insufficient in the main memory, the parsing unit collectively secures a buffer area for storing the reference information for r records in the main memory. Predetermined processing (overhead) is required to secure the buffer area. By allocating a buffer area for each r record corresponding to unit processing instead of allocating a buffer area for each record, it is possible to process variable-length records while suppressing overhead.

After the reference information representing the position and length of each cell included in the character string S _i is calculated, the buffer boundary unlocking unit releases the buffer boundary lock of thread i + 1. The thread corresponding to i + 1> T does not exist, and the buffer boundary lock of the nonexistent thread is not released.

After releasing the buffer boundary lock of thread i + 1, the encoding unit is a processing unit character which is a character string for r records to be processed included in the _{combined character string CS i from the text data based on the information specified by the reference information.} column PS _i, select _j, selected processing unit character string PS _i, encoding information E _{i, j} is a predetermined original finite _set, the process for encoding the _j, performed using a cache memory. The start end of the processing unit character string PS _{i, j} is the start end of any record, and the end end of the processing unit character string PS _{i, j is the end end of any record.} The concatenation character string CS ₀ when i = 0 is S ₀ , and the concatenation character string CS _i when i ≧ 1 is the concatenation character string S _i immediately after the concatenation character string CS _i-1 . J is a positive integer, and j = 0, ..., J-1. For example, if the number of characters in the string S _i is equal to or greater than the number of characters r record content, the encoding unit, the string S _i from the processing unit character string PS _i, choose a _j, or string S _i and the string The _{processing unit character strings PS i and j} are selected from the character string in which _{S i-1 is combined.} When the number of characters in the character string S _i is less than the number of characters for r records, the encoding unit selects the _{processing unit character strings PS i and j} from the character string obtained by combining the _{character strings S i} to the character string S _i'. However, 0 ≦ i'≦ i-1. When i = 0, the encoding unit selects the processing unit character strings PS ₀ , 0, ..., PS _{0, J-1 from the beginning of the character string S 0.} When i ≧ 1, the encoding unit _{selects the processing unit character string PS i, 0} , ..., PS _{i, J-1} starting from the character not selected as the processing unit character string in the _{character string S i-1.} To do. For J ≧ _{2, PS i, j} followed by _{PS i,} immediately after the _j-1.

The encoding unit encodes each processing unit character string PS _{i, j.} In text data, data is arranged in a code-oriented manner (data is arranged in the order of record 1, record 2, ..., Record W), and all records correspond to the same "attribute set". In general, the processing speed is faster when the same type of data is continuously processed than when different types of data are continuously processed. Therefore, it is desirable that the encoding unit continuously encodes r cells of r records corresponding to the same attribute information in the processing unit character strings PS _{i and j.} Encoding section processing unit character string based on the information specified by the reference information PS _i, select _j, encoding selected processing unit character string PS _i, a _j encoding information E _i, the _j. High-speed processing is possible by storing the reference information for r records, the processing unit character strings PS _{i, j} , and the encoding information E _{i, j} , which are required in the process of this processing, in the cache memory and performing the calculation. As described above, such processing is possible by using the number of records r corresponding to _{C / (Scsv} + _Senc + _Sref).

The calculation unit uses the cache memory to perform a process of performing a specific "calculation" on _{the encoding information E i and j to obtain the calculation values SS i and j} and storing them in the main memory. It is desirable that the calculation unit continuously performs "calculation" corresponding to r cells for r records corresponding to the same attribute information. Even in this processing process, high-speed processing is possible by storing the reference information for r records, the processing unit character strings PS _{i, j} , and the encoding information E _{i, j} in the cache memory and performing the calculation.

After the calculated values SS _{i and j} are obtained, the parallelism unlocking unit releases the parallelism lock of the thread i + _np . However, the thread corresponding to i + n _p > T does not exist, and the parallelism lock of the nonexistent thread is not released. After that, the processing unit that has been processing the thread i is released, and the processing unit can perform the processing of another thread.

<Arithmetic logic unit (high-speed version)>
When each cell of the text file contains only characters that can determine whether or not it represents the target of "calculation" by itself, the process of specifying the position and length of each cell of the text file can be performed in parallel. , Even faster calculation is possible. For example, in the case of a text file that does not use escape characters, such parallel processing is possible. An arithmetic unit that can handle such a text file has a main memory, a cache memory, and a plurality of processing units. Each processing unit has a reading unit, a file reading unlocking unit, a parsing unit, a cell specifying unit, a buffer boundary unlocking unit, an encoding unit, an arithmetic unit, and a parallelism unlocking unit. These plurality of processing units are assigned to the processing of any thread. The processing unit that processes thread i performs the following processing. It is assumed that the file read lock of thread 0, the buffer boundary lock, and _{the concurrency lock of threads 0, ..., N p} -1 are released in the initial state.

Reading unit, after the file read lock and parallelism locking thread i is released, stored in the main memory by reading the string S _i can be stored from a string in a region of the file buffer size f of the text file. This detail is the same as the arithmetic unit (general version).

After the character string S _i is stored in the main memory, the file read unlock unit releases the file read lock of thread i + 1. This detail is the same as the arithmetic unit (general version).

Parser stores to calculate the reference information representative of the location and length of each cell in a string S _i in a main memory. The parsing unit can start this process before the buffer boundary lock of thread i is released. That is, the parser, in i ≧ 1, before the calculation of the reference information indicating the position and length of each cell in the string S _i-1 is completed, the position of each cell in the string S _i and You can start calculating the reference information that represents the length. For example, parser identifies the cell of the string S _i, stored in the main memory to calculate the reference information of each cell identified. For example, the parsing unit identifies a cell based on information located at the boundary of the cell (for example, a delimiter or a line feed), calculates the reference information of each identified cell, and stores it in the main memory. When i = 0, the _{terminal character included in the character string S 0} may be the end of the cell or not the end of the cell. If the terminal character included in the character string S ₀ is not the terminal of the cell, the parsing unit cannot identify the cell containing the terminal character, and the reference information thereof cannot be calculated. For i ≧ 1, the start of the character string S _i is a cell may not be the beginning of when the cell is a beginning, end in the string S _i characters if the cell is the end of the cell It may not be the end. When starting character of the string S _i is not a beginning of the cell, parsing unit can not identify the cell that contains the start character string S _i, not even be calculated the reference information. If the character terminating in a character string S _i is not the end of the cell, parsing unit can not identify the cell containing the end character, not even be calculated the reference information. The terminal character _{included in the terminal character string ST-1} is the terminal of the cell. When the area for storing the reference information is insufficient in the main memory, the parsing unit collectively secures a buffer area for storing the reference information for r records in the main memory. This makes it possible to process variable-length records while suppressing overhead.

When i ≧ 1, the character string S _i is a character string immediately following the character string _{S i-1.} When i ≧ 1, the cell identification part is included in the character string S _{i-1 by using the reference information, the character string S i-1} and the character string S _i -1 after the buffer boundary lock of the thread i is released. stored in the main memory to obtain information a _i corresponding to the position of the cell immediately following the last cell. Information A _i is, for example, _{information representing a record to which a cell immediately following the last cell included in the character string S i-1} belongs and information representing an attribute corresponding to the cell (for example, the number from the beginning of the record). Information indicating whether or not it corresponds to the attribute of), or information indicating the position and length of the cell immediately following the last cell included _{in the character string Si-1.} When the end of the character string S _{i-1 is} the end of the cell, the _{first cell of the character string S i} becomes "the cell immediately following the last cell included in the character string S _i-1". In this case, _{the storage of the information Ai in} the main memory may be omitted. On the other hand, _{when the end of the character string S i-1} is not the end of the cell, the character string S _i uses the character string S _i-1 and the character string S _i, and "the last cell included in the _{character string S i-1".} obtain information a _i generates a cell "that immediately follows. As a result, information corresponding to the position of the cell whose perspective portion could not be specified can be obtained. The record to which each cell of the text file belongs and the attribute corresponding to the cell (for example, information indicating the number of the attribute from the beginning of the record) can be specified by the reference information and the information _Ai. When i = 0, the cell specific part does nothing.

The buffer boundary unlocking unit releases the buffer boundary lock of thread i + 1 after the _{information Ai is obtained.} This detail is the same as the arithmetic unit (general version). After that, the processing unit that has been processing the thread i is released, and the processing unit can perform the processing of another thread.

After that, the encoding unit selects the _{processing unit character strings PS i and j} , which are the character strings for the r records to be processed included in the combined character string CS _{i , based on the reference information and the information specified by the information A i.} , The process of encoding the processing unit character strings PS _{i, j into the encoding information E i, j} , which is the source of a predetermined finite set, is performed using the cache memory. This detail is the same as that of the arithmetic unit (general version) except _{that the information Ai} is used in addition to the reference information.

The calculation unit uses the cache memory to perform a process of performing a specific "calculation" on _{the encoding information E i and j to obtain the calculation values SS i and j} and storing them in the main memory. This detail is the same as the arithmetic unit (general version).

After the calculated values SS _{i and j} are obtained, the parallelism unlocking unit releases the parallelism lock of the thread i + _np . This detail is the same as the arithmetic unit (general version).

[First Embodiment]
The first embodiment will be described with reference to the drawings. In the first embodiment, the parameter setting device sets the number of records r, the number of parallels _np , and the file buffer size f handled in one unit processing, and the arithmetic unit (general version) can use escape characters in CSV (general version). An example of performing secret sharing (calculation) of a Comma-Separated Values) file (text file) will be described. In the following, the differences from the items described so far will be mainly described, and the items already explained may be omitted.

<Structure>
As illustrated in FIG. 1, the arithmetic system 1 of the present embodiment includes a parameter setting device 11, an arithmetic device 12, and N server devices 13-1 to 13-N. However, N is a positive integer of 2 or more. Information can be transmitted from the parameter setting device 11 to the arithmetic unit 12, and information can be transmitted from the arithmetic unit 12 to the server devices 13-1 to 13-N. Information may be transmitted via a network, may be performed by using other communication means, or may be transmitted via a portable recording medium.

As illustrated in FIG. 2, the parameter setting device 11 includes an input unit 111a, an output unit 111b, a storage unit 112, a control unit 113, a maximum size setting unit 114a, a minimum size setting unit 114b, an encoding size setting unit 114c, and a calculation size. It has a setting unit 114d, a reference size setting unit 114e, a processing unit calculation unit 114f, a parallel number calculation unit 114g, and a buffer size calculation unit 114h. The parameter setting device 11 executes each process under the control of the control unit 113. Each value obtained by the parameter setting device 11 is stored in the storage unit 112, and is read out from the storage unit 112 as needed and used for other processing.

As illustrated in FIG. 3, the arithmetic unit 12 includes an input unit 121a, an output unit 121b, an auxiliary storage unit 122, a main memory 123, a control unit 125, and a processing unit 126-1 to 126-Q. However, Q is an integer of 2 or more. The arithmetic unit 12 executes each process under the control of the control unit 125.

As illustrated in FIG. 4, the processing unit 126-q (where q = 1, ..., Q) includes a cache memory 1260-q, a reading unit 1261-q, a perspective unit 1262-q, and an encoding unit 1265-q. It has a calculation unit 1266-q, a file read unlock unit 1267-q, a buffer boundary unlock unit 1268-q, and a parallelism unlock unit 1269-q.

<Parameter setting process>
The parameter setting process of the parameter setting device 11 will be described with reference to FIG.
The attribute information of the text data to be processed is input to the input unit 111a of the parameter setting device 11 (FIG. 2) and stored in the storage unit 112. The attribute information may be read from the text data or may be given from other than the text data (step S111a).

Maximum size setting unit 114a sets the maximum value _{S csv} size of one record of a string of text file and outputs the attribute information read from the storage unit 112 as an input (step S114a).

_{The minimum size setting unit 114b sets and outputs the minimum value s csv} of the size of the character string for one record by inputting the attribute information read from the storage unit 112 (step S114b).

The encoding size setting unit 114c takes the attribute information read from the storage unit 112 as input, and uses the character string for one record as the source of the predetermined finite set based on the information representing the "predetermined finite set" to which the encoding information belongs. Set and output _{the maximum value Senc} of the total size of the encoding information obtained by encoding (conversion). The "predetermined finite set" to which the encoding information of the present embodiment belongs is a finite set to which secret sharing is performed, and is predetermined (step S114c).

The calculation size setting unit 114d receives the attribute information read from the storage unit 112 as input, and is obtained by secret sharing (calculation) of the encoding information for one record based on a predetermined finite set to which the encoding information belongs and the secret sharing method. Set and output _{the maximum value S ss} of the total size of the secret sharing value (calculated value). The secret sharing method of this embodiment is predetermined (step S114d).

The reference size setting unit 114e takes the attribute information read from the storage unit 112 as input, sets and outputs _{the total size Sref of the reference information representing the position and length of each cell for one record in the text file (} Step S114e).

The processing unit calculation unit 114f takes S _{csv, Senc,} and _Sref as inputs, and has _{a function value r of C / (S csv} + _Senc + _Sref ) (the number of records r processed in one unit processing, that is, the processing unit. The number of records r) included in the character string is obtained and output. The cache memory size C may be a predetermined one or may be an input one (step S114f).

The parallel number calculation unit 114g takes S _csv, s _csv , S _ref , _Senc , I and r as inputs, and sets the function value n _p (parallel number n _p in arithmetic processing) of _{f 0} / Ir r S _csv. Get and output. f ₀ is a function value of _{s csv} · M / (s _csv + _Senc + max ( _Sref , _Sss)). The main memory size M may be a predetermined one or may be an input one (step S114g).

The buffer size calculation unit 114h takes f ₀ and n _p as inputs, and _{obtains a function value f of f 0} / n _p (file buffer size f of data collectively read from a character string of a text file during arithmetic processing). Is output (step S114h).

The output unit 111b outputs r, n _p , and f obtained as described above (step S111b).

<Calculation processing>
The arithmetic processing of the arithmetic unit 12 will be described with reference to FIGS. 6 to 12.
As illustrated in FIG. 6, r, n _p , and f output from the parameter setting device 11 are input to the input unit 121a of the arithmetic unit 12 (FIG. 3) and stored in the auxiliary storage unit 122 (step S111aa). ). Further, the text data to be processed is input to the input unit 121a and stored in the auxiliary storage unit 122. Text data is illustrated in FIGS. 9 to 12. The text data illustrated in FIG. 9 is a CSV file in which each cell is enclosed in double quotation marks. When the double quotation mark """is used as the cell value,""" is added as an escape character before it. For example, "4selddks""k304kdkk400-03d" represents the value of the secret sharing target "4selddks" k304kdkk400-03d ". As illustrated in FIG. 10, this text file contains W records rec (1), ..., Rec (W), and each of the records rec (w) (where w = 1, ..., W) is arbitrary. It contains G long cell cells (w, g) (where g = 1, ..., G), and each cell cell (w, g) contains any number of characters. The cell cell (w, g) corresponds to the g-th attribute att (g) from the beginning of the record rec (w) (step S111ab).

After that, r, n _p , and f are read from the auxiliary storage unit 122 into the main memory 123, and the arithmetic processing of threads i = 0, ..., T-1 is executed. The arithmetic processing is started from the thread of i = 0. In the initial state, the file read lock and buffer boundary lock of thread 0 and _{the concurrency lock of threads 0, ..., N p} -1 are released. The control unit 125 allocates the unused processing unit 126-q of the processing units 126-1 to 126-Q to the thread i, and a plurality of threads execute the processing of each thread i in parallel as much as possible (step S126). ). Each secret sharing value obtained in this way is output from the output unit 121b, sent to each server device 13-1 to 13-N, and stored in each server device 13-1 to 13-N (step S111b). .. The details of the processing of thread i will be described below.

≪Processing of thread i≫
As illustrated in FIGS. 7 and 8, the reading unit 1261-q of the processing unit 126-q that performs the processing of the thread i determines whether both the file reading lock and the concurrency lock of the thread i are released. The file read lock and parallelism lock of thread 0 are released in the initial state (step S1261a-q). If both the file read lock and the parallelism lock of thread i are not released, the determination in step S1261a-q is repeated.

On the other hand, when both the file read lock and the concurrency lock of thread i are released, the read unit 1261-q reads the file buffer size f from the main memory 123 and sets the area of the file buffer size f in the main memory 123. Secure. Furthermore, the reading unit 1261-q reads the string S _i can be stored in an area of the file buffer size f from a string of text files stored in the auxiliary storage unit 122. In the example of FIG. 11, the following is read as _{the character string S 0.}
“Ishida”, “Taro”, “1990/2/8”, “100-0002”, “sjeifdfgjrrf”, “45dkfjkejdf5”
"Ishida", "Jiro", "1985/5/2", "111-0112", "25df4d4ed", "1s4dlccclseed"
“Ishida”, “Hanako”, “2001/4/8”, “111-2222”, “5d4e4d4ffg”, “skekdjjfaae”
“Sato”, “Taro”, “1992/7/11”, “111-0345”, “dlekd4f3e”, “4selddks”
In the example of FIG. 12, the following are read as a string S _1.
“K304kdkk400-03d”
“Sato”, “Jiro”, “1989/8/21”, “123-0434”, “dkesopd445e”, “4ssjdejdoseae3230dds”
“Sato”, “Hanako”, “1995/2/3”, “145-0234”, “skdeofl4s3d3”, “skek94kdskd4dc”
"Tanaka", "Taro", "1992/3/23", "134-0134", "dj394949495kf", "47s52 \ n5412485d"
"Tanaka", "Jiro", "1979/4/21", "11"
Reading unit 1261-q stores the string _{S i} in the area of the file buffer size f secured in the main memory 123 (Step S1261b-q in FIG. 7, _R i in FIG. 8).

After the character string S _i is stored in the main memory 123, the file read unlocking unit 1267-q releases the file read lock of the thread i + 1 (step S1267-q in FIG. 7, UR _{i + 1 in} FIG. 8).

The parsing unit 1262-q determines whether or not the buffer boundary lock of thread i has been released. The buffer boundary lock of thread 0 is released in the initial state (step S1262a-q). If the buffer boundary lock of thread i is not released, the determination in step S1262a-q is repeated.

On the other hand, when the buffer boundary lock of the thread i is released, the parsing unit 1262-q determines whether or not i ≧ 1 (step S1262b−q). If not i ≧ 1 (i.e., the case of i = 0), parser 1262-q parses the string _{S i} read from the main memory 123, the position and length of each cell included in the string _{S i} The reference information representing the above is calculated and stored in the main memory 123. _{For example, in the case of the character string S 0} illustrated in FIG. 11, the parsing unit 1262-q parses the character string S ₀ and cells "Ishida", "Taro", "1990/2/8", "100-0002", and "sjeifdfgjrrf". "45dkfjkejdf5""Ishida""Jiro""1985/5/2""111-0112""25df4d4ed""1s4dlccclseed""Ishida""Hanako" 2001/4/8 "" 111-2222 "" 5d4e4d4ffg "" skekdjjfaae "Sato""Taro""1992/7/11""111-0345""dlekd4f3e" are identified and their reference information is calculated. Since the end of the last "4selddks" is not the end of the cell, thread 0 does not calculate the reference information for "4selddks". The parsing unit 1262-q reads r from the main memory 123 when the area for storing the reference information in the main memory 123 becomes insufficient, and sets the buffer area for storing the reference information for the r records in the main memory 123. Secure all together. Thereafter, the processing proceeds to step S1268-q (Step S1262c-q in FIG. 7, _P i in FIG. 8). On the other hand, when i ≧ 1, the parsing unit 1262-q outputs the parsing result (reference information of each specified cell and information for identifying a character not included in the cell) in thread i-1 from the main memory 123. Read and identify the characters that are not included in the cell specified by thread i-1 in the character string S _i-1. When the end of the character string S _{i-1 is} the end of the cell, there is no character in the character string S _i-1 that is not included in the cell specified by the thread i-1 (step S1262d-q). Then parser 1262-q reads the string S _i from the main memory 123, to parse the string which is the concatenation of the characters and not in the specified cell and a string S _i thread i-1, this Reference information representing the position and length of each cell included in the character string is calculated and stored in the main memory 123. Reference information when the end of the string S _i-1 is the end of the cell, parsing unit 1262-q is to parse the string S _i, representing the position and length of each cell in the string S _i Is calculated and stored in the main memory 123. _{For example, in the case of the character strings S 0} and S ₁ illustrated in FIGS. 11 and 12, the perspective portion 1262-q is a character " _{1 in the character string S i-1} that is not included in the cell specified by the thread i-1." The character string "4selddks""k304kdkk400-03d" which is a combination of "4selddks""and the character string Si _-1.
“Sato”, “Jiro”, “1989/8/21”, “123-0434”, “dkesopd445e”, “4ssjdejdoseae3230dds”
“Sato”, “Hanako”, “1995/2/3”, “145-0234”, “skdeofl4s3d3”, “skek94kdskd4dc”
"Tanaka", "Taro", "1992/3/23", "134-0134", "dj394949495kf", "47s52 \ n5412485d"
"Tanaka", "Jiro", "1979/4/21", "11"
And each cell included in this string "4selddks""k304kdkk400-03d""Sato""Jiro""1989/8/21""123-0434""dkesopd445e""4ssjdejdoseae3230dds""Sato""Hanako""1995/2/3""145-0234""skdeofl4s3d3""skek94kdskd4dc""Tanaka""Taro""1992/3/23""134-0134""dj394949495kf""47s52 \ n5412485d""Tanaka""Jiro" The reference information indicating the position and length of "1979/4/21" is calculated and stored in the main memory 123. Since the end of the last "11 is not the end of the cell, thread 1 does not calculate the reference information of" 11. The parsing unit 1262-q reads r from the main memory 123 when the area for storing the reference information in the main memory 123 becomes insufficient, and sets the buffer area for storing the reference information for the r records in the main memory 123. Secure all together. Thereafter, the processing proceeds to step S1268-q (Step S1262e-q in FIG. 7, _P i in FIG. 8).

In step S1268-q, the buffer boundary unlocking unit 1268-q releases the buffer boundary lock of thread i + 1. However, the thread corresponding to i + 1> T does not exist, and the buffer boundary lock of the nonexistent thread is not released (step S1268-q in FIG. 7, UB _{i + 1 in} FIG. 8).

_{After that, the encoding unit 1265-q is a processing unit character string PS i, j} which is a character string for r records to be processed included in the _{combined character string CS i} from the text data based on the information specified by the reference information. (However, j = 0, ..., J-1) is selected _{, and the reference information for the r records corresponding to the processing unit character strings PS i, j} and the processing unit character strings PS _{i, j} is stored in the cache memory 1260-q. Store. In the examples of FIGS. 11 and 12 in which r = 2, the _{processing unit character string PS 0, 0 is selected from the combined character string CS 0} = S _0, and the processing unit character string is selected from the combined character string CS ₁ = S ₀ + S _1. PS _1,0 and PS _1,1 are selected (step S1263-q). The encoding unit 1265-q uses the processing unit character strings PS _{i, j} and the reference information of the cache memory 1260-q, and uses the processing unit character strings PS _{i, j} as the source of the predetermined finite set of encoding information E _{i, j.} And the encoding information E _{i and j} are stored in the cache memory 1260-q (step S1265-q in FIG. 7, E _i in FIG. 8).

_{The calculation unit 1266-q performs secret sharing of the encoding information E i and j} read from the cache memory 1260-q, obtains the secret sharing value (calculation value) SS _{i and j} , and stores them in the main memory 123. At this time, since it is not necessary to store the reference information for the r records corresponding to the processing unit character strings PS _{i, j in the main memory 123, the secret sharing values SS i, 0} , ..., SS _{i, J-1} May be overwritten in the area where the reference information for this r record is stored (step S1266-q in FIG. 7, SS _i in FIG. 8).

After that, the parallelism unlocking unit 1269-q _{reads np} from the main memory 123 and releases the parallelism lock of the thread i + _np. However, the thread corresponding to i + n _p > T does not exist, and the parallelism lock of the nonexistent thread is not released. After that, the control unit 125 releases the assignment of the processing unit 126-q to the thread i. As a result, the processing unit 126-q can be assigned to another thread (step S1269-q in FIG. 7, UP _{i + np in} FIG. 8).

[Second Embodiment]
In the second embodiment, the parameter setting device sets the number of records r, the number of parallels _np , and the file buffer size f handled in one unit processing, and the arithmetic device (high-speed version) prohibits the use of escape characters. An example of performing secret sharing (calculation) of a (Comma-Separated Values) file (text file) will be described.

<Structure>
As illustrated in FIG. 1, the arithmetic system 2 of the present embodiment includes a parameter setting device 11, an arithmetic device 22, and N server devices 13-1 to 13-N. Information can be transmitted from the parameter setting device 11 to the arithmetic unit 22, and information can be transmitted from the arithmetic unit 22 to the server devices 13-1 to 13-N.

As illustrated in FIG. 2, the parameter setting device 11 includes an input unit 111a, an output unit 111b, a storage unit 112, a control unit 113, a maximum size setting unit 114a, a minimum size setting unit 114b, an encoding size setting unit 114c, and a calculation size. It has a setting unit 114d, a reference size setting unit 114e, a processing unit calculation unit 114f, a parallel number calculation unit 114g, and a buffer size calculation unit 114h. The parameter setting device 11 executes each process under the control of the control unit 113. Each value obtained by the parameter setting device 11 is stored in the storage unit 112, and is read out from the storage unit 112 as needed and used for other processing.

As illustrated in FIG. 3, the arithmetic unit 12 includes an input unit 121a, an output unit 121b, an auxiliary storage unit 122, a main memory 123, a control unit 125, and a processing unit 226-1 to 226-Q. However, Q is an integer of 2 or more. The arithmetic unit 12 executes each process under the control of the control unit 125.

As illustrated in FIG. 4, the processing unit 126-q (where q = 1, ..., Q) includes a cache memory 1260-q, a reading unit 1261-q, a perspective unit 1262-q, and an encoding unit 1265-q. It has a calculation unit 1266-q, a file read unlock unit 1267-q, a buffer boundary unlock unit 1268-q, and a parallelism unlock unit 1269-q.

<Parameter setting process>
It is the same as the first embodiment.

<Calculation processing>
The arithmetic processing of the arithmetic unit 22 will be described with reference to FIGS. 6 and 13 to 18.
As illustrated in FIG. 6, r, n _p , and f output from the parameter setting device 11 are input to the input unit 121a of the arithmetic unit 22 (FIG. 3) and stored in the auxiliary storage unit 122 (step S111aa). ). Further, the text data to be processed is input to the input unit 121a and stored in the auxiliary storage unit 122. Text data is illustrated in FIGS. 15 to 18. The text data illustrated in FIG. 15 is a CSV file in which each cell is not enclosed in double quotation marks. The use of escape characters in the text data of the present embodiment is not permitted, and each cell contains only characters that can independently determine whether or not it represents the target of secret sharing (calculation). As illustrated in FIG. 16, this text file contains W records rec (1), ..., Rec (W), and each of the records rec (w) (where w = 1, ..., W) is arbitrary. It contains G long cell cells (w, g) (where g = 1, ..., G), and each cell cell (w, g) contains any number of characters. The cell cell (w, g) corresponds to the g-th attribute att (g) from the beginning of the record rec (w) (step S211ab).

After that, r, n _p , and f are read from the auxiliary storage unit 122 into the main memory 123, and the arithmetic processing of threads i = 0, ..., T-1 is executed. The arithmetic processing is started from the thread of i = 0. In the initial state, the file read lock of thread 0 and _{the parallelism lock of threads 0, ..., N p} -1 are released. The control unit 125 allocates the unused processing unit 226-q of the processing units 226-1 to 226-Q to the thread i, and a plurality of threads execute the processing of each thread i in parallel as much as possible (step S226). ). Each secret sharing value obtained in this way is output from the output unit 121b, sent to each server device 13-1 to 13-N, and stored in each server device 13-1 to 13-N (step S111b). .. The details of the processing of thread i will be described below.

≪Processing of thread i≫
As illustrated in FIGS. 13 and 14, the reading unit 1261-q of the processing unit 226-q that performs the processing of thread i determines whether both the file reading lock and the concurrency lock of thread i have been released. The file read lock and parallelism lock of thread 0 are released in the initial state (step S1261a-q). If both the file read lock and the parallelism lock of thread i are not released, the determination in step S1261a-q is repeated.

On the other hand, when both the file read lock and the concurrency lock of thread i are released, the read unit 1261-q reads the file buffer size f from the main memory 123 and sets the area of the file buffer size f in the main memory 123. Secure. Furthermore, the reading unit 1261-q reads the string S _i can be stored in an area of the file buffer size f from a string of text files stored in the auxiliary storage unit 122. In the example of FIG. 17, the following is read as _{the character string S 0.}
Ishida, Taro, 1990/2 / 8,100-0002, Shibuya-ku, Tokyo 〇〇〇, 03-3234-5678
Ishida, Jiro, 2000/4 / 2,274-16, Enoshima, Fujisawa City, Kanagawa Prefecture 〇〇〇, 03-9999-9999
Ishida, Hanako, 1985/6 / 2,352-725, Minato-ku, Tokyo 〇〇〇, 03-1111-9999
Sato, Taro, 2001/5 / 1,100-0002, Chiyoda-ku, Tokyo 〇〇〇, 03-3234-5678
Sato, in the following example 18, the following are read as a string S _1.
Ro, 2001/6 / 2,274-16, Enoshima, Fujisawa City, Kanagawa Prefecture 〇〇〇, 03-9999-9999
Sato, Hanako, 2002/7 / 2,352-725, Shinjuku, Shinjuku-ku, Tokyo 〇〇〇, 03-1111-9999
Tanaka, Taro, 2001/1 / 1,100-0002, Chiyoda-ku, Tokyo 〇〇〇, 03-1234-5678
Tanaka, Jiro, 2001/1 / 2,251-0036, Fujisawa, Kanagawa Prefecture Enoshima thousand reading unit 1261-q stores the string S _i in the area of the file buffer size f secured in the main memory 123 (FIG. 13 step S1261b-q, _R i in FIG. 14) of the.

After the character string S _i is stored in the main memory 123, the file read unlocking unit 1267-q releases the file read lock of the thread i + 1 (step S1267-q in FIG. 13, UR _{i + 1 in} FIG. 14).

Parser 2262-q parses the string S _i read from the main memory 123, and stores the calculated reference information representative of the location and length of each cell in a string S _i in a main memory 123. _{For example, in the case of the character string S 0} illustrated in FIG. 17, the perspective unit 2262-q parses the character string S ₀ and cells "Ishida", "Taro", "1990/2/8", "100-0002", and "Tokyo". Shibuya-ku, Tokyo 〇〇〇 ”“ 03-3234-5678 ”“ Ishida ”“ Jiro ”“ 2000/4/2 ”“ 274-16 ”“ Enoshima 〇〇〇, Fujisawa City, Kanagawa Prefecture ”“ 03-9999-9999 ”“ Ishida, Hanako, June 2, 1985, 352-725, Minato-ku, Tokyo, 〇〇〇, 03-1111-9999, Sato, Taro, May 1, 2001, 100-0002 "○○○, Chiyoda-ku, Tokyo""03-3234-5678""Sato" is specified, and the reference information for them is calculated. Since the last "next" end is not the cell end, thread 0 does not calculate the "next" reference information. _{For example, in the case of the character string S 0} illustrated in FIG. 17, the perspective unit 2262-q parses the character string S ₁ and cells “2001/6/2” “274-16” “Enoshima, Fujisawa-shi, Kanagawa 〇〇 〇 ”“ 03-9999-9999 ”“ Sato ”“ Hanako ”“ 2002/7/2 ”“ 352-725 ”“ Shinjuku, Shinjuku-ku, Tokyo 〇 〇 〇 ”“ 03-1111-9999 ”“ Tanaka ”“ Taro ” Identify "2001/1/1""100-0002""Chiyoda-ku, Tokyo 〇〇〇""03-1234-5678""Tanaka""Jiro""2001/1/2""251-0036" Calculate those reference information. Since the beginning of the first "ro" is not the beginning of the cell, and the end of the last "Enoshima, Fujisawa-shi, Kanagawa" is not the end of the cell, in thread 1, "ro" and "Enoshima, Fujisawa-shi, Kanagawa" The reference information of "○○" is not calculated. When the area for storing the reference information in the main memory 123 becomes insufficient, the perspective unit 2262-q reads r from the main memory 123 and uses the buffer area for storing the reference information for the r record as the main memory. Secure all at 123. The parsing unit 2262-q can start this process before the buffer boundary lock of thread i is released. That is, the perspective unit 2262-q calculates the reference information of each cell included in the character string S _{i before the calculation of the reference information of each cell included in the character string S i-1 is completed in i ≧ 1.} It can be started _(P i in step S2262-q, 14 in FIG. 13).

After that, the cell identification unit 2264-q determines whether or not the buffer boundary lock of the thread i has been released. The buffer boundary lock of thread 0 is released in the initial state (step S2264a-q). If the buffer boundary lock of thread i is not released, the determination in step S2264-q is repeated.

On the other hand, when the buffer boundary lock of the thread i is released and i ≧ 1, the cell identification unit 2264-q uses the reference information, the character string S _i-1, and the character string S _i , and the character is displayed. obtains information a _i corresponding to the position of the cell immediately following the last cell of the column contains S _i-1 stored in the main memory. On the other hand, when the buffer boundary lock of thread i is released and i = 0, the cell identification unit 2264-q does nothing (step S2264b-q).

After that, the buffer boundary unlocking unit 1268-q releases the buffer boundary lock of thread i + 1. However, the thread corresponding to i + 1> T does not exist, and the buffer boundary lock of the nonexistent thread is not released (step S1268-q in FIG. 13, UB _{i + 1 in} FIG. 14).

After that, instead of the processing unit 126-q, the encoding unit 1265-q and the calculation unit 1266-q of the processing unit 226-q perform the processing of steps S1265-q, S1266-q, and S1269-q described in the first embodiment. (Steps S1265-q, S1266-q, S1269-q in FIG. 13, E _i , SS _i , UP _{i + np} in FIG. 14).

[Other variants]

The present invention is not limited to the above-described embodiment. For example, in the first embodiment and the second embodiment, the parameter setting device sets the number of records r, the number of parallels n _p , and the file buffer size f handled in one unit processing, but the parameter setting device sets the file buffer size f. It may be an embodiment in which is not set. Further, in the first embodiment and the second embodiment, the CSV (Comma-Separated Values) file is exemplified as the text file, but the other text files described above may be processed. Further, in the first embodiment and the second embodiment, an example in which secret sharing is performed as an "operation" has been described, but other operations may be performed as an "operation".

The various processes described above are not only executed in chronological order according to the description, but may also be executed in parallel or individually as required by the processing capacity of the device that executes the processes. In addition, it goes without saying that changes can be made as appropriate without departing from the spirit of the present invention.

Each of the above devices is, for example, a general-purpose or dedicated computer including a processor (hardware processor) such as a CPU (central processing unit) and a memory such as a RAM (random-access memory) and a ROM (read-only memory). Is composed of executing a predetermined program. This computer may have one processor and memory, or may have a plurality of processors and memory. This program may be installed in a computer or may be recorded in a ROM or the like in advance. Further, a part or all of the processing units are configured by using an electronic circuit that realizes a processing function without using a program, instead of an electronic circuit (circuitry) that realizes a function configuration by reading a program like a CPU. You may. The electronic circuits constituting one device may include a plurality of CPUs.

When the above configuration is realized by a computer, the processing contents of the functions that each device should have are described by a program. By executing this program on a computer, the above processing function is realized on the computer. The program describing the processing content can be recorded on a computer-readable recording medium. An example of a computer-readable recording medium is a non-transitory recording medium. Examples of such recording media are magnetic recording devices, optical disks, opto-magnetic recording media, semiconductor memories, and the like.

The distribution of this program is performed, for example, by selling, transferring, renting, or the like a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Further, the program may be stored in the storage device of the server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network.

A computer that executes such a program first, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads the program stored in its own storage device and executes the process according to the read program. Another form of execution of this program may be for the computer to read the program directly from a portable recording medium and perform processing according to the program, and each time the program is transferred from the server computer to this computer. , Sequentially, the processing according to the received program may be executed. Even if the above processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition without transferring the program from the server computer to this computer. Good.

Instead of executing a predetermined program on a computer to realize the processing functions of the present device, at least a part of these processing functions may be realized by hardware.

1,2 Arithmetic system 11 Parameter setting device 12,22 Arithmetic device 13-1 to 13-N Server device

Y. Shafranovich, “RFC4180: Common Format and MIME Type for Comma-Separated Values (CSV) Files,” [online], October, 2005, SolidMatrix Technologies, Inc., [Search January 6, 2018], Internet < http://www.ietf.org/rfc/rfc4180.txt> Dai Igarashi, Koji Chida, Hiroki Hamada, Katsumi Takahashi, "Secure Database Operations Using An Improved 3-party Veriable Secure Function Evaluation," InSCIS2011,201. A. Shamir, "How to Share a Secret", Communications of the ACM, November 1979, Volume 22, Number 11, pp.612-613.

FIG. 1 is a block diagram illustrating an arithmetic system of an embodiment. FIG. 2 is a block diagram illustrating the functional configuration of the parameter setting device of the embodiment. FIG. 3 is a block diagram illustrating the functional configuration of the arithmetic unit of the embodiment. FIG. 4 is a block diagram illustrating the functional configuration of the processing unit of the embodiment. FIG. 5 is a flow chart for exemplifying the parameter setting process of the embodiment. FIG. 6 is a flow chart for exemplifying the arithmetic processing of the embodiment. FIG. 7 is a flow chart for exemplifying the processing of the thread i of the embodiment. FIG. 8 is a conceptual diagram for exemplifying the processing of each thread of the embodiment. FIG. 9 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 10 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 11 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 12 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 13 is a flow chart for exemplifying the processing of the thread i of the embodiment. FIG. 14 is a conceptual diagram for exemplifying the processing of each thread of the embodiment. FIG. 15 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 16 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 17 is a conceptual diagram for exemplifying a text file of an embodiment. FIG. 18 is a conceptual diagram for exemplifying a text file of an embodiment.

Hereinafter, embodiments of the present invention will be described.
[Overview]
First, an outline will be described.
<Text file>
In each embodiment, arithmetic processing is performed on the character string of the text file. This text file contains W records, each of which contains G cells of arbitrary length, and each of the cells contains any number of characters. However, the length of each cell has an upper limit according to the attribute of each cell. W and G are integers greater than or equal to 1. For example, at least one of W and G is an integer greater than or equal to 2. W may be an integer of 2 or more, G may be an integer of 2 or more, and both W and G may be an integer of 2 or more. When W is an integer of 2 or more, there is information for specifying the record delimiter between the records adjacent to each other. For example, there is a line break between records adjacent to each other, and a plurality of records are separated from each other by a line break. Further, when G is an integer of 2 or more, there is information for specifying the delimiter between cells between cells adjacent to each other. For example, there is a delimiter or newline between cells adjacent to each other, and multiple cells are separated from each other by the delimiter or newline. An example of a delimiter is the comma ",". As another example, there may be tabs or line breaks between cells adjacent to each other, and half-width spaces or line breaks may exist between cells adjacent to each other. When W is an integer of 2 or more, the number G of cells included in each record is the same as each other. The G cells of each record correspond to the attribute information (also called "schema"). The attribute information represents what kind of attribute information each cell has, and is information for specifying or estimating at least the maximum value and the minimum value of the size (data amount) of the character string represented by each cell. Includes. For example, attribute information contains information indicating what kind of finite set of elements a cell represents. For example, the attribute information may represent "a cell represents a remainder (mod p) modulo p (p is a positive integer)" or "a predetermined number of cells (for example, 10)". it may represent "things is an element string represented by the finite field (for example, extension field GF (2 ^8)), the integer" cell is a predetermined integer (e.g., signed 32-bit integer ) Is a character string. ” Each of the G attribute information may have a one-to-one correspondence with each of the G cells of each record (that is, one attribute information may represent the attribute of one cell). One attribute information may correspond to a plurality of cells (for example, G cells) of each record (that is, one attribute information may represent an attribute of a plurality of cells). In the former case, the attributes of a plurality of cells belonging to one record may be different from each other or may be the same as each other. When W is an integer of 2 or more, the "set of G attributes" corresponding to G cells of all records are the same as each other. That is, the attribute att (g) of the gth cell (where g = 1, ..., G) possessed by all the records is the same as each other. In addition, the attribute information may represent the type of information represented by the cell. Further, the attribute information may or may not be included in the text file (for example, the header of the text file may include the attribute information). Examples of text files are CSV (Comma-Separated Values) files, TSV (tab-separated values) files, SSV (space-separated values) files , and the like. These are collectively referred to as CSV (character-separated values) files and DSV (delimiter-separated values) files.

The above-mentioned "encoding" and "operation" are executed for each processing unit character string which is a character string for r records of a text file. Processing The processing executed for each unit character string is called "unit processing". The processing unit calculating section represents the record number r to be processed in unit process once to obtain the _"C / function value of _{(S csv + S enc + S} ref) " _{(C / (S} csv ₊ S as a record number r _enc + S Obtain the function value of _{ref) and output it.} The cache memory size C may be a predetermined one or may be an input one. The maximum value S _csv is obtained by the maximum size setting unit, the maximum value _Senc is obtained by the encoding size setting unit, and the total size S _ref is obtained by the reference size setting unit. .. The "function value of α" may be α itself or another value corresponding to α. Examples of "function values of α" are the smallest integer greater than or equal to α, the largest integer less than or equal to α, and the integer closest to α. For example, r = C / (S _csv + S _enc + S _ref ), r = ROUNDUP (C / (S _csv + S _enc + S _ref )), or r = ROUNDDOWN It may be (C / (S _csv + S _enc + S _ref )) or r = ROUND (C / (S _csv + S _enc + S _ref )). However, ROUNDUP (α) is a rounding function that rounds up α to an integer value, ROUNDDOWN (α) is a rounding function that rounds down α to an integer value, and ROUND (α) is a rounding function that rounds α to the integer closest to α. Is. Here, in S _csv + _Senc + _Sref , until the processing unit reads the character string for one record from the text file, encodes it into the encoded information while referring to the reference information, and performs "calculation" such as secret sharing. Represents the memory size required for the processing (hereinafter referred to as "a series of processing for one record"). If this memory size is less than or equal to the cache memory size, a series of processes for one record can be executed at high speed without reading data from the main memory on the way. C / ( _Scsv + _Senc + _{Sref ) indicates how many times the memory size (Scsv} + _Senc + _Sref ) required for "a series of processing for one record" can be secured in the cache memory. By using the character string of the number of records r corresponding to C / (S _csv + _Senc + _Sref ) as the processing unit character string, the number of accesses to the main memory when processing r records is reduced and the speed is increased. Can perform operations.

When i ≧ 1, the character string S _i is a character string immediately following the character string _{S i-1.} When i ≧ 1, the cell identification part is included in the character string S _{i-1 by using the reference information, the character string S i-1} and the character string S _i -1 after the buffer boundary lock of the thread i is released. stored in the main memory to obtain information a _i corresponding to the position of the cell immediately following the last cell. Information A _i is, for example, _{information representing a record to which a cell immediately following the last cell included in the character string S i-1} belongs and information representing an attribute corresponding to the cell (for example, the number from the beginning of the record). Information indicating whether or not it corresponds to the attribute of), or information indicating the position and length of the cell immediately following the last cell included _{in the character string Si-1.} When the end of the character string S _{i-1 is} the end of the cell, the _{first cell of the character string S i} becomes "the cell immediately following the last cell included in the character string S _i-1". In this case, _{the storage of the information Ai in} the main memory may be omitted. On the other hand, if the end of the string S _i-1 is not the end of the cell, the cell specific portion using a string S _i-1 and the string S _i, of the last cell included in the "string S _i-1 obtain information a _i generates a cell "immediately following. As a result, information corresponding to the position of the cell whose perspective portion could not be specified can be obtained. The record to which each cell of the text file belongs and the attribute corresponding to the cell (for example, information indicating the number of the attribute from the beginning of the record) can be specified by the reference information and the information _Ai. When i = 0, the cell specific part does nothing.

On the other hand, when the buffer boundary lock of the thread i is released, the parsing unit 1262-q determines whether or not i ≧ 1 (step S1262b−q). If not i ≧ 1 (i.e., the case of i = 0), parser 1262-q parses the string _{S i} read from the main memory 123, the position and length of each cell included in the string _{S i} The reference information representing the above is calculated and stored in the main memory 123. _{For example, in the case of the character string S 0} illustrated in FIG. 11, the parsing unit 1262-q parses the character string S ₀ and cells "Ishida", "Taro", "1990/2/8", "100-0002", and "sjeifdfgjrrf". "45dkfjkejdf5""Ishida""Jiro""1985/5/2""111-0112""25df4d4ed""1s4dlccclseed""Ishida""Hanako""2001/4/8""111-2222""5d4e4d4ffg"" Identify "skekdjjfaae", "Sato", "Taro", "1992/7/11", "111-0345", and "dlekd4f3e", and calculate their reference information. Since the end of the last "4selddks" is not the end of the cell, thread 0 does not calculate the reference information for "4selddks". The parsing unit 1262-q reads r from the main memory 123 when the area for storing the reference information in the main memory 123 becomes insufficient, and sets the buffer area for storing the reference information for the r records in the main memory 123. Secure all together. Thereafter, the processing proceeds to step S1268-q (Step S1262c-q in FIG. 7, _P i in FIG. 8). On the other hand, when i ≧ 1, the parsing unit 1262-q outputs the parsing result (reference information of each specified cell and information for identifying a character not included in the cell) in thread i-1 from the main memory 123. Read and identify the characters that are not included in the cell specified by thread i-1 in the character string S _i-1. When the end of the character string S _{i-1 is} the end of the cell, there is no character in the character string S _i-1 that is not included in the cell specified by the thread i-1 (step S1262d-q). Then parser 1262-q reads the string S _i from the main memory 123, to parse the string which is the concatenation of the characters and not in the specified cell and a string S _i thread i-1, this Reference information representing the position and length of each cell included in the character string is calculated and stored in the main memory 123. Reference information when the end of the string S _i-1 is the end of the cell, parsing unit 1262-q is to parse the string S _i, representing the position and length of each cell in the string S _i Is calculated and stored in the main memory 123. _{For example, in the case of the character strings S 0} and S ₁ illustrated in FIGS. 11 and 12, the perspective portion 1262-q is a character " _{1 in the character string S i-1} that is not included in the cell specified by the thread i-1." The character string "4selddks""k304kdkk400-03d" which is a combination of "4selddks""and the character string Si _-1.
“Sato”, “Jiro”, “1989/8/21”, “123-0434”, “dkesopd445e”, “4ssjdejdoseae3230dds”
“Sato”, “Hanako”, “1995/2/3”, “145-0234”, “skdeofl4s3d3”, “skek94kdskd4dc”
"Tanaka", "Taro", "1992/3/23", "134-0134", "dj394949495kf", "47s52 \ n5412485d"
"Tanaka", "Jiro", "1979/4/21", "11"
And each cell included in this string "4selddks""k304kdkk400-03d""Sato""Jiro""1989/8/21""123-0434""dkesopd445e""4ssjdejdoseae3230dds""Sato""Hanako""1995/2/3""145-0234""skdeofl4s3d3""skek94kdskd4dc""Tanaka""Taro""1992/3/23""134-0134""dj394949495kf""47s52 \ n5412485d""Tanaka""Jiro" The reference information indicating the position and length of "1979/4/21" is calculated and stored in the main memory 123. Since the end of the last "11 is not the end of the cell, thread 1 does not calculate the reference information of" 11. The parsing unit 1262-q reads r from the main memory 123 when the area for storing the reference information in the main memory 123 becomes insufficient, and sets the buffer area for storing the reference information for the r records in the main memory 123. Secure all together. Thereafter, the processing proceeds to step S1268-q (Step S1262e-q in FIG. 7, _P i in FIG. 8).

_{The calculation unit 1266-q performs secret sharing of the encoding information E i and j} read from the cache memory 1260-q, obtains the secret sharing value (calculation value) SS _{i and j} , and stores them in the main memory 123. At this time, since it is not necessary to store the reference information for the r records corresponding to the processing unit character strings PS _{i, j in the main memory 123, the secret sharing values SS i, 0} , ..., SS _{i, J-1} May be overwritten in the area where the reference information for this r record is stored (step S126 5- q in FIG. 7, SS _i in FIG. 8).

As illustrated in FIG. 3, the arithmetic unit 2 2 has an input unit 121a, an output portion 121b, an auxiliary storage unit 122, a main memory 123, control unit 125, and a processing unit 226-1~226-Q. However, Q is an integer of 2 or more. The arithmetic unit 12 executes each process under the control of the control unit 125.

As illustrated in FIG. 4, the processing unit 2 26-q (where q = 1, ..., Q) includes a cache memory 1260-q, a reading unit 1261-q, a perspective unit 2 262-q, and a cell specifying unit 2264. It has −q, an encoding unit 1265-q, an arithmetic unit 1266-q, a file read unlocking unit 1267-q, a buffer boundary unlocking unit 1268-q, and a parallelism unlocking unit 1269-q.

On the other hand, when both the file read lock and the concurrency lock of thread i are released, the read unit 1261-q reads the file buffer size f from the main memory 123 and sets the area of the file buffer size f in the main memory 123. Secure. Furthermore, the reading unit 1261-q reads the string S _i can be stored in an area of the file buffer size f from a string of text files stored in the auxiliary storage unit 122. In the example of FIG. 17, the following is read as _{the character string S 0.}
Ishida, Taro, 1990/2 / 8,100-0002, Shibuya-ku, Tokyo 〇〇〇, 03-3234-5678
Ishida, Jiro, 2000/4 / 2,274-16, Enoshima, Fujisawa City, Kanagawa Prefecture 〇〇〇, 03-9999-9999
Ishida, Hanako, 1985/6 / 2,352-725, Minato- ku , Tokyo 〇〇〇, 03-1111-9999
Sato, Taro, 2001/5 / 1,100-0002, Chiyoda-ku, Tokyo 〇〇〇, 03-3234-5678
Sato, in the following example 18, the following are read as a string S _1.
Ro, 2001/6 / 2,274-16, Enoshima, Fujisawa City, Kanagawa Prefecture 〇〇〇, 03-9999-9999
Sato, Hanako, 2002/7 / 2,352-725, Shinjuku, Shinjuku-ku, Tokyo 〇〇〇, 03-1111-9999
Tanaka, Taro, 2001/1 / 1,100-0002, Chiyoda-ku, Tokyo 〇〇〇, 03-1234-5678
Tanaka, Jiro, 2001/1 / 2,251-0036, Fujisawa, Kanagawa Prefecture Enoshima thousand reading unit 1261-q stores the string S _i in the area of the file buffer size f secured in the main memory 123 (FIG. 13 step S1261b-q, _R i in FIG. 14) of the.

Parser 2262-q parses the string S _i read from the main memory 123, and stores the calculated reference information representative of the location and length of each cell in a string S _i in a main memory 123. _{For example, in the case of the character string S 0} illustrated in FIG. 17, the perspective unit 2262-q parses the character string S ₀ and cells "Ishida", "Taro", "1990/2/8", "100-0002", and "Tokyo". Shibuya-ku, Tokyo 〇〇〇 ”“ 03-3234-5678 ”“ Ishida ”“ Jiro ”“ 2000/4/2 ”“ 274-16 ”“ Enoshima 〇〇〇, Fujisawa City, Kanagawa Prefecture ”“ 03-9999-9999 ”“ Ishida, Hanako, June 2, 1985, 352-725, Minato- ku, Tokyo, 〇〇〇, 03-1111-9999, Sato, Taro, May 1, 2001, 100-0002 "○○○, Chiyoda-ku, Tokyo""03-3234-5678""Sato" is specified, and the reference information for them is calculated. Since the last "next" end is not the cell end, thread 0 does not calculate the "next" reference information. For example, if the string S ₁ illustrated in FIG. 1 8, parser 2262-q, the cell "2001/6/2" parses the string S ₁ "274-16", "Fujisawa Enoshima 〇 〇〇 ”“ 03-9999-9999 ”“ Sato ”“ Hanako ”“ 2002/7/2 ”“ 352-725 ”“ Shinjuku, Shinjuku-ku, Tokyo 〇〇〇 ”“ 03-1111-9999 ”“ Tanaka ”“ Taro "2001/1/1""100-0002""Chiyoda-ku, Tokyo 〇〇〇""03-1234-5678""Tanaka""Jiro""2001/1/2""251-0036""KanagawaPrefecture" Identify "Fujisawa City Enoshima OOOO" and calculate the reference information for them. The first of the beginning of the "low" nose damage at the beginning of the cell, reference information of the thread 1 "wax" is not calculated. When the area for storing the reference information in the main memory 123 becomes insufficient, the perspective unit 2262-q reads r from the main memory 123 and uses the buffer area for storing the reference information for the r record as the main memory. Secure all at 123. The parsing unit 2262-q can start this process before the buffer boundary lock of thread i is released. That is, the perspective unit 2262-q calculates the reference information of each cell included in the character string S _{i before the calculation of the reference information of each cell included in the character string S i-1 is completed in i ≧ 1.} It can be started _(P i in step S2262-q, 14 in FIG. 13).

After that, the cell identification unit 2264-q determines whether or not the buffer boundary lock of the thread i has been released. The buffer boundary lock of thread 0 is released in the initial state (step S2264a-q). If the buffer boundary lock of thread i is not released, the determination in step S2264 a-q is repeated.

After that, instead of the processing unit 126-q, the encoding unit 1265-q and the calculation unit 1266-q of the processing unit 226-q perform the steps S1263-q, S1265-q , and S1269-q described in the first embodiment. process is executed (step S1263-q, S1265-q in FIG. _{13, S 1269-q, E} i in FIG. _{14, SS i, UP i +} np).

Claims (11)

It is a parameter setting device for arithmetic processing on character strings in text files.
The text file contains W records, each of the records contains G cells of arbitrary length, each of the cells contains arbitrary characters, and W and G are integers of 1 or more. G cells correspond to attribute information,
C is the cache memory size, M is the main memory size,
A maximum size setting unit that sets _{the maximum value Scsv} of the size of the character string for one record of the text file by inputting the attribute information, and
A minimum size setting unit that sets _{the minimum value s csv} of the size of the character string for one record by inputting the attribute information, and
An encoding size setting unit that sets _{the maximum value Sensor} of the total size of the encoding information obtained by encoding the character string for one record based on a predetermined finite set, and
An operation size setting unit that sets _{the maximum value Sss} of the total size of the operation values obtained by performing a specific operation on the encoding information for one record, and
A reference size setting unit for setting _{the total size Sref} of reference information representing the position and length of each of the cells for one record, and a reference size setting unit.
The encoding and the operation are processes executed for each processing unit character string which is a character string for r records of the text file, and a function value _{of C / (Scsv} + _Senc + _{Sref) is obtained as the number of records r.} Processing unit calculation unit and
I is the maximum value of the encoding and the number of iterations of the operations are performed for each of the processing units string _{is S ref} ≧ _S when the _{ss max (S ref, S ss} ) = S ref, S ref < when _{S ss max (S ref, S} ss) a _{= S ss,} a function value of _{f 0} is _{s csv · M / (s csv} + S enc + max (S ref, S ss)), in the arithmetic processing A parallel number calculation unit that obtains a function value of f ₀ / I · r · S _{csv as a parallel number n p, and a parallel number calculation unit.}
Parameter setting device having. The parameter setting device according to claim 1.
A parameter setting device further including a buffer size calculation unit that obtains a function value _{of f 0} / _np as a file buffer size f of data collectively read from a character string of the text file during the arithmetic processing. The parameter setting device according to claim 1 or 2.
The maximum value I of the number of iterations is a parameter defined so that the ratio of the total processing amount of the preprocessing to the total processing amount for performing the encoding and the operation of the processing unit character string is equal to or less than a predetermined value. Setting device. An arithmetic unit that performs arithmetic processing on a character string in a text file.
The text file contains W records, each of the records contains G cells of arbitrary length, each of the cells contains any character, W and G are integers greater than or equal to 1, f. Is the defined file buffer size, n _p is the number of parallels, r is a positive integer representing the number of records, i represents each thread, and i ∈ {0, ..., T-1}. T is a positive integer representing the number of threads corresponding to the size of the character string of the text file, 1 ≤ n _p ≤ T, and in the initial state, the file read lock and buffer boundary lock of thread 0 and the thread 0, ..., The parallelism lock of n _{p -1 is released,}
It has a main memory, a cache memory, and a plurality of processing units.
Of the plurality of processing units, the processing unit that performs the processing of thread i is
Wherein after a file read lock and parallelism locking thread i is released, the reading unit for storing a character string in a text file into the main memory by reading the file buffer size string S _i can be stored in an area of f When,
A file read unlocking unit that unlocks the file read lock of thread i + 1 after the character string S _{i is stored in the main memory,}
After the buffer boundary locking thread i is canceled, and a parser for the character string to calculate a reference information representative of the location and length of each cell included in the S _i and stored in the main memory,
A buffer boundary unlocking unit that releases the buffer boundary lock of thread i + 1 after the reference information representing the position and length of each cell included in the character string S _{i is calculated.}
The combined character string CS ₀ when i = 0 is S ₀ , and the combined character string CS _i when i ≧ 1 is the combined character string S _i immediately after the combined character string CS _i-1 . , J is a positive integer, j = 0, ..., J-1, and is a character string for r records to be processed included in the _{combined character string CS i based on the information specified by the reference information.} A process of selecting a certain processing unit character string PS _{i, j} and encoding the processing unit character string PS _{i, j into encoding information E i, j} , which is a source of a predetermined finite set, is performed using the cache memory. Encoding part to do and
A calculation unit that uses the cache memory to perform a process of performing a specific operation on the encoding information E _{i, j to obtain an operation value SS i, j and storing the operation value SS i, j in the main memory.}
A parallelism unlocking unit that unlocks the parallelism of threads i + _np after the calculated values SS _{i and j are obtained.}
Arithmetic logic unit. An arithmetic unit that performs arithmetic processing on a character string in a text file.
The text file contains W records, each of the records contains G cells of arbitrary length, each of the cells contains any character, W and G are integers greater than or equal to 1, f. Is the defined file buffer size, n _p is the number of parallels, r is a positive integer representing the number of records, i represents each thread, and i ∈ {0, ..., T-1}. T is a positive integer representing the number of threads corresponding to the size of the character string of the text file, 1 ≤ n _p ≤ T, and the file read lock of thread 0 and thread 0, ..., n _p -1 in the initial state. The parallelism lock of
It has a main memory, a cache memory, and a plurality of processing units.
Of the plurality of processing units, the processing unit that performs the processing of thread i is
Wherein after a file read lock and parallelism locking thread i is released, the reading unit for storing a character string in a text file into the main memory by reading the file buffer size string S _i can be stored in an area of f When,
A file read unlocking unit that unlocks the file read lock of thread i + 1 after the character string S _{i is stored in the main memory,}
A parsing unit to be stored in the main memory by calculating a reference information representative of the location and length of each cell included in the string S _i,
When i ≧ 1, the character string S _i is a character string _{immediately following the character string S i-1} , and after the buffer boundary lock of the thread i is released, the reference information and the character string S _{i-1 are released.} And the cell identification part that obtains _{the information A i} corresponding to the position of the cell immediately after the last cell included in the _{character string S i-1} by using the character string S _i and the character string S i.
A buffer boundary unlocking unit that releases the buffer boundary lock of thread i + 1 after the information A _{i is obtained, and a buffer boundary unlocking unit.}
The combined character string CS ₀ when i = 0 is S ₀ , and the combined character string CS _i when i ≧ 1 is the combined character string S _i immediately after the combined character string CS _i-1 . , J is a positive integer, j = 0, ..., J-1, and the processing target r included in the concatenation character string CS _{i is based on the reference information and the information specified by the information A i. The process of selecting the processing unit character strings PS i, j} , which are character strings for records, and encoding the processing unit character strings PS _{i, j into the encoding information E i, j} , which is the source of a predetermined finite set, is described above. The encoding part that uses the cache memory and
A calculation unit that uses the cache memory to perform a process of performing a specific operation on the encoding information E _{i, j to obtain an operation value SS i, j and storing the operation value SS i, j in the main memory.}
A parallelism unlocking unit that unlocks the parallelism of threads i + _np after the calculated values SS _{i and j are obtained.}
Arithmetic logic unit. The arithmetic unit according to claim 5.
The cell contains only characters that can be determined by itself whether or not it represents the target of the operation.
In i ≧ 1, the perspective unit is the position of each cell included in the _{character string S i} before the calculation of the reference information representing the position and length of each cell included in the _{character string S i-1 is completed.} And an arithmetic unit that initiates the calculation of reference information representing length. The arithmetic unit according to any one of claims 4 to 6.
C is the cache memory size of the cache memory, M is the main memory size of the main memory, and
S _csv is the maximum value of the character string size for one record of the text file.
s _csv is the minimum value of the size of the character string for one record.
S _enc is the maximum value of the total size of the encoded information obtained by encoding the string of the one record based on the finite set,
S _ss is the maximum value of the total size of the calculated values obtained by performing the operation on the encoding information for the one record.
S _ref is the total size of reference information representing the position and length of each of the cells for the one record.
The number of records r is a function value of _{C / (S csv} + _Senc + _Sref).
I is the maximum number of iterations of the encoding and the calculation is performed the processing unit character string _{PS i,} for each _j, when the _{S ref ≧ S ss max (S} ref, S ss) be a _{= S ref} , _S ref _<when the _{S ss max (S ref, S} ss) a _{= S ss,} is a function value of _{f 0} is _{s csv · M / (s csv} + S enc + max (S ref, S ss)), The parallel number n _p is a function value of _{f 0} / I · r · _Scsv.
An arithmetic unit in which the file buffer size f is _{a function value of f 0} / _np. The arithmetic unit according to any one of claims 4 to 7.
The parsing unit is an arithmetic unit that collectively secures a buffer area for storing the reference information for the r record in the main memory when the area for storing the reference information becomes insufficient in the main memory. .. A method of executing processing of each part of the apparatus according to any one of claims 1 to 8. A program for operating a computer as the device according to any one of claims 1 to 8. A computer-readable recording medium in which a program for operating a computer as any of the devices 1 to 8 is stored.

Images