JP2024042360A - Data processing program, data processing method, and data processing device - Google Patents

Abstract

An object of the present invention is to reduce variations in processing amount of a plurality of calculation units.
A storage unit 11 stores 2N (N is an integer greater than or equal to 2) pieces of data used for calculations in combination with each of a plurality of partner data. From the 2N pieces of data, the processing unit 12 identifies the top N first data and the bottom N second data in the result of sorting based on the number of partner data to be operated on. The processing unit 12 sends each of the N first data items to each of the N calculation units from the first calculation unit to the N-th calculation unit in descending order of the number of partner data to be calculated. assign. The processing unit 12 sends each of the lower N second data to each of the N calculation units from the first calculation unit to the N-th calculation unit in ascending order of the number of partner data to be calculated. assign. The processing unit 12 uses the N calculation units to execute operations on N data out of the 2N data in parallel based on the results of allocation of the 2N data to the N calculation units.
[Selection diagram] Figure 1

Description

The present invention relates to a data processing program, a data processing method, and a data processing device.

A method called pattern mining is used to analyze data. In pattern mining, combinations of data that satisfy certain conditions may be extracted from a data set. Operations on each of a plurality of combinations of data can be executed in parallel by a plurality of arithmetic units such as a CPU (Central Processing Unit) included in a computer, for example. Here, methods are being considered to make the execution of parallel processing by computers more efficient.

For example, in a multiprocessor system, a schedule control method has been proposed in which the OS (Operating System) scheduler allocates suspended threads to the CPU with the least load, thereby simultaneously allocating suspended threads and performing load distribution processing. be.

In addition, when dispatching threads in batches, the dispatch timing is adjusted so that the threads are dispatched to the same CPU as the previous time, thereby increasing the possibility that data in the cache provided for each CPU will be reused. There is also a proposal.

Also, in a computer system with a multi-core processor, a method has been proposed in which the OS scheduler dynamically assigns each thread to each core using the number of cycles per instruction, called the CPI (Cycles Per Instruction) rate.

Additionally, multithreaded processing collects statistical data, such as the number of object, memory, or register locks issued while a task is executing, and adjusts the number of threads in subsequent processing cycles based on the statistical data. There are also system proposals.

International Publication No. 2007/017932 Japanese Patent Application Laid-Open No. 7-302246 US Patent Application Publication No. 2008/0059712 US Patent Application Publication No. 2017/0031708

When performing calculations on all combinations of elements of the first data set and elements of the second data set using multiple calculation units, each element of the first data set is responsible for the calculation regarding that element. It is conceivable to allocate an arithmetic unit. In this case, redundant combination operations may be omitted. Therefore, the number of elements of the second data set used as a combination partner may vary for each element of the first data set. Therefore, for example, if the elements of the first data set are cyclically assigned to each calculation unit in descending order of the number of elements of the second data set to be combined, the amount of processing performed by each calculation unit will vary. arise. Variations in the processing amount of each calculation unit become a factor that delays the completion of calculations for all combinations.

In one aspect, the present invention aims to reduce variations in processing amount of a plurality of arithmetic units.

In one aspect, a data processing program is provided. This data processing program causes a computer to execute the following process. The computer identifies the top N first data and the bottom N second data in the sorting result by the number of the partner data to be operated on, from 2N (N is an integer of 2 or more) pieces of data used in the operation by combination with each of the multiple partner data. The computer assigns each of the top N first data to each of the N operation units from the first operation unit to the Nth operation unit in descending order of the number of the partner data to be operated on. The computer assigns each of the bottom N second data to each of the N operation units from the first operation unit to the Nth operation unit in ascending order of the number of the partner data to be operated on. The computer executes the operation on N data of the 2N data in parallel by the N operation units based on the result of the assignment of the 2N data to the N operation units.

Also, in one aspect, a computer-implemented data processing method is provided. Further, in one aspect, a data processing device including a storage section and a processing section is provided.

In one aspect, variations in the amount of processing of the plurality of calculation units can be reduced.

FIG. 1 is a diagram illustrating a data processing device according to a first embodiment. FIG. 7 is a diagram illustrating an example of hardware of a data processing device according to a second embodiment. It is a figure which shows the example of extraction of the combination of goods. FIG. 13 is a diagram showing an example of calculation of the number of people who purchased all three types of products. FIG. 3 is a diagram showing an example of calculation for a combination of two types of products. FIG. 3 is a diagram showing an example of calculation for a combination of three types of products. It is a diagram showing an example of hardware of a CPU. FIG. 3 is a diagram illustrating an example of data allocation to multiple cores. FIG. 2 is a diagram illustrating a functional example of a data processing device. It is a flowchart which shows the example of a combination calculation. FIG. 3 is a diagram illustrating a comparative example of data allocation to multiple cores. FIG. 3 is a diagram showing a comparison of the number of combinations calculated by each core. FIG. 7 is a diagram showing an example (part 1) of cache use according to the second embodiment. FIG. 7 is a diagram showing an example (part 2) of cache use according to the second embodiment. FIG. 13 illustrates an example of an execution order of combination calculations according to the third embodiment; FIG. 7 is a diagram illustrating an example of the execution order of combination calculations when the number of products is 10. FIG. 13 illustrates an example of cache usage according to the third embodiment; FIG. 13 illustrates a continuation of an example of cache usage according to the third embodiment; It is a figure which shows the cache use example (continued) of 3rd Embodiment. It is a flowchart which shows the example of a combination calculation. FIG. 7 is a diagram showing an example of a sector cache according to a fourth embodiment. FIG. 12 is a diagram illustrating an example of cache use according to the fourth embodiment. FIG. 23 illustrates a continuation of the cache usage example according to the fourth embodiment; FIG. 12 is a diagram showing an example of cache use (continued) according to the fourth embodiment. It is a flowchart which shows the example of a combination calculation. It is a flowchart which generalizes the combination calculation of a 2nd embodiment. It is a flowchart which generalizes combination calculation of a 3rd embodiment. It is a flowchart which generalizes combination calculation of a 4th embodiment.

The present embodiment will be described below with reference to the drawings.
[First embodiment]
A first embodiment will be described.

FIG. 1 is a diagram illustrating a data processing apparatus according to a first embodiment.
The data processing device 10 uses a plurality of calculation units to perform calculations on combinations of elements of the first data set and elements of the second data set. The data processing device 10 has a storage section 11 and a processing section 12.

The storage unit 11 may be a volatile semiconductor memory such as a RAM (Random Access Memory), or may be a nonvolatile storage such as an HDD (Hard Disk Drive) or a flash memory. The processing unit 12 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the processing unit 12 may include a specific purpose electronic circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor executes a program stored in a memory such as a RAM (or the storage unit 11).

Here, a set of multiple processors is sometimes referred to as a "multiprocessor." Further, a processor having multiple processor cores is sometimes referred to as a "multi-core processor." The processing unit 12 is, for example, a multiprocessor or a multicore processor.

The processing unit 12 has N calculation units. N is an integer of 2 or more. In FIG. 1, a case where N=2 is illustrated. When N=2, the processing section 12 includes calculation sections 12a and 12b. For example, when the processing unit 12 is a multiprocessor, the calculation units 12a and 12b may be processors included in the processing unit 12. For example, when the processing unit 12 is a multi-core processor, the calculation units 12a and 12b may be processor cores included in the processing unit 12. The calculation units 12a and 12b are used for combination calculations.

Here, the target combination is a combination of elements of the first data set and elements of the second data set. The first data set used for this target combination calculation includes 2N pieces of data. In the example of N=2, the first data set includes four data d1, d2, d3, and d4 as elements of the first data set. The elements of the second data set that are to be combined with the elements of the first data set are referred to as "partner data." In the combination calculation, a calculation unit is assigned to each element of the first data set in charge of calculation of the element. This is because by executing the calculations for all combinations corresponding to one element of the first data set in one thread, there is less need to replace data used in the calculations, and the calculations can be performed efficiently.

Furthermore, in the calculation of combinations, calculations for overlapping combinations are omitted. By omitting operations for duplicate combinations, unnecessary operations can be omitted. For example, if the combination of data d1 and certain partner data and the combination of data d2 and other partner data correspond to the same combination, it is only necessary to perform calculations on data d1 with the corresponding partner data; The calculation between d2 and the other partner data is omitted. For this reason, the data d1 to d4 differ in the number of partner data to be calculated in combination. Therefore, the processing unit 12 allocates data d1 to d4 to each calculation unit as follows.

From the N pieces of data, the processing unit 12 sorts the upper N pieces of first data and the lower N pieces of second data in the sorting result based on the number of partner data to be operated on (the number of combinations with the partner data). Identify. Table 11a illustrates the number of combinations of each of data d1 to d4 with partner data in an example where N=2. The number of partner data to be calculated for each of the data d1 to d4 can also be said to be the number of partner data to be combined to each of the data d1 to d4. For example, the table 11a may be stored in the storage unit 11. It is assumed that the number of combinations of data d1 to d4 with each other's data is as follows.

The number of combinations of data d1 and partner data is m1. The number of combinations of data d2 and partner data is m2. m2<m1. The number of combinations of data d3 and partner data is m3. m3<m2. The number of combinations of data d4 and partner data is m4. m4<m3. Note that m1 to m4 are all positive integers. In this case, the processing unit 12 specifies the top two pieces of data d1 and d2 and the bottom two pieces of data d3 and d4 in the number of combinations.

The processing unit 12 allocates each of the N pieces of high-order first data to each of the N pieces of calculation units from the first calculation unit to the N-th calculation unit in descending order of the number of combinations. The processing unit 12 allocates each of the lower N second data to each of the N calculation units from the first calculation unit to the N-th calculation unit in ascending order of the number of combinations.

In the above example of N=2, the calculation unit 12a may be the first calculation unit, and the calculation unit 12b may be the second calculation unit. The processing unit 12 allocates the two data d1 and d2 with the highest number of combinations to the respective calculation units 12a and 12b in descending order of the number of combinations. That is, the processing unit 12 allocates data d1 to the calculation unit 12a, and allocates data d2 to the calculation unit 12b. Furthermore, the processing unit 12 allocates the two lowest data d3 and d4 in the number of combinations to the calculation units 12a and 12b, respectively, in ascending order of the number of combinations. That is, the processing unit 12 allocates data d4 to the calculation unit 12a and data d3 to the calculation unit 12b.

Regarding the above allocation, when looking at the order of the calculation units 12b and 12a, it can be said that the data d2 and d1 are allocated to the calculation units 12b and 12a, respectively, in ascending order of the number of combinations. Similarly, it can be said that the data d3 and d4 are respectively allocated to the calculation units 12b and 12a in descending order of the number of combinations.

Further, the allocation processing by the processing unit 12 described above may be executed by either the calculation units 12a or 12b, or may be executed by another calculation unit (control calculation unit) included in the processing unit 12. . In FIG. 1, illustration of the other calculation units is omitted.

The processing unit 12 uses the N calculation units to execute operations on N data out of the 2N data in parallel based on the results of allocation of the 2N data to the N calculation units. In the above example where N=2, the processing unit 12 causes the calculation units 12a and 12b to execute calculations on the two pieces of data in parallel based on the results of allocation of data d1 to d4 to the calculation units 12a and 12b.

FIG. 1 shows an example 20 of execution of calculations by the calculation units 12a and 12b. Execution example 20 shows an example of the execution time of calculations by each of the calculation units 12a and 12b. The direction from left to right on the horizontal axis is the positive direction of time. For example, the calculation unit 12a performs calculations on data d1, and then performs calculations on data d4. The calculation unit 12b performs calculations on data d2, and then performs calculations on data d3. In this way, the processing unit 12 uses the calculation units 12a and 12b to simultaneously execute calculations on two pieces of data in parallel.

According to the data processing device 10 of the first embodiment, from 2N pieces of data used for calculations in combination with each of a plurality of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of data processed by each other, the top N pieces of data are sorted according to the number of piece of pieces of pieces of pieces of pieces of pieces of pieces of pieces of pieces of data that are to be calculated. The first data and the lower N second data are specified. Each of the N pieces of first data at a higher rank is assigned to each of the N pieces of calculation parts from the first calculation part to the Nth calculation part in descending order of the number of partner data to be calculated. The lower N pieces of second data are respectively assigned to the N pieces of calculation units from the first calculation unit to the Nth calculation unit in ascending order of the number of partner data to be calculated. Based on the results of allocation of 2N data to the N calculation units, operations on N data out of the 2N data are executed in parallel by the N calculation units. This makes it possible to reduce variations in the amount of processing performed by the plurality of arithmetic units.

Here, for example, it is conceivable to cyclically allocate the data d1, d2, d3, and d4 to the calculation units 12a and 12b in descending order of the number of combinations with partner data (number of partner data to be calculated). In the case of cyclic allocation, data d1 and d3 are allocated to the calculation unit 12a, and data d2 and d4 are allocated to the calculation unit 12b, respectively. However, in this case, the difference between the processing amount for the data d1 and d3 handled by the calculation section 12a and the processing amount for the data d2 and d4 handled by the calculation section 12b becomes relatively large. As a result, the total calculation time for the data d1 and d3 by the calculation unit 12a becomes relatively long, and the end of the entire calculation is delayed.

On the other hand, according to the data processing device 10, as shown in execution example 20, there is a difference between the amount of processing for data d1 and d4 handled by the calculation unit 12a and the amount of processing for data d2 and d3 handled by the calculation unit 12b. can be reduced. As a result, the difference between the calculation time of the calculation unit 12a and the calculation time of the calculation unit 12b is reduced, and the delay in completing the entire calculation is reduced.

[Second embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram illustrating a hardware example of a data processing device according to the second embodiment.

The data processing device 100 includes a CPU 101, a RAM 102, an HDD 103, a GPU 104, an input interface 105, a media reader 106, and a communication interface 107. These units included in the data processing device 100 are connected to a bus inside the data processing device 100. The CPU 101 corresponds to the processing unit 12 of the first embodiment. RAM 102 or HDD 103 corresponds to storage unit 11 in the first embodiment.

The CPU 101 is a processor that executes program instructions. The CPU 101 loads at least a portion of the programs and data stored in the HDD 103 into the RAM 102 and executes the programs. The CPU 101 is a multi-core processor that includes multiple processor cores. The processor cores may also be referred to as CPU cores. Hereinafter, the processor cores will be referred to as cores.

The RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the CPU 101 and data used for calculations by the CPU 101. Note that the data processing device 100 may include a type of memory other than RAM, or may include a plurality of memories.

The HDD 103 is a nonvolatile storage device that stores software programs such as an OS (Operating System), middleware, and application software, and data. Note that the data processing device 100 may include other types of storage devices such as flash memory and SSD (Solid State Drive), or may include a plurality of nonvolatile storage devices.

The GPU 104 outputs images to a display 111 connected to the data processing device 100 in accordance with instructions from the CPU 101. The display 111 may be any type of display, such as a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD), a plasma display, or an organic electro-luminescence (OEL) display.

The input interface 105 acquires an input signal from the input device 112 connected to the data processing apparatus 100 and outputs it to the CPU 101. As the input device 112, a mouse, a touch panel, a touch pad, a pointing device such as a trackball, a keyboard, a remote controller, a button switch, etc. can be used. Further, a plurality of types of input devices may be connected to the data processing apparatus 100.

The media reader 106 is a reading device that reads programs and data recorded on the recording medium 113. For example, a magnetic disk, an optical disk, a magneto-optical disk (MO: Magneto-Optical disk), a semiconductor memory, etc. can be used as the recording medium 113. Magnetic disks include flexible disks (FD: Flexible Disks) and HDDs. Optical disks include compact discs (CDs) and digital versatile discs (DVDs).

For example, the media reader 106 copies programs and data read from the recording medium 113 to other recording media such as the RAM 102 and the HDD 103. The read program is executed by the CPU 101, for example. Note that the recording medium 113 may be a portable recording medium, and may be used for distributing programs and data. Further, the recording medium 113 and the HDD 103 are sometimes referred to as computer-readable recording media.

Communication interface 107 is connected to network 114 and communicates with other information processing devices via network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or a router, or a wireless communication interface connected to a wireless communication device such as a base station or access point.

Note that the data processing device 100 may be an HPC (High-Performance Computing) system that includes a plurality of CPUs 101 and is used for large-scale calculations.
The data processing device 100 is used for pattern mining. In pattern mining, the data processing device 100 uses pattern mining to find combinations of conditions for which a predetermined number or more of samples satisfy the conditions. Below, as an example, pattern mining that is performed to plan purchase predictions, advertisements, web postings, etc. in the marketing field will be illustrated. However, the data processing device 100 can also be applied to pattern mining in other fields, such as the political field, such as predicting election turnout, and the medical field, such as discovering the causes of diseases.

FIG. 3 is a diagram showing an example of extracting product combinations.
For example, in the marketing field, pattern mining may be performed in the following cases. The first example is a case where a combination of products is sought in which the number of people who purchased all the products in the combination is equal to or greater than a certain number, such as 100 people. A second example is a case where it is investigated whether the number of people who have purchased both of two specific products is greater than or equal to a certain number, such as 100 people. Alternatively, pattern mining may be performed for purposes other than the first and second examples.

In the example of the second embodiment, a case is exemplified in which the data processing device 100 selects a combination of k arbitrary products and extracts a combination of products for which the number of customers who purchased all of them is s_min or more. do. k is an integer of 2 or more. s_min is an integer greater than or equal to 1.

The product purchase history data 200 shows the purchase history of "product 1", "product 2", "product 3", etc. for the customer ID. "0" in the product column indicates that the product was not purchased, and "1" indicates that it was purchased. The output list 210 is output by the data processing device 100 based on the product purchase history data 200. The output list 210 shows combinations of k products for which the number of people who purchased all of the combinations of k products is equal to or greater than s_min. Next, a calculation example when k=3 will be explained.

FIG. 4 is a diagram showing an example of calculating the number of people who purchased all three types of products.
For example, based on the product purchase history data 200, the number of people who purchased all of "Product 1,""Product2," and "Product 3" is calculated as follows. First, the data processing device 100 calculates the logical product (AND) of the "product 1" column and the "product 2" column of the product purchase history data 200 (step ST1). The data corresponding to the column of each product is a vector of 0's and 1's. And _{(1, 2)} is the result of calculating the logical product of the "Product 1" column and the "Product 2" column. For example, if the column for "Product 1" is "1111" and the column for "Product 2" is "0110", And _{(1, 2)} becomes "0110".

Next, the data processing device 100 calculates the logical product And _{(1, 2 , 3)} of And (1, 2) and the column of "product 3" (step ST2). If the column of "Product 3" is "0100", And _{(1, 2, 3)} becomes "0100".

Then, the data processing device 100 calculates the result of step ST2, that is, the total sum Sum _{(1, 2, 3)} of the elements of And _(1, 2, 3) (step ST3). For And _{(1, 2, 3)} , Sum _{(1, 2, 3)} = 0 + 1 + 0 + 0 = 1. For example, if Sum _{(1, 2, 3)} is greater than or equal to s_min, the data processing device 100 adds the combination of “product 1,” “product 2,” and “product 3” to the output list 210.

Next, an example of calculation for the combination of two types of products exemplified in step ST1 will be described.
FIG. 5 is a diagram showing an example of a calculation for a combination of two types of products.

As an example, the number of products d_x=10. The matrix 201 illustrates each operation of logical product And _(x,y) corresponding to the combination of two products x and y. Column x corresponds to product x in the product purchase history data 200. Row y corresponds to product y in the product purchase history data 200. That is, "column 1", "column 2", ..., "column 10" correspond to "product 1", "product 2", ..., "product 10", respectively. Also, "row 1", "row 2", ..., "row 10" correspond to "product 1", "product 2", ..., "product 10", respectively.

One square of the matrix 201 indicates one logical product And _{(x, y) operation for each combination of columns of products x and y} . For example, the cell in the 7th row and 8th column of the matrix 201 indicates the logical product And _{(8, 7)} of the column "Product 7" and the column "Product 8" of the product purchase history data 200. When the number of products d_x=10, the total number of combinations of two product columns is ₁₀ C ₂ =45. Note that a square in the matrix 201 with diagonal lines indicates that the calculation corresponding to the square is not executed.

The data processing device 100 performs calculations for combinations of three types of products based on the logical product And _{(x, y)} .
FIG. 6 is a diagram showing an example of calculation for a combination of three types of products.

The matrix 202 illustrates each operation of the logical product And _{(x, y, z)} corresponding to the combination of data of And _{(x, y) and data of product z (other data) different from products x and y.} do. Column z in matrix 202 corresponds to product z in product purchase history data 200. A row of matrix 202 corresponds to And _(x,y) . Note that in the figure, the rows of the matrix 202 are identified by labels ( _{x, y) corresponding to And (} x, y). The number of z values on the horizontal axis of matrix 202 is d_x. The number of (x, y) on the vertical axis of matrix 202 is _{d_x} C ₂ .

The rows of the matrix 202 are arranged in the order of (9, 10), (8, 10), (8, 9), (7, 10), (7, 9), ..., (1, 2). The columns are arranged in the order of z from 10 to 1. The number of combinations to be calculated for each And _{(x, y)} , that is, the number of partner data to be operated on, is specified in advance as the number of z to be combined for (x, y) so that there is no duplication of combinations. The arrangement order of (x, y) corresponding to a row of the matrix 202 is determined by the number of combinations (number of z) to calculate And _{(x, y)} data corresponding to the row with respect to the And _{(x, y).} ) is sorted in descending order.

One square of the matrix 202 indicates one logical product operation And _{(x, y , z) for the combination of And (x,} y) and the column of product z. When the number of products d_x=10, the total number of combinations of And _{(x, y)} and the column of product z is ₁₀ C ₃ =120. Note that a square in the matrix 202 with diagonal lines indicates that the calculation corresponding to the square is not executed.

The logical AND operations shown in the matrix 201 and the matrix 202 can be executed in parallel in multiple threads by allocating the operation corresponding to one row to each core of the CPU 101 as one thread.

FIG. 7 is a diagram showing an example of CPU hardware.
CPU 101 has cores 121, 122, 123, 124 and cache memory 125. Cores 121 to 124 are processor cores that each execute operations in parallel. The cache memory 125 stores data used for calculations by each of the cores 121-124. Cache memory 125 can be accessed faster than RAM 102 from cores 121 to 124. Data used for calculations is loaded into the cache memory 125 from the RAM 102 . Furthermore, data stored in the cache memory 125 may be written to the RAM 102. Cache memory 125 is shared by cores 121-124. For example, data loaded into the cache memory 125 for the operation of one core can be reused for the operation of other cores.

For example, with respect to the matrix 202, data used for calculations is allocated to the cores 121 to 124 as follows.
FIG. 8 is a diagram illustrating an example of data allocation to multiple cores.

Matrix 300 describes a portion of matrix 202. In the example of the second embodiment, a set of eight (4×2=8) And _{(x, y)} data with adjacent sort orders is set for the four cores 121 to 124 as follows. Each data is assigned to each core.

Specifically, each of the four pieces of data with the highest number of combinations is allocated to each of the four cores from the first core 121 to the fourth core 124 in descending order of the number of combinations. Further, each of the four pieces of data with the lowest number of combinations is allocated to each of the four cores from the first core 121 to the fourth core 124 in ascending order of the number of combinations. Note that the "number of combinations" corresponds to the number of partner data to be calculated.

Then, in the example of the matrix 300, data is assigned to the cores 121 to 124 in a nested manner as follows. And _{(9, 10)} , which has a combination number of “8”, is assigned to the core 121. And _{(8, 10)} , which has a combination number of “7”, is assigned to the core 122. And _{(8, 9)} , which has a combination number of “7”, is assigned to the core 123. And _{(7, 10)} , which has a combination number of “6”, is assigned to the core 124. Further, And _{(7, 9)} , which has a combination number of “6”, is assigned to the core 124. And _{(7, 8)} , which has a combination number of “6”, is assigned to the core 123. And _{(6, 10)} , which has a combination number of “5”, is assigned to the core 122. And _{(6, 9)} , which has a combination number of “5”, is assigned to the core 121.

For other data corresponding to rows in matrix 300, cores 121 to 124 are similarly assigned in a nested manner.
Then, the cores 121 to 124 can execute operations in the row direction of the matrix 300 in parallel using four threads. For example, the core 121 uses one thread to perform eight logical product operations combining And _{(9, 10)} and each of the columns of products z=8 to 1. The core 122 executes, in one thread, seven logical product operations combining And _{(8, 10)} and each of the columns of products z=7 to 1. The core 123 executes, in one thread, seven logical product operations combining And _{(8, 9)} and each of the columns of products z=7 to 1. The core 124 executes, in one thread, six logical product operations combining And _{(7, 10)} and each of the columns of products z=6 to 1.

When the core 124 completes the logical product calculation for the six ways, the core 124 uses one thread to calculate the logical product for the six ways combining And _{(7, 9)} and each column of products z=6 to 1. Execute. Similarly, the cores 121 to 123 sequentially execute operations on the assigned data.

Regarding the matrix 201, similarly to the method illustrated in FIG. can be assigned.

FIG. 9 is a diagram illustrating a functional example of the data processing device.
The data processing device 100 includes a data storage section 130, a cache storage section 140, an allocation section 150, and an arithmetic control section 160. The data storage unit 130 is realized by a storage area of the RAM 102 or the HDD 103. Cache storage unit 140 is realized by a storage area of cache memory 125. The allocation unit 150 and the calculation control unit 160 are realized by the CPU 101 executing a program stored in the RAM 102.

The data storage unit 130 stores all data used in the matrix 201 and the matrix 202. For example, the data storage unit 130 stores product purchase history data 200.
Cache storage unit 140 stores part of the data stored in data storage unit 130. A portion of the data stored in the data storage section 130 is loaded into the cache storage section 140 in accordance with the execution of calculations by the calculation control section 160. Cache storage unit 140 has a fixed size.

The allocation unit 150 allocates data corresponding to the rows of the matrix 201 and the matrix 202 to each of the cores 121 to 124. The data allocation method used is the method illustrated in FIG. 8.

The calculation control unit 160 uses the cores 121 to 124 to execute calculations on each data in parallel based on the data allocation result by the allocation unit 150. The calculation control unit 160 stores the calculation results in the data storage unit 130.

Next, a processing procedure by the data processing device 100 will be explained.
FIG. 10 is a flowchart showing an example of combination calculation.
(S10) The calculation control unit 160 performs k=2 combination calculations using the cores 121 to 124, and outputs the logical product And _{(x, y)} of each combination as a list A.

(S11) The allocation unit 150 allocates each data in list A, that is, the logical product And _{(x, y)} , to each core in a nested manner. In the assignment in step S11, the method illustrated in FIG. 8 is used.

(S12) The arithmetic control unit 160 repeatedly performs the combination calculation of three arbitrary products shown in steps S13 to S16 below using the cores 121 to 124. The total number of combination calculations is _{d_x} C ₃ times.

(S13) Each of the cores 121 to 124 selects the combination And _{(x, y)} for which it is responsible, and selects a product z that is part of an unselected combination (x, y, z).
(S14) Each of the cores 121 to 124 calculates the logical product And _{(x, y, z) of the three products x, y, and z} .

(S15) Each of the cores 121 to 124 counts the number of people who purchased three products x, y, z Sum _{(x, y, z) from the logical product And (x, y, z )} .
(S16) Each of the cores 121 to 124 adds Sum _{(x, y, z)} to list B.

(S17) When the calculation control unit 160 completes the calculations for all combinations of three arbitrary products shown in steps S13 to S16, it advances the process to step S18.
(S18) The calculation control unit 160 outputs list B. List B is stored in the data storage section 130. Then, the combination calculation process ends.

For example, from the list B output in step S18, a combination of products x, y, z for which Sum _{(x, y, z)} is greater than or equal to s_min is added to the output list 210.
Next, a comparative example for the data allocation shown in FIG. 8 will be described.

FIG. 11 is a diagram illustrating a comparative example of data allocation to a plurality of cores.
The matrix 400 is a part of the matrix 202. For example, it is possible to cyclically assign data corresponding to each row of the matrix 400 to four cores C1, C2, C3, and C4 in descending order of the number of combinations. In this case, the core C1 is assigned And _(9,10) , the core C2 is assigned And _(8,10) , the core C3 is assigned And _(8,9 ), the core C4 is assigned And _(7,10) , and the core C1 is assigned And _(7,9) . However, in the assignment method of the comparative example, the more the number of cores increases, the greater the difference in the amount of processing of each core becomes. When the difference in the amount of processing between the cores becomes large, processing is biased toward a specific core, and the overall processing time becomes longer.

Furthermore, as another method for comparison, a part of the matrix 400 may be divided into four parts and allocated to the cores C1 to C4. For example, a first region in the first to fifth rows and first to fifth columns from the left of the matrix 400, a second region in the first to fifth rows and sixth to tenth columns from the left, and a second region in the first to fifth rows and sixth to tenth columns from the left; It can be divided into a third area in the 1st to 10th rows and 1st to 5th columns, and a fourth area in the 6th to 10th rows and the 6th to 10th columns. Then, each logical product operation included in the divided areas is allocated to the cores C1 to C4 for each area. However, even with this allocation method, the difference in processing amount between the cores C1 to C4 becomes large. For example, when assigning the first region to core C1, the second region to core C2, the third region to core C3, and the fourth region to core C4, the number of combinations that cores C2 and C4 are responsible for is 25, and core C1 is responsible for The number of combinations handled by the core C3 becomes nine, and the number of combinations handled by the core C3 becomes one. For this reason, the difference in processing amount between the cores C1 and C3 becomes extremely large even in the first to tenth rows and first to tenth columns of the matrix 400, and it is difficult to bridge the difference. Become. Furthermore, in this method, data for the same product z can only be used in at most two of the four cores.

FIG. 12 is a diagram showing a comparison of the number of combinations calculated by each core.
Table 500 shows the number of combinations calculated by each of cores 121 to 124 when using the method of the comparative example shown in FIG. 11 and when using the method of data processing apparatus 100 illustrated in FIG. 8. The number of cores is 4. The number of products d_x=10. The number of combinations calculated by a certain core is the sum of the number of combinations corresponding to each piece of data (x, y) assigned to the core.

In the method of the comparative example shown in FIG. 11, the number of combinations calculated by each of the cores 121, 122, 123, and 124 is 33, 31, 29, and 27. In the method of the comparative example, the maximum value of the difference in processing amount between cores is 6.

On the other hand, in the method of the data processing apparatus 100 of FIG. 8, the number of combinations calculated by each of the cores 121, 122, 123, and 124 is 30, 30, 30, and 30. In the data processing device 100, the maximum value of the difference in processing amount between cores is zero. Therefore, in the data processing apparatus 100, the calculation time can be reduced by three combinations (=33-30) as compared to the method of the comparative example.

As described above, according to the data processing device 100 of the second embodiment, variations in the processing amount of the plurality of cores can be reduced. As a result, it is possible to reduce the delay in completing calculations for all combinations.

By the way, the cores 121 to 124 share the cache storage unit 140 in the above calculation, for example, as follows.
FIG. 13 is a diagram showing an example (part 1) of cache use according to the second embodiment.

Here, the calculation for one combination of And _{(x, y)} and z is defined as one step. Table 600 shows combinations (z, (x, y)) of calculation targets of cores 121 to 124 in each step. The steps proceed in ascending order of step numbers, such as steps 1, 2, . . . In addition, in the figure, the core 121 is abbreviated as "core 1," the core 122 as "core 2," the core 123 as "core 123," and the core 124 as "core 4." It is assumed that the time required to calculate one combination is the same for cores 121 to 124.

Furthermore, the order of loading data into the cache memory unit 140 of each core in each step is assumed to be core 121 being the fastest, core 122 being second fastest, core 123 being third fastest, and core 124 being slowest. When there is no free space in the cache memory unit 140, data that has been used the longest is preferentially deleted from the cache memory unit 140.

For example, in step 1, the cores 121 to 124 execute the following combination (z, (x, y)) operation. The core 121 executes the operation (8, (9, 10)). Core 122 executes the operation (7, (8, 10)). The core 123 executes the operation (7, (8, 9)). Core 124 executes the operation (6, (7, 10)).

Immediately after step 1 is completed, the data held in the cache storage unit 140 is stored in order from the oldest to the oldest: And _{(9, 10)} , "Product 8", And _{(8, 10)} , And _{(8, 9)} , "Product 7" , And _{(7, 10)} , becomes "product 6". Here, “product z” corresponds to the column of product z in the product purchase history data 200. In addition, in the figure, the time since the last use of data stored in the cache storage unit 140 is longer as it goes to the left side of the figure (older usage history), and the time that it has been last used is longer as it goes to the right side of the figure. It has been used for a short time (the usage history is new).

Next, in step 2, cores 121 to 124 perform the calculations for the following combinations (z, (x, y)). Core 121 performs the calculation (7, (9, 10)). Core 122 performs the calculation (6, (8, 10)). Core 123 performs the calculation (6, (8, 9)). Core 124 performs the calculation (5, (7, 10)).

Immediately after step 2 is completed, the data held in the cache storage unit 140 is stored in order from the oldest to the oldest: "Product 8", And _{(9, 10)} , "Product 7", And _{(8, 10)} , "Product 6", And _{(7, 10)} , becomes "product 5".

In this way, in the second embodiment, the cores 121 to 124 perform calculations in the row direction of the matrix 300 in the order from the larger z to the smaller z. Each cell of the matrix 300 in FIG. 13 has a step number written in which the calculation for that cell is executed.

FIG. 14 is a diagram showing an example (part 2) of cache use according to the second embodiment.
For example, when new data D is added to the cache storage unit 140 during the execution of step x, if there is no free space in the cache storage unit 140, the data D with the oldest usage history (for example "Product 8" data) is deleted.

When the data for "product 8" is deleted from the cache storage unit 140, the data in the cache storage unit 140 cannot be reused in other calculations. For this reason, when the data for "product 8" is used in other calculations, the data for "product 8" is reloaded into the cache storage unit 140. For example, each row of the matrix 300 handled by the cores 121 to 124 is divided into blocks of four rows. In this case, when each core moves on to the calculation of the next block, after calculating the first block of the matrix 300, there may be no data for "product z" in the cache. In that case, the data for "product z" is reloaded.

[Third embodiment]
Next, a third embodiment will be described. The points that are different from the second embodiment described above will be mainly explained, and the explanations of the common points will be omitted.

The data processing device 100 according to the third embodiment provides a function that reduces the number of times data is loaded into the cache memory 125 compared to the second embodiment.
FIG. 15 is a diagram illustrating an example of the execution order of combination calculations according to the third embodiment.

Matrix 700 describes a portion of matrix 202.
In the third embodiment, cores 121 to 124 perform operations on a certain row in matrix 700 using z in a first order. Then, the cores 121 to 124 execute the operation in the next row using z in the reverse order from the first order. The first order is, for example, descending order of z. The first order may be ascending order of z.

Each cell of the matrix 700 contains the number of steps indicating the execution order of combination calculation for the data And _{(x, y)} corresponding to the row. The data And _{(x, y)} corresponding to each row is assigned to the cores 121 to 124 in the same manner as in the second embodiment. For example, the cores 121 to 124 execute calculations for the data And _{(x, y)} they are responsible for using the other data in descending order of z, and then execute calculations for the data And _{(x', y')} they are responsible for using the other data in ascending order of z.

More specifically, in an example in which And _{(9, 10)} and And _{(6, 9)} are assigned to the core 121, the core 121 assigns And _{(9, 10)} in the order of z=..., 3, 2, 1. After performing the calculation, the calculation is performed for And _{(6, 9)} in the order of z=1, 2, 3, . . . In this manner, in the third embodiment, the cores 121 to 124 execute calculations using the partner data of the combination in the looped order. As a result, when a certain combination of calculations is performed, partner data that has already been loaded into the cache memory 125 can be easily used for another combination of calculations.

FIG. 16 is a diagram showing an example of the execution order of combination calculations when the number of products is 10.
Matrix 700a shows the execution order of all combination calculations by cores 121 to 124 when the number of products d_x = 10. It is assumed that core 121 is assigned to And _{(9, 10)} , core 122 is assigned to And _{(8, 10)} , core 123 is assigned to And _{(8, 9)} , and core 124 is assigned to And _{(7, 10)} . After that, data is assigned to each core in a nested manner using the method of the second embodiment illustrated in FIG.

FIG. 17 is a diagram showing an example of cache use according to the third embodiment.
Table 800 illustrates the storage state of data in cache storage unit 140 for each step number (step) of matrix 700a. The (x, y) item in the table 800 indicates data for And _{(x, y)} . The item z in the table 800 indicates data corresponding to the "product z" column of the product purchase history data 200.

Cells surrounded by thick frames in the table 800 indicate data stored in the cache storage unit 140, that is, data on the cache. Note that the cache storage unit 140 can store up to 10 pieces of data in each step. In other words, if the total number of data in the cache that corresponds to the And _{(x, y)} data and the "product z" column exceeds 10, the data that has been used the longest since the last time is stored in the cache storage section. 140.

Furthermore, hatched cells with thin dots in the table 800 indicate data loaded from the data storage unit 130 to the cache storage unit 140, that is, data loaded from the RAM 102 to the cache memory 125.

Furthermore, the diagonally hatched cells in the table 800 indicate data deleted from the cache storage unit 140 due to cache overflow. The data to be deleted is the data with the oldest usage history in the cache storage unit 140.

Furthermore, the darkly dotted hatched cells in table 800 indicate data used in calculations at each step. The usage history of the data used in the calculation is updated to the latest in the cache storage unit 140.

Table 800 exemplifies steps with step numbers 1 to 18 (step=1 to 18) among the steps in matrix 700a.
FIG. 18 is a diagram showing an example of cache usage (continued) according to the third embodiment.

Table 800a exemplifies step numbers 19 to 26 (step=19 to 26) among the steps in matrix 700a.
FIG. 19 is a diagram showing an example of cache use (continued) according to the third embodiment.

Table 800b exemplifies steps with step numbers 27 to 30 (step=27 to 30) among the steps in matrix 700a. Note that the table 800b also exemplifies a step with step number 31 (step=31) in which each data is finally deleted from the cache storage unit 140.

In the examples of tables 800 to 800b, the number of loads from RAM 102 to cache memory 125 is 53 times in total.
FIG. 20 is a flowchart showing an example of combination calculation.

The third embodiment differs from the second embodiment in that step S13a of the procedure in FIG. 10 is replaced with step S13a. Therefore, step S13a will be mainly explained below, and explanations of other steps will be omitted. Step S13a is executed next to step S12.

(S13a) Each of the cores 121 to 124 selects the combination And _{(x, y)} for which it is responsible, and selects products z that are part of an unselected combination (x, y, z) in a wraparound order. Here, the selection method for the wraparound order uses the method exemplified by the matrix 700 in Fig. 15 or the matrix 700a in Fig. 16. Then, the process proceeds to step S14.

In this way, the data processing device 100 increases the possibility of reusing the data loaded in the cache memory 125 by executing the combination calculation using the partner data of the combination in the z-folding order. The number of times the file is loaded can be reduced. As a result, the data processing device 100 can reduce the overhead associated with loading to the cache memory 125, and can speed up combinatorial calculations.

[Fourth embodiment]
Next, a fourth embodiment will be described. Items that are different from the second and third embodiments described above will be mainly described, and descriptions of common items will be omitted.

In the fourth embodiment, the data processing device 100 provides a function to further reduce the number of times data is loaded into the cache memory 125. The fourth embodiment is applied when k is 3 or more.

FIG. 21 is a diagram showing an example of a sector cache according to the fourth embodiment.
The cache storage unit 140 of the fourth embodiment has sectors 141 and 142. The sectors 141 and 142 are storage areas obtained by further dividing the entire storage area of the cache storage unit 140. The identification number of sector 141 is "#0". The identification number of sector 142 is "#1". Sector cache is a software-controllable cache mechanism. By using a sector cache, the storage area of the cache memory 125 can be divided into a plurality of sectors in this way, and data that can be reused and data that cannot be reused can be stored separately for each sector.

Here, in the examples of tables 800 to 800b illustrated in Figures 17 to 19, the data of "product z" is more reusable than And _{(x, y} ), because when k ≥ 3, And _{(x, y)} is never referenced again after processing of one row of matrix 700a is completed.

Therefore, the cores 121 to 124 store And _{(x, y)} and the data of "product z" in different sectors. For example, the cores 121 to 124 store data of And _{(x, y)} in the sector 141. The cores 121 to 124 store data of “product z” in the sector 142. For example, which data is placed in which sector can be specified in the source code of the program executed by the cores 121 to 124, such as arranging the array of data a in the sector 141. The capacity ratio of each of the sectors 141 and 142 can be specified, for example, in units of WAY. The value obtained by multiplying the unit size s per WAY by the WAY value w of the sector becomes the size M (=w×s) of the sector. s and w are positive real numbers.

For example, the size of the sector 141 is the smallest size M1 of the sizes corresponding to the WAY value w (=w x s) that is larger than the data size for the number of cores (for example, the size for 4 data for 4 cores). The WAY value w1 corresponds to (=w1×s). w1 is a positive real number. The allocation unit 150 may determine the WAY value to be allocated to the sector 141 in the storage area of the cache storage unit 140 based on the data size, and allocate the remaining WAY value to the sector 142.

FIG. 22 is a diagram showing an example of cache use according to the fourth embodiment.
The table 900 illustrates the storage state of data in the cache storage unit 140 for each step number (step) of the matrix 700a. The (x, y) item in the table 900 indicates data for And _{(x, y)} . The item z in the table 900 indicates data corresponding to the "product z" column of the product purchase history data 200.

The number of cores is 4. The number of products is d_x = 10. Also, the memory area of the cache memory unit 140 is assumed to be capable of holding a maximum of 10 pieces of data. Furthermore, the memory area of the cache memory unit 140 is assumed to be divided in a ratio of sector #0:sector #1 = 4:6.

In the fourth embodiment, the data of And _{(x, y)} is placed in sector 141 (sector #0), and the data of "product z" is placed in sector 142 (sector #1), which is different from the third embodiment. The form is different from that of

Squares surrounded by thick frames in the table 900 indicate data stored in the cache storage unit 140, that is, data on the cache. Note that the cache storage unit 140 can store up to 10 pieces of data in each step.

Furthermore, hatched cells with thin dots in the table 900 indicate data loaded from the data storage unit 130 to the cache storage unit 140, that is, data loaded from the RAM 102 to the cache memory 125.

Further, the diagonally hatched cells in the table 900 indicate data deleted from the cache storage unit 140 due to cache overflow. Data deletion is performed sector by sector. The data to be deleted is the data with the oldest usage history within the sector.

Furthermore, the darkly dotted hatched cells in the table 900 indicate data used in the calculation of each step. The usage history of the data used in calculations is updated to the latest within the corresponding sector.

Table 900 exemplifies steps with step numbers 1 to 19 (step=1 to 19) among the steps in matrix 700a.
FIG. 23 is a diagram showing an example of cache use (continued) according to the fourth embodiment.

Table 900a exemplifies steps with step numbers 20 to 28 (step=20 to 28) among the steps in matrix 700a.
FIG. 24 is a diagram showing an example of cache usage (continued) according to the fourth embodiment.

Table 900b exemplifies steps with step numbers 29 and 30 (step=29, 30) among the steps in matrix 700a.
FIG. 25 is a flowchart showing an example of combination calculation.

The fourth embodiment differs from the third embodiment in that step S10a is executed before step S10 in the procedure of FIG. Therefore, step S10a will be mainly explained below, and explanations of other steps will be omitted.

(S10a) The allocation unit 150 specifies the sector cache ratio, that is, the ratio of sectors 141 and 142, and specifies that data z be allocated to sector 142 (sector #1). As described above, the allocation unit 150 determines the size of the sector 141 (sector #0) in the cache storage unit 140 based on the number of cores and the data size per column of the product purchase history data 200. , and the remainder is the size of sector 142 (sector #1). The process then proceeds to step S10.

In this way, by using the sector cache, the data processing device 100 can further reduce the number of times the cache memory 125 is loaded. For example, in the tables 900 to 900b in FIGS. 22 to 24, the number of times the data is loaded from the RAM 102 to the cache memory 125 is 44 times in total.

In the example of the third embodiment shown in Figures 17 to 19, the total number of loads is 53. For example, in the third embodiment, there are cases where data with high reusability (e.g. data with z=1) is deleted from the cache and then loaded again, as in steps 22 and 23 in Figure 18. Such cases cause the number of loads to increase.

On the other hand, by using the sector cache, the data processing device 100 can prevent the data of "product z" from being evicted from the cache due to loading the data of And _{(x, y)} . For example, when loading the data of And _{(x, y)} , the data of And _{(x, y)} with the longest time since last use is evicted from sector 141, and the data of the new And _{(x, y)} is evicted. Data is stored in sector 141. As a result, the data of "product z" is retained in the sector 142 of the cache memory 125 for a long time, and the number of times the data is loaded from the RAM 102 to the cache memory 125 can be further reduced.

In this way, in the fourth embodiment, by using the sector cache, the number of loads can be reduced more than in the third embodiment. Moreover, as a result, the data processing device 100 can further reduce the overhead associated with loading into the cache memory 125, and can further speed up the combination calculation.

By the way, when pattern mining processing is executed in multiple threads, processing time may become long due to variations in processing load for each thread (core) and inefficient use of the cache memory 125.

For example, as a countermeasure against variations in processing load among cores, it is conceivable that the scheduler of the OS constantly monitors the processing load of the cores and allocates processing to cores with less load. However, such a method requires a program for constant monitoring. Furthermore, the OS scheduler merely distributes processing such as programs other than pattern mining processing to each core.

In contrast, the data processing device 100 does not use a scheduler to allocate threads to cores, but rather determines the allocation before executing the combinational calculation process. Therefore, implementation is easy. Further, the method of the data processing device 100 is more useful for leveling the processing amount of each core as the matrix data to be handled is closer to a triangular matrix. That is, the method of the data processing device 100 can make the loads of each core more uniform as the matrix data to be handled is closer to a triangular matrix. Furthermore, by increasing the reusability of data loaded into the cache memory 125, the data processing device 100 can reduce the number of times data is loaded from the RAM 102 to the cache memory 125, reducing the overall processing time for combinational calculations. can.

Next, an example in which the processing procedures of the second to fourth embodiments are generalized will be described.
FIG. 26 is a flowchart generalizing the combination calculation of the second embodiment.
(S20) The calculation control unit 160 determines whether the number n of product combinations to be calculated is n=2. Here, n is an integer of 2 or more. If n≠2, the process proceeds to step S21. If n=2, the process proceeds to step S22.

(S21) The calculation control unit 160 performs k=n-1 combination calculations using the cores 121 to 124, and outputs the logical product And _{(x, . . .)} of each combination as a list A. The process then proceeds to step S23.

(S22) The calculation control unit 160 sets the other party's data d to list A. List A is a list of partner data d. The process then proceeds to step S23.
(S23) The allocation unit 150 allocates each data in list A, that is, the logical product And _{(x, . . .)} to each core in a nested manner. In the assignment in step S23, the method illustrated in FIG. 8 is used. (x,...) indicates a combination of n-1 products.

(S24) The arithmetic control unit 160 repeatedly performs the combination calculation for arbitrary n products shown in steps S25 to S27 below using the cores 121 to 124. The total number of combination calculations is _{d_x} C _n times.

(S25) Each of the cores 121 to 124 selects the combination And _(x,...) for which the core is in charge, and selects the data of the product d that is the unselected combination (x,..., d) (i.e. , select the other party's data d). (x,...,d) indicates a combination of n products. Each of the cores 121 to 124 stores n products x, . ．． ．． , d, and _(x,...,d) .

(S26) Each of the cores ₁₂₁ to 124 calculates n products x, . ．． ．． , _d is counted.
(S27) Each of the cores 121 to 124 adds Sum _{(x, . . . , d)} to list B.

(S28) When the calculation control unit 160 finishes calculations for all combinations of arbitrary n products shown in steps S25 to S27, it advances the process to step S29.
(S29) The calculation control unit 160 outputs list B. List B is stored in the data storage section 130. List B is a list of total purchases of products (x,...,d). Then, the combination calculation process ends.

FIG. 27 is a flowchart generalizing the combination calculation of the third embodiment.
The procedure in FIG. 27 differs from the procedure in FIG. 26 in that step S24a is executed after step S24 in FIG. Therefore, step S24a will be mainly explained below, and explanations of other steps will be omitted.

(S24a) Each of the cores 121 to 124 selects the combination And _(x,...) for which it is in charge, and returns the product d that is the unselected combination (x,..., d). Select in the order shown. Here, the method exemplified in matrix 700 in FIG. 15 and matrix 700a in FIG. 16 is used as the selection method for the order of folding. The process then proceeds to step S25.

FIG. 28 is a flowchart generalizing the combination calculation of the fourth embodiment.
The procedure in FIG. 28 is for the case where n≧3, and differs from the procedure in FIG. 27 in that step S20a is executed instead of step S20 of the procedure in FIG. 27, and step S22 is not executed. Therefore, step S20a will be mainly explained below, and explanations of other steps will be omitted. Note that since n≧3 in the procedure of FIG. 28, k≧2 in step S21.

(S20a) The allocation unit 150 specifies the sector cache ratio, that is, the ratio of sectors 141 and 142, and specifies that data z be allocated to sector 142 (sector #1). As described above, the allocation unit 150 determines the size of the sector 141 (sector #0) in the cache storage unit 140 based on the number of cores and the data size per column of the product purchase history data 200. , and the remainder is the size of sector 142 (sector #1). The process then proceeds to step S21.

In this way, the procedures of the second to fourth embodiments can be generalized.
As described in the second to fourth embodiments, the data processing device 100 executes the following processing.

The allocation unit 150 selects the top Nth pieces of data from 2N (N is an integer of 2 or more) data to be used for calculations in combination with each of a plurality of partner data in the sorting result based on the number of partner data to be calculated. 1 data and the lower N pieces of second data are specified. The allocation unit 150 assigns each of the N first data items to each of the N calculation units from the first calculation unit to the N-th calculation unit in descending order of the number of partner data to be calculated. assign. The allocation unit 150 assigns each of the N pieces of lower second data to each of the N pieces of calculation units from the first calculation unit to the Nth calculation unit in ascending order of the number of partner data to be calculated. assign. The calculation control unit 160 causes the N calculation units to execute calculations on N data out of the 2N data in parallel based on the results of allocation of 2N data to the N calculation units.

Thereby, the data processing device 100 can reduce variations in the processing amount of the plurality of calculation units (N calculation units). Cores 121 to 124 are examples of four calculation units. Note that, as illustrated in FIGS. 5 and 6, the functions of the data processing device 100 are such that the closer the data to be handled is to a triangular matrix, the smaller the variation in the processing amount of the plurality of calculation units can be. Furthermore, in the example of the matrix 201 in FIG. 5, x data corresponding to a column is an example of counterpart data for y data corresponding to a row. In the example of the matrix 202 in FIG. 6, z data corresponding to a column is an example of counterpart data to And _{(x, y)} data corresponding to a row.

Furthermore, when performing a calculation on each of the N pieces of high-order first data, the N calculation units may select the partner data to be used for the calculation from among the plurality of partner data in the first order. When performing an operation on each of the lower N pieces of second data, the N calculation units may select the partner data to be used for the operation from among the plurality of partner data in an order opposite to the first order. .

Thereby, the data processing device 100 can reduce the number of times that the other party's data is loaded onto the cache memory 125 shared by the N calculation units. As a result, the data processing device 100 can reduce the overhead associated with loading into the cache memory 125, and can speed up calculations.

Furthermore, the allocation unit 150 may divide the storage area of the cache memory 125 that is accessed by the N calculation units into a first storage area and a second storage area. Then, the N calculation units load the data on which the operation is to be performed out of the 2N pieces of data into the first storage area, and load the partner data on which the operation is to be performed among the plurality of partner data into the second storage area. You can also load it into

Thereby, the data processing device 100 can suppress the expulsion of counterpart data with a high possibility of reuse from the cache memory 125, and further reduce the number of times that counterpart data is loaded onto the cache memory 125. Sector 141 is an example of a first storage area. Sector 142 is an example of a second storage area.

For example, the allocation unit 150 determines the size of the first storage area based on the value obtained by multiplying the first size, which is the size of each of the 2N pieces of data, by N. Thereby, the data processing device 100 can appropriately determine the size of the first storage area, and can efficiently reduce the number of times that the other party's data is loaded onto the cache memory 125. For example, the data processing device 100 determines the size of the first storage area to be the minimum necessary size among the usable storage areas of the cache memory 125 for the number N of calculation units, and sets the remaining size to the second storage area. The size shall be . In this way, the size of the second storage area can be made relatively large, and the size that can hold the other party's data becomes large. Therefore, the data processing device 100 can further reduce the number of times the other party's data is loaded onto the cache memory 125.

Note that the information processing in the first embodiment can be realized by causing the processing unit 12 to execute a program. Further, the information processing according to the second embodiment can be realized by causing the CPU 101 to execute a program. The program can be recorded on a computer-readable recording medium 113.

For example, the program can be distributed by distributing the recording medium 113 on which the program is recorded. Alternatively, the program may be stored in another computer and distributed via a network. For example, the computer may store (install) a program recorded on the recording medium 113 or a program received from another computer in a storage device such as the RAM 102 or the HDD 103, and read and execute the program from the storage device. good.

10 data processing device 11 storage unit 11a table 12 processing unit 12a, 12b calculation unit 20 execution example

Claims (6)

to the computer,
From 2N (N is an integer of 2 or more) data used for calculations in combination with each of multiple partner data, the upper N first data and the lower N pieces of second data and
Allocating each of the N pieces of first data at the higher rank to each of the N pieces of calculation units from the first calculation unit to the N-th calculation unit in descending order of the number of the partner data to be calculated, and Allocating each of the lower N pieces of second data to each of the N pieces of calculation units from the first calculation unit to the Nth calculation unit in ascending order of the number of the partner data to be calculated. ,
Based on the result of allocation of the 2N data to the N calculation units, the N calculation units execute the calculation on N data of the 2N data in parallel;
A data processing program that executes processing. In executing the calculation on each of the N first data items of the higher order, selecting the partner data to be used for the calculation among the plurality of partner data in a first order;
In executing the operation on each of the N pieces of second data at the lower level, selecting the partner data to be used for the operation from among the plurality of partner data in an order opposite to the first order;
The data processing program according to claim 1. The storage area of the cache memory accessed by the N calculation units is divided into a first storage area and a second storage area, and the data to be executed for the calculation among the 2N pieces of data is stored in the first storage area. loading into the second storage area, and loading the partner data that is the target of the operation among the plurality of partner data into the second storage area;
The data processing program according to claim 2. 4. The data processing program according to claim 3, wherein the size of the first storage area is determined based on a value obtained by multiplying a first size, which is the size of each of the 2N pieces of data, by N. The computer is
From 2N (N is an integer greater than or equal to 2) data used in an operation in combination with each of multiple partner data, the upper N first data and the lower N pieces of second data, and
Allocating each of the N pieces of high-rank first data to each of the N pieces of calculation units from the first calculation unit to the N-th calculation unit in descending order of the number of the partner data to be calculated, and assigning each of the lower N second data to each of the N calculation units from the first calculation unit to the N-th calculation unit in ascending order of the number of the partner data to be calculated; ,
Based on the result of allocation of the 2N data to the N calculation units, the N calculation units execute the calculation on N data among the 2N data in parallel;
Data processing methods. A storage unit that stores 2N (N is an integer equal to or greater than 2) pieces of data used in a calculation based on a combination with each of a plurality of pieces of partner data;
a processing unit which identifies, from the 2N pieces of data, the top N pieces of first data and the bottom N pieces of second data in a sorting result by the number of pieces of partner data to be operated on, assigns the top N pieces of first data to each of the N pieces of calculation units from the first calculation unit to the Nth calculation unit in descending order of the number of pieces of partner data to be operated on, and assigns the bottom N pieces of second data to each of the N pieces of calculation units from the first calculation unit to the Nth calculation unit in ascending order of the number of pieces of partner data to be operated on, and executes the calculation on N pieces of data of the 2N pieces of data in parallel by the N calculation units based on the allocation result of the 2N pieces of data to the N calculation units;
A data processing device comprising:

Images