WO2023005264A1 - 一种数据处理方法和装置 - Google Patents

一种数据处理方法和装置 Download PDF

Info

Publication number
WO2023005264A1
WO2023005264A1 PCT/CN2022/087297 CN2022087297W WO2023005264A1 WO 2023005264 A1 WO2023005264 A1 WO 2023005264A1 CN 2022087297 W CN2022087297 W CN 2022087297W WO 2023005264 A1 WO2023005264 A1 WO 2023005264A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
sample
processed
fragments
sampling
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/087297
Other languages
English (en)
French (fr)
Inventor
郑瑞峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Wodong Tianjun Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Publication of WO2023005264A1 publication Critical patent/WO2023005264A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system

Definitions

  • the present disclosure relates to the technical field of big data, and in particular to a data processing method and device.
  • Data skew has always been a major pain point in the field of big data. It refers to the fact that in the process of distributed data parallel processing, due to uneven data fragmentation, a large amount of data is concentrated on one or a small number of computing nodes, resulting in the processing of a small number of tasks. The speed is much lower than the average speed, slowing down the whole calculation process.
  • One of the common data skew situations is called data inflation, which is manifested as: the size of the input fragment is not significantly larger than the average level, but the corresponding data only contains a few different keys, resulting in the output of the corresponding task after the Join connection is executed. The amount of data is much larger than other tasks.
  • an embodiment of the present invention provides a data processing method and device to solve the technical problem of being unable to handle data expansion.
  • a data processing method including:
  • a target fragment is selected from the plurality of fragments, and data fragmentation is performed on the target fragment.
  • the data to be processed in the data table is sampled in an adaptive manner to obtain the sample data distribution of the data table, including:
  • the sample data distribution of the data table is obtained according to the sampling result of the data to be processed.
  • the data to be processed is sampled using a reservoir sampling algorithm, including:
  • obtaining the sample data distribution of the data table according to the sampling result of the data to be processed includes:
  • the sampling result of the data to be processed includes a plurality of samples
  • each data table For each data table, divide the data amount of the data to be processed by the accumulated sampling amount of each sample to obtain the weight of each sample in the data table, thereby obtaining the sample data distribution of the data table.
  • calculating the output data volumes corresponding to the connection operations of the multiple fragments according to the sample data distribution of the data table including:
  • each of the data tables contains the target samples
  • the output data volume corresponding to the connection operation of the fragments is calculated.
  • calculating the amount of output data corresponding to the connection operation of the fragments includes:
  • the weights of each target sample in each data table are multiplied and accumulated, so as to obtain the output data volume corresponding to the connection operation of the shards.
  • the weights of each target sample in each data table are multiplied and accumulated, so as to obtain the output data volume corresponding to the connection operation of the fragments, including:
  • connection operation is an inner connection
  • the weights of each target sample in each data table are multiplied and accumulated, and then the weights of each target sample in the left join data table are added to obtain the split The amount of output data corresponding to the connection operation;
  • connection operation is right connection, multiply the weights of each target sample in each data table and accumulate the multiplication, and then add the weight of each target sample in the right connection data table to obtain the connection of the fragments The amount of output data corresponding to the operation;
  • the weights of each target sample in each data table are multiplied and accumulated, and then the weights of each target sample in the left join data table and right join data table are added to obtain the The amount of output data corresponding to the connection operation of the above shards.
  • the target fragment is screened out from the plurality of fragments, and data fragmentation is performed on the target fragment, including:
  • a data processing device including:
  • a sampling module configured to sample the data to be processed in the data table in an adaptive manner to obtain the sample data distribution of the data table
  • a calculation module configured to divide the data to be processed into a plurality of fragments, and respectively calculate the output data volume corresponding to the connection operation of the plurality of fragments according to the sample data distribution of the data table;
  • the sharding module is configured to select a target shard from the multiple shards according to the output data volume corresponding to the connection operation of the multiple shards, and perform data sharding on the target shard.
  • an electronic device including:
  • processors one or more processors
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the foregoing embodiments.
  • a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-mentioned embodiments is implemented.
  • Figure 1 is a schematic diagram of the basic principle of Spark AQE
  • Figure 2 is a schematic diagram of the basic principle of the dynamic optimization strategy in Spark AQE
  • FIG. 3 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present disclosure
  • FIG. 4 is a schematic diagram of the main flow of a data processing method according to a reference embodiment of the present disclosure
  • FIG. 5 is a schematic diagram of the main flow of a data processing method according to another reference embodiment of the present disclosure.
  • 6a and 6b are schematic comparison diagrams before and after adopting the data processing method of the embodiment of the present disclosure.
  • FIG. 7 is a schematic diagram of main modules of a data processing device according to an embodiment of the disclosure.
  • FIG. 8 is an exemplary system architecture diagram to which embodiments of the present disclosure can be applied.
  • Fig. 9 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present disclosure.
  • the dynamic optimization strategy in Spark AQE will count the size of each slice. When A0 is found to be significantly larger than the median of each slice, it will trigger the division of A0. At the same time, in order to ensure the smooth progress of the calculation, B0 Make a copy, as shown in Figure 2. After such processing, although the total number of tasks has increased (from 4 to 5), the data of each task is more uniform, and the calculation time is almost the same, which brings better performance to the entire query calculation.
  • data inflation which is manifested as: the size of the input fragment is not significantly larger than the average level, but the corresponding data only contains a few different keys, resulting in the output of the corresponding task after the Join connection is executed. The amount of data is much larger than other tasks.
  • the frequency of the key on the A3 side and the B3 side are N_A and N_B respectively, then in the result after the Join connection, the frequency of the key is N_A*N_B.
  • Data bloat occurs when N_A and N_B are normal to the input shard size, but N_A*N_B is larger than the output shard size.
  • the embodiments of the present disclosure aim at predicting the output data volume of the Join connection through self-adaptive data statistics and analysis in a distributed environment, and based on this, analyze the data slices to handle data bloat.
  • Fig. 3 is a schematic diagram of a main flow of a data processing method according to an embodiment of the disclosure.
  • the data processing method may include:
  • step 301 the data to be processed in the data table is sampled in an adaptive manner to obtain the sample data distribution of the data table.
  • the data to be processed in the data table is sampled in an adaptive manner, and the sampling logic is executed in a distributed manner on the execution end (Executor).
  • the sample data can be collected from the data table through adaptive data sampling, so as to calculate the sample data distribution of the data table.
  • Executor In the Spark computing engine, the node that actually executes specific computing tasks.
  • step 301 may include: obtaining the data to be processed from the data table, so as to obtain the key in the data to be processed; judging whether the cumulative number of processed items is greater than or equal to the first quantity threshold; The key in the processing data is written into the counting table, and the key in the counting table is used as the sampling result; if so, the reservoir sampling algorithm is used to sample the data to be processed; according to the sampling result of the data to be processed, the The sample data distribution for the data table.
  • a first quantity threshold (MapSize) may be preset, and then incremental updates are performed in the form of streams. Each time a piece of data to be processed is obtained from the data table, the key is extracted from the piece of data to be processed, and the cumulative number of processed pieces is recorded in a cumulative manner.
  • the data to be processed includes a key and its corresponding value
  • the embodiment of the present disclosure only needs the key, so it is only necessary to extract the key from the data to be processed.
  • sampling the data to be processed using a reservoir sampling algorithm includes: judging whether the cumulative number of processed items is greater than or equal to a second quantity threshold; if not, writing the key in the data to be processed into the sample An array; if so, use a reservoir sampling algorithm to replace a key in the sample array with a key in the data to be processed; use the key in the sample array as the sampling result.
  • a second number threshold (ArraySize) needs to be preset, and the second number threshold is greater than the first number threshold.
  • the algorithm After entering the sampling mode, if the cumulative number of processed items is less than the second quantity threshold, write the key extracted from the data to be processed into the sampling array, and if the cumulative processed number is greater than or equal to the second quantity threshold, use reservoir sampling The algorithm replaces a key in the sampling array with a key extracted from the data to be processed, and finally uses all the keys in the sampling array as the sampling result.
  • all the keys in the counting table are written into the sampling array, so as to obtain the sampling results directly from the sampling array, and all the keys in the sampling array are the sampling results.
  • obtaining the sample data distribution of the data table according to the sampling result of the data to be processed includes: separately calculating the cumulative sampling amount of each sample; wherein the sampling result of the data to be processed includes a plurality of samples; For each data table, divide the data amount of the data to be processed by the accumulated sampling amount of each sample to obtain the weight of each sample in the data table, thereby obtaining the sample data distribution of the data table.
  • the sampling result, fragmented data amount C, and sampling amount S are sent back to the scheduling end (Driver), and the scheduling end assigns a weight to each sample C/S, and aggregate and accumulate the sample weights corresponding to each execution end (the weights of each sample are aggregated and accumulated separately).
  • Driver In the Spark computing engine, the node responsible for scheduling, summarizing, and assigning tasks.
  • Step 302 Divide the data to be processed into a plurality of fragments, and respectively calculate the output data volumes corresponding to the connection operations of the plurality of fragments according to the sample data distribution of the data table.
  • the hash algorithm can be used to divide the data to be processed into multiple fragments. As shown in Figure 1, table A is divided into four fragments A0, A1, A2 and A3 according to the hash algorithm, and table B is divided into two Shards B0 and B1; then, according to the sample data distribution of Table A and Table B calculated in step 301, the output data volume corresponding to the Join of each slice is calculated respectively.
  • Join connection SQL is used to combine the rows from two or more tables to obtain the calculation logic of the matching relationship. According to the semantics, it can be divided into: InnerJoin (inner connection), LeftJoin (left connection), RightJoin (right connection), FullOuterJoin (full connection), etc.
  • calculating the output data volumes corresponding to the connection operations of the plurality of fragments according to the sample data distribution of the data table includes: for any one of the plurality of fragments, from the sampling results Filter out the target samples assigned to the shards; wherein, each of the data tables contains the target samples; calculate the output corresponding to the connection operation of the shards according to the weight of each target sample in the data table The amount of data.
  • the sample data distribution of table A and table B is obtained.
  • the target samples A_X and B_X falling in fragment X are selected from the sampling results of table A and table B, and then According to the weight of each target sample A_X and B_X, calculate the output data volume corresponding to the Join of slice X. It should be pointed out that both Table A and Table B contain target samples, and there may be multiple target samples.
  • calculating the output data volume corresponding to the connection operation of the fragments includes: multiplying the weights of each target sample in each data table and accumulating the multiplication, Thus, the output data volume corresponding to the connection operation of the fragments is obtained.
  • the following formula can be used to calculate the output data volume corresponding to the Join of each shard:
  • w A (x) and w B (x) are the weights of sample x in the data of table A and table B respectively.
  • the weights of each target sample in each data table are multiplied and accumulated, so as to obtain the output data volume corresponding to the join operation of the slice, including: if the type of join operation is an inner join, then The weights of each target sample in each data table are multiplied and accumulated, so as to obtain the output data volume corresponding to the connection operation of the fragment; The weights in are multiplied and accumulated, and then the weights of each target sample in the left join data table are added to obtain the output data volume corresponding to the join operation of the slice; if the join operation type is right join, the The weights of each target sample in each data table are multiplied and accumulated, and then the weights of each target sample in the right connection data table are added to obtain the corresponding output data volume of the connection operation of the fragment; if the connection operation The type of is full connection, the weights of each target sample in each data table are multiplied and accumulated, and then the weights of each target sample in the left join data table and right join data table are added to obtain
  • w A (x) and w B (x) are the weights of sample x in the data of table A and table B respectively.
  • Step 303 According to the output data volume corresponding to the connection operation of the plurality of fragments, select a target fragment from the plurality of fragments, and perform data fragmentation on the target fragment.
  • the shards can be sorted based on the output data volume to filter out the target shard and perform data sharding on the target shard to control the sharding The size of the output data volume.
  • step 303 may include: sorting the output data volumes corresponding to the connection operations of the plurality of fragments, and selecting a median fragment from the plurality of fragments; Screen out target fragments whose output data volume is greater than or equal to the preset multiple of the output data volume corresponding to the connection operation of the median fragmentation; divide each of the target fragments into a plurality of fragments.
  • the fragments are first sorted based on the amount of output data, and then the median fragment of the median is selected from these fragments, and finally the fragments with the output data amount greater than or equal to the median are screened out.
  • the target slice of the output data volume W multiple (preset parameter) corresponding to the connection operation of slices.
  • a hash algorithm may be used to perform data sharding on the target shard, so as to control the output data volume of the shard.
  • the embodiments of the present disclosure sample the data to be processed in the data table in an adaptive manner to obtain the sample data distribution of the data table, and calculate respectively according to the sample data distribution of the data table
  • the connection operation of multiple fragments corresponds to the output data volume, thereby screening out the technical means of target fragments and performing data fragmentation on them, which solves the technical problem that the existing technology cannot handle data expansion.
  • the embodiment of the disclosure obtains the sample data distribution of the data table through adaptive data adoption to obtain the overall distribution of the data, and then calculates the output data volume of each distribution according to the sample data distribution, and accurately screens out the targets that need to be re-sharded. Fragmentation, so as to deal with data expansion in a limited way and avoid data expansion.
  • Fig. 4 is a schematic diagram of a main flow of a data processing method according to a reference embodiment of the present disclosure.
  • the step of sampling the data to be processed in the data table in an adaptive manner may include:
  • Step 401 obtain data to be processed from a data table, so as to obtain a key in the data to be processed.
  • Step 402 whether the cumulative number of processed items is greater than or equal to the first quantity threshold; if yes, execute step 403; if not, execute step 406.
  • Step 403 whether the cumulative number of processed items is greater than or equal to the second quantity threshold; if yes, execute step 404; if not, execute step 405.
  • Step 404 Replace a key in the sampling array with a key in the data to be processed by using a reservoir sampling algorithm, and use the key in the sampling array as a sampling result.
  • Step 405 write the key in the data to be processed into the sampling array, and use the key in the sampling array as the sampling result.
  • Step 406 write the key in the data to be processed into the count table, and use the key in the count table as the sampling result.
  • Fig. 5 is a schematic diagram of the main flow of a data processing method according to another reference embodiment of the present disclosure.
  • the step of sampling the data to be processed in the data table in an adaptive manner may include:
  • Step 501 sampling the data to be processed in the data table in an adaptive manner.
  • step 501 may include: obtaining the data to be processed from the data table, so as to obtain the key in the data to be processed; judging whether the cumulative number of processed items is greater than or equal to the first quantity threshold; The key in the processing data is written into the counting table, and the key in the counting table is used as the sampling result; if so, the reservoir sampling algorithm is used to sample the data to be processed; according to the sampling result of the data to be processed, the The sample data distribution for the data table.
  • a counting mode or a sampling mode can be adaptively adopted, and the sampling logic is executed in a distributed manner at the execution end.
  • each time a piece of data to be processed is obtained from the data table the key is extracted from the piece of data to be processed, and the cumulative number of processed pieces is recorded in a cumulative manner. If the cumulative number of processed pieces is less than the first Quantity threshold, then enter the counting mode, in this mode, write the key into the counting table (CountMap), and use all the keys in the counting table as the sampling results; if the cumulative number of processed items is greater than or equal to the first quantity threshold, that is, counting When the size of the table reaches the preset first quantity threshold, the sampling mode is entered, that is, the data to be processed is sampled using the reservoir sampling algorithm to ensure that each key in the entire stream is sampled and retained with equal probability.
  • Step 502 calculating the cumulative sampling amount of each sample respectively.
  • the sampling result of the data to be processed includes a plurality of samples, and the cumulative sampling amount of each sample is calculated from the sampling results.
  • Step 503 for each data table, divide the data amount of the data to be processed by the cumulative sampling amount of each sample to obtain the weight of each sample in the data table, thereby obtaining the sample of the data table data distribution.
  • each execution terminal (Executor) After the sampling of each execution terminal (Executor) is completed, it will send the sampling result, fragmented data volume C, and sampling volume S back to the scheduling terminal (Driver).
  • the scheduling terminal assigns weight C/S to each sample, and assigns each The sample weights corresponding to the execution end are summarized and accumulated (the weights of each sample are summarized and accumulated separately), so as to obtain the sample data distribution of each data table.
  • Step 504 divide the data to be processed into multiple fragments.
  • the hash algorithm can be used to divide the data to be processed into multiple fragments.
  • Step 505 for any one of the multiple slices, filter out the target samples assigned to the slice from the sampling results.
  • Each of the data tables contains the target samples assigned to the shards.
  • Step 506 according to the weight of each target sample in the data table, calculate the output data volume corresponding to the connection operation of the fragments.
  • the weights of each target sample in each data table are multiplied and accumulated, so as to obtain the output data amount corresponding to the connection operation of the shards.
  • the following formula can be used to calculate the output data volume corresponding to the Join of each shard:
  • w A (x) and w B (x) are the weights of sample x in the data of table A and table B respectively.
  • Step 507 sort the output data volumes corresponding to the connection operations of the plurality of fragments, and select a median fragment with a median value from the plurality of fragments.
  • Step 508 Screen out target shards whose output data volume is greater than or equal to a preset multiple of the output data volume corresponding to the connection operation of the median shard from the plurality of shards.
  • Step 509 dividing each target segment into multiple segments.
  • the hash algorithm can be used to shard the data of the target shard to control the size of the output data of the shard.
  • the input data (Shuffle Read Size) of fragment 5 is 1.8MiB, which is not much different from other fragments; but its output data volume (Output Size) is 75.5MiB, far exceeding other fragments (median is 853.4KiB).
  • This example contains a total of 10 shards, and the running time of the data-inflated shard is 17 seconds, which is far longer than the 2 seconds of other shards. Since the existing technical solution only considers the size of the input data volume, it cannot solve the problem of data expansion.
  • the input data is divided into 72 pieces from the original 10 pieces according to the estimated output data size; After that, the output of all shards is more uniform, and the execution time of all shards does not exceed 2 seconds, so that the total running time is optimized from 17 seconds to 2 seconds, and the efficiency is greatly improved.
  • Fig. 7 is a schematic diagram of the main modules of a data processing device according to an embodiment of the present disclosure.
  • the data processing device 700 includes a sampling module 701, a calculation module 702 and a slice module 703; wherein, the sampling module 701 uses The data to be processed in the data table is sampled in an adaptive manner to obtain the sample data distribution of the data table; the calculation module 702 is used to divide the data to be processed into multiple fragments, according to the The sample data distribution calculates the output data volume corresponding to the connection operation of the plurality of fragments respectively; the fragmentation module 703 is used to obtain the The target shard is screened out, and data sharding is performed on the target shard.
  • sampling module 701 is also used for:
  • the sample data distribution of the data table is obtained according to the sampling result of the data to be processed.
  • sampling module 701 is also used for:
  • sampling module 701 is also used for:
  • the sampling result of the data to be processed includes a plurality of samples
  • each data table For each data table, divide the data amount of the data to be processed by the accumulated sampling amount of each sample to obtain the weight of each sample in the data table, thereby obtaining the sample data distribution of the data table.
  • the calculating module 702 is also used for:
  • each of the data tables contains the target samples
  • the output data volume corresponding to the connection operation of the fragments is calculated.
  • the calculating module 702 is also used for:
  • the weights of each target sample in each data table are multiplied and accumulated, so as to obtain the output data volume corresponding to the connection operation of the shards.
  • the calculating module 702 is also used for:
  • connection operation is an inner connection
  • the weights of each target sample in each data table are multiplied and accumulated, and then the weights of each target sample in the left join data table are added to obtain the split The amount of output data corresponding to the connection operation;
  • connection operation is right connection, multiply the weights of each target sample in each data table and accumulate the multiplication, and then add the weight of each target sample in the right connection data table to obtain the connection of the fragments The amount of output data corresponding to the operation;
  • the weights of each target sample in each data table are multiplied and accumulated, and then the weights of each target sample in the left join data table and right join data table are added to obtain the The amount of output data corresponding to the connection operation of the above shards.
  • the fragmentation module 703 is also used for:
  • the embodiments of the present disclosure sample the data to be processed in the data table in an adaptive manner to obtain the sample data distribution of the data table, and calculate respectively according to the sample data distribution of the data table
  • the connection operation of multiple fragments corresponds to the output data volume, thereby screening out the technical means of target fragments and performing data fragmentation on them, which solves the technical problem that the existing technology cannot handle data expansion.
  • the embodiment of the disclosure obtains the sample data distribution of the data table through adaptive data adoption to obtain the overall distribution of the data, and then calculates the output data volume of each distribution according to the sample data distribution, and accurately screens out the targets that need to be re-sharded. Fragmentation, so as to deal with data expansion in a limited way and avoid data expansion.
  • FIG. 8 shows an exemplary system architecture 800 to which the data processing method or data processing device of the embodiments of the present disclosure can be applied.
  • a system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805.
  • the network 804 is used as a medium for providing communication links between the terminal devices 801 , 802 , 803 and the server 805 .
  • Network 804 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.
  • terminal devices 801, 802, 803 Users can use terminal devices 801, 802, 803 to interact with server 805 through network 804 to receive or send messages and the like.
  • Various communication client applications can be installed on the terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (just examples).
  • the terminal devices 801, 802, and 803 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
  • the server 805 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by users using the terminal devices 801 , 802 , and 803 (just an example).
  • the background management server can analyze and process the received data such as item information query requests, and feed back the processing results to the terminal device.
  • the data processing method provided by the embodiment of the present disclosure is generally executed by the server 805 , and correspondingly, the data processing device is generally disposed in the server 805 .
  • terminal devices, networks and servers in FIG. 8 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.
  • FIG. 9 shows a schematic structural diagram of a computer system 900 suitable for implementing a terminal device according to an embodiment of the present disclosure.
  • the terminal device shown in FIG. 9 is only an example, and should not limit the functions and scope of use of this embodiment of the present disclosure.
  • a computer system 900 includes a central processing unit (CPU) 901 that can be programmed according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random-access memory (RAM) 903 Instead, various appropriate actions and processes are performed.
  • ROM read-only memory
  • RAM random-access memory
  • various programs and data required for the operation of the system 900 are also stored.
  • the CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904.
  • An input/output (I/O) interface 905 is also connected to the bus 904 .
  • the following components are connected to the I/O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 908 including a hard disk, etc. and a communication section 909 including a network interface card such as a LAN card, a modem, or the like.
  • the communication section 909 performs communication processing via a network such as the Internet.
  • a drive 910 is also connected to the I/O interface 905 as needed.
  • a removable medium 911 such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc. is mounted on the drive 910 as necessary so that a computer program read therefrom is installed into the storage section 908 as necessary.
  • embodiments of the present disclosure include a computer program, including a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts.
  • the computer program may be downloaded and installed from a network via communication portion 909 and/or installed from removable media 911 .
  • this computer program is executed by a central processing unit (CPU) 901, the above-described functions defined in the system of the present disclosure are performed.
  • CPU central processing unit
  • the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two.
  • a computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • each block in a flowchart or block diagram may represent a module, program segment, or portion of code that includes one or more logical functions for implementing specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a A combination of dedicated hardware and computer instructions.
  • the modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware.
  • the described modules can also be set in the processor, for example, it can be described as: a processor includes a sampling module, a calculation module and a slice module, wherein the names of these modules do not constitute the its own limitations.
  • the present disclosure also provides a computer-readable medium, which may be included in the device described in the above-mentioned embodiments, or exist independently without being assembled into the device.
  • the above-mentioned computer-readable medium carries one or more programs.
  • the device implements the following method: sampling the data to be processed in the data table in an adaptive manner, and obtaining the The sample data distribution of the data table; the data to be processed is divided into a plurality of fragments, and the output data volume corresponding to the connection operation of the plurality of fragments is calculated respectively according to the sample data distribution of the data table; according to the The output data volume corresponding to the connection operation of multiple shards, the target shard is selected from the multiple shards, and data sharding is performed on the target shard.
  • the sample data distribution of the data table is obtained, and the connection operations of multiple fragments are respectively calculated according to the sample data distribution of the data table Corresponding output data volume, so as to filter out the technical means of target fragmentation and data fragmentation, so it overcomes the technical problem of being unable to deal with data expansion in the prior art.
  • the embodiment of the disclosure obtains the sample data distribution of the data table through adaptive data adoption to obtain the overall distribution of the data, and then calculates the output data volume of each distribution according to the sample data distribution, and accurately screens out the targets that need to be re-sharded. Fragmentation, so as to deal with data expansion in a limited way and avoid data expansion.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Complex Calculations (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本公开公开了一种数据处理方法和装置，涉及大数据技术领域。该方法的一具体实施方式包括：采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。该实施方式能够解决无法处理数据膨胀的技术问题。

Description

一种数据处理方法和装置
相关申请的交叉引用
本申请要求享有2021年7月30日提交的题为“一种数据处理方法和装置”的中国专利申请No.202110871021.0的中国专利申请的优先权，在此全文引用上述中国专利申请公开的内容以作为本申请的一部分或全部。
技术领域
本公开涉及大数据技术领域，尤其涉及一种数据处理方法和装置。
背景技术
数据倾斜一直是大数据领域里面的一个主要痛点，它是指在分布式的数据并行处理过程中，由于数据分片不均匀，导致大量数据集中到一个或少数计算节点上，导致少数任务的处理速度远低于平均速度，拖慢整个计算环节。其中一种常见的数据倾斜情况被称为数据膨胀，其表现为：输入分片大小并没有显著大于平均水平，但其对应数据只含有少数不同的key，导致Join连接执行后，对应任务输出的数据量远大于其他任务。
在实现本公开过程中，发明人发现现有技术中至少存在如下问题：
现有计算引擎基于输入分片的大小进行划分，不能处理数据膨胀的场景。而且，在实际生产环境中，情况更为复杂，多种数据倾斜场景(包含数据膨胀)往往一同出现，而现有技术方案均不能处理包含数据膨胀的情况。
发明内容
有鉴于此，本发明实施例提供一种数据处理方法和装置，以解决无法处理数据膨胀的技术问题。
为实现上述目的，根据本公开实施例的一个方面，提供了一种数据处理方法，包括：
采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；
将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；
根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
可选地，采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布，包括：
从数据表中获取待处理数据，以获取所述待处理数据中的key；
判断累计处理条数是否大于等于第一数量阈值；若否，则将所述待处理数据中的key写入计数表，将所述计数表中的key作为采样结果；若是，则采用蓄水池采样算法对所述待处理数据进行采样；
根据所述待处理数据的采样结果得到所述数据表的样本数据分布。
可选地，采用蓄水池采样算法对所述待处理数据进行采样，包括：
判断累计处理条数是否大于等于第二数量阈值；若否，则将所述待处理数据中的key写入采样数组；若是，则采用蓄水池采样算法将所述采样数组中的一个key置换为所述待处理数据中的key；
将所述采样数组中的key作为采样结果。
可选地，根据所述待处理数据的采样结果得到所述数据表的样本数据分布，包括：
分别计算每个样本的累计采样量；其中，所述待处理数据的采样结果包括多个样本；
对于每个数据表，将所述待处理数据的数据量除以所述每个样本的累计采样量，得到所述数据表中每个样本的权重，从而得到所述数据表的样本数据分布。
可选地，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量，包括：
对于所述多个分片中的任意一个分片，从采样结果中筛选出被分配至所述分片的目标样本；其中，每个所述数据表包含所述目标样本；
根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量。
可选地，根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量，包括：
将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量。
可选地，将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量，包括：
若连接操作的类型为内连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量；
若连接操作的类型为左连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；
若连接操作的类型为右连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；
若连接操作的类型为全连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表和右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量。
可选地，根据所述多个分片的连接操作对应的输出数据量，从所 述多个分片中筛选出目标分片，对所述目标分片进行数据分片，包括：
对所述多个分片的连接操作对应的输出数据量进行排序，从所述多个分片中筛选出中位数的中位分片；
从所述多个分片中筛选出输出数据量大于等于所述中位分片的连接操作对应的输出数据量预设倍数的目标分片；
将每个所述目标分片划分为多个分片。
另外，根据本公开实施例的另一个方面，提供了一种数据处理装置，包括：
采样模块，用于采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；
计算模块，用于将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；
分片模块，用于根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
根据本公开实施例的另一个方面，还提供了一种电子设备，包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序，
当所述一个或多个程序被所述一个或多个处理器执行时，所述一个或多个处理器实现上述任一实施例所述的方法。
根据本公开实施例的另一个方面，还提供了一种计算机可读介质，其上存储有计算机程序，所述程序被处理器执行时实现上述任一实施例所述的方法。
上述的非惯用的可选方式所具有的进一步效果将在下文中结合具体实施方式加以说明。
附图说明
附图用于更好地理解本公开，不构成对本公开的不当限定。其中：
图1是Spark AQE的基本原理示意图；
图2是Spark AQE中的动态优化策略基本原理示意图；
图3是根据本公开实施例的数据处理方法的主要流程的示意图；
图4是根据本公开一个可参考实施例的数据处理方法的主要流程的示意图；
图5是根据本公开另一个可参考实施例的数据处理方法的主要流程的示意图；
图6a和6b是采用本公开实施例的数据处理方法前后的对比示意图；
图7是根据本公开实施例的数据处理装置的主要模块的示意图；
图8是本公开实施例可以应用于其中的示例性系统架构图；
图9是适于用来实现本公开实施例的终端设备或服务器的计算机系统的结构示意图。
具体实施方式
以下结合附图对本公开的示范性实施例做出说明，其中包括本公开实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本公开的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。
以Spark、Hive为首的分布式计算引擎，都针对数据倾斜提供了一定的处理能力。以Spark为例，其自从3.0版本开始，在运行时中引入了自适应查询执行优化(AQE：Adaptive Query Execution)功能，可以基于已完成环节(Stage)的统计信息，来自适应的、动态的调整和优化后续计算环节。
AQE中一个很重要的策略为：动态优化倾斜的Join连接。如图1所示，以表A连接(Join)表B为例介绍其基本原理，假定表A的分 片A0数据量明显大于其他分片，在执行哈希数据分发(hash shuffle)后，处理A0、B0的计算任务的数据量明显大于其他分片，导致整个计算环节被拖慢。shuffle：分布式计算中，数据在网络上分发的过程；一般会基于Hash值进行shuffle，对应同一Hash值的数据会被放在同一分片中。
Spark AQE中的动态优化策略，会统计每个分片的大小，在发现A0显著的大于各个分片的中位数时，会触发对A0的划分，同时为了保证计算的顺利进行，会对B0进行复制，如图2所示。经过这样的处理后，虽然总任务数增多(由4个变成了5个)，但每个任务的数据更加均匀，计算耗时差不多，为整个查询计算带来了更好的性能。
实际生产环境中，存在多种多样的数据倾斜场景。其中一种常见的数据倾斜情况被称为数据膨胀，其表现为：输入分片大小并没有显著大于平均水平，但其对应数据只含有少数不同的key，导致Join连接执行后，对应任务输出的数据量远大于其他任务。现有计算引擎基于输入分片的大小进行划分，不能处理这种数据膨胀的场景。例如，输入的分片A3和B3的大小不存在倾斜情况，但是A3和B3中都存在一个频繁key＝Hot_Sku。该key在A3侧和B3侧的频次分别为N_A和N_B，则在Join连接后的结果中，该key的频次为N_A*N_B。当N_A和N_B较输入分片尺寸正常，但N_A*N_B较输出分片尺寸较大时，则会发生数据膨胀。
为了解决现有技术中存在的数据膨胀的问题，本公开实施例针对在分布式环境下，通过自适应的数据统计和分析，对Join连接的输出数据量进行预估，并据此对数据分片进行划分，以处理数据膨胀。
图3是根据本公开实施例的数据处理方法的主要流程的示意图。作为本公开的一个实施例，如图3所示，所述数据处理方法可以包括：
步骤301，采用自适应方式对数据表中的待处理数据进行采样，得 到所述数据表的样本数据分布。
首先采用自适应方式对数据表中的待处理数据进行采样，采样逻辑以分布式的形式执行在执行端(Executor)。通过自适应数据采样可以从数据表中采集到样本数据，从而计算出数据表的样本数据分布。Executor：Spark计算引擎中，实际执行具体计算任务的节点。
可选地，步骤301可以包括：从数据表中获取待处理数据，以获取所述待处理数据中的key；判断累计处理条数是否大于等于第一数量阈值；若否，则将所述待处理数据中的key写入计数表，将所述计数表中的key作为采样结果；若是，则采用蓄水池采样算法对所述待处理数据进行采样；根据所述待处理数据的采样结果得到所述数据表的样本数据分布。可以预先设定第一数量阈值(MapSize)，之后以流的形式进行增量更新。每次从数据表中获取一条待处理数据时，从该条待处理数据中提取出key，并以累计方式记录累计处理条数，如果累计处理条数小于第一数量阈值，则进入计数模式(Counting)，在该模式下，将该key写入计数表(CountMap)，并将计数表中的所有key作为采样结果；如果累计处理条数大于等于第一数量阈值，即计数表的大小达到预设的第一数量阈值，则进入采样模式(Sampling)，即采用蓄水池采样算法对所述待处理数据进行采样，保证整个流中的每个key以等概率被采样保留。蓄水池抽样(Reservoir Sampling)：一种流式采样算法，能够在o(n)时间内对n个数据进行等概率随机抽取。
需要说明的是，待处理数据中包含key及其对应的value，本公开实施例只需要key，因此只需要从待处理数据中提取出key。
可选地，采用蓄水池采样算法对所述待处理数据进行采样，包括：判断累计处理条数是否大于等于第二数量阈值；若否，则将所述待处理数据中的key写入采样数组；若是，则采用蓄水池采样算法将所述采样数组中的一个key置换为所述待处理数据中的key；将所述采样数组 中的key作为采样结果。在采样模式下，需要预先设定第二数量阈值(ArraySize)，第二数量阈值大于第一数量阈值。进入采样模式后，如果累计处理条数小于第二数量阈值，则将从待处理数据中提取出的key写入采样数组，如果累计处理条数大于等于第二数量阈值，则采用蓄水池采样算法将所述采样数组中的一个key置换为从待处理数据中提取出的key，最后将采样数组中的所有key作为采样结果。
在本公开的实施例中，当进入采样模式时，将计数表中的所有key全部写入采样数组，以便于直接从采样数组中获取采样结果，采样数组中的所有key即为采样结果。首先尝试进行精确采样，在累计处理条数大于等于第一数量阈值的情况下，会回退到蓄水池采样。
可选地，根据所述待处理数据的采样结果得到所述数据表的样本数据分布，包括：分别计算每个样本的累计采样量；其中，所述待处理数据的采样结果包括多个样本；对于每个数据表，将所述待处理数据的数据量除以所述每个样本的累计采样量，得到所述数据表中每个样本的权重，从而得到所述数据表的样本数据分布。在本公开的实施例中，各个执行端(Executor)的采样执行完毕后，会将采样结果、分片数据量C、采样量S发送回调度端(Driver)，调度端为每个样本赋予权重C/S，并将每个执行端对应的样本权重进行汇总累加处理(每个样本的权重分别汇总累加)。Driver：Spark计算引擎中，负责调度、汇总、分配任务的节点。
如图1所示，以表A为例，将每个map任务中各个样本的权重分别汇总累加，从而得到表A的样本数据分布。同理，也可以得到表B的样本数据分布。
步骤302，将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量。
可以采用哈希算法将待处理数据划分为多个分片，如图1所示，根据哈希算法将表A划分为四个分片A0、A1、A2和A3，将表B划分为两个分片B0和B1；然后根据步骤301计算出的表A和表B的样本数据分布，分别计算出每个分片的Join对应的输出数据量。
Join连接：SQL中用于把来自两个或多个表的行结合起来，获取匹配关系的计算逻辑。按照语义可以分为：InnerJoin(内连接)、LeftJoin(左连接)、RightJoin(右连接)、FullOuterJoin(全连接)等。
可选地，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量，包括：对于所述多个分片中的任意一个分片，从采样结果中筛选出被分配至所述分片的目标样本；其中，每个所述数据表包含所述目标样本；根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量。通过自适应采样，获得了表A和表B的样本数据分布，对于某个分片X，从表A和表B的采样结果中筛选出落在分片X中的目标样本A_X和B_X，然后根据各个目标样本A_X和B_X的权重，计算分片X的Join对应的输出数据量。需要指出的是，表A和表B中均包含目标样本，目标样本可以是多个。
可选地，根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量，包括：将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量。具体地，可以采用如下公式计算各个分片的Join对应的输出数据量：
Figure PCTCN2022087297-appb-000001
其中，w A(x)和w B(x)分别为表A和表B的数据中，样本x的权重。
可选地，将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量，包括：若连接 操作的类型为内连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量；若连接操作的类型为左连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；若连接操作的类型为右连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；若连接操作的类型为全连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表和右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量。为了准确地计算分片的Join对应的输出数据量，不同类型的Join，采用不同的计算公式：
如果JoinType为内连接(InnerJoin)：
Estimated Size(X)＝∑ x∈A_X∩B_Xw A(x)*w B(x)；
如果JoinType为左连接(LeftJoin)：
Estimated Size(X)＝∑ x∈A_X∩B_Xw A(x)*w B(x)+∑ x∈A_Xw A(x)；
如果JoinType为右连接(RightJoin)：
Estimated Size(X)＝∑ x∈A_X∩B_Xw A(x)*w B(x)+∑ x∈B_Xw B(x)；
如果JoinType为全连接(FullOuterJoin)：
Estimated Size(X)＝∑ x∈A_X∩B_Xw A(x)*w B(x)+∑ x∈A_Xw A(x)+∑ x∈B_Xw B(x)；
其中，w A(x)和w B(x)分别为表A和表B的数据中，样本x的权重。
步骤303，根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
由于计算出了每个分片的Join对应的输出数据量，因此可以基于输出数据量对各个分片进行排序，从而筛选出目标分片，并对目标分片进行数据分片，以控制分片的输出数据量的大小。
可选地，步骤303可以包括：对所述多个分片的连接操作对应的输出数据量进行排序，从所述多个分片中筛选出中位数的中位分片；从所述多个分片中筛选出输出数据量大于等于所述中位分片的连接操作对应的输出数据量预设倍数的目标分片；将每个所述目标分片划分为多个分片。在本公开的实施例中，先基于输出数据量对各个分片进行排序，然后从这些分片中筛选出中位数的中位分片，最后筛选出输出数据量大于等于所述中位分片的连接操作对应的输出数据量W倍数(预设参数)的目标分片。
在本公开的实施例中，筛选出目标分片之后，可以采用哈希算法对目标分片进行数据分片，以控制分片的输出数据量的大小。
根据上面所述的各种实施例，可以看出本公开实施例通过采用自适应方式对数据表中的待处理数据进行采样，得到数据表的样本数据分布，根据数据表的样本数据分布分别计算多个分片的连接操作对应的输出数据量，从而筛选出目标分片并对其进行数据分片的技术手段，解决了现有技术中无法处理数据膨胀的技术问题。本公开实施例通过自适应数据采用得到数据表的样本数据分布，以获得数据的整体分布，然后根据样本数据分布计算出各个分配的输出数据量，准确地筛选出需要再次进行数据分片的目标分片，从而有限地处理数据膨胀，避免发生数据膨胀。
图4是根据本公开一个可参考实施例的数据处理方法的主要流程的示意图。作为本公开的又一个实施例，如图4所示，采用自适应方式对数据表中的待处理数据进行采样的步骤可以包括：
步骤401，从数据表中获取待处理数据，以获取所述待处理数据中的key。
步骤402，累计处理条数是否大于等于第一数量阈值；若是，则执行步骤403；若否，则执行步骤406。
步骤403，累计处理条数是否大于等于第二数量阈值；若是，则执行步骤404；若否，则执行步骤405。
步骤404，采用蓄水池采样算法将所述采样数组中的一个key置换为所述待处理数据中的key，将所述采样数组中的key作为采样结果。
步骤405，将所述待处理数据中的key写入采样数组，将所述采样数组中的key作为采样结果。
步骤406，将所述待处理数据中的key写入计数表，将所述计数表中的key作为采样结果。
另外，在本公开一个可参考实施例中数据处理方法的具体实施内容，在上面所述数据处理方法中已经详细说明了，故在此重复内容不再说明。
图5是根据本公开另一个可参考实施例的数据处理方法的主要流程的示意图。作为本公开的另一个实施例，如图5所示，采用自适应方式对数据表中的待处理数据进行采样的步骤可以包括：
步骤501，采用自适应方式对数据表中的待处理数据进行采样。
可选地，步骤501可以包括：从数据表中获取待处理数据，以获取所述待处理数据中的key；判断累计处理条数是否大于等于第一数量阈值；若否，则将所述待处理数据中的key写入计数表，将所述计数表中的key作为采样结果；若是，则采用蓄水池采样算法对所述待处理数据进行采样；根据所述待处理数据的采样结果得到所述数据表的样本数据分布。通过预先设定第一数量阈值，可以自适应地采用计数模式或者采用采样模式，采样逻辑以分布式的形式执行在执行端。
在本公开的实施例中，每次从数据表中获取一条待处理数据时，从该条待处理数据中提取出key，并以累计方式记录累计处理条数，如果累计处理条数小于第一数量阈值，则进入计数模式，在该模式下，将该key写入计数表(CountMap)，并将计数表中的所有key作为采样结果；如果累计处理条数大于等于第一数量阈值，即计数表的大小达到预设的第一数量阈值，则进入采样模式，即采用蓄水池采样算法对所述待处理数据进行采样，保证整个流中的每个key以等概率被采样保留。
步骤502，分别计算每个样本的累计采样量。
其中，所述待处理数据的采样结果包括多个样本，从采用结果中统计出每个样本的累计采样量。
步骤503，对于每个数据表，将所述待处理数据的数据量除以所述每个样本的累计采样量，得到所述数据表中每个样本的权重，从而得到所述数据表的样本数据分布。
各个执行端(Executor)的采样执行完毕后，会将采样结果、分片数据量C、采样量S发送回调度端(Driver)，调度端为每个样本赋予权重C/S，并将每个执行端对应的样本权重进行汇总累加处理(每个样本的权重分别汇总累加)，从而得到每个数据表的样本数据分布。
步骤504，将所述待处理数据划分为多个分片。
可以采用哈希算法将待处理数据划分为多个分片。
步骤505，对于所述多个分片中的任意一个分片，从采样结果中筛选出被分配至所述分片的目标样本。
每个所述数据表包含所述目标样本，目标样本被分配至所述分片。
步骤506，根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量。
具体地，将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量。具体地，可以采用如下公式计算各个分片的Join对应的输出数据量：
Figure PCTCN2022087297-appb-000002
其中，w A(x)和w B(x)分别为表A和表B的数据中，样本x的权重。
为了准确地计算分片的Join对应的输出数据量，不同类型的Join，采用不同的计算公式，不再赘述。
步骤507，对所述多个分片的连接操作对应的输出数据量进行排序，从所述多个分片中筛选出中位数的中位分片。
步骤508，从所述多个分片中筛选出输出数据量大于等于所述中位分片的连接操作对应的输出数据量预设倍数的目标分片。
步骤509，将每个所述目标分片划分为多个分片。
筛选出目标分片之后，可以采用哈希算法对目标分片进行数据分片，以控制分片的输出数据量的大小。
另外，在本公开另一个可参考实施例中数据处理方法的具体实施内容，在上面所述数据处理方法中已经详细说明了，故在此重复内容不再说明。
在实践中，本公开实施例提供的方法可以有效地处理数据膨胀的问题，以下为例：
如图6a所示，分片5的输入数据(Shuffle Read Size)为1.8MiB，与其他分差别不大；但是其输出数据量(Output Size)为75.5MiB，远超其他分片(中位数为853.4KiB)。本例中共包含10个分片，数据膨胀的分片运行时长17秒，远超其他分片的运行时长2秒。由于现有技术方案仅仅考虑输入数据量的大小，故不能解决数据膨胀问题。
如图6b所示，采用本公开实施例提供的方法后，依据预估的输出数据量大小，对输入数据进行切分，从原有的10个分片，切为72个分片；切分之后，所有分片的输出更加均匀，且所有分片的执行时间都不超过2秒，使得总运行时长从17秒优化为2秒，效率获得极大提升。
图7是根据本公开实施例的数据处理装置的主要模块的示意图，如图7所示，所述数据处理装置700包括采样模块701、计算模块702和分片模块703；其中，采样模块701用于采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；计算模块702用于将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；分片模块703用于根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
可选地，所述采样模块701还用于：
从数据表中获取待处理数据，以获取所述待处理数据中的key；
判断累计处理条数是否大于等于第一数量阈值；若否，则将所述待处理数据中的key写入计数表，将所述计数表中的key作为采样结果；若是，则采用蓄水池采样算法对所述待处理数据进行采样；
根据所述待处理数据的采样结果得到所述数据表的样本数据分布。
可选地，所述采样模块701还用于：
判断累计处理条数是否大于等于第二数量阈值；若否，则将所述 待处理数据中的key写入采样数组；若是，则采用蓄水池采样算法将所述采样数组中的一个key置换为所述待处理数据中的key；
将所述采样数组中的key作为采样结果。
可选地，所述采样模块701还用于：
分别计算每个样本的累计采样量；其中，所述待处理数据的采样结果包括多个样本；
对于每个数据表，将所述待处理数据的数据量除以所述每个样本的累计采样量，得到所述数据表中每个样本的权重，从而得到所述数据表的样本数据分布。
可选地，所述计算模块702还用于：
对于所述多个分片中的任意一个分片，从采样结果中筛选出被分配至所述分片的目标样本；其中，每个所述数据表包含所述目标样本；
根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量。
可选地，所述计算模块702还用于：
将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量。
可选地，所述计算模块702还用于：
若连接操作的类型为内连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量；
若连接操作的类型为左连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；
若连接操作的类型为右连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在右连接数据表中的 权重，从而得到所述分片的连接操作对应的输出数据量；
若连接操作的类型为全连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表和右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量。
可选地，所述分片模块703还用于：
对所述多个分片的连接操作对应的输出数据量进行排序，从所述多个分片中筛选出中位数的中位分片；
从所述多个分片中筛选出输出数据量大于等于所述中位分片的连接操作对应的输出数据量预设倍数的目标分片；
将每个所述目标分片划分为多个分片。
根据上面所述的各种实施例，可以看出本公开实施例通过采用自适应方式对数据表中的待处理数据进行采样，得到数据表的样本数据分布，根据数据表的样本数据分布分别计算多个分片的连接操作对应的输出数据量，从而筛选出目标分片并对其进行数据分片的技术手段，解决了现有技术中无法处理数据膨胀的技术问题。本公开实施例通过自适应数据采用得到数据表的样本数据分布，以获得数据的整体分布，然后根据样本数据分布计算出各个分配的输出数据量，准确地筛选出需要再次进行数据分片的目标分片，从而有限地处理数据膨胀，避免发生数据膨胀。
需要说明的是，在本公开所述数据处理装置的具体实施内容，在上面所述数据处理方法中已经详细说明了，故在此重复内容不再说明。
图8示出了可以应用本公开实施例的数据处理方法或数据处理装置的示例性系统架构800。
如图8所示，系统架构800可以包括终端设备801、802、803，网 络804和服务器805。网络804用以在终端设备801、802、803和服务器805之间提供通信链路的介质。网络804可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。
用户可以使用终端设备801、802、803通过网络804与服务器805交互，以接收或发送消息等。终端设备801、802、803上可以安装有各种通讯客户端应用，例如购物类应用、网页浏览器应用、搜索类应用、即时通信工具、邮箱客户端、社交平台软件等(仅为示例)。
终端设备801、802、803可以是具有显示屏并且支持网页浏览的各种电子设备，包括但不限于智能手机、平板电脑、膝上型便携计算机和台式计算机等等。
服务器805可以是提供各种服务的服务器，例如对用户利用终端设备801、802、803所浏览的购物类网站提供支持的后台管理服务器(仅为示例)。后台管理服务器可以对接收到的物品信息查询请求等数据进行分析等处理，并将处理结果反馈给终端设备。
需要说明的是，本公开实施例所提供的数据处理方法一般由服务器805执行，相应地，所述数据处理装置一般设置在服务器805中。
应该理解，图8中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。
下面参考图9，其示出了适于用来实现本公开实施例的终端设备的计算机系统900的结构示意图。图9示出的终端设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图9所示，计算机系统900包括中央处理单元(CPU)901，其可以根据存储在只读存储器(ROM)902中的程序或者从存储部分908 加载到随机访问存储器(RAM)903中的程序而执行各种适当的动作和处理。在RAM 903中，还存储有系统900操作所需的各种程序和数据。CPU 901、ROM 902以及RAM903通过总线904彼此相连。输入/输出(I/O)接口905也连接至总线904。
以下部件连接至I/O接口905：包括键盘、鼠标等的输入部分906；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分907；包括硬盘等的存储部分908；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分909。通信部分909经由诸如因特网的网络执行通信处理。驱动器910也根据需要连接至I/O接口905。可拆卸介质911，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器910上，以便于从其上读出的计算机程序根据需要被安装入存储部分908。
特别地，根据本公开公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开公开的实施例包括一种计算机程序，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分909从网络上被下载和安装，和/或从可拆卸介质911被安装。在该计算机程序被中央处理单元(CPU)901执行时，执行本公开的系统中限定的上述功能。
需要说明的是，本公开所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存 储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块可以通过软件的方式实现，也可以通过硬件的方式来实现。所描述的模块也可以设置在处理器中，例如，可以描述为：一种处理器包括采样模块、计算模块和分片模块，其中，这些模块的名称在某种情况下并不构成对该模块本身的限定。
作为另一方面，本公开还提供了一种计算机可读介质，该计算机 可读介质可以是上述实施例中描述的设备中所包含的；也可以是单独存在，而未装配入该设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被一个该设备执行时，该设备实现如下方法：采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
根据本公开实施例的技术方案，因为通过采用自适应方式对数据表中的待处理数据进行采样，得到数据表的样本数据分布，根据数据表的样本数据分布分别计算多个分片的连接操作对应的输出数据量，从而筛选出目标分片并对其进行数据分片的技术手段，所以克服了现有技术中无法处理数据膨胀的技术问题。本公开实施例通过自适应数据采用得到数据表的样本数据分布，以获得数据的整体分布，然后根据样本数据分布计算出各个分配的输出数据量，准确地筛选出需要再次进行数据分片的目标分片，从而有限地处理数据膨胀，避免发生数据膨胀。
上述具体实施方式，并不构成对本公开保护范围的限制。本领域技术人员应该明白的是，取决于设计要求和其他因素，可以发生各种各样的修改、组合、子组合和替代。任何在本公开的精神和原则之内所作的修改、等同替换和改进等，均应包含在本公开保护范围之内。

Claims (11)

  1. 一种数据处理方法，包括：
    采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；
    将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；
    根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
  2. 根据权利要求1所述的方法，其中，采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布，包括：
    从数据表中获取待处理数据，以获取所述待处理数据中的key；
    判断累计处理条数是否大于等于第一数量阈值；若否，则将所述待处理数据中的key写入计数表，将所述计数表中的key作为采样结果；若是，则采用蓄水池采样算法对所述待处理数据进行采样；
    根据所述待处理数据的采样结果得到所述数据表的样本数据分布。
  3. 根据权利要求2所述的方法，其中，采用蓄水池采样算法对所述待处理数据进行采样，包括：
    判断累计处理条数是否大于等于第二数量阈值；若否，则将所述待处理数据中的key写入采样数组；若是，则采用蓄水池采样算法将所述采样数组中的一个key置换为所述待处理数据中的key；
    将所述采样数组中的key作为采样结果。
  4. 根据权利要求2所述的方法，其中，根据所述待处理数据的采样结果得到所述数据表的样本数据分布，包括：
    分别计算每个样本的累计采样量；其中，所述待处理数据的采样结果包括多个样本；
    对于每个数据表，将所述待处理数据的数据量除以所述每个样本 的累计采样量，得到所述数据表中每个样本的权重，从而得到所述数据表的样本数据分布。
  5. 根据权利要求4所述的方法，其中，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量，包括：
    对于所述多个分片中的任意一个分片，从采样结果中筛选出被分配至所述分片的目标样本；其中，每个所述数据表包含所述目标样本；
    根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量。
  6. 根据权利要求5所述的方法，其中，根据所述数据表中各个目标样本的权重，计算所述分片的连接操作对应的输出数据量，包括：
    将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量。
  7. 根据权利要求6所述的方法，其中，将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量，包括：
    若连接操作的类型为内连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，从而得到所述分片的连接操作对应的输出数据量；
    若连接操作的类型为左连接，则将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；
    若连接操作的类型为右连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据量；
    若连接操作的类型为全连接，将各个目标样本在各个数据表中的权重相乘并将乘积累加，然后加上各个目标样本在左连接数据表和右连接数据表中的权重，从而得到所述分片的连接操作对应的输出数据 量。
  8. 根据权利要求1所述的方法，其中，根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片，包括：
    对所述多个分片的连接操作对应的输出数据量进行排序，从所述多个分片中筛选出中位数的中位分片；
    从所述多个分片中筛选出输出数据量大于等于所述中位分片的连接操作对应的输出数据量预设倍数的目标分片；
    将每个所述目标分片划分为多个分片。
  9. 一种数据处理装置，包括：
    采样模块，用于采用自适应方式对数据表中的待处理数据进行采样，得到所述数据表的样本数据分布；
    计算模块，用于将所述待处理数据划分为多个分片，根据所述数据表的样本数据分布分别计算所述多个分片的连接操作对应的输出数据量；
    分片模块，用于根据所述多个分片的连接操作对应的输出数据量，从所述多个分片中筛选出目标分片，对所述目标分片进行数据分片。
  10. 一种电子设备，包括：
    一个或多个处理器；
    存储装置，用于存储一个或多个程序，
    当所述一个或多个程序被所述一个或多个处理器执行时，所述一个或多个处理器实现如权利要求1-8中任一所述的方法。
  11. 一种计算机可读介质，其上存储有计算机程序，所述程序被处理器执行时实现如权利要求1-8中任一所述的方法。
PCT/CN2022/087297 2021-07-30 2022-04-18 一种数据处理方法和装置 Ceased WO2023005264A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110871021.0A CN113590322B (zh) 2021-07-30 2021-07-30 一种数据处理方法和装置
CN202110871021.0 2021-07-30

Publications (1)

Publication Number Publication Date
WO2023005264A1 true WO2023005264A1 (zh) 2023-02-02

Family

ID=78252498

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/087297 Ceased WO2023005264A1 (zh) 2021-07-30 2022-04-18 一种数据处理方法和装置

Country Status (2)

Country Link
CN (1) CN113590322B (zh)
WO (1) WO2023005264A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116737769A (zh) * 2023-06-15 2023-09-12 中国平安财产保险股份有限公司 一种数据倾斜点查找方法、装置、计算机设备及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113590322B (zh) * 2021-07-30 2025-07-15 北京沃东天骏信息技术有限公司 一种数据处理方法和装置
CN114756405A (zh) * 2022-03-29 2022-07-15 浪潮云信息技术股份公司 一种基于分片技术的表格全量采集方法及相关装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150331724A1 (en) * 2014-05-16 2015-11-19 Teradata Us, Inc. Workload balancing to handle skews for big data analytics
CN107145394A (zh) * 2017-04-28 2017-09-08 中国人民解放军国防科学技术大学 一种针对数据倾斜的均衡负载处理方法及装置
CN112000467A (zh) * 2020-07-24 2020-11-27 广东技术师范大学 一种数据倾斜处理方法、装置、终端设备及存储介质
CN112905596A (zh) * 2021-03-05 2021-06-04 北京中经惠众科技有限公司 数据处理的方法、装置、计算机设备以及存储介质
CN113590322A (zh) * 2021-07-30 2021-11-02 北京沃东天骏信息技术有限公司 一种数据处理方法和装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102799486B (zh) * 2012-06-18 2014-11-26 北京大学 一种MapReduce系统中的数据采样和划分方法
CN107066612A (zh) * 2017-05-05 2017-08-18 郑州云海信息技术有限公司 一种基于SparkJoin操作的自适应数据倾斜调整方法
CN108804626B (zh) * 2018-05-31 2019-11-22 华中科技大学 一种基于分布感知的二元等值连接倾斜优化方法和系统
CN109376012A (zh) * 2018-10-10 2019-02-22 电子科技大学 一种针对异构环境的基于Spark的自适应任务调度方法
CN112711588B (zh) * 2019-10-25 2024-04-05 北京沃东天骏信息技术有限公司 多表连接的方法和装置
CN113032446B (zh) * 2019-12-24 2024-07-09 阿里巴巴集团控股有限公司 一种分布式查询系统的数据处理方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150331724A1 (en) * 2014-05-16 2015-11-19 Teradata Us, Inc. Workload balancing to handle skews for big data analytics
CN107145394A (zh) * 2017-04-28 2017-09-08 中国人民解放军国防科学技术大学 一种针对数据倾斜的均衡负载处理方法及装置
CN112000467A (zh) * 2020-07-24 2020-11-27 广东技术师范大学 一种数据倾斜处理方法、装置、终端设备及存储介质
CN112905596A (zh) * 2021-03-05 2021-06-04 北京中经惠众科技有限公司 数据处理的方法、装置、计算机设备以及存储介质
CN113590322A (zh) * 2021-07-30 2021-11-02 北京沃东天骏信息技术有限公司 一种数据处理方法和装置

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
HU ZHONG-KUI, QU BO;HUANG BIN;LI WEN-YANG: "A Load Balanced Equi-Join Algorithm Based on Virtual Processor Range Partition", XIANDAI JISUANJI (ZHUANYE BAN)/ MODERN COMPUTER (PROFESSIONAL EDITION), XIANDAI JISUANJI ZAZHISHE, CHINA, no. 3, 1 January 2016 (2016-01-01), China , pages 3 - 7, XP093030120, ISSN: 1007-1423, DOI: 10.3969/j.issn.1007-1423.2016.03.001 *
HUANG ZICHUN: "Research on Partition Loading Balance Based on Spark Data Skew", CHINA DOCTORAL DISSERTATIONS/MASTER'S THESES FULL-TEXT DATABASE (MONTHLY), 15 February 2021 (2021-02-15), pages 1 - 61, XP093030131 *
ZHANG ZHANFENG, WANG WEN-LI;GENG SHAN-SHAN;JIA ZHI-TING: "Research on Data Skew of Spark", CHINESE MASTER’S THESES FULL-TEXT DATABASE, AGRICULTURAL SCIENCE AND TECHNOLOGY, 1 January 2020 (2020-01-01), pages 1 - 7, XP093030125, ISSN: 1001-9383, DOI: 10.16191/j.cnki.hbkx.2020.01.001 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116737769A (zh) * 2023-06-15 2023-09-12 中国平安财产保险股份有限公司 一种数据倾斜点查找方法、装置、计算机设备及存储介质

Also Published As

Publication number Publication date
CN113590322A (zh) 2021-11-02
CN113590322B (zh) 2025-07-15

Similar Documents

Publication Publication Date Title
US10922316B2 (en) Using computing resources to perform database queries according to a dynamically determined query size
CN111786895A (zh) 动态全局限流的方法和装置
CN112925859B (zh) 数据存储方法和装置
CN103748579B (zh) 在映射化简框架中处理数据
WO2023005264A1 (zh) 一种数据处理方法和装置
US20080168179A1 (en) Method and apparatus for providing load diffusion in data stream correlations
CN113312355B (zh) 一种数据管理的方法和装置
CN108536650A (zh) 生成梯度提升树模型的方法和装置
Liu et al. SP-Partitioner: A novel partition method to handle intermediate data skew in spark streaming
EP3356951B1 (en) Managing a database of patterns used to identify subsequences in logs
US10249070B2 (en) Dynamic interaction graphs with probabilistic edge decay
CN107885783B (zh) 获取搜索词高相关分类的方法和装置
CN110019087A (zh) 数据处理方法及其系统
US11200231B2 (en) Remote query optimization in multi data sources
CN107145394B (zh) 一种针对数据倾斜的均衡负载处理方法及装置
CN111723089A (zh) 一种基于列式存储格式处理数据的方法和装置
CN113760521A (zh) 一种虚拟资源的分配方法和装置
CN113392863A (zh) 一种机器学习训练数据集的获取方法、获取装置及终端
US10609206B1 (en) Auto-repairing mobile communication device data streaming architecture
CN115858467A (zh) 针对键值数据库的文件处理方法、装置、电子设备和介质
CN113536088A (zh) 流式系统数据处理方法、装置、电子设备及存储介质
CN108804224B (zh) 一种基于Spark框架的中间数据权重设置方法
US10067678B1 (en) Probabilistic eviction of partial aggregation results from constrained results storage
US10185729B2 (en) Index creation method and system
CN112862554B (zh) 一种订单数据的处理方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22847883

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 13.05.2024)

122 Ep: pct application non-entry in european phase

Ref document number: 22847883

Country of ref document: EP

Kind code of ref document: A1