CN111400012A - Data parallel processing method, device, equipment and storage medium - Google Patents

Data parallel processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN111400012A
CN111400012A CN202010199891.3A CN202010199891A CN111400012A CN 111400012 A CN111400012 A CN 111400012A CN 202010199891 A CN202010199891 A CN 202010199891A CN 111400012 A CN111400012 A CN 111400012A
Authority
CN
China
Prior art keywords
data
splitting
subtasks
sub
processed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010199891.3A
Other languages
Chinese (zh)
Inventor
陈军
唐成山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202010199891.3A priority Critical patent/CN111400012A/en
Publication of CN111400012A publication Critical patent/CN111400012A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/5018—Thread allocation

Abstract

The embodiment of the invention provides a data parallel processing method, a data parallel processing device, data parallel processing equipment and a storage medium. The method comprises the following steps: if the task to be processed supports the fragmentation, acquiring a data splitting rule; splitting the task to be processed into a plurality of subtasks according to the splitting rule; distributing the subtasks to corresponding threads for processing to obtain a plurality of sub results; and combining the sub-results to obtain a data processing result. According to the data parallel processing method provided by the embodiment of the invention, the task to be processed is divided into the plurality of subtasks, then the plurality of subtasks are distributed to the corresponding threads for parallel processing, and finally, the sub-results are combined to obtain the final data processing result, so that the batch processing time of single job processing can be reduced, and the data processing efficiency is improved.

Description

Data parallel processing method, device, equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data parallel processing method, a data parallel processing device, data parallel processing equipment and a storage medium.
Background
A business bank core system is constructed based on an IBM mainframe, online and batch processing on the mainframe are based on files, no network loss exists in the calculation process, but after the mainframe is moved to an X86 open system, the loss of a CPU (central processing unit), network communication, IO (input/output) and the like of an X86 system is increased, a single machine cannot meet the service processing of more than 10 hundred million grades of a large bank, the bottleneck of a database is solved by vertically splitting the service field of service data and horizontally splitting the data, but in the batch processing, the data files of one batch reach million grades, and if the data files are processed by a single machine, efficient calculation cannot be realized. Therefore, it is important to improve the data processing efficiency.
Disclosure of Invention
Embodiments of the present invention provide a data parallel processing method, apparatus, device, and storage medium, which split a task into multiple pieces according to a certain proportion by data fragmentation, and run in parallel on different machines to reduce the batch processing time for processing a single job, thereby improving the data processing efficiency.
In a first aspect, an embodiment of the present invention provides a data parallel processing method, including:
if the task to be processed supports the fragmentation, acquiring a data splitting rule;
splitting the task to be processed into a plurality of subtasks according to the splitting rule;
distributing the subtasks to corresponding threads for processing to obtain a plurality of sub results;
and combining the sub-results to obtain a data processing result.
Further, obtaining a data splitting rule includes:
acquiring fragmentation parameters according to the identity codes of the tasks to be processed;
and determining a data splitting rule according to the fragmentation parameters.
Further, splitting the task to be processed into a plurality of subtasks according to a set splitting rule, including:
registering a splitting table;
registering information of the split plurality of subtasks into the split table; the splitting table is used for recording splitting progress.
Further, the information of the subtasks includes: the identity identification code, the input file path, the output file path and the number of records of the subtasks.
Further, distributing the multiple subtasks to corresponding threads for processing, and obtaining multiple sub results, including:
and distributing the subtasks to corresponding threads for processing according to the information of the subtasks and the locking mechanism, and obtaining a plurality of sub results.
In a second aspect, an embodiment of the present invention further provides a data parallel processing apparatus, including:
the data splitting rule obtaining module is used for obtaining a data splitting rule if the task to be processed supports the fragment;
the subtask acquisition module is used for splitting the task to be processed into a plurality of subtasks according to the splitting rule;
the sub-result acquisition module is used for distributing the plurality of sub-tasks to corresponding threads for processing to obtain a plurality of sub-results;
and the data processing result acquisition module is used for merging the sub-results to obtain a data processing result.
Further, the data splitting rule obtaining module is further configured to:
acquiring fragmentation parameters according to the identity codes of the tasks to be processed;
and determining a data splitting rule according to the fragmentation parameters.
Further, the subtask obtaining module is further configured to:
registering a splitting table;
registering information of the split plurality of subtasks into the split table; the splitting table is used for recording splitting progress. .
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the data parallel processing method according to the embodiment of the present invention when executing the computer program.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processing apparatus, implements the data parallel processing method according to the embodiment of the present invention.
In the embodiment of the invention, if the task to be processed supports fragmentation, a data splitting rule is obtained; splitting a task to be processed into a plurality of subtasks according to a splitting rule; distributing the plurality of subtasks to corresponding threads for processing to obtain a plurality of sub results; and combining the results to obtain a data processing result. According to the data parallel processing method provided by the embodiment of the invention, the task to be processed is divided into the plurality of subtasks, then the plurality of subtasks are distributed to the corresponding threads for parallel processing, and finally, the sub-results are combined to obtain the final data processing result, so that the batch processing time of single job processing can be reduced, and the data processing efficiency is improved.
Drawings
FIG. 1 is a flow chart of a data parallel processing method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a method for parallel processing of data according to a first embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a parallel data processing apparatus according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device in a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data parallel processing method according to an embodiment of the present invention, where the embodiment is applicable to a case of batch processing of data, and the method may be executed by a data parallel processing apparatus, where the apparatus may be composed of hardware and/or software, and may be generally integrated in a device with a data parallel processing function, where the device may be an electronic device such as a server or a server cluster. As shown in fig. 1, the method specifically comprises the following steps:
and step 110, if the task to be processed supports the fragmentation, acquiring a data splitting rule.
The task to be processed may be batch data, for example: one pending task containing 200 million records. In this embodiment, it is first determined whether the task to be processed supports data fragmentation, and if so, the data fragmentation rule required by the task to be processed is obtained.
Specifically, the manner of obtaining the data splitting rule may be: acquiring fragmentation parameters according to the identity codes of the tasks to be processed; and determining a data splitting rule according to the fragmentation parameters.
The identity code of the task to be processed may be the name of the task to be processed. In this embodiment, if the to-be-processed task supports the fragmentation, the fragmentation parameter table is read according to the identification code of the to-be-processed task, and then the data splitting rule is obtained according to the fragmentation parameter.
And step 120, splitting the task to be processed into a plurality of subtasks according to the splitting rule.
Specifically, the process of splitting the task to be processed into a plurality of subtasks according to the set splitting rule may be: registering a splitting table; and registering the information of the split plurality of subtasks into the split table.
The splitting table is used for recording splitting progress. The information of the subtasks may include: the identity identification code, the input file path, the output file path and the number of records of the subtasks. In this embodiment, the main task and the subtasks are recorded in the splitting table, and after the information of all the subtasks is registered in the splitting table, the state of the main task is updated to be "split completed". In this embodiment, parameter adjustment may also be performed according to the task identification code and the information of the subtask.
Step 130, allocating the multiple subtasks to the corresponding threads for processing, and obtaining multiple sub results.
Specifically, the manner of allocating the multiple subtasks to the corresponding threads for processing may be: and distributing the plurality of subtasks to corresponding threads for processing according to the information of the subtasks and the locking mechanism, and obtaining a plurality of sub results.
In this embodiment, the split table is used to record subtask information, and a locking mechanism of the database is used to realize that each subtask is processed by only one thread, and does not complete one subtask, and update the number of records for completing the main task.
And step 140, combining the plurality of sub-results to obtain a data processing result.
In this embodiment, after the processed sub-results are obtained, the main thread merges the plurality of sub-results to obtain the data processing result.
According to the technical scheme of the embodiment, if the task to be processed supports fragmentation, a data splitting rule is obtained; splitting a task to be processed into a plurality of subtasks according to a splitting rule; distributing the plurality of subtasks to corresponding threads for processing to obtain a plurality of sub results; and combining the results to obtain a data processing result. According to the data parallel processing method provided by the embodiment of the invention, the task to be processed is divided into the plurality of subtasks, then the plurality of subtasks are distributed to the corresponding threads for parallel processing, and finally, the sub-results are combined to obtain the final data processing result, so that the batch processing time of single job processing can be reduced, and the data processing efficiency is improved.
Optionally, fig. 2 is a flowchart of a data parallel processing method according to an embodiment of the present invention, which is used to further explain the above embodiment, where the method specifically includes: acquiring a task to be processed, judging whether the task to be processed supports fragmentation, if so, reading a fragmentation parameter table according to a task name, and acquiring a data fragmentation rule according to a fragmentation parameter; registering a splitting table, and registering main task and subtask information into the splitting table; allocating the plurality of subtasks to corresponding threads for processing by using an exclusive lock mechanism to obtain a plurality of sub results, and updating the state in the splitting table to be completed; and when all the subtasks are processed, combining the multiple sub-results to obtain a data processing result.
Example two
Fig. 3 is a schematic structural diagram of a data parallel processing apparatus according to a second embodiment of the present invention. As shown in fig. 3, the apparatus includes: a data splitting rule obtaining module 210, a subtask obtaining module 220, a sub result obtaining module 230, and a data processing result obtaining module 240.
A data splitting rule obtaining module 210, configured to obtain a data splitting rule if the to-be-processed task supports the fragment;
the subtask obtaining module 220 is configured to split the to-be-processed task into a plurality of subtasks according to a splitting rule;
a sub-result obtaining module 230, configured to allocate multiple sub-tasks to corresponding threads for processing, so as to obtain multiple sub-results;
and a data processing result obtaining module 240, configured to combine the multiple sub-results to obtain a data processing result.
Optionally, the data splitting rule obtaining module 210 is further configured to:
acquiring fragmentation parameters according to the identity codes of the tasks to be processed;
and determining a data splitting rule according to the fragmentation parameters.
Optionally, the subtask obtaining module 220 is further configured to:
registering a splitting table;
registering the information of the split multiple subtasks into a split table; the splitting table is used for recording splitting progress.
Optionally, the information of the subtask includes: the identity identification code, the input file path, the output file path and the number of records of the subtasks.
Optionally, the sub-result obtaining module 230 is further configured to:
and distributing the plurality of subtasks to corresponding threads for processing according to the information of the subtasks and the locking mechanism, and obtaining a plurality of sub results.
The device can execute the methods provided by all the embodiments of the invention, and has corresponding functional modules and beneficial effects for executing the methods. For details not described in detail in this embodiment, reference may be made to the methods provided in all the foregoing embodiments of the present invention.
EXAMPLE III
Fig. 4 is a schematic structural diagram of a computer device according to a third embodiment of the present invention. FIG. 4 illustrates a block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention. The computer device 312 shown in FIG. 4 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention. Device 312 is a computing device for typical data parallel processing functions.
As shown in FIG. 4, computer device 312 is in the form of a general purpose computing device. The components of computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 that couples the various system components including the storage device 328 and the processors 316.
Bus 318 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Computer device 312 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 312 and includes both volatile and nonvolatile media, removable and non-removable media.
Storage 328 may include computer system readable media in the form of volatile Memory, such as Random Access Memory (RAM) 330 and/or cache Memory 332. The computer device 312 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 334 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a Compact disk-Read Only Memory (CD-ROM), a Digital Video disk (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 318 by one or more data media interfaces. Storage 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
Program 336 having a set (at least one) of program modules 326 may be stored, for example, in storage 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which may comprise an implementation of a network environment, or some combination thereof. Program modules 326 generally carry out the functions and/or methodologies of embodiments of the invention as described herein.
The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), and may also communicate with one or more devices that enable a user to interact with the computer device 312, and/or with any device (e.g., Network card, modem, etc.) that enables the computer device 312 to communicate with one or more other computing devices (i.e., I/O interface 322. this communication may be via input/output (I/O) interface 322. and, further, the computer device 312 may communicate with one or more networks (e.g., local Area Network (L O Network, &ttlttransition = L "&ttg L &lt/t &ttg AN), Wide Area Network (R) Area (WAN) and/or public Network, such as the Internet) via Network adapter 320 via bus 318. as shown, the Network adapter 320 communicates with other modules of the computer device 312 via bus 318. it should be appreciated that other modules of the computer device 312 may be used in conjunction with other modules including, a Wide Area array of disk drive systems (RAID, RAID array, Redundant storage drive, RAID array, and/or disk drive (R) storage system, and/or disk redundancy array of RAID redundancy, and the like.
The processor 316 executes various functional applications and data processing by executing programs stored in the storage device 328, for example, to implement the data parallel processing method provided by the above-described embodiment of the present invention.
Example four
An embodiment of the present invention provides a computer-readable storage medium. The computer readable medium of the present invention described above may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText transfer protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). examples of communications networks include local area networks ("L AN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a source text input by a user, and translating the source text into a target text corresponding to a target language; acquiring historical correction behaviors of the user; and correcting the target text according to the historical correction behaviors to obtain a translation result, and pushing the translation result to a client where the user is located.
Computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including but not limited to AN object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.
For example, without limitation, exemplary types of hardware logic that may be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CP L D), and so forth.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. A method for parallel processing of data, comprising:
if the task to be processed supports the fragmentation, acquiring a data splitting rule;
splitting the task to be processed into a plurality of subtasks according to the splitting rule;
distributing the subtasks to corresponding threads for processing to obtain a plurality of sub results;
and combining the sub-results to obtain a data processing result.
2. The method of claim 1, wherein obtaining the data splitting rule comprises:
acquiring fragmentation parameters according to the identity codes of the tasks to be processed;
and determining a data splitting rule according to the fragmentation parameters.
3. The method according to claim 1, wherein splitting the to-be-processed task into a plurality of subtasks according to a set splitting rule comprises:
registering a splitting table;
registering information of the split plurality of subtasks into the split table; the splitting table is used for recording splitting progress.
4. The method of claim 3, wherein the information of the subtasks comprises: the identity identification code, the input file path, the output file path and the number of records of the subtasks.
5. The method of claim 3, wherein assigning the plurality of subtasks to corresponding threads for processing to obtain a plurality of sub-results comprises:
and distributing the subtasks to corresponding threads for processing according to the information of the subtasks and the locking mechanism, and obtaining a plurality of sub results.
6. A data parallel processing apparatus, comprising:
the data splitting rule obtaining module is used for obtaining a data splitting rule if the task to be processed supports the fragment;
the subtask acquisition module is used for splitting the task to be processed into a plurality of subtasks according to the splitting rule;
the sub-result acquisition module is used for distributing the plurality of sub-tasks to corresponding threads for processing to obtain a plurality of sub-results;
and the data processing result acquisition module is used for merging the sub-results to obtain a data processing result.
7. The apparatus of claim 6, wherein the data splitting rule obtaining module is further configured to:
acquiring fragmentation parameters according to the identity codes of the tasks to be processed;
and determining a data splitting rule according to the fragmentation parameters.
8. The apparatus of claim 6, wherein the subtask obtaining module is further configured to:
registering a splitting table;
registering information of the split plurality of subtasks into the split table; the splitting table is used for recording splitting progress.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data parallel processing method according to any of claims 1-5 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when executed by a processing device, implements a data parallel processing method according to any one of claims 1 to 5.
CN202010199891.3A 2020-03-20 2020-03-20 Data parallel processing method, device, equipment and storage medium Pending CN111400012A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010199891.3A CN111400012A (en) 2020-03-20 2020-03-20 Data parallel processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010199891.3A CN111400012A (en) 2020-03-20 2020-03-20 Data parallel processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN111400012A true CN111400012A (en) 2020-07-10

Family

ID=71434374

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010199891.3A Pending CN111400012A (en) 2020-03-20 2020-03-20 Data parallel processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111400012A (en)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782403A (en) * 2020-07-17 2020-10-16 Oppo广东移动通信有限公司 Data processing method and device and electronic equipment
CN112130972A (en) * 2020-08-24 2020-12-25 北京思特奇信息技术股份有限公司 Task processing method and system
CN112162859A (en) * 2020-09-24 2021-01-01 成都长城开发科技有限公司 Data processing method and device, computer readable medium and electronic equipment
CN112162644A (en) * 2020-09-24 2021-01-01 深圳哇哇鱼网络科技有限公司 Hardware acceleration method, terminal and medium for multi-language input method
CN112307105A (en) * 2020-11-03 2021-02-02 平安普惠企业管理有限公司 Timing task running method, device, equipment and storage medium based on multithreading
CN112364001A (en) * 2020-11-03 2021-02-12 北京红山信息科技研究院有限公司 Cleaning method and device for multidimensional space big data, computer equipment and medium
CN112486966A (en) * 2020-11-26 2021-03-12 中国国家铁路集团有限公司 Expired data cleaning method and device and electronic equipment
CN112818338A (en) * 2021-01-22 2021-05-18 支付宝(杭州)信息技术有限公司 Program running method and system
CN113010286A (en) * 2021-03-12 2021-06-22 京东数字科技控股股份有限公司 Parallel task scheduling method and device, computer equipment and storage medium
CN113110935A (en) * 2021-04-13 2021-07-13 中国邮政储蓄银行股份有限公司 Distributed batch job processing system
CN113392252A (en) * 2021-06-01 2021-09-14 上海徐毓智能科技有限公司 Data processing method and device
CN113407429A (en) * 2021-06-23 2021-09-17 中国建设银行股份有限公司 Task processing method and device
CN115062022A (en) * 2022-06-16 2022-09-16 北京汉端科技有限公司 Aircraft manual splitting method, device, electronic equipment and computer readable medium
CN115361382A (en) * 2022-08-10 2022-11-18 北京友友天宇系统技术有限公司 Data processing method, device, equipment and storage medium based on data group
CN115378937A (en) * 2022-08-10 2022-11-22 北京友友天宇系统技术有限公司 Distributed concurrency method, device and equipment for tasks and readable storage medium
CN115426266A (en) * 2022-10-09 2022-12-02 中国工商银行股份有限公司 Configuration issuing method and device
CN116401070A (en) * 2023-06-06 2023-07-07 昆山嘉提信息科技有限公司 Multi-MCU data parallel processing method and device
CN115426266B (en) * 2022-10-09 2024-04-30 中国工商银行股份有限公司 Configuration issuing method and device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107688500A (en) * 2017-07-26 2018-02-13 阿里巴巴集团控股有限公司 A kind of distributed task scheduling processing method, device, system and equipment
CN109308214A (en) * 2017-07-27 2019-02-05 北京京东尚科信息技术有限公司 Data task processing method and system
CN109408214A (en) * 2018-11-06 2019-03-01 北京字节跳动网络技术有限公司 A kind of method for parallel processing of data, device, electronic equipment and readable medium
CN110673959A (en) * 2019-10-14 2020-01-10 京东数字科技控股有限公司 System, method and apparatus for processing tasks
CN110781159A (en) * 2019-10-28 2020-02-11 柏科数据技术（深圳）股份有限公司 Ceph directory file information reading method and device, server and storage medium

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107688500A (en) * 2017-07-26 2018-02-13 阿里巴巴集团控股有限公司 A kind of distributed task scheduling processing method, device, system and equipment
CN109308214A (en) * 2017-07-27 2019-02-05 北京京东尚科信息技术有限公司 Data task processing method and system
CN109408214A (en) * 2018-11-06 2019-03-01 北京字节跳动网络技术有限公司 A kind of method for parallel processing of data, device, electronic equipment and readable medium
CN110673959A (en) * 2019-10-14 2020-01-10 京东数字科技控股有限公司 System, method and apparatus for processing tasks
CN110781159A (en) * 2019-10-28 2020-02-11 柏科数据技术（深圳）股份有限公司 Ceph directory file information reading method and device, server and storage medium

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782403A (en) * 2020-07-17 2020-10-16 Oppo广东移动通信有限公司 Data processing method and device and electronic equipment
CN112130972A (en) * 2020-08-24 2020-12-25 北京思特奇信息技术股份有限公司 Task processing method and system
CN112162859A (en) * 2020-09-24 2021-01-01 成都长城开发科技有限公司 Data processing method and device, computer readable medium and electronic equipment
CN112162644A (en) * 2020-09-24 2021-01-01 深圳哇哇鱼网络科技有限公司 Hardware acceleration method, terminal and medium for multi-language input method
CN112307105A (en) * 2020-11-03 2021-02-02 平安普惠企业管理有限公司 Timing task running method, device, equipment and storage medium based on multithreading
CN112364001A (en) * 2020-11-03 2021-02-12 北京红山信息科技研究院有限公司 Cleaning method and device for multidimensional space big data, computer equipment and medium
CN112486966A (en) * 2020-11-26 2021-03-12 中国国家铁路集团有限公司 Expired data cleaning method and device and electronic equipment
CN112818338A (en) * 2021-01-22 2021-05-18 支付宝(杭州)信息技术有限公司 Program running method and system
CN113010286A (en) * 2021-03-12 2021-06-22 京东数字科技控股股份有限公司 Parallel task scheduling method and device, computer equipment and storage medium
CN113110935A (en) * 2021-04-13 2021-07-13 中国邮政储蓄银行股份有限公司 Distributed batch job processing system
CN113392252A (en) * 2021-06-01 2021-09-14 上海徐毓智能科技有限公司 Data processing method and device
CN113392252B (en) * 2021-06-01 2023-01-17 上海徐毓智能科技有限公司 Data processing method and device
CN113407429A (en) * 2021-06-23 2021-09-17 中国建设银行股份有限公司 Task processing method and device
CN115062022A (en) * 2022-06-16 2022-09-16 北京汉端科技有限公司 Aircraft manual splitting method, device, electronic equipment and computer readable medium
CN115062022B (en) * 2022-06-16 2023-08-18 北京汉端科技有限公司 Aircraft manual splitting method, device, electronic equipment and computer readable medium
CN115361382A (en) * 2022-08-10 2022-11-18 北京友友天宇系统技术有限公司 Data processing method, device, equipment and storage medium based on data group
CN115378937A (en) * 2022-08-10 2022-11-22 北京友友天宇系统技术有限公司 Distributed concurrency method, device and equipment for tasks and readable storage medium
CN115426266A (en) * 2022-10-09 2022-12-02 中国工商银行股份有限公司 Configuration issuing method and device
CN115426266B (en) * 2022-10-09 2024-04-30 中国工商银行股份有限公司 Configuration issuing method and device
CN116401070A (en) * 2023-06-06 2023-07-07 昆山嘉提信息科技有限公司 Multi-MCU data parallel processing method and device

Similar Documents

Publication Publication Date Title
CN111400012A (en) Data parallel processing method, device, equipment and storage medium
US11544623B2 (en) Consistent filtering of machine learning data
US10102480B2 (en) Machine learning service
US11100420B2 (en) Input processing for machine learning
US10866973B2 (en) Test data management
CN109508326B (en) Method, device and system for processing data
US20190050402A1 (en) Deferred update of database hashcode in blockchain
US10402406B2 (en) Predictive database for computer processes
US11222016B2 (en) Dynamic combination of processes for sub-queries
US11200231B2 (en) Remote query optimization in multi data sources
US9501377B2 (en) Generating and implementing data integration job execution design recommendations
US9652227B2 (en) Assigning an annotation to a variable and a statement in a source code of a software application
CN110428319B (en) Data comparison platform, method, equipment and storage medium
US10425483B2 (en) Distributed client based cache for keys using demand fault invalidation
CN115964646A (en) Heterogeneous graph generation for application microservices
CN109597697B (en) Resource matching processing method and device
CN111782614B (en) Data access method, device, equipment and storage medium
US9632837B2 (en) Systems and methods for system consolidation
CN111581098A (en) Interface data transfer storage method, device, server and storage medium
CN113312195A (en) Data processing method, device, equipment and storage medium
US8799318B2 (en) Function module leveraging fuzzy search capability
US10528400B2 (en) Detecting deadlock in a cluster environment using big data analytics
CN112148705A (en) Data migration method and device
US20230214394A1 (en) Data search method and apparatus, electronic device and storage medium
US20230237095A1 (en) Metadata for Graph Connected Databases

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20220920

Address after: 25 Financial Street, Xicheng District, Beijing 100033

Applicant after: CHINA CONSTRUCTION BANK Corp.

Address before: 25 Financial Street, Xicheng District, Beijing 100033

Applicant before: CHINA CONSTRUCTION BANK Corp.

Applicant before: Jianxin Financial Science and Technology Co.,Ltd.

TA01 Transfer of patent application right