CN109033184B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN109033184B CN109033184B CN201810678074.9A CN201810678074A CN109033184B CN 109033184 B CN109033184 B CN 109033184B CN 201810678074 A CN201810678074 A CN 201810678074A CN 109033184 B CN109033184 B CN 109033184B
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- data processing
- target
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data processing method and a device, wherein the method comprises the following steps: reading data to be processed one by one in batch from a target database server corresponding to a target access address according to the target access address in the data processing configuration file; according to the thread concurrency number in the data processing configuration file, distributing the read data to be processed one by one to a plurality of data processing threads with the total number being the same as the thread concurrency number; controlling a plurality of data processing threads to perform data processing on the data to be processed which are distributed respectively one by one in parallel to obtain a feedback data file corresponding to the data to be processed one by one; and writing the obtained feedback data files into a target database server in batches. The data processing method can share the task pressure of batch processing data item by item of the database, and realize efficient processing data item by item so as to expand the application scene range of the database.
Description
Technical Field
The invention relates to the technical field of database management, in particular to a data processing method and device.
Background
With the continuous development of scientific technology, the requirement of the big data processing technology on the data batch processing performance of the database is more and more strict, and the database required by the big data processing technology has extremely strong data processing performance in different application scenes. However, currently, when a plurality of databases are applied to an application environment that needs to process a large amount of data sequentially and one by one, a large amount of time is consumed in the data query and acquisition process due to the architectural characteristics of the databases, so that the data processing efficiency of the databases is reduced, and the databases cannot have high-strength data processing performance in similar application environments.
Disclosure of Invention
In order to overcome the above disadvantages in the prior art, an object of the present invention is to provide a data processing method and apparatus, where the data processing method can share the task pressure of batch processing data item by item of a database, and implement and process efficient processing data item by item to expand the application scene range of the database.
As for a method, an embodiment of the present invention provides a data processing method, where the method includes:
reading correspondingly matched data to be processed one by one in batches from a target database server corresponding to a target access address according to the target access address in a data processing configuration file;
according to the thread concurrency number in the data processing configuration file, distributing the read data to be processed one by one to a plurality of data processing threads of which the total number is the same as the thread concurrency number;
controlling the plurality of data processing threads to perform data processing on the data to be processed which are distributed respectively one by one in parallel to obtain a feedback data file corresponding to the data to be processed one by one;
and writing the obtained feedback data files into the target database server in batches.
As for an apparatus, an embodiment of the present invention provides a data processing apparatus, including:
the data reading module is used for reading the correspondingly matched data to be processed one by one in batches from the target database server corresponding to the target access address according to the target access address in the data processing configuration file;
the data distribution module is used for distributing the read data to be processed one by one to a plurality of data processing threads with the total number being the same as the thread concurrent number according to the thread concurrent number in the data processing configuration file;
the processing module is used for controlling the data processing threads to process the data to be processed one by one which is distributed to the data processing threads in parallel to obtain a feedback data file corresponding to the data to be processed one by one;
and the data feedback module is used for writing the obtained feedback data files into the target database server in batches.
Compared with the prior art, the data processing method and the data processing device provided by the embodiment of the invention have the following beneficial effects: the data processing method can share the task pressure of batch processing data item by item of the database, and realize efficient processing data item by item so as to expand the application scene range of the database. Firstly, the method reads the corresponding matched data to be processed one by one in batch from the target database server corresponding to the target access address according to the target access address in the data processing configuration file. Then, the method allocates the read data to be processed item by item to a plurality of data processing threads with the total number being the same as the thread concurrency number according to the thread concurrency number in the data processing configuration file. Then, the method controls the data processing threads to process the data to be processed one by one distributed to the data to be processed one by one in parallel to obtain a feedback data file corresponding to the data to be processed one by one. And finally, writing the obtained feedback data files into the target database server in batch by the method. The data processing method can realize parallel and efficient processing of data item by item, and the electronic equipment executing the data processing method shares the task pressure of batch data item by item processing corresponding to the database running on the target database server, so that the target database server does not need to sequentially process the read data to be processed item by item, and the application scene range of the database is correspondingly expanded.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments are briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention, and therefore should not be considered as limiting the scope of the claims of the present invention, and it is obvious for those skilled in the art that other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a block diagram of an electronic device according to an embodiment of the present invention.
Fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating the sub-steps included in step S220 shown in fig. 2.
Fig. 4 is a flowchart illustrating the sub-steps included in step S230 shown in fig. 2.
Fig. 5 is a second flowchart of a data processing method according to an embodiment of the invention.
Fig. 6 is a block diagram of the data processing apparatus shown in fig. 1 according to an embodiment of the present invention.
FIG. 7 is a block diagram of the data distribution module shown in FIG. 6.
FIG. 8 is a block diagram of the process module shown in FIG. 6.
Fig. 9 is another block diagram of the data processing apparatus shown in fig. 1 according to an embodiment of the present invention.
Icon: 10-an electronic device; 11-a memory; 12-a processor; 13-a communication unit; 100-a data processing device; 110-a data reading module; 120-a data distribution module; 130-processing module; 140-a data feedback module; 121-dividing sub-modules; 122-an allocation submodule; 131-a processing control sub-module; 132-data merge sub-module; 133-a feedback generation submodule; 150-file configuration module.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Fig. 1 is a block diagram of an electronic device 10 according to an embodiment of the present invention. In the embodiment of the present invention, the electronic device 10 is in communication connection with a target database server running a database, and shares the task pressure of batch processing data on the target database server one by reading data to be processed on the target database server and performing parallel processing on the read data to be processed one by one efficiently, so that the target database server does not need to perform sequential processing on the read data to be processed one by one, thereby expanding the application scene range of the database running the target database server correspondingly.
The data to be processed one by one is data which needs to be processed one by one in sequence, the database can be a distributed database, and a target database server running the database is a server serving as a main control node in a corresponding database system; the database may also be a clustered database, and the target database server running the database is the server running the database alone. The database may be, but is not limited to, a greenplus database, an Oracle database, etc.; the electronic device 10 may be, but is not limited to, a server, a Personal Computer (PC), a tablet PC, a Personal Digital Assistant (PDA), a Mobile Internet Device (MID), and the like.
In the present embodiment, the electronic device 10 includes a data processing apparatus 100, a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12 and the communication unit 13 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
In this embodiment, the Memory 11 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an electrically Erasable Programmable Read-Only Memory (EEPROM), and the like. The memory 11 may be used to store programs that are executed by the processor 12 upon receiving execution instructions.
In this embodiment, the processor 12 may be an integrated circuit chip having signal processing capabilities. The Processor 12 may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), and the like. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In this embodiment, the communication unit 13 is configured to establish a communication connection between the electronic device 10 and a target database server through a network, and to send and receive data through the network.
In this embodiment, the data processing apparatus 100 includes at least one software functional module which can be stored in the memory 11 in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the electronic device 10. The processor 12 may execute executable modules stored in the memory 11, such as software functional modules and computer programs included in the data processing apparatus 100. In this embodiment, the electronic device 10 reads the data to be processed on the target database server one by one through the data processing apparatus 100, and performs parallel and efficient data processing on the read data to be processed one by one to share the task pressure of batch data processing one by one corresponding to the target database server, so that the target database server does not need to perform sequential data processing one by one on the read data to be processed one by one, thereby expanding the range of the database application scenario on the target database server.
It is understood that the configuration shown in fig. 1 is only a schematic configuration of the electronic device 10, and that the electronic device 10 may include more or less components than those shown in fig. 1, or have a different configuration than that shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.
Fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present invention. In the embodiment of the present invention, the data processing method is applied to the electronic device 10, and specific flows and steps of the data processing method shown in fig. 2 are described in detail below.
Step S210, reading the corresponding matched data to be processed one by one in batch from the target database server corresponding to the target access address according to the target access address in the data processing configuration file.
In this embodiment, the data processing configuration file is used to implement that the electronic device 10 performs data processing item by item on to-be-processed data item by item in a target database server, a target access address corresponding to the target database server is recorded in the data processing configuration file, the electronic device 10 accesses the target database server corresponding to the target access address based on the target access address in the data processing configuration file, and batch reads the to-be-processed data item by item that needs to be processed by data processing item by item from the target database server.
The data processing configuration file further comprises a target data object and a target reading number, the target data object is used for representing the data to be processed one by one corresponding to be processed one by one, the target reading number is used for representing the number of the read data to be processed one by one, and the target reading number can be represented in the form of a configuration data reading start number and a data reading end number. The step of reading the corresponding matched data to be processed one by one in batch from the target database server corresponding to the target access address according to the target access address in the data processing configuration file comprises the following steps:
and sending a data reading instruction comprising the target data object and the target reading number to a target database server corresponding to the target access address so as to acquire the data to be processed corresponding to the target data object one by one from the target database server.
And the number of the acquired data corresponding to the data to be processed one by one is the same as the target reading number. After the electronic device 10 accesses a corresponding target database server based on the target access address and sends the data reading instruction, the target database server searches for piece-by-piece processing data, which is correspondingly matched with the target data object and the target reading number, in a database running on the target database server according to the target data object and the target reading number in the data reading instruction, and sends the searched piece-by-piece processing data to the electronic device 10.
Step S220, according to the thread concurrency number in the data processing configuration file, allocating the read data to be processed one by one to a plurality of data processing threads, the total number of which is the same as the thread concurrency number.
In this embodiment, the data processing configuration file further records a thread concurrency number of data processing threads for processing data to be processed one by one, where the thread concurrency number is used to indicate a number of data processing threads that can be run in parallel in the electronic device 10. After the data to be processed item by item is read, the electronic device 10 allocates the data to be processed item by item according to the number of concurrent threads in the data processing configuration file and the number of data items of the data to be processed item by item, so as to allocate the data to be processed item by item to a plurality of data processing threads of which the total number is the same as the number of concurrent threads.
Optionally, please refer to fig. 3, which is a flowchart illustrating the sub-steps included in step S220 shown in fig. 2. In this embodiment, the step S220 may include a sub-step S221 and a sub-step S222.
And a substep S221, dividing the data to be processed one by one into a plurality of sub data to be processed one by one according to the thread concurrency number and the data number of the data to be processed one by one.
In this embodiment, the total number of the sub-data processed one by one and the thread concurrence number are equal to the number of the data to be processed one by one. The electronic device 10 may equally divide the number of the data to be processed one by one according to the number of concurrent threads, so that the number of the data to be processed sub-data is the same for each sub-data; the electronic device 10 may also divide the number of data to be processed one by one according to the data processing capability of each data processing thread, so that the number of data to be processed sub-data one by one is at least one.
And a substep S222, correspondingly allocating a plurality of copies of the processing sub-data to the plurality of data processing threads.
In this embodiment, the data processing thread corresponds to one piece of processing sub-data. If the electronic device 10 divides the number of the data to be processed one by one in an average division manner, the electronic device 10 may randomly allocate the divided multiple sets of the sub-data to the multiple data processing threads, and the electronic device 10 may also allocate the divided multiple sets of the sub-data to be processed one by one according to the thread numbers of the multiple data processing threads. If the electronic device 10 divides the number of data to be processed one by one according to the degree of the data processing capabilities of the data processing threads, the electronic device 10 may correspondingly allocate the divided multiple pieces of sub-data to the data processing threads, so that the data processing threads with strong data processing capabilities can process the sub-data with the number of data as large as possible, and the data processing threads with weak data processing capabilities can process the sub-data with the number of data as small as possible.
Referring to fig. 2 again, in step S230, the multiple data processing threads are controlled to perform data processing on the data to be processed respectively allocated to the multiple data processing threads in parallel, so as to obtain a feedback data file corresponding to the data to be processed one by one.
In this embodiment, after allocating the data to be processed one by one (the sub-data to be processed one by one) corresponding to the number of data pieces to each data processing thread of the plurality of data processing threads, the electronic device 10 controls the plurality of data processing threads to perform data processing one by one on the sub-data to be processed one by one respectively allocated to the data processing threads in parallel, so as to obtain the feedback data file corresponding to the data to be processed one by one after the corresponding processing. In an implementation manner of this embodiment, the finally obtained feedback data file is a feedback data set in which each piece of processed sub data is processed, and the number of the feedback data file is only one; in another implementation manner of this embodiment, the finally obtained feedback data file is corresponding feedback data after each piece of processed sub data is processed, and the number of the feedback data file is the same as the total number of the pieces of processed sub data.
Optionally, please refer to fig. 4, which is a flowchart illustrating the sub-steps included in step S230 shown in fig. 2. In this embodiment, if the number of the finally obtained feedback data files is only one, the step S230 may include a substep S231, a substep S232, and a substep S233.
And a substep S231 of performing data processing on the piece-by-piece processing sub-data allocated to each data processing thread in parallel by controlling each data processing thread to obtain corresponding result data.
In this embodiment, the data processing configuration file further records a data processing logic code or a data processing logic program for implementing data processing. The electronic device 10 may perform data processing on the piece-by-piece processing sub-data allocated to each data processing thread according to the data processing logic code or the data processing logic program by controlling each data processing thread in parallel, so as to obtain result data correspondingly generated after each data processing thread executes the processing flow. Each data processing thread processes the data of the sub-data matched with the data processing thread one by one in a processing mode one by one.
And a substep S232, performing data merging on the result data corresponding to each data processing thread to obtain a corresponding result data set.
In this embodiment, after all the data processing threads complete the data processing flow, the electronic device 10 performs data merging on the result data corresponding to each data processing thread to obtain a corresponding result data set.
And a substep S233, performing data format conversion on the result data set to obtain a feedback data file corresponding to the read data to be processed one by one.
In this embodiment, the electronic device 10 performs data format conversion on the result data set by writing the result data set into a data file that can be identified and processed by the target database server, so as to obtain a feedback data file corresponding to the read data to be processed one by one.
In this embodiment, if the number of the finally obtained feedback data files is the same as the total number of the sub-data processed piece by piece, the electronic device 10 may obtain, by executing the sub-step S231, result data that is correspondingly generated after each data processing thread executes the processing flow, perform data format conversion on each result data in a manner of writing the result data into a data file that can be identified and processed by the target database server, and obtain a feedback data file corresponding to the read data to be processed piece by piece, so as to ensure that the number of the finally obtained feedback data files is the same as the total number of the sub-data processed piece by piece.
Step S240, writing the obtained feedback data files into the target database server in batch.
In this embodiment, after obtaining the feedback data file corresponding to the data to be processed one by one, the electronic device 10 sends the obtained feedback data file to the target database server corresponding to the target access address in batch according to the target access address in the data processing configuration file, so that the target database server loads the obtained feedback data file into the database running on the target database server, and thus the target database server does not need to perform sequential one by one processing on the read data to be processed one by one, the task pressure of batch one by one processing data of the target database server is reduced, and the application scene range of the database is correspondingly expanded.
Fig. 5 is a second schematic flow chart of the data processing method according to the embodiment of the invention. In the embodiment of the present invention, the data processing method may further include step S209.
Step S209 configures the target access address, the thread concurrency number, the target data object, and the target reading number in the data processing configuration file.
In this embodiment, before the step S210 in the step S209, the operation and maintenance personnel of the target database server may configure the target access address, the thread concurrency number, the target data object, and the target reading number in the data processing configuration file by using a visual configuration mode at the electronic device 10. The data processing logic code or the data processing logic program included in the data processing configuration file may also be manually modified and configured at the electronic device 10 by an operation and maintenance worker according to a requirement, and a configuration mode corresponding to the manual modification and configuration may also be a visual configuration mode.
Fig. 6 is a block diagram of the data processing apparatus 100 shown in fig. 1 according to an embodiment of the present invention. In the embodiment of the present invention, the data processing apparatus 100 includes a data reading module 110, a data distributing module 120, a processing module 130, and a data feedback module 140.
The data reading module 110 is configured to read, in batches, the corresponding matched data to be processed one by one from the target database server corresponding to the target access address according to the target access address in the data processing configuration file.
In this embodiment, the data processing configuration file further includes a target data object and a target reading number, and the manner of reading, by the data reading module 110, the corresponding and matched piece-by-piece data to be processed in batch from the target database server corresponding to the target access address according to the target access address in the data processing configuration file includes:
and sending a data reading instruction comprising the target data object and the target reading number to a target database server corresponding to the target access address so as to acquire the data to be processed one by one corresponding to the target data object from the target database server, wherein the data number corresponding to the data to be processed one by one is the same as the target reading number.
The data reading module 110 may execute step S210 shown in fig. 2, and the specific execution process may refer to the above detailed description of step S210.
The data distribution module 120 is configured to distribute the read data to be processed one by one to a plurality of data processing threads, of which the total number is the same as the number of concurrent threads, according to the number of concurrent threads in the data processing configuration file.
In this embodiment, the data distribution module 120 may execute step S220 shown in fig. 2, and the specific execution process may refer to the above detailed description of step S220.
Optionally, please refer to fig. 7, which is a block diagram illustrating the data distribution module 120 shown in fig. 6. In this embodiment, the data distribution module 120 may include a dividing sub-module 121 and a distribution sub-module 122.
The dividing submodule 121 is configured to divide the data to be processed one by one into multiple sub-pieces of sub-processing data one by one according to the number of concurrent threads and the number of data pieces of the data to be processed one by one, where a total number of sub-pieces of the sub-processing data one by one is the same as the number of concurrent threads.
The dividing sub-module 121 may perform the sub-step S221 shown in fig. 3, and the detailed implementation process may refer to the detailed description of the sub-step S221 above.
The distributing submodule 122 is configured to correspondingly distribute multiple copies of the piece-by-piece processing sub data to the multiple data processing threads, where each data processing thread corresponds to one piece of the piece-by-piece processing sub data.
The sub-module 122 may perform the sub-step S222 shown in fig. 3, and the detailed process may refer to the above detailed description of the sub-step S222.
Referring to fig. 6 again, the processing module 130 is configured to control the multiple data processing threads to perform data processing on the data to be processed respectively allocated to the multiple data processing threads in parallel, so as to obtain a feedback data file corresponding to the data to be processed one by one.
In this embodiment, the processing module 130 may execute step S230 shown in fig. 2, and the specific execution process may refer to the above detailed description of step S230.
Optionally, please refer to fig. 8, which is a block diagram of the processing module 130 shown in fig. 6. In this embodiment, the processing module 130 may include a processing control sub-module 131, a data merging sub-module 132, and a feedback generation sub-module 133.
The processing control sub-module 131 is configured to control each data processing thread to perform data processing on the piece-by-piece processing sub-data allocated to the data processing thread in parallel, so as to obtain corresponding result data.
The processing control sub-module 131 may execute the sub-step S231 shown in fig. 4, and the detailed execution process may refer to the detailed description of the sub-step S231 above.
The data merging submodule 132 is configured to perform data merging on the result data corresponding to each data processing thread to obtain a corresponding result data set.
The data merging sub-module 132 may perform the sub-step S232 shown in fig. 4, and the specific implementation process may refer to the above detailed description of the sub-step S232.
The feedback generation submodule 133 is configured to perform data format conversion on the result data set to obtain a feedback data file corresponding to the read data to be processed one by one.
The feedback generation sub-module 133 may perform the sub-step S233 shown in fig. 4, and the detailed implementation process may refer to the above detailed description of the sub-step S233.
Referring to fig. 6 again, the data feedback module 140 is configured to write the obtained feedback data file into the target database server in batch.
In this embodiment, the data feedback module 140 may execute step S240 shown in fig. 2, and the specific execution process may refer to the above detailed description of step S240.
Fig. 9 is a block diagram of another data processing apparatus 100 shown in fig. 1 according to an embodiment of the present invention. In the embodiment of the present invention, the data processing apparatus 100 may further include a file configuration module 150.
The file configuration module 150 is configured to configure a target access address, a thread concurrency number, a target data object, and a target reading number in the data processing configuration file.
In this embodiment, the file configuration module 150 may execute step S209 shown in fig. 5, and the specific execution process may refer to the above detailed description of step S209.
In summary, in the data processing method and the data processing apparatus provided in the embodiments of the present invention, the data processing method can share the task pressure of batch processing data item by item of the database, and realize and process efficient data item by item to expand the application scene range of the database. Firstly, the method reads the corresponding matched data to be processed one by one in batch from the target database server corresponding to the target access address according to the target access address in the data processing configuration file. Then, the method allocates the read data to be processed item by item to a plurality of data processing threads with the total number being the same as the thread concurrency number according to the thread concurrency number in the data processing configuration file. Then, the method controls the data processing threads to process the data to be processed one by one distributed to the data to be processed one by one in parallel to obtain a feedback data file corresponding to the data to be processed one by one. And finally, writing the obtained feedback data files into the target database server in batch by the method. The data processing method can realize parallel and efficient processing of data item by item, and the electronic equipment executing the data processing method shares the task pressure of batch data item by item processing corresponding to the database running on the target database server, so that the target database server does not need to sequentially process the read data to be processed item by item, and the application scene range of the database is correspondingly expanded.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
reading correspondingly matched data to be processed one by one in batches from a target database server corresponding to a target access address according to the target access address in a data processing configuration file;
according to the thread concurrency number in the data processing configuration file, distributing the read data to be processed one by one to a plurality of data processing threads of which the total number is the same as the thread concurrency number;
controlling the plurality of data processing threads to perform data processing on the data to be processed which are distributed respectively one by one in parallel to obtain a feedback data file corresponding to the data to be processed one by one;
and writing the obtained feedback data files into the target database server in batches.
2. The method according to claim 1, wherein the data processing configuration file further includes a target data object and a target reading number, and the step of batch reading the corresponding matched data to be processed one by one from the target database server corresponding to the target access address according to the target access address in the data processing configuration file comprises:
and sending a data reading instruction comprising the target data object and the target reading number to a target database server corresponding to the target access address so as to acquire the data to be processed one by one corresponding to the target data object from the target database server, wherein the data number corresponding to the data to be processed one by one is the same as the target reading number.
3. The method according to claim 1, wherein the step of allocating the read data to be processed item by item according to the number of concurrent threads in the data processing configuration file to a plurality of data processing threads with a total number equal to the number of concurrent threads comprises:
dividing the data to be processed one by one into a plurality of sub data to be processed one by one according to the thread concurrency number and the data number of the data to be processed one by one, wherein the total number of the sub data to be processed one by one is the same as the thread concurrency number;
and correspondingly distributing a plurality of copies of the processing sub-data to the data processing threads, wherein each data processing thread corresponds to one copy of the processing sub-data one by one.
4. The method according to claim 3, wherein the step of controlling the plurality of data processing threads to perform data processing on the data to be processed respectively allocated to the data to be processed one by one in parallel to obtain the feedback data file corresponding to the data to be processed one by one comprises:
parallelly controlling each data processing thread to process the data of the sub-data one by one allocated to the data processing thread one by one to obtain corresponding result data;
merging the result data corresponding to each data processing thread to obtain a corresponding result data set;
and converting the data format of the result data set to obtain a feedback data file corresponding to the read data to be processed one by one.
5. The method according to any one of claims 1-4, further comprising:
and configuring the target access address, the thread concurrency number, the target data object and the target reading number in the data processing configuration file.
6. A data processing apparatus, characterized in that the apparatus comprises:
the data reading module is used for reading the correspondingly matched data to be processed one by one in batches from the target database server corresponding to the target access address according to the target access address in the data processing configuration file;
the data distribution module is used for distributing the read data to be processed one by one to a plurality of data processing threads with the total number being the same as the thread concurrent number according to the thread concurrent number in the data processing configuration file;
the processing module is used for controlling the data processing threads to process the data to be processed one by one which is distributed to the data processing threads in parallel to obtain a feedback data file corresponding to the data to be processed one by one;
and the data feedback module is used for writing the obtained feedback data files into the target database server in batches.
7. The apparatus of claim 6, wherein the data processing configuration file further includes a target data object and a target reading number, and the manner for the data reading module to read the corresponding and matched pieces of data to be processed in batch from the target database server corresponding to the target access address according to the target access address in the data processing configuration file includes:
and sending a data reading instruction comprising the target data object and the target reading number to a target database server corresponding to the target access address so as to acquire the data to be processed one by one corresponding to the target data object from the target database server, wherein the data number corresponding to the data to be processed one by one is the same as the target reading number.
8. The apparatus of claim 6, wherein the data distribution module comprises:
the dividing submodule is used for dividing the data to be processed one by one into a plurality of sub-data to be processed one by one according to the thread concurrency number and the data number of the data to be processed one by one, wherein the total number of the sub-data to be processed one by one is the same as the thread concurrency number;
and the distribution submodule is used for correspondingly distributing a plurality of copies of the piece-by-piece processing subdata to the plurality of data processing threads, wherein each data processing thread corresponds to one piece of piece-by-piece processing subdata.
9. The apparatus of claim 8, wherein the process module comprises:
the processing control submodule is used for parallelly controlling each data processing thread to process the data of the processing subdata allocated to the data processing thread one by one to obtain corresponding result data;
the data merging submodule is used for carrying out data merging on the result data corresponding to each data processing thread to obtain a corresponding result data set;
and the feedback generation submodule is used for performing data format conversion on the result data set to obtain a feedback data file corresponding to the read data to be processed one by one.
10. The apparatus according to any one of claims 6-9, further comprising:
and the file configuration module is used for configuring the target access address, the thread concurrency number, the target data object and the target reading number in the data processing configuration file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810678074.9A CN109033184B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810678074.9A CN109033184B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109033184A CN109033184A (en) | 2018-12-18 |
| CN109033184B true CN109033184B (en) | 2021-08-17 |
Family
ID=64610780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810678074.9A Active CN109033184B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109033184B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222075B (en) * | 2019-04-25 | 2021-11-19 | 视联动力信息技术股份有限公司 | Method for responding to data query, video networking system and mserver system |
| CN110895490A (en) * | 2019-11-29 | 2020-03-20 | 深圳乐信软件技术有限公司 | Data batch processing system, method, equipment and storage medium |
| CN114116803A (en) * | 2021-11-30 | 2022-03-01 | 中国建设银行股份有限公司 | Method, device and equipment for processing big data file and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105975331A (en) * | 2016-04-26 | 2016-09-28 | 浪潮(北京)电子信息产业有限公司 | Data parallel processing method and apparatus |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916296B (en) * | 2010-08-29 | 2012-12-19 | 武汉天喻信息产业股份有限公司 | Mass data processing method based on files |
| CN104657111A (en) * | 2013-11-20 | 2015-05-27 | 方正信息产业控股有限公司 | Parallel computing method and device |
| CN104239133B (en) * | 2014-09-26 | 2018-03-30 | 北京国双科技有限公司 | A kind of log processing method, device and server |
| CN104376082B (en) * | 2014-11-18 | 2019-06-18 | 中国建设银行股份有限公司 | A method of the data in data source file are imported into database |
| CN104715076B (en) * | 2015-04-13 | 2019-03-12 | 东信和平科技股份有限公司 | A kind of data processing of multithread and device |
| JP6432450B2 (en) * | 2015-06-04 | 2018-12-05 | 富士通株式会社 | Parallel computing device, compiling device, parallel processing method, compiling method, parallel processing program, and compiling program |
-
2018
- 2018-06-27 CN CN201810678074.9A patent/CN109033184B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105975331A (en) * | 2016-04-26 | 2016-09-28 | 浪潮(北京)电子信息产业有限公司 | Data parallel processing method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109033184A (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108549583B (en) | Big data processing method and device, server and readable storage medium | |
| CN106961454B (en) | File downloading method and device and terminal equipment | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| CN108959292B (en) | Data uploading method, system and computer readable storage medium | |
| CN109033184B (en) | Data processing method and device | |
| CN110096336B (en) | Data monitoring method, device, equipment and medium | |
| CN106302780B (en) | Method, device and system for batch data transmission of cluster equipment and server | |
| CN114416352B (en) | Computing power resource allocation method and device, electronic equipment and storage medium | |
| CN111506401B (en) | Automatic driving simulation task scheduling method and device, electronic equipment and storage medium | |
| CN110633135A (en) | Asynchronous task allocation method and device, computer equipment and storage medium | |
| CN106302640A (en) | Data request processing method and device | |
| CN109597810B (en) | Task segmentation method, device, medium and electronic equipment | |
| CN111988429A (en) | Algorithm scheduling method and system | |
| CN114531477B (en) | Method and device for configuring functional components, computer equipment and storage medium | |
| CN112422450A (en) | Computer equipment, and flow control method and device for service request | |
| CN116302708A (en) | Data backup method, device, equipment and storage medium based on load balancing | |
| CN115392501A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN111580948A (en) | Task scheduling method and device and computer equipment | |
| CN112052144B (en) | Information management method, device, electronic equipment and storage medium | |
| CN110912967A (en) | Service node scheduling method, device, equipment and storage medium | |
| CN111382141B (en) | Master-slave architecture configuration method, device, equipment and computer readable storage medium | |
| CN111767126A (en) | System and method for distributed batch processing | |
| CN108920278B (en) | Resource allocation method and device | |
| CN111431951A (en) | Data processing method, node equipment, system and storage medium | |
| US20220129313A1 (en) | Introspection of a containerized application in a runtime environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |