CN109033184A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN109033184A CN109033184A CN201810678074.9A CN201810678074A CN109033184A CN 109033184 A CN109033184 A CN 109033184A CN 201810678074 A CN201810678074 A CN 201810678074A CN 109033184 A CN109033184 A CN 109033184A
- Authority
- CN
- China
- Prior art keywords
- data
- handle
- data processing
- target
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present invention provides a kind of data processing method and device, which comprises according to the target access address in data processing configuration file, batch is read to handle data one by one from the corresponding targeting database server in target access address；Total number and the concurrent the same number of multiple data processing threads of thread are distributed to handle data one by one by what is read according to the concurrent number of thread in data processing configuration file；It controls multiple data processing threads and concurrently carries out data mart modeling processing to handle data one by one to what is be individually assigned to, obtain feedback data file corresponding with to handle data one by one；The obtained feedback data files in batch is written in targeting database server.The data processing method can sharing data library batch one by one handle data role pressure, realize and to efficiently one by one data mart modeling handle, to expand the application scenarios range of the database.
Description
Technical field
The present invention relates to database management technology fields, in particular to a kind of data processing method and device.
Background technique
With the continuous development of science and technology, requirement of the big data processing technique to the batch data process performance of database
More stringent, the required database of big data processing technique should can have at extremely strong data in different application scenarios
Rationality energy.But presently, many databases are being applied to the application for needing sequentially to be handled one by one high-volume data
When in environment, it can be taken considerable time because of the architected features of own database during data query obtains, to drop
The data-handling efficiency of low database makes database that can not have high-intensitive data processing performance in similar application environment.
Summary of the invention
In order to overcome above-mentioned deficiency in the prior art, the purpose of the present invention is to provide a kind of data processing method and dresses
Set, the data processing method can sharing data library batch one by one handle data role pressure, realize and to efficiently by
Data working process, to expand the application scenarios range of the database.
For method, the embodiment of the present invention provides a kind of data processing method, which comprises
According to the target access address in data processing configuration file from number of targets corresponding with the target access address
According at the server of library batch read Corresponding matching to handle data one by one；
According to the concurrent number of thread in the data processing configuration file will read described in handle data one by one
Distribute to total number and the concurrent the same number of multiple data processing threads of the thread；
The multiple data processing threads are controlled concurrently to add to what is be individually assigned to handle data progress data one by one
Work processing is obtained with described to handle the corresponding feedback data file of data one by one；
The obtained feedback data files in batch is written in the targeting database server.
For device, the embodiment of the present invention provides a kind of data processing equipment, and described device includes:
Data read module, for according to the target access address in data processing configuration file from the target access
At the corresponding targeting database server in address batch read Corresponding matching to handle data one by one；
Data allocation module, the institute for will be read according to the concurrent number of thread in the data processing configuration file
It states and distributes to total number and the concurrent the same number of multiple data processing threads of the thread to handle data one by one；
Working process module, for controlling the multiple data processing threads concurrently to being individually assigned to locate one by one
It manages data and carries out data mart modeling processing, obtain with described to handle the corresponding feedback data file of data one by one；
Data feedback module, for the obtained feedback data files in batch to be written to the target database service
In device.
In terms of existing technologies, data processing method provided in an embodiment of the present invention and device have below beneficial to effect
Fruit: the data processing method can sharing data library batch one by one handle data role pressure, realize and to efficiently by
Data working process, to expand the application scenarios range of the database.Firstly, the method is configured according to the data processing
Reading is corresponding in batches from targeting database server corresponding with the target access address for target access address in file
It is matched to handle data one by one.Then, the method will according to the concurrent number of thread in the data processing configuration file
What is read described distributes to the concurrent the same number of multiple data processing lines of total number and the thread to handle data one by one
Journey.Then, the method control the multiple data processing threads concurrently to be individually assigned to one by one handle data into
Row data working process is obtained with described to handle the corresponding feedback data file of data one by one.Finally, the method will obtain
The feedback data files in batch be written in the targeting database server.The data processing method can be realized simultaneously
The efficient data mart modeling one by one of row is handled, and the electronic equipment by executing the data processing method is shared the target database and taken
Batch corresponding to the database that runs handles the role pressure of data one by one on business device so that the targeting database server without
It need to be sequentially handled one by one to handle data one by one what is be read, correspondingly expand the application scenarios model of the database
It encloses.
To enable the above objects, features and advantages of the present invention to be clearer and more comprehensible, present pre-ferred embodiments are cited below particularly,
And cooperate appended attached drawing, it is described in detail below.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of the claims in the present invention protection scope, for those of ordinary skill in the art, what is do not made the creative labor
Under the premise of, it can also be obtained according to these attached drawings other relevant attached drawings.
Fig. 1 is a kind of block diagram of electronic equipment provided in an embodiment of the present invention.
Fig. 2 is one of the flow diagram of data processing method provided in an embodiment of the present invention.
Fig. 3 is a kind of flow diagram for the sub-step that step S220 shown in Fig. 2 includes.
Fig. 4 is a kind of flow diagram for the sub-step that step S230 shown in Fig. 2 includes.
Fig. 5 is the two of the flow diagram of data processing method provided in an embodiment of the present invention.
Fig. 6 is a kind of block diagram of data processing equipment shown in Fig. 1 provided in an embodiment of the present invention.
Fig. 7 is a kind of block diagram of data allocation module shown in Fig. 6.
Fig. 8 is a kind of block diagram of working process module shown in Fig. 6.
Fig. 9 is another block diagram of data processing equipment shown in Fig. 1 provided in an embodiment of the present invention.
Icon: 10- electronic equipment；11- memory；12- processor；13- communication unit；100- data processing equipment；
110- data read module；120- data allocation module；130- working process module；140- data feedback module；121- is divided
Submodule；122- distribution sub module；131- handles control submodule；132- data merge submodule；133- feedback generates submodule
Block；150- file configuration module.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.The present invention being usually described and illustrated herein in the accompanying drawings is implemented
The component of example can be arranged and be designed with a variety of different configurations.
Therefore, the detailed description of the embodiment of the present invention provided in the accompanying drawings is not intended to limit below claimed
The scope of the present invention, but be merely representative of selected embodiment of the invention.Based on the embodiments of the present invention, this field is common
Technical staff's every other embodiment obtained without creative efforts belongs to the model that the present invention protects
It encloses.
It should also be noted that similar label and letter indicate similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, does not then need that it is further defined and explained in subsequent attached drawing.
With reference to the accompanying drawing, it elaborates to some embodiments of the present invention.In the absence of conflict, following
Feature in embodiment and embodiment can be combined with each other.
Fig. 1 is please referred to, is a kind of block diagram of electronic equipment 10 provided in an embodiment of the present invention.Implement in the present invention
In example, the electronic equipment 10 has the targeting database server of database to communicate to connect with operation, by reading number of targets
According on the server of library to handle data one by one, and to described in reading to one by one handle data carry out it is parallel efficiently one by one
The mode of data mart modeling processing, shares batch corresponding to database on the targeting database server and handles data one by one
Role pressure, so that the targeting database server is not necessarily to sequentially be handled one by one to handle data one by one what is be read,
To expand the application scenarios range of the corresponding database run of the targeting database server.
Wherein, described to handle data is one by one the data for needing sequentially to be handled one by one, and the database can be
Distributed data base, the then targeting database server for running the database are to serve as master control section in correspondence database system
The server of point；The database is also possible to concentrating type database, then runs the targeting database server of the database i.e.
For the server of the isolated operation database.The database may be, but not limited to, Greenplum database, Oracle number
According to library etc.；The electronic equipment 10 may be, but not limited to, and server, is put down at PC (personal computer, PC)
Plate computer, personal digital assistant (personal digital assistant, PDA), mobile internet surfing equipment (mobile
Internet device, MID) etc..
In the present embodiment, the electronic equipment 10 includes data processing equipment 100, memory 11, processor 12 and leads to
Believe unit 13.The memory 11, processor 12 and each element of communication unit 13 directly or indirectly electrically connect between each other
It connects, to realize the transmission or interaction of data.For example, these elements can pass through one or more communication bus or signal between each other
Line, which is realized, to be electrically connected.
In the present embodiment, the memory 11 may be, but not limited to, random access memory (Random Access
Memory, RAM), read-only memory (Read Only Memory, ROM), programmable read only memory (Programmable
Read-Only Memory, PROM), Erasable Programmable Read Only Memory EPROM (Erasable Programmable Read-Only
Memory, EPROM), electrically erasable programmable read-only memory (Electric Erasable Programmable Read-
Only Memory, EEPROM) etc..The memory 11 can be used for storing program, and the processor 12 is executed instruction receiving
Afterwards, described program is executed.
In the present embodiment, the processor 12 can be a kind of IC chip of processing capacity with signal.
The processor 12 can be general processor, including central processing unit (Central Processing Unit, CPU), network
Processor (Network Processor, NP) etc..May be implemented or execute disclosed each method in the embodiment of the present invention,
Step and logic diagram.General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.
In the present embodiment, the communication unit 13 is used to establish the electronic equipment 10 and target database by network
Communication connection between server, and pass through the network sending and receiving data.
In the present embodiment, the data processing equipment 100 can be with software or firmware (firmware) including at least one
Form be stored in the memory 11 or be solidificated in the electronic equipment 10 operating system (operating system,
OS the software function module in).The executable module stored in the memory 11 can be performed in the processor 12, such as described
Software function module included by data processing equipment 100 and computer program etc..In the present embodiment, the electronic equipment 10
By the data processing equipment 100 read on targeting database server to handle data one by one, and to the institute read
It states and carries out parallel efficiently data mart modeling processing one by one to handle data one by one, it is right to share the targeting database server institute
The batch answered handles the role pressure of data one by one, so that the targeting database server is not necessarily to being read to handle one by one
Data are sequentially handled one by one, to expand the database application scene domain on the targeting database server.
It is understood that structure shown in FIG. 1 is only a kind of structural schematic diagram of electronic equipment 10, the electronic equipment
10 may also include than shown in Fig. 1 more perhaps less component or with the configuration different from shown in Fig. 1.Shown in Fig. 1
Each component can using hardware, software, or its combination realize.
It referring to figure 2., is one of the flow diagram of data processing method provided in an embodiment of the present invention.Of the invention real
It applies in example, the data processing method is applied to above-mentioned electronic equipment 10, below to the tool of data processing method shown in Fig. 2
Body process and step are described in detail.
Step S210, according to the target access address in data processing configuration file from corresponding with the target access address
Targeting database server at batch read Corresponding matching to handle data one by one.
In the present embodiment, the data processing configuration file is directed to target database for realizing the electronic equipment 10
Data mart modeling is carried out one by one to handling data one by one handling, record has pair in the data processing configuration file in server
The target access address of targeting database server is answered, the electronic equipment 10 handles the mesh in configuration file based on the data
Mark access address accesses the corresponding targeting database server in target access address, and from the targeting database server
Batch read need one by one data mart modeling processing to handle data one by one.
It wherein, further include that target data objects and target read number, the target in the data processing configuration file
Data object is for indicating in requisition for, to handle data one by one, the target is read described in progress one by one data mart modeling processing
Number is used for the number of data to handle data one by one for indicating to be read, and the target reads number can be by configuring number
It is numbered according to reading and the form of reading data finish number is indicated.The mesh according in data processing configuration file
Mark access address from targeting database server corresponding with the target access address batch read Corresponding matching to by
Item handle data the step of include:
To the transmission of the corresponding targeting database server in the target access address include the target data objects and
The target reads the data read command of number, to obtain and the target data pair from the targeting database server
As corresponding to handle data one by one.
Wherein, what is got is described identical as target reading number to handle the corresponding number of data of data one by one.
The corresponding targeting database server of target access address access is based in the electronic equipment 10 and sends the data
Read instruction after, the targeting database server by according in the data read command the target data objects and institute
It states target and reads number, searched and the target data objects and institute in the database run on the targeting database server
That states that target reads number Corresponding matching is sent to institute to handle data one by one to handle data one by one, and described in finding
State electronic equipment 10.
Step S220, according to the concurrent number of thread in the data processing configuration file will read described in one by one
Processing data distribute to total number and the concurrent the same number of multiple data processing threads of the thread.
In the present embodiment, in the data processing configuration file also record have for treat one by one processing data carry out by
The concurrent number of the thread of the data processing threads of data working process, the concurrent number of thread is for indicating that the electronics is set
The data processing threads number that can be concurrently run in standby 10.The electronic equipment 10 read it is described to handle number one by one
It, will be according to the concurrent number of thread and the number of data to handle data one by one in the data processing configuration file according to rear
It is allocated to described to handle data one by one, to handle one by one, data distribute to total number and the thread is concurrent by described
The same number of multiple data processing threads.
It optionally, referring to figure 3., is a kind of flow diagram for the sub-step that step S220 shown in Fig. 2 includes.?
In the present embodiment, the step S220 may include sub-step S221 and sub-step S222.
Sub-step S221, according to the concurrent number of the thread and the number of data to handle data one by one will it is described to
Processing data are divided into more parts and handle subdata one by one one by one.
In the present embodiment, the corresponding total number of processing subdata and the concurrent number of the thread one by one, it is all by
The sum of the number of data of item processing subdata is equal to the number of data to handle data one by one.The electronic equipment 10 can root
Average division is carried out to the number of data to handle data one by one according to the concurrent number of the thread, so that every part is handled one by one
The number of data of subdata is identical；The electronic equipment 10 can also be according to the data-handling capacity power journey of each data processing threads
Degree divides the number of data to handle data one by one, so that every part handles the number of data of subdata at least one by one
It is one.
Sub-step S222 will handle one by one subdata correspondence and distribute to the multiple data processing threads described in more parts.
In the present embodiment, the corresponding portion of data processing threads handles subdata one by one.If the electronic equipment 10 according to
The mode averagely divided divides the number of data to handle data one by one, then the electronic equipment 10 can at random by
Subdata is handled one by one described in more parts that division obtains and distributes to the multiple data processing threads, and the electronic equipment 10 can also
It is allocated according to the thread number of the multiple data processing threads to handling subdata one by one described in division obtain more parts.
If the electronic equipment 10 is according to the data-handling capacity degree of strength of the multiple data processing threads to described to locate one by one
The number of data of reason data is divided, then handles subdata one by one described in the electronic equipment 10 can obtain division more parts
Correspondence distributes to the multiple data processing threads, so that the strong data processing threads of data-handling capacity can handle number as far as possible
According to the processing subdata one by one more than item number, it is few that the weak data processing threads of data-handling capacity can handle as far as possible number of data
Subdata is handled one by one.
Referring once again to Fig. 2, step S230, control the multiple data processing threads concurrently to be individually assigned to
Processing data carry out data mart modeling processing one by one, obtain with described to handle the corresponding feedback data file of data one by one.
In the present embodiment, the electronic equipment 10 is for each data processing line in the multiple data processing threads
Journey be assigned with corresponding data item number after handling data (handling subdata one by one) one by one, the multiple data processing will be controlled
It carries out one by one formula data mart modeling to the subdata of processing one by one being individually assigned to thread parallel to handle, after obtaining alignment processing
With described to handle the corresponding feedback data file of data one by one.It is finally obtained in an embodiment of the present embodiment
The feedback data file be every part one by one processing subdata it is processed after feedback data set, the text of the feedback data file
Number of packages mesh is only one；In the another embodiment of the present embodiment, the finally obtained feedback data file is every part
Corresponding feedback data after processing subdata is processed one by one, the number of files of the feedback data file and processing one by one
The total number of data is identical.
It optionally, referring to figure 4., is a kind of flow diagram for the sub-step that step S230 shown in Fig. 2 includes.?
In the present embodiment, if the number of files of the finally obtained feedback data file is only one, the step S230 can be with
Including sub-step S231, sub-step S232 and sub-step S233.
Sub-step S231, each data processing threads of parallel control are one by one to the institute for being assigned to the data processing threads
It states processing subdata one by one and carries out data mart modeling processing, obtain corresponding result data.
In the present embodiment, also record has the data handled for realizing data mart modeling in the data processing configuration file
Handle logical code or data process method program.The electronic equipment 10 can be by concurrently controlling each data processing threads
According to the data process method code or data process method program to being assigned to described in the data processing threads one by one
It handles subdata and carries out data mart modeling processing, obtain each data processing threads and executed the corresponding result generated after process flow
Data.Wherein, each data processing threads are processing one by one matched to the data processing threads in a manner of handling one by one
Data carry out data mart modeling processing.
Result data corresponding to each data processing threads is carried out data merging, obtained corresponding by sub-step S232
Result data collection.
In the present embodiment, the electronic equipment 10 completes data mart modeling process flow in all data processing threads
Afterwards, by the way that result data corresponding to each data processing threads is carried out data merging, corresponding result data collection is obtained.
Sub-step S233 carries out Data Format Transform to the result data collection, and what is obtained and read is described to one by one
Handle the corresponding feedback data file of data.
In the present embodiment, the electronic equipment 10 is taken by the way that the result data collection is written to the target database
Business device is capable of the mode in the data file of identifying processing, carries out Data Format Transform to the result data collection, obtains and read
That gets is described to handle the corresponding feedback data file of data one by one.
In the present embodiment, if the number of files of the finally obtained feedback data file is handled one by one with described
The total number of subdata is identical, then the electronic equipment 10 can obtain each data by way of executing above-mentioned sub-step S231
Processing thread has executed the corresponding result data generated after process flow, and uses to each result data and be written to the target
Database server is capable of the mode in the data file of identifying processing, carries out Data Format Transform to each result data, obtains
To with read described in handle the corresponding feedback data file of data one by one, to ensure the finally obtained feedback data
The number of files of file is identical as the processing total number of subdata one by one.
The obtained feedback data files in batch is written in the targeting database server by step S240.
In the present embodiment, the electronic equipment 10 is being obtained with described to handle data corresponding feedback data text one by one
After part, by according to the target access address in the data processing configuration file by the obtained feedback data files in batch
It is sent to the targeting database server corresponding with the target access address, so that the targeting database server will
The feedback data file got is loaded into the database run on the targeting database server, to make the mesh
Mark database server reduces the target database without sequentially being handled one by one to handle data one by one what is be read
The batch of server handles the role pressure of data one by one, and has correspondingly expanded the application scenarios range of the database.
Referring to figure 5., be data processing method provided in an embodiment of the present invention flow diagram two.Of the invention real
It applies in example, the data processing method can also include step S209.
Step S209, to the target access address in data processing configuration file, the concurrent number of thread, target data objects
And target reads number and is configured.
In the present embodiment, the step S209 is in front of the step S210, the targeting database server
Operation maintenance personnel can be by way of using visual configuration at the electronic equipment 10 in the data processing configuration file
Target access address, the concurrent number of thread, target data objects and target read number and configured.Wherein, the data
The data process method code or data process method program for including in processing configuration file can also be by operation maintenance personnels according to demand
Modification configuration by hand is carried out at the electronic equipment 10, the modification by hand, which configures corresponding configuration mode, to be visualization
Configuration mode.
Fig. 6 is please referred to, is that a kind of box of data processing equipment 100 shown in Fig. 1 provided in an embodiment of the present invention shows
It is intended to.In embodiments of the present invention, the data processing equipment 100 include data read module 110, data allocation module 120,
Working process module 130 and data feedback module 140.
The data read module 110, for according to the target access address in data processing configuration file from it is described
At the corresponding targeting database server in target access address batch read Corresponding matching to handle data one by one.
It in the present embodiment, further include that target data objects and target read number in the data processing configuration file,
The data read module 110 according to the target access address in data processing configuration file from the target access address pair
The mode to handle data one by one of batch reading Corresponding matching includes: at the targeting database server answered
To the transmission of the corresponding targeting database server in the target access address include the target data objects and
The target reads the data read command of number, to obtain and the target data pair from the targeting database server
As corresponding to handle data one by one, wherein described read number to handle the corresponding number of data of data and the target one by one
It is identical.
Wherein, the data read module 110 can execute step S210 shown in Fig. 2, and specific implementation procedure can
Referring to above to the detailed description of step S210.
The data allocation module 120, for that will be read according to the concurrent number of thread in the data processing configuration file
That gets described distributes to the concurrent the same number of multiple data processing threads of total number and the thread to handle data one by one.
In the present embodiment, the data allocation module 120 can execute step S220 shown in Fig. 2, specifically hold
Row process can refer to above to the detailed description of step S220.
Optionally, Fig. 7 is please referred to, is a kind of block diagram of data allocation module 120 shown in Fig. 6.In this reality
It applies in example, the data allocation module 120 may include dividing submodule 121 and distribution sub module 122.
The division submodule 121, for according to the concurrent number of the thread and the data to handle data one by one
Item number is divided into described more parts to handling data one by one and handles subdata one by one, wherein the subdata of processing one by one is corresponding
Total number is identical as the concurrent number of the thread.
Wherein, the division submodule 121 can execute sub-step S221 shown in Fig. 3, and specific implementation procedure can
Referring to the detailed description of above sub-paragraphs S221.
The distribution sub module 122 distributes to the multiple data for will handle one by one subdata correspondence described in more parts
Thread is handled, wherein the corresponding portion of each data processing threads handles subdata one by one.
Wherein, the distribution sub module 122 can execute sub-step S222 shown in Fig. 3, and specific implementation procedure can
Referring to the detailed description of above sub-paragraphs S222.
Referring once again to Fig. 6, the working process module 130, for controlling the multiple data processing threads concurrently
To being individually assigned to carry out data mart modeling processing to handle data one by one, obtain with it is described corresponding anti-to one by one handle data
Present data file.
In the present embodiment, the working process module 130 can execute step S230 shown in Fig. 2, specifically hold
Row process can refer to above to the detailed description of step S230.
Optionally, Fig. 8 is please referred to, is a kind of block diagram of working process module 130 shown in Fig. 6.In this reality
It applies in example, the working process module 130 may include processing control submodule 131, data merge submodule 132 and feedback is given birth to
At submodule 133.
The processing control submodule 131, for each data processing threads of parallel control one by one to being assigned to this
The subdata of processing one by one of data processing threads carries out data mart modeling processing, obtains corresponding result data.
Wherein, the processing control submodule 131 can execute sub-step S231 shown in Fig. 4, specifically execute
Journey can refer to the detailed description of above sub-paragraphs S231.
The data merge submodule 132, for result data corresponding to each data processing threads to be carried out data
Merge, obtains corresponding result data collection.
Wherein, the data, which merge submodule 132, can execute sub-step S232 shown in Fig. 4, specifically execute
Journey can refer to the detailed description of above sub-paragraphs S232.
The feedback generates submodule 133, for carrying out Data Format Transform to the result data collection, obtains and reads
To it is described to handle the corresponding feedback data file of data one by one.
Wherein, the feedback, which generates submodule 133, can execute sub-step S233 shown in Fig. 4, specifically execute
Journey can refer to the detailed description of above sub-paragraphs S233.
Referring once again to Fig. 6, the data feedback module 140, for writing the obtained feedback data files in batch
Enter into the targeting database server.
In the present embodiment, the data feedback module 140 can execute step S240 shown in Fig. 2, specifically hold
Row process can refer to above to the detailed description of step S240.
Fig. 9 is please referred to, is another box of data processing equipment 100 shown in Fig. 1 provided in an embodiment of the present invention
Schematic diagram.In embodiments of the present invention, the data processing equipment 100 can also include file configuration module 150.
The file configuration module 150, for the target access address in data processing configuration file, thread number of concurrent
Mesh, target data objects and target read number and are configured.
In the present embodiment, the file configuration module 150 can execute step S209 shown in Fig. 5, specifically hold
Row process can refer to above to the detailed description of step S209.
In conclusion in data processing method provided in an embodiment of the present invention and device, the data processing method energy
Enough sharing data library batches handle the role pressure of data one by one, realize and to efficiently data mart modeling is handled one by one, to expand
The application scenarios range of the database.Firstly, the method is according to the target access address in the data processing configuration file
From targeting database server corresponding with the target access address batch read Corresponding matching to handle data one by one.
Then, the method according to the concurrent number of thread in the data processing configuration file will read described in handle one by one
Data distribute to total number and the concurrent the same number of multiple data processing threads of the thread.Then, the method controls institute
Multiple data processing threads are stated concurrently to being individually assigned to carry out data mart modeling processing to handle data one by one, are obtained and institute
It states to handle the corresponding feedback data file of data one by one.Finally, the feedback data files in batch that the method will obtain
It is written in the targeting database server.The data processing method can be realized at parallel efficient data mart modeling one by one
Reason, it is right that the electronic equipment by executing the data processing method shares the database institute run on the targeting database server
The batch answered handles the role pressure of data one by one, so that the targeting database server is not necessarily to being read to handle one by one
Data are sequentially handled one by one, have correspondingly expanded the application scenarios range of the database.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of data processing method, which is characterized in that the described method includes:
According to the target access address in data processing configuration file from target database corresponding with the target access address
At server batch read Corresponding matching to handle data one by one；
According to the concurrent number of thread in the data processing configuration file will read described in handle one by one data distribution
To total number and the concurrent the same number of multiple data processing threads of the thread；
The multiple data processing threads are controlled concurrently to carry out at data mart modeling to what is be individually assigned to handle data one by one
Reason is obtained with described to handle the corresponding feedback data file of data one by one；
The obtained feedback data files in batch is written in the targeting database server.
2. the method according to claim 1, wherein further including target data in the data processing configuration file
Object and target read number, and the target access address according in data processing configuration file is from the target access
At the corresponding targeting database server in location batch read Corresponding matching to handle data one by one the step of include:
It to the transmission of the corresponding targeting database server in the target access address include the target data objects and described
Target reads the data read command of number, to obtain and the target data objects pair from the targeting database server
Answer to handle data one by one, wherein described to handle the corresponding number of data of data and target reading number phase one by one
Together.
3. the method according to claim 1, wherein the thread according in the data processing configuration file
Concurrent number will read described in handle one by one data distribute to total number and the thread is concurrently equal in number multiple
The step of data processing threads includes:
It is drawn described to handle data one by one according to the concurrent number of the thread and the number of data to handle data one by one
It is divided into more parts and handles subdata one by one, wherein the corresponding total number of the subdata of processing one by one and the concurrent number phase of the thread
Together；
Subdata correspondence will be handled one by one described in more parts and distributes to the multiple data processing threads, wherein each data processing line
The corresponding portion of journey handles subdata one by one.
4. according to the method described in claim 3, it is characterized in that, the multiple data processing threads of control are concurrently right
What is be individually assigned to carries out data mart modeling processing to handle data one by one, obtains with described to handle the corresponding feedback of data one by one
The step of data file includes:
The each data processing threads of parallel control handle subnumber to being assigned to described in the data processing threads one by one one by one
According to data mart modeling processing is carried out, corresponding result data is obtained；
Result data corresponding to each data processing threads is subjected to data merging, obtains corresponding result data collection；
Data Format Transform is carried out to the result data collection, is obtained described corresponding instead to handle data one by one with what is read
Present data file.
5. method described in any one of -4 according to claim 1, which is characterized in that the method also includes:
Number is read to the target access address in data processing configuration file, the concurrent number of thread, target data objects and target
Mesh is configured.
6. a kind of data processing equipment, which is characterized in that described device includes:
Data read module, for according to the target access address in data processing configuration file from the target access address
At corresponding targeting database server batch read Corresponding matching to handle data one by one；
Data allocation module, for according to the concurrent number of thread in the data processing configuration file will read described in
Processing data distribute to total number and the concurrent the same number of multiple data processing threads of the thread one by one；
Working process module, for controlling the multiple data processing threads concurrently to being individually assigned to handle number one by one
According to data mart modeling processing is carried out, obtain with described to handle the corresponding feedback data file of data one by one；
Data feedback module, for the obtained feedback data files in batch to be written to the targeting database server
In.
7. device according to claim 6, which is characterized in that further include target data in the data processing configuration file
Object and target read number, the data read module according to the target access address in data processing configuration file from institute
State the mode to handle data one by one that batch at the corresponding targeting database server in target access address reads Corresponding matching
Include:
It to the transmission of the corresponding targeting database server in the target access address include the target data objects and described
Target reads the data read command of number, to obtain and the target data objects pair from the targeting database server
Answer to handle data one by one, wherein described to handle the corresponding number of data of data and target reading number phase one by one
Together.
8. device according to claim 6, which is characterized in that the data allocation module includes:
Divide submodule, for according to the concurrent number of the thread and the number of data to handle data one by one will it is described to
Processing data are divided into more parts and handle subdata one by one one by one, wherein the corresponding total number of the subdata of processing one by one with it is described
The concurrent number of thread is identical；
Distribution sub module distributes to the multiple data processing threads for will handle one by one subdata correspondence described in more parts,
In the corresponding portion of each data processing threads handle subdata one by one.
9. device according to claim 8, which is characterized in that the working process module includes:
Control submodule is handled, for each data processing threads of parallel control one by one to being assigned to the data processing threads
It is described one by one processing subdata carry out data mart modeling processing, obtain corresponding result data；
Data merge submodule, for result data corresponding to each data processing threads to be carried out data merging, obtain pair
The result data collection answered；
Feedback generate submodule, for the result data collection carry out Data Format Transform, obtain and read it is described to
The corresponding feedback data file of processing data one by one.
10. the device according to any one of claim 6-9, which is characterized in that described device further include:
File configuration module, for the target access address in data processing configuration file, the concurrent number of thread, target data
Object and target read number and are configured.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810678074.9A CN109033184B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810678074.9A CN109033184B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109033184A true CN109033184A (en) | 2018-12-18 |
| CN109033184B CN109033184B (en) | 2021-08-17 |
Family
ID=64610780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810678074.9A Active CN109033184B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109033184B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222075A (en) * | 2019-04-25 | 2019-09-10 | 视联动力信息技术股份有限公司 | A kind of method, view networked system and the mserver system of response data inquiry |
| CN110895490A (en) * | 2019-11-29 | 2020-03-20 | 深圳乐信软件技术有限公司 | Data batch processing system, method, equipment and storage medium |
| CN114116803A (en) * | 2021-11-30 | 2022-03-01 | 中国建设银行股份有限公司 | Method, device and equipment for processing big data file and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916296A (en) * | 2010-08-29 | 2010-12-15 | 武汉天喻信息产业股份有限公司 | Mass data processing method based on files |
| CN104239133A (en) * | 2014-09-26 | 2014-12-24 | 北京国双科技有限公司 | Log processing method, device and server |
| CN104376082A (en) * | 2014-11-18 | 2015-02-25 | 中国建设银行股份有限公司 | Method for importing data in data source file to database |
| CN104657111A (en) * | 2013-11-20 | 2015-05-27 | 方正信息产业控股有限公司 | Parallel computing method and device |
| CN104715076A (en) * | 2015-04-13 | 2015-06-17 | 东信和平科技股份有限公司 | Multi-threaded data processing method and device |
| CN105975331A (en) * | 2016-04-26 | 2016-09-28 | 浪潮(北京)电子信息产业有限公司 | Data parallel processing method and apparatus |
| US20160357703A1 (en) * | 2015-06-04 | 2016-12-08 | Fujitsu Limited | Parallel computing apparatus, compiling apparatus, and parallel processing method |
-
2018
- 2018-06-27 CN CN201810678074.9A patent/CN109033184B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916296A (en) * | 2010-08-29 | 2010-12-15 | 武汉天喻信息产业股份有限公司 | Mass data processing method based on files |
| CN104657111A (en) * | 2013-11-20 | 2015-05-27 | 方正信息产业控股有限公司 | Parallel computing method and device |
| CN104239133A (en) * | 2014-09-26 | 2014-12-24 | 北京国双科技有限公司 | Log processing method, device and server |
| CN104376082A (en) * | 2014-11-18 | 2015-02-25 | 中国建设银行股份有限公司 | Method for importing data in data source file to database |
| CN104715076A (en) * | 2015-04-13 | 2015-06-17 | 东信和平科技股份有限公司 | Multi-threaded data processing method and device |
| US20160357703A1 (en) * | 2015-06-04 | 2016-12-08 | Fujitsu Limited | Parallel computing apparatus, compiling apparatus, and parallel processing method |
| CN105975331A (en) * | 2016-04-26 | 2016-09-28 | 浪潮(北京)电子信息产业有限公司 | Data parallel processing method and apparatus |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222075A (en) * | 2019-04-25 | 2019-09-10 | 视联动力信息技术股份有限公司 | A kind of method, view networked system and the mserver system of response data inquiry |
| CN110222075B (en) * | 2019-04-25 | 2021-11-19 | 视联动力信息技术股份有限公司 | Method for responding to data query, video networking system and mserver system |
| CN110895490A (en) * | 2019-11-29 | 2020-03-20 | 深圳乐信软件技术有限公司 | Data batch processing system, method, equipment and storage medium |
| CN114116803A (en) * | 2021-11-30 | 2022-03-01 | 中国建设银行股份有限公司 | Method, device and equipment for processing big data file and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109033184B (en) | 2021-08-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7783627B2 (en) | Database retrieval with a unique key search on a parallel computer system | |
| US20130232133A1 (en) | Systems and methods for performing a nested join operation | |
| CN108959146A (en) | Data-storage system | |
| CN108959292A (en) | A kind of data uploading method, system and computer readable storage medium | |
| CN109033184A (en) | Data processing method and device | |
| CN109614402B (en) | Multidimensional data query method and device | |
| CN110162388A (en) | A kind of method for scheduling task, system and terminal device | |
| CN110309142B (en) | Method and device for rule management | |
| CN101872335A (en) | CPU console redirecting method and system and CPUs | |
| CN102982116A (en) | Multi-media transfer method and system based on cloud | |
| CN115168162B (en) | Multi-gray-scale issuing method and device based on ingess controller in container environment and storage medium | |
| CN110147507A (en) | A kind of method, apparatus obtaining short chained address and server | |
| CN108268503A (en) | A kind of storage of database, querying method and device | |
| CN102201922B (en) | Data charging method and relevant apparatus | |
| CN112328656B (en) | Service query method, device, equipment and storage medium based on middle platform architecture | |
| CN113296959B (en) | Service processing method and device based on AOP functional component and computer equipment | |
| US20180278472A1 (en) | System and method for performing mass renaming of list of items at run-time with variable differentiation factor | |
| CN113918305B (en) | Node scheduling method, node scheduling device, electronic equipment and readable storage medium | |
| CN115328938A (en) | Inventory query method and device | |
| CN115292580A (en) | Data query method and device, computer equipment and storage medium | |
| CN113077318A (en) | Service processing method and device | |
| CN112988874A (en) | Data processing method, system, computing device and readable storage medium | |
| CN110033145B (en) | Financial sharing job order separating method and device, equipment and storage medium | |
| CN112183799A (en) | Task allocation method and device for synthesizing task list | |
| CN109815295A (en) | Distributed type assemblies data lead-in method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |