CN108920272A - A kind of data processing method, device, computer equipment and storage medium - Google Patents

A kind of data processing method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN108920272A
CN108920272A CN201810585017.6A CN201810585017A CN108920272A CN 108920272 A CN108920272 A CN 108920272A CN 201810585017 A CN201810585017 A CN 201810585017A CN 108920272 A CN108920272 A CN 108920272A
Authority
CN
China
Prior art keywords
node server
destination node
data
state
data processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810585017.6A
Other languages
Chinese (zh)
Inventor
高越
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Life Insurance Company of China Ltd
Original Assignee
Ping An Life Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Life Insurance Company of China Ltd filed Critical Ping An Life Insurance Company of China Ltd
Priority to CN201810585017.6A priority Critical patent/CN108920272A/en
Publication of CN108920272A publication Critical patent/CN108920272A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/08Insurance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/50Indexing scheme relating to G06F9/50
    • G06F2209/508Monitor

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Accounting & Taxation (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • Technology Law (AREA)
  • General Business, Economics & Management (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a kind of data processing method, device, computer equipment and storage medium, the method includes:Receive the declaration form traffic table that client is sent,Data fragmentation is carried out to the declaration form traffic table using preset sharding method,Obtain P target service table,Each node server is detected,Obtain destination node server,And the state score of each destination node server is calculated according to use state,Obtain the task allocation proportion of each destination node server,And P target service table is distributed in proportion in destination node server and is stored,And establish identification information,Target service table,And the mapping relations between destination node server,So that the data in declaration form traffic table are in the way of load balancing of having the right,It is assigned in each destination node server and is stored,Guarantee that the data volume of each destination node server storage is within server handling ability,Be conducive to improve the treatment effeciency and Treatment Stability of declaration form traffic table.

Description

A kind of data processing method, device, computer equipment and storage medium
Technical field
The present invention relates to field of computer technology more particularly to a kind of data processing method, device, computer equipment and deposit Storage media.
Background technique
With the development and the improvement of people's living standards of society, more and more people begin to focus on various insurance businesses, The service surface that the declaration form business of insurance field is related to is wider, records more and more, declaration form business with inside declaration form traffic table Data volume in table becomes abnormal huge.
If database purchase has mass data, when carrying out data processing task, database can face very big pressure, this So that needing to take a significant amount of time, and be easy to cause system crash in the storage and processing to declaration form traffic table, guarantor is caused The low problem low with stability of single traffic table treatment effeciency, this wants basic calculation frame storage system in scale and system Ask etc. proposes higher challenge.
For this problem, current Main is to carry out depositing for declaration form traffic table using the extending transversely of database Storage, but database also has the bottleneck of oneself when extension, and performance boost extending transversely itself is no more than 10% or so, to solution Certainly declaration form traffic table is due to the data volume problem that treatment effeciency is low, stability is low caused greatly, and the effect is unsatisfactory.
Summary of the invention
Based on this, it is necessary in view of the above technical problems, provide one kind be conducive to improve declaration form traffic table treatment effeciency and Data processing method, device, computer equipment and the storage medium of Treatment Stability.
A kind of data processing method, including:
Receive the declaration form traffic table that client is sent, wherein the declaration form traffic table includes a plurality of data record and every The identification information of the data record;
Data fragmentation is carried out to the data record in the declaration form traffic table using preset sharding method, obtains P A target service table, wherein P is positive integer;
Each node server in node server set is detected using preset detection mechanism, obtains node The state of server, and the node server that the state is activation is determined as destination node server;
For each destination node server, the use state of the destination node server is obtained, and according to described Use state calculates the state score of the destination node server;
The task allocation proportion of each destination node server is calculated according to the state score;
According to the task allocation proportion, the P target service tables are distributed in the destination node server and are deposited Storage;
Establish the mapping relations between the identification information, the target service table and the destination node server.
A kind of data processing equipment, including:
Data reception module, for receiving the declaration form traffic table of client transmission, wherein the declaration form traffic table includes more The identification information of data record and every data record;
Data fragmentation module, for use preset sharding method to the data record in the declaration form traffic table into Row data fragmentation obtains P target service table, wherein P is positive integer;
Node detecting module, for using preset detection mechanism to each node server in node server set It is detected, obtains the state of node server, and the node server that the state is activation is determined as destination node clothes Business device;
Score value computing module obtains making for the destination node server for being directed to each destination node server With state, and calculate according to the use state state score of the destination node server;
Ratio-dependent module, the task for calculating each destination node server according to the state score are distributed Ratio;
Data allocation module, for according to the task allocation proportion, the P target service tables to be distributed to the mesh It is stored in mark node server;
Module is established in mapping, for establishing the identification information, the target service table and the destination node service Mapping relations between device.
A kind of computer equipment, including memory, processor and storage are in the memory and can be in the processing The computer program run on device, the processor realize the step of above-mentioned data processing method when executing the computer program Suddenly.
A kind of computer readable storage medium, the computer-readable recording medium storage have computer program, the meter The step of calculation machine program realizes above-mentioned data processing method when being executed by processor.
Above-mentioned data processing method, device, computer equipment and storage medium receive the declaration form business that client is sent Table carries out data fragmentation to the declaration form traffic table using preset sharding method, P target service table is obtained, using explorer System detects each node server in node server set, the state of obtained node server, and obtains shape State is the node server of activation as destination node server, for each destination node server, according to the destination node The use state of server calculates state score, and obtains the task distribution ratio of the destination node server according to state score Example, and then according to task allocation proportion, P target service table is distributed in these destination node servers and is stored, and is based on Data Identification establishes the mapping relations between declaration form traffic table, target service table and destination node server, so that declaration form business Data in table are assigned in each destination node server and are stored in the way of load balancing of having the right, and guarantee each The data volume of destination node server storage is within server handling ability, is conducive to the processing effect for improving declaration form traffic table Rate and Treatment Stability.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention Example, for those of ordinary skill in the art, without any creative labor, can also be according to these attached drawings Obtain other attached drawings.
Fig. 1 is the application environment schematic diagram of data processing method provided in an embodiment of the present invention;
Fig. 2 is the implementation flow chart of data processing method provided in an embodiment of the present invention;
Fig. 3 is the implementation flow chart of step S2 in data processing method provided in an embodiment of the present invention;
Fig. 4 is the implementation flow chart of step S4 in data processing method provided in an embodiment of the present invention;
Fig. 5 is the implementation flow chart that mapping relations update in data processing method provided in an embodiment of the present invention;
Fig. 6 is to be handled to obtain implementing result to declaration form traffic table in data processing method provided in an embodiment of the present invention Implementation flow chart;
Fig. 7 is the schematic diagram of data processing equipment provided in an embodiment of the present invention;
Fig. 8 is the schematic diagram of computer equipment provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts Example, shall fall within the protection scope of the present invention.
Referring to Fig. 1, Fig. 1 shows the application environment of data processing method provided in an embodiment of the present invention.The data processing Method is applied in the data processing scene of the declaration form traffic table of insurance field.The data processing scene includes center service Device, node server and client, wherein between central server and node server, between central server and client It is attached by network, user sends declaration form traffic table to central server by client, and central server is by declaration form Traffic table is divided into multiple target service tables, and multiple target service tables are distributed to node server, and node server receives simultaneously Target service table is stored, client specifically can be, but not limited to be mobile phone, tablet computer, personal computer Intelligent terminals such as (PersonalComputer, PC), central server and node serve implement body can use independent clothes The server cluster of business device or multiple servers composition is realized.
Referring to Fig. 2, being applied in this way Fig. 2 shows a kind of data processing method provided in an embodiment of the present invention in Fig. 1 In central server for be illustrated, details are as follows:
S1:Receive the declaration form traffic table that client is sent, wherein declaration form traffic table includes a plurality of data record and every number According to the identification information of record.
Specifically, central server receives client by the network transmission protocol and sends declaration form traffic table.
Wherein, declaration form traffic table is a kind of traffic table for insurance field comprising a plurality of data record and every number According to the identification information of record, the data record of declaration form traffic table is more, and can increase with the time, is continuously increased data record. Declaration form traffic table can be needed according to actual business to formulate, for example, the data record that certain declaration form traffic table includes is:Data Mark, insurer's title, insured amount, insurance period and status etc..
Wherein, the network transmission protocol includes but is not limited to:Internet Control Message agreement (Internet Control Message Protocol, ICMP), address resolution protocol (ARP Address Resolution Protocol, ARP) and text Part transport protocol (File Transfer Protocol, FTP) etc..
S2:Data fragmentation is carried out to the data record in declaration form traffic table using preset sharding method, obtains P target Traffic table, wherein P is positive integer.
Specifically, data fragmentation is carried out to the data record in declaration form traffic table by preset sharding method, will protected Single traffic table splits into P target service table, and the data record in each target service table has certain relevance, is conducive to The optimization of follow-up data record storage and the raising of follow-up data treatment effeciency.
Wherein, preset sharding method can be by, come fragment, be also possible to according to dimension using modulus algorithm come into Row fragment can also be the function of the partition table provided according to database to carry out fragment, and specific sharding method can be according to reality The needs of border application are configured, and are not particularly limited herein.
Preferably, preset sliced fashion used in the embodiment of the present invention is by dimension fragment and partition table sliced fashion In conjunction with.
S3:Each node server in node server set is detected using preset detection mechanism, is obtained The state of node server, and the node server that state is activation is determined as destination node server.
Specifically, by preset detection mechanism, each node server in node server set is detected, It finds state and is the node server of activation, and the node server is determined as destination node server.
Wherein, preset detection mechanism refers to tasting to each node server progress availability using probe timing Examination, specific embodiment, which can be, sends ping curl order to each node server by central server, such as The result that fruit returns is normal, then illustrates that the node server state illustrates the node server if returning to exception for activation State is delay machine.
S4:For each destination node server, the use state of the destination node server is obtained, and according to using shape State calculates the state score of the destination node server.
Specifically, each destination node server due to its own hardware configuration it is different, operation conditions is different, meeting Cause its performance to have differences, therefore, before target service table is distributed to destination node server, needs first according to target The use state of node server determines the state score of the destination node server, and then determine the data record of distribution Quantity.
Wherein, some parameter informations when use state refers to the operation of destination node server, these parameters include but not It is limited to:Central processing unit model, memory model and disk space parameter etc..
Wherein, referred to according to the state score that use state calculates the destination node server through currently used state institute The respective weights of the various parameters and pre-set various parameters that are related to obtain the state point of destination node server Value.
S5:The task allocation proportion of each destination node server is calculated according to state score.
Specifically, after the state score for calculating each destination node server, by the shape of all destination node servers State score value adds up, and obtains the sum of all destination node server state score values, and then be directed to each destination node server, The state score of the destination node server and the ratio of the sum of all destination node server state score values are calculated, as the mesh Mark the task allocation proportion of node server.
For example, in a specific embodiment, there are two destination node servers, respectively first object node serve Device and the second destination node server, the state score of first object node server are 3, the shape of the second destination node server State score value is 7, and according to foregoing description, the task allocation proportion that first object node server is calculated is 30%, the first mesh The task allocation proportion for marking node server is 70%.
S6:According to task allocation proportion, P target service table is distributed in destination node server and is stored.
Specifically, according to task allocation proportion obtained in step S5, the data record in P target service table is distributed To destination node server, each destination node server stores the data record after receiving the data record.
It is worth noting that lower target service table of ordinary circumstance is stored in a destination node server, if mesh It is more to mark data record in traffic table, when can not directly distribute according to task allocation proportion, by the data in the target service table Record, which is distributed in two or more destination node servers, to be stored.
S7:Establish the mapping relations between identification information, target service table and destination node server.
Specifically, target service table is being distributed in destination node server after storage, central server establishes mark Know the mapping relations between information, target service table and destination node server, when so as to subsequent progress data processing, energy The destination node server being quickly found out where the corresponding identification information of data record.
For example, in a specific embodiment, please referring to table one, table one shows identification information, target service table, and Mapping relations between destination node server, wherein declaration form number is the identification information of data record.
Table one
Declaration form number Target service table Destination node server
1-40000 First object traffic table First object node server
40001-60000 Second target service table First object node server
60001-80000 Third target service table Second destination node server
80001-110000 4th target service table Second destination node server
110001-160000 5th target service table Third destination node server
160001-180000 5th target service table 4th destination node server
In the present embodiment, the declaration form traffic table that client is sent is received, using preset sharding method to the declaration form industry Business table carries out data fragmentation, obtains P target service table, is taken using detection mechanism to each node in node server set Business device detected, the state of obtained node server, and obtain state be activate node server as destination node Server calculates state score, and root according to the use state of the destination node server for each destination node server The task allocation proportion of the destination node server is obtained according to state score, and then according to task allocation proportion, by P target industry Business table is distributed in these destination node servers and stores, and be based on Data Identification, establish declaration form traffic table, target service table with Mapping relations between destination node server, so that the data in declaration form traffic table in the way of load balancing of having the right, are divided It is fitted in each destination node server and is stored, guarantee that the data volume of each destination node server storage is in server Within processing capacity, be conducive to the treatment effeciency and Treatment Stability that improve declaration form traffic table.
In one embodiment, as shown in figure 3, in step S2, i.e., using preset sharding method in declaration form traffic table Data record carries out data fragmentation, obtains P target service table, specifically comprises the following steps:
S21:Fragment is carried out to the data in declaration form traffic table according to preset dimension, generates interim fragment table.
Specifically, fragment is carried out to the data in declaration form traffic table according to preset dimension fragment, it will be in same range Relevant data record is assigned to together, and interim fragment table is generated.
Wherein, preset dimension can be according to time dimension, be also possible to region dimension, specifically can be according to practical feelings Condition is configured, and is not particularly limited herein.
S22:Using preset partition table generating mode, fragment is carried out to each interim fragment table, obtains P target service Table.
Specifically, the subregion table function carried by using database carries out fragment to each interim fragment table, obtains P Target service table.
Wherein, preset partition table generating mode refers to the function of establishing partition table of carrying using oracle database, Oracle database provides powerful subregion table function, in oracle database, if a table is set up as partition table, Logically or a complete table, only the data record in table is stored among multiple table spaces physically.
It is worth noting that interim fragment table is obtained by preset dimension in step S21, in these interim fragment tables Data record dimensional properties having the same, such as carry out by region dimension the data record in the interim fragment table that fragment obtains For the data of same region, this sharding method makes the interim fragment table in part there is a situation where that data record is more, for needle The relevance in table between data record both ensure that by way of establishing partition table to interim fragment table to such case, The interim fragment table that data record is more, data volume is big is made to be divided into multiple target service tables again.
In the present embodiment, fragment is carried out to the data in declaration form traffic table according to preset dimension, generates interim fragment Table, and preset partition table generating mode is used, fragment is carried out to each interim fragment table, obtains P target service table, so that Declaration form traffic table comprising mass data record is split into multiple target service tables, and the data note in each target service table Record has certain relevance, is conducive to the raising of the storage optimization and follow-up data treatment effeciency of follow-up data record.
In one embodiment, as shown in figure 4, in step S4, that is, it is directed to each destination node server, obtains the target section The use state of point server, and the state score of the destination node server is calculated according to use state, it specifically includes as follows Step:
S41:For each destination node server, the central processing unit model Q of the destination node server is obtained1, magnetic Disk space parameter Q2With memory model Q3
Specifically, the use state of each destination node server is not quite similar, i.e., different destination node servers Hardware configuration is different, wherein hardware configuration includes but is not limited to:Central processing unit model and memory model etc., current Disk space size is also different, and the use state of destination node server determines the storage of its data and processing capacity, because This needs first to obtain the use state of the destination node server for each destination node server, i.e. the destination server Central processing unit model, disk space parameter and memory model.
S42:The corresponding state score S of each destination node server is calculated according to following formula:
S=J1+Q2×J2+J3
Wherein, J1For central processing unit model Q1Corresponding preset weights, J2For the corresponding default power of disk space parameter Value, J3For memory model Q3Corresponding preset weights.
Specifically, for the central processing unit of different model, disk space parameter and memory model, it is respectively right to preset respectively The weight answered uses formula S=J after the use state for getting destination node server1+Q2×J2+J3Calculate target section The state score of point server.
For example, in a specific embodiment, in node server set, the model of central processing unit includes:E3- 1231v3, E5-2620v4 and E5-2680v2, corresponding preset weights are:0.1,0.3 and 0.6, memory model includes: KVR24N17D8/8-SP and KVR26N19S8/8, corresponding preset weights are 0.4 and 0.6, the preset threshold of disk space parameter It is 0.5, gets the model E5-2620v4 of the central processing unit of some destination node server, disk space parameter is 0.8 Terabyte (Terabyte, TB), memory model KVR24N17D8/8-SP, the state score of the destination node server are:S =0.3+0.8 × 0.5+0.4=1.1.
In the present embodiment, for each destination node server, the use state of the destination node server is obtained, is made Include but is not limited to state:Central processing unit model, disk space parameter and memory model etc., and then according to use state meter The state score of the destination node server is calculated, so that each destination node server obtains corresponding according to its use state State score, an important measurement index of the state score as subsequent load balancing of having the right, is conducive to central server root According to the use state of destination node server, the declaration form traffic table data within the destination node server handling ability are obtained Amount, and then the distribution of task is carried out to it, it ensure that the stability in data handling procedure.
In one embodiment, central server detects destination node server, in time to the destination node of delay machine The data record stored on server is shifted, as shown in figure 5, after step s 7, the data processing method further include as Lower step:
S81:Destination node server is detected using detection mechanism, obtains the state of destination node server.
Specifically, the mapping relations established between identification information, target service table and destination node server it Afterwards, destination node server is detected by the explorer formulation phase, obtains the state of destination node server, alternatively, When receiving processing request of the client to declaration form traffic table, detection mechanism is activated to detect destination node server, Obtain the state of destination node server.
S82:If the state of destination node server obtains the destination node server according to mapping relations for delay machine Corresponding target service table.
Specifically, when step S81 monitors that the state of destination node server is delay machine, pass through identification information, target It is corresponding to obtain the destination node server that the state is delay machine for mapping relations between traffic table and destination node server Target service table.
S83:By state be delay machine the corresponding target service table of destination node server by way of load balancing, point It is dealt into other destination node servers in node server set.
It specifically, is the destination node of delay machine by state according to step S3 to the method for the step S6 load balancing described The corresponding target service table of server is distributed in other destination node servers in node server set, avoids carrying out When data processing, due to partial target node server delay machine and cause partial data to be lost, guarantee declaration form traffic table in Data record is effectively stored, and the stability for improving data processing is conducive to.
S84:The destination node server that state is delay machine is removed from node server set.
Specifically, central server and node server set have been built into a distributed type assemblies, are collected by distribution The reliable coordination system of group removes the destination node server that state is delay machine.
Wherein, reliable coordination system specifically can be ZooKeeper system, and ZooKeeper is one distributed, open The distributed application program coordination service of source code, it provides Consistency service for Distributed Application, and the function of providing includes:Configuration Maintenance, domain name service, distributed synchronization, group service etc..When receiving event information, for example, connection time-out, node data change, Child node changes, and corresponding behavior can be called to handle event.
S85:Update the mapping relations between identification information, target service table and destination node server.
Specifically, in mapping relations, the state in step S82 that finds is the destination node server of delay machine, and deleting should State is the corresponding relationship between the destination node server and identification information, target service table of delay machine, and then according to step S83 Described in corresponding relationship between the target service table that increases newly in destination node server, with the destination node server, it is right Mapping relations between identification information, target service table and destination node server are updated.
In the present embodiment, destination node server is detected using detection mechanism, obtains destination node server State, and then by state be delay machine the corresponding target matrix of destination node server be re-distributed to node server collection In other destination node servers in conjunction, and update between identification information, target service table and destination node server Mapping relations make, since the delay machine of destination node server causes partial data to be lost, to be conducive to mention when carrying out data processing The stability of high data processing.
In one embodiment, central server driving destination node server executes the process instruction of declaration form traffic table, and The result of return is summarized to obtain final implementing result, as shown in fig. 6, after step s 7, the data processing method It further include following steps:
S91:If receive client transmission for declaration form traffic table processing request, obtain processing request in include Target data mark and target data identify corresponding process instruction.
Specifically, client and central server are based on transmission control protocol (Transmission Control Protocol, TCP) network communication is carried out, when the processing for declaration form traffic table for monitoring that client is sent is requested, obtain The identification information for the data record for including in processing request, identifies as target data, and obtains target data mark and correspond to Process instruction.
Wherein, TCP provides a kind of connection-oriented, reliable byte stream service.Mean that two use TCP towards connection Application (client and central server) TCP connection must be first established before packet exchanging data with one another.In a TCP In connection, only two sides communicate with one another.
It should be understood that can have different process instructions for different target data marks, the processing that client is sent is asked In asking, it may include multiple target data marks and each target data identify corresponding process instruction.
For example, in a specific embodiment, the identification information for the data record for including in the processing request got is Number 50001 identifies, the corresponding process instruction got is to number 100000 using these identification informations as target data The record of insuring for obtaining and increasing newly in corresponding data record in this month is identified in target data.
S92:According to mapping relations, obtains target data and identify corresponding destination node server, taken as data processing Business device.
Specifically, it according to the mapping relations between identification information, target service table and destination node server, obtains It is stored with the destination node server of target data identification information, and using the destination node server as data processing service Device.
It is worth noting that target data, which identifies corresponding destination node server, can be one, be also possible to it is multiple, Specific number determines according to the actual situation.
By taking the target data mark obtained in step S91 as an example, according to mapping relations, number 50001 is obtained to number 60000 target data identifies corresponding data record and is stored in first object node server, obtains number 60001 to number 100000 target data identifies corresponding data record and is stored in the second destination node server, by first object node serve Device is as the first data processing server, using the second destination node server as the second data processing server.
S93:Process instruction is sent to data processing server.
Specifically, the process instruction got in step S91 is sent to corresponding data processing service by central server Device, processing server executes the process instruction after receiving process instruction, and obtains implementing result.
By taking the first data processing server and the second data processing server in step S92 as an example, central server to The two processing servers are sent in target data and identify the record of insuring for obtaining that this month increases newly in corresponding data record Instruction, the first data processing server is after receiving the instruction, from the number 50001 of storage to the number of targets of number 60000 According to the record of insuring for obtaining and increasing newly in corresponding data record in this month is identified, amount to 300, the second data processing server exists After receiving the instruction, identifies in corresponding data record and obtain from the target data of number 60001 to the number 100000 of storage The record of insuring that this month is newly-increased is taken, amounts to 1100.
S94:If receiving the implementing result of data processing server return, according to mapping relations, by implementing result into Row merges, and obtains objective result.
Specifically, when receiving the implementing result that one or more processing servers return, all implementing results are pressed It according to mapping relations, merges, obtains final objective result.
By taking the implementing result that the first data processing server in step S93 and the second data processing server obtain as an example, 1100 that the 300 datas record that first data processing server sends over is sended over the second data processing server Data record is summarized, and this 1400 data is recorded and is ranked up according to the identification information of data record, will be sorted Data record afterwards is as final objective result.
S95:Objective result is sent to client.
Specifically, the obtained objective result of step S94 is sent to client.
In the present embodiment, when the processing for declaration form traffic table for receiving client transmission is requested, acquisition processing The target data mark and target data for including in request identify corresponding process instruction, and according to mapping relations, obtain target The corresponding destination node server of Data Identification, is sent to data processing as data processing server, and then by process instruction Server is handled, and when receiving the implementing result of data processing server return, implementing result is merged, is obtained Objective result, and objective result is sent to client so that when needing to carry out data processing to declaration form traffic table, pass through by The identification information of data record to be treated and corresponding process instruction are sent to corresponding destination node server and execute, The plenty of time has been saved, the efficiency of data processing is improved.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit It is fixed.
In one embodiment, a kind of data processing equipment is provided, in the data processing equipment and above-described embodiment at data Reason method corresponds.As shown in fig. 7, the data processing equipment includes data reception module 10, data fragmentation module 20, node Module 70 is established in detecting module 30, score value computing module 40, ratio-dependent module 50, data allocation module 60 and mapping.Each function Detailed description are as follows for energy module:
Data reception module 10, for receiving the declaration form traffic table of client transmission, wherein declaration form traffic table includes a plurality of The identification information of data record and every data record;
Data fragmentation module 20, for using preset sharding method to carry out data to the data record in declaration form traffic table Fragment obtains P target service table, wherein P is positive integer;
Node detecting module 30, for using preset detection mechanism to each node serve in node server set Device is detected, and obtains the state of node server, and the node server that state is activation is determined as destination node service Device;
Score value computing module 40 obtains the use of the destination node server for being directed to each destination node server State, and calculate according to use state the state score of the destination node server;
Ratio-dependent module 50, for calculating the task allocation proportion of each destination node server according to state score;
Data allocation module 60, for according to task allocation proportion, P target service table to be distributed to destination node service It is stored in device;
Module 70 is established in mapping, for establishing reflecting between identification information, target service table and destination node server Penetrate relationship.
Further, data fragmentation module 20 includes:
Fragment table generation unit 21 is generated for carrying out fragment to the data in declaration form traffic table according to preset dimension Interim fragment table;
Traffic table acquiring unit 22 divides each interim fragment table for using preset partition table generating mode Piece obtains P target service table.
Further, score value computing module 40 includes:
Parameter acquiring unit 41 obtains the center of the destination node server for being directed to each destination node server Processor model Q1, disk space parameter Q2With memory model Q3
Score value computing unit 42, for calculating the corresponding state score S of each destination node server according to following formula:
S=J1+Q2×J2+J3
Wherein, J1For central processing unit model Q1Corresponding preset weights, J2For the corresponding default power of disk space parameter Value, J3For memory model Q3Corresponding preset weights.
Further, which further includes:
State detection module 81 obtains destination node for detecting using detection mechanism to destination node server The state of server;
Delay machine processing module 82, if the state for destination node server is delay machine, according to mapping relations, obtaining should The corresponding target service table of destination node server;
Data migration module 83, it is negative for passing through the corresponding target service table of destination node server that state is delay machine Balanced mode is carried, is distributed in other destination node servers in node server set;
Node remove module 84, for removing the destination node server that state is delay machine from node server set;
Map updating module 85, for updating reflecting between identification information, target service table and destination node server Penetrate relationship.
Further, which further includes:
Request module 91 obtains if the processing for declaration form traffic table for receiving client transmission is requested The target data mark for including in processing request and target data is taken to identify corresponding process instruction;
Node selection module 92, for obtaining target data and identifying corresponding destination node server according to mapping relations, As data processing server;
Instruction sending module 93, for process instruction to be sent to data processing server;
As a result it is closed if the implementing result for receiving data processing server return according to mapping determining module 94 System, implementing result is merged, objective result is obtained;
As a result sending module 95, for objective result to be sent to client.
Specific about data processing equipment limits the restriction that may refer to above for data processing method, herein not It repeats again.Modules in above-mentioned data processing equipment can be realized fully or partially through software, hardware and combinations thereof.On Stating each module can be embedded in the form of hardware or independently of in the processor in computer equipment, can also store in a software form In memory in computer equipment, the corresponding operation of the above modules is executed in order to which processor calls.
In one embodiment, a kind of computer equipment is provided, which can be server, internal junction Composition can be as shown in Figure 8.The computer equipment include by system bus connect processor, memory, network interface and Database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The memory packet of the computer equipment Include non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer program and data Library.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating The database of machine equipment is for storing declaration form traffic table and mapping relations.The network interface of the computer equipment is used for and outside Terminal passes through network connection communication.To realize a kind of data processing method when the computer program is executed by processor.
In one embodiment, a kind of computer equipment is provided, including memory, processor and storage are on a memory And the computer program that can be run on a processor, processor realize above-described embodiment data processing side when executing computer program The step of method, such as step S1 shown in Fig. 2 to step S7.Alternatively, processor realizes above-mentioned implementation when executing computer program The function of each module/unit of example data processing equipment, such as module shown in Fig. 7 10 is to module 70.To avoid repeating, here It repeats no more.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated Machine program realizes the step of above-described embodiment data processing method when being executed by processor, alternatively, computer program is by processor The function of each module/unit of above-described embodiment data processing equipment is realized when execution, to avoid repeating, which is not described herein again.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, To any reference of memory, storage, database or other media used in each embodiment provided by the present invention, Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations;Although referring to aforementioned reality Applying example, invention is explained in detail, those skilled in the art should understand that:It still can be to aforementioned each Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features;And these are modified Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all It is included within protection scope of the present invention.

Claims (10)

1. a kind of data processing method, which is characterized in that the data processing method includes:
Receive the declaration form traffic table that client is sent, wherein the declaration form traffic table includes described in a plurality of data record and every The identification information of data record;
Data fragmentation is carried out to the data record in the declaration form traffic table using preset sharding method, obtains P mesh Mark traffic table, wherein P is positive integer;
Each node server in node server set is detected using preset detection mechanism, obtains node serve The state of device, and the node server that the state is activation is determined as destination node server;
For each destination node server, the use state of the destination node server is obtained, and according to the use The state score of the state computation destination node server;
The task allocation proportion of each destination node server is calculated according to the state score;
According to the task allocation proportion, the P target service tables are distributed in the destination node server and are stored;
Establish the mapping relations between the identification information, the target service table and the destination node server.
2. data processing method as described in claim 1, which is characterized in that described to use preset sharding method to the guarantor Single traffic table carries out data fragmentation, and obtaining P target service table includes:
Fragment is carried out to the data in the declaration form traffic table according to preset dimension, generates interim fragment table;
Using preset partition table generating mode, fragment is carried out to each interim fragment table, obtains the P target services Table.
3. data processing method as described in claim 1, which is characterized in that described to be directed to each destination node service Device obtains the use state of the destination node server, and the shape of the destination node server is calculated according to the use state State score value includes:
For each destination node server, the central processing unit model Q of the destination node server is obtained1, disk space parameter Q2With memory model Q3
The corresponding state score S of each destination node server is calculated according to following formula:
S=J1+Q2×J2+J3
Wherein, J1For the central processing unit model Q1Corresponding preset weights, J2It is corresponding default for the disk space parameter Weight, J3For the memory model Q3Corresponding preset weights.
4. data processing method as described in claim 1, which is characterized in that establish the identification information, the mesh described After marking the mapping relations between traffic table and the node server, the data processing method further includes:
The destination node server is detected using the detection mechanism, obtains the state of destination node server;
If the state of the destination node server obtains the destination node server according to the mapping relations for delay machine Corresponding target service table;
By the state be delay machine the corresponding target service table of destination node server by way of load balancing, be distributed to In other destination node servers in node server set;
The destination node server that the state is delay machine is removed from the node server set;
Update the mapping relations between the identification information, the target service table and the destination node server.
5. such as the described in any item data processing methods of Claims 1-4, which is characterized in that establish the mark letter described After mapping relations between breath, the target service table and the node server, the data processing method further includes:
If receiving the processing for the declaration form traffic table that the client is sent to request, obtain in the processing request The target data mark for including and the target data identify corresponding process instruction;
According to the mapping relations, obtains the target data and identify corresponding destination node server, taken as data processing Business device;
The process instruction is sent to the data processing server;
If receiving the implementing result that the data processing server returns, according to the mapping relations, by implementing result into Row merges, and obtains objective result;
The objective result is sent to the client.
6. a kind of data processing equipment, which is characterized in that the data processing equipment includes:
Data reception module, for receiving the declaration form traffic table of client transmission, wherein the declaration form traffic table includes a plurality of number According to the identification information of record and every data record;
Data fragmentation module, for being counted using preset sharding method to the data record in the declaration form traffic table According to fragment, P target service table is obtained, wherein P is positive integer;
Node detecting module, for being carried out using preset detection mechanism to each node server in node server set Detection obtains the state of node server, and the node server that the state is activation is determined as destination node server;
Score value computing module, for being directed to each destination node server, obtain the destination node server uses shape State, and calculate according to the use state state score of the destination node server;
Ratio-dependent module, for calculating the task distribution ratio of each destination node server according to the state score Example;
Data allocation module, for according to the task allocation proportion, the P target service tables to be distributed to the target section It is stored in point server;
Mapping establish module, for establish the identification information, the target service table and the destination node server it Between mapping relations.
7. data processing equipment as claimed in claim 6, which is characterized in that the data fragmentation module includes:
Fragment table generation unit, for carrying out fragment to the data in the declaration form traffic table according to preset dimension, generation is faced When fragment table;
Traffic table acquiring unit carries out fragment to each interim fragment table for using preset partition table generating mode, Obtain the P target service tables.
8. data processing equipment as claimed in claim 6, which is characterized in that the score value computing module includes:
Parameter acquiring unit obtains the centre of the destination node server for being directed to each destination node server Manage type Q1, disk space parameter Q2With memory model Q3
Score value computing unit, for calculating the corresponding state score S of each destination node server according to following formula:
S=J1+Q2×J2+J3
Wherein, J1For the central processing unit model Q1Corresponding preset weights, J2It is corresponding default for the disk space parameter Weight, J3For the memory model Q3Corresponding preset weights.
9. a kind of computer equipment, including memory, processor and storage are in the memory and can be in the processor The computer program of upper operation, which is characterized in that the processor realized when executing the computer program as claim 1 to Any one of 5 data processing methods.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists In the step of realization data processing method as described in any one of claim 1 to 5 when the computer program is executed by processor Suddenly.
CN201810585017.6A 2018-06-08 2018-06-08 A kind of data processing method, device, computer equipment and storage medium Pending CN108920272A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810585017.6A CN108920272A (en) 2018-06-08 2018-06-08 A kind of data processing method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810585017.6A CN108920272A (en) 2018-06-08 2018-06-08 A kind of data processing method, device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN108920272A true CN108920272A (en) 2018-11-30

Family

ID=64420254

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810585017.6A Pending CN108920272A (en) 2018-06-08 2018-06-08 A kind of data processing method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN108920272A (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857549A (en) * 2019-01-04 2019-06-07 平安科技(深圳)有限公司 Image processing method, system, equipment and medium based on load balancing
CN110428019A (en) * 2019-08-09 2019-11-08 绵阳德川鸿丰环保科技有限公司 Intelligent garbage classification method and modularization intelligent garbage classification processing system
CN110489416A (en) * 2019-07-23 2019-11-22 中国平安财产保险股份有限公司 A kind of information storage means and relevant device based on data processing
CN111209263A (en) * 2020-01-14 2020-05-29 中国建设银行股份有限公司 Data storage method, device, equipment and storage medium
CN111241143A (en) * 2020-01-09 2020-06-05 湖南华博信息技术有限公司 Distributed calculation method and system for water supply amount and water fee
CN111597056A (en) * 2020-05-18 2020-08-28 北京思特奇信息技术股份有限公司 Distributed scheduling method, system, storage medium and device
CN111603765A (en) * 2020-04-07 2020-09-01 广州多益网络股份有限公司 Server distribution method, system and storage medium
CN112015731A (en) * 2020-07-08 2020-12-01 广州乐摇摇信息科技有限公司 Information distribution method and system based on multi-terminal equipment
CN112291292A (en) * 2019-07-26 2021-01-29 浙江宇视科技有限公司 Data storage method, device, equipment and medium
CN112468548A (en) * 2020-11-13 2021-03-09 苏州智加科技有限公司 Data processing method, device, system, server and readable storage medium
CN112671905A (en) * 2020-12-23 2021-04-16 广州三七互娱科技有限公司 Service scheduling method, device and system
CN112667656A (en) * 2020-12-07 2021-04-16 南方电网数字电网研究院有限公司 Transaction data processing method and device, computer equipment and storage medium
CN112769957A (en) * 2021-02-05 2021-05-07 上海哔哩哔哩科技有限公司 Point-to-point network scheduling method and system
CN113110803A (en) * 2021-04-19 2021-07-13 浙江中控技术股份有限公司 Data storage method and device
CN113242302A (en) * 2021-05-11 2021-08-10 鸬鹚科技(深圳)有限公司 Data access request processing method and device, computer equipment and medium
CN113242283A (en) * 2021-04-29 2021-08-10 西安点告网络科技有限公司 Server dynamic load balancing method, system, equipment and storage medium
CN113407108A (en) * 2020-03-16 2021-09-17 北京沃东天骏信息技术有限公司 Data storage method and system
CN113703826A (en) * 2021-07-29 2021-11-26 北京三快在线科技有限公司 Method, apparatus, device and storage medium for responding to data processing request
CN114253733A (en) * 2021-12-24 2022-03-29 苏州浪潮智能科技有限公司 Memory management method and device, computer equipment and storage medium
CN115114012A (en) * 2021-08-12 2022-09-27 腾讯科技(深圳)有限公司 Task allocation method and device, electronic equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104158904A (en) * 2014-08-28 2014-11-19 重庆邮电大学 Cloud-assisted cooperative downloading method of mobile P2P network
CN105912601A (en) * 2016-04-05 2016-08-31 国电南瑞科技股份有限公司 Partition storage method for distributed real-time memory database of energy management system
WO2017008697A1 (en) * 2015-07-10 2017-01-19 努比亚技术有限公司 Data transmission method and device, multi-channel routing method, and user equipment

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104158904A (en) * 2014-08-28 2014-11-19 重庆邮电大学 Cloud-assisted cooperative downloading method of mobile P2P network
WO2017008697A1 (en) * 2015-07-10 2017-01-19 努比亚技术有限公司 Data transmission method and device, multi-channel routing method, and user equipment
CN105912601A (en) * 2016-04-05 2016-08-31 国电南瑞科技股份有限公司 Partition storage method for distributed real-time memory database of energy management system

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857549A (en) * 2019-01-04 2019-06-07 平安科技(深圳)有限公司 Image processing method, system, equipment and medium based on load balancing
CN110489416B (en) * 2019-07-23 2024-02-09 中国平安财产保险股份有限公司 Information storage method based on data processing and related equipment
CN110489416A (en) * 2019-07-23 2019-11-22 中国平安财产保险股份有限公司 A kind of information storage means and relevant device based on data processing
CN112291292A (en) * 2019-07-26 2021-01-29 浙江宇视科技有限公司 Data storage method, device, equipment and medium
CN112291292B (en) * 2019-07-26 2023-04-07 浙江宇视科技有限公司 Data storage method, device, equipment and medium
CN110428019A (en) * 2019-08-09 2019-11-08 绵阳德川鸿丰环保科技有限公司 Intelligent garbage classification method and modularization intelligent garbage classification processing system
CN111241143A (en) * 2020-01-09 2020-06-05 湖南华博信息技术有限公司 Distributed calculation method and system for water supply amount and water fee
CN111209263A (en) * 2020-01-14 2020-05-29 中国建设银行股份有限公司 Data storage method, device, equipment and storage medium
CN113407108A (en) * 2020-03-16 2021-09-17 北京沃东天骏信息技术有限公司 Data storage method and system
CN111603765A (en) * 2020-04-07 2020-09-01 广州多益网络股份有限公司 Server distribution method, system and storage medium
CN111597056A (en) * 2020-05-18 2020-08-28 北京思特奇信息技术股份有限公司 Distributed scheduling method, system, storage medium and device
CN112015731A (en) * 2020-07-08 2020-12-01 广州乐摇摇信息科技有限公司 Information distribution method and system based on multi-terminal equipment
CN112468548A (en) * 2020-11-13 2021-03-09 苏州智加科技有限公司 Data processing method, device, system, server and readable storage medium
CN112667656A (en) * 2020-12-07 2021-04-16 南方电网数字电网研究院有限公司 Transaction data processing method and device, computer equipment and storage medium
CN112671905A (en) * 2020-12-23 2021-04-16 广州三七互娱科技有限公司 Service scheduling method, device and system
CN112671905B (en) * 2020-12-23 2023-04-28 广州三七互娱科技有限公司 Service scheduling method, device and system
CN112769957A (en) * 2021-02-05 2021-05-07 上海哔哩哔哩科技有限公司 Point-to-point network scheduling method and system
CN112769957B (en) * 2021-02-05 2023-09-22 上海哔哩哔哩科技有限公司 Point-to-point network scheduling method and system
CN113110803B (en) * 2021-04-19 2022-10-21 浙江中控技术股份有限公司 Data storage method and device
CN113110803A (en) * 2021-04-19 2021-07-13 浙江中控技术股份有限公司 Data storage method and device
CN113242283A (en) * 2021-04-29 2021-08-10 西安点告网络科技有限公司 Server dynamic load balancing method, system, equipment and storage medium
CN113242302A (en) * 2021-05-11 2021-08-10 鸬鹚科技(深圳)有限公司 Data access request processing method and device, computer equipment and medium
CN113703826A (en) * 2021-07-29 2021-11-26 北京三快在线科技有限公司 Method, apparatus, device and storage medium for responding to data processing request
CN115114012A (en) * 2021-08-12 2022-09-27 腾讯科技(深圳)有限公司 Task allocation method and device, electronic equipment and storage medium
CN114253733A (en) * 2021-12-24 2022-03-29 苏州浪潮智能科技有限公司 Memory management method and device, computer equipment and storage medium
CN114253733B (en) * 2021-12-24 2024-01-12 苏州浪潮智能科技有限公司 Memory management method, device, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
CN108920272A (en) A kind of data processing method, device, computer equipment and storage medium
US11323347B2 (en) Systems and methods for social graph data analytics to determine connectivity within a community
US20210326885A1 (en) Method and Apparatus of Identifying a Transaction Risk
CN109729131B (en) Application request processing method and device and router
CN107368259A (en) A kind of method and apparatus that business datum is write in the catenary system to block
CN107395659A (en) A kind of method and device of service handling and common recognition
US9043317B2 (en) System and method for event-driven prioritization
WO2011047474A1 (en) Systems and methods for social graph data analytics to determine connectivity within a community
WO2011106897A1 (en) Systems and methods for conducting more reliable assessments with connectivity statistics
CN103631933B (en) Distributed duplication elimination system-oriented data routing method
CN107832407A (en) For generating the information processing method, device and readable storage medium storing program for executing of knowledge mapping
WO2020100108A1 (en) Systems and method for scoring entities and networks in a knowledge graph
CN110601922A (en) Method and device for realizing comparison experiment, electronic equipment and storage medium
CN111310784B (en) Resource data processing method and device
CN110827028A (en) Data acquisition and transaction system and method based on block chain
CN106326012B (en) Web application cluster caching utilizes method and system
CN110310129A (en) Risk Identification Method and its system
CN110661719A (en) Flow load balancing method and device
CN109377383A (en) Product data synchronous method, device, computer equipment and storage medium
CN111553594B (en) Resource allocation method, device, computer equipment and computer readable storage medium
CN110502549B (en) User data processing method and device, computer equipment and storage medium
US11442632B2 (en) Rebalancing of user accounts among partitions of a storage service
CN111598390B (en) Method, device, equipment and readable storage medium for evaluating high availability of server
CN107491265B (en) Method and device for distributing internet protocol IP disk
CN112559460B (en) File storage method, device, equipment and storage medium based on artificial intelligence

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination